Overview
On Site
Depends on Experience
Contract - W2
Contract - 1 Month(s)
Skills
Agile
Amazon Web Services
Analytical Skill
Business Data
Cloud Computing
Collaboration
Communication
Computer Science
Job Details
We are looking for Test Manager / Test Architect for our client in Eden Prairie, MN
Job Title: Test Manager / Test Architect
Job Type: Contract
Job Description:
- The Lead Data Engineer will be responsible for overseeing the design, testing, and implementation of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
- This role involves leading data migration and transformation initiatives, ensuring data quality, and mentoring junior engineers while collaborating with cross-functional teams to meet business data needs.
- Manage testing and validation of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into scalable data solutions.
- Ensure data quality, integrity, and governance across all migration and transformation processes.
- Mentor junior engineers and provide technical guidance within the team.
- Develop reusable frameworks and automation tools to streamline migration and data processing workflows.
- Stay up to date with emerging technologies, trends, and best practices in cloud data engineering and migration.
- Expertise in Python and PySpark for large-scale data processing.
- Strong understanding of distributed computing, data modeling, and ETL/ELT methodologies.
- Proficiency in Azure Databricks and Azure data services such as Azure Data Lake and Azure Synapse.
- Experience with CI/CD pipelines and version control tools such as Git and Azure DevOps.
- Excellent problem-solving, analytical, and communication skills.
- Proven ability to lead and manage multiple projects simultaneously.
- Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
- 12+ years of experience in data engineering with hands-on expertise in Python and PySpark.
- 3+ years of experience working with Azure Databricks and related Azure data services.
- Azure Data Engineer Associate or equivalent Azure certification.
- Experience with other cloud platforms such as AWS or Google Cloud Platform.
- Familiarity with Delta Lake, MLflow, and other tools within the Databricks ecosystem.
- Experience working in Agile or Scrum environments.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.