Job Title: Azure Data Engineer
Design and implement robust Azure data platforms with comprehensive ETL testing expertise. Build scalable pipelines using Databricks, Data Lakes, and Synapse while leading data quality assurance through automated testing frameworks.
Location: Toronto, ON
Experience Level: Senior (5+ years Azure Data Engineering)
Role Summary
Execute end-to-end Azure data engineering solutions with specialized focus on ETL/ELT testing, data integration, and automated test frameworks. Leverage deep expertise in Azure Databricks, Data Lakes, and Synapse Analytics to ensure production-grade data quality and reliability.
Key Responsibilities
Data Platform Development
- Architect ETL/ELT pipelines integrating diverse data sources into Azure Data Lake and Azure Synapse Analytics.
- Implement scalable data transformations using Azure Databricks (PySpark, Spark SQL, Scala notebooks).
- Design data warehousing solutions optimizing for analytics, reporting, and machine learning workloads.
Comprehensive Data Testing
- Develop ETL testing frameworks validating data integrity, completeness, accuracy, and performance.
- Create automated test scripts using SQL, Python, and Scala for pipeline validation.
- Implement cloud-based testing strategies specific to Azure Data Factory (ADF) pipelines and Databricks jobs.
Quality Assurance Leadership
- Write comprehensive test cases covering data ingestion, transformation logic, and output validation.
- Execute data testing across unit, integration, and end-to-end scenarios ensuring zero production defects.
- Perform performance testing and optimization of Spark jobs and ADF pipelines.
Required Technical Expertise
Technology Area | Core Skills |
Azure Platform | Azure Databricks, Azure Data Lake, Azure Synapse Analytics, Azure Data Factory (ADF) |
Programming | SQL (complex queries, performance tuning), Python (PySpark, pandas), Scala (Spark) |
Testing | ETL testing, ADF pipeline testing, automated test frameworks, data quality validation |
Data Engineering | ETL/ELT concepts, data integration, data warehousing, pipeline orchestration |
DevOps | Azure DevOps, CI/CD for data pipelines, Git, test automation |
Experience Profile
- Strong ETL concepts, data integration, data warehousing foundation
- Hands-on Azure Databricks (PySpark/Spark SQL/Scala notebooks)
- Proven data testing expertise (ETL testing, cloud testing)
- Automated test framework implementation experience
- SQL/Python/Scala proficiency for test script development
- ADF pipeline testing and validation experience
- Strong debugging/problem-solving across data platforms
Key Differentiators
- 5+ years Azure Data Engineering production experience
- Demonstrated test automation reducing manual validation by 70%+
- Experience preventing production data quality incidents
- Multiple successful ADF pipeline deployments to production
- Performance optimization case studies (before/after metrics)
Keywords: Azure Data Engineer, Toronto, Azure Databricks, Azure Data Lake, Azure Synapse Analytics, ETL testing, data testing, cloud testing, automated test frameworks, SQL testing, Python PySpark testing, Scala Spark testing, Azure Data Factory ADF testing, data pipeline testing, data quality validation, ETL concepts, data integration, data warehousing, test automation, debugging, problem solving, Azure DevOps, CI/CD data pipelines, production data quality, Spark performance testing, data validation frameworks
About VDart Group
VDart Group is a global leader in technology, product, and talent solutions, serving Fortune 500 clients in 13 countries. With over 4,000 professionals worldwide, we deliver innovation, operational excellence, and measurable outcomes across industries. Guided by our commitment to People, Purpose, and Planet, VDart is recognized with an EcoVadis Bronze Medal and as a UN Global Compact member, reflecting our dedication to sustainable practices.
Never miss an opportunity! Create an alert based on the job you applied for.