Lead Data Engineer Azure Data Platform (Python/PySpark)

Job Title: Lead Data Engineer Azure Data Platform (Python/PySpark)

Lead enterprise data transformation initiatives as a hands-on technical leader architecting scalable Azure data platforms. Mentor 5-10 person data engineering teams while driving pipeline optimization and business stakeholder alignment.

Toronto, ON, Canada

Key Responsibilities

Data Platform Architecture

• Architect end-to-end data pipelines using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Gen2 for enterprise-scale data processing.
• Develop production-grade PySpark transformations and Python ETL processes handling petabyte-scale datasets with complex business logic.
• Lead data modeling initiatives implementing star schemas, dimensional modeling, and SCD Type 2 for analytics and reporting platforms.

Team Leadership & Technical Excellence

• Mentor and lead data engineering teams of 5-10 engineers through code reviews, technical standards, and architectural decision-making.

• Implement enterprise data quality frameworks using Great Expectations, Azure Monitor, and Azure Purview for governance and compliance.

Real-Time & Advanced Analytics

• Design real-time streaming solutions leveraging Azure Event Hubs, Stream Analytics, and Delta Lake for low-latency analytics use cases.

• Optimize Spark jobs for cost-efficiency, performance, and scalability across Azure Databricks clusters.

Required Technical Expertise

Leadership Requirements

• Lead developer experience managing data engineering teams through complete SDLC with demonstrated success mentoring junior engineers and driving technical excellence.
• Executive stakeholder communication: Translate complex technical concepts into business value during requirement sessions and roadmap presentations.
• Agile/Scrum mastery: Lead sprint ceremonies using JIRA, Confluence, and cross-functional collaboration with data scientists, analysts, and business teams.

Experience Profile

• 7+ years data engineering experience including 5+ years hands-on Azure platform expertise
• Multiple long-term enterprise projects (1.5+ years each) building production data platforms
• Hands-on Python and PySpark demonstrated across 3+ major projects with complex transformations

Keywords: Lead Data Engineer, Azure Data Engineer, Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Gen2, PySpark, Python, Pandas, NumPy, Delta Lake, MLflow, star schema, dimensional modeling, SCD Type 2, data quality, Great Expectations, Azure Purview, Airflow MWAA, Databricks Workflows, Spark SQL, T-SQL, Terraform, ARM templates, Azure DevOps, CI/CD, Azure Monitor, Log Analytics, Monte Carlo, data pipeline, ETL, ELT, data lake, data warehouse, real-time streaming, Azure Event Hubs, Stream Analytics, team leadership, stakeholder management, Agile, Scrum, JIRA, Confluence, technical mentorship, code review, enterprise data platform

About VDart Group
VDart Group is a global leader in technology, product, and talent solutions, serving Fortune 500 clients in 13 countries. With over 4,000 professionals worldwide, we deliver innovation, operational excellence, and measurable outcomes across industries. Guided by our commitment to People, Purpose, and Planet, VDart is recognized with an EcoVadis Bronze Medal and as a UN Global Compact member, reflecting our dedication to sustainable practices.