Seeking an experienced Data Platform Manager to lead the design, development, and optimization of enterprise data platforms within Microsoft Azure. This position will oversee Azure Databricks architecture, data engineering, ETL/ELT pipelines, data governance, and cloud infrastructure integration. The ideal candidate combines hands-on expertise in Databricks, PySpark, SQL, and Azure data services with technical leadership experience managing data engineering teams and delivering scalable analytics solutions.
Lead, mentor, and provide technical direction to data engineers, including code reviews, development standards, and production troubleshooting.
Design, build, and maintain scalable enterprise data platforms using Azure Databricks and Microsoft Azure.
Architect and implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers).
Oversee the development of batch and real-time ETL/ELT pipelines using PySpark, Scala, SQL, and Apache Spark.
Integrate data from APIs, relational databases, streaming platforms, and other enterprise sources.
Develop and optimize data ingestion, transformation, enrichment, and validation processes.
Integrate Azure Databricks with Azure Data Factory, Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Synapse.
Implement Unity Catalog for data governance, access controls, data lineage, privacy, and security.
Optimize Databricks clusters, Spark jobs, query execution, and cloud resource utilization for performance and cost efficiency.
Support streaming data integrations using Apache Kafka and other distributed processing technologies.
Implement CI/CD pipelines and deployment automation using Azure DevOps.
Collaborate with data scientists and analysts to support machine learning models, analytics, and business intelligence initiatives.
Establish data engineering standards, quality controls, monitoring, and operational best practices.
Partner with business stakeholders to translate data requirements into scalable technical solutions.
Support cloud infrastructure security, networking, identity management, and access controls.
Bachelor's degree in Computer Science, Engineering, or a related field.
Five to seven or more years of hands-on experience in data engineering or data architecture.
Two to four years of direct experience with Azure Databricks and Azure cloud technologies.
Strong hands-on experience designing and implementing Azure Databricks Lakehouse architectures.
Advanced programming experience with Python, PySpark, SQL, Scala, and PowerShell.
Experience developing batch and streaming ETL/ELT data pipelines.
Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse.
Strong understanding of Apache Spark, Delta Lake, and distributed data processing.
Experience with relational SQL and NoSQL databases.
Knowledge of data modeling, schema design, and database architecture.
Experience with data governance, security, and Unity Catalog.
Experience optimizing Spark workloads, Databricks clusters, and cloud infrastructure.
Knowledge of Azure networking, VNETs, Key Vault, and identity management.
Experience with data orchestration, automation, and CI/CD processes.