Focus: Scalability, Performance Optimization & Data Integration
Experience: 8+ years in Data Engineering, including 3+ years of hands-on Azure Databricks experience.
Role Overview
We are looking for a highly experienced, hands-on Azure Databricks Data Engineer to build and optimize scalable, high-performance data pipelines for a large-scale healthcare analytics platform.
The engineer will integrate an existing Python-based data processing and enrichment engine with Azure Databricks, reuse existing transformations, and enable bidirectional data movement between the engine, Azure Data Lake, and Microsoft Fabric.
The primary focus is scalability, performance optimization, near-real-time processing, and efficient Azure resource utilization.
Key Responsibilities
Data Lake Integration: Build and optimize data pipelines between existing Python processing services, Azure Databricks, ADLS Gen2, and Microsoft Fabric using Bronze, Silver, and Gold architecture.
Spark Performance Optimization: Optimize PySpark and Spark SQL workloads involving billions of records, complex joins, large-table MERGE operations, data skew, shuffles, and unnecessary table scans.
Scalability & Resource Management: Design scalable multi-tenant pipelines with efficient cluster sizing, autoscaling, parallel processing, workload orchestration, and Azure compute quota management.
Streaming & Incremental Processing: Implement high-performance CDC, micro-batch, and incremental pipelines using Structured Streaming, Event Hubs, and Azure Data Factory. Support updates, deletes, retries, and replay.
Transformation Reuse: Integrate existing Python and SQL-based business transformations, ensuring data consistency, correctness, and minimal redevelopment.
Data Integration & Quality: Enable reliable bidirectional data flows between the existing processing engine and the lakehouse, with automated reconciliation, schema evolution, monitoring, and failure recovery.
Fabric Integration: Publish optimized and curated datasets to Microsoft Fabric for downstream analytics and reporting.
Required Technical Skills
Strong hands-on expertise in Azure Databricks, Apache Spark, PySpark, and advanced SQL.
Deep understanding of Spark internals, query execution plans, partitioning, clustering, joins, shuffles, memory optimization, and performance tuning.
Experience with Delta Lake, ADLS Gen2, Azure Data Factory, Azure Event Hubs, and Structured Streaming.
Proven experience designing multi-terabyte, multi-tenant, and high-volume data pipelines.
Strong knowledge of CDC, incremental processing, schema evolution, data reconciliation, and distributed processing.
Experience optimizing Databricks clusters, job orchestration, concurrency, and Azure compute costs.
Strong Python development and production troubleshooting skills.
Preferred Skills
Microsoft Fabric, OneLake, Apache Iceberg, and Unity Catalog.
Kafka, RabbitMQ, Debezium, and asynchronous data integration.
AWS-to-Azure cross-cloud integration.
Healthcare data processing and XML/JSON transformation experience.
Ideal Candidate
A strong, hands-on data engineer who has successfully optimized production-scale Databricks pipelines and can independently diagnose performance bottlenecks, improve processing speed, and design scalable, cost-efficient solutions.
Must be able to demonstrate real-world Spark/Databricks performance improvements with measurable results.
