Azure Databricks Data Engineer

Industry: Government / Public Health
Location: Indianapolis, IN
Work Arrangement: Hybrid
Duration: 12+ months, with potential extension or conversion to full-time
Interview: Webcam
Candidates: Local candidates only

Job Description

Seeking an experienced Azure Databricks Data Engineer to build, optimize, and maintain enterprise data pipelines and lakehouse solutions. The ideal candidate will have strong hands-on experience with PySpark, Spark SQL, Delta Lake, Azure Synapse Analytics, Azure Data Factory, and data governance.

Key Responsibilities

• Build batch and streaming pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.

• Implement medallion architecture (Bronze, Silver, Gold), incremental ingestion, and Delta Lake optimization.

• Develop and optimize dimensional data models, SQL queries, and warehouse workloads in Azure Synapse Analytics.

• Orchestrate workflows using Databricks Jobs, Azure Data Factory, and Synapse pipelines.

• Implement Unity Catalog permissions, data lineage, and security controls for sensitive data.

• Optimize Spark workloads, troubleshoot pipeline failures, and monitor production jobs.

• Implement CI/CD, Git version control, and secure access to Azure Data Lake Storage Gen2.

• Support data governance, technical documentation, and integration across enterprise systems.

Mandatory Technical Requirements

Candidates must demonstrate 4 years of experience in the following required areas:

• PySpark, Spark SQL, and Databricks notebooks, including transformations, joins, aggregations, and window functions.

• Delta Lake, ACID transactions, MERGE operations, schema evolution, and table optimization.

• Medallion architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.

• Unity Catalog, permissions, lineage, schemas, and storage access.

• Spark performance tuning, cluster sizing, partitioning, shuffle, skew, and caching.

• Azure Data Factory, Databricks Workflows/Jobs, dependencies, scheduling, and source control.

• Azure Synapse Analytics, dimensional modeling, star schemas, SCD Type 1/Type 2, and SQL performance tuning.

• ADLS Gen2, Parquet/Delta formats, Entra ID, managed identities, and Key Vault.

• Git-based CI/CD using Azure DevOps or GitHub.

• Pipeline monitoring, alerting, troubleshooting, and production support.

• Microsoft lakehouse patterns and interoperability between Azure Synapse and Databricks.

Education and Experience

Candidates must meet the stated education and experience criteria through a relevant master's degree, a related bachelor's degree with additional relevant experience, or an accepted equivalent experience/certification pathway.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.

• Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.

• Power BI connectivity to Synapse, including Import vs. DirectQuery and performance considerations.

• Experience handling sensitive health data and applying HIPAA-aligned security and governance controls.

Important

Local candidates only. Applicants must be available for the hybrid work arrangement in Indianapolis, IN. Please submit an updated resume highlighting hands-on Azure Databricks, PySpark, Delta Lake, and Azure Synapse experience.