Databricks Lead - Chicago IL

Overview

Hybrid
Depends on Experience
Accepts corp to corp applications
Contract - W2
Contract - Independent
Contract - 12 Month(s)

Skills

Databricks
PySpark
Data Engineering
Data Governance
Data Processing

Job Details

Role: Databricks Lead

Locations: Chicago, IL (Hybrid Onsite)

Duration: 12+ Months Contract

Note: Candidate needs to be in the office 3-4 Days every week. Local or candidates from adjacent states only.

Key Responsibilities:

Platform Leadership:

  • Lead the design and deployment of large-scale data solutions on Databricks.
  • Establish best practices for notebook development, job orchestration, and cluster management.
  • Stay current with Databricks features and recommend improvements.

Data Engineering & Pipeline Development

  • Build robust batch and streaming data pipelines using PySpark, Delta Lake, and Structured Streaming.
  • Implement Medallion architecture (Bronze, Silver, Gold layers) for data curation.
  • Optimize Spark jobs for performance and scalability.

Cloud Integration:

  • Integrate Databricks with AWS or Azure services (e.g., S3, EC2, Lambda, Glue, Azure Data Factory, Event Hub).
  • Ensure secure and compliant cloud architecture.

Data Governance:

  • Implement data governance using Unity Catalog and Azure Purview.
  • Define access controls, lineage, and cataloging standards.

Cost Optimization:

  • Monitor DBU consumption and cluster utilization.
  • Implement autoscaling, auto-termination, and right-sizing strategies.

Technical Leadership:

  • Mentor junior engineers and conduct code reviews.
  • Lead technical discussions and decision-making.
  • Collaborate with data scientists, analysts, and business teams.

Required Skills & Qualifications:

  • Bachelor s or Master s degree in Computer Science, Engineering, or related field.
  • 8+ years in data engineering, with 3+ years in Databricks and Spark.
  • Expertise in PySpark, Delta Lake, Structured Streaming.
  • Experience with cloud platforms (AWS or Azure).
  • Familiarity with CI/CD, Git, and DevOps practices.
  • Strong understanding of data lakehouse architecture.

Preferred Qualifications:

  • Databricks Certified Data Engineer.
  • Microsoft Certified: Azure Data Engineer Associate.
  • Experience with Kafka, Event Hub, and real-time data processing.
  • Knowledge of Dynamics 365, Power BI, and enterprise integration.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.