Databricks Platform Lead - Schaumburg, IL (Hybrid)

Overview

Hybrid
Depends on Experience
Contract - W2
Contract - Independent
Contract - 12 month(s)
No Travel Required

Skills

Azure
aws
Spark
Databricks
data engineering
PySpark
delta lake
data lakehouse

Job Details

Position: Databricks Platform Lead

Duration: 12 Months

Location: Schaumburg, IL (Hybrid)

 

Key Responsibilities:


Platform Leadership
Lead the design and deployment of large-scale data solutions on Databricks.
Establish best practices for notebook development, job orchestration, and cluster management.
Stay current with Databricks features and recommend improvements.

Data Engineering & Pipeline Development
Build robust batch and streaming data pipelines using PySpark, Delta Lake, and Structured Streaming.
Implement Medallion architecture (Bronze, Silver, Gold layers) for data curation.
Optimize Spark jobs for performance and scalability.

Cloud Integration
Integrate Databricks with AWS or Azure services (e.g., S3, EC2, Lambda, Glue, Azure Data Factory, Event Hub).
Ensure secure and compliant cloud architecture.

Data Governance
Implement data governance using Unity Catalog and Azure Purview.
Define access controls, lineage, and cataloging standards.

Cost Optimization
Monitor DBU consumption and cluster utilization.
Implement auto scaling, auto-termination, and right-sizing strategies.

Technical Leadership
Mentor junior engineers and conduct code reviews.
Lead technical discussions and decision-making.
Collaborate with data scientists, analysts, and business teams.

Required Skills & Qualifications
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
10+ years in data engineering, with 3+ years in Databricks and Spark.
Expertise in PySpark, Delta Lake, Structured Streaming.
Experience with cloud platforms (AWS or Azure).
Familiarity with CI/CD, Git, and DevOps practices.
Strong understanding of data lakehouse architecture.

Preferred Qualifications
Databricks Certified Data Engineer.
Microsoft Certified: Azure Data Engineer Associate.
Experience with Kafka, Event Hub, and real-time data processing.
Knowledge of Dynamics 365, Power BI, and enterprise integration.

Soft Skills
Leadership and mentoring.
Strategic thinking and problem-solving.
Excellent communication and stakeholder management.
Adaptability and continuous learning.


Best Regards,

Vishal

Truth Lies in  Heart

 

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.