Overview
Remote
Full Time
Skills
FOCUS
Customer Engagement
Management
Machine Learning (ML)
Team Leadership
Mentorship
Continuous Improvement
Collaboration
DevOps
Optimization
Regulatory Compliance
Data Quality
Access Control
Data Engineering
IT Management
Workflow
Unity
Migration
Electronic Health Record (EHR)
Machine Learning Operations (ML Ops)
Continuous Integration
Continuous Delivery
Cloud Computing
Google Cloud
Google Cloud Platform
Python
SQL
Apache Spark
Apache Kafka
Streaming
Communication
Leadership
Problem Solving
Conflict Resolution
Innovation
Databricks
Microsoft
Microsoft Azure
Amazon Web Services
Data Analysis
Data Governance
Professional Services
Job Details
Lead Data Engineer - Databricks & MLOps Focus
We are seeking a highly experienced and strategic Lead Data Engineer with deep expertise in Databricks, MLOps, and cloud-native data engineering. This role is ideal for a technically
strong leader who thrives in consulting environments, excels at client engagement, and can drive complex data initiatives from architecture to execution.
Key Responsibilities:
Lead Architecture & Development: Design and implement scalable, high-performance data pipelines and MLOps workflows using Databricks, Delta Lake, and MLflow.
Migration Leadership: Oversee the migration of AWS EMR Spark jobs to Databricks, optimizing for performance, cost, and maintainability.
Client Consulting: Act as a trusted advisor to clients, providing strategic guidance on Databricks architecture, MLOps best practices, and data platform modernization.
Model Operationalization: Collaborate with data scientists to productionize machine learning models and manage the full ML lifecycle.
Team Leadership: Mentor and lead data engineering teams, fostering a culture of innovation, quality, and continuous improvement.
Cross-functional Collaboration: Work closely with cloud architects, DevOps, and business stakeholders to align technical solutions with business goals.
Innovation & Optimization: Identify opportunities to improve data workflows, automation, and platform capabilities using modern tools and frameworks.
Governance & Compliance: Implement data governance practices using tools like Unity Catalog, ensuring data quality, lineage, and access control.
Required Skills & Experience:
7+ years of experience in data engineering, with at least 2 years in a technical leadership or consulting role.
Expert-level proficiency in Databricks, including Delta Lake, MLflow, Job Workflows, and Unity Catalog.
Proven experience migrating and optimizing Apache Spark workloads from EMR or other platforms to Databricks.
Strong understanding of MLOps principles, CI/CD for data pipelines, and model deployment strategies.
Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
Proficiency in Python, SQL, and Spark.
Experience with streaming data technologies like Kafka, Structured Streaming, or Delta Live Tables.
Excellent communication, leadership, and consulting skills, with a proven ability to engage clients and stakeholders.
Strong problem-solving skills and a passion for innovation and continuous learning.
Preferred Qualifications:
Certifications such as:
- Databricks Certified Data Engineer - Professional
- Microsoft Certified: Azure Data Engineer Associate
- AWS Certified Data Analytics - Specialty
Experience implementing medallion architecture (bronze-silver-gold layers).
Familiarity with data governance and observability tools.
Background in consulting firms or professional services with a track record of delivering high-impact solutions.
Note: Job Description and Background Check
Candidates may be subjected to a Background Check /Drug Test as required by the end client before the assignment starts.
We are seeking a highly experienced and strategic Lead Data Engineer with deep expertise in Databricks, MLOps, and cloud-native data engineering. This role is ideal for a technically
strong leader who thrives in consulting environments, excels at client engagement, and can drive complex data initiatives from architecture to execution.
Key Responsibilities:
Lead Architecture & Development: Design and implement scalable, high-performance data pipelines and MLOps workflows using Databricks, Delta Lake, and MLflow.
Migration Leadership: Oversee the migration of AWS EMR Spark jobs to Databricks, optimizing for performance, cost, and maintainability.
Client Consulting: Act as a trusted advisor to clients, providing strategic guidance on Databricks architecture, MLOps best practices, and data platform modernization.
Model Operationalization: Collaborate with data scientists to productionize machine learning models and manage the full ML lifecycle.
Team Leadership: Mentor and lead data engineering teams, fostering a culture of innovation, quality, and continuous improvement.
Cross-functional Collaboration: Work closely with cloud architects, DevOps, and business stakeholders to align technical solutions with business goals.
Innovation & Optimization: Identify opportunities to improve data workflows, automation, and platform capabilities using modern tools and frameworks.
Governance & Compliance: Implement data governance practices using tools like Unity Catalog, ensuring data quality, lineage, and access control.
Required Skills & Experience:
7+ years of experience in data engineering, with at least 2 years in a technical leadership or consulting role.
Expert-level proficiency in Databricks, including Delta Lake, MLflow, Job Workflows, and Unity Catalog.
Proven experience migrating and optimizing Apache Spark workloads from EMR or other platforms to Databricks.
Strong understanding of MLOps principles, CI/CD for data pipelines, and model deployment strategies.
Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
Proficiency in Python, SQL, and Spark.
Experience with streaming data technologies like Kafka, Structured Streaming, or Delta Live Tables.
Excellent communication, leadership, and consulting skills, with a proven ability to engage clients and stakeholders.
Strong problem-solving skills and a passion for innovation and continuous learning.
Preferred Qualifications:
Certifications such as:
- Databricks Certified Data Engineer - Professional
- Microsoft Certified: Azure Data Engineer Associate
- AWS Certified Data Analytics - Specialty
Experience implementing medallion architecture (bronze-silver-gold layers).
Familiarity with data governance and observability tools.
Background in consulting firms or professional services with a track record of delivering high-impact solutions.
Note: Job Description and Background Check
Candidates may be subjected to a Background Check /Drug Test as required by the end client before the assignment starts.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.