Lead Data Engineer Databricks (REMOTE) (Only Ex-Charter consultant )

Overview

Remote
$40 - $60
Contract - W2
Contract - 12 Month(s)

Skills

Amazon Web Services
Apache Kafka
Apache Spark
Cloud Computing
Communication
Continuous Delivery
Continuous Integration
Data Analysis
Data Engineering
Data Governance
Databricks
DevOps
Electronic Health Record (EHR)
Good Clinical Practice
Google Cloud Platform
Machine Learning (ML)
Machine Learning Operations (ML Ops)
Management
Mentorship
Microsoft
Microsoft Azure
Migration
SQL
Streaming
Unity
Workflow

Job Details

Job Summary:

We are seeking a strategic and technically hands-on Lead Data Engineer with Databricks, MLOps, and cloud-native data engineering expertise. The role involves leading high-impact data initiatives, Spark workload migration, and guiding clients through modernization and architecture decisions.

Key Responsibilities:

  • Design and implement scalable data pipelines and MLOps workflows using Databricks, Delta Lake, and MLflow

  • Lead Spark workload migration from AWS EMR to Databricks

  • Act as a consultant/advisor to clients on Databricks architecture and best practices

  • Operationalize ML models, manage full ML lifecycle with CI/CD integration

  • Collaborate with DevOps, Architects, and Stakeholders

  • Ensure data governance with tools like Unity Catalog

  • Lead and mentor data engineering teams

  • Optimize workflows, innovate data platform capabilities

    Skill / Technology
    7+ years data engineering
    2+ years in consulting or technical leadership
    Expert with Databricks (incl. Workflows, Unity Catalog)
    Delta Lake
    MLflow
    AWS EMR to Databricks migration
    Spark (Structured Streaming, Spark SQL)
    MLOps principles & ML CI/CD
    Python programming
    SQL (Advanced)
    Cloud platforms (AWS / Azure / Google Cloud Platform)
    Kafka / Delta Live Tables / Streaming frameworks
    Strong stakeholder communication skills
    Client consulting experience
    Databricks Certified Data Engineer Professional
    Azure Data Engineer Associate (Microsoft)
    AWS Certified Data Analytics Specialty
    Medallion Architecture (Bronze/Silver/Gold layers)
    Data governance/observability tooling (e.g., Unity Catalog)
    Ex-Charter consultant experience
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

About Bridge Flair LLC