Databricks Engineer

Overview

Remote
Depends on Experience
Contract - W2
No Travel Required

Skills

Azure Databricks
ETL
PySpark

Job Details

Lead Databricks Engineer Role Summary

  1. Architecture & Engineering Leadership:
    Leads the end-to-end architecture, design, and implementation of big data pipelines on Databricks Lakehouse Platform, integrating with Azure, AWS, or Google Cloud Platform.

  2. Data Pipeline & ETL Expertise:
    Develops scalable, high-performance ETL/ELT pipelines using PySpark, Delta Lake, and SQL, enabling real-time and batch processing of large datasets.

  3. Team & Project Oversight:
    Mentors junior engineers, drives Agile delivery, performs code reviews, and ensures best practices in data engineering and CI/CD (using Git, Azure DevOps, or Jenkins).

  4. Cross-Platform Integration:
    Integrates Databricks with external systems such as Kafka, Snowflake, Power BI, Synapse, and cloud storage (S3, ADLS), ensuring data quality, security, and governance.

  5. Performance, Security & Governance:
    Optimizes jobs and clusters for cost and performance. Implements Unity Catalog, role-based access controls, and compliance policies for secure enterprise data usage.

Must-Have Skills:

  • Databricks (PySpark, Delta Lake, MLflow, Unity Catalog)

  • Cloud (Azure Databricks, AWS/Google Cloud Platform optional)

  • Python, SQL, Spark

  • CI/CD, Git, DevOps, Terraform (optional)

  • Data Governance, Performance Tuning

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.