Databricks

Databricks Data Engineer

Location: Dallas, TX (Hybrid/Onsite)
Employment Type: W2

Job Summary

We are looking for a skilled Databricks Data Engineer to design, develop, and optimize scalable data pipelines on the Databricks Lakehouse Platform. The ideal candidate should have hands-on experience with Spark, PySpark, Delta Lake, cloud platforms (Azure/AWS), and modern ETL/ELT architectures.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and PySpark.
• Build and optimize batch and streaming data pipelines using Spark Structured Streaming and Auto Loader.
• Implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Develop and manage Delta Live Tables (DLT) and Databricks Workflows.
• Optimize Spark jobs for performance, scalability, and cost.
• Work with Unity Catalog for data governance, lineage, and security.
• Integrate Databricks with cloud storage such as Azure Data Lake Storage (ADLS), Amazon S3, or Google Cloud Storage.
• Collaborate with data analysts, BI developers, and data scientists to deliver analytics-ready datasets.
• Implement CI/CD using Git, Azure DevOps, GitHub Actions, or similar tools.
• Monitor pipeline performance and troubleshoot production issues.
• Follow Agile/Scrum methodologies and participate in sprint planning.

Required Skills

• 5+ years of experience in Data Engineering.
• 3+ years of hands-on Databricks experience.
• Strong expertise in:
  • Databricks
  • Apache Spark
  • PySpark
  • Spark SQL
  • Python
  • SQL
  • Delta Lake
  • Delta Live Tables (DLT)
  • Unity Catalog

Preferred Skills

• Databricks Certified Data Engineer Associate/Professional.
• Experience with Kafka or Event Hubs.
• Experience with dbt.
• Knowledge of Power BI or Tableau.
• Exposure to MLflow and AI/ML workloads.
• Terraform or Infrastructure as Code experience.