Azure Databricks Engineer (DLT & Streaming)

Overview

Hybrid
Depends on Experience
Accepts corp to corp applications
Contract - Independent
Contract - W2
Contract - 9 Month(s)
No Travel Required
Unable to Provide Sponsorship

Skills

Analytics
Streaming
Real-time
Regulatory Compliance
Scalability
Management
Microsoft Azure
Problem Solving
Storage
Information Security Governance
PySpark
Data Validation
Continuous Integration
Data Lake
Data Quality
Communication
Conflict Resolution
Continuous Delivery
Apache Kafka
Apache Spark
Business Intelligence
Databricks
DevOps
Machine Learning (ML)

Job Details

Azure Databricks Engineer (DLT & Streaming)

Location: Houston, TX

Key Responsibilities

  • Design, develop, and maintain end-to-end data pipelines using Azure Databricks

  • Build and manage Delta Live Tables (DLT) pipelines with data quality checks and expectations

  • Implement real-time and near-real-time streaming solutions using Structured Streaming

  • Integrate data from multiple sources including Azure Event Hubs, Kafka, ADLS Gen2, and APIs

  • Optimize Databricks jobs for performance, scalability, and cost efficiency

  • Develop and maintain Delta Lake architectures (Bronze, Silver, Gold layers)

  • Implement data validation, error handling, monitoring, and alerting mechanisms

  • Collaborate with data engineers, data scientists, and BI teams to support analytics and ML use cases

  • Ensure security, governance, and compliance using Azure-native tools

  • Participate in code reviews, CI/CD pipelines, and best practices implementation

Required Skills & Qualifications

  • Strong experience with Azure Databricks

  • Hands-on expertise in Delta Live Tables (DLT)

  • Experience with Databricks Structured Streaming

  • Proficiency in PySpark / Spark SQL

  • Strong knowledge of Delta Lake and medallion architecture

  • Experience with Azure Data Lake Storage (ADLS Gen2)

  • Familiarity with Azure Event Hubs / Kafka for streaming ingestion

  • Experience with Azure DevOps or CI/CD pipelines

  • Understanding of data quality, monitoring, and pipeline orchestration

  • Strong problem-solving and communication skills

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.