Databricks Engineering Lead

Our client is a well-established consulting firm. They seek a senior-level Databricks engineering leader with deep, hands-on experience building modern data platforms in Azure Databricks. This role is designed for an engineer who is highly technical, code-driven, and comfortable operating at scale within distributed systems.

This opportunity goes well beyond traditional data integration or ETL work. The focus is on Spark internals, real-time data processing, performance tuning, and architecting scalable lakehouse solutions. The individual in this role will also act as a technical point of leadership, helping guide best practices across teams.

What You ll Be Doing

• Architecting and developing scalable batch and streaming data pipelines using PySpark and Azure Databricks
  
• Designing and implementing real-time data workflows leveraging Structured Streaming
  
• Building declarative lakehouse pipelines using Delta Live Tables and Lakehouse Declarative Pipelines
  
• Driving Spark optimization and Databricks cost-efficiency initiatives
  
• Partnering closely with architects, analysts, and business stakeholders to translate requirements into robust technical solutions
  
• Providing technical leadership through code reviews, mentorship, and architectural input
  
• Ensuring platform reliability through monitoring, tuning, and proactive performance management
  

Required Experience

• Significant hands-on experience with Azure Databricks in production environments
  
• Advanced PySpark development expertise and strong core programming skills
  
• Deep understanding of distributed data processing frameworks and Spark execution behavior
  
• Demonstrated success with:
  
• Spark performance optimization
  
• Databricks cost management
  
• Streaming data architectures
  
• Delta Live Tables (DLT) and declarative pipeline patterns
  
• Experience within the broader Azure data ecosystem (e.g., ADLS, Synapse, Azure Data Factory)
  
• Strong communication skills and the ability to collaborate across technical and non-technical teams
  
• Ability and willingness to work onsite in Houston
  

Preferred Background

Exposure to data platforms in energy, utilities, or industrial environments