Overview
On Site
Depends on Experience
Accepts corp to corp applications
Contract - W2
Contract - Independent
Contract - 12 Month(s)
No Travel Required
Skills
Azure Databricks
Python
Spark Streaming
Lead
Data engineer
Job Details
Job Title: ADB Technical Lead - Azure Databricks/Python/Spark Streaming
Location: Pleasanton, CA - Onsite
Location: Pleasanton, CA - Onsite
Exp: 15
About the Role:
- We are seeking a hands-on and technically strong Azure Databricks (ADB) Technical Lead to architect and drive the development of large-scale batch and streaming data pipelines on Azure Cloud.
- This role requires expertise in Python, Databricks Notebooks, Apache Spark (including Structured Streaming), and real-time integration with Kafka.
- You will work with both relational databases like DB2 and NoSQL systems such as MongoDB, with a strong focus on performance optimization and scalable architecture.
- The ideal candidate excels at leading technical teams, mentoring others, and communicating effectively across business and technology stakeholders.
Required:
- 15+ years of experience in date engineering, with 7+ years in a technical leadership role.
- Deep hands-on experience with Azure Databricks and Apache Spark, including Structured Streaming
- Proficiency in Python for building scalable and reusable date workflows.
- Experience with Kafka for real-time data ingestion and event streaming
- Strong experience integrating with relational (DB2, PostgreSQL) and NoSQL (MongoDB) databases.
- Demonstrated expertise in performance tuning and scaling Spark jobs in production environments, Strong communication and analytical skills, with the ability to work cross-functionally and lead through influence.
Preferred:
- Experience with Java or Scala in Spark streaming or Kafka connector development
- Familianty with Azure services like Data Lake, Data Factory, Synapse, and Event Hubs.
- Knowledge of CI/CD pipelines, GitHub workflows, and Infrastructure as Code (e.g., Terraform).
- Background in building data platforms in regulated or large-scale enterprise environments.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.