Lead Databricks Engineer/Architect

Overview

Remote
Depends on Experience
Contract - W2
Contract - 12 Month(s)

Skills

Extract
Transform
Load
Databricks
Data Warehouse
Data Modeling

Job Details

Job Title: Lead Databricks Engineer
Location: Remote
Job Type: Contract to hire
Experience Level: Mid-Senior Level

Job Summary:

We are looking for a Lead Databricks Engineer to join and lead our data engineering team. This role combines hands-on development with team leadership responsibilities. The ideal candidate will have strong expertise in Databricks and cloud data platforms, as well as a proven ability to guide and mentor a team of engineers. You will play a key role in architecting and building scalable data pipelines, enabling advanced analytics and driving our data strategy forward.

Key Responsibilities:

  • Lead a team of data engineers, providing technical guidance, mentorship, and support for project delivery.

  • Design, develop, and optimize scalable ETL pipelines using Apache Spark on Databricks.

  • Drive architecture decisions for data processing frameworks and cloud infrastructure.

  • Collaborate with stakeholders including data scientists, analysts, and business teams to understand data requirements and translate them into technical solutions.

  • Implement and manage data lakes, data warehouses, and Delta Lake environments on cloud platforms.

  • Monitor and optimize Databricks cluster performance, ensuring cost-efficiency and scalability.

  • Oversee integration with various structured and unstructured data sources across cloud and on-prem environments.

  • Establish and enforce best practices around coding, testing, security, and DevOps for the team.

  • Create and manage CI/CD pipelines using tools such as Azure DevOps, GitHub Actions, or similar.

  • Document technical architecture, workflows, and development processes.

Required Skills and Qualifications:

  • Bachelor s or Master s degree in Computer Science, Engineering, or a related field.

  • 5 8 years of experience in data engineering, with at least 2 years in a technical leadership role.

  • Extensive hands-on experience with Databricks, Apache Spark, and Delta Lake.

  • Strong experience with at least one major cloud platform (Azure, AWS, or Google Cloud Platform).

  • Proficiency in Python, SQL, and optionally Scala.

  • Experience managing and mentoring data engineers or developers.

  • Sound understanding of data modeling, data warehousing, and distributed data processing.

  • Familiarity with infrastructure-as-code tools (Terraform, ARM templates, etc.).

  • Strong problem-solving, communication, and leadership skills.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.