Databricks Engineer

Overview

Remote
Depends on Experience
Accepts corp to corp applications
Contract - W2
Contract - Independent
Contract - 12 Month(s)

Skills

Amazon Web Services
Apache Spark
Artificial Intelligence
Cloud Computing
Conflict Resolution
Computer Science
Data Modeling
Databricks
Data Security
Encryption
Extract
Transform
Load
Good Clinical Practice
Google Cloud Platform
Identity Management
Machine Learning (ML)
SQL
Prompt Engineering

Job Details

Job Title : Databricks Engineer
Location : Remote
Duration : Long Term Contract
Job Description
Senior Databricks Engineer to join our Data Engineering team. This role is critical to designing, developing, and optimizing scalable data solutions using the Databricks Lakehouse platform. The ideal candidate possesses deep expertise in Databricks, Delta Lake, and cloud-based data architectures (Azure). Experience with ML and AI prompt engineering is a plus. This role will involve end-to-end solution design, including transforming business requirements into technical designs and implementation strategies for production.
Responsibilities include:
  • Design, develop, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Scala, or SQL).
  • Architect and optimize Delta Lake implementations to support reliable and performant data lakes.
  • Implement and enforce data quality, governance, and observability best practices within Databricks.
  • Collaborate with data scientists, analysts, and stakeholders to understand data needs and translate requirements into robust engineering solutions.
  • Lead the design and implementation of data solutions, including end-to-end technical strategies for production.
  • Monitor and optimize data workflows for cost efficiency and performance, especially in cloud environments (Azure, AWS, or Google Cloud Platform).
  • Lead migration efforts from legacy data systems to Databricks and modern cloud-native architectures.
  • Mentor junior engineers and establish best practices for code quality, version control, and CI/CD processes.
  • Participate in data modeling and help enforce medallion architecture standards (bronze, silver, gold layers).
Desired Experience/Education:
  • Bachelor s or Master s degree in Computer Science, Engineering, or related field.
  • 5+ years of experience in data engineering, with at least 2+ years hands-on with Databricks.
  • Expert-level proficiency in Spark (PySpark and/or Scala) and SQL.
  • Deep experience with Delta Lake, data lakehouse architectures, and advanced performance optimization techniques (e.g., adaptive query execution, optimized writes, Z-ordering, caching strategies).
  • Experience with data lineage and data catalog tools (e.g., Unity Catalog, Azure Purview).
  • Proficiency in building and managing streaming data pipelines using Structured Streaming in Databricks.
  • Strong knowledge of data encryption, masking, and data privacy best practices.
  • Strong experience with orchestration tools (e.g., Airflow, Azure Data Factory, Databricks Workflows).
  • Proficiency with monitoring and logging solutions
  • Proven experience in technical design, solution architecture, and implementation strategy.
  • Strong knowledge of Azure Data Lake, Azure Data Factory, and Azure Synapse (or equivalents in AWS/Google Cloud Platform).
  • Proficiency in CI/CD for data pipelines (e.g., GitHub Actions, Azure DevOps, Jenkins).
  • Solid understanding of data security, identity management, and access control in the cloud.
  • Familiarity with Infrastructure-as-Code (IaC) tools (e.g., Terraform, ARM templates, CloudFormation).
Other Desired Skills/Certifications:
  • Experience with ML and AI prompt engineering
  • Experience implementing medallion architecture in production environments.
  • Databricks certifications (e.g., Databricks Certified Data Engineer Professional).
  • Familiarity with MLflow, Unity Catalog, and Databricks Delta Live Tables (DLT).
  • Excellent problem-solving and troubleshooting skills.
  • Strong communication skills, both written and verbal.
  • Ability to work independently and lead cross-functional initiatives.
  • Passion for automation, reusability, and performance optimization.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.