Databricks DevOps Engineer

  • Nashville, TN
  • Posted 1 day ago | Updated 1 day ago

Overview

On Site
Depends on Experience
Contract - W2
Contract - 12 Month(s)

Skills

Azure DevOps Pipelines
YAML builds
Azure Databricks
Databricks CLI
Databricks REST APIs
Python
PowerShell
or Bash
PySpark
ADLS Gen2
Key Vault
ADF/Synapse

Job Details

Databricks DevOps Engineer

Develop and maintain CI/CD pipelines in Azure DevOps to deploy notebooks, jobs, and workflows to Azure Databricks.

Integrate Databricks Repos with Azure Repos (Git) for source control of PySpark/R notebooks.

Automate job runs in Databricks using Databricks CLI, REST API, or DevOps tasks.

Maintain infrastructure-as-code practices using Terraform (optional).

Monitor and troubleshoot pipeline failures, notebook issues, and deployment inconsistencies.

Collaborate with Data Engineers and Platform teams to maintain development, staging, and production environments.
Implement best practices for version control, testing, and notebook promotion in a multi-environment setup.

Store and manage Databricks secrets, tokens, and configuration securely within Azure Key Vault and DevOps variable groups.

Required Skills:
Strong experience with Azure DevOps Pipelines, YAML builds, and release workflows.

Strong knowledge of cloud platforms like Azure, AWS, or Google Cloud Platform.
Hands-on experience with Azure Databricks: clusters, jobs, Repos, and workspace configuration.
Proficiency in Databricks CLI, Databricks REST APIs, and automating notebook execution.

Proficiency in scripting languages like Python, PowerShell, or Bash
Experience with PySpark, Delta Lake, and SQL for data engineering tasks.
Familiarity with Git for source control (Azure Repos or GitHub).
Working knowledge of Azure services: ADLS Gen2, Key Vault, ADF/Synapse.
Understanding of DevOps practices in a data platform context

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.