Azure Databricks Platform Administrator

Job Title: Azure Databricks Platform Engineer
Location: Remote
Employment Type: Contract / Long-Term Consulting Engagement

Must Have Skills:
Network Connectivity Configuration (NCC)
Azure Access Connectors
Role Overview
We are seeking an experienced Azure Databricks Platform Engineer to architect, administer, and optimize our enterprise Databricks environment within the Azure cloud ecosystem. This role is critical to enabling large-scale data engineering, analytics, and AI/ML workloads while ensuring security, governance, cost optimization, and operational excellence. This is a hands-on technical position requiring deep expertise in Databricks platform administration, Unity Catalog, Spark performance tuning, cloud networking, and infrastructure automation.
Key Responsibilities
Databricks Platform Engineering
Architect, deploy, and operate Azure Databricks workspaces across Development, QA, and Production environments.
Configure and manage Databricks clusters, jobs, pipelines, and notebooks for enterprise-scale workloads.
Implement Unity Catalog for data governance, including RBAC, lineage, metadata management, and access controls.
Configure Databricks Network Connectivity Configuration (NCC) for secure workspace networking, including private endpoints, route tables, subnets, and group policies.
Implement Databricks Access Connectors for external storage access (ADLS Gen2), including RBAC policies, managed identity, and Unity Catalog integration.
Migration & Modernization
Lead migration initiatives from on-premises SQL Server, SSIS, and legacy data platforms to Azure Databricks.
Design and implement Lakehouse architectures using Delta Lake and ADLS Gen2.
Build and optimize ETL/ELT pipelines using Databricks, Azure Data Factory, and Azure Synapse Analytics.
Performance & Cost Optimization
Optimize Apache Spark workloads through performance tuning, cluster sizing, and execution strategy improvements.
Implement auto-scaling and cost-control measures to reduce cloud infrastructure spend.
Monitor and optimize Databricks job performance, resource utilization, and billing.
Security & Compliance
Enforce Zero Trust security model using Microsoft Entra ID, RBAC, Privileged Identity Management (PIM), and Conditional Access.
Implement HIPAA/SOX/GDPR compliance using Azure Policy, Defender for Cloud, and Microsoft Sentinel.
Manage Azure Key Vault for secure credential and secret management.
Implement data lineage, cataloging, and classification using Microsoft Purview or similar tools.
Automation & CI/CD
Automate Databricks infrastructure provisioning using Terraform, ARM templates, or Bicep.
Build and maintain CI/CD pipelines using Azure DevOps or GitHub Actions for Databricks deployments.
Implement Infrastructure as Code (IaC) best practices for workspace, cluster, and policy configurations.
Collaboration & Support
Partner with data engineering, data science, infrastructure, security, and DevOps teams.
Provide L2/L3 production support, troubleshoot platform issues, and conduct root cause analysis.
Mentor team members on Databricks best practices, governance, and platform optimization.
Required Qualifications
Experience & Skills
3+ years of hands-on experience with Azure Databricks administration, workspaces, clusters, jobs, and notebooks.
Strong experience implementing Unity Catalog governance, RBAC, lineage, and access controls.
Deep understanding of Apache Spark, PySpark, Spark SQL, and performance tuning.
Experience with Delta Lake architecture, table management, and Lakehouse patterns.
Proven experience migrating SQL Server, SSIS, and on-prem workloads to Azure Databricks.
Strong experience with Azure services: ADLS Gen2, Azure SQL, Azure Data Factory, Azure Synapse, VNets, and Private Endpoints.
Understanding of cloud networking: VNets, subnets, private endpoints, route tables, and NSGs.
Experience with Azure Key Vault, RBAC, managed identity, Entra ID, and compliance frameworks (HIPAA/SOX).
Hands-on experience with Terraform, ARM templates, or Bicep for infrastructure provisioning.
Experience with Azure DevOps, GitHub Actions, or similar CI/CD tools for Databricks deployments.
Familiarity with data governance tools like Microsoft Purview, Alation, or similar lineage solutions.
Strong troubleshooting, analytical, and problem-solving skills.
Excellent communication and collaboration skills with the ability to work across engineering, security, governance, and operations teams.
 
Preferred Qualifications
Experience with real-time streaming using Azure Event Hub or Service Bus.
Experience with Kubernetes (AKS) and containerized workloads.
Databricks certification (Platform Administrator or equivalent).
Experience with AI/ML workloads using Databricks ML and MLflow.
Familiarity with NVIDIA AI Infrastructure and GPU workloads.
Experience with CyberArk or other privileged access management solutions.