Job Summary:
We are seeking a highly skilled Databricks DevOps Engineer or Senior Databricks Workspace Administrator to join our team. The candidate will be responsible for designing, implementing, and maintaining CI/CD pipelines, automating infrastructure, integrating version control, and supporting MLOps workflows to ensure seamless deployment and monitoring of data and AI initiatives. This role requires both hands-on technical expertise and the ability to guide and mentor junior team members.
Key Responsibilities:
• Design, implement, and maintain CI/CD pipelines for Databricks notebooks, libraries, and workflows using Azure DevOps or GitHub Actions.
• Automate infrastructure provisioning using ARM templates or Databricks REST APIs.
• Integrate Databricks with version control (Git) for collaborative development.
• Implement monitoring, logging, and alerting using Seq logger, Azure Monitor, and Log Analytics.
• Support MLOps workflows for deploying and monitoring machine learning models.
• Troubleshoot, optimize, and maintain Databricks workspaces, clusters, jobs, and pipelines.
• Collaborate with cross-functional teams to ensure compliance, security, and operational efficiency.
• Mentor junior engineers and provide technical guidance in best practices for Databricks administration and CI/CD processes.
• Document infrastructure, processes, and workflows for knowledge sharing and operational continuity.
Required Skills & Qualifications:
• Experience: 5–10 years in DevOps, Cloud Engineering, or Databricks administration.
• Strong experience in Databricks administration (clusters, jobs, repos, Unity Catalog).
• Hands-on expertise with Azure DevOps, Git, and CI/CD pipelines.
• Proficiency in Infrastructure as Code (IaC).
• Scripting skills in Python, PowerShell, or Bash for automation.
• Solid understanding of cloud networking, security, and compliance in Azure.
• Strong troubleshooting, performance optimization, and debugging skills.
• Ability to work independently and collaboratively in cross-functional teams.
• Excellent communication, documentation, and stakeholder management skills.
Preferred Experience:
• Experience supporting MLOps workflows, model deployment, and monitoring.
• Prior experience in large-scale enterprise Databricks deployments.
• Familiarity with monitoring and logging tools in Azure (e.g., Log Analytics, Application Insights).
• Exposure to cloud governance, cost optimization, and security best practices in Azure.
Education:
• Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field.
Other Details:
• Work Arrangement: Hybrid or onsite as per project requirements.
• Reporting: Reports to the Data/AI Delivery Lead or Cloud Infrastructure Manager.
• Certifications (Optional but Preferred):
o Databricks Certified Data Engineer Associate
o Azure DevOps Engineer Expert
o Azure Solutions Architect
Never miss an opportunity! Create an alert based on the job you applied for.
