Role : Databricks Platform & CI/CD Engineer
Location : Remote
Position Summary
We are seeking a hands-on Senior Databricks Platform & CI/CD Engineer to support the implementation, administration, automation, and operationalization of our enterprise Databricks platform.
The contractor will play a key role in establishing platform standards, Unity Catalog governance, deployment automation, CI/CD frameworks, and AI/ML platform(nice to have). This role will partner closely with Data Engineering, AI Engineering, Architecture, Governance, and Enterprise Platform.
This position focuses on Databricks platform engineering and DevOps enablement, not cloud infrastructure provisioning. Environment provisioning and foundational cloud services are managed by the enterprise platform teams. The selected engineer will be responsible for configuring and operating Databricks services within approved enterprise environments.
Key Responsibilities
Databricks Platform Administration
- Configure and administer Databricks workspaces.
- Implement and manage Unity Catalog governance standards.
- Configure catalogs, schemas, permissions, and access controls.
- Develop and maintain Databricks platform standards and best practices.
- Monitor platform health, utilization, performance, and operational readiness.
- Assist with troubleshooting platform and deployment issues.
CI/CD & DevOps Automation
- Design and implement CI/CD pipelines for Databricks solutions.
- Integrate Databricks with Azure DevOps and GitHub Enterprise.
- Build deployment automation using Databricks Asset Bundles (DAB).
- Automate promotion across Development, Test, UAT, and Production environments.
- Implement source control strategies, pull request workflows, and release management processes.
- Develop automated validation, testing, and deployment processes.
- Create reusable deployment templates and engineering accelerators.
Governance & Security
- Implement Unity Catalog security and governance controls.
- Configure role-based access management (RBAC).
- Support PHI/PII data governance requirements.
- Configure audit logging, monitoring, and data lineage capabilities.
- Implement enterprise security and compliance standards.
- Partner with Governance, Privacy, Security, and Architecture teams to ensure adherence to organizational requirements.
AI/ML Platform Enablement
- Configure and support MLflow environments.
- Enable model lifecycle management and deployment processes.
- Support feature engineering and model-serving capabilities.
- Assist with the deployment and operationalization of AI, Machine Learning, Generative AI, and Agentic AI solutions.
- Help establish best practices for AI platform governance and monitoring.
Production Support & Operational Readiness
- Establish platform monitoring and alerting capabilities.
- Support production deployments and incident resolution.
- Develop operational runbooks and troubleshooting procedures.
- Identify platform optimization opportunities.
- Partner with operations teams to ensure platform reliability and supportability.
Required Qualifications
Experience
- 5+ years of experience in Data Platform Engineering, DevOps Engineering, or Cloud Data Engineering.
- 3+ years of hands-on Databricks platform experience.
- 3+ years of experience building and supporting CI/CD pipelines.
- Experience supporting enterprise-scale production environments.
- Experience collaborating across architecture, engineering, security, and governance organizations.
Required Technical Skills
Databricks
- Unity Catalog
- Delta Lake
- Databricks Workflows
- Delta Live Tables (DLT)
- Databricks Asset Bundles (DAB)
- Databricks Repos
- Databricks Jobs
DevOps & CI/CD
- Azure DevOps
- GitHub Enterprise
- Git Branching Strategies
- CI/CD Pipeline Development
- Release Management
- Automated Testing Frameworks
Programming
Governance & Security
- Role-Based Access Control (RBAC)
- Data Governance
- Data Lineage
- Audit Controls
- Access Management
- PHI/PII Data Handling
Preferred Qualifications
- MLflow
- Databricks Model Serving
- Databricks Apps
- Vector Search
- Azure OpenAI
- Generative AI platforms
- Agentic AI frameworks
- Healthcare industry experience
- Experience working in regulated environments
Operations
- Production Support
- Monitoring & Alerting
- Incident Management
- Performance Optimization
- Operational Runbooks
Primary Focus Areas
- Databricks Platform Administration (25%)
- CI/CD & Release Automation (35%)
- Unity Catalog & Security Governance (35%)
- AI/ML Platform Enablement (10%)
Never miss an opportunity! Create an alert based on the job you applied for.