Senior Databricks DevOps Engineer

1. Position Overview

Satsyil Corp is seeking an experienced Senior Databricks DevOps Engineer to design, automate, and maintain enterprise-scale Databricks environments. In this role, you will build the platform foundations that enable data engineers, data scientists, and analysts to work efficiently and securely. You will own CI/CD pipelines, infrastructure as code, workspace governance, and operational reliability across cloud-based Lakehouse environments, while serving as a technical leader and mentor on the team.

2. Key Roles & Responsibilities

·        Design, deploy, and manage Databricks workspaces across AWS, Azure, or Google Cloud Platform using infrastructure as code (Terraform, Databricks Terraform Provider, ARM/Bicep, or CloudFormation).

·        Build and maintain CI/CD pipelines for notebooks, jobs, Delta Live Tables, ML models, and libraries using Databricks Asset Bundles, Azure DevOps, GitHub Actions, GitLab CI, or Jenkins.

·        Implement and administer Unity Catalog, including metastore configuration, catalog and schema design, access controls, data lineage, and audit logging.

·        Configure and optimize clusters, SQL warehouses, cluster policies, and job compute to balance performance and cost.

·        Establish security best practices, including network isolation (VNet/VPC injection, Private Link), secrets management, SCIM provisioning, SSO integration, and role-based access control.

·        Develop monitoring, alerting, and observability solutions for platform health, job performance, and cost tracking using tools such as Datadog, Azure Monitor, CloudWatch, or Grafana.

·        Automate environment promotion (dev, test, prod) and enforce release management, version control, and change management standards.

·        Support MLOps workflows using MLflow, Model Registry, and Model Serving.

·        Troubleshoot platform, pipeline, and performance issues, and lead root-cause analysis for production incidents.

·        Partner with data engineering, security, and architecture teams to define platform standards and reference architectures.

·        Mentor junior engineers and contribute to technical documentation, runbooks, and knowledge sharing.

·        Ensure compliance with organizational and regulatory requirements (e.g., FedRAMP, NIST 800-53, FISMA, HIPAA, or SOC 2, as applicable).

3. Candidate Qualifications & Requirements

Required Experience & Technical Skills:

·        Experience: 8+ years of experience in DevOps, cloud infrastructure, or platform engineering, with at least 3 years working hands-on with Databricks.

·        IaC & Cloud Infrastructure: Strong experience with Terraform, the Databricks Terraform Provider, and deep expertise in at least one major cloud platform (Azure, AWS, or Google Cloud Platform) including networking, IAM, and storage.

·        CI/CD & Automation: Proven experience building CI/CD pipelines for data platforms; proficiency in Python, Bash, or PowerShell for automation, plus familiarity with Databricks REST API and CLI.

·        Architecture & Governance: Solid understanding of Apache Spark, Delta Lake, Lakehouse architecture, and hands-on experience with Unity Catalog administration and data governance.

·        Workflows: Strong knowledge of Git-based workflows and branching strategies.

·        Visa Type: US Ctizenship

·        Clearance & Citizenship: U.S. Citizenship required; must possess or have the ability to obtain Public Trust / Secret clearance.

·        Communication & Education: Excellent communication skills to work with technical and non-technical stakeholders. Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.

Preferred Qualifications & Nice-to-Haves:

·        Experience with Databricks Asset Bundles (DABs), Delta Live Tables (DLT), containerization (Docker, Kubernetes), and MLOps practices using MLflow at scale.

·        Familiarity with FinOps and cloud cost optimization for data workloads.

·        Prior experience supporting federal government programs or regulated industries.

4. Preferred Certifications

Candidates possessing any of the following certifications will be prioritized:

·        Databricks Certified Data Engineer Professional

·        Databricks Certified Platform Administrator

·        AWS Certified DevOps Engineer – Professional

·        HashiCorp Certified: Terraform Associate