Databricks/Cloud Engineer

We are seeking a Databricks / Cloud Engineer to serve as the hands-on technical owner of the Databricks platform supporting an Enterprise Data Platform. This role is responsible for end-to-end platform operations, security, governance, and cost controls, ensuring the environment is compliant, reliable, and scalable for secure analytics and AI/ML workloads.

Key Responsibilities

Platform Administration & Operations

• Administer Databricks accounts and workspaces across SDLC environments; standardize configuration, naming conventions, and operational patterns.
• Configure and maintain clusters/compute, job compute, SQL warehouses, runtimes, libraries, repos, and workspace settings.
• Implement monitoring, alerting, operational dashboards, and health checks; maintain runbooks and operational procedures.
• Provide Tier 2/3 operational support, including incident troubleshooting, root cause analysis, and preventive remediation.
• Manage change control for upgrades, feature rollouts, and configuration changes with documented impact and rollback plans.

Security, Access Control & Governance

• Enforce least-privilege access across workspaces, jobs, clusters, SQL warehouses, repos, and secrets using group-based RBAC.
• Configure and manage secrets, credential handling, and key management integrations.
• Enable audit logging and access/event visibility; support security reviews and audit evidence requests.
• Administer Unity Catalog governance, including metastores, catalogs, schemas, tables, ownership, grants, and environment/domain patterns.
• Configure external locations, storage credentials, and governed access to cloud object storage.
• Partner with governance stakeholders to support metadata, lineage, classification/tagging, and retention controls.

Cloud Integration & Networking

• Coordinate secure connectivity and guardrails with cloud and network teams, including private connectivity, egress controls, and firewall/proxy requirements.
• Configure required cloud integrations for governed data access and service connectivity (roles, permissions, endpoints, storage integrations).

Cost Management & Performance Optimization

• Implement cost controls such as cluster policies, auto-termination, workload sizing standards, scheduling, and capacity planning.
• Produce usage and cost insights with optimization recommendations addressing idle compute, oversized clusters, and inefficient workloads.
• Monitor and optimize platform performance, including SQL warehouse tuning, autoscaling, Photon enablement, and Delta Lake optimization best practices.

Automation, DevOps & Resiliency

• Automate administration and configuration using APIs, CLI, and Infrastructure-as-Code (e.g., Terraform).
• Implement Databricks Asset Bundles (DABs) and CI/CD workflows for standardized deployment across environments.
• Design and maintain backup and disaster recovery procedures for workspace assets, Unity Catalog metadata, and job configurations; perform periodic DR testing aligned to RTO/RPO.
• Administer Delta Live Tables (DLT) pipelines and coordinate with data engineering teams on pipeline health and remediation.
• Manage third-party integrations, including BI tools (e.g., Power BI) and external metadata/catalog systems.

Enablement & Stakeholder Support

• Facilitate user onboarding, access provisioning, training coordination, and self-service documentation.
• Support SLA/SLO definition, operational reporting, stakeholder communication, and vendor coordination with Databricks account teams.

Qualifications

• 7+ years of cloud or data platform operations experience, including 4+ years supporting Databricks or similar platforms.
• Hands-on Databricks administration experience, including workspaces, clusters/compute policies, jobs, SQL warehouses, repos, runtimes, and Databricks CLI.
• Strong Unity Catalog administration experience (metastores, catalogs/schemas, grants, service principals, external locations, storage credentials).
• Identity & Access Management expertise, including SSO, SCIM provisioning, RBAC, service principals, and least-privilege patterns.
• Security fundamentals covering secrets management, audit logging, access monitoring, and secure connectivity.
• AWS cloud expertise, including IAM roles/policies, S3 security patterns, VPC concepts, logging/monitoring integrations, and KMS.
• Automation experience using Terraform, CLI, scripting, and REST APIs.
• CI/CD experience for jobs, notebooks, and configuration promotion across environments.
• Strong understanding of lakehouse architectures (Delta Lake, separation of storage and compute).
• SQL proficiency and data engineering fundamentals; working knowledge of Python and/or Scala for troubleshooting.
• Experience supporting regulated environments (e.g., FedRAMP, HIPAA, SOC 2) with audit-ready controls and evidence collection.Experience with Databricks serverless compute, Workspace Git integrations (e.g., GitLab), and modern DevOps-enabled workspace features.

• Familiarity with AWS PrivateLink, Secrets Manager/Systems Manager, CloudWatch/CloudTrail, cross-account access patterns.

Education & Certifications

• Bachelor’s degree in a related field or equivalent practical experience.
• Databricks Platform Administrator or Databricks AWS Platform Architect.
• Databricks Certified Data Engineer (Associate or Professional).
• AWS Certified Solutions Architect (Associate or Professional).