Senior Databricks Administrator (AWS)

Job Title: Senior Databricks Administrator (AWS)

Job Summary

We are seeking an experienced Senior Databricks Administrator to manage, optimize, and govern our Databricks platform in an AWS cloud environment. This role is responsible for ensuring platform reliability, security, performance, and cost efficiency while supporting data engineering and AI/ML workloads. The ideal candidate will have strong expertise in Databricks administration, Apache Spark, cloud infrastructure, and DevOps practices.

Key Responsibilities

• Administer and manage Databricks workspaces in AWS, including setup, configuration, and ongoing maintenance.
• Configure and optimize Databricks clusters, including performance tuning and resource management.
• Manage job scheduling, workflows, and orchestration within Databricks.
• Implement and maintain access controls using IAM, SCIM, and role-based access control (RBAC).
• Enforce cluster policies, governance standards, and best practices across the platform.
• Integrate Databricks with cloud storage solutions such as Amazon S3.
• Monitor platform health, performance, and availability, and troubleshoot issues proactively.
• Ensure data security, encryption, and compliance with organizational and regulatory standards.
• Support CI/CD pipelines and infrastructure automation using tools like Terraform and scripting.
• Collaborate with data engineering and analytics teams to support data pipelines and workloads.
• Optimize cost and resource utilization for Databricks workloads.
• Provide support for Databricks SQL, notebooks, and job execution environments.

Required Qualifications

• 8+ years of experience in Databricks administration in AWS environments.
• Strong knowledge of Databricks workspace management, cluster configuration, and job orchestration.
• Hands-on experience with Apache Spark, including performance tuning and troubleshooting.
• Expertise in IAM, SCIM, and RBAC for access and identity management.
• Experience integrating Databricks with cloud storage (e.g., Amazon S3).
• Solid understanding of data security, encryption, and compliance frameworks.
• Experience with DevOps tools such as Terraform, CI/CD pipelines, and scripting.
• Strong troubleshooting, monitoring, and performance optimization skills.

Preferred Qualifications

• Experience in enterprise or government environments.
• Hands-on experience with Databricks Unity Catalog for governance and access control.
• Knowledge of cost management and optimization strategies in Databricks.
• Experience supporting AI/ML workloads, including Databricks ML and MLflow.
• Familiarity with data lake and lakehouse architectures.
• Working knowledge of Python, SQL, or Scala.

Key Skills

• Databricks Administration
• AWS Cloud (S3, IAM)
• Apache Spark
• Data Governance & Security
• CI/CD & Terraform
• Performance Optimization
• Monitoring & Troubleshooting