Databricks Administrator

Job Title: Databricks Administrator

Job Summary

We are looking for a Databricks Admin with strong hands-on experience in Databricks, AWS, and modern data platforms. The ideal candidate should have experience designing and supporting scalable data pipelines, administering the Databricks platform, optimizing performance and cloud costs, implementing governance, and troubleshooting production issues.

Key Responsibilities

• Design, build, and maintain scalable data pipelines using AWS and Databricks.
• Develop ETL/ELT solutions using PySpark, Spark SQL, Delta Lake, Python, and SQL.
• Create and manage Databricks clusters, SQL Warehouses, and cluster policies.
• Configure and manage Unity Catalog, Metastore, RBAC, and data governance.
• Implement Delta Sharing and support secure data access across teams.
• Monitor and optimize Spark jobs, cluster performance, and Databricks workloads.
• Drive cloud cost optimization (FinOps), including DBU optimization, tagging, budgets, and resource utilization.
• Automate infrastructure and deployments using Terraform, Databricks Asset Bundles, GitHub Actions, or other CI/CD tools.
• Work with AWS services such as S3, Glue, EMR, Lambda, Athena, Redshift, IAM, and CloudWatch.
• Investigate and resolve production issues, perform root cause analysis, and improve platform reliability.
• Collaborate with architects, developers, and business teams to deliver reliable data solutions.

Required Skills

• 5+ years of Data Engineering experience in AWS.
• Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience creating and managing Databricks clusters and platform administration.
• Hands-on experience with Unity Catalog, Metastore, and Databricks governance.
• Experience with Delta Sharing.
• Strong Spark performance tuning and memory optimization skills.
• Experience with AWS services including S3, Glue, EMR, Lambda, Athena, Redshift, IAM, and CloudWatch.
• Strong Python and SQL skills.
• Experience with cloud cost optimization (FinOps) and Databricks DBU optimization.
• Experience with Terraform, Infrastructure as Code (IaC), GitHub Actions, or similar CI/CD tools.
• Strong troubleshooting skills with experience supporting production data pipelines.

Preferred Skills

• Databricks or AWS certifications.
• Experience with MLflow, Delta Live Tables, or Databricks Asset Bundles.
• Experience with AI-driven automation for platform operations.
• Knowledge of modern Lakehouse architecture and cloud data governance.

Interview Expectations

Candidates should be able to explain real-world experience with:

• Databricks cluster creation and optimization.
• Unity Catalog and Metastore setup.
• Delta Sharing implementation.
• Spark performance tuning and memory optimization.
• Databricks cost (DBU) optimization and FinOps practices.
• Terraform and CI/CD automation.
• Resolving slow-running Databricks jobs.
• Troubleshooting and resolving production data pipeline failures with root cause analysis.