Databricks Administrator

Databricks Administrator
Location : Remote
Experience
7+ Years of IT Experience with 3 5+ years of Databricks Administration
Job Summary
We are seeking an experienced Databricks Administrator to manage, secure, optimize, and support enterprise Databricks environments. The ideal candidate will have expertise in Databricks workspace administration, Unity Catalog, cluster management, security, performance optimization, automation, and cloud platforms such as Azure, AWS, or Google Cloud Platform. The role involves working closely with data engineers, data scientists, architects, security teams, and cloud administrators to deliver a secure and scalable data analytics platform.
Key Responsibilities
Databricks Administration
Install, configure, and administer Databricks workspaces.
Manage multiple Databricks environments (Development, QA, UAT, Production).
Configure and maintain clusters and cluster policies.
Manage SQL Warehouses and Databricks Jobs.
Monitor workspace health and performance.
Upgrade and maintain Databricks Runtime versions.
Optimize cluster utilization and reduce infrastructure costs.
Security & Governance
Implement Unity Catalog governance.
Configure Role-Based Access Control (RBAC).
Manage users, groups, service principals, and permissions.
Integrate Databricks with Azure AD, AWS IAM, or Okta.
Configure Single Sign-On (SSO).
Manage secrets using Databricks Secret Scopes or cloud key management services.
Ensure compliance with organizational security policies.
Workspace Administration
Create and manage workspaces.
Configure notebooks and repositories.
Manage Git integration.
Administer workspace permissions.
Manage cluster access policies.
Configure job permissions.
Manage libraries and dependencies.
Infrastructure Management
Deploy Databricks using Terraform.
Automate infrastructure provisioning.
Configure networking:
VPC/VNet
Private Link
Secure Cluster Connectivity
NSG/Security Groups
Firewalls
Configure storage accounts (ADLS, S3).
Manage DBFS and external storage.
Performance Optimization
Tune Spark clusters.
Optimize autoscaling.
Configure instance pools.
Monitor cluster performance.
Troubleshoot slow jobs.
Optimize notebook execution.
Manage caching strategies.
Monitoring & Troubleshooting
Monitor Databricks clusters.
Configure alerts.
Review audit logs.
Troubleshoot:
Cluster failures
Job failures
Authentication issues
Permission issues
Connectivity issues
Library conflicts
Performance bottlenecks
Automation
Automate workspace deployment.
Automate cluster creation.
Automate user provisioning.
Automate permission management.
Automate backup processes.
Automate scheduled jobs using Terraform, Azure DevOps, GitHub Actions, or Jenkins.
CI/CD
Implement CI/CD pipelines.
Git integration.
Azure DevOps Pipelines.
GitHub Actions.
Jenkins.
Deployment automation.
Release management.
Data Governance
Unity Catalog
Data Lineage
Catalog Management
Schema Management
Access Policies
Table Permissions
Row-Level Security
Column-Level Security
Data Masking
Backup & Disaster Recovery
Backup workspace configurations.
Backup notebooks.
Backup cluster configurations.
Disaster recovery planning.
Workspace migration.
Restore procedures.
Required Skills
Databricks
Databricks Workspace Administration
Unity Catalog
Cluster Administration
SQL Warehouses
Jobs
Notebooks
Repos
DBFS
Delta Lake
Delta Live Tables
MLflow
Apache Spark
Spark Administration
Spark SQL
Performance Tuning
Cluster Optimization
Cloud Platforms
Azure
AWS
Google Cloud Platform
Infrastructure as Code
Terraform
ARM Templates
CloudFormation
Scripting
Python
Bash
PowerShell
DevOps
Git
Azure DevOps
Jenkins
GitHub Actions
CI/CD
Monitoring
Azure Monitor
CloudWatch
Datadog
Prometheus
Grafana
Splunk
Security
Azure AD
AWS IAM
OAuth
SAML
LDAP
RBAC
Secret Scopes
Key Vault
AWS Secrets Manager
Databases
SQL Server
Snowflake
Oracle
PostgreSQL
MySQL
Preferred Qualifications
Databricks Certified Data Engineer Associate/Professional
Databricks Certified Administrator (where applicable)
Azure Administrator Associate
AWS Solutions Architect Associate
Experience with Lakehouse Architecture
Experience with Unity Catalog Governance
Experience with Delta Lake Optimization
Knowledge of Apache Iceberg
Experience supporting enterprise data platforms
Soft Skills
Strong analytical and troubleshooting skills
Excellent verbal and written communication
Ability to work with cross-functional teams
Strong documentation skills
Ability to support production environments
Customer-focused mindset
Experience working in Agile/Scrum environments
Nice-to-Have Skills
Azure Synapse Analytics
Microsoft Fabric
Kafka
Airflow
dbt
Kubernetes
Docker
Python-based automation
AI/ML platform administration
Cost optimization and FinOps experience