Job Summary
We are seeking an experienced Databricks Administrator responsible for administering, configuring, and supporting the Databricks platform in an AWS cloud environment to enable scalable data engineering, analytics, and AI/ML workloads.
This role focuses on ensuring platform reliability, security, compliance, governance, and cost optimization, while supporting data engineers, analysts, and data scientists across enterprise data initiatives.
The ideal candidate will manage workspace configuration, cluster policies, access controls, cloud integrations, and platform monitoring to support ongoing enterprise analytics operations.
Key Responsibilities
- Understand business objectives, technical challenges, and identify alternative solutions.
- Perform studies and cost/benefit analysis for platform and architecture improvements.
- Analyze user requirements, operational procedures, and existing system capabilities.
- Recommend enhancements to automate processing and improve current systems.
- Collaborate with business and technical teams to understand:
- Data input requirements
- Reporting needs
- Operational workflows
- Output formats
- Prepare detailed technical documentation covering:
- User needs
- Program functions
- Development steps
- System modifications
- Review current platform capabilities, specifications, and scheduling limitations to determine feasibility of requested changes.
Databricks Administration Responsibilities
- Administer and support Databricks workspaces in AWS cloud environment
- Configure and manage Databricks clusters, job scheduling, and workspace settings
- Manage user access, permissions, and role assignments using:
- Implement and enforce:
- Cluster policies
- Workspace governance standards
- Security controls
- Integrate Databricks with cloud storage services such as:
- Amazon Web Services Amazon S3
- Monitor platform health, performance, and availability
- Troubleshoot cluster issues and Spark performance bottlenecks
- Support Databricks SQL, notebooks, and job orchestration
- Ensure compliance with enterprise data security and encryption standards
- Support automation using:
- Terraform
- CI/CD pipelines
- Scripting tools
- Optimize platform usage and control infrastructure costs
CANDIDATE SKILLS AND QUALIFICATIONS
Minimum Requirements:
Candidates that do not meet or exceed the minimum stated requirements (skills/experience) will be displayed to customers but may not be chosen for this opportunity. |
Years | Required/Preferred | Experience |
8 | Required | Experience administering Databricks workspaces in a cloud environment - AWS |
8 | Required | Strong understanding of Databricks cluster configuration, job scheduling, and workspace management. |
8 | Required | Experience managing user access, roles, and permissions using IAM, SCIM, and role-based access control (RBAC). |
8 | Required | Proficiency with Apache Spark concepts, including performance tuning and troubleshooting. |
8 | Required | Experience integrating Databricks with cloud storage services (e.g., S3,) |
8 | Required | Experience implementing and enforcing cluster policies and workspace governance standards. |
8 | Required | Familiarity with Databricks SQL, notebooks, and job orchestration. |
8 | Required | Experience monitoring platform health, performance, and availability. |
8 | Required | Understanding of data security, encryption, and compliance requirements. |
8 | Required | Experience with DevOps or automation tools (Terraform, CI/CD pipelines, scripting). |
4 | Preferred | Experience administering Databricks in an enterprise or government environment. |
4 | Preferred | Experience with Databricks Unity Catalog for data governance and access control. |
4 | Preferred | Knowledge of cost management and optimization for Databricks workloads. |
4 | Preferred | Experience supporting AI/ML workloads using Databricks ML and MLflow. |
4 | Preferred | Familiarity with data lake and lakehouse architectures. |
4 | Preferred | Knowledge of Python, SQL, or Scala for administration and troubleshooting. |
Never miss an opportunity! Create an alert based on the job you applied for.
