The ideal candidate will have deep expertise in Databricks platform administration, AWS cloud architecture, Terraform-based infrastructure automation, and DevOps practices, along with the ability to build proof-of-concept solutions for emerging Databricks capabilities and translate them into scalable enterprise design patterns.
This role will partner closely with data engineering, data science, cloud platform, and governance teams to ensure the platform is secure, scalable, cost-efficient, and aligned with enterprise data strategy.
________________________________________
Key Responsibilities
Databricks Platform Administration
? Design and maintain enterprise Databricks workspace architecture, including networking, security, and compute governance.
? Implement and enforce cluster policies, job policies, compute guardrails, and platform governance standards.
? Manage and optimize Databricks compute environments including job clusters, interactive clusters, and serverless compute.
? Implement Unity Catalog for centralized data governance and access management.
? Manage workspace configurations, platform integrations, and operational reliability.
________________________________________
AWS Cloud Architecture for Databricks
Design and maintain secure and scalable Databricks infrastructure leveraging AWS services including:
? IAM roles and cross-account access architecture
? VPC networking, subnets, and secure connectivity
? PrivateLink and secure endpoint configuration
? S3 data lake architecture
? AWS KMS encryption and key management
? CloudWatch monitoring and logging integration
Ensure the platform aligns with enterprise security, compliance, and networking standards.
________________________________________
Infrastructure as Code & DevOps Automation
? Build and maintain Databricks infrastructure using Terraform.
? Implement infrastructure-as-code frameworks for workspace provisioning, cluster policies, and platform configurations.
? Develop automation using:
o Databricks CLI
o Databricks REST APIs
o Python and shell scripting
? Design and maintain CI/CD pipelines for Databricks workloads and platform infrastructure.
? Integrate deployments with Git-based development workflows (GitHub, GitLab, Azure DevOps).
? Enable standardized environment promotion frameworks across Dev, Test, and Production environments.
________________________________________
Platform Cost Governance & Optimization
? Monitor and analyze DBU consumption and compute utilization across workloads.
? Provide recommendations on cost optimizations
? Implement guardrails such as:
o Cluster policies
o Budget alerts
o Resource tagging
o Usage monitoring dashboards
? Identify and implement cost optimization strategies across:
o Compute configurations
o Job scheduling patterns
o Serverless vs classic compute usage
o Storage lifecycle management
? Build dashboards for cost transparency.
________________________________________
Data Platform Enablement
Provide architectural guidance and best practices for data engineering and data science workloads, including:
Data Engineering
? Delta Lake architecture and performance optimization
? Medallion architecture (Bronze/Silver/Gold)
? Batch and streaming data ingestion
? Large-scale Spark workloads
Data Science & AI
? MLflow experimentation and model lifecycle management
? Feature Store usage
? Machine learning pipelines and model deployment frameworks.
________________________________________
Innovation & Proof of Concept Development
Evaluate and operationalize new Databricks capabilities by building Proof of Concepts (POCs) and translating them into enterprise standards.
Areas of focus include:
? Databricks Serverless Compute
? Delta Live Tables
? Auto Loader
? Lakehouse AI capabilities
? Databricks Feature Store
? Vector Search and LLM integrations
? Lakehouse monitoring and observability frameworks
Define enterprise reference architectures and reusable design patterns based on POC outcomes.
________________________________________
Required Technical Skills
Databricks Platform
? Databricks workspace administration
? Cluster and job policy design
? Unity Catalog governance
? Delta Lake architecture
? Databricks REST APIs and CLI automation
? Performance optimization and troubleshooting
AWS Cloud Platform
Strong hands-on experience with:
? AWS IAM
? VPC networking and security
? S3 data lake architecture
? AWS KMS encryption
? CloudWatch monitoring
? PrivateLink connectivity
? Cross-account access architecture
? Enterprise cloud security practices
________________________________________
Infrastructure & Automation
? Terraform for Databricks and AWS infrastructure
? Infrastructure as Code design patterns
? Databricks CLI and API automation
? Python / Bash scripting
? Platform automation frameworks
________________________________________
DevOps & DataOps
? CI/CD pipeline development
? Git-based version control workflows
? Automated deployment and release processes
? Monitoring, logging, and observability frameworks
? Incident management and root cause analysis
Data Engineering & Analytics Skills
? Apache Spark / PySpark
? SQL optimization
? Delta Lake performance tuning
? Streaming frameworks
? Data pipeline orchestration
? Data modeling and ETL best practices
________________________________________
Preferred Qualifications
? Databricks Certified Data Engineer / Databricks Platfor Architect
? AWS Certified Solutions Architect
? Experience designing enterprise-scale Lakehouse platforms
? Experience supporting large-scale ML workloads
? Experience implementing data governance frameworks
Never miss an opportunity! Create an alert based on the job you applied for.
