Scope of Work – Azure Databricks Platform Administration
The responsibilities for the Azure Databricks platform administration should include, but are not limited to:
• Platform Setup & Configuration
o Provisioning and configuring Azure Databricks workspaces
o Managing clusters, pools, and job configurations
o Integration with Azure services (ADLS, Snowflake, Monte Carlo, Solidatus, Atlan, Key Vault, Kafka, etc.)
• User Access & Security Management
o Managing role-based access control (RBAC) and permissions
o Integration with Azure Active Directory (AAD)
o Implementing security best practices, including data encryption and secrets management
• Cluster & Job Management
o Monitoring cluster performance, uptime, and cost optimization
o Troubleshooting failed jobs and workflows
o Scheduling and managing automated jobs
• Performance Optimization
o Tuning Spark jobs and clusters for optimal performance
o Managing autoscaling and resource utilization
o Identifying and resolving performance bottlenecks
• Monitoring & Incident Management
o Setting up monitoring, alerting, and logging (Azure Monitor, Log Analytics)
o Proactive issue detection and resolution
o Incident response and root cause analysis
• Cost Management & Optimization
o Tracking usage and optimizing costs across environments
o Providing recommendations for resource efficiency
• CI/CD & Release Management
o Supporting deployment pipelines for Databricks notebooks and code
o Version control integration (e.g., Git)
o Managing environment promotion (Dev, Test, Prod)
• Backup, Recovery & Compliance
o Ensuring backup strategies and disaster recovery readiness
o Supporting compliance requirements and audit activities
• Documentation & Support
o Maintaining operational runbooks and documentation
o Providing periodic reporting and status updates
o Offering L2/L3 support for platform-related issues
Never miss an opportunity! Create an alert based on the job you applied for.