Senior Databricks Admin

• Design, implement, and manage scalable and secure Databricks platforms to ensure optimal performance and reliability.
• Create and manage Databricks workspaces and clusters, including right-sizing and lifecycle management of resources.
• Automate infrastructure provisioning and deployment using Infrastructure-as-Code tools like Terraform, GitLab CI/CD, and scripting languages (Python, Bash, PowerShell).
• Monitor platform health, performance, and cost by building dashboards and integrating with monitoring tools.
• Troubleshoot and resolve issues related to cluster configuration, job failures, and performance bottlenecks within Databricks environments.
• Collaborate with cross-functional teams including data scientists, ML engineers, and product teams to deploy and maintain data pipelines and services on Databricks.
• Implement security and governance policies, including user management, access controls, and compliance adherence using Unity Catalog and other Databricks security features.
• Optimize compute and storage resources for cost efficiency without compromising performance, leveraging features like Delta Lake and Photon.
• Stay updated with new Databricks features and coordinate with product teams to evaluate and adopt them.
• Document platform architecture, best practices, troubleshooting guides, and provide mentorship to junior engineers on platform usage and optimization

What you'll need

• Required hands-on experience in architecting, deploying, managing, and supporting Azure Databricks Platform, Azure Datalake and related solutions.
• Required hands-on experience working with Cloud provider tools and architecture (Azure) .
• Required hands-on experience using scripting languages and command-line (CLI) tools such as Ansible, SQL, Terraform, Azure CLI, Databricks CLI, PowerShell and/or Bash.
• Preferred experience in management and tuning of data platforms like Azure ADLS storage and Databricks.
• Knowledge working with and maintaining CI/CD pipelines and versioning tools such as Git preferred.
• Knowledge with a broad set of data ingestion and data pipeline patterns, and associated tools like (Qlik Replicate, Qlik Compose, Databricks, Kafka, Matillion ELT and Snowflake Services)
• Experience with working in an agile product development team

Excellent communication skills and ability to articulate system designs and patterns to varying levels of leadership