Databricks SME / Azure Databricks Platform Lead-NYC, NY – 4 Days onsite/Week

Note: The candidate must have excellent communication skills, as they will be reporting directly to senior management.

Overview:

Our client is seeking a Databricks Subject Matter Expert (SME) to lead the redesign and implementation of a new Azure Databricks platform. The current environment requires significant improvement, and this role will be responsible for establishing a scalable, secure, and enterprise-grade Databricks architecture from the ground up.

This individual will act as the technical lead and platform owner, driving best practices across architecture, governance, security, and operational excellence while partnering with engineering, data, and business teams.

This initiative will have significant senior leadership visibility and is considered a critical transformation effort within the firm.

Key Responsibilities:

• Lead the end-to-end implementation of a new Azure Databricks platform, redesigning the current environment to meet enterprise standards for scalability, security, and performance
• Define and implement best-practice architecture for Databricks, including hub-and-spoke networking, private endpoints, secure cluster policies, and workspace isolation
• Establish and own Databricks governance using Unity Catalog, including fine-grained access control, data lineage, data sharing, and compliance alignment
• Build and manage infrastructure using Terraform (IaC) to provision and standardize Azure and Databricks environments across development, test, and production
• Develop CI/CD pipelines to support automated deployment of Databricks workspaces, jobs, and infrastructure components
• Partner with data engineering and analytics teams to enable scalable Lakehouse architectures and optimize workloads for performance and cost efficiency
• Implement operational best practices, including monitoring, alerting, performance tuning, and cost optimization across Databricks workloads
• Ensure platform security and compliance, integrating Azure services such as Entra ID (Azure AD), Key Vault, and secure networking configurations (VNet injection, NSGs, Private Link)
• Act as the Databricks SME and technical advisor, guiding architectural decisions and mentoring internal teams on platform usage and best practices
• Present architectural designs, progress updates, and recommendations to senior leadership, clearly communicating complex technical concepts to both technical and non-technical stakeholders

Required Qualifications:

• Proven experience as a Databricks SME / Platform Engineer, with hands-on ownership of Azure Databricks implementations
• Deep expertise with Databricks Unity Catalog (required), including governance, lineage, and cross-workspace data management
• Strong experience designing and deploying enterprise Databricks architectures in Azure, including secure networking and workspace design
• Advanced knowledge of Terraform and Infrastructure as Code (IaC) for provisioning cloud and Databricks environments
• Experience building and managing CI/CD pipelines for data platforms and infrastructure deployments
• Strong understanding of Azure services, including Entra ID (Azure AD), VNet injection, NSGs, Private Link, Key Vault, and storage integrations (ADLS)
• Demonstrated experience in performance optimization, cost management, and operational reliability for large-scale data platforms
• Exceptional presentation and communication skills, with the ability to confidently engage senior stakeholders and clearly articulate technical strategy, trade-offs, and progress
• Ability to work cross-functionally and lead platform transformation initiatives in complex enterprise environments

Preferred Qualifications:

• Experience working in financial services, asset management, or regulated environments
• Background in Lakehouse architecture and large-scale data platform modernization
• Prior experience remediating or re-architecting underperforming data platforms

Key Traits for Success:

• Strong ownership mindset with the ability to lead from design through implementation
• Hands-on technical leadership combined with strategic thinking
• Ability to operate in fast-paced environments and drive change across teams
• Executive presence, with the ability to present to and influence senior leadership

Summary:

This is a high-impact role for a Databricks expert who can lead a platform transformation, replacing an underperforming environment with a modern, scalable, and well-governed Azure Databricks ecosystem.

Given the senior visibility and strategic importance of this initiative, strong communication, presentation, and stakeholder management skills are just as critical as technical expertise.