Senior Azure Databricks (Platform) Architect

Position: Senior Azure Databricks (Platform) Architect

Location: Remote

Duration: 6 Months Contract

Role Overview:

We're seeking a hands-on Senior Azure Databricks Architect to lead this migration and separation effort end-to-end — covering platform architecture, tenant-to-tenant migration strategy, Unity Catalog and governance, security and networking, Infrastructure as Code, CI/CD, and cutover execution. The ideal candidate has led Azure Databricks migrations, cloud carve-outs, or tenant separation initiatives; data engineering experience is a plus, but deep Databricks architecture and migration expertise is the core requirement.

Key Responsibilities:

Platform Assessment & Migration Planning

•      Assess & document. Audit workspaces, clusters, jobs, secrets, identities, networking, storage, and metastore/Unity Catalog; identify dependencies, risks, and migration blockers.

•      Plan the migration. Define sequencing, cutover approach, validation plan, rollback procedure, and post-migration support model.

Databricks Architecture & Target Environment Design

•      Design target-state architecture. Workspace strategy, network topology, governance model, and Unity Catalog/metastore strategy (external locations, storage credentials, catalog/schema structure, access control) in the Client's tenant.

•      Establish platform standards. Cluster policies, compute patterns, naming conventions, environment promotion, monitoring, and cost management.

Security, Identity & Governance

•      Rebuild identity & access. Microsoft Entra ID, SSO/SCIM, RBAC, service principals, managed identities, and least-privilege access.

•      Re-establish security controls & governance. Key Vault, private endpoints/Private Link, VNet peering, NSGs, storage firewalls, and Unity Catalog governance policies with auditability.

Tenant-to-Tenant Migration Execution

•      Lead migration execution. Move or recreate workspaces, notebooks, jobs, cluster configs, secrets, CI/CD assets, and integrations in the target tenant.

•      Manage cutover & validate readiness. Coordinate migration windows, go/no-go decisions, hypercare, and post-migration validation of access, job execution, security, and monitoring.

Infrastructure Automation & DevOps

•      Automate & rebuild pipelines. Provision environments with Terraform (preferred) or Bicep/ARM; rebuild CI/CD in Azure DevOps or GitHub Actions for platform, notebook, and job deployment.

Data Engineering Support (Secondary)

•      Support pipeline migration. Partner with data engineering on ETL/ELT, Databricks Workflows, and Delta Live Tables affected by the separation; validate data parity and job reliability.

Stakeholder Coordination & Handoff

•      Coordinate & document. Manage dependencies and cutover readiness across infrastructure, security, application, and business teams; produce architecture diagrams, runbooks, and handoff documentation.

Required Qualifications:

Platform Architecture, Migration & Tenant Separation

•      8+ years in cloud data platform architecture or enterprise platform operations, including 5+ years hands-on Azure Databricks architecture, administration, and operations.

•      Proven migration leadership on Azure Databricks tenant-to-tenant migrations, cloud carve-outs, divestitures, or comparable enterprise-scale platform migrations.

•      Cutover and rollback expertise, including migration plans, runbooks, validation checklists, and hypercare — with strong risk management and stakeholder coordination for live production transitions.

•      Deep governance expertise with Unity Catalog or Hive metastore — catalog/schema design, external locations, permissions, and lineage.

•      Proven ability to design scalable, secure, cost-aware Databricks environments across Dev/Test/UAT/Prod with monitoring, runbooks, and support models.

Azure Infrastructure, Security & Automation

•      Strong Azure networking and security experience (VNets, Private Link, NSGs, Key Vault, managed identities, Entra ID).

•      Strong Terraform experience (ARM/Bicep acceptable); CI/CD experience with Azure DevOps or GitHub Actions.

Communication & Leadership

•      Ability to lead technical decisions across platform, security, and data teams, and communicate migration risks, tradeoffs, and readiness decisions to technical and business stakeholders.