Resident Solutions Architect – Databricks

Location: 100% Remote – USA Experience: 15+ Years overall (5+ Years hands-on Databricks Platform experience required) Databricks Certification: Mandatory

Job Summary

We are seeking a hands-on, customer-facing Resident Solutions Architect with deep Databricks Platform expertise, combined with strong CI/CD and Terraform skills. This is an architect-level role for someone who can work directly with enterprise clients, design scalable Lakehouse and data platform solutions, and stay hands-on through build, deployment, and production support — including infrastructure-as-code delivery of the Databricks platform itself.

Key Responsibilities

• Serve as a resident, customer-facing Databricks Solutions Architect, providing architectural guidance, best practices, and hands-on delivery for enterprise Lakehouse implementations.
• Design and implement scalable Lakehouse architectures using PySpark, Spark SQL, and Delta Lake.
• Own Unity Catalog governance and security design, including data access controls, lineage, and cross-workspace governance patterns.
• Build and operate production data pipelines using Delta Live Tables (DLT) and Databricks Workflows.
• Administer Databricks Workspaces and Accounts, including provisioning, access management, and platform configuration.
• Provision and manage Databricks infrastructure using Terraform, including workspace, cluster, Unity Catalog, and job/workflow resources as reusable, version-controlled modules.
• Design and implement CI/CD pipelines for Databricks assets (notebooks, DLT pipelines, jobs, ML models) using GitHub Actions, GitLab CI, Jenkins, Azure DevOps, or equivalent, including Databricks Asset Bundles.
• Perform Spark performance tuning — partitioning, caching, Adaptive Query Execution (AQE), and data skew mitigation — to optimize cost and performance at scale.
• Design and manage SQL Warehouses for BI and analytics workloads.
• Build streaming data solutions using Kafka and Structured Streaming.
• Architect Databricks solutions across AWS, Azure, or Google Cloud Platform, tailored to each cloud's native services and security model.
• Partner directly with client engineers, business stakeholders, and executives to translate business problems into Databricks/Lakehouse solutions and measurable outcomes.
• Guide client teams through migration from legacy ETL platforms (e.g., Informatica) to Databricks.

Required Skills

• 15+ years of experience in Data Engineering, Cloud Engineering, Solutions Architecture, or Platform Engineering.
• Databricks Certification required (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Solutions Architect, or equivalent).
• 5+ years of hands-on Databricks Platform experience, including:
  • PySpark, Spark SQL, and Delta Lake
  • Lakehouse Architecture design and implementation
  • Unity Catalog governance and security
  • Delta Live Tables (DLT)
  • Databricks Workflows
  • Workspace and Account Administration
  • SQL Warehouses
• Strong hands-on Terraform experience with Databricks, including reusable modules for workspace, cluster, and Unity Catalog provisioning.
• Strong CI/CD experience (GitHub Actions, GitLab CI/CD, Jenkins, Azure DevOps, or equivalent), including CI/CD for Databricks Asset Bundles, notebooks, and jobs.
• Strong Spark performance tuning expertise — partitioning, caching, AQE, and data skew resolution.
• Kafka and Structured Streaming experience.
• Strong experience with AWS, Azure, or Google Cloud Platform.
• Proven customer-facing experience providing architectural guidance and Databricks best practices to enterprise clients.
• Strong communication and stakeholder-management skills, with the ability to translate business needs into practical Lakehouse solutions.

Preferred Skills

• MLflow for model tracking and lifecycle management.
• Databricks AI capabilities (AI/BI, Databricks Assistant, Mosaic AI).
• Experience migrating legacy ETL platforms (e.g., Informatica) to Databricks.
• Familiarity with the latest Databricks features — Genie, Lakebase, and Databricks Apps.
• Experience working with enterprise clients in a consulting or professional-services environment.

Kubernetes, Docker, and cloud-native application architecture.