Location: San Francisco, CA (Remote)
Work Arrangement: Remote
Employment Type: Contract
Duration: 8–12 Months
Experience Required: 12–15+ Years
Position Overview
We are seeking experienced Databricks Solution Architects with strong banking industry experience and expertise in enterprise data engineering, cloud data platforms, and modern data warehouse solutions. The ideal candidates will have successfully delivered multiple end-to-end Databricks implementations and possess hands-on architecture and development experience. This role requires deep knowledge of the Databricks Lakehouse Platform, Apache Spark, Delta Lake, cloud-native data solutions, and enterprise data platform modernization.
Primary Skills
Databricks Architecture
Databricks Lakehouse Platform
Apache Spark
Data Engineering
Delta Lake
Cloud Data Platforms
Secondary Skills
Structured Streaming
ETL/ELT Pipelines
Unity Catalog
Databricks Workflows
MLflow
Delta Live Tables
MLOps
DevOps and CI/CD
AWS
Microsoft Azure
Google Cloud Platform (Google Cloud Platform)
Key Responsibilities
-
Design and implement scalable, cloud-native data platforms using Databricks.
-
Lead end-to-end Databricks implementations, from architecture and design through development and production deployment.
-
Develop and optimize large-scale ETL/ELT pipelines using Databricks, Delta Lake, Structured Streaming, and modern data engineering frameworks.
-
Design enterprise data architecture aligned with banking business requirements and technical standards.
-
Optimize Apache Spark workloads through partitioning strategies, execution plan analysis, memory management, and performance tuning.
-
Implement and manage Databricks Lakehouse capabilities, including Unity Catalog, Workflows, MLflow, and Delta Live Tables.
-
Establish DevOps and CI/CD practices for production-grade data engineering solutions.
-
Apply MLOps principles to support machine learning lifecycle management, model deployment, and monitoring.
-
Troubleshoot complex data platform challenges and recommend scalable, reliable, and cost-efficient architecture solutions.
-
Provide technical leadership and architectural guidance to engineering and delivery teams.
Required Qualifications
-
12–15+ years of experience in data engineering, data platforms, data analytics, and modern data warehouse solutions.
-
10+ years of consulting and client-facing delivery experience.
-
Demonstrated success delivering at least 6–8 end-to-end Databricks implementations.
-
Hands-on experience as a Databricks developer, technical lead, or solution architect.
-
Strong expertise in cloud-native data platform architecture across AWS, Azure, or Google Cloud Platform.
-
Deep hands-on proficiency in at least one major cloud platform.
-
Advanced knowledge of Apache Spark, including performance optimization, partitioning, execution plans, memory management, and runtime internals.
-
Extensive experience with Databricks, Delta Lake, Structured Streaming, and scalable ETL/ELT pipelines.
-
Experience implementing CI/CD and DevOps practices using Azure DevOps, GitHub Actions, GitLab CI/CD, or Jenkins.
-
Working knowledge of MLOps, machine learning lifecycle management, model deployment, and monitoring.
-
Strong understanding of enterprise data architecture, distributed processing, and performance optimization.
-
Relevant experience in the banking or financial services industry.
-
Strong problem-solving and solution architecture capabilities.
Preferred Qualifications
-
Databricks Data Engineering Professional certification or an equivalent advanced Databricks certification.
-
Completion of recommended Databricks learning paths and coursework.
-
Databricks Solutions Architect Champion program participation.
-
Experience with advanced Databricks Lakehouse Platform capabilities and enterprise implementations.
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, and gender identity), national origin, age, disability, genetic information, protected veteran status, or any other characteristic protected by applicable federal, state, or local law.
We are committed to providing reasonable accommodations to qualified applicants with disabilities and for sincerely held religious beliefs, as required by applicable law. Applicants who require a reasonable accommodation during the hiring process may request assistance.
Employment may be contingent upon the successful completion of job-related pre-employment screening, which may include background and reference checks and, where permitted by applicable law and applicable to the position, drug testing. All such screening will be conducted in accordance with applicable federal, state, and local laws.
