Resident Solution Architect – Databricks & Spark

Job Title: Databricks Resident Solutions Architect (RSA)

Experience: 12+ Years (with min. 5+ years deep specialization in Databricks/Spark)
Location: Remote

Role Overview

We are looking for an experienced Resident Solution Architect with strong expertise in the Databricks Lakehouse Platform to design, architect, and deliver scalable, secure, and production-grade Data & AI solutions.

The ideal candidate should have deep hands-on technical expertise, strong consulting experience, and proven experience leading enterprise-level data platform implementations and migrations across cloud environments.

 

Key Responsibilities

1. Solution Architecture & Delivery

• Design and build production-ready reference architectures using Lakehouse and Delta Lake best practices
• Architect scalable big data and AI solutions on Databricks
• Lead migrations (ETL/ELT, data warehouses, legacy systems) to modern Lakehouse architecture
• Provide architecture consulting, cluster optimization, and performance tuning
• Implement data governance and security using Unity Catalog

 

2. Customer Engagement & Delivery Management

• Scope, plan, and manage technical engagements
• Drive end-to-end project delivery (Design → Development → Deployment → Optimization)
• Manage timelines, risks, and deliverables
• Provide support for complex production issues

 

3. Platform Engineering & DevOps

• Implement CI/CD pipelines for code and infrastructure
• Deploy infrastructure using Terraform and Databricks Asset Bundles (DAB)
• Set up job scheduling, monitoring, and production management
• Establish best practices for version control and automation

 

4. Optimization & Continuous Improvement

• Monitor and optimize data pipelines and ML models
• Improve system performance and cost efficiency
• Contribute reusable assets and documentation
• Enable customer teams through knowledge transfer

 

5. Leadership & Enablement

• Mentor and train customer teams
• Provide technical leadership across engagements
• Support pre-sales activities and architecture discussions

 

Required Skills & Qualifications

Experience (Mandatory)

12+ years of hands-on experience in:

• Data Engineering
• Data Platforms
• Data Analytics
• Data Warehousing
• Big Data technologies (Kafka, Data Lakes, Cloud-native solutions)

 

Cloud Expertise

Hands-on experience in at least one:

• AWS
• Microsoft Azure
• Google Cloud Platform (GCP)

 

Programming Skills

• Python
• SQL
• Scala

 

Databricks Expertise

Strong hands-on experience with:

• Databricks SQL
• Apache Spark
• Delta Lake
• MLflow
• Unity Catalog
• Delta Live Tables (DLT)

 

Migration & Architecture

• Experience leading enterprise workload migrations
• Strong knowledge of ETL/ELT design patterns
• Experience modernizing legacy data systems

 

Deployment & Automation

• Databricks Asset Bundles (DAB)
• Terraform
• CI/CD pipelines
• Infrastructure as Code (IaC)

 

Certifications 

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional (Preferred)

 

Soft Skills

• Excellent communication
• Strong stakeholder management
• Leadership & mentoring ability
• Consulting and problem-solving skills