Solution Architect Data Engineering (Azure Databricks)

Solution Architect – Data Engineering (Azure Databricks)

Experience: 10+ Years

Education: Bachelor’s Degree (4-year) – Computer Science / Engineering / Related Field

 

 

 

Role Summary

• We are seeking a Solution Architect to design and drive enterprise-scale data engineering architectures on Azure Databricks and cloud platforms. This role will define end-to-end data solutions, architecture patterns, and engineering standards to support analytics, AI/ML, and operational use cases.
• The ideal candidate will combine deep technical expertise with strong business alignment, enabling scalable, secure, and high-performing data platforms.
• Solution architecture is critical to ensuring standardized, scalable, and governed data ecosystems across domains and use cases

 

Key Responsibilities

1. Architecture & Solution Design

• Define and implement end-to-end data architecture (ingestion → transformation → consumption)
• Design scalable Lakehouse architectures (Databricks, Delta Lake)
• Develop reference architectures, reusable frameworks, and design patterns
• Align architecture with performance, scalability, security, and cost optimization goals

 

2. Data Modeling & Domain Architecture

• Define canonical data models, dimensional schemas, and semantic layers
• Architect enterprise data across domains:
• Claims, Member, Provider, Eligibility
• Ensure data consistency, interoperability, and reuse across teams

 

3. Azure Databricks & Cloud Enablement

• Architect solutions using:

• Azure Databricks (Mandatory)

• Azure Data Services (ADF, ADLS, Synapse)
• Guide engineering teams on:
• Spark optimization, Delta Lake design, and data pipeline best practices
• Enable multi-cloud integrations when required

 

4. Governance, Security & Compliance

• Define and enforce:
• Data governance frameworks (lineage, quality, metadata)
• Access control, RBAC, and data security policies
• Ensure compliance with:
• HIPAA, HITRUST, and enterprise regulatory standards
• Establish data quality SLAs and auditability mechanisms

 

5. Engineering & Platform Alignment

Work closely with:

• Data Engineering teams
• Platform Engineering teams
• AI/ML Engineering teams
• Provide architectural oversight for pipelines, workflows, and platform capabilities
• Drive standardization across CI/CD, DataOps, and MLOps practices

 

6. Stakeholder & Business Engagement

• Collaborate with business, product, and analytics teams to:
• Translate requirements into scalable technical solutions
• Lead architecture reviews, design approvals, and governance forums (ARB)
• Communicate architecture decisions and trade-offs to leadership

 

Required Skills

Strong experience in:

• Azure Databricks (Mandatory) – PySpark, Delta Lake
• Cloud Architecture (Azure preferred)

Deep expertise in:

• Data Architecture & Data Modeling (OLTP, OLAP, Dimensional)
• Lakehouse / Data Warehouse design patterns
• Hands-on experience with:
• ETL/ELT pipelines and orchestration frameworks
• CI/CD and DevOps practices (Azure DevOps, Git)
• Metadata, lineage, and governance tools (Unity Catalog, Purview)

 

Preferred Skills

Experience with:

• AI/ML data architectures (feature stores, model pipelines)
• Multi-cloud environments (Azure + Google Cloud Platform/Snowflake)

Strong understanding of:

• Medallion architecture (Bronze–Silver–Gold)
• Domain Experience (Good to Have)
• US Healthcare ecosystem:
• Claims, Member, Provider datasets

Familiarity with:

• FHIR / HL7 / EMR data models
• Payment Integrity / Risk Adjustment use cases