Principal Solution Architect - Data and AI Platform

Introduction

We are seeking a Principal Solution Architect – Data & AI Platform to lead the design and enablement of AI-ready data capabilities on a modern Databricks Lakehouse foundation. This role will focus on building the architecture patterns required to support LLM-powered analytics, conversational AI, agentic AI workflows, governed data products, semantic layers, metadata intelligence, RAG, vector search, and secure enterprise data access.

Responsibilities

• Define and drive the Data and AI platform architecture using Databricks, Delta Lake, Unity Catalog, and modern lakehouse design patterns.
• Architect secure LLM and GenAI enablement patterns, including RAG, embeddings, vector search, semantic search, prompt orchestration, model serving, grounding, and AI evaluation.
• Design agentic AI capabilities to support data platform operations such as metadata discovery, catalog enrichment, lineage analysis, data quality investigation, documentation generation, pipeline troubleshooting, test generation, and operational runbook support.
• Establish architecture standards for AI-ready data products, certified metrics, semantic layers, business definitions, metadata, lineage, data contracts, and reusable data engineering patterns.
• Guide modernization of key workloads into a scalable Databricks Lakehouse architecture using Bronze, Silver, and Gold data zones, governed Delta patterns, workflows, observability, and CI/CD-enabled delivery.
• Define responsible AI and governance controls, including human-in-the-loop approvals, prompt and response governance, auditability, least-privilege access, entitlement-aware responses, and sensitive data protection.
• Partner with AI/ML and data engineering teams to productionize AI-driven insights and recommendations across business domains such as merchant analytics, authorization, disputes, chargebacks, interchange, risk, fraud, and operational anomalies.
• Provide technical leadership through architecture reviews, solution design, standards creation, proof-of-concepts, reusable frameworks, and implementation guidance.
• Collaborate with product and business stakeholders to translate strategic data and AI needs into practical platform capabilities and delivery roadmaps.

Requirements

Required Qualifications

• 10+ years of experience in data architecture, solution architecture, data engineering, analytics platforms, or AI/data platforms.
• Strong hands-on Databricks experience, including Delta Lake, Unity Catalog, Spark/PySpark, SQL, Databricks Workflows, medallion architecture, performance optimization, and production data engineering patterns.
• Experience designing and implementing LLM/GenAI solutions, including RAG, vector search, embeddings, conversational AI, prompt engineering, model serving, AI evaluation, and hallucination mitigation.
• Understanding of agentic AI architecture, including tool use, workflow automation, agent guardrails, approval patterns, audit trails, and safe execution models.
• Strong knowledge of data architecture concepts such as data products, semantic layers, data contracts, metadata, cataloging, lineage, observability, data quality, and governance.
• Experience designing secure data platforms using RBAC, ABAC, row-level security, column-level security, masking, encryption, audit logging, and entitlement-aware data delivery.
• Ability to create architecture diagrams, standards, decision records, implementation playbooks, and reusable design patterns.
• Strong communication and influencing skills with the ability to work across engineering, AI/ML, product, governance, security, operations, and executive stakeholders.

Preferred Qualifications

• Experience with Databricks Mosaic AI, MLflow, Model Serving, Vector Search, Feature Store, Lakehouse Monitoring, or advanced Unity Catalog capabilities.
• Experience with Azure data and AI services such as ADLS, Synapse, Data Factory, Event Hubs, Azure OpenAI, Azure AI Search, Azure API Management, Azure DevOps, or GitHub Actions.
• Background in payments, merchant acquiring, financial services, merchant analytics, authorization, disputes, chargebacks, interchange, settlement, funding, risk, or fraud.
• Familiarity with LLM and agent frameworks such as LangChain, LangGraph, Semantic Kernel, LlamaIndex, AutoGen, CrewAI, or similar technologies.
• Experience with enterprise cataloging and governance tools such as Microsoft Purview, Collibra, Alation, Informatica, or Unity Catalog.
• Experience building AI assistants, data copilots, analytics copilots, recommendation engines, or self-service analytics platforms.
• Databricks, Azure, AI/ML, data engineering, or cloud architecture certifications are a plus.