Data AI Architect

Job Description - Data AI Architect

We are looking for a hands-on Data & AI Architect to design, build, and govern the foundational data infrastructure. This is a senior individual contributor and technical leadership role someone who can architect at scale and remain deeply engaged in execution.

Responsibilities

• Drive the adoption of AI-assisted data engineering practices including code generation, automated testing, and documentation to accelerate team velocity
• Design and own end-to-end data architecture for the platform, including the lakehouse, medallion layers, and data contracts across domains
• Lead the architecture and implementation of Databricks environments on AWS including Unity Catalog governance, Delta Live Tables, and MLflow for model lifecycle
• Define and enforce data modeling standards: conceptual, logical, and physical models for insurance data domains (policies, claims, exposure, renewals)
• Architect real-time and batch data pipelines that ingest, transform, and serve data to AI models, analytics dashboards, and client-facing applications
• Partner with data scientists, ML engineers, and product teams to embed AI/ML models into production data workflows and ensure inference pipelines are scalable and observable
• Establish data quality, lineage, and observability frameworks using tools like Great Expectations, dbt, or equivalent
• Define cloud infrastructure patterns on AWS (S3, Glue, EMR, Lambda, Redshift, SageMaker) aligned with security, cost, and performance targets
• Mentor data engineers and analysts; set the technical bar and review architecture across the team

Skillset needed

• 12+ years of experience in data engineering or data architecture roles, with at least 5 years in a senior/lead/architect capacity
• Deep expertise in Databricks Delta Lake, Unity Catalog, DLT, Workflows, MLflow; Databricks certifications preferred
• Strong hands-on AWS experience: S3, Glue, IAM, Redshift, SageMaker, Lambda, VPC networking
• Mastery of data modeling dimensional, relational, and NoSQL patterns; experience with dbt a strong plus
• Proficiency in Python and SQL for pipeline development, transformation logic, and model integration
• Experience designing lakehouse architectures (medallion/bronze-silver-gold patterns) in enterprise contexts
• Experience building AI for data engineering
• Familiarity with AI/ML integration patterns feature stores, model serving, drift monitoring, A/B frameworks
• Strong understanding of data governance: lineage, cataloging, access control, PII handling
• Insurance or financial services domain experience is a meaningful plus understanding of policy, claims, exposure, or renewal data structures
• Excellent communication skills; able to influence cross-functional stakeholders and explain architectural decisions to both engineers and business leaders

Nice to have

• Experience with Kafka or event streaming platforms for real-time data architectures
• Exposure to LLM integration patterns RAG pipelines, vector databases, prompt orchestration (LangChain, LlamaIndex)
• Knowledge of insurance data standards or regulatory data frameworks