Job Title: Azure Databricks & Agentic AI Architect
Location : Chicago IL
Mandatory skills
Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering
Role Summary:
We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.
The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.
Key Responsibilities
Agentic Data Engineering Leadership
Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
Define autonomous workflows using AI Agents for:
Data ingestion
Data mapping
Schema evolution
Data quality remediation
Metadata Enrichment
Pipeline optimization
Root cause analysis
Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.
AI-Driven Data Lifecycle (AI-DLC)
Lead architecture for AI-enabled Data Development Lifecycle across:
Requirement analysis
Data modeling
Pipeline generation
Automated testing
Code review
Documentation
Deployment
Monitoring
Implement AI copilots to accelerate developer productivity.
Enable automated lineage creation and intelligent impact analysis.
Lakehouse & Data Platform Architecture
Design scalable Lakehouse platforms using:
Azure Databricks
Delta Lake
Unity Catalog
ADLS Gen2
Databricks Workflows
Delta Live Tables
Enterprise GenAI Integration
Architect RAG-based solutions using enterprise data assets.
Design agent orchestration frameworks using:
Azure OpenAI
LangGraph
Semantic Kernel
AutoGen
MCP-enabled architectures
Build domain-specific AI agents supporting Data Engineering and Analytics teams.
AI Governance & Responsible AI
Define guardrails for enterprise GenAI adoption.
Implement:
Prompt governance
Observability
Cost monitoring
Auditability
Explainability
Security controls
Establish governance models for autonomous AI agents.
AI-Powered Platform Optimization
Design self-healing data pipelines.
Implement AI-driven:
Incident triage
Failure prediction
Capacity planning
Cost optimization
SLA monitoring
Enable intelligent workload placement and model routing.
Technical Skills
Data Platform
Azure Databricks
Delta Lake
Unity Catalog
Azure Data Factory
AI & Agentic Frameworks
Azure OpenAI
Knowledge Graph
RAG Architecture
LangChain
LangGraph
MCP Protocol
Vector Databases
AI Agent Orchestration
Data Engineering
PySpark
Spark SQL
Python
SQL
ELT/ETL Modernization
DevOps & AI-DLC
Azure DevOps
GitHub Actions
CI/CD
MLOps
LLMOps
Evaluation Frameworks
AI Testing Frameworks
Leadership Expectations
Drive AI-First Data Engineering transformation.
Define enterprise patterns, accelerators, and reusable AI agents.
Mentor architects, data engineers, and AI engineers.
Lead executive conversations on AI adoption, ROI, and transformation roadmaps.
Preferred Certifications
Databricks Certified Data Engineer Professional
Databricks Certified Solution Architect
Microsoft Azure Solution Architect (AZ-305)
Azure Data Engineer (DP-203)
Microsoft Applied Skills Azure OpenAI
Generative AI / Agentic AI Certifications
Success Metrics
30-50% Data Engineering productivity improvement.
Reduction in manual pipeline development effort.
Improved data quality and governance compliance.
Measurable ROI from Agentic AI adoption.
Expansion of reusable AI agents across programs.
Ideal Candidate Profile
A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.
Experience: - 16-20 Years
Location: - Chicago, IL(Hybrid- 3 days/week in office)
Educational Qualifications: -
- Engineering Degree BE/ME/BTech/MTech/BSc/MSc.
Technical certification in multiple technologies is desirable.
Never miss an opportunity! Create an alert based on the job you applied for.