Job Title: Azure Databricks & Agentic AI Architect
Location: - Chicago, IL(Hybrid- 3 days/week in office)
Employment Type: Long term contract
Job Description:-
Ideal Candidate Profile
A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.
Mandatory skills
- Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering
We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.
The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.
Key Responsibilities
Agentic Data Engineering Leadership
- Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
- Define autonomous workflows using AI Agents for:
- Data ingestion
- Data mapping
- Schema evolution
- Data quality remediation
- Metadata Enrichment
- Pipeline optimization
- Root cause analysis
- Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.
AI-Driven Data Lifecycle (AI-DLC)
- Lead architecture for AI-enabled Data Development Lifecycle across:
- Requirement analysis
- Data modeling
- Pipeline generation
- Automated testing
- Code review
- Documentation
- Deployment
- Monitoring
- Implement AI copilots to accelerate developer productivity.
- Enable automated lineage creation and intelligent impact analysis.
Lakehouse & Data Platform Architecture
- Design scalable Lakehouse platforms using:
- Azure Databricks
- Delta Lake
- Unity Catalog
- ADLS Gen2
- Databricks Workflows
- Delta Live Tables
Enterprise GenAI Integration
- Architect RAG-based solutions using enterprise data assets.
- Design agent orchestration frameworks using:
- Azure OpenAI
- LangGraph
- Semantic Kernel
- AutoGen
- MCP-enabled architectures
- Build domain-specific AI agents supporting Data Engineering and Analytics teams.
AI Governance & Responsible AI
- Define guardrails for enterprise GenAI adoption.
- Implement:
- Prompt governance
- Observability
- Cost monitoring
- Auditability
- Explainability
- Security controls
- Establish governance models for autonomous AI agents.
AI-Powered Platform Optimization
- Design self-healing data pipelines.
- Implement AI-driven:
- Incident triage
- Failure prediction
- Capacity planning
- Cost optimization
- SLA monitoring
- Enable intelligent workload placement and model routing.
Technical Skills
- Data Platform
- Azure Databricks
- Delta Lake
- Unity Catalog
- Azure Data Factory
AI & Agentic Frameworks
- Azure OpenAI
- Knowledge Graph
- RAG Architecture
- LangChain
- LangGraph
- MCP Protocol
- Vector Databases
- AI Agent Orchestration
Data Engineering
- PySpark
- Spark SQL
- Python
- SQL
- ELT/ETL Modernization
DevOps & AI-DLC
- Azure DevOps
- GitHub Actions
- CI/CD
- MLOps
- LLMOps
- Evaluation Frameworks
- AI Testing Frameworks
Leadership Expectations
- Drive AI-First Data Engineering transformation.
- Define enterprise patterns, accelerators, and reusable AI agents.
- Mentor architects, data engineers, and AI engineers.
- Lead executive conversations on AI adoption, ROI, and transformation roadmaps.
Preferred Certifications
- Databricks Certified Data Engineer Professional
- Databricks Certified Solution Architect
- Microsoft Azure Solution Architect (AZ-305)
- Azure Data Engineer (DP-203)
- Microsoft Applied Skills - Azure OpenAI
- Generative AI / Agentic AI Certifications
Success Metrics
- 30-50% Data Engineering productivity improvement.
- Reduction in manual pipeline development effort.
- Improved data quality and governance compliance.
- Measurable ROI from Agentic AI adoption.
- Expansion of reusable AI agents across programs.
Never miss an opportunity! Create an alert based on the job you applied for.