The role You own the spec, the system design and the guardrails for every agentic solution the POD ships. You turn a business epic into an approved spec with tests and design the AI system behind it. The engineers build it with coding agents under your direction. You sign off the design and the evaluation evidence before release.
What you will do
· Refine epics with the product owner, split them into testable stories and write acceptance criteria.
· Draft specs with AI agents and weigh design options. Choose the right Azure approach for each use case: Copilot Studio, Azure AI Foundry Agents, Microsoft Agent Framework, or a combination.
· Design the AI system end to end:
· RAG: chunking, indexing, hybrid search, reranking, grounding and citations.
· Multi-agent: when to use one agent and when several, orchestration patterns, tool contracts, hand-offs and failure handling.
· LLM selection and handling: model choice per task, routing, fallbacks, retries, caching, rate limits and structured outputs.
· Token and cost management: context budgets, prompt compression, model tiering and cost per transaction.
· Decision APIs: call deterministic rules and decision services for policy decisions, and keep the LLM for interpretation, extraction and drafting.
· Set the document intelligence, vector store and context engineering approach for each use case.
· Define guardrails: business rules, data access, human approval points, and responsible AI and compliance limits.
· Define the evaluation strategy: test sets, metrics, pass thresholds, and the release criteria.
· Define the observability standard: what is traced, logged and alerted on for every agent.
· Design A/B tests for prompts, models and retrieval changes, and decide when a variant wins.
· Review design fit on every pull request, read evaluation evidence against the approved rules, and clear or reject findings.
· Confirm release gates and sign off the design record before go/no-go.
· Capture lessons and update the reusable patterns, templates and prompts.
Must have
· Production delivery of agentic or LLM solutions, including tool calling, orchestration, RAG and multi-agent designs.
· Depth in the Microsoft AI stack: Azure AI Foundry, Copilot Studio, Microsoft Agent Framework, Azure OpenAI, Azure AI Search, and Entra ID and Azure security.
· Document intelligence design: Azure AI Document Intelligence or equivalent, layout and table extraction, handling scanned and low-quality records, and field-level confidence and validation.
· Vector database and retrieval design: embedding model choice, index and chunking strategy, hybrid and filtered search, and retrieval quality measurement.
· Context engineering: how to assemble instructions, retrieved data, memory and tool results within a token budget, and how to test context changes.
· Evaluation depth: golden datasets, LLM-as-judge and human review, groundedness, relevance, accuracy and safety metrics, regression evals in CI, and setting thresholds the business accepts.
· Observability: OpenTelemetry instrumentation for agents (traces, spans, metrics and logs), Azure Monitor and Application Insights or equivalent, and tracking latency, token use, cost and quality in production.
· Experimentation: A/B and shadow testing for prompts, models and retrieval, with sound success criteria and rollback.
· Spec-driven development with coding agents (GitHub Copilot, Claude Code or similar) in daily delivery.
· Enterprise integration and data architecture, including APIs, event and batch pipelines, and CI/CD.
· Strong written communication. You can turn policy into precise, testable specs and defend design decisions with business and technical leaders.
· Microsoft certifications (Azure AI Engineer Associate / Azure AI Apps and Agents Developer Associate/Agentic AI Business Solutions Architect).
Nice to have
· US healthcare payer experience: claims adjudication, Medicaid or managed care, utilization management or appeals, HEDIS and CAHPS.
· Familiarity with Facets, Edifecs or similar core payer platforms.
· HIPAA, PHI handling and audit-ready AI controls.
