Role: QA Engineer AI Applications Domain: Healthcare / Medicaid Analytics Technology Focus: Generative AI, Agentic AI, AWS Bedrock, Databricks Level: Senior QA Engineer
Role Summary
We are looking for an experienced QA Engineer to validate an enterprise Generative AI and Agentic AI solution built
using AWS Bedrock and Databricks. The role will focus on conversational AI, agent workflows, natural-language-to-SQL,
KPI assessments, surveillance capabilities, data accuracy, security, and end-to-end AI quality. The candidate should also
have a basic understanding of the US healthcare / Medicaid domain so that business scenarios and AI responses can be
validated in the correct functional context.
Key Responsibilities
Define and execute test scenarios for Generative AI and Agentic AI workflows.
Validate natural-language questions and AI-generated responses against agreed source-of-truth results.
Test NL-to-SQL generation, query execution, validation, repair, and result interpretation.
Validate AI agent workflows, tool/function calls, routing, retries, fallback behavior, and error handling.
Test KPI threshold alerts, automated assessments, root-cause analysis, evidence, and recommended actions.
Validate surveillance/anomaly-detection workflows against expected historical outcomes.
Perform functional, integration, regression, API, data, and end-to-end testing.
Validate AI responses across different roles/personas and entitlement scenarios.
Test ambiguous questions, unsupported requests, refusal behavior, and negative scenarios.
Validate PHI/PII handling, role-based access, and security controls.
Validate data/results against Databricks and governed business definitions.
Work with healthcare/business SMEs to understand use cases, expected results, business rules, and KPI definitions.
Support performance/concurrency testing, UAT, defect triage, production-readiness validation, and hypercare.
Work with AI Engineers and the AI Architect to isolate failures across data/context, prompts, models, agents, and
application logic.
Must-Have Skills
Strong experience in QA/testing of enterprise applications.
Strong API testing experience.
Good SQL and data-validation skills.
Experience with functional, integration, regression, and end-to-end testing.
Experience testing complex data-driven applications.
Understanding of Generative AI / LLM concepts.
Ability to validate AI-generated results against expected business outcomes.
Experience with test automation tools/frameworks.
Basic understanding of US healthcare / Medicaid domain concepts and healthcare data.
Strong defect analysis, troubleshooting, communication, and documentation skills.
Basic Healthcare Domain Knowledge
Basic understanding of the US healthcare and Medicaid ecosystem and common MMIS concepts.
Familiarity with common healthcare data domains such as members/beneficiaries, eligibility, providers,
claims/encounters, and pharmacy data.
Basic understanding of healthcare KPIs, measures, business rules, and the importance of validating results against an
approved source of truth.
Awareness of HIPAA, PHI/PII handling, least-privilege access, and role/persona-based data access.
Ability to work with healthcare SMEs to translate business scenarios into test cases and expected outcomes.
Pharmacy / PBM / Medicaid claims knowledge is a plus, but deep domain expertise is not required.
Preferred Skills
Hands-on experience testing Generative AI / LLM applications.
Experience testing Agentic AI / AI agents.
AWS Bedrock exposure.
Databricks / Unity Catalog experience.
Natural Language-to-SQL testing.
RAG / grounding validation.
AI evaluation frameworks and automated scoring.
Prompt/model regression testing.
API automation and performance/load testing.
Healthcare / Medicaid / Pharmacy domain experience.
PHI/PII and healthcare-security testing experience.
AI-Specific Testing Experience Preferred
Response accuracy and factual consistency
SQL/result accuracy
Hallucination and unsupported-answer handling
Grounding and context retrieval
Tool-call execution
Conversation/follow-up context
Agent workflow failures and recovery
Role/persona-based response differences
Model/prompt regression
Latency and cost-related behavior
Never miss an opportunity! Create an alert based on the job you applied for.
