Senior AI QA Engineer

Job Title: Senior AI QA Engineer

Location: Remote

Duration: 12 Months + Extension

Bill Rate: $87/hr

Job Type: W2 Contract

Client: To Be Discussed Later

Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD

Key Responsibilities:

• Define and execute test scenarios for Generative AI and Agentic AI workflows.
• Validate natural-language questions and AI-generated responses against agreed source-of-truth results.
• Test NL-to-SQL generation, query execution, validation, repair, and result interpretation.
• Validate AI agent workflows, tool/function calls, routing, retries, fallback behavior, and error handling.
• Test KPI threshold alerts, automated assessments, root-cause analysis, evidence, and recommended actions.
• Validate surveillance/anomaly-detection workflows against expected historical outcomes.
• Perform functional, integration, regression, API, data, and end-to-end testing.
• Validate AI responses across different roles/personas and entitlement scenarios.
• Test ambiguous questions, unsupported requests, refusal behavior, and negative scenarios.
• Validate PHI/PII handling, role-based access, and security controls.
• Validate data/results against Databricks and governed business definitions.
• Work with healthcare/business SMEs to understand use cases, expected results, business rules, and KPI definitions.
• Support performance/concurrency testing, UAT, defect triage, production-readiness validation, and hypercare.
• Work with AI Engineers and the AI Architect to isolate failures across data/context, prompts, models, agents, and application logic.

Must-Have Skills:

• Strong experience in QA/testing of enterprise applications.
• Strong API testing experience.
• Good SQL and data-validation skills.
• Experience with functional, integration, regression, and end-to-end testing.
• Experience testing complex data-driven applications.
• Understanding of Generative AI / LLM concepts.
• Ability to validate AI-generated results against expected business outcomes.
• Experience with test automation tools/frameworks.
• Basic understanding of US healthcare / Medicaid domain concepts and healthcare data.
• Strong defect analysis, troubleshooting, communication, and documentation skills.

Equal opportunity employer: We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.