Must Have Technical/Functional Skills
• AI/ML/LLM Quality Engineering experience (testing, validation, evaluation frameworks)
• Expertise in AI evaluation methods (deterministic & non-deterministic systems)
• Validation frameworks, golden datasets, and test harness
• Scalable testing strategies for enterprise AI platforms
• Experience embedding quality practices into AI SDLC / delivery lifecycle
• Model evaluation, monitoring, and continuous quality improvement
• Agent/LLM evaluation and autonomy threshold
• Cross-functional collaboration with engineering, data, and architecture ________________________________________
Roles & Responsibilities
• Build the AI QA function as a dedicated capability within the broader AI delivery organization.
• Define quality engineering practices for AI systems, including evaluation methods for deterministic and non-deterministic outputs.
• Develop, adopt AI for QE capabilities and improve outcomes of the Quality Engineering function using new AI capabilities
• Establish validation criteria, golden datasets, and test harnesses to support safe deployment and continuous model or prompt changes.
• Work with engineering and synthetic data teams to ensure quality controls are embedded into the AI delivery lifecycle.
• Establish baseline KPIs for QE function and incorporate mechanism to track maturity
• Help define thresholds for agent maturity and autonomy, including when systems can move from analysis to decision and action under appropriate oversight.
Never miss an opportunity! Create an alert based on the job you applied for.