AI/ML Test Lead

Position :AI/ML Test Lead

Location : Chicago, IL

Job Description : 

The AI/ML Test Lead will own endtoend quality assurance for AIdriven solutions across Ulta’s Enterprise AI.This role is responsible for defining and executing an Enterprise AI testing strategy, ensuring functional correctness, data accuracy, AI output quality, robustness, performance, and Responsible AI compliance. 

The role blends handson testing leadership, AI/MLspecific validation, and onsite stakeholder collaboration, working closely with Product Owners, AI Engineering Leads and Data teams. 

Key Responsibilities 

1. AI Testing Strategy & POD QA Ownership 

• Own AI/ML testing strategy and execution for assigned AI POD(s). 
• Define test approaches for AI use cases, covering:  
• Data pipelines and feature inputs 
• ML / LLM outputs 
• Agentic workflows and orchestration 
• Align testing with Enterprise AI QA strategy, leveraging existing data and analytics test controls while adding AIspecific validation layers 

2. Functional & Data Validation 

• Validate source data, Gold layer tables, KPIs, dimensions, rollups, and semantic models used by AI systems. 
• Ensure AI outputs (insights, recommendations, summaries) are:  
• Grounded in verified data 
• Consistent with business logic 
• Reproducible and explainable 
• Partner with Data and Analytics teams to ensure upstream data quality assumptions are met.  

3. AI / LLM / Agent Testing 

• Design and execute AIspecific test cases, including:  
• Prompt validation and prompt regression 
• Agent workflow testing (tools, steps, decision paths) 
• Hallucination and invalid response detection 
• Output consistency and relevance scoring 
• Support shiftleft testing (preprod validation of prompts, agents, workflows) and shiftright testing (postproduction monitoring and feedback loops). 

4. NonFunctional & Responsible AI Testing 

• Own nonfunctional testing for AI solutions:  
• Performance and latency 
• Scalability and reliability 
• Failuremode and exception testing 
• Validate Responsible AI controls, including:  
• Bias and fairness checks 
• Unsafe or toxic output detection 
• Output explainability and traceability 
• Track and report AI quality metrics such as accuracy, error rate, hallucination rate, and output stability.  

5. Test Planning, Automation & Metrics 

• Create AI test plans, scenarios, and acceptance criteria aligned to sprint goals. 
• Identify opportunities for test automation (data checks, regression prompts, API validations). 
• Define and track AI testing KPIs, feeding insights back to engineering and product teams. 
• Ensure defects, risks, and quality gaps are clearly logged and communicated. 

6. Stakeholder Collaboration (Onsite Role) 

• Act as the onsite QA interface for Ulta stakeholders. 
• Collaborate with:  
• AI Engineering Leads 
• Data Engineering & Analytics teams 
• Product Owners and Business SMEs 
• Provide clear quality status, risk visibility, and go/nogo recommendations for releases. 

Required Skills & Experience 

Core QA & Testing Experience 

• 10+ years of experience in QA / Testing roles, with leadership responsibility 
• Strong experience defining test strategy, test plans, and execution ownership 
• Experience testing dataheavy and analyticsdriven systems 

AI / ML / Data Testing Skills 

• Handson experience testing AI/ML or LLMbased solutions 
• Strong understanding of:  
• Data validation and KPI testing 
• AI output correctness and consistency 
• Promptbased and agentbased systems 
• Familiarity with AI evaluation metrics (accuracy, error rate, hallucination rate, relevance)