AI/ML Test Lead

Position 1 : AI/ML Test Lead

Location : Chicago, IL

Job Description :

The AI/ML Test Lead will own end‑to‑end quality assurance for AI‑driven solutions across Ulta’s Enterprise AI.This role is responsible for defining and executing an Enterprise AI testing strategy, ensuring functional correctness, data accuracy, AI output quality, robustness, performance, and Responsible AI compliance. 

The role blends hands‑on testing leadership, AI/ML‑specific validation, and onsite stakeholder collaboration, working closely with Product Owners, AI Engineering Leads and Data teams. 

Key Responsibilities 

1. AI Testing Strategy & POD QA Ownership 

• Own AI/ML testing strategy and execution for assigned AI POD(s). 

• Define test approaches for AI use cases, covering:  

• Data pipelines and feature inputs 

• ML / LLM outputs 

• Agentic workflows and orchestration 

• Align testing with Enterprise AI QA strategy, leveraging existing data and analytics test controls while adding AI‑specific validation layers 

2. Functional & Data Validation 

• Validate source data, Gold layer tables, KPIs, dimensions, rollups, and semantic models used by AI systems. 

• Ensure AI outputs (insights, recommendations, summaries) are:  

• Grounded in verified data 

• Consistent with business logic 

• Reproducible and explainable 

• Partner with Data and Analytics teams to ensure upstream data quality assumptions are met.  

3. AI / LLM / Agent Testing 

• Design and execute AI‑specific test cases, including:  

• Prompt validation and prompt regression 

• Agent workflow testing (tools, steps, decision paths) 

• Hallucination and invalid response detection 

• Output consistency and relevance scoring 

• Support shift‑left testing (pre‑prod validation of prompts, agents, workflows) and shift‑right testing (post‑production monitoring and feedback loops). 

4. Non‑Functional & Responsible AI Testing 

• Own non‑functional testing for AI solutions:  

• Performance and latency 

• Scalability and reliability 

• Failure‑mode and exception testing 

• Validate Responsible AI controls, including:  

• Bias and fairness checks 

• Unsafe or toxic output detection 

• Output explainability and traceability 

• Track and report AI quality metrics such as accuracy, error rate, hallucination rate, and output stability.  

5. Test Planning, Automation & Metrics 

• Create AI test plans, scenarios, and acceptance criteria aligned to sprint goals. 

• Identify opportunities for test automation (data checks, regression prompts, API validations). 

• Define and track AI testing KPIs, feeding insights back to engineering and product teams. 

• Ensure defects, risks, and quality gaps are clearly logged and communicated. 

6. Stakeholder Collaboration (Onsite Role) 

• Act as the onsite QA interface for Ulta stakeholders. 

• Collaborate with:  

• AI Engineering Leads 

• Data Engineering & Analytics teams 

• Product Owners and Business SMEs 

• Provide clear quality status, risk visibility, and go/no‑go recommendations for releases. 

Required Skills & Experience 

Core QA & Testing Experience 

• 10+ years of experience in QA / Testing roles, with leadership responsibility 

• Strong experience defining test strategy, test plans, and execution ownership 

• Experience testing data‑heavy and analytics‑driven systems 

AI / ML / Data Testing Skills 

• Hands‑on experience testing AI/ML or LLM‑based solutions 

• Strong understanding of:  

• Data validation and KPI testing 

• AI output correctness and consistency 

• Prompt‑based and agent‑based systems 

• Familiarity with AI evaluation metrics (accuracy, error rate, hallucination rate, relevance)