Sr Quality Analyst with AI/ML and LLM testing

New York, NY, US • Posted 1 day ago • Updated 1 day ago
Full Time
On-site
$100,000 - $120,000/yr
Fitment

Dice Job Match Score™

🎯 Assessing qualifications...

Job Details

Skills

  • LLM
  • AI/ML
  • automation test

Summary

Role: Sr Quality Analyst with AI/ML and LLM testing

Location: New York

Duration: Full time

Role Summary

This role requires deep understanding of AI fundamentals, handson experience with LLM testing, guardrail design, data sanitization,

and evaluation of agent trajectories. The tester will work closely with AI engineers, data scientists, and product teams to ensure safe, reliable, and highquality AI behavior.

5 8+ years in QA, with at least 2 3 years in AI/ML or LLM testing.

Strong understanding of AI fundamentals: embeddings, vector stores, RAG, LLMs, agentic frameworks.

Handson experience testing:

  • LLM prompts and outputs
  • Agentic flows (e.g., LangChain, AutoGen, Semantic Kernel, AIP agents)
  • Model evaluation metrics

Experience writing guardrails, safety rules, and evaluation prompts.

Knowledge of data sanitization, PII detection, and data governance.

Familiarity with pipeline validation and MLOps workflows.

Ability to evaluate model performance using LowPick Score or similar ranking metrics.

Strong Python skills for test automation and evaluation scripting.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 501494924
  • Position Id: 115872-3-1
  • Posted 1 day ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Hybrid in Jersey City, New Jersey

Today

Easy Apply

Contract, Third Party

$1 - $1

Jersey City, New Jersey

Yesterday

Full-time

USD 104,500.00 - 150,000.00 per year

New York, New York

2d ago

Easy Apply

Full-time, Third Party

Depends on Experience

New York, New York

Today

Full-time

USD 190,000.00 - 211,000.00 per year

Search all similar jobs