job summary:
We're looking for a Senior Product Quality Analyst to deliver high-quality human judgments that provide ground truth for evaluation, support model fine-tuning, and guide the alignment of LLM-based evaluators. You'll lead annotation projects and drive improvements in annotation quality.
What you'll do
Lead end-to-end execution of human annotation projects, from intake and scoping to delivery.
Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
Manage vendors and coordinate contracts and related reviews with legal, security, privacy, finance, and accounting.
Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to improve guidelines and outputs.
location: New York, New York
job type: Contract
salary: $52 - 54 per hour
work hours: 8am to 5pm
education: Bachelors
responsibilities:
Location: New York or Boston (in-office), or remote with working hours in Eastern Time
AI Foundation develops the core ML and AI systems that power personalized experiences. These systems rely on human judgment to ground their intelligence in the nuances of culture, personal taste, and real-world listening habits.
We're looking for a Senior Product Quality Analyst to deliver high-quality human judgments that provide ground truth for evaluation, support model fine-tuning, and guide the alignment of LLM-based evaluators. You'll lead annotation projects and drive improvements in annotation quality.
What you'll do
- Lead end-to-end execution of human annotation projects, from intake and scoping to delivery.
- Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
- Manage vendors and coordinate contracts and related reviews with legal, security, privacy, finance, and accounting.
- Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
- Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
- Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to improve guidelines and outputs.
Who you are- Background in product quality, evaluation, data operations, or related roles.
- Hands-on experience designing, coordinating, or reviewing data annotation workflows for product development.
- Familiarity with ML-driven products and human evaluation of AI/LLM-powered features.
- Ability to translate product and modeling needs into clear annotation guidelines and evaluation rubrics.
- Skilled at managing vendors and coordinating delivery across internal teams.
- Excellent communication and stakeholder management skills, including setting expectations and resolving ambiguous requirements.
Where you'll beThis role is based in New York or Boston (in office) or remote working Eastern Time hours.
qualifications:
Background in product quality, evaluation, data operations, or related roles.
Hands-on experience designing, coordinating, or reviewing data annotation workflows for product development.
Familiarity with ML-driven products and human evaluation of AI/LLM-powered features.
Ability to translate product and modeling needs into clear annotation guidelines and evaluation rubrics.
Skilled at managing vendors and coordinating delivery across internal teams.
Excellent communication and stakeholder management skills, including setting expectations and resolving ambiguous requirements.
skills:
AI,data operations,LLM-based,LLM-powered,ML-driven,communication,coordinating,accounting,annotation,data annotation,consult,finance,managing vendors,product development,Product Quality,rubrics,security,setting expectations,stakeholder management skills,workflows
Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.
At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact
Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).
This posting is open for thirty (30) days.
![]()
