Avenue Code, LLC
Avenue Code, LLC

AI/ML Engineer

Hybrid in New York, NY, US • Posted 5 hours ago • Updated 5 hours ago
Contract W2
12 Months
No Travel Required
Hybrid
$75 - $80/hr
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • AI
  • ML

Summary

AI/ML Engineer
Location: New York or Boston (in-office), or remote with working hours in Eastern Time
 
AI Foundation develops the core ML and AI systems that power personalized experiences on Spotify. These systems rely on human judgment to ground their intelligence in the nuances of culture, personal taste, and real-world listening habits.
 
We’re looking for a Senior Product Quality Analyst to deliver high-quality human judgments that provide ground truth for evaluation, support model fine-tuning, and guide the alignment of LLM-based evaluators. You’ll lead annotation projects and drive improvements in annotation quality.
What you’ll do
  • Lead end-to-end execution of human annotation projects, from intake and scoping to delivery.
  • Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
  • Manage vendors and coordinate contracts and related reviews with legal, security, privacy, finance, and accounting.
  • Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
  • Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
  • Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to improve guidelines and outputs.
Who you are
  • Background in product quality, evaluation, data operations, or related roles.
  • Hands-on experience designing, coordinating, or reviewing data annotation workflows for product development.
  • Familiarity with ML-driven products and human evaluation of AI/LLM-powered features.
  • Ability to translate product and modeling needs into clear annotation guidelines and evaluation rubrics.
  • Skilled at managing vendors and coordinating delivery across internal teams.
  • Excellent communication and stakeholder management skills, including setting expectations and resolving ambiguous requirements.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10383289
  • Position Id: 9127515
  • Posted 5 hours ago
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.
More jobs at Avenue Code, LLC in New York, NY