Apply Now

AI Evaluations Engineer

Hybrid in New York, NY, US • Posted 1 hour ago • Updated 27 minutes ago

Contract W2

On-site

ADDSOURCE

Fitment

Dice Job Match Score™

To see how well you match this job, please log in or create an account.

Once logged in, be sure to complete your profile to get the most accurate match score.

🔢 Crunching numbers...

Job Details

Skills

Amazon Web Services
Metrics
Spring Framework
regression testing
CloudWatch
Workflows
Sybase
Infrastructure Management
Mentoring
Identity and Access Management
Artificial Intelligence
Generative AI
Large Language Models
Python (Programming Language)
Java (Programming Language)
IBM DB2
Knowledge of Finance
Data Control
Amazon S3
Functional Programming
COBOL (Programming Language)
Audit Planning
Programme Evaluations
A/B Testing
Delivery of Packages
Agentic-AI
CrewAI
LangGraph Framework
Remote Monitoring

Amazon Web Services
Metrics
Spring Framework
regression testing
CloudWatch
Workflows
Sybase
Infrastructure Management
Mentoring
Identity and Access Management
Artificial Intelligence
Generative AI
Large Language Models
Python (Programming Language)
Java (Programming Language)
IBM DB2
Knowledge of Finance
Data Control
Amazon S3
Functional Programming
COBOL (Programming Language)
Audit Planning
Programme Evaluations
A/B Testing
Delivery of Packages
Agentic-AI
CrewAI
LangGraph Framework
Remote Monitoring

Summary

Role: AI Evaluations Engineer (Senior)

Location: US (remote)

Duration: 12 Months +

Key Responsibilities

Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)
Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows
Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety
Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices

Must Have

5-7+ years building production-grade ML or GenAI systems; advanced Python
Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring
LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity
Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety
Agentic framework experience (Strands, LangGraph, CrewAI or similar)
AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals

Should Have

Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)
Agent-as-a-Judge framework; HITL evaluation pipelines
Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction
RAG quality evaluation (faithfulness, context precision, recall)
CI/CD integration for eval regression; observability tooling (tracing, structured logging)
Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design
Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)
Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91122967
Position Id: 2026-4603
Posted 1 hour ago

Company Info

About ADDSOURCE

AddSource is a premier staffing and workforce solutions partner headquartered in Delaware, USA, with operations in Alberta, Canada under the brand name AddSource.

We specialize in connecting top-tier talent with leading organizations across diverse industry sectors.

With a strong commitment to excellence, our mission is to deliver innovative, customized staffing solutions that empower clients to achieve their goals while helping candidates advance their careers.

Go to company profile

Contact the job poster

Rahil Khan

Recruiter @ ADDSOURCE

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AIDDLC Engineer

Remote or Hybrid in New York, New York

•

Today

Role: AIDDLC Engineer (Senior) Location: US (remote) Duration: 12 Months + Key Responsibilities Lead customers through the full AIDDLC - Assess, Design, Develop, Deploy & Operate - as both methodology practitioner and technical builder Drive architecture decisions and framework execution across all phases; own supporting tooling and automation delivery Run AI readiness workshops, use case prioritization, and maturity assessments with business and technical stakeholders Own engagements end-to-en

Easy Apply

Contract

Search all similar jobs

Remote jobs at ADDSOURCE