Search Jobs | Dice.com

1125 results (55 new)

Group by company

AI Evaluation Engineer

Acunor Infotech

Remote or New Jersey • Today

Easy Apply

Contract, Third Party

AI Evaluation Engineer

Tandavix llc

Remote • Today

Easy Apply

Contract

50 - 60

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or Texas • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or South Carolina • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or New York, New York • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or Michigan • Today

Part-time

USD 50.00 per hour

Evaluation Engineer (AI Models)

Exl Neo Technologies

New York, New York • Yesterday

Easy Apply

Full-time, Part-time, Third Party, Contract

AI Engineer - Algorithm Evaluation & Agentic Systems

Apple, Inc.

Sunnyvale, California • Today

Full-time

Machine Learning Engineer - AI Evaluation & LLM Systems

Apple, Inc.

Cupertino, California • Today

Full-time

AIML - Sr Applied AI Scientist - GenAI Model Autograding, Evaluation

Apple, Inc.

Cupertino, California • Today

Full-time

Staff ML Engineer - Embodied AI Evaluation Foundations

General Motors

Sunnyvale, California • Today

Full-time

USD 189,400.00 - 300,600.00 per year

AIML - Applied AI Scientist, Image Autograder Systems, Evaluation

Apple, Inc.

Cupertino, California • Today

Full-time

Senior Director - AI Evaluation Platform

ServiceNow, Inc.

Santa Clara, California • Today

Full-time

USD 279,100.00 - 488,400.00 per year

AI/ML Evaluation Specialist, Human Data

Apple, Inc.

Cupertino, California • Today

Full-time

Senior Applied Scientist, Multilingual AI Evaluation

Apple, Inc.

Washington • Today

Full-time

Principal Engineer Tech Lead, Embodied AI & Off-Board Performance Evaluation

Motional

Remote or Boston, Massachusetts • Today

Full-time

USD 200,000.00 - 275,000.00 per year

Applied AI Researcher, Agent Systems & Evaluation

Nuro Inc.

Mountain View, California • Today

Full-time

USD 193,930.00 - 352,290.00 per year

AI Governance and Evaluation Manager (Remote From Anywhere In CO)

State of Colorado-OIT

Remote or Colorado • Today

Full-time

USD 145,000.00 - 160,000.00 per year

Lead, Search & Evaluation, AI and Automation Drug Discovery

Takeda pharmaceutical

Cambridge, Massachusetts • Today

Full-time

USD 177,000.00 - 278,080.00 per year

Director, Search & Evaluation (Platforms & AI)

Novartis

Cambridge, England • Today

Full-time

USD 176,400.00 - 327,600.00 per year

AI Engineer

GDH

Remote • Today

Easy Apply

Third Party, Contract

Depends on Experience

AI Partnerships Lead - Technology Vendor & Start-up Evaluation, VP

Citi

Jersey City, New Jersey • Today

Full-time

USD 142,320.00 - 213,480.00 per year

Applied AI Lead - Evaluation & Measurement

Leonardo DRS

Frederick, Maryland • Today

Full-time

Director, AI & Data Partner Evaluation

AstraZeneca

Boston, Massachusetts • Today

Full-time

USD 164,521.60 - 246,782.40 per year

Machine Learning Engineer - AI & ML Evaluation Frameworks

Apple, Inc.

Cupertino, California • Today

Full-time

AI Engineer

Randstad Digital

Charlotte, North Carolina • Today

Contract

USD52 - USD62

AI Engineer

S Linx LLC

Mountain View, California • 8d ago

Easy Apply

Contract, Third Party

60 - 65

AI Engineer – Neural Networks & Knowledge Graph, Phoenix, AZ (Onsite)

Tror

Phoenix, Arizona • Today

Easy Apply

Third Party, Contract

Depends on Experience

Business Consultant - AI/ML Engineer

NTT DATA Americas, Inc

Hybrid in Sandy Springs, Georgia • Yesterday

Easy Apply

Contract, Third Party

$60 - $70

Acunor Infotech

Job Details

Skills

LLM
healthcare
RAG
Evaluation

Summary

AI Evaluation Engineer

Location: USA Remote
Employment Type: Contract

Position Summary

We are seeking an AI Evaluation Engineer to establish and maintain the quality standards used to determine whether production AI and LLM systems are ready to launch.

This role will build centralized evaluation frameworks, create golden datasets, establish measurable quality thresholds, audit AI evaluation results, and implement continuous production monitoring. The successful candidate will bring strong technical evaluation expertise along with the independence and analytical rigor required to objectively determine whether AI systems meet production quality and safety standards.

Key Responsibilities

Design and build centralized LLM/AI evaluation frameworks and reusable evaluation templates.
Establish evaluation standards that can be adopted consistently across multiple AI engineering teams.
Create and maintain golden datasets in partnership with business, domain, and subject-matter experts.
Develop evaluation suites covering functional quality, accuracy, safety, reliability, and domain-specific requirements.
Implement and assess LLM-as-a-judge evaluation approaches while accounting for their limitations and failure modes.
Define measurable pass/fail thresholds and production-readiness criteria for AI applications.
Independently review and audit evaluation suites developed by individual engineering teams.
Build regression testing approaches for LLM applications, agents, prompts, retrieval systems, and model changes.
Establish production monitoring for AI quality degradation, drift, regressions, and incidents.
Analyze and report evaluation pass rates, quality trends, regressions, and production incidents.
Partner with AI engineers, platform engineers, product teams, and domain experts to continuously improve AI quality.

Required Qualifications

4+ years of experience in ML/LLM evaluation, AI quality engineering, applied research engineering, or related areas.
Hands-on experience designing and implementing AI/LLM evaluation frameworks.
Experience developing golden/reference datasets and measurable evaluation criteria.
Strong understanding of LLM-as-a-judge methodologies and associated failure modes.
Experience evaluating LLM applications, agents, RAG systems, prompts, or other probabilistic AI systems.
Strong statistical and analytical skills, including evaluation methodologies for relatively small sample sizes.
Experience establishing quality thresholds and regression criteria.
Strong engineering skills with the ability to build reusable evaluation tooling and automation.
Ability to independently assess system quality and challenge release decisions when evaluation evidence does not meet established standards.

Preferred Qualifications

Healthcare, clinical, or other safety-critical AI evaluation experience.
Experience designing medical-accuracy or domain-specific evaluation suites.
AI red-teaming or adversarial testing experience.
Experience with production AI monitoring, drift detection, and continuous evaluation.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91026517
Position Id: 2026-1944
Posted 4 hours ago

Company Info

About Acunor Infotech

Acunor stands for Accurate North. Similar to North Star, we strive to provide accurate guidance and help our clients in every step of their journey towards a sustainable digital organization. Here at Acunor, we’ve been on a journey to upskill IT practitioners and we are ready to help IT organisations and individuals accelerate their upskilling journey and create the workforce of tomorrow, today.

Go to company profile

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AI Engineering Manager

Remote or New Jersey

•

Today

AI Engineering Manager / Lead AI Engineer Location: Remote United States only Work Hours: Must work Pacific Time (PST/PDT) business hours Employment Type: Long-Term Contract Openings: 3 About the Role We are seeking an experienced AI Engineering Manager / Lead AI Engineer who can operate as a true player-coach, combining engineering leadership with hands-on technical expertise. This role will manage engineers across a central AI platform team and multiple delivery pods, including both interna

Easy Apply

Third Party, Contract

Applied AI Engineer

Remote or New Jersey

•

Today

Applied AI Engineer Location: Remote United States, Canada Work Hours: Must work Pacific Time (PST/PDT) business hours Employment Type: Long-Term Contract About the Role We are seeking a hands-on Applied AI Engineer to design, build, evaluate, and productionize LLM-powered agents and AI workflows. This is a production engineering role rather than a research-only or proof-of-concept position. You will turn business use cases into reliable AI systems using an AWS-native platform with Amazon Bedr

Easy Apply

Contract, Third Party

$DOE

Acunor Infotech

Job Details

Skills

LLM
healthcare
RAG
Evaluation

Summary

AI Evaluation Engineer

Location: USA Remote
Employment Type: Contract

Position Summary

We are seeking an AI Evaluation Engineer to establish and maintain the quality standards used to determine whether production AI and LLM systems are ready to launch.

Key Responsibilities

Design and build centralized LLM/AI evaluation frameworks and reusable evaluation templates.
Establish evaluation standards that can be adopted consistently across multiple AI engineering teams.
Create and maintain golden datasets in partnership with business, domain, and subject-matter experts.
Develop evaluation suites covering functional quality, accuracy, safety, reliability, and domain-specific requirements.
Implement and assess LLM-as-a-judge evaluation approaches while accounting for their limitations and failure modes.
Define measurable pass/fail thresholds and production-readiness criteria for AI applications.
Independently review and audit evaluation suites developed by individual engineering teams.
Build regression testing approaches for LLM applications, agents, prompts, retrieval systems, and model changes.
Establish production monitoring for AI quality degradation, drift, regressions, and incidents.
Analyze and report evaluation pass rates, quality trends, regressions, and production incidents.
Partner with AI engineers, platform engineers, product teams, and domain experts to continuously improve AI quality.

Required Qualifications

4+ years of experience in ML/LLM evaluation, AI quality engineering, applied research engineering, or related areas.
Hands-on experience designing and implementing AI/LLM evaluation frameworks.
Experience developing golden/reference datasets and measurable evaluation criteria.
Strong understanding of LLM-as-a-judge methodologies and associated failure modes.
Experience evaluating LLM applications, agents, RAG systems, prompts, or other probabilistic AI systems.
Strong statistical and analytical skills, including evaluation methodologies for relatively small sample sizes.
Experience establishing quality thresholds and regression criteria.
Strong engineering skills with the ability to build reusable evaluation tooling and automation.
Ability to independently assess system quality and challenge release decisions when evaluation evidence does not meet established standards.

Preferred Qualifications

Healthcare, clinical, or other safety-critical AI evaluation experience.
Experience designing medical-accuracy or domain-specific evaluation suites.
AI red-teaming or adversarial testing experience.
Experience with production AI monitoring, drift detection, and continuous evaluation.

Dice Id: 91026517
Position Id: 2026-1944
Posted 4 hours ago

Company Info

About Acunor Infotech

Go to company profile

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AI Engineering Manager

Remote or New Jersey

•

Today

Easy Apply

Third Party, Contract

Applied AI Engineer

Remote or New Jersey

•

Today

Easy Apply

Contract, Third Party

$DOE