Search Jobs | Dice.com

169 results (1 new)

Group by company

AI Evaluation Engineer

Acunor Infotech

Remote or New Jersey • Today

Easy Apply

Third Party, Contract

AI Evaluation Engineer

Tandavix llc

Remote • Today

Easy Apply

Contract

50 - 60

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or Texas • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or South Carolina • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or New York, New York • Today

Part-time

USD 50.00 per hour

AI Evaluation Engineer (Python, QA or Security)

MindLift

Remote or Michigan • Today

Part-time

USD 50.00 per hour

Principal Engineer Tech Lead, Embodied AI & Off-Board Performance Evaluation

Motional

Remote or Boston, Massachusetts • Today

Full-time

USD 200,000.00 - 275,000.00 per year

AI Governance and Evaluation Manager (Remote From Anywhere In CO)

State of Colorado-OIT

Remote or Colorado • Today

Full-time

USD 145,000.00 - 160,000.00 per year

AI Engineer

GDH

Remote • Today

Easy Apply

Third Party, Contract

Depends on Experience

AI Partnerships Lead - Technology Vendor & Start-up Evaluation, VP

Citi

Jersey City, New Jersey • Today

Full-time

USD 142,320.00 - 213,480.00 per year

Agentic AI Forward Deployment Engineer / Architect

Photon

Remote • 15d ago

Easy Apply

Contract, Third Party

Depends on Experience

AI Engineer

System One

Remote • 7d ago

Easy Apply

Contract

$52.85

AI Engineer

Holtec International

Camden, New Jersey • Today

Full-time

USD 120,000.00 per year

AI Engineer

Kobie Marketing

Remote or Detroit, Michigan • Today

Full-time

AI Engineer

ANALYTICA

Remote or Washington, District of Columbia • Today

Full-time

Artificial Intelligence/Machine Learning (AI/ML) Engineer

ANALYTICA

Remote or Bethesda, Maryland • Today

Full-time

AI Engineer Tech Lead

NTT DATA, Inc.

Remote or Dallas, Texas • Today

Full-time

Senior ML Ops Engineer

Mitchell Martin, Inc.

Remote • 26d ago

Easy Apply

Full-time, Third Party

$105000 - $150000

AI Field Engineer - Enterprise

Fireworks AI

Remote or New York, New York • 24d ago

Full-time

USD 200,000.00 - 260,000.00 per year

AI Field Engineer - AI Natives

Fireworks AI

Remote or New York, New York • 24d ago

Full-time

USD 200,000.00 - 260,000.00 per year

AI Foundational Model Engineer

NTT DATA, Inc.

Jersey City, New Jersey • Today

Full-time

USD 139,872.00 - 209,808.00 per year

Senior AI/ML Engineer

Johnson & Johnson

Hopewell Township, New Jersey • Today

Full-time

USD 155,000.00 - 175,000.00 per year

AI Field Engineer - Strategic Partnerships

Fireworks AI

Remote or New York, New York • 24d ago

Full-time

USD 200,000.00 - 260,000.00 per year

AI / Machine Learning Engineer

Cybermedia Technologies LLC

Remote • Today

Full-time

Senior AI Engineer, Banking Technology

Citi

Jersey City, New Jersey • Today

Full-time

USD 142,320.00 - 213,480.00 per year

Senior AI Engineer

The Chemours Company

Remote • Today

Full-time

USD 126,067.00 - 196,980.00 per year

AI Foundation Model Engineer

NTT DATA, Inc.

Jersey City, New Jersey • Today

Full-time

AI Engineer

StatusNeo Inc.

Parsippany-Troy Hills, New Jersey • 6d ago

Easy Apply

Full-time

Depends on Experience

Principal Software Engineer - AI Engineer

JPMorgan Chase & Co.

Jersey City, New Jersey • Today

Full-time

USD 204,250.00 - 285,000.00 per year

Acunor Infotech

Job Details

Skills

LLM
healthcare
RAG
Evaluation

Summary

AI Evaluation Engineer

Location: USA Remote
Employment Type: Contract

Position Summary

We are seeking an AI Evaluation Engineer to establish and maintain the quality standards used to determine whether production AI and LLM systems are ready to launch.

This role will build centralized evaluation frameworks, create golden datasets, establish measurable quality thresholds, audit AI evaluation results, and implement continuous production monitoring. The successful candidate will bring strong technical evaluation expertise along with the independence and analytical rigor required to objectively determine whether AI systems meet production quality and safety standards.

Key Responsibilities

Design and build centralized LLM/AI evaluation frameworks and reusable evaluation templates.
Establish evaluation standards that can be adopted consistently across multiple AI engineering teams.
Create and maintain golden datasets in partnership with business, domain, and subject-matter experts.
Develop evaluation suites covering functional quality, accuracy, safety, reliability, and domain-specific requirements.
Implement and assess LLM-as-a-judge evaluation approaches while accounting for their limitations and failure modes.
Define measurable pass/fail thresholds and production-readiness criteria for AI applications.
Independently review and audit evaluation suites developed by individual engineering teams.
Build regression testing approaches for LLM applications, agents, prompts, retrieval systems, and model changes.
Establish production monitoring for AI quality degradation, drift, regressions, and incidents.
Analyze and report evaluation pass rates, quality trends, regressions, and production incidents.
Partner with AI engineers, platform engineers, product teams, and domain experts to continuously improve AI quality.

Required Qualifications

4+ years of experience in ML/LLM evaluation, AI quality engineering, applied research engineering, or related areas.
Hands-on experience designing and implementing AI/LLM evaluation frameworks.
Experience developing golden/reference datasets and measurable evaluation criteria.
Strong understanding of LLM-as-a-judge methodologies and associated failure modes.
Experience evaluating LLM applications, agents, RAG systems, prompts, or other probabilistic AI systems.
Strong statistical and analytical skills, including evaluation methodologies for relatively small sample sizes.
Experience establishing quality thresholds and regression criteria.
Strong engineering skills with the ability to build reusable evaluation tooling and automation.
Ability to independently assess system quality and challenge release decisions when evaluation evidence does not meet established standards.

Preferred Qualifications

Healthcare, clinical, or other safety-critical AI evaluation experience.
Experience designing medical-accuracy or domain-specific evaluation suites.
AI red-teaming or adversarial testing experience.
Experience with production AI monitoring, drift detection, and continuous evaluation.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91026517
Position Id: 2026-1944
Posted 2 hours ago

Company Info

About Acunor Infotech

Acunor stands for Accurate North. Similar to North Star, we strive to provide accurate guidance and help our clients in every step of their journey towards a sustainable digital organization. Here at Acunor, we’ve been on a journey to upskill IT practitioners and we are ready to help IT organisations and individuals accelerate their upskilling journey and create the workforce of tomorrow, today.

Go to company profile

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AI Engineering Manager

Remote or New Jersey

•

Today

AI Engineering Manager / Lead AI Engineer Location: Remote United States only Work Hours: Must work Pacific Time (PST/PDT) business hours Employment Type: Long-Term Contract Openings: 3 About the Role We are seeking an experienced AI Engineering Manager / Lead AI Engineer who can operate as a true player-coach, combining engineering leadership with hands-on technical expertise. This role will manage engineers across a central AI platform team and multiple delivery pods, including both interna

Easy Apply

Third Party, Contract

Applied AI Engineer

Remote or New Jersey

•

Today

Applied AI Engineer Location: Remote United States, Canada Work Hours: Must work Pacific Time (PST/PDT) business hours Employment Type: Long-Term Contract About the Role We are seeking a hands-on Applied AI Engineer to design, build, evaluate, and productionize LLM-powered agents and AI workflows. This is a production engineering role rather than a research-only or proof-of-concept position. You will turn business use cases into reliable AI systems using an AWS-native platform with Amazon Bedr

Easy Apply

Contract, Third Party

$DOE

Acunor Infotech

Job Details

Skills

LLM
healthcare
RAG
Evaluation

Summary

AI Evaluation Engineer

Location: USA Remote
Employment Type: Contract

Position Summary

We are seeking an AI Evaluation Engineer to establish and maintain the quality standards used to determine whether production AI and LLM systems are ready to launch.

Key Responsibilities

Design and build centralized LLM/AI evaluation frameworks and reusable evaluation templates.
Establish evaluation standards that can be adopted consistently across multiple AI engineering teams.
Create and maintain golden datasets in partnership with business, domain, and subject-matter experts.
Develop evaluation suites covering functional quality, accuracy, safety, reliability, and domain-specific requirements.
Implement and assess LLM-as-a-judge evaluation approaches while accounting for their limitations and failure modes.
Define measurable pass/fail thresholds and production-readiness criteria for AI applications.
Independently review and audit evaluation suites developed by individual engineering teams.
Build regression testing approaches for LLM applications, agents, prompts, retrieval systems, and model changes.
Establish production monitoring for AI quality degradation, drift, regressions, and incidents.
Analyze and report evaluation pass rates, quality trends, regressions, and production incidents.
Partner with AI engineers, platform engineers, product teams, and domain experts to continuously improve AI quality.

Required Qualifications

4+ years of experience in ML/LLM evaluation, AI quality engineering, applied research engineering, or related areas.
Hands-on experience designing and implementing AI/LLM evaluation frameworks.
Experience developing golden/reference datasets and measurable evaluation criteria.
Strong understanding of LLM-as-a-judge methodologies and associated failure modes.
Experience evaluating LLM applications, agents, RAG systems, prompts, or other probabilistic AI systems.
Strong statistical and analytical skills, including evaluation methodologies for relatively small sample sizes.
Experience establishing quality thresholds and regression criteria.
Strong engineering skills with the ability to build reusable evaluation tooling and automation.
Ability to independently assess system quality and challenge release decisions when evaluation evidence does not meet established standards.

Preferred Qualifications

Healthcare, clinical, or other safety-critical AI evaluation experience.
Experience designing medical-accuracy or domain-specific evaluation suites.
AI red-teaming or adversarial testing experience.
Experience with production AI monitoring, drift detection, and continuous evaluation.

Dice Id: 91026517
Position Id: 2026-1944
Posted 2 hours ago

Company Info

About Acunor Infotech

Go to company profile

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AI Engineering Manager

Remote or New Jersey

•

Today

Easy Apply

Third Party, Contract

Applied AI Engineer

Remote or New Jersey

•

Today

Easy Apply

Contract, Third Party

$DOE