AI Evaluation & Benchmarking Engineer

Remote • Posted 5 hours ago • Updated 5 hours ago
Contract W2
Contract Independent
12 Months
No Travel Required
Able to Sponsor
Remote
$50 - $65/hr
Fitment

Dice Job Match Score™

📋 Comparing job requirements...

Job Details

Skills

  • Artificial Intelligence
  • Benchmarking
  • Machine Learning (ML)
  • Data Structure

Summary

Job Description::
 
* Hands-on reinforcement learning experience.
* Experience using LLMs for agents, evaluation, reasoning, automation, or benchmark workflows.
* Strong Python experience for ML, data workflows, experimentation, and analysis.
* Experience designing and running experiments with statistical and analytical rigor.
* Strong understanding of evaluation metrics, scoring frameworks, performance comparison, and benchmark design.
* Experience analyzing structured logs, run outputs, model/agent performance, and experiment results.
* Ability to work across APIs, logs, CLI/tools, data structures, and platform workflows.
* Strong communication skills to translate experiment findings into platform improvement requirements.
* Ability to work inside client-owned repositories, infrastructure, workflows, and security controls.
 
Preferred Skills
 
* Experience with game environments, simulation environments, Gym-like interfaces, RL environments, or agentic AI test harnesses.
* Experience benchmarking LLM agents, RL policies, autonomous agents, or hybrid AI systems.
* Experience with experiment tracking, run comparison tools, metrics dashboards, or evaluation pipelines.
* Experience with prompt engineering, agent orchestration, tool use, and LLM evaluation frameworks.
* Experience with data visualization and performance analytics.
* Experience working with externally developed algorithms, reproducible experiments, and version-controlled evaluation workflows.

 

 
 
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: RTX1c3151
  • Position Id: 9023452
  • Posted 5 hours ago
Contact the job poster
RB

Raghav Badiger

Recruiter @ Highbrow
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Remote

3d ago

Easy Apply

Full-time

Depends on Experience

Remote

23d ago

Full-time

220000 - 500000

Remote

Today

Full-time

Remote

Today

Full-time

USD 372,000.00 - 600,000.00 per year

Search all similar jobs