AI Evaluation Engineer

Job Title: AI Evaluation Engineer
Location: Remote – Working PST schedule
Employment Type: Contract 6 months + extension opportunity
Job Description:
We are looking for engineers to join us on a 6-month contract (with the possibility of extension) our Engineering Team. The primary work is split between engineering work to port external benchmarks to run on internal infrastructure and developing novel model evaluations. You should be comfortable with fast execution speed, high velocity learning, and engineering work with clear documentation and sharp debugging.
Responsibilities

• Porting new external benchmarks to the team ʼ s internal infrastructure so they can be run as part of their evaluation stack for new model releases.
• Keeping up to date with new evals and benchmarks, pitching the team on porting newly released evals.
• Performing rigorous quality control for new and existing evals.
• Implementing novel evaluations to measure dangerous capabilities and safety of frontier models.

Requirements

• Strong Python coding experience and writing clean code fast.
• Working in a small team on a large, shared codebase.
• Experience designing and building model evaluations.
• Detail-oriented, with tenacity to dig through transcripts to identify and resolve issues.
• Ability to quickly and independently learn new skills and frameworks.
• Team player with strong communication skills.

In addition, it would be advantageous if you have

• Demonstrated research experience in the evals space.
• Experience with agentic evaluations and working with Docker.