Apply Now

Principal Coding Annotator / LLM Evaluation Engineer

Remote • Posted 1 hour ago • Updated 1 hour ago

Full Time

On-site

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

Art
Large Language Models (LLMs)
Artificial Intelligence
Workflow
Software Engineering
Code Refactoring
Debugging
Reasoning
Software Development
Python
Code Review
Research
Fluency
English
Team Leadership
Mentorship
Evaluation

Summary

This is a contracting engagement - initially 6 months - with potential for long term engagement.

Location: Paris or London-based preferred; alternatively Europe remote for strong candidates

We are building and evaluating state-of-the-art large language models (LLMs) and are looking for experienced software engineers to join our evaluation and annotation team. This role sits at the intersection of real-world software engineering, model evaluation, and applied AI , and is critical to improving model reliability, reasoning, and code quality.

You will design challenging coding tasks, evaluate model outputs against rigorous benchmarks, identify failure modes, and contribute to reinforcement learning and model improvement workflows.

This is not a junior annotation role. We are looking for practitioners with deep hands-on coding experience who can think like both an engineer and an evaluator.

What You'll Do

Evaluate coding tasks involving software vulnerabilities, exploit verification, and security patches.
Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
Identify and document model failures, edge cases, and reasoning gaps.
Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
Build or configure coding environments to support evaluation and reinforcement learning (RL).
Follow detailed annotation and evaluation guidelines with high consistency.

What We're Looking For

5+ years of professional software development experience .
Strong Python skills (required).
Knowledge of at least one additional programming language (bonus).
Experience with professional code review, coding annotation, LLM/code evaluation, or benchmark design is a plus, but not required.
Hands-on experience with vulnerability research, exploit reproduction or verification , or i mplementing, backporting, or validating security patches .
Proven ability to apply structured evaluation criteria and write clear technical feedback.
Fluent in English (written and spoken) .
Team lead or mentoring experience is a strong plus.

Why This Role

Work hands-on with cutting-edge LLMs.
Apply real-world engineering judgment to model evaluation and improvement.
High-impact, technical work with a focused, senior team.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: vonqfeed2
Position Id: 967609876
Posted 1 hour ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Remote or Grenoble, Auvergne-Rhône-Alpes

•

Today

This is a contracting engagement - initially 6 months - with potential for long term engagement. Location: Paris or London-based preferred; alternatively Europe remote for strong candidates We are building and evaluating state-of-the-art large language models (LLMs) and are looking for experienced software engineers to join our evaluation and annotation team. This role sits at the intersection of real-world software engineering, model evaluation, and applied AI , and is critical to improving m

Full-time

ML Engineer - LLM Evaluation & Automation

Remote

•

Today

We are seeking a highly skilled Machine Learning Engineer who specializes in leveraging Large Language Models (LLMs) for automated evaluation and quality assessment. In this role, you will design and build systems that automatically measure and improve the accuracy, relevance, and consistency of model outputs. You will lead initiatives to create evaluation pipelines, develop metrics, and deliver actionable insights for continuous improvements. This position requires strong technical expertise, a

Full-time

Remote Software Engineer

Remote

•

Yesterday

About Us: Based in San Francisco, California, Turing is the worlds leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing supports customers in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, plus top AI researchers who specialize in software engineering, logical reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to

Easy Apply

Contract

200 - 300

Remote Senior Software Developer

Remote

•

Yesterday

Easy Apply

Contract

200 - 300

Search all similar jobs

Remote jobs at Braintrust