Principal/Staff Data Scientist, GenAI/LLM Evaluation
Our direct client, a prestigious global firm, is creating a new advanced-technology organization. They are standing up AI evaluation as a first-class function and need a genuinely senior leader to build it.
The firm’s products and agents will do various business functions which must be auditable. The standard is correct results every time, not “good enough for a personal assistant.” Because large language models are probabilistic, this takes real science. You will decide how to measure non-deterministic systems. You will find where they fail, including the corner cases nobody predicted. And you will design controls for the behavior that can’t be made deterministic.
This is not a software-engineering or agent-building role. Other teams build the agents and own deployment and DevOps. You define how we know whether the agents can be trusted, and you build the team that does it.
Note: this work can be performed from home, anywhere in the US. It is also open as a full-time salaried position as an alternative to long-term contract.
What you’ll do
- Establish and lead the evaluation function: set the vision, hire and develop leaders and build a team that runs with minimal day-to-day oversight.
- Design evaluation harnesses and architecture for agentic and generative AI systems, both during development and in production.
- Apply statistical and mathematical rigor to stochastic systems. That means deciding whether behavior can be made deterministic, how many runs are enough, how to separate real regressions from noise, and how to tell “succeeds at least once” from “succeeds every time.”
- Where determinism isn’t possible, define fallback measures and compensating controls that detect and contain failures.
- Monitor agents after deployment, detect failures and unexpected corner cases, and feed what you learn back into evaluation.
- Define what “correct” means for multi-step work: accuracy against known-true data, sound reasoning and tool use, conclusions traceable to sources, and compliance with accounting and regulatory expectations.
- Build and calibrate model-based graders by measuring agreement with human reviewers, checking for bias and watching for drift.
- Partner with the engineering and DevOps teams who own agent development and release pipelines. Your results shape ship decisions.
Must have
- A strong data science foundation: probability, statistics and linear algebra
- Experience in both traditional machine learning and generative AI. Hands-on GenAI work is essential, since the evaluation problem is most severe there.
- Hands-on experience designing or establishing evaluation infrastructure or architecture for ML or AI systems, built in practice and not just studied
- The ability to walk through specific initiatives, lessons learned, problems encountered and how you solved them
- Senior leadership experience: building teams, developing leaders and setting technical direction
- Enough software engineering background to understand how systems are built and operated. About 5–6 years is sufficient if the rest of your career is in ML, data science or AI engineering.
- Working understanding of CI/CD, deployment and production concepts. You won’t own them.
- BS/MS or higher in a quantitative field such as statistics, data science, computer science or engineering. An advanced degree is a plus.
Nice to have
- Regulated-industry exposure (audit, tax, accounting, financial services, regtech)
- Experience making decisions explainable and auditable
- Production monitoring and observability for ML or LLM systems
- Experience with multi-agent systems or retrieval-based systems
- Security and adversarial testing of AI (prompt injection, data leakage)
- Hands-on use of public agent benchmarks or evaluation frameworks. Tell us which ones and what you’d change.
Immediate interview and start. Send your resume today, with a short note on the evaluation systems you’ve built and what you learned from them.
By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Benchmark IT, LLC and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here: bmarkits.com/privacy-policy
