Principal/Staff Data Scientist, GenAI/LLM Evaluation, Contract-to-Hire

Remote • Posted 30+ days ago • Updated 1 day ago
Contract W2
Contract Independent
12 Months
No Travel Required
Remote
Depends on Experience
Company Branding Image
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Software Engineering
  • Technical Lead
  • Technical Architect
  • Python
  • Automating Testing
  • ML-Operations
  • LLM's
  • Agent Architecture
  • Retrieval-based Systems
  • Security and Privacy Compliance
  • Computer Science
  • Verification Infrastructure
  • Statistics
  • Machine learning
  • Generative AI
  • CI/CD
  • AI Engineering
  • ML
  • Data Science

Summary

Principal/Staff Data Scientist, GenAI/LLM Evaluation

Our direct client, a prestigious global firm, is creating a new advanced-technology organization. They are standing up AI evaluation as a first-class function and need a genuinely senior leader to build it.

The firm’s products and agents will do various business functions which must be auditable. The standard is correct results every time, not “good enough for a personal assistant.” Because large language models are probabilistic, this takes real science. You will decide how to measure non-deterministic systems. You will find where they fail, including the corner cases nobody predicted. And you will design controls for the behavior that can’t be made deterministic.

This is not a software-engineering or agent-building role. Other teams build the agents and own deployment and DevOps. You define how we know whether the agents can be trusted, and you build the team that does it.

Note: this work can be performed from home, anywhere in the US. Ideally a contract-to-hire as they are looking to get a strong person in the role right away, it could be also open as a full-time salaried position from the start.  

What you’ll do

  • Establish and lead the evaluation function: set the vision, hire and develop leaders and build a team that runs with minimal day-to-day oversight.
  • Design evaluation harnesses and architecture for agentic and generative AI systems, both during development and in production.
  • Apply statistical and mathematical rigor to stochastic systems. That means deciding whether behavior can be made deterministic, how many runs are enough, how to separate real regressions from noise, and how to tell “succeeds at least once” from “succeeds every time.”
  • Where determinism isn’t possible, define fallback measures and compensating controls that detect and contain failures.
  • Monitor agents after deployment, detect failures and unexpected corner cases, and feed what you learn back into evaluation.
  • Define what “correct” means for multi-step work: accuracy against known-true data, sound reasoning and tool use, conclusions traceable to sources, and compliance with accounting and regulatory expectations.
  • Build and calibrate model-based graders by measuring agreement with human reviewers, checking for bias and watching for drift.
  • Partner with the engineering and DevOps teams who own agent development and release pipelines. Your results shape ship decisions.


Must have

  • A strong data science foundation: probability, statistics and linear algebra
  • Experience in both traditional machine learning and generative AI. Hands-on GenAI work is essential, since the evaluation problem is most severe there.
  • Hands-on experience designing or establishing evaluation infrastructure or architecture for ML or AI systems, built in practice and not just studied
  • The ability to walk through specific initiatives, lessons learned, problems encountered and how you solved them
  • Senior leadership experience: building teams, developing leaders and setting technical direction
  • Enough software engineering background to understand how systems are built and operated. About 5–6 years is sufficient if the rest of your career is in ML, data science or AI engineering.
  • Working understanding of CI/CD, deployment and production concepts. You won’t own them.
  • BS/MS or higher in a quantitative field such as statistics, data science, computer science or engineering. An advanced degree is a plus.

Nice to have

  • Regulated-industry exposure (audit, tax, accounting, financial services, regtech)
  • Experience making decisions explainable and auditable
  • Production monitoring and observability for ML or LLM systems
  • Experience with multi-agent systems or retrieval-based systems
  • Security and adversarial testing of AI (prompt injection, data leakage)
  • Hands-on use of public agent benchmarks or evaluation frameworks. Tell us which ones and what you’d change.

Immediate interview and start. Send your resume today, with a short note on the evaluation systems you’ve built and what you learned from them.

 

By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Benchmark IT, LLC and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here: bmarkits.com/privacy-policy

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10208703
  • Position Id: 5612
  • Posted 30+ days ago

Company Info

About Benchmark IT- Technology Talent

Since 2007, Benchmark IT has delivered superior technology staffing solutions to employers throughout the New York metropolitan area and beyond. Our experienced recruiters consistently identify best-fit talent by combining the latest recruiting technology with seasoned insight and perspective.

Whether you’re an employer or a job seeker, we approach every recruiting assignment with keen precision, dedication to ethics, and personalized service. Our best-in-class recruiting process and high-touch approach produce outstanding, long-lasting placements.

Connecting the right talent with the right role at the right company creates success. Knowing how to put it all together is our expertise. Contact us today to get started.

About_Company_One
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.