Senior AI/ML Ops Engineer
Agent Evaluation, Observability & Production Reliability
About the Role
We are looking for a Senior AI/ML Ops Engineer to own the evaluation and reliability backbone of our AI agent ecosystem. You will design and build a comprehensive evaluation framework from the ground up, then use it to measure, monitor, and continuously improve a portfolio of production LLM-powered agents. This is a high-visibility role that sits at the intersection of ML engineering, platform operations, and quality: when a critical agent misbehaves in production, you will be the person teams look to for fast, rigorous triage and durable resolution.
You will work cross-functionally with agent developers, product owners, and infrastructure teams to define what “good” looks like for each agent, encode it into automated evaluations, wire those evaluations into CI/CD so regressions are caught before they ship, and build the reporting that gives leadership a clear, trustworthy view of agent quality over time.
What You’ll Do
Design and build an end-to-end evaluation framework for LLM-based agents, covering offline evaluation, regression testing, online monitoring, and human-in-the-loop review.
Develop and maintain evaluation suites across multiple agents: golden datasets, task-completion and accuracy metrics, LLM-as-judge rubrics, safety and guardrail checks, latency and cost benchmarks.
Integrate evaluations into CI/CD pipelines as quality gates, so every prompt, model, tool, or orchestration change is automatically evaluated before promotion to production.
Build monitoring, alerting, and observability for agents in production, including tracing of multi-step agent runs, drift detection, and anomaly detection on quality and behavioral metrics.
Create dashboards and recurring reporting that communicate agent performance, quality trends, incident history, and evaluation coverage to engineering teams and leadership.
Lead the triage and resolution of highly complex, critical agent issues in production: reproduce failures, isolate root causes across prompts, models, tools, retrieval, and infrastructure, and drive fixes through to verified resolution.
Partner cross-functionally with agent developers, product, and platform teams to define acceptance criteria, prioritize fixes, and establish runbooks, severity levels, and escalation paths for agent incidents.
Establish and champion best practices for agent versioning, release management, rollback, canary deployments, and A/B evaluation of agent changes.
Continuously improve the evaluation platform itself: expand coverage, reduce evaluation runtime and cost, and automate away manual review wherever quality allows.
Must Have
5+ years of experience in ML engineering, MLOps, platform engineering, or SRE, including 2+ years working hands-on with LLMs or LLM-powered applications in production.
Demonstrated experience building evaluation systems for ML or LLM applications: test harnesses, benchmark datasets, automated scoring (including LLM-as-judge approaches), and regression detection.
Strong software engineering skills in Python (and ideally TypeScript), with a track record of building reliable, well-tested internal platforms and tooling.
Deep familiarity with CI/CD systems (e.g., GitHub Actions, GitLab CI, Jenkins, Buildkite) and experience embedding automated quality gates into deployment pipelines.
Experience with observability and monitoring stacks (e.g., OpenTelemetry, Datadog, Grafana/Prometheus) and, ideally, LLM-specific observability tools (e.g., LangSmith, Langfuse, Arize Phoenix, Braintrust, W&B Weave).
Proven ability to debug complex distributed systems under pressure, including production incident response, root-cause analysis, and blameless postmortems.
Excellent cross-functional communication: able to translate evaluation results into clear findings and recommendations for both engineers and non-technical stakeholders.
Comfort with ambiguity and a builder’s mindset: this role starts with a blank page and ends with the evaluation platform the whole organization relies on.
Experience with agentic frameworks and orchestration patterns (e.g., multi-agent systems, tool use, RAG pipelines) and their distinct failure modes.
Experience with prompt management, model routing, or fine-tuning workflows and evaluating changes across model versions and providers.
Background in statistics or experimentation (A/B testing, significance testing, sampling strategies for human review).
Design and build reusable AI agent skills, plugins and maintain internal marketplace infrastructure to extend and scale Data, AIML capabilities across the organization.
Expertise in causal inference and measurement strategy including causal graphs, ontologies, and knowledge graphs to drive rigorous, decision grade data analysis.
Experience operating in regulated or high-stakes domains where agent errors carry real business or customer impact.
Success in the first 90 days
A production-grade evaluation framework is in place, with automated evaluation suites covering every critical agent.
CI/CD quality gates catch agent regressions before release, with clear pass/fail criteria trusted by agent teams.
Production agents have real-time quality monitoring and alerting, with documented runbooks and severity-based escalation paths.
Leadership receives regular, reliable reporting on agent quality, and time-to-resolution for critical agent incidents has measurably decreased.
Never miss an opportunity! Create an alert based on the job you applied for.