Design, develop, and deploy AI agents that automate engineering workflows translating requirements into working agentic systems that configure experiments, analyze results, identify regressions, and recommend follow-up actions.
Build and maintain the evaluation harness for agentic systems: regression suites, golden sets, and scoring rubrics for non-deterministic outputs, integrated into the release pipeline so changes ship on evidence, not intuition.
Architect the ingestion and storage pipeline for agent telemetry at scale, keeping the query surface fast and actionable during live incidents.
Instrument cost, latency, and token consumption per agent, per tool, and per routing path, surfacing efficiency signals to engineers and architects shaping platform decisions.
Advance the trace and event model for agent execution tool calls, routing decisions, handoffs, retries, terminal outcomes so any production run can be reconstructed and explained.
Collaborate with data science partners on anomaly detection, drift detection, and failure clustering across agent execution history.
Deliver instrumentation as composable libraries, making observability standards the path of least resistance for agent authors across the organization.
Lead design reviews and shape observability and evaluation best practices across the agentic ecosystem.
Never miss an opportunity! Create an alert based on the job you applied for.
