AI Observability engineer

Remote in San Francisco, CA, US • Posted 20 hours ago • Updated 39 minutes ago
Full Time
Part Time
On-site
Fitment

Dice Job Match Score™

🫥 Flibbertigibetting...

Job Details

Skills

  • AI Observability
  • AIOps

Summary

AI Observability engineer

Location: remote long term contract only on w2

AI Observability Engineer Tasks

  • Design and implement endtoend observability for AI agents, models, MCPs, and data pipelines
  • Instrument agents for traces, metrics, and logs covering prompts, tool calls, responses, latency, errors, and cost
  • Monitor agent behavior, reliability, and performance across single and multiagent systems
  • Build and operate an evaluation framework (offline + continuous) for agentic systems
  • Define offline golden test suites, regression sets, and scenariobased evaluations
  • Implement continuous, inproduction evaluations to detect quality and safety drift with alerts and thresholds\
  • Implement AI quality and safety metrics (hallucination rate, grounding accuracy, tool success rate, confidence scores)
  • Detect and alert on model drift, data drift, and concept drift impacting agent outcomes
  • Implement HumanintheLoop (HITL) review workflows for approvalgated agent actions
  • Enforce and log approvals for sensitive or highrisk tool actions
  • Define HITL triggers using confidence thresholds, escalation policies, and reviewer queues
  • Feed human feedback back into prompt updates, retrieval tuning, and agent policy improvements
  • Instrument MCPs for request/response observability and correlate MCP telemetry with agent traces
  • Integrate observability and evaluation checks into CI/CD pipelines to enable safe rollout, canarying, and rollback
  • Build dashboards and alerts for agent health, quality, safety, and usage trends
  • Ensure security, privacy, and compliance observability, including PII detection and audit logging
  • Optimize observability cost and performance across logs, metrics, traces, and evaluation runs
  • Experience implementing AI observability using AWS cloud services and opensource tooling

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 90884655
  • Position Id: OOJ - 8318-7343-1772828309
  • Posted 20 hours ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

San Francisco, California

9d ago

Easy Apply

Full-time

Depends on Experience

San Francisco, California

Yesterday

Full-time

USD 55.00 per hour

San Francisco, California

Yesterday

Full-time

San Francisco, California

Today

Full-time

USD 139,000.00 - 174,000.00 per year

Search all similar jobs