We are seeking an Observability and Evaluation Engineer with strong AI/ML expertise to build, monitor, evaluate, and optimize AI-driven applications, LLMs, and Agentic AI systems. The ideal candidate will have hands-on experience in observability, telemetry, monitoring, distributed systems, and AI model evaluation frameworks.
<>Required Skills
- 7+ years of experience in Software Engineering, SRE, MLOps, DevOps, or Observability Engineering.
- Strong experience in Observability, Monitoring, Telemetry, Logging, and Distributed Tracing.
- Hands-on expertise with OpenTelemetry, Grafana, Prometheus, Splunk, Datadog, Dynatrace, or New Relic.
- Experience building and monitoring AI/ML applications, LLMs, and RAG-based systems.
- Strong programming skills in Python and familiarity with Java or Golang.
- Experience with LLM evaluation frameworks, prompt testing, model performance analysis, and quality metrics.
- Knowledge of Vector Databases, embeddings, and Retrieval-Augmented Generation (RAG).
- Experience with AWS, Azure, or Google Cloud Platform cloud platforms.
- Strong understanding of Kubernetes, Docker, CI/CD, and MLOps practices.
<>Preferred Skills
- Experience with LangChain, LangGraph, LlamaIndex, MLflow, Arize, TruLens, LangSmith, or Weights & Biases.
- Knowledge of Agentic AI architectures and evaluation methodologies.
- Familiarity with AI safety, hallucination detection, and model guardrails.
- Experience with Kafka, Elasticsearch, and event-driven architectures.
- Understanding of SLI/SLO implementation for AI systems.
<>Key Responsibilities
- Design and implement observability frameworks for AI/ML and distributed applications.
- Build telemetry, logging, monitoring, and tracing solutions using OpenTelemetry and industry-standard tools.
- Develop evaluation pipelines for LLMs, AI agents, and RAG systems.
- Measure model accuracy, latency, reliability, hallucination rates, and business KPIs.
- Create dashboards, alerts, and AI system health monitoring solutions.
- Analyze production telemetry and model performance data to identify optimization opportunities.
- Establish AI quality metrics, evaluation standards, and benchmarking frameworks.
- Collaborate with AI Engineers, Data Scientists, Platform Teams, and SREs to improve system reliability and model effectiveness.
<>Education
- Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, or a related field.
<>Keywords
Observability, Evaluation Engineering, AI/ML, LLM, Agentic AI, RAG, OpenTelemetry, Monitoring, Telemetry, Logging, Distributed Tracing, LangChain, LangGraph, LangSmith, TruLens, MLflow, Arize, Python, Kubernetes, Docker, AWS, Azure, Google Cloud Platform, MLOps, SRE, Prometheus, Grafana, Datadog, Splunk.
