Apply Now

Observability and Evaluation Engineer

Charlotte, NC, US • Posted 9 hours ago • Updated 9 hours ago

Contract Independent

Contract Corp To Corp

Contract W2

12 Months

No Travel Required

Able to Sponsor

On-site

$60 - $65/hr

IMR Soft LLC

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

AI
LLM
Observability
Evaluation

Summary

Observability and Evaluation Engineer

Charlotte, NC

12 months

Implements telemetry, traces, dashboards, evaluation suites, s, service objectives, runbooks, and readiness evidence for priority agent releases.

LLM and agent evaluation; tracing and telemetry; metrics and dashboards; ing; SLOs; test automation; prompt and model performance analysis; Python; production operations.

We are seeking an Observability and Evaluation Engineer to build the monitoring, evaluation, and reliability infrastructure behind our priority AI agent releases. This role owns the full observability lifecycle — from telemetry and tracing to evaluation suites, alerting, and production readiness — ensuring agent systems are measurable, reliable, and accountable at scale.

Key Responsibilities

Design and implement telemetry pipelines and distributed tracing for LLM and agent-based systems
Build dashboards and metrics to monitor model/agent performance, latency, cost, and reliability
Develop and maintain evaluation suites for LLM and agent quality, accuracy, and regression testing
Define and track Service Level Objectives (SLOs) and error budgets for agent services
Implement alerting systems to proactively detect degradation, drift, or failures
Create and maintain runbooks for incident response and operational troubleshooting
Produce production readiness evidence and documentation for agent release approvals
Conduct prompt and model performance analysis to identify optimization opportunities
Build and maintain test automation frameworks supporting continuous evaluation
Collaborate with engineering teams to support production operations of agent-based systems

Required Skills & Experience

Strong hands-on experience with LLM and agent evaluation methodologies and frameworks
Proficiency in tracing and telemetry tools (e.g., OpenTelemetry, Datadog, Grafana, or similar)
Experience building metrics, dashboards, and alerting systems for production systems
Solid understanding of SLOs, error budgets, and reliability engineering practices
Strong Python programming skills
Experience with test automation for ML/LLM systems
Background in production operations and incident/runbook management
Analytical skills for prompt and model performance evaluation

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91133294
Position Id: 19650-10625-1788963323
Posted 9 hours ago

Company Info

About IMR Soft LLC

We're building a great company, and we're very excited about the future of the company.

IMR Soft. is one of the fastest-growing, USA based Information Technology company specializing in software development and IT Services. Founded in 2017, by a team of experienced professionals with a solid base in IT and Management. Headquartered in Princeton, New Jersey.

We help clients Create the Future. We deliver innovative information technology solutions and unlimited services that benefit our clients by maximizing their performance. We combine tech expertise and business intelligence to catalyze change and deliver results. In a world that is constantly changing, companies are faced with the challenge of continually adapting to a dynamic environment.

Go to company profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Principal GenAI Engineer

Hybrid in New York, New York

•

9d ago

Principal GenAI Engineer Location: NYC, NY - 3 Days WFO Employment Type: Full Time About the Role hiring a Principal GenAI Engineer with strong expertise in LLMs to lead enterprise-scale AI implementations for Fortune 500 clients. This role focuses on building RAG systems that combine structured semantic reasoning with advanced LLM architectures to deliver scalable, explainable, production-grade AI solutions. What Were Looking For 8-13 years of experience in ML/AI systems 2+ years hands-on ex

Easy Apply

Full-time, Third Party

$180000 - $200000

GenAI Engineer (Agentic AI)

Hybrid in New York, New York

•

6d ago

Position: GenAI Engineer (Agentic AI) Location; NYC, NY ( In person interview must) Duration: 12 months Role Overview We are looking for a hands-on GenAI Engineer to build and scale agent-based AI applications, with a strong focus on Claude Agent SDK (Anthropic). The role centers on developing intelligent, production-grade workflows for document processing, onboarding, and operational automation, integrating GenAI capabilities into enterprise platforms. Key Responsibilities Build agent-

Easy Apply

Contract, Third Party

$65 - $70

AI Guardrails Engineer

Concord, California

•

Today

AIGuardrails Engineer Concord, CA 12 months contract We are seeking an AI Guardrails Engineer to strengthen the safety and control posture of our AI and agent systems. This role automates adversarial testing, validates guardrail effectiveness, and packages repeatable evidence that supports model and agent enablement gates ensuring releases meet responsible AI and regulatory standards before approval. Key Responsibilities Design and automate adversarial and red-team testing programs for LLMs a

Easy Apply

Third Party, Contract

$60 - $65

UAT lead

Sunnyvale, California

•

12d ago

UAT lead 6-12 months Sunnyvale, CA 8+ years of experience managing User Acceptance Testing (UAT) as a test lead for complex, highly integrated projects Understanding of sales and sales operations processes in consumer channel sales or similar environment is highly preferred Well versed in multiple styles of Release and Change Management processes including Agile/Scrum, Waterfall, and Iterative; Experience in an Agile environment is a must Hands-on experience with Salesforce platform is pref

Easy Apply

Third Party, Contract

$45 - $50

Search all similar jobs

More jobs at IMR Soft LLC in Charlotte, NC