Senior AI Quality Engineer – GenAI / LLM Evaluation & Agentic AI

Hybrid in Austin, TX, US • Posted 7 hours ago • Updated 7 hours ago
Contract Independent
Contract W2
12 Months
No Travel Required
Hybrid
Depends on Experience
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • AI Quality Engineer
  • LLM
  • Python
  • Testing
  • Evaluation

Summary

Job Title: Senior AI Quality Engineer – GenAI / LLM Evaluation & Agentic AI

Work Location: Austin, TX - Hybrid

Contract duration: Long term contract

Visa: USC/Only

 

Detailed JD:

We are seeking an AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for our next-generation conversational AI and agentic systems. In this role, you will design automated evaluation pipelines, lead Guardrail teaming and safety assessments, and develop test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.

 

Key Responsibilities

  • Design & Build AI Eval Pipelines: Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows.
  • LLM-as-a-Judge & Heuristic Scoring: Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators.
  • Adversarial & Safety Testing (Guardrail Validations): Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases.
  • Partner with ML/Platform Engineers: Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors.
  • Synthetic Test Data Generation: Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets.
  • CI/CD Integration: Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: mategr
  • Position Id: Nilesh_payal 212
  • Posted 7 hours ago
Contact the job poster
Murtuza Tajir

Murtuza Tajir

Recruiter @ ClifyX

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.