AI Quality Engineer

Hybrid in Austin, TX, US • Posted 8 hours ago • Updated 8 hours ago
Contract W2
12 Months
No Travel Required
Hybrid
Depends on Experience
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • AI
  • QA
  • LLM
  • Automation
  • RAG
  • Prompt

Summary

Position : AI Quality Engineer

Location: Austin, TX (Hybrid Onsite)

Duration: 12+ Months Contract

 

Job Description:

AI Quality Engineering (AI QE) Engineer

We are seeking an AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for our next-generation conversational AI and agentic systems. In this role, you will design automated evaluation pipelines, lead Guardrail teaming and safety assessments, and develop test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.

Key Responsibilities

- Design & Build AI Eval Pipelines: Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows.

- LLM-as-a-Judge & Heuristic Scoring: Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators.

- Adversarial & Safety Testing (Guardrail Validations): Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases.

- Partner with ML/Platform Engineers: Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors.

- Synthetic Test Data Generation: Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets.

- CI/CD Integration: Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.

 

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: mategr
  • Position Id: AS-317856
  • Posted 8 hours ago
Contact the job poster
Sandesh Sawant

Sandesh Sawant

Recruiter @ ClifyX

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.