Position : AI Quality Engineer
Location: Austin, TX (Hybrid Onsite)
Duration: 12+ Months Contract
Job Description:
AI Quality Engineering (AI QE) Engineer
We are seeking an AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for our next-generation conversational AI and agentic systems. In this role, you will design automated evaluation pipelines, lead Guardrail teaming and safety assessments, and develop test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.
Key Responsibilities
- Design & Build AI Eval Pipelines: Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows.
- LLM-as-a-Judge & Heuristic Scoring: Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators.
- Adversarial & Safety Testing (Guardrail Validations): Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases.
- Partner with ML/Platform Engineers: Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors.
- Synthetic Test Data Generation: Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets.
- CI/CD Integration: Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.