AI Quality Engineer (GenAI / RAG / Agentic AI) - Pheonix, AZ - Contract

Phoenix, AZ, US • Posted 1 day ago • Updated 1 day ago
Contract Corp To Corp
Contract W2
Contract Independent
12 Months
No Travel Required
On-site
$0 - $0/yr
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Assembly
  • Autogen
  • Automated Testing
  • Banking
  • Business Strategy
  • Docker
  • Embedded Systems
  • Evaluation
  • Generative Artificial Intelligence (AI)
  • Continuous Delivery
  • Continuous Integration
  • Cloud Computing
  • Cloud Testing
  • Computer Science
  • PASS
  • Python
  • Quality Assurance
  • RESTful
  • Management
  • Microservices
  • Microsoft Azure
  • Orchestration
  • Insurance
  • Kubernetes
  • LangChain
  • LangSmith
  • Data Science
  • Git
  • Health Care
  • Information Technology
  • Vector Databases
  • Workflow
  • Security QA
  • Regression Testing
  • Reporting
  • Risk Management
  • Reasoning
  • Recovery
  • Recruiting
  • Regression Analysis
  • SQL
  • Selenium
  • Semantics
  • Testing
  • Vertex

Summary

We at NJ TECH INC are focused on hiring highly skilled professionals who are excited by the opportunity to make a true impact on their careers as well as on our clients' business. We power our clients’ success—and drive our consultants’ career growth.

We are seeking an experienced and outstanding AI Quality Engineer (GenAI / RAG / Agentic AI)  for one of our esteemed clients.

Role: AI Quality Engineer (GenAI / RAG / Agentic AI)
Work Location: Pheonix, AZ
Work Mode: Onsite
Employment Type: Long-Term Contract

Job Summary
We are seeking a hands-on AI Quality Engineer to design and implement quality-engineering strategies for Generative AI, Retrieval-Augmented Generation (RAG), and Agentic AI applications.
The ideal candidate will have strong Python automation skills and hands-on experience evaluating AI applications to ensure accuracy, reliability, safety, performance, and production readiness. The role involves building automated evaluation frameworks, establishing quality benchmarks, and integrating AI testing into CI/CD pipelines.
Key Responsibilities
  • Design and implement end-to-end testing strategies for LLM, RAG, and Agentic AI applications.
  • Build automated evaluation frameworks to measure answer correctness, relevance, groundedness, faithfulness, hallucination, retrieval quality, tool selection, and agent task completion.
  • Develop and maintain golden datasets, benchmark suites, and prompt/model regression test packs.
  • Test RAG pipelines covering document ingestion, chunking, embeddings, vector retrieval, reranking, context assembly, and response generation.
  • Validate multi-step agent workflows, including planning, reasoning, tool execution, memory, exception handling, guardrails, and human-in-the-loop controls.
  • Perform negative, boundary, adversarial, prompt-injection, jailbreak, and sensitive-data leakage testing.
  • Validate APIs, microservices, enterprise integrations, and downstream systems accessed by AI agents.
  • Automate functional, API, integration, and end-to-end testing using Python, PyTest, Playwright, Selenium, and REST API frameworks.
  • Integrate AI evaluations and automated tests into CI/CD pipelines with actionable reporting and quality gates.
  • Monitor production quality across accuracy, latency, reliability, token consumption, and cost.
  • Identify and troubleshoot failures across prompts, retrieval, data, models, orchestration, tools, and application code.
Required Skills
AI / GenAI Quality Engineering
  • Strong understanding of LLMs, Generative AI, RAG, embeddings, vector search, and Agentic AI.
  • Experience evaluating non-deterministic AI outputs and defining acceptance thresholds.
  • Knowledge of RAG evaluation, prompt regression, hallucination detection, grounding, and agent/tool-call validation.
  • Ability to create representative golden datasets and benchmark suites.
Automation Engineering
  • Strong hands-on experience with Python and PyTest.
  • REST API testing and automation using Playwright or Selenium.
  • Experience building reusable test automation frameworks.
  • Experience integrating automated tests with Git-based CI/CD pipelines.
  • Working knowledge of SQL, microservices, Docker, and Kubernetes.
AI Evaluation & Observability
  • Hands-on exposure to DeepEval, RAGAS, Giskard, LangSmith, Langfuse, or custom Python evaluation frameworks.
  • Understanding of LLM-as-a-Judge, deterministic checks, human evaluation, tracing, and production observability.
Cloud & Platform
  • Experience with Azure OpenAI / Azure AI Foundry, Amazon Bedrock, or Google Vertex AI.
  • Working knowledge of vector databases, model endpoints, identity and access controls, secrets management, and cloud monitoring.
Quality Metrics
  • Response Quality: Answer relevance, correctness, faithfulness, groundedness, and hallucination rate.
  • RAG Quality: Context precision/recall, Retrieval Precision@K and Recall@K, and citation accuracy.
  • Agent Quality: Task-success rate, tool-call accuracy, workflow completion, and recovery rate.
  • Non-functional: P95 latency, availability, throughput, token usage, and cost per transaction.
  • Safety: Prompt-injection resistance, jailbreak success rate, toxicity, bias, and PII leakage.
  • Regression: Automated pass rate, model/prompt variance, and escaped defect rate.
Preferred Qualifications
  • Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science, or a related discipline.
  • Experience testing production GenAI or RAG applications in regulated enterprise environments such as banking, insurance, or healthcare.
  • Working knowledge of LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, or comparable orchestration frameworks.
  • Exposure to Responsible AI, model risk management, security testing, and AI governance.
  • Relevant cloud, testing, security, or AI certifications.
Success Measures
  • Reduced AI defects escaping into production.
  • Measurable improvement in grounding, retrieval, and answer quality.
  • High automated regression coverage across prompts, models, RAG pipelines, and agent workflows.
  • Reliable AI quality gates embedded in delivery pipelines.
  • Reduced manual validation effort and faster release cycles without compromising safety or reliability.
Ideal Candidate Profile
A strong quality-engineering and automation professional with hands-on GenAI/RAG knowledge, advanced Python skills, and demonstrated ability to measure AI quality programmatically and diagnose failures across the complete AI application stack.

NJ TECH INC is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, We cannot sponsor any visa at this time

NJ TECH INC, is a globally managed IT service, consulting and Business solutions partner. Our "High-Performance Business" strategy builds up our expertise in technology and consulting. Our consulting plays a major role in helping clients to achieve their objectives in the highest level; ultimately creating sustainable value for customers. Come, transform your career with us and be a part of our high-performing team
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 91092521
  • Position Id: 9144791
  • Posted 1 day ago
Contact the job poster
NS

Neil Saks

Recruiter @ NJTECH INC.

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.