AI Quality Engineer

Job#: 3044569

Job Description:
AI Quality Engineer

Location: Merrifield, Virginia (Hybrid) preferred. Open to Remote U.S.

Role Overview

An experienced AI Quality Engineer is sought to establish and execute the quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions. This is a specialized engineering position focused on validating AI systems, Retrieval-Augmented Generation (RAG) architectures, agent workflows, and orchestration frameworks. The successful candidate will serve as an independent reviewer and certification authority responsible for evaluating the quality and effectiveness of AI-generated outputs and agent behavior.

Key Responsibilities

• Define and implement AI certification, validation, and production readiness standards for enterprise AI agents and intelligent automation solutions.
• Develop and execute evaluation frameworks that measure accuracy, relevance, groundedness, completeness, hallucination rates, and user satisfaction.
• Design and maintain AI validation datasets, benchmark scenarios, and regression test suites using platforms such as LangSmith and Azure AI Foundry.
• Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search, Azure AI Foundry, LangChain, and LangGraph.
• Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps, Azure Functions, Azure Databricks, and related cloud-native services.
• Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, and retrieval pipelines.
• Validate security, governance, auditability, and compliance controls.
• Create production readiness and certification checklists covering observability, monitoring, logging, and operational readiness.
• Analyze LangSmith traces, agent execution logs, and AI evaluation metrics to identify quality concerns.
• Partner with development teams to remediate quality, accuracy, security, and performance issues prior to production deployment.
• Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities and developer experience improvements.

Required Qualifications

Education: A Bachelor's Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Information Systems, or a related technical field is required.

Experience: This role requires 5+ years of experience in software quality engineering, test architecture, platform engineering, software development, or machine learning engineering. A minimum of 2+ years of experience working directly with Generative AI, Large Language Models (LLMs), RAG architectures, or AI agents is necessary.

Technical Skills: Candidates must have experience with one or more of the following: Azure AI Foundry, Azure OpenAI, LangChain, LangGraph, LangSmith, AI agent frameworks, or RAG architectures. Experience with cloud-native application architectures and Azure services is also required. A strong understanding of DevSecOps, CI/CD, and enterprise software development practices is expected.

Preferred Qualifications

• Experience building or validating enterprise AI agents and multi-agent systems.
• Experience within Platform Engineering, Developer Experience (DevEx), or Site Reliability Engineering (SRE) organizations.
• Experience with AI observability platforms such as LangSmith.
• Experience validating Azure AI Search implementations and vector databases.
• Experience implementing AI governance or Responsible AI frameworks.
• Experience in financial services, banking, or other regulated industries.
• Familiarity with Azure DevOps, GitHub, and enterprise developer platforms.
• Knowledge of identity and access management concepts including Entra ID and managed identities.

Everforth Apex is a world-class IT services company that serves thousands of clients across the globe. When you join Everforth Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRateds Best of Staffing in Talent Satisfaction in the United States and Great Place to Work in the United Kingdom and Mexico.

Everforth Apex uses a virtual recruiter as part of the application process. Click for more details. By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Everforth Apex and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy at

Everforth Apex Benefits Overview: Everforth Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Everforth Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Everforth Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA. Everforth Apex has a dedicated customer service team for our Consultants that can address questions around benefits and other resources, as well as a certified Career Coach. You can access a full list of our benefits, programs, support teams and resources within our 'Welcome Packet' as well, which an Everforth Apex team member can provide.

Everforth Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Everforth Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law.

If you require an accommodation under the Americans with Disabilities Act to participate in an interview with a virtual recruiter or to use our website for a search or application, please contact our Benefits Department at or . Please note that this contact information is strictly to be used for medical ADA accommodations and that no other inquiries will be answered.

UnitedHealthcare creates and publishes the Transparency in Coverage Machine-Readable Files on behalf of Everforth Apex Systems.