AI Automation QA Engineer - 100% Onsite and In-Person interview in Maryland

Job Description:

Summary:
Lead quality management for AI-driven information systems, ensuring accuracy, reliability, and performance of ML models, chatbots, and generative AI applications, with a focus on probabilistic testing within Agile environments.

Responsibilities

• Establish and monitor capable processes; drive root cause analysis, corrective actions, and continuous improvement.
• Define QA strategies and quality plans that align with industry standards, regulations, and customer requirements.
• Develop and implement lifecycle and QA methodologies and metrics.
• Validate AI/LLM outputs, including hallucination detection, prompt–response accuracy, and metrics such as BLEU and perplexity.
• Develop and maintain automated test suites for AI systems using PyTest, Selenium, and AI-focused tools like RAGAS and promptfoo; integrate into CI/CD pipelines.
• Monitor AI metrics such as latency, cost, and model drift; collaborate with data scientists, developers, and product managers.
• Design, develop, and document test plans, test cases, test data, regression and integration suites; execute tests and report results.
• Drive QA lifecycle in Agile, coordinate test strategies, enhance CI flows, and troubleshoot system issues.

Requirements

• At least 8 years of information systems QA experience.
• Minimum 5 years with statistical methods and quality standards.
• 2–5+ years in software QA, including 1–2 years specifically in AI/ML or automation testing.
• Strong QA/process knowledge and experience with automation tools; excellent written and verbal communication.
• Proficiency in Python (for GenAI libraries such as LangChain and LlamaIndex) and API testing tools (Postman, REST).
• Solid understanding of ML concepts, data pipelines, and NLP.
• Familiarity with CI/CD tools (Jenkins, GitLab), Git, Docker, and Kubernetes.

Education & Domain Preference

• Bachelor’s in Engineering, Computer Science, Information Systems, or related field.

Preferred: Experience with State government programs and systems such as Child Support, Integrated Eligibility, Child Welfare, Adult Protective Services, and Juvenile Justice, plus experience analyzing and troubleshooting production issues.