We are looking for a AI Automation QA Engineer who are local to DMV area. This position requried 100% Onsite and In-Person Interview in Baltimore,MD.
AI/LLM Testing: Validating AI outputs, including GenAI hallucination detection, prompt-response accuracy, and evaluating models using metrics like BLEU or perplexity.
Test Automation: Developing and maintaining automated test suites for AI systems, including integrating tools like PyTest, Selenium, or specialized AI frameworks such as RAGAS and promptfoo into CI/CD pipelines.
Collaboration: Working closely with data scientists, developers, and product managers to define, test, and implement AI features within Agile environments.
Performance Monitoring: Monitoring AI system metrics such as latency, cost, and model drift
Develop and document test cases, testing plans and procedures in an agile environment.
Develop, execute and coordinate IT software tests and evaluate results to ensure compliance with applicable regulations.
Design and prepare all test data needed. Reviews test results and evaluates for conformance to design.
Develop and document test cases, testing plans and procedures in an agile environment.
Develop and maintain automated regression and integration test plans for validation.
General Experience: The proposed candidate must have at least eight (8) years of information systems quality assurance experience. 2–5+ years of experience in software quality assurance, with at least 1–2 years specifically in AI/ML or automation testing.
Technical Skills: Proficiency in Python (essential for GenAI libraries like LangChain and LlamaIndex) and API testing tools (Postman, REST).
Domain Knowledge: Solid understanding of machine learning concepts, data pipelines, and NLP (natural language processing).
Drive software quality assurance lifecycle within an Agile process
Establish and coordinate test strategies with development/product teams
Design and implement test plans and test cases
Develop and execute automated UI and functional tests
Enhance and maintain automated CI flows
Never miss an opportunity! Create an alert based on the job you applied for.