Job Description:
The Quality Assurance Consultant provides quality management for information systems using the standard methodologies, techniques, and metrics for assuring product quality and key activities in quality management. This individual is responsible for performing the following tasks:
- Establishing capable processes, monitoring and control of critical processes and product mechanisms for feedback of performance, implementing effective root cause analysis and corrective action system, and continuous process improvement;
- Providing strategic quality plans in targeted areas of the organization;
- Providing QA strategies to ensure continuous production of products consistent with established industry standards, government regulations, and customer requirements; and
- Developing and implementing life cycle and QA methodologies and educating and implementing QA metrics. Ensures the accuracy, reliability, and performance of AI-driven systems, including machine learning models, chatbots, and generative AI (GenAI) applications.
Education: This position requires a Bachelor’s degree from an accredited college or university in Engineering, Computer Science, Information Systems or a related discipline.
Responsibilites:
● AI/LLM Testing: Validating AI outputs, including GenAI hallucination detection, prompt-response accuracy, and evaluating models using metrics like BLEU or perplexity.
● Test Automation: Developing and maintaining automated test suites for AI systems, including integrating tools like PyTest, Selenium, or specialized AI frameworks such as RAGAS and promptfoo into CI/CD pipelines.
● Collaboration: Working closely with data scientists, developers, and product managers to define, test, and implement AI features within Agile environments.
● Performance Monitoring: Monitoring AI system metrics such as latency, cost, and model drift
● Develop and document test cases, testing plans and procedures in an agile environment.
● Develop, execute and coordinate IT software tests and evaluate results to ensure compliance with applicable regulations.
● Design and prepare all test data needed. Reviews test results and evaluates for conformance to design.
● Develop and document test cases, testing plans and procedures in an agile environment.
● Develop and maintain automated regression and integration test plans for validation.
● Communicate test results and feedback with business and IT stakeholders
General Experience:
The candidate must have at least eight (8) years of information systems quality assurance experience.
2–5+ years of experience in software quality assurance, with at least 1–2 years specifically in AI/ML or automation testing.
Tools & Frameworks: Familiarity with CI/CD tools (Jenkins, GitLab), version control (Git), and containerization (Docker, Kubernetes).
Specialized Experience:
The candidate must have at least five (5) years of experience working with statistical methods and quality standards. This individual must have working QA/process knowledge, use of automation tools within the QA process of building software and possess superior written and verbal communication skills.
Technical Skills: Proficiency in Python (essential for GenAI libraries like LangChain and LlamaIndex) and API testing tools (Postman, REST).
Domain Knowledge: Solid understanding of machine learning concepts, data pipelines, and NLP (natural language processing).
● Drive software quality assurance lifecycle within an Agile process
● Establish and coordinate test strategies with development/product teams
● Design and implement test plans and test cases
● Develop and execute automated UI and functional tests
● Enhance and maintain automated CI flows
● Troubleshoot issues in systems under test
Never miss an opportunity! Create an alert based on the job you applied for.