Lead GenAI Quality Assurance Analyst

Title: Lead GenAI Quality Assurance Analyst

Location: Remote

Terms of Employment

Contract, 6 Months (Likely Extension)

This position is remote. Candidates who are local to the DMV area and willing to attend onsite quarterly PI planning sessions in Reston, VA will be prioritized. However, candidates can be based anywhere in the United States.

The selected candidate must be comfortable working standard Eastern time zone hours.

Overview & Responsibilities

Our client is looking for a Lead GenAI Quality Assurance Analyst to join a forward-thinking team at a large health insurance provider pioneering new solutions in the exciting field of Generative AI! We are seeking a highly motivated and experienced professional play a crucial role in ensuring the quality and accuracy of cutting-edge GenAI projects. This is a unique opportunity to work with Large Language Models (LLMs), advanced AI tools, and a talented team dedicated to "breaking boundaries" in a fast-paced, innovative environment. You'll be instrumental in shaping how we test and validate complex AI systems that learn and evolve. You will

Develop and execute test strategies for Generative AI applications, focusing on validating outputs from Large Language Models (LLMs) and similar AI systems.

Utilize and gain expertise in GenAI tools such as Compass, Amazon Kendra, and Amazon SageMaker.

Employ contextual expression comparison tools and methodologies to assess the accuracy and relevance of AI-generated content, working towards defined accuracy thresholds.

Design and implement Python-based automation scripts for testing GenAI functionalities.

Perform back-end validation as needed, though this is a lower priority.

Collaborate within an agile-like environment on "speeding bullet" projects requiring minimal supervision and a proactive approach.

Participate in understanding and mitigating issues like AI hallucination and ethical biases in GenAI outputs.

Required Qualifications

A minimum of 1 year of direct experience in testing and validating Generative AI systems or components, including Large Language Models (LLMs).

A minimum of 8 years of overall experience in software quality assurance and testing.

Strong proficiency in Python for test automation.

Demonstrable ability to write SQL code on the fly to define data populations for testing.

Proven experience in triaging and fixing Java code on the fly.

Excellent problem-solving skills and the ability to work independently and take initiative on fast-moving projects.

Familiarity with testing methodologies for non-deterministic systems where outputs can vary.