Lead GenAI Quality Assurance Analyst (W2 only)

Overview

Hybrid
$55 - $65
Contract - W2
Contract - 6 Month(s)

Skills

Amazon SageMaker
Artificial Intelligence
Large Language Models (LLMs)
Java
SQL
Python
Generative Artificial Intelligence (AI)
Test Methods
Testing
Scripting
Amazon Web Services

Job Details

Terms of Employment

  • Contract, 6 Months (Likely Extension)
  • This position is remote. Candidates who are local to the DMV area and willing to attend onsite quarterly PI planning sessions in Reston, VA will be prioritized. However, candidates can be based anywhere in the United States.
  • The selected candidate must be comfortable working standard Eastern time zone hours.

Overview & Responsibilities

Our client is looking for a Lead GenAI Quality Assurance Analyst to join a forward-thinking team at a large health insurance provider pioneering new solutions in the exciting field of Generative AI! We are seeking a highly motivated and experienced professional play a crucial role in ensuring the quality and accuracy of cutting-edge GenAI projects. This is a unique opportunity to work with Large Language Models (LLMs), advanced AI tools, and a talented team dedicated to "breaking boundaries" in a fast-paced, innovative environment. You'll be instrumental in shaping how we test and validate complex AI systems that learn and evolve. You will

  • Develop and execute test strategies for Generative AI applications, focusing on validating outputs from Large Language Models (LLMs) and similar AI systems.
  • Utilize and gain expertise in GenAI tools such as Compass, Amazon Kendra, and Amazon SageMaker.
  • Employ contextual expression comparison tools and methodologies to assess the accuracy and relevance of AI-generated content, working towards defined accuracy thresholds.
  • Design and implement Python-based automation scripts for testing GenAI functionalities.
  • Perform back-end validation as needed, though this is a lower priority.
  • Collaborate within an agile-like environment on "speeding bullet" projects requiring minimal supervision and a proactive approach.
  • Participate in understanding and mitigating issues like AI hallucination and ethical biases in GenAI outputs.

Required Qualifications

  • A minimum of 1 year of direct experience in testing and validating Generative AI systems or components, including Large Language Models (LLMs).
  • A minimum of 8 years of overall experience in software quality assurance and testing.
  • Strong proficiency in Python for test automation.
  • Demonstrable ability to write SQL code on the fly to define data populations for testing.
  • Proven experience in triaging and fixing Java code on the fly.
  • Excellent problem-solving skills and the ability to work independently and take initiative on fast-moving projects.
  • Familiarity with testing methodologies for non-deterministic systems where outputs can vary.

Preferred Qualifications

  • Hands-on experience with the GenAI tool "Compass."
  • Experience with Amazon Web Services (AWS) AI tools, particularly Amazon Kendra and Amazon SageMaker.
  • Experience with various contextual expression comparison tools and techniques.
  • Experience working on projects involving knowledge bases and querying/feedback mechanisms for AI models.
  • Located in an area allowing for occasional (e.g., quarterly) travel to an office for PI planning.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.