Senior QA Engineer AI / LLM Testing

San Jose, CA, US • Posted 3 hours ago • Updated 3 hours ago

Contract Corp To Corp

Contract W2

On-site

Fitment

Dice Job Match Score™

🧠 Analyzing your skills...

Job Details

Skills

Test Plans
User Experience
Test Suites
Multilingual
Reasoning
Product Management
Quality Assurance
Regression Analysis
LangChain
Amazon Web Services
RESTful
Testing
Machine Learning (ML)
Test Cases
UI
Automated Testing
Selenium
Cucumber
Scripting
Python
Java
JavaScript
Artificial Intelligence
Productivity
Analytical Skill
Documentation
Communication
Collaboration
SANS

Summary

Design and run intelligent test strategies for LLM and agent based systems, combining classic QA and test automation skills with a deep understanding of agentic automation, and conversational AI,

Responsibilities:

Design and execute end to end test plans for UX / Uber orchestrator/ Sub Agents . Build and maintain agentic AI test suites validating autonomous .
Develop test data and synthetic scenarios for edge cases, ambiguous queries, and multilingual requests.
Evaluate agent behavior through reasoning traces, tool use decisions, and memory usage.
Implement and improve automated testing frameworks leveraging AI
for conversational flows, APIs, and integrations.
Collaborate with Product management / cross functional teams to refine requirements and acceptance criteria.
Document defects, test results, and insights clearly for stakeholders.

Skill Set:

Minimum 5 years of experience in software QA / test engineering. Strong understanding of testing principles: functional, regression, integration, performance.
Hands on experience testing AI or ML systems such as LLM chatbots or recommendation engines.
Familiarity with key concepts of multi agentic systems, agentic frameworks (e.g., LangChain), AWS AgentCore.
Scripting skills in Python and experience with REST API testing
Hands on experience with AI and Machine Learning (ML) techniques to enhance test automation, such as generating test cases, predicting potential defects, or updating test code for user interface changes.
Hands on experience with test automation frameworks and tools like Selenium, Playwright or Cucumber using scripting languages like Python, Java, or JavaScript and leveraging AI to improve productivity .
Excellent analytical, documentation, and communication skills.
Collaborate with cross functional teams for requirements clarifications and acceptance criteria.
Document defects, test results, and insights clearly for stakeholders.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10236892
Position Id: OOJ - 4450-3451-1774470628
Posted 3 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AI/ML Engineer

Santa Clara, California

•

Today

Role: AI/ ML Engineer Duration: 12+ months Location: Santa Clara, CA (Hybrid 3-4 days onsite) Top 3 Skills: The 3 skill setor experience below requires Python programming expertise. Data Analytics Data Generation Strategies/SFT experience Model Fine Tuning Skills Job Overview:As a Machine Learning Engineer, you'll work alongside Clients research and engineering teams, focused on AI Safety for LLMs, including multi-lingual, multi-modal, and reasoning models. We value expertise in data scie

Easy Apply

Third Party, Contract

Depends on Experience

AI QA Automation Engineer

Hybrid in Cupertino, California

•

Today

Role: AI QA Automation Engineer Location: Cupertino, CA OR Austin, TX (Hybrid T-W-T Onsite) Duration: 6 Months Job Description: QA Engineer focused on testing AI/ML systems and building automation frameworks to ensure model quality, reliability, and performance. Key Responsibilities Design and implement automated tests for AI/ML featuresValidate model outputs for accuracy, consistency, and edge casesBuild and maintain automation frameworks and evaluation pipelinesPerform data-driven testing for

Easy Apply

Contract

$55 - $70

Software Tester with AI

Fremont, California

•

2d ago

Locations: Fremont, CA (2 rounds, both in person interview)C2H Please looks for the profiles with good company experience. Software Tester with AIWhat to ExpectAs a Senior Software Engineer in Test (SDET) on the FinTech Team, you will engineer the automated defense systems that protect Tesla s financial backbone. We are moving beyond traditional static test scripts. Your mission is to architect intelligent, self-healing automation frameworks completely in Python that leverage Generative AI to dy

Easy Apply

Contract

$60 - $65

AI QA Automation Engineer

Hybrid in Cupertino, California

•

Today

Key ResponsibilitiesDesign and implement automated tests for AI/ML featuresValidatemodel outputsfor accuracy, consistency, and edge casesBuild and maintain automation frameworks and evaluation pipelinesPerformdata-driven testingfor datasets and model behaviorIdentify issues related to model performance, bias, and reliabilityNice to HaveExposure to LLMs or multimodal systemsUnderstanding of ML concepts and evaluation metricsExperience with Python and test automation tools

Easy Apply

Contract

Depends on Experience

Search all similar jobs