Mid-Level AI Software Test Engineer / AI Test Engineer - W2 Only

We are currently looking for a Mid-Level AI Software Test Engineer / AI Quality Engineer for a long-term contract opportunity in Ann Arbor, MI.

Position Details

Job Title: Mid-Level AI Software Test Engineer
Location: Ann Arbor, MI
Contract Duration: 18 Months with Possible Extension
Employment Type: Contract
Work Arrangement: Onsite
Schedule:

• First 6 months: 100% onsite

• After 6 months: 4 days onsite / 1 day remote

• Candidate can choose their remote day

Position Summary

We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and Generative AI solutions.

This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready.

The ideal candidate will have a strong software testing background, experience building automated test frameworks, and exposure to AI technologies such as LLMs, AI agents, RAG, MCP integrations, and machine learning models.

Key Responsibilities

AI Quality Engineering

• Develop testing strategies for AI applications, platforms, and services

• Validate AI model outputs for accuracy, consistency, reliability, and safety

• Perform functional, integration, end-to-end, regression, and performance testing

• Create test cases for prompt-driven, agentic, and retrieval-based AI workflows

• Validate AI guardrails, business rules, permissions, and governance controls

• Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations

Test Automation

• Build and maintain automated test frameworks for AI applications

• Develop automated evaluation pipelines for AI responses and workflows

• Integrate AI testing into CI/CD pipelines

• Implement automated quality scoring and regression detection

• Create reusable test data, mocks, simulators, and validation frameworks

Platform & Integration Testing

• Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities

• Validate integrations with external systems, data sources, and enterprise services

• Test performance, reliability, scalability, and resiliency of AI workloads

• Execute load and stress testing for AI services

Collaboration

• Partner with software engineers, AI engineers, product owners, architects, and security teams

• Participate in design reviews and provide quality feedback

• Contribute to test strategy, quality standards, and best practices

• Support production readiness reviews and defect triage

Required Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or related field

• 3–6 years of software testing, QA automation, or quality engineering experience

• Experience developing automated test solutions using one or more of:

  • Python

  • Java

  • JavaScript/TypeScript

  • C#

• Experience with API testing and automation tools

• Strong understanding of:

  • Test Automation

  • SDLC

  • Agile methodologies

  • CI/CD pipelines

• Experience testing distributed systems, web applications, and APIs

Preferred Qualifications

• Experience testing:

  • Generative AI applications

  • LLM-based systems

  • AI agents

  • RAG applications

  • MCP-based integrations

• Familiarity with OpenAI, Claude, Gemini, or Azure OpenAI

• Experience building AI evaluation and benchmarking frameworks

• Experience testing cloud-native applications on Azure, AWS, or Google Cloud Platform

• Knowledge of responsible AI, AI governance, risk management, privacy, and compliance

Technical Skills

AI Testing

• Prompt Testing

• Response Evaluation

• Hallucination Detection

• Agent Workflow Validation

• RAG Validation

• AI Safety Testing

• AI Regression Testing

• AI Benchmarking

Quality Engineering

• Test Automation

• API Testing

• Integration Testing

• Performance Testing

• Load Testing

• Security Testing

• Defect Analysis

• Root Cause Investigation

Tools & Technologies

• Selenium

• Playwright

• Postman

• JUnit / PyTest

• GitHub Actions

• Jenkins

• Docker

• Kubernetes

• Azure / AWS / Google Cloud Platform

Typical Projects

• Testing AI chat assistants and copilots

• Validating MCP tools and AI agent workflows

• Evaluating RAG search quality and grounding accuracy

• Automating AI response evaluation frameworks

• Testing AI-powered workflow automation

• Performance testing AI services and orchestration platforms

Experience Level

3–6 years of software quality engineering experience, with approximately 1–3 years of exposure to AI/ML or Generative AI technologies.

Equivalent Titles

• AI Software Test Engineer

• AI Quality Engineer

• Generative AI Test Engineer

• AI Automation Engineer

• AI Validation Engineer

• SDET – AI