We are currently looking for a Mid-Level AI Software Test Engineer / AI Quality Engineer for a long-term contract opportunity in Ann Arbor, MI.
Position Details
Job Title: Mid-Level AI Software Test Engineer
Location: Ann Arbor, MI
Contract Duration: 18 Months with Possible Extension
Employment Type: Contract
Work Arrangement: Onsite
Schedule:
First 6 months: 100% onsite
After 6 months: 4 days onsite / 1 day remote
Candidate can choose their remote day
Position Summary
We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and Generative AI solutions.
This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready.
The ideal candidate will have a strong software testing background, experience building automated test frameworks, and exposure to AI technologies such as LLMs, AI agents, RAG, MCP integrations, and machine learning models.
Key Responsibilities
AI Quality Engineering
Develop testing strategies for AI applications, platforms, and services
Validate AI model outputs for accuracy, consistency, reliability, and safety
Perform functional, integration, end-to-end, regression, and performance testing
Create test cases for prompt-driven, agentic, and retrieval-based AI workflows
Validate AI guardrails, business rules, permissions, and governance controls
Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations
Test Automation
Build and maintain automated test frameworks for AI applications
Develop automated evaluation pipelines for AI responses and workflows
Integrate AI testing into CI/CD pipelines
Implement automated quality scoring and regression detection
Create reusable test data, mocks, simulators, and validation frameworks
Platform & Integration Testing
Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities
Validate integrations with external systems, data sources, and enterprise services
Test performance, reliability, scalability, and resiliency of AI workloads
Execute load and stress testing for AI services
Collaboration
Partner with software engineers, AI engineers, product owners, architects, and security teams
Participate in design reviews and provide quality feedback
Contribute to test strategy, quality standards, and best practices
Support production readiness reviews and defect triage
Required Qualifications
Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or related field
3–6 years of software testing, QA automation, or quality engineering experience
Experience developing automated test solutions using one or more of:
Python
Java
JavaScript/TypeScript
C#
Experience with API testing and automation tools
Strong understanding of:
Test Automation
SDLC
Agile methodologies
CI/CD pipelines
Experience testing distributed systems, web applications, and APIs
Preferred Qualifications
Experience testing:
Familiarity with OpenAI, Claude, Gemini, or Azure OpenAI
Experience building AI evaluation and benchmarking frameworks
Experience testing cloud-native applications on Azure, AWS, or Google Cloud Platform
Knowledge of responsible AI, AI governance, risk management, privacy, and compliance
Technical Skills
AI Testing
Quality Engineering
Test Automation
API Testing
Integration Testing
Performance Testing
Load Testing
Security Testing
Defect Analysis
Root Cause Investigation
Tools & Technologies
Typical Projects
Testing AI chat assistants and copilots
Validating MCP tools and AI agent workflows
Evaluating RAG search quality and grounding accuracy
Automating AI response evaluation frameworks
Testing AI-powered workflow automation
Performance testing AI services and orchestration platforms
Experience Level
3–6 years of software quality engineering experience, with approximately 1–3 years of exposure to AI/ML or Generative AI technologies.
Equivalent Titles
Never miss an opportunity! Create an alert based on the job you applied for.
