Location: Ann Arbor, MI
Salary: $43.00 USD Hourly - $48.00 USD Hourly
Description: Job Title: Mid-Level AI Software Development Engineer in Test (SDET)
Location: Ann Arbor, MI (100% On-site for the first 6 months; 4 days On-site / 1 day Remote thereafter)
Duration: 12+ months contract
Note: This is a 12+ month contract position. The first 6 months require 100% on-site attendance in Ann Arbor, MI. Following the initial 6 months, the schedule transitions to a hybrid model with 4 days on-site and 1 remote day of your choosing per week. About the job We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines traditional software quality engineering with emerging AI validation techniques to ensure our enterprise AI systems are reliable, safe, accurate, performant, and production-ready. You will take independent ownership of AI testing and automation efforts, partnering closely with developers, architects, and product teams to enable the safe and scalable delivery of intelligent applications.
Responsibilities - Develop comprehensive testing strategies for AI applications, platforms, and services, designing functional, integration, end-to-end, regression, and performance tests.
- Validate AI model outputs for accuracy, consistency, reliability, and safety, creating test cases for prompt-driven, agentic, and retrieval-based (RAG) workflows.
- Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations, while validating AI guardrails, business rules, permissions, and governance controls.
- Build and maintain automated test frameworks and evaluation pipelines for AI responses, integrating automated quality scoring and regression detection into CI/CD pipelines.
- Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities to validate integrations with external enterprise systems and data sources.
- Execute load and stress testing to verify the performance, scalability, and resiliency of AI workloads.
- Partner with software engineers, AI engineers, product owners, architects, and security teams to provide quality feedback during design reviews and support production readiness triage.
Minimum qualifications - Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field.
- 3-6 years of experience in software testing, QA automation, or quality engineering.
- Strong hands-on experience developing automated test solutions using Python, Java, JavaScript/TypeScript, or C#.
- Experience testing distributed systems, web applications, and APIs utilizing automation tools and testing frameworks (e.g., Selenium, Playwright, Postman, JUnit, PyTest).
- Deep understanding of test automation principles, the software development lifecycle (SDLC), Agile methodologies, and CI/CD pipelines (e.g., GitHub Actions, Jenkins, Docker, Kubernetes).
Preferred qualifications - 1-3 years of direct exposure to testing Generative AI applications, LLM-based systems, AI agents, RAG applications, or MCP-based integrations.
- Familiarity with leading foundation models and environments such as OpenAI, Claude, Gemini, or Azure OpenAI.
- Hands-on experience building evaluation and benchmarking frameworks for AI solutions (e.g., automating AI response evaluation, evaluating RAG search grounding accuracy).
- Experience testing cloud-native applications on enterprise cloud platforms (Azure, AWS, or Google Cloud Platform).
- Knowledge of responsible AI practices, AI governance, and enterprise risk management requirements.
Medical, dental, and vision insurance are available to qualified candidates who meet eligibility requirements.
By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates (collectively "Judge") to such phone number regarding job opportunities, your job application, and for other related purposes. Message & data rates apply and message frequency may vary. Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared with third parties for marketing/promotional purposes. Reply STOP to opt out of receiving telephone calls and text messages from Judge and HELP for help.
Contact: This job and many more are available through The Judge Group. Please apply with us today! 
