Mid-Level AI Software Test Engineer

Location: Ann Arbor, MI Salary: $43.00 USD Hourly - $48.00 USD Hourly Description: Our client is currently seeking a Mid-Level AI Software Test Engineer

Location: Ann Arbor, MI (100% Onsite)

Duration: 18 months

Note: This position requires 100% onsite attendance for the first 6 months. Following the initial 6-month period, the schedule transitions to a hybrid model consisting of 4 days onsite and 1 remote day of the candidate's choosing per week.

About the job:

We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines traditional software quality engineering with emerging AI validation techniques to ensure our systems are reliable, safe, accurate, performant, and production-ready.

You will be responsible for establishing reliable automated AI testing coverage, detecting quality issues before production deployment, and improving overall confidence in AI model behavior. Example projects include testing AI chat assistants, validating external tool integrations and agent workflows, evaluating RAG search quality and grounding accuracy, and performance testing AI services. This role is highly collaborative and is suitable for an engineer who can independently own AI testing and automation efforts while partnering closely with developers, architects, and product teams to deliver safe, scalable features.

Work Location & Schedule Details: This position requires 100% onsite attendance for the first 6 months. Following the initial 6-month period, the schedule transitions to a hybrid model consisting of 4 days onsite and 1 remote day of the candidate's choosing per week.

Minimum qualifications:

Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related technical field.

3 years of experience in software testing, QA automation, or quality engineering.

Experience developing automated test solutions using Python, Java, JavaScript/TypeScript, or C#.

Experience with API testing and test automation tools, such as Selenium, Playwright, Postman, JUnit, or PyTest.

Experience working with software development lifecycles (SDLC), Agile methodologies, and CI/CD pipelines including GitHub Actions or Jenkins.

Experience testing distributed systems, web applications, and APIs.

Preferred qualifications:

1 to 3 years of applied experience or exposure to AI/ML or Generative AI technologies.

Experience testing Generative AI applications, LLM-based systems, AI agents, RAG applications, and MCP-based integrations.

Familiarity with foundational models and platforms, including OpenAI, Claude, Gemini, or Azure OpenAI.

Experience building evaluation and benchmarking frameworks for AI solutions.

Experience testing cloud-native applications on Azure, AWS, or Google Cloud Platform, as well as familiarity with Docker and Kubernetes.

Knowledge of responsible AI, AI governance, AI risk management, and compliance practices.

Responsibilities:

Develop comprehensive testing strategies and execute functional, integration, end-to-end, regression, and performance tests for AI applications and services.

Validate AI model outputs, prompt-driven workflows, agentic systems, and retrieval-based processes for accuracy, consistency, reliability, and safety.

Perform adversarial, negative, and edge-case testing to identify model failures, hallucinations, and validate AI guardrails and business rules.

Build and maintain automated test frameworks, evaluation pipelines, reusable test data, and simulators, integrating them directly into CI/CD pipelines for quality scoring.

Test AI workflows, APIs, tool-calling capabilities, and integrations with external enterprise systems and data sources.

Execute load and stress testing to verify the performance, reliability, scalability, and resiliency of AI workloads.

Partner with software engineers, AI developers, product owners, and security teams during design reviews to provide quality feedback, drive best practices, and support production readiness and defect triage.

By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates (collectively "Judge") to such phone number regarding job opportunities, your job application, and for other related purposes. Message & data rates apply and message frequency may vary. Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared with third parties for marketing/promotional purposes. Reply STOP to opt out of receiving telephone calls and text messages from Judge and HELP for help.
Contact:
This job and many more are available through The Judge Group. Please apply with us today!