AI/LLM QA Automation Engineer (SDET)

Alpharetta, GA, US • Posted 7 days ago • Updated 7 days ago
Contract W2
Contract Corp To Corp
On-site
$75 - $78/hr
Fitment

Dice Job Match Score™

👤 Reviewing your profile...

Job Details

Skills

  • Artificial Intelligence
  • Natural Language Processing
  • Automated Testing
  • Quality Assurance

Summary

We re hiring a QA Automation Engineer with strong SDET experience to test and validate AI/LLM-powered applications. This role combines traditional automation with modern AI validation techniques to ensure accuracy, reliability, and safety of intelligent systems.

What you ll do:

  • Build and maintain automation frameworks for UI and API testing

  • Validate AI/LLM outputs using golden datasets, scoring, and statistical methods

  • Test prompt behavior, regression, and handle non-deterministic outputs

  • Perform RAG validation (retrieval quality, grounding, citations)

  • Ensure AI safety (hallucination, toxicity, PII checks)

  • Integrate tests into CI/CD pipelines

  • Test microservices, distributed systems, and cloud-based applications

  • Monitor systems using observability tools and ensure data privacy compliance

  • Conduct performance, load, and resilience testing for AI systems

Requirements:

  • 3+ years in QA Automation / SDET roles

  • Strong coding in Python / Java / TypeScript

  • Experience with Playwright, Cypress, Selenium, pytest, Postman, or REST Assured

  • CI/CD experience (Git, Jenkins, GitHub Actions, GitLab CI)

  • Solid understanding of test design techniques

  • Exposure to AI/LLM testing and handling probabilistic outputs

  • Experience testing APIs, microservices, and cloud environments

Nice to have:

  • Knowledge of NLP concepts (embeddings, temperature, context window)

  • Experience with LangChain, LlamaIndex, or similar tools

  • Observability tools (Datadog, Grafana, ELK, OpenTelemetry)

  • Docker/Kubernetes and performance testing experience

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10112696
  • Position Id: 8895206
  • Posted 7 days ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Atlanta, Georgia

Today

Easy Apply

Contract

$50 - $65

Hybrid in Alpharetta, Georgia

11d ago

Easy Apply

Contract, Third Party

Depends on Experience

Atlanta, Georgia

Today

Easy Apply

Contract

$54.42 - $59.42

Sandy Springs, Georgia

6d ago

Easy Apply

Contract

Search all similar jobs