Apply Now

AI Automation Engineer

Bellevue, WA, US • Posted 7 hours ago • Updated 1 hour ago

Contract W2

12 Months

No Travel Required

On-site

Depends on Experience

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

Automated Testing
Testing
Artificial Intelligence
Python
PyTest
LLM Evaluation
DeepEval
Ragas
Promptfoo
Locust
JMeter
Machine Learning Operations (ML Ops)
Apache JMeter
API
Performance Testing
Telecommunications

Summary

Job Description:

Design and build end-to-end AI testing frameworks covering unit, functional, integration, and regression testing including automated test case generation and model output validation against defined quality benchmarks.
Develop load and performance testing simulating high-concurrency scenarios to validate scalability, latency SLAs, and platform stability under peak load.
Build LLM evaluation pipelines covering relevance, grounded Ness, and hallucination metrics integrated into CI/CD workflows to enforce automated quality gates before every model or agent deployment.
Own regression testing strategy across the AI platform maintaining automated suites that validate agent behavior, RAG pipeline accuracy, and API contracts after every release to prevent quality degradation.

Must Have Skills:

Overall IT experience: Minimum 14+ years
AI Experience- 4 Years
Experience with LLM Evaluation frameworks like Prompt foo, DeepEval, Ragas etc.
Handon experience with AI Automation frameworks using Python PyTest, Unit test
Experience with Load and performance testing using Locust, JMeter etc.

Good to have skills

AIOps/Mlops knowledge
Observability for AI
Experience working on projects for Telecom operators like T-Mobile, AT&T, Verizon or OSS/BSS area,

Soft Skills

Ability to work in fast-paced, multi-disciplinary, data-driven environments.
Work closely with offshore teams with some overlap with IST.
Excellent stakeholder management

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91099677
Position Id: 9078336
Posted 7 hours ago

Contact the job poster

Anuradha Karod

Recruiter @ Talent Groups

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Bellevue, Washington

•

Today

AI Automation Engineer LLM Evaluation, Onsite We have an immediate need for an experienced AI Test Automation Engineer with strong expertise in GenAI, Agentic AI, LLM evaluation, and Python-based test automation to design and build end-to-end AI testing frameworks. This is an onsite position based in Bellevue, WA for a candidate with strong hands-on experience in LLM evaluation frameworks such as Promptfoo, DeepEval, and Ragas, along with Python automation using PyTest and Unittest. The ideal

Easy Apply

Contract

Software Development Engineer Intern (Global E-Commerce-Quality Platform & AI Test Automation) - 2027 Summer

Seattle, Washington

•

Today

Responsibilities Global E-Commerce Team is a content E-commerce business with international short video products as the carrier. It is committed to becoming the first choice for users to discover and purchase good products at affordable prices. The global E-commerce business team hopes to provide users with a more tailored and efficient consumption experience, enabling merchants to receive reliable platform services in different scenarios such as live E-commerce, and short video content E-comme

Full-time

USD 42.75 - 42.75 per hour

AI Engineer

Remote

•

6d ago

About Digipulse Digipulse Technologies, Inc (DTI) is a software solutions company offering focused IT services to fortune 1000 clients. We provide services in enterprise application development, system integration & support services for Insurance, Financial, Hospitality, Telecommunication, Pharmaceutical & Banking sectors.About the RoleRequired Qualifications Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.5-10 years of experience in software engineering

Easy Apply

Full-time, Third Party

Depends on Experience

Content Automation Specialist [211488]

No location provided

•

Today

Step into a pivotal role with a leading organization at the forefront of digital innovation, shaping the future of how people interact with technology. As an Aquent talent, you will directly influence the quality and efficiency of widely used platforms, leaving a lasting impact on how millions engage with technology globally. This is an exceptional opportunity to contribute to cutting-edge AI experiences, designing scalable evaluation systems, pioneering automated testing workflows, and developi

Full-time

Search all similar jobs

More jobs at Talent Groups in Bellevue, WA

AI Automation Engineer

Dice Job Match Score™

Job Details

Skills

Summary

Anuradha Karod

Similar Jobs