Job Title: AI Automation Engineer
Location: Bellevue, WA Onsite
Job Type: Full-Time
Compensation: $150,000/Year
Experience: 15+ Years of Overall IT Experience, including 3+ Years of AI/ML Experience
Job Summary
We are seeking a highly experienced AI Automation Engineer to design, develop, and implement automated testing, evaluation, observability, and quality frameworks for AI/ML platforms and applications.
The ideal candidate will have strong expertise in Python, SQL, ETL pipelines, AI/ML, AIOps, MLOps, LLMOps, AgentOps, system design, and MCP client-server architecture. The candidate will be responsible for building scalable AI testing and evaluation frameworks, ensuring the quality, reliability, performance, and observability of AI-powered systems.
Key Responsibilities
Design and build end-to-end AI testing frameworks covering unit, functional, integration, and regression testing.
Develop automated test case generation frameworks and validate AI/ML model outputs against defined quality benchmarks.
Build and execute load and performance testing frameworks to simulate high-concurrency scenarios and validate scalability, latency SLAs, and platform stability.
Develop LLM evaluation pipelines to measure relevance, groundedness, factual accuracy, hallucination, and other quality metrics.
Integrate AI/LLM evaluation frameworks into CI/CD pipelines and establish automated quality gates before model, agent, or application deployments.
Own and maintain the AI regression testing strategy, ensuring automated test suites validate platform functionality after every release.
Develop automated testing for AI agents, RAG pipelines, LLM applications, and API contracts.
Design and implement AIOps, MLOps, LLMOps, and AgentOps workflows for the complete AI lifecycle.
Build observability solutions for AI systems covering model performance, agent behavior, latency, errors, data quality, and system health.
Design and implement scalable ETL/data pipelines supporting AI/ML workflows and evaluation frameworks.
Develop AI automation solutions using Python and SQL.
Design robust system architectures for AI-powered applications, testing platforms, and automation frameworks.
Design and implement MCP (Model Context Protocol) client-server architectures and integrations.
Collaborate with data scientists, ML engineers, software engineers, DevOps teams, architects, and business stakeholders.
Identify quality, performance, reliability, and scalability risks across AI systems and implement automated solutions to address them.
Establish engineering best practices for AI quality assurance, observability, automation, and continuous delivery.
Required Technical Skills
15+ years of overall IT experience.
3+ years of hands-on AI/ML experience.
Strong experience with:
Python
SQL
ETL/Data Pipelines
Machine Learning / Artificial Intelligence
AIOps
MLOps
LLMOps
AgentOps
AI Observability
Strong understanding of AI/ML testing and automation frameworks.
Experience with LLM evaluation methodologies, including relevance, groundedness, hallucination detection, and model output validation.
Experience building RAG pipeline testing and evaluation frameworks.
Strong knowledge of CI/CD automation and quality gates.
Experience with load, performance, scalability, and high-concurrency testing.
Strong system design and architecture experience.
Hands-on understanding of MCP client-server architecture and design.
Experience designing automated workflows across the AI/ML lifecycle.
Preferred Qualifications
Experience with enterprise-scale Generative AI and LLM applications.
Experience testing and monitoring AI agents and agentic workflows.
Experience with RAG architectures, vector databases, embeddings, and retrieval evaluation.
Experience implementing AI/ML observability and monitoring solutions.
Knowledge of cloud-based AI/ML platforms and modern DevOps practices.
Experience working in large-scale enterprise environments.
Strong communication, analytical, troubleshooting, and problem-solving skills.
Education
Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science, Artificial Intelligence, or a related field preferred.
Work Location
Bellevue, WA Onsite
Employment Type & Compensation
Full-Time | $150,000 per annum
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91171926
- Position Id: OOJ - 1427-431-1786110968
- Posted 10 hours ago
Never miss an opportunity! Create an alert based on the job you applied for.