AI Quality Engineering Engineer

Job Title: AI QE Engineer

Location: Canada

Role Summary

We are looking for an AI Quality Engineering Engineer with strong automation expertise and hands-on experience validating LLMs, GenAI workflows, and AI-driven applications. This role involves building automated test suites, validating AI outputs, ensuring system reliability, and contributing to the quality strategy for next-generation AI features.

Core Experience

• 4 6 years as a Software Engineer, SDET, or Automation Engineer.
• Strong coding skills in Python, TypeScript, or Java.
• Hands-on experience developing automation scripts, tools, or frameworks.
• Practical experience using LLMs, prompt engineering, and evaluating AI-generated outputs.
• Familiarity with agentic AI systems and exposure to tools like LangGraph, AutoGen, CrewAI.
• Basic understanding of Model Context Protocol (MCP) or context-aware workflow automation (nice to have).

AI / ML Technologies

• Practical experience with AI/ML frameworks: LangChain, Hugging Face, GPT models, vector databases, RAG pipelines.
• Experience with ML/DL libraries such as:
  Scikit-learn, PyTorch, TensorFlow, Keras, Transformers, OpenCV.
• Ability to work with embeddings, similarity search, and content evaluation metrics.

GenAI & AI Agent Development

• Ability to integrate or build GenAI components, including RAG pipelines or agent-based workflows.
• Support model evaluation tasks such as:
  • Output quality checks
  • Hallucination detection
  • Prompt validation
  • Regression checks for model updates

Automation & Quality Engineering

• Experience building automation using Python, PyTest, Selenium, Playwright, or API testing libraries.
• Ability to design and execute automated tests for:
  • Functional
  • Integration
  • API
  • Basic performance & reliability testing
• Hands-on experience testing RESTful APIs and building automated API suites.

Cloud, DevOps & CI/CD

• Exposure to deploying AI or automation solutions on AWS.
• Working knowledge of CI/CD pipelines, including:
  • Automated testing
  • Model validation steps
  • Versioning & artifact management

SDLC & Collaboration

• Strong understanding of the software development lifecycle, including requirements, development, testing, and defect analysis.
• Ability to collaborate with Developers, Data Scientists, and QE teams, clearly communicating progress, risks, and results.
• Skilled in documenting and tracking defects and participating in defect triage.