Senior QA Analyst – Sierra AI Platform Requirement

Senior QA Analyst – Sierra AI Platform

Function: Quality Assurance / Testing Engagement Type: Project-Based
Experience Level: Senior / Experienced Reports To: QA Lead / Project Manager

Role Overview

We are seeking an experienced QA professional with demonstrated hands-on expertise in the Sierra AI platform to join our team. The candidate will be responsible for ensuring the quality, reliability, and performance of Sierra AI-powered implementations across conversational AI workflows, agent configurations, and enterprise integrations. This role is critical to delivery success within the PMO framework and requires someone who can hit the ground running with minimal ramp-up.

Note: Proven hands-on experience with Sierra AI is mandatory. Candidates without direct, demonstrable Sierra AI experience will not be considered.

Key Responsibilities

• Sierra AI Testing: Design, execute, and maintain comprehensive test plans specific to Sierra AI agent flows, knowledge base configurations, conversation routing logic, and AI response quality.
• Functional & Regression Testing: Validate end-to-end Sierra AI workflows including intent recognition, escalation paths, fallback handling, and CRM/system integrations.
• Prompt & Response Validation: Test LLM-based prompts and AI-generated responses for accuracy, tone adherence, and compliance with business rules defined in Sierra.
• Integration Testing: Verify Sierra AI integrations with backend systems (CRM, ticketing, knowledge management platforms) and confirm data flow integrity.
• Defect Management: Log, track, and drive resolution of defects through the SDLC; communicate issues clearly to developers, architects, and the PMO.
• Test Automation: Leverage or build automation frameworks suited for AI/conversational platform testing where applicable.
• UAT Support: Assist business stakeholders during User Acceptance Testing; document sign-off criteria and outcomes for PMO records.
• Quality Metrics & Reporting: Produce test summary reports, defect dashboards, and QA status updates aligned with PMO reporting cadence.
• Risk Identification: Proactively flag quality risks related to Sierra AI behavior, configuration drift, or model updates.

Mandatory Requirements

• Hands-on Sierra AI experience is non-negotiable — candidates must be able to demonstrate practical knowledge of the platform (e.g., agent design, topic configuration, testing workflows within Sierra).
• Minimum 5+ years of overall QA/testing experience.
• Minimum 1+ years of direct, project-based experience working with Sierra AI (formerly Sierra, the AI customer service platform).
• Demonstrated experience testing conversational AI / virtual agent / chatbot platforms.
• Proficiency in writing and executing test cases for AI/ML-driven applications, including non-deterministic output validation strategies.
• Strong understanding of REST APIs and experience with API testing tools (Postman, REST Assured, or equivalent).
• Familiarity with defect tracking tools (Jira, Azure DevOps, or similar).

Preferred Qualifications

• Experience with other AI/CX platforms (e.g., Salesforce Einstein, Genesys, ServiceNow AI) in addition to Sierra.
• Exposure to LLM evaluation techniques — hallucination detection, relevance scoring, confidence thresholds.
• Background in BPO, insurance, financial services, or healthcare domains where Sierra AI is commonly deployed.
• Knowledge of Agile/Scrum methodology and ability to work within sprint-based delivery cycles.
• Scripting skills in Python or JavaScript for test automation purposes.

Soft Skills & Expectations

• Strong verbal and written communication skills for cross-functional collaboration.
• Ability to work independently and manage testing timelines with minimal supervision.
• Comfortable working within PMO governance structures — providing timely status updates, maintaining documentation, and participating in project reviews.
• Detail-oriented with a proactive, solution-first mindset.
• Able to manage ambiguity inherent in AI-based testing environments.