AI/ML with testing Experience - SFO, CA - Hybrid

Title: AI/ML Engineer with Testing Experience

Location: SFO, CA

Hybrid Model 3 days onsite

Key Responsibilities

Strategy & Leadership

• Define and own the end-to-end quality strategy, test approach, and release readiness criteria across squads.
• Lead, mentor, and grow a team of QA engineers; establish career paths, skill matrices, and a culture of continuous improvement.
• Drive shift-left testing practices, ensuring quality gates in PRs, CI/CD, and design reviews.

Automation & Frameworks

• Architect and maintain Python-based automation frameworks (e.g., PyTest, Selenium, Playwright, Robot Framework) for UI, API, integration, and end-to-end tests.
• Implement data-driven and behavior-driven testing (BDD) with tools like Behave/Cucumber where applicable.
• Standardize test design patterns (Page Object, Screenplay, fixtures, test data services) and enforce code quality (linting, type hints, reviews).

AI-Enabled Quality

• Integrate AI-assisted testing (e.g., intelligent test case generation, flaky test detection, failure clustering, anomaly detection in logs).
• Collaborate with Data Science/ML teams to validate ML models, including dataset integrity, bias checks, model drift monitoring, and functional/non-functional validation of inference services.
• Evaluate and, where appropriate, adopt AI-powered test platforms (e.g., Mabl, Testim) or build in-house utilities using scikit-learn/PyTorch/TensorFlow for prioritization and defect prediction.

CI/CD & DevOps Quality

• Embed tests into CI/CD pipelines (GitHub Actions/Jenkins/Azure DevOps/GitLab CI), enabling parallelization, shards, and caching.
• Define and monitor quality gates (code coverage, mutation testing, static analysis, performance thresholds).
• Orchestrate environment management using Docker/Kubernetes, service mocks, test data services, and synthetic data generation.

Quality Operations

• Establish metrics and reporting (DRE, escape rate, MTTR, flaky rate, coverage, defect aging) with dashboards (Grafana/PowerBI).
• Lead root cause analyses and drive corrective/preventive actions (CAPA).
• Partner with Product and Engineering on release planning, risk assessment, and sign-off.

Required Skills & Experience

• Python: Advanced proficiency; building robust test frameworks, utilities, parsers, and CLI tools; strong OOP and familiarity with concurrency (asyncio), typing, packaging.
• Automation: Hands-on with PyTest, Selenium/Playwright, Requests, Robot Framework; API testing (REST/GraphQL), contract testing (Pact), and service virtualization/mocking.
• AI/ML Knowledge: Understanding of ML lifecycle (data prep, model training/evaluation, drift monitoring), and AI-assisted testing concepts (prioritization, flaky test detection, anomaly detection). Ability to use pandas, NumPy, scikit-learn for analytics.
• CI/CD & DevOps: Experience integrating tests into pipelines, containerized testing, environment orchestration, and test parallelization.
• Performance & Reliability: Exposure to load/stress testing (JMeter/Locust/k6) and reliability checks (resilience, chaos testing basics).
• Cloud & Tools: Familiarity with AWS/Google Cloud Platform/Azure, Docker/K8s; version control (Git), issue tracking (Jira/Azure Boards).
• Leadership: Proven experience leading QA teams, setting standards, coaching, and delivering across multiple releases.