Data / AI QE Lead — Retail eCommerce
Location-San Ramon, California or Beverly Hills, CA (5 Days Onsite)
Job Type-Long Term Contract
Role Summary
The Data / AI QE Lead will define the quality engineering strategy for data pipelines, machine learning models, and AI-powered features across the retail eCommerce platform. This role bridges traditional data quality assurance and emerging AI/ML validation disciplines, ensuring that customer-facing capabilities — including product recommendations, personalization, search relevance, demand forecasting, and pricing intelligence — perform accurately, fairly, and reliably at scale.
The role will build foundational QE practices for data and AI, partner with data engineering, data science, and product teams, and translate complex model behavior into measurable, business-aligned quality standards.
Key Responsibilities
Data Quality Engineering
Define and own the QE strategy for data assets including customer, product, inventory, transaction, and behavioral event data
Design and implement data validation frameworks covering completeness, accuracy, consistency, timeliness, and referential integrity
Lead testing of ETL/ELT pipelines, data lake and warehouse layers (raw, curated, consumption), and real-time streaming pipelines
Establish data contract testing practices between producing and consuming systems
Build automated data quality monitors and alerting that operate continuously in production environments
Partner with data governance and data stewardship teams to align QE standards with enterprise data policies
AI / ML Model Quality & Validation
Lead quality validation for ML models powering eCommerce capabilities: product recommendations, personalized search, dynamic pricing, demand forecasting, propensity models, and generative AI features
Define model evaluation frameworks including offline metrics and online business metrics (CTR, conversion rate, AOV, revenue lift)
Design and execute A/B and shadow testing strategies to validate model performance before and during production rollout
Assess and test for model fairness, bias, and regulatory compliance across customer segments and product categories
Validate model monitoring and drift detection systems to ensure production models remain within acceptable performance thresholds
Define rollback and circuit-breaker criteria for AI features that degrade customer experience
eCommerce Platform Integration Testing
Drive end-to-end quality of data flows from customer interaction events through to AI feature delivery on site, app, and email channels
Test integrations between the eCommerce platform and downstream data consumers including CDP, CRM, marketing automation, and analytics tools
Validate real-time personalization pipelines for homepage, PDP, cart, and post-purchase experiences
Ensure data quality for key eCommerce events: product views, add-to-cart, checkout, order confirmation, returns, and search queries
Test search and browse relevance improvements driven by ML rankers and query understanding models
Test Automation & Observability
Build and scale automated data and AI testing frameworks integrated into CI/CD and model deployment pipelines
Define and enforce data quality SLAs and embed automated gates into pipeline orchestration (Airflow, dbt, Spark, etc.)
Implement observability tooling for data pipelines and AI model inputs/outputs in collaboration with data and ML engineering
Drive adoption of synthetic data and data masking strategies to support safe, representative testing environments
Establish version-controlled, repeatable test datasets for regression testing of ML models across release cycles
Cross-Functional Partnership
Collaborate with data scientists, data engineers, product managers, and business analysts to define acceptance criteria for data and AI deliverables
Champion a culture of data quality ownership across data producers and consumers in the eCommerce organization
Qualifications
Required
7+ years in data or quality engineering, with at least 2 years leading a team or technical discipline
Proven experience testing data pipelines (batch and streaming) across modern data stack technologies (Spark, Kafka, Airflow, dbt, Snowflake, BigQuery, Databricks, or similar)
Hands-on experience with ML model evaluation techniques, including offline metrics and online experimentation (A/B testing)
Strong SQL skills and proficiency in Python for data validation scripting and test automation
Familiarity with eCommerce data domains: customer behavior, product catalog, order management, inventory, and digital marketing
Excellent ability to communicate data and AI quality concepts to technical and non-technical stakeholders
Preferred
Familiarity with Generative AI applications (RAG pipelines, LLM-powered features) and emerging AI QE practices
Knowledge of data privacy regulations (GDPR, CCPA) and their implications for test data management
Experience in high-scale eCommerce environments (peak traffic events, flash sales, seasonal demand spikes)
Never miss an opportunity! Create an alert based on the job you applied for.
