Job Details
Title: A/B Test Data Scientist (Website Experimentation)
Location: Sunnyvale, CA / Austin, TX
Duration: Full Time
Job Description:
Role Overview
As an A/B Test Data Scientist, you will be the statistical engine behind our website’s optimization strategy. You will design, execute, and analyze controlled experiments to improve user experience, conversion rates, and business growth. You will work at the intersection of product, engineering, and marketing to ensure every website change is backed by rigorous data.
Key Responsibilities
Phase | Core Duties |
Strategy & Design | Formulate clear hypotheses and define primary/secondary metrics. Determine required sample sizes, power, and test duration. |
Execution | Oversee randomization and user segmentation. Partner with engineers to ensure proper tracking and data integrity. |
Analysis | Apply statistical methods (p-values, confidence intervals) to determine significance. Conduct post-test deep dives to identify segment-specific impacts. |
Strategy & Reporting | Translate complex data into clear, actionable recommendations for stakeholders. Maintain an experimentation roadmap and archive learnings. |
Required Skills & Qualifications
Statistical Expertise: Mastery of hypothesis testing, experimental design, and probability. Knowledge of Bayesian vs. Frequentist approaches is a plus.
Programming: Proficiency in SQL for data extraction and Python or R for advanced statistical analysis.
Tools: Experience with experimentation platforms (e.g., Optimizely, VWO, or Launch Darkly) and web analytics (e.g., Google Analytics 4).
Visualization: Ability to build dashboards in Tableau, Power BI, or Looker to monitor live tests.
Soft Skills: Strong critical thinking and the ability to explain "why" a result occurred to non-technical partners.
Experience & Education
Education: Bachelor’s degree in a quantitative field (Statistics, Math, CS, Economics); Master’s or PhD preferred.
Experience: 2–5 years in a data science or product analytics role, specifically focused on digital experimentation
Never miss an opportunity! Create an alert based on the job you applied for.
