A/B tests and other experimental designs -Sr. Data Analyst(Only w2)

Title: Sr. Data Analyst

Location: Orlando, FL 32819

Duration: 12 Months Contract

Job Description:

• Client is seeking a Senior Data Analyst to lead experimentation and causal analysis efforts that inform product, marketing, and business decisions. This person will design, execute, and interpret A/B tests and other experimental frameworks, applying rigorous statistical methods to separate true causal impact from noise. This role partners closely with Product, Engineering, and Marketing stakeholders to translate business questions into testable hypotheses and turn results into clear, actionable recommendations.
• The ideal candidate combines strong statistical fundamentals with hands-on experience with statistics, machine learning and AI; and is comfortable owning experiments end-to-end (design, instrumentation review, analysis, readout) and can clearly communicate methodology and results to both technical and non-technical audiences.

Responsibilities

• Design, implement, and analyze A/B tests and other experimental designs (e.g., multivariate tests, holdout groups, switchback experiments) across product and marketing initiatives.
• Apply causal inference techniques including difference-in-differences, propensity score matching, synthetic control, regression discontinuity, and instrumental variables for scenarios where randomized experiments aren't feasible.
• Determine appropriate statistical tests (t-tests, chi-square, ANOVA, non-parametric alternatives, sequential testing, Bayesian methods) based on data characteristics and business context.
• Calculate sample size and power requirements, and advise stakeholders on minimum detectable effect, test duration, and experiment feasibility before launch.
• Build reusable Python-based analysis pipelines and tooling (e.g., using pandas, NumPy, SciPy, statsmodels) to standardize experiment analysis and reduce time-to-insight.
• Write and optimize SQL to extract, transform, and validate experiment and behavioral data from large, complex datasets.
• Partner with Product Managers, Engineers, and Marketers to scope experiments, define success metrics and guardrail metrics, and ensure proper randomization and instrumentation.
• Proactively identify data quality issues, sample ratio mismatches, novelty effects, and other threats to experiment validity, and flag them before they affect decisions.
• Communicate experiment design, methodology, and results clearly to stakeholders with varying levels of technical expertise, through written reports and presentations.
• Build dashboards and visualizations (e.g., in Tableau, Looker, or Python plotting libraries) to track experiment health and results in near real time.
• Contribute to the evolution of the company's experimentation platform, standards, and playbooks.

Required Qualifications

• Bachelor's or Master's degree in Statistics, Data Science, Economics, Mathematics, Computer Science, or a related quantitative field.
• 5+ years of hands-on experience in data analysis, with 3+ years specifically designing and analyzing A/B tests or experiments in an industry setting.
• Strong proficiency in Python for statistical analysis and data manipulation (pandas, NumPy, SciPy, statsmodels, or similar).
• Demonstrated expertise in statistical hypothesis testing (t-tests, chi-square, ANOVA) and in causal inference methods beyond simple A/B testing (e.g., diff-in-diff, matching methods, regression discontinuity, uplift modeling).
• Practical understanding of experimental design concepts: randomization, power analysis, minimum detectable effect, novelty/primacy effects, network effects, and common pitfalls (e.g., peeking, multiple comparisons, Simpson's paradox).
• Ability to translate ambiguous business questions into rigorous, testable analyses and communicate the resulting trade-offs clearly.
• Experience presenting technical findings to non-technical stakeholders, including executives, in a clear and compelling way.
• Comfort working with large, sometimes messy, real-world data and identifying/troubleshooting data integrity issues.