Data Analyst · Senior · Theory

A product manager looks at A/B test results every day and stops the test as soon as the p-value drops below 0.05. What's wrong with that, and how would you fix it?

~4 min by voice

How to approach it

This checks whether you understand the peeking problem: repeated checks with stopping at the first significant result break control of the type I error. The second thing they look at: whether you can offer a working alternative, a fixed design or sequential methods, understanding their cost and the organizational side.

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

ab testing sequential testing

Similar questions

A company runs dozens of A/B tests at the same time, and analysts calculate the results by hand in notebooks. Design the data side of an experimentation platform that calculates results automatically. Mid-level What are the most common mistakes people make when running A/B tests? Junior How would you plan an A/B test of a new checkout button in an online store? Junior You're planning an A/B test of a new onboarding flow, and the baseline conversion is ten percent. How would you figure out how many users you need and how long the test should run? Mid-level Design an internal A/B testing platform for a product with ten million users and a hundred concurrent experiments. What components would it consist of? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min