Data Analyst · Senior · Theory

A product manager looks at A/B test results every day and stops the test as soon as the p-value drops below 0.05. What's wrong with that, and how would you fix it?

~4 min by voice

How to approach it

This checks whether you understand the peeking problem: repeated checks with stopping at the first significant result break control of the type I error. The second thing they look at: whether you can offer a working alternative, a fixed design or sequential methods, understanding their cost and the organizational side.

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

ab testing sequential testing

Similar questions

A company runs dozens of A/B tests at the same time, and analysts calculate the results by hand in notebooks. Design the data side of an experimentation platform that calculates results automatically. Mid-level Your A/B tests run for four weeks, and the business wants decisions faster without losing sensitivity. What methods would you use? Senior In an A/B test you compare ten metrics across five segments, and in one segment one metric went up significantly. The product manager wants to roll the feature out to that segment. What's your take? Mid-level What are the most common mistakes people make when running A/B tests? Junior How would you plan an A/B test of a new checkout button in an online store? Junior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min