Question library
Questions from real interviews. Drill any of them by voice in a couple of minutes, with an interviewer, a score and feedback.
How would you plan an A/B test of a new checkout button in an online store?JuniorWhat are the most common mistakes people make when running A/B tests?JuniorIn an A/B test you compare ten metrics across five segments, and in one segment one metric went up significantly. The product manager wants to roll the feature out to that segment. What's your take?Mid-levelYou're planning an A/B test of a new onboarding flow, and the baseline conversion is ten percent. How would you figure out how many users you need and how long the test should run?Mid-levelA company runs dozens of A/B tests at the same time, and analysts calculate the results by hand in notebooks. Design the data side of an experimentation platform that calculates results automatically.Mid-levelA product manager looks at A/B test results every day and stops the test as soon as the p-value drops below 0.05. What's wrong with that, and how would you fix it?SeniorIn an experiment you randomize users, and the target metric is CTR, that is, clicks divided by impressions. How do you correctly calculate the variance and test for significance?SeniorYour A/B tests run for four weeks, and the business wants decisions faster without losing sensitivity. What methods would you use?SeniorDesign an internal A/B testing platform for a product with ten million users and a hundred concurrent experiments. What components would it consist of?Senior