Data Analyst · Product Analyst · Senior · Theory

In an experiment you randomize users, and the target metric is CTR, that is, clicks divided by impressions. How do you correctly calculate the variance and test for significance?

~4 min by voice

How to approach it

The interviewer checks whether the candidate understands the mismatch between the unit of randomization and the unit of analysis, and can correctly estimate the variance of a ratio metric (delta method, bootstrap, linearization) instead of applying a naive z-test on impressions.

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

ab testing ratio metrics

Similar questions

A company runs dozens of A/B tests at the same time, and analysts calculate the results by hand in notebooks. Design the data side of an experimentation platform that calculates results automatically. Mid-level What are the most common mistakes people make when running A/B tests? Junior How would you plan an A/B test of a new checkout button in an online store? Junior You're planning an A/B test of a new onboarding flow, and the baseline conversion is ten percent. How would you figure out how many users you need and how long the test should run? Mid-level A product manager looks at A/B test results every day and stops the test as soon as the p-value drops below 0.05. What's wrong with that, and how would you fix it? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min