Ctrl K
Sign inStart free
Data Scientist · Senior · Theory

A churn model showed a ROC-AUC of 0.93 on validation, but after it went into production, retention based on its recommendations didn't improve, and quality on fresh data is much lower. What causes would you check?

~4 min by voice

How to approach it

The interviewer wants to see whether you can separate two different problems: degraded model quality (leakage, wrong validation, skew, drift) and the lack of business impact (predicting churn isn't the same as the effect of an intervention; you need uplift and an A/B test).

Have you seen this question in a real interview?
Answers0
Avg. score—
Strong—
Practiced0
Topics
machine learningdata leakage
Similar questionsWhat is model overfitting, and how do you detect it? JuniorYou trained a churn prediction model, the ROC AUC on validation is 0.97, but after deploying to production the quality is much worse. What causes would you check? Mid-level

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Sign in to answer

Best with headphones · ~4 min