Ctrl K
Sign inStart free
Data Scientist · Mid-level · Theory

You trained a churn prediction model, the ROC AUC on validation is 0.97, but after deploying to production the quality is much worse. What causes would you check?

~4 min by voice

How to approach it

The interviewer checks whether the candidate recognizes that suspiciously high validation quality is more likely a symptom of data leakage and a flawed validation scheme than a success, and whether they can systematically work through the gap between the offline metric and production: skew, drift, how it's measured.

Have you seen this question in a real interview?
Answers0
Avg. score—
Strong—
Practiced0
Topics
machine learningdata leakage
Similar questionsWhat is model overfitting, and how do you detect it? JuniorA churn model showed a ROC-AUC of 0.93 on validation, but after it went into production, retention based on its recommendations didn't improve, and quality on fresh data is much lower. What causes would you check? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Sign in to answer

Best with headphones · ~4 min