A daily pipeline loads yesterday's data into the warehouse. Sometimes it fails halfway through, and after a restart duplicates show up in the table. How would you redesign it?
~4 min by voice
How to approach it
This checks whether the candidate understands pipeline idempotency: a rerun for the same period should give the same result, rather than being patched with retries and after-the-fact duplicate cleanup. Plus whether they know the specific mechanisms: overwriting a partition, MERGE, atomic switching, parameterizing the date.
The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.