Data Engineer · Mid-level · Theory

A daily pipeline loads yesterday's data into the warehouse. Sometimes it fails halfway through, and after a restart duplicates show up in the table. How would you redesign it?

~4 min by voice

How to approach it

This checks whether the candidate understands pipeline idempotency: a rerun for the same period should give the same result, rather than being patched with retries and after-the-fact duplicate cleanup. Plus whether they know the specific mechanisms: overwriting a partition, MERGE, atomic switching, parameterizing the date.

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

etl idempotency

Similar questions

Write a handler for creating a payment that takes an idempotency key and the payment data and returns the result. A repeat request with the same key must not charge the money a second time, even if two identical requests arrive simultaneously on different instances of the service; you have a database and an external payment provider. Senior Design a system of daily batch pipelines for several hundred tables with dependencies between them. How would you handle reruns, backfills and late-arriving data? Senior You need to deliver changes from a multi-terabyte production PostgreSQL database to an analytics warehouse with no more than five minutes of latency. How would you design this pipeline? Senior A client sends a POST request to create a payment, the response never arrives because of a timeout, and the client retries the request. How do you make sure the money isn't charged twice? Mid-level A client calls your create-payment endpoint, the response is lost to a timeout, and the client retries the request. How would you design the API so the money isn't charged twice? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min