Delivery Guarantees and Idempotency
The intuition: a courier and a package
Picture a courier bringing you a package. They can work in different ways:
- Toss it over the fence and drive off. Fast, but if the package lands in a puddle, nobody will know. That's at-most-once: delivered no more than once, and maybe not at all.
- Keep ringing the doorbell until you sign. If you signed but the receipt got lost, the courier comes back again, and now you have two packages. That's at-least-once: guaranteed delivery, but duplicates are possible.
- Exactly one package, always. That's exactly-once. Sounds perfect, but in a distributed system it's nearly impossible to honestly guarantee.
Queues work the same way. A message can get lost on the way. Or it can be processed, but the acknowledgment (ack) never reaches the broker, so the broker sends the message again.
The three guarantees side by side
| Guarantee | When to ack | Risk | Where it fits |
|---|---|---|---|
| At-most-once | before processing | lost messages | metrics, logs, "like was seen" |
| At-least-once | after processing | duplicates | payments, orders, almost anything important |
| Exactly-once | transactionally | complexity, limitations | within a single ecosystem (Kafka Streams) |
Why is exactly-once mostly marketing? Kafka can do exactly-once, but only as long as the data stays inside Kafka. The moment your handler writes to someone else's database or calls a payment API, the broker can no longer control it. Between "done" and "acknowledged" there's always a window where a failure can happen.
So in practice you build it like this: at-least-once + an idempotent handler = effectively exactly-once. Duplicates arrive, but processing them again changes nothing.
In the interview, say it plainly: "There's no exactly-once across different systems. I'll take at-least-once and make the consumer idempotent." That's exactly what interviewers want to hear.
Idempotency and the deduplication key
An idempotent operation gives the same result when repeated. SET balance = 100 is idempotent; balance = balance + 10 is not.
If an operation can't be idempotent by nature (charging money, sending an email), an idempotency key comes to the rescue. It's a unique ID for the message or request: message_id, order_id, an Idempotency-Key header. The sender generates it, and it stays the same across retries. The consumer remembers the keys it has processed and skips repeats.
The key condition: recording the key and the business effect must happen in one transaction. Otherwise a window for failure opens up between them again.
BEGIN;
INSERT INTO processed_messages (message_id) VALUES ('m-42')
ON CONFLICT DO NOTHING; -- 0 rows? it's a duplicate, bail out
UPDATE accounts SET balance = balance - 500 WHERE id = 7;
COMMIT;Trade-offs worth naming:
- the key table grows, so old entries are cleaned up by TTL (for example, kept for 7 days). That has to be longer than the longest possible retry;
- if the effect lives in an external system (Stripe, an SMS gateway), pass the key along. Good APIs accept an
Idempotency-Key.
Outbox: a database write and an event that stay in sync
A classic trap: a service saves an order to the database and publishes an event to Kafka. These are two different systems with no shared transaction.
- Saved to the database, crashed before publishing: the event is lost, and the warehouse never hears about the order.
- Published, but the database transaction rolled back: an event about an order that doesn't exist is now wandering around the system.
The outbox pattern solves this: the event is written to an outbox table in the same transaction as the order. A separate relay process reads outbox and publishes the events to the broker.
How the relay works:
- It picks up unpublished rows: by polling every N ms, or through CDC, that is, by reading the replication log (WAL).
- It publishes them to Kafka.
- It marks the rows as sent.
If the relay crashes between steps 2 and 3, the event goes out again. That's at-least-once once more, so consumers have to be idempotent. It all fits together: the outbox guarantees the event isn't lost, and idempotency protects against duplicates.
Trade-offs: added latency (from polling, usually tens of milliseconds), extra load on the database, and one more component to monitor.
Retries with exponential backoff
A handler failed because an external service went down. If every consumer retries immediately and all at once, you'll finish that service off yourself (the thundering herd effect). So you double the delay and add random jitter:
delay = min(cap, base × 2^attempt) + random(0, delay)
| Attempt | base × 2^n (base = 1 s) | With cap = 30 s |
|---|---|---|
| 1 | 1 s | 1 s |
| 2 | 2 s | 2 s |
| 3 | 4 s | 4 s |
| 4 | 8 s | 8 s |
| 5 | 16 s | 16 s |
| 6 | 32 s | 30 s |
That's 61 seconds of waiting over 6 attempts. After that, the message goes to the DLQ (dead letter queue): someone handles it manually or with a separate process, and the main queue doesn't get stuck.
Retries are only safe on top of idempotency. Retrying a non-idempotent operation means charging someone twice.
Tell temporary errors from permanent ones: a timeout or a 503 is worth retrying, but a 400 "invalid JSON" is pointless to retry, so send that message straight to the DLQ.
How it sounds in an interview
"The broker gives at-least-once, and we ack after processing. Every message carries a message_id. In a single transaction, the consumer writes the key to a deduplication table and applies the effect. Events from the service are published through an outbox, so the database write and the event can't drift apart. Retries use exponential backoff with jitter, capped at 30 seconds, and after 6 attempts the message goes to a DLQ."
Common mistakes
- Promising exactly-once "because Kafka supports it" and forgetting about external systems.
- Generating a new idempotency key on every retry. Then it's useless.
- Checking the key and applying the effect in different transactions.
- Writing to the database and then publishing, one after the other in code, and calling that reliable.
- Retrying without jitter and without an attempt limit, then being surprised by infinite loops and a flood of requests.
If you can link these four ideas into one chain (at-least-once → idempotency → outbox → retries with a DLQ), you're already answering like a strong mid-level engineer, edging toward senior.