Replication

Intuition: the master ledger and its copies

Picture an accounting department with one master ledger. Only the chief accountant is allowed to write in it. Each branch office keeps a copy of the ledger, and a courier regularly delivers the new pages. Any employee can look things up in the nearest copy. That way the chief accountant doesn't drown in questions like "How much do we have in the account?"

There are two catches:

• the courier is sometimes late, so a copy shows yesterday's numbers;
• if the chief accountant gets sick, someone at a branch has to take over fast.

That's replication. Let's go through each catch one at a time.

The setup: leader and followers

The rules are simple:

1. Only the leader accepts writes.
2. The leader records changes in a log (the WAL in PostgreSQL, the binlog in MySQL) and sends it to the followers.
3. Reads can go to replicas. That's how you scale workloads with far more reads than writes.

Let's do the math: how many replicas do you need

Say we have a news feed at 20,000 RPS, and 90% of requests are reads.

What	Calculation	Result
Reads	20,000 × 0.9	18,000 RPS
Writes	20,000 × 0.1	2,000 RPS
Reads one node can handle	roughly	~5,000 RPS
Read replicas	18,000 / 5,000 ≈ 3.6 → 4, plus 1 spare	5 replicas
Writes	all go to the leader	2,000 RPS on a single node

The key takeaway for the interview: replication scales reads, not writes. If the leader can't keep up with writes, you need sharding, not more replicas.

Synchronous or asynchronous

The most important question: does the leader wait for the replicas to confirm before telling the client "written"?

	Synchronous	Asynchronous	Semi-synchronous
Leader waits for	all replicas	nobody	at least one replica
Write latency	high	minimal	moderate
Data loss if the leader dies	none	possible	almost none
If a replica goes down	writes stall	nothing breaks	wait for another replica

In practice, people most often choose asynchronous or semi-synchronous replication: one synchronous replica plus the rest asynchronous. A fully synchronous setup with several replicas is too fragile: one slow replica drags down the whole system.

Replication lag and "read your own writes"

With asynchronous replication, a replica falls behind the leader. Usually by milliseconds, but under load by seconds or even minutes. That's replication lag.

The classic bug looks like this: a user changes their avatar, the page reloads, and the old avatar is still there. The write went to the leader, but the read hit a lagging replica. The user decides the site is broken.

We need a read-your-writes guarantee: users always see their own changes. Here's how to provide it:

• Read your own data from the leader. The owner's profile, settings, and cart are read from the leader; everyone else's data comes from replicas.
• A window after writing. For N seconds after a write (say, 5–10), all of that user's reads go to the leader. The timestamp can live in a cookie or the session.
• Log position. Remember the LSN (the position of the user's last write) and read only from a replica that has caught up to it.
• Sticky replica. Each user always reads from the same replica. That keeps data from "jumping back in time" (monotonic reads).

In the interview, say it outright: "The replicas are asynchronous, so there can be lag. For read-your-writes, I read the owner's data from the leader or wait until the replica catches up to the LSN of their write." That instantly shows maturity.

Leader failure: failover

The leader is down. There's nowhere to write. Failover happens in three steps:

1. Detect. The leader has stopped answering heartbeats for longer than a timeout, usually 10–30 seconds.
2. Pick a new leader. Take the replica with the most up-to-date log. The decision is made by consensus: Patroni + etcd, an orchestrator, or cloud Multi-AZ.
3. Switch the clients over. Update DNS or a VIP, or reconfigure the proxy (PgBouncer, HAProxy). The remaining replicas start following the new leader.

What can go wrong — and this is worth naming to the interviewer:

• Lost writes. With asynchronous replication, the old leader's last transactions may not have reached the replicas. They'll be gone.
• Split-brain. The old leader "comes back to life" and also accepts writes. Now there are two leaders, and the data diverges. The protection is fencing: the old node is forbidden from writing, for example via STONITH or an epoch number.
• A bad timeout. Too short, and you get false failovers during network hiccups. Too long, and you get extended downtime.

Multi-leader, briefly

You can allow writes to several leaders, usually one per data center. A user in Europe writes to the European leader, a user in Asia to the Asian one, and the leaders exchange changes.

• Pros: low write latency across regions; survives the loss of an entire data center.
• Cons: write conflicts. Two people change the same field at the same moment in different data centers. These are resolved by last-write-wins (at the risk of losing data), with CRDTs, or with application logic.

Bring up multi-leader only when the problem involves multiple regions and low-latency writes matter. Otherwise it adds more complexity than value.

What it sounds like in an interview

"The load is mostly reads, so I'll run PostgreSQL in a leader–follower setup: writes go to the leader, and reads are spread across 4–5 replicas. Replication is semi-synchronous: one synchronous replica protects against data loss, the rest are asynchronous. Because of lag, I read the owner's profile from the leader. If the leader dies, Patroni fails over to the synchronous replica, and the old node gets fenced off. If we hit a wall on writes, that's a sharding conversation."

Common mistakes

• Saying that replicas speed up writes. They don't.
• Forgetting about replication lag and quietly reading from replicas right after a write.
• Promising "zero data loss" with asynchronous replication.
• Describing failover without mentioning split-brain.
• Proposing multi-leader "just in case" without naming the conflict problem.

Get these five points down, and you'll explain replication more confidently than half the candidates out there.