Fault Tolerance and Observability

Everything breaks — and that's normal

Picture a café. The cook is out sick, the espresso machine is broken, the delivery driver is stuck in traffic. A good café doesn't close because of it: they brew coffee in a French press, the second cook covers the shift, and guests are told honestly, "dessert will be 20 minutes." In a bad café, one failure stops everything.

Systems are the same. Networks drop packets, disks die, a neighboring service hangs. In an interview, nobody expects a system that never fails. They expect you to know in advance how it will fail, and to keep one breakdown from dragging everything else down with it.

The design: two availability zones

An availability zone is a separate data center with its own power and network. The load balancer constantly polls the services (/health) and stops sending traffic wherever they don't respond. The database is replicated to the second zone, and in a disaster the replica becomes the primary (failover).

The math shows why this works (ideally, with independent failures):

Configuration	Availability	Downtime per month
One zone, 99.9%	99.9%	~43 min
Two zones at 99.9% each	1 − 0.001² = 99.9999%	~3 sec

In the interview, add an honest caveat: zone failures aren't fully independent (shared deploys, shared DNS), so the real number is lower. That caveat earns you a point.

Four defensive techniques

1. Timeouts. Without a timeout, a request to a hung service waits forever and holds a thread. A minute later you're out of threads, and your service is "dead" too. The rule is simple: every network call has a timeout, and it's shorter than the timeout of whoever is calling you.

2. Retries with exponential backoff and jitter. Transient failures are cured by retrying, but you have to retry carefully:

• the pauses grow: 100 ms, 200, 400… (exponential backoff);
• randomness (jitter) is added to the pause, or a thousand clients will hit again in the very same millisecond;
• retry only idempotent operations or operations with an idempotency key;
• a cap on attempts is mandatory.

The danger here is a retry storm. If three layers of services each make 3 attempts, the bottom layer gets 3 × 3 × 3 = 27× the load exactly when it's already struggling. The fix: retry in one layer only and introduce a retry budget (for example, no more than 10% of traffic).

client:
  timeout: 300ms
  retries: 3
  backoff: exponential   # 100ms, 200ms, 400ms
  jitter: full           # pause = random(0, backoff)
  circuit_breaker:
    error_threshold: 50%
    open_duration: 30s

3. Circuit breaker. It works like the breaker in your home's electrical panel. If a dependency fails too often, the breaker trips open, and requests to it are rejected immediately instead of waiting for a timeout. After a pause, the breaker lets a couple of trial requests through (half-open). If they succeed, the circuit closes again. This gives the sick service time to recover.

4. Graceful degradation. Serving part of it beats serving nothing. If the recommendations service is down, show "popular" items from the cache. If reviews are down, the product page still opens. Split features ahead of time into critical (payments, login) and nice-to-have (recommendations, like counters). Push heavy, non-urgent work into a queue.

Technique	Protects against	Cost
Timeout	hangs	false failures if set too short
Retry + jitter	transient failures	extra load, requires idempotency
Circuit breaker	cascading failures	some requests are rejected immediately
Degradation	total outage	the user sees a reduced version
Second zone	a data center going down	2× the cost, replication lag

Observability: how to tell what broke

Protection is useless if you can't see the problem. Observability stands on three pillars:

• Metrics — numbers over time, cheap and fast. They answer "is something wrong?".
• Logs — detailed events. They answer "what exactly happened?". Write them structured (JSON) and with a request_id.
• Traces — the path of a single request through every service, with the time spent at each step. They answer "where is it slow?". An end-to-end trace_id is passed in the headers.

Two mnemonics tell you which metrics to collect:

Method	For	What to watch
RED	services (requests)	Rate — requests/s, Errors — error share, Duration — latency (p50/p99)
USE	resources (CPU, disk, connection pool)	Utilization — load, Saturation — wait queue, Errors — errors

Look at p99, not the average. A 50 ms average can hide the 1% of users who wait 3 seconds.

SLIs and SLOs in plain terms

• SLI — what we measure. For example, "the share of requests that succeed in under 300 ms".
• SLO — what target we set. For example, "99.9% over 30 days".
• Error budget — how much you're allowed to "break": 100% − SLO.

An example with 10 million requests a month and a 99.9% SLO:

• an error budget of 0.1% → 10,000 bad requests;
• in time, that's ~43 minutes of downtime.

While there's budget left, the team ships boldly. Once the budget is spent, releases freeze and the team works on reliability. That's how an SLO turns the "speed vs. stability" argument into arithmetic.

How it sounds in an interview

"I'll spread the service across two zones behind a load balancer with health checks; the database is a primary plus a replica in the other zone with automatic failover. Every call has a timeout, retries with exponential backoff and jitter only for idempotent operations, and circuit breakers on external dependencies. If recommendations go down, I'll show a cached top list. Monitoring: RED metrics for services, USE for resources, tracing via trace_id. The SLO is 99.9% of requests succeeding in under 300 ms, with alerts on error budget burn rate."

That's 30 seconds, and it already has every keyword.

Common mistakes

• Retries without jitter or a cap. That's how you DDoS your own system.
• Retrying a non-idempotent operation. Like charging a customer twice.
• Default timeouts. These are often 30–60 s, or even infinite.
• "We'll add a replica and that's it." No mention of replication lag or split-brain during failover.
• Promising a 100% SLO. That target is unreachable and forbids any change at all.
• Alerting on CPU instead of user-facing symptoms. Wake someone up for errors and latency, not for processor load.

You've just assembled the toolkit real production teams use: isolating failures, smart retries, a backup zone and eyes on all of it. Name these techniques and their cost, and the interviewer will see someone who thinks about the system at 3 a.m., not just on the whiteboard.