Vertical and Horizontal Scaling

Intuition: a coffee shop at rush hour

Picture a coffee shop with a line out the door every morning. You can do one of two things:

• Hire a champion barista and buy them a more powerful espresso machine. That's vertical scaling: still one counter, just more powerful.
• Open three more counters with regular baristas. That's horizontal scaling: more identical workers.

The champion has a ceiling: no human gets faster than a certain point, and their salary grows faster than their speed. If they call in sick, the shop is closed. Several counters are cheaper, survive one barista being out sick, and can grow almost without limit. But you need someone at the door sending customers to a free counter, in other words a load balancer.

Vertical vs horizontal

	Vertical (scale up)	Horizontal (scale out)
What you do	more CPU/RAM/SSD for one machine	add more machines
Complexity	almost none, the code doesn't change	you need stateless code and a load balancer
Limit	the biggest machine available	practically unlimited
Cost	grows non-linearly, top-tier hardware is expensive	grows roughly linearly
Fault tolerance	a single point of failure	losing one machine doesn't take the service down
Upgrades	often with downtime	one machine at a time, no downtime

In an interview, don't say vertical scaling is bad. It's a perfectly good first step: simple, fast, no rewrite needed. Databases often live this way for years. You need horizontal scaling when you hit the ceiling or have availability requirements.

Run the numbers

Say one API instance handles ~2,000 RPS, and peak traffic is 10,000 RPS.

• Baseline need: 10,000 / 2,000 = 5 instances
• Keep utilization at 70% or below for headroom: 5 / 0.7 ≈ 8 instances
• Plan for losing one zone (N+1): 9–10 instances

This calculation is worth walking through out loud. The interviewer sees that you think about peaks and failures, not just the average.

Stateless: the key requirement for scaling out

Horizontal scaling only works if any instance can handle any request. That means the service has to be stateless: it keeps nothing in its own memory that the next request will need.

The classic trap is the user session. If it lives in the memory of API #1 and the next request lands on API #2, the user gets "logged out." Your options:

Approach	How it works	Trade-off
Sticky sessions	the load balancer pins a client to one server	uneven load; sessions are lost if that server dies
Shared store (Redis)	the session lives in Redis under a session_id, and any API can read it	+1 network hop; Redis has to be replicated
Client-side token (JWT)	the state is signed and carried in the token itself	hard to revoke early; the token grows

The default interview answer: "Services are stateless; I keep sessions in Redis with a TTL, or use a short-lived JWT plus a refresh token." Mention sticky sessions as a crutch for legacy systems.

Diagram

Notice that the APIs multiplied, but there's still one database. Sooner or later it becomes the bottleneck. You scale it vertically, with read replicas, and with sharding, but that's a topic for upcoming lessons. In an interview, say it plainly: "The next bottleneck is the DB."

Load balancer

L4 or L7

• L4 (transport layer) sees only IP and port and never looks inside the traffic. It's very fast and cheap and works for any TCP/UDP.
• L7 (application layer) reads HTTP: path, headers, cookies. It can route /api/* and /static/* to different services, terminate TLS, and do cookie-based sticky sessions. It's slower and burns more CPU.

A common combination: L4 at the edge for raw throughput, with L7 behind it for smart routing.

Algorithms

Algorithm	Idea	When it shines
Round robin	take turns: 1, 2, 3, 1, 2, 3…	similar requests, identical servers
Weighted RR	more powerful servers get more	mixed hardware
Least connections	send to whoever has the fewest active connections	requests of varying length, WebSockets
Consistent hashing	the server is picked by hashing a key (user_id)	you need a local cache or key affinity; adding a server moves only ~1/N of keys

Here's how round robin spreads six requests across three servers:

Health checks

The load balancer regularly polls each instance, for example GET /health every 5 seconds. After several failed checks in a row, it takes the instance out of rotation, and after several successful ones it brings it back.

• Liveness checks whether the process is alive.
• Readiness checks whether the instance is ready to take traffic: it's warmed up and has a DB connection.

Don't make health checks "deep" all the way down. If /health checks the DB, a one-second DB hiccup makes the load balancer drop every instance at once.

The load balancer itself shouldn't be a single point of failure either. You run a pair in active-passive mode with a floating IP, or use a managed cloud LB.

How it sounds in an interview

"I'll start with one server and add resources to it as needed; that's the simplest option. For growth and fault tolerance, I make the API stateless and move sessions to Redis. In front of the instances I put an L7 load balancer with least connections and health checks, and the LB itself is redundant. With a 10k RPS peak and 2k per instance, that's about 9–10 machines with headroom. After that, the database becomes the bottleneck."

Common mistakes

• Jumping straight to "hundreds of microservices" without mentioning the simpler vertical option.
• Scaling the API horizontally but leaving sessions in process memory.
• Forgetting that the load balancer and the database can also be points of failure.
• Sizing for average load instead of peak, and not planning for N+1.
• Naming an algorithm with no reason. Always add why you picked it.

You've just worked through the foundation of the whole scaling chapter. Next, we'll build on it.