Horizontal versus vertical scaling

01Larger machines or more machines

A photo API runs on one worker with two CPU cores and 8 GB memory. Tests show that it safely serves 200 requests/s and CPU is the limiting resource.

There are two different changes:

1. Vertical scaling: replace that worker with one having eight cores and 32 GB. The worker count stays one; each worker has more resources. Scaling down gives each worker fewer resources.
2. Horizontal scaling: run several workers and distribute requests among them. Scaling in removes workers after arranging what happens to their requests and sessions.

Neither change guarantees proportional throughput. Extra cores cannot bypass an exclusive lock that lets only one request proceed at a time. Extra API workers cannot make a saturated shared database accept more work.

Capacity and failure protection are separate. One large worker may handle every healthy request, but its crash removes the entire worker. Several workers need enough capacity left after a crash or while one is being replaced during a deployment.

02Plan routing, state and replacement

For a larger worker, follow the replacement

1. Measure the limiting resource. More memory helps when frequently used data does not fit; it does not fix a one-at-a-time lock.
2. Prepare the change. Some platforms resize in place; others restart or replace the machine. Stop assigning requests to the old worker. Let accepted requests finish; if a response is lost, retry using the same request key and its saved result so a completed purchase is not repeated.
3. Restore access to saved data. Attach persistent storage or reconnect to shared services, check the new worker, then switch traffic. Keep a way to return to the previous setup.

For more workers, follow one shopper

1. A load balancer chooses a healthy worker for each request. It needs enough surviving workers, not merely enough configured instances.
2. The shopper's cart must be recoverable. If the cart exists only in worker A's memory, routing the next request to B gives B no cart to read. Save the cart in a shared store, carry a small signed login token while storing the changing cart separately, or transfer it before removing A.
3. Handle sessions deliberately. Sending a shopper back to A is sticky routing. It improves locality but cannot recover A's memory after a crash. Tokens need size, privacy and revocation rules too.

03Worked example: survive one failed worker

Peak demand is 900 requests/s. Each ordinary worker safely handles 200/s.

1. Five healthy workers provide 1,000/s. After one fails, four provide only 800/s, below demand.
2. Six workers survive one loss. Five survivors provide 1,000/s, assuming reasonably balanced requests and sufficient database capacity.
3. Test a vertical alternative separately. If each larger worker is measured at 1,000/s, two such workers can survive one loss and still handle 900/s. One larger worker cannot offer that same failure protection.
4. Include deployment time. If one of six ordinary workers is drained for replacement and another crashes, only four remain: 800/s. The six-worker plan does not cover those two simultaneous absences.
5. Reduce the fleet safely when demand falls. At 300/s demand, three ordinary workers leave 400/s after one loss. Stop new routing to a removed worker, finish short requests and save progress or reconnect long sessions elsewhere.

Suppose an ordinary worker costs one unit/hour and a large worker costs four. Six ordinary workers cost six units/hour; two large ones cost eight. These are illustrative prices, not claims about actual hardware. Compare tested capacity, discounts, spare capacity and operating effort before deciding.

04A useful scaling sequence

Application scaling and database scaling solve different problems:

• Read replicas copy records. They can answer reads only where their delay is acceptable; copying does not divide primary write ownership.
• Partitions split records. For example, different customer IDs can go to different database servers, spreading independent writes. Queries touching several partitions need more coordination.
• Popular records can still overload one partition. A document database and a relational database can both encounter this problem. Their labels do not decide whether partitioning is possible.

Choose larger workers when the measured resource gain provides sufficient spare capacity with simpler operation. Choose more workers when requests divide usefully and the extra routing and state handling justify their cost.

Before scaling down or in, check memory needs, traffic bursts, surviving capacity and shared dependencies. In the interview, describe the before-and-after request path, how a cart survives, what one failure removes, and what the change costs.