Load balancing

01One front door, several workers

A shopper requests the product catalog. Four application servers, A through D, can all answer the request. The client uses one public address; a load balancer receives the request and chooses a server behind that address. These application servers are often called backends.

Work through capacity before choosing the routing rule:

1. Tests show that each server handles 350 representative requests per second within the response-time target.
2. At a peak of 900 requests per second, four servers receive about 900 / 4 = 225 each under even routing.
3. If D is removed, three servers receive about 900 / 3 = 300 each.
4. Compare 300 with the tested limit of 350. The survivors have some margin, provided the request mix and routing remain similar.

The balancer makes these servers usable as a pool. It cannot make a slow shared database faster or decide which shopper may claim the last item.

Two routing approaches inspect different information:

• A transport-layer balancer chooses a destination using connection information, such as addresses and ports. It need not read the HTTP request.
• An application-layer balancer can read HTTP information, such as the website name or /catalog path, and route accordingly.
• If the balancer ends the client’s encrypted connection so it can read HTTP, that is TLS termination. TLS protects that connection; the separate connection to the application needs its own protection.

02Follow health and safe replacement

D starts successfully, but is still loading the catalog data needed to answer requests. Its process exists, yet sending shoppers to it would cause errors. Separate two checks:

• A liveness check asks whether restarting this process is likely to help.
• A readiness check asks whether this instance can handle new requests now.

For a planned replacement, the operator and routing layer follow this sequence:

1. Mark D unready so the routing layer stops selecting it for new work.
2. Allow time for that change to reach the routing layer.
3. Let requests already running on D finish, up to a stated deadline. This is draining.
4. Close remaining long-lived connections according to the reconnect policy, then stop D.
5. Start the replacement. Route new requests to it only after its required data and services are ready.

A WebSocket connection cannot simply move intact to another server. The client must reconnect, and the application needs saved state from which it can resume. A session that exists only in D’s memory may disappear with D.

Health checks also have limits:

• An active check periodically probes a server. The interval creates a delay before a failure is noticed.
• Passive observations use real responses, errors, and timeouts. A healthy flag should not override repeated failed requests.
• An optional recommendation service can fail while the catalog remains useful. Marking every catalog instance unready for that optional failure can unnecessarily remove the whole pool.

Some traffic may still reach a failed instance before the routing layer reacts. Safe deadlines and retry rules remain necessary.

03State, overload, and shared limits

Suppose one shopper makes many requests. Always routing that shopper to the same server may reuse information already loaded there, but can overload that server while others are idle. This routing preference is session affinity. It does not copy session data or preserve it after a crash.

Check the complete request path:

1. Measure how much work each application server is doing, rather than checking request totals alone.
2. Check the shared database. If all servers wait for the same slow query, adding application servers can increase the database’s waiting queue.
3. Set maximum counts for running requests and queued requests. When the queue limit is reached, reject new excess work rather than keep adding it. Remove requests that are still queued when their caller’s deadline expires; do not start work after that deadline.
4. Carry the caller’s time budget through the path. From a 300 ms allowance, 100 ms already spent leaves at most 200 ms for remaining work and returning the response.
5. Retry only if time remains and the operation can safely repeat. A purchase may have committed even if its response disappeared. Repeated reads also add load to an overloaded service.

The balancer is part of the required path too. One unprotected balancer process can make every healthy application server unreachable if it fails.

Test specific disruptions:

• Remove one application server at peak demand and measure the survivors’ response times.
• Replace a server while requests and long-lived connections are active.
• Start a replacement with an empty cache or unloaded data.
• Fail part of the balancer’s serving path.
• Slow the shared database and verify that waiting remains bounded.

Successful balancing means callers receive useful responses within their limits, including during the named failure. Equal request counts are only one possible means to that outcome.