Distributed systems: scalability, reliability, availability and efficiency

01Distributed system quality attributes: definitions

A distributed system consists of independent computers that coordinate by exchanging messages. One part can fail while others keep running; this is a partial failure. An online shop might run its request-handling application on two machines, keep live copies of orders on several database machines, and send receipts through a background worker. The customer sees one checkout experience, even though these parts can fail or respond at different times.

The qualities below answer different questions about that experience. Scalability asks whether the service can handle a larger workload while maintaining its targets. Reliability is the ability to perform the specified function correctly under stated conditions over a period of time. Availability is the degree to which the service is usable when requested, often measured as successful eligible requests divided by all eligible requests. Efficiency asks how much useful work it gets from its resources. Manageability asks how safely operators can observe, configure, operate, and change it. The related term serviceability focuses on diagnosing and repairing faults.

02Vertical scaling, horizontal scaling and serial bottlenecks

At first, one application server handles 500 checkout requests/s. Vertical scaling replaces it with a larger machine: more CPU, memory, or faster disks. It can be the simplest improvement, but hardware has practical limits and a single machine still fails as one unit.

Horizontal scaling adds machines. Put two application servers behind a load balancer, and either can handle a request if essential state is stored outside the process. This can grow application capacity and tolerate one application failure if the survivor can meet the admitted workload. It does not automatically double database write capacity.

Some work remains serialized: operations must take turns because they update the same protected state. Adding application machines does not remove that ordering requirement. This matters both for the time one checkout takes and for how many checkouts can update the same inventory record.

For a separate latency calculation, suppose one request spends 80 ms on parallelizable work and 20 ms executing a serialized operation on one inventory key, excluding queue wait. Making the first part four times faster yields 80/4 + 20 = 40 ms, a 2.5× improvement, not 4×. Even infinitely fast application work cannot eliminate the remaining 20 ms. This is the intuition behind a serial bottleneck: improve the part that limits the actual operation.

The workload also matters. Adding nodes can help independent product lookups while thousands of purchases of the same final item still contend on one record. Measure distribution, not just total QPS.

03Availability and error-budget calculations

An error budget is the amount of unsuccessful service allowed by the chosen availability target over a defined measurement window. The target supplies the permitted fraction; the number of requests or the duration of the window turns it into a count or time allowance. Choose that denominator before interpreting an outage.

At 10:00 the only order database stops responding. Automated detection fires at 10:01. An operator finishes failover and verifies writes at 10:07. Checkout was unavailable for seven minutes, not merely the six minutes spent repairing after detection. Monitoring delay is part of the user impact.

For a simple recurring up/down model, availability can be approximated by mean uptime / (mean uptime + mean downtime). Real services have partial and correlated failures, so a single formula is not a substitute for measuring user requests. Faster detection and repair can improve availability even when the underlying failure frequency is unchanged.

Dependencies also affect the result. In a deliberately simplified model, if two required dependencies are independently available 99.9% of the time, the path is available 0.999 × 0.999 = 99.8001% of the time before other failure sources. Redundant alternatives instead help only when at least one is usable and routing can reach it. Shared power, bad configuration and overload make failures correlated, so multiplying advertised service percentages is not a production reliability proof.

04Reliability, durability and failure domains

If order O17 commits but its response is lost, a retry can create O18 and charge again. Save the result under a stable request ID so a retry returns O17. Save the order and request result in one atomic database transaction: both commit or neither does. An external payment is outside that transaction. Reuse the same payment identifier under the provider’s retry rules, and check an uncertain result before issuing another charge.

Durability is retention of acknowledged data. Replicated records can survive a machine loss if the acknowledgment and recovery protocol make that promise. A backup can restore an earlier state after accidental deletion. Both require verification; merely drawing duplicate cylinders does not prove an acknowledged purchase survives.

A failure domain is a set of components that one event can disable together, such as machines sharing a power supply or deployment zone. A network partition prevents some machines from communicating even though they may still be running. Replica placement must match the failures the service is meant to survive.

The following failure sequence shows which records and identifiers must survive a lost response. (1) Purchase key K17 requests $25 and the order authority records O17. (2) Payment action charge-O17 produces confirmed provider charge C81. (3) The application response is lost. (4) Retrying K17 returns O17/C81 rather than allocating O18 or a new charge identity. If the provider response was lost instead, the charge remains unknown until lookup or the provider’s documented same-key retry resolves it. A timeout establishes uncertainty, not failure.

05Resource efficiency and communication cost

Efficiency is useful outcomes divided by the resources spent. For O17, ten internal RPCs—remote procedure calls—may each transfer a small record. One giant catalog transfer may use fewer messages but far more bytes. Count both messages and data size, then account for network distance and repeated work.

Suppose design A makes ten sequential 5 ms calls and design B makes two 20 ms calls. Their network wait contributions are about 50 and 40 ms respectively in this simplified example. A third design could batch data into one call, but might waste bytes or postpone the response. Message count alone does not identify the best design.

Mixed machine sizes, topology, and uneven load make ideal linear speedup unlikely. Compare designs with the same workload and objective.

06Manageability, monitoring and safe change

An operator should be able to answer what failed, which customers are affected, and which action is safe. Attach one request/trace ID to O17 across services, record state transitions without payment secrets, and measure both successful outcomes and latency. A health endpoint that only says the process is alive does not prove orders can commit.

Use different controls for different problems. Readiness decides whether an instance receives new requests. A restart policy decides when to restart its process. Admission control limits accepted work so existing requests can finish. If a dependency fails, accept less work where necessary; restarting otherwise healthy application processes will not fix that dependency.

Roll out a new version to a small fraction first, compare outcomes, and retain a rollback path. A database change should let old and new application versions coexist during the rollout. Stop assigning new work to a known dead instance, and bound or shed the excess traffic if survivors lack capacity. Separately, avoid ejecting or repeatedly restarting every live instance merely because a shared dependency is slow: that reaction can reduce useful capacity further. Readiness, restart policy and admission control have different jobs.

Candidate explanation: “I separate checkout availability from order correctness. I can temporarily refuse new purchases when I cannot confirm which database node is allowed to update inventory, while keeping browsing available. I add application redundancy, make retries return the original order, and measure the full checkout outcome. My recovery plan includes detection, failover, validation, and enough remaining capacity.”

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.