Characteristics of distributed systems

01Different words promise different things

A document service accepts an upload and later returns its file to the owner. Several different promises must be checked:

• People must be able to complete valid requests. Availability measures successful service under a stated counting rule.
• A confirmed file must survive the covered failures. That is durability, such as retaining it after one machine is destroyed.
• Requests must continue to behave correctly over the assessed period. This is reliability; returning the wrong document quickly is not correct service.
• Required operations may need to continue after a specified fault. That behavior is fault tolerance, such as reads continuing after one server fails.
• Growing demand must remain affordable and meet quality targets. This is scalability.
• The same useful work may use less CPU, memory or money. That improves efficiency.
• Operators must diagnose problems, change settings and restore service safely. This is manageability, also called serviceability.

Use the document to distinguish them:

1. Its bytes can remain saved while the read API is unreachable: durability may hold while availability falls.
2. Doubling request capacity can still leave each response too slow: capacity alone is not the response-time promise.
3. Halving CPU at the same measured throughput and response time improves efficiency; it does not prove a new maximum load or data-loss guarantee.
4. A visible last-completed job and a practiced restore help operators recover; green replica hostnames alone do not show that ability.

A distributed system uses communicating processes or machines. One may work while another fails or a response is lost, called partial failure. Always name the operation and the fault being covered instead of treating all these properties as one promise.

02Make the promise measurable

For document reads, first decide which requests count and what success means. Separate expected client errors, timeouts and partial results before calculating.

For an assumed one million valid reads/month and a 99.9% success target:

1. Count successful valid reads and divide by valid reads. This measured ratio is a service-level indicator (SLI).
2. Require the ratio to be at least 99.9% over the month. That target is a service-level objective (SLO).
3. Calculate the permitted unsuccessful fraction:100% −99.9% =0.1%.
4. Multiply one million by 0.1% to obtain 1,000 unsuccessful reads under this definition.

This request budget is not automatically a number of downtime minutes: busy periods and quiet periods contain different request counts. Also measure response time; correct bytes returned after a minute may still miss the useful read target.

Now choose two recovery limits for a named failure:

• The recovery point objective (RPO) limits missing recent changes. An RPO of 30 seconds permits at most that recent interval of acknowledged changes to be absent after recovery.
• The recovery time objective (RTO) limits elapsed restoration time. For this document example, an RTO of 15 minutes means the service must again authorize downloads and return complete files from its recovered state within that time. A responding health endpoint alone does not meet the target; the separate RPO governs how much recent data may be missing.

If recovery takes 8 minutes but loses 20 seconds of changes, it meets those two example limits. It does not meet a zero-loss promise.

A30-second RPO is a requirement to demonstrate, not evidence that copying always finishes within 30 seconds. Check which bytes and records exist at success and how the proposed restore actually uses them.

03Trace a failure domain and a dependency

Call one confirmed document d8. Suppose its metadata has copies in two zones, but its original bytes exist only in zone A. Follow the failure:

1. The service confirms the complete upload.
2. Zone A is permanently destroyed.
3. Another zone can recover the ownership row, but not the original file.
4. The API may restart quickly, yet downloading d8 fails. A zero-loss document promise was broken.

A zone is an example of a failure domain: resources that can fail together. Two copies sharing the same power source or destroyed zone do not protect against that whole-domain loss. Different zones still share risks such as a bad deletion command or software change; retain suitable history and test restoration.

A separate asynchronous-copy example has the same risk. If a confirmed write reaches only the primary while its remote replica is 20 seconds behind, losing the primary five seconds later may lose that write. Quick promotion cannot reconstruct bytes the replica never received.

Required dependencies affect read success. Under a simplified independence assumption:

1. The application succeeds with probability 0.999.
2. Its required database also succeeds with probability 0.999.
3. Both are needed, giving 0.999 × 0.999 = 0.998001, or 99.8001%.

Real failures may be shared, so that arithmetic is an illustration to test, not production evidence.

Keep optional work optional:

• If recommendations fail, return the permitted document without them when the product allows it.
• If ownership cannot be checked, do not guess permission.
• Measure successful requests, slow response times, resource limits and unfinished-work age together.

Returning a defined useful subset is graceful degradation. It preserves the agreed core behavior instead of treating every dependency as equally optional or equally mandatory.