LearnastraTHE DESIGN COURSE

Understand the building blocks / CHAPTER 16

Redundancy and replication

Use extra capacity and data copies to survive stated failures without confusing replicas with backups.

Intermediate1h of deliberate study10 questions

Builds on Characteristics of distributed systems, Data partitioning.

By the end, you can
  • Distinguish spare capacity, replicas, and backups.
  • Compare synchronous and asynchronous acknowledgment.
  • Reason about lag, promotion, and correlated failure.

01Copies, spares, and history solve different failures

A document service saves a new document and keeps copies on other machines. Ask which failure each extra resource is meant to cover before counting servers.

  • An extra application instance supplies spare processing capacity when another instance fails.
  • Replication maintains additional copies of data, potentially allowing reads or takeover when a data-serving machine fails.
  • A backup preserves recoverable earlier state, which can repair mistakes that current copies have already repeated.

These are forms of redundancy: extra resources used to preserve specified behavior after a failure. They are not interchangeable.

For this example, call the saved document d51. The primary accepts changes for this data, then sends them to two followers. A follower keeps a copy and may serve permitted reads or take over later. Neither role guarantees that the follower already has the newest document.

Follow an accidental deletion:

  1. A user or program deletes the document at the primary by mistake.
  2. Replication sends that deletion to the followers as it would any other accepted change.
  3. Every current copy can now lack the document.
  4. Recovering yesterday’s version requires retained history or a suitable backup.

Conversely, yesterday’s backup may lack valid updates from today and may take time to restore. State both the permitted lost work and the restoration time rather than calling either mechanism a complete recovery plan.

02Put the success response at a named boundary

When may the service tell the client that saving d51 succeeded? The success response is an acknowledgment. Its timing determines which copies must have reached which step.

A follower can make progress in stages:

  1. Receive the bytes, possibly only into memory.
  2. Persist them under the configured storage and failure protection.
  3. Apply the change to the data its normal queries read.

Receiving is not the same as safely saving, and safely saving is not necessarily the same as making the change query-visible.

Example policy Required point before success Consequence
Local acknowledgment The primary safely saves the change. Less remote waiting; a not-yet-copied change can be lost with the primary.
Required remote durable copy The primary and the specified remote copy safely save it. Better survival of the covered loss; confirmation may wait or fail if that copy is unavailable.
All required replicas apply Every replica required by the policy makes the change query-visible. Broader immediate visibility, with potentially more waiting.

Waiting for a specified remote step is synchronous replication. Allowing some copying to happen after success is asynchronous replication. Name the actual step; “synchronous” alone does not say all followers have applied the change.

Trace the asynchronous loss window:

  1. At time zero, the primary safely logs d51 and sends success.
  2. Neither follower has received that change yet.
  3. Half a second later, the primary’s only current durable copy is permanently destroyed.
  4. A follower takes over, but cannot reconstruct d51 from the data it has.

Three configured servers did not prevent this loss. The acknowledgment and copy placement must match the promised failure coverage.

03Choose safe placement and failover

Suppose every copy is in the same zone. A zone-wide destruction can remove all of them together. Such a group of resources that can fail together is a failure domain.

Choose placement and recovery deliberately:

  1. Put required copies in locations independent of the failure being covered. Cross-zone copies can cover a zone loss, while adding communication delay and operational work.
  2. Retain suitable history for deletion and shared software mistakes, which can affect copies across zones too.
  3. Select a replacement with the required accepted data and enough capacity to serve the new read-and-write workload.
  4. Establish the replacement’s legitimate permission to write. A missing heartbeat can mean the old primary is unreachable rather than stopped. If it resumes, storage must reject its obsolete write permission. Later chapters explain that check in detail.
  5. Redirect clients and resolve uncertain requests using their original request identifiers and saved outcomes.

Reads need their own rule:

  • If a reader must see a completed write, use a source known to have applied it.
  • Alternatively, wait until the selected follower has applied that version, or return the defined failure.
  • A follower’s received log entry alone does not make an ordinary query show the new document.

Measure copy and apply delay separately, write waiting, promotion time, and restore results. Test machine loss, zone loss, interrupted takeover, and restoration of old data. A green server count is useful operational information, but the recovery claim is about what callers can still read and safely change after the actual failure.

The acknowledgment boundary controls the loss window

The acknowledgment boundary controls the loss windowThe primary saves the document and returns local success. The remote copy has not received the change yet. The primary’s current copy is destroyed half a second later. The follower can take over only with the data it actually has; it cannot recreate the missing document. Numbered arrows correspond to the connection key below. All connections are directional.Client writes d51Success promisemattersPrimary durablelogd51 at time 0Local successBefore remote copycatches upRemote followerStill missing d51Primary lostAt 0.5 secondsPromoted stateMay lack acknowledgedd5112345The acknowledgment boundary controls the loss windowThe primary saves the document and returns local success. The remote copy has not received the change yet. The primary’s current copy is destroyed half a second later. The follower can take over only with the data it actually has; it cannot recreate the missing document. Numbered arrows correspond to the connection key below. All connections are directional.Client writes d51Success promisemattersPrimary durablelogd51 at time 0Local successBefore remote copycatches upRemote followerStill missing d51Primary lostAt 0.5 secondsPromoted stateMay lack acknowledgedd5112345The acknowledgment boundary controls the loss windowThe primary saves the document and returns local success. The remote copy has not received the change yet. The primary’s current copy is destroyed half a second later. The follower can take over only with the data it actually has; it cannot recreate the missing document. Numbered arrows correspond to the connection key below. All connections are directional.Client writes d51Success promisemattersPrimary durablelogd51 at time 0Local successBefore remote copycatches upRemote followerStill missing d51Primary lostAt 0.5 secondsPromoted stateMay lack acknowledgedd5112345

Follow the numbered arrows

  1. Client writes d51 → Primary durable logcommit locally
  2. Primary durable log → Local successasynchronous policy returns
  3. Primary durable log → Remote followercopy has not arrived yet
  4. Primary lost → Primary durable logdestroys sole current copy
  5. Remote follower → Promoted stateavailable recovery candidate
The primary confirms document d51 before a follower receives it. Destroying the primary during that interval loses the only saved copy. Waiting for a follower’s durable-save acknowledgment before confirmation would close this primary-loss window, at the cost of remote waiting.
Trace the diagram in words
  1. The primary saves the document and returns local success.
  2. The remote copy has not received the change yet.
  3. The primary’s current copy is destroyed half a second later.
  4. The follower can take over only with the data it actually has; it cannot recreate the missing document.

DRAW. CALCULATE. EXPLAIN.

Choose a zero-loss boundary

20 minutes · guided practice

A service promises that destroying one zone will not lose a confirmed document. Its primary is local and its remote follower normally trails by one second.

  1. Show a failure that violates the promise under local acknowledgment.
  2. State what must be safely saved before confirmation.
  3. Explain the cost and what happens when the required remote copy is unreachable.
Your self-check
Compare the worked solution
  1. The primary confirms the document locally. Before the remote follower saves it, the primary’s zone is destroyed. The follower takes over without the confirmed document. A typical one-second delay still contains this loss window.
  2. Before confirming, the application waits for replies establishing that both the document’s metadata and its referenced file version have been durably saved outside the zone that may be lost. A received packet or metadata-only reply is insufficient.
  3. This adds remote waiting. If the required copy cannot be reached, confirmation may wait or fail according to the policy; it cannot honestly claim the same guarantee by skipping the copy.
  4. Destroy the primary zone immediately after success in a recovery test. Verify the complete document can be retrieved, including its bytes rather than only its metadata row.

The checkboxes record your own assessment. The worked solution gives you a way to check the reasoning.

RETRIEVE BEFORE YOU REVEAL

Keep the important ideas.

Say the answer aloud, then turn the card over.

CARD 01Where is the replication promise made?

At the success response, name both the required copies and their progress:

  1. Received means bytes arrived.
  2. Persisted means they were saved under the configured failure protection.
  3. Applied means normal queries can read the change.

These are different milestones.

CARD 02Why are replicas not backups?
  1. Replicas follow accepted changes, including a mistaken deletion.
  2. Backups or retained history preserve an earlier state.
  3. Restoration must be tested against permitted lost work and recovery time.

A current copy is not automatically recoverable history.

CARD 03What does safe failover need?
  1. Select a replacement with the required data and capacity.
  2. Give it legitimate write permission.
  3. Make storage reject the old writer’s obsolete permission.
  4. Redirect clients and recover uncertain requests from saved outcomes.
Practise this topic in the revision deck ↗

CHECK THE REASONING

Can you explain the choice?

Answer every question, then finish for an immediate score and an explanation for each answer. Aim for 80%, then revisit any gaps.

1 / 10The primary acknowledges d51 before followers receive it, then its disk is lost. What is possible?
2 / 10Which remote milestone most directly supports a durable-copy promise?
3 / 10A follower has logged d51 but not applied it. What may a normal local query observe?
4 / 10All copies are in one zone and that zone is destroyed. What did the placement fail to cover?
5 / 10Why retain a backup when replicas exist?
6 / 10A required remote replica is unreachable under a synchronous policy. What can happen?
7 / 10The old primary misses heartbeats but still runs. Which protection matters during takeover?
8 / 10What should a failover capacity test include?
9 / 10What distinguishes replication from extra application workers?
10 / 10What makes a backup claim reviewable?
0 of 10 answered

CLOSE THE LOOP

Explain it without the page.

Summarise the mechanism, one trade-off, and one failure mode. Then mark this chapter complete.

Go to the source