Make the right trade-off / CHAPTER 44
Primary-replica versus peer replication
Explain which replicas may accept writes, how conflicts are resolved, and what happens when the current writer fails.
Builds on Strong versus eventual consistency, Data partitioning.
- Explain primary ownership and acknowledgment policies.
- Identify peer-write conflict and repair requirements.
- Design failover without split-brain writes.
01Where can a write be accepted?
Order records are copied onto several machines so one machine's failure need not remove the only copy. Decide which machine may accept a change and what happens if two places receive changes together.
- Primary-replica: one current primary orders writes for a defined set of records and sends changes to followers. Different data partitions can have different primaries; this does not imply one global writer for the entire database.
- Peer, multi-leader or leaderless approaches: more than one node may accept or coordinate requests for the relevant data. The exact protocol decides how replicas acknowledge and resolve them.
Multiple entry points can reduce geographic distance or preserve selected operations during failures. They do not automatically make conflicting stock or balance changes safe. Every design still needs a write-order or conflict rule, a definition of success and a recovery plan.
02Acknowledgment determines exposure to failure
Place order 42 on primary P with followers A and B. These machines hold one shard: a subset of records, such as orders for a range of customer IDs.
Trace replica progress
- Receive: a follower gets update bytes over the network.
- Persist: it saves them through the failures covered by its storage guarantee.
- Apply: it changes the state queries can read.
A write response may wait for different steps. “Synchronous” alone does not prove every follower already answers current queries. Asynchronous success can precede followers receiving the new write.
Now explain blocking an old primary, called fencing
- The coordination system grants
Pwrite permission number 12. Requests to the replicas or storage carryingP's writes include that number. - After the coordination system selects one replacement and recovers the committed orders, it grants that primary number 13. The replicas or storage that accept writes record 13 before the replacement starts changing records.
- Every write-accepting participant checks the permission number together with accepting the change, as one indivisible operation. Once 13 is current, a request carrying 12 fails that check and cannot change the record. Checking 12 earlier and later performing an unchecked write would leave another race; changing a hostname cannot enforce this either.
- For example, require two of the three replicas to save every accepted write and to adopt the replacement’s permission. After
AandBadopt 13, isolatedPwith 12 cannot obtain the two acceptances needed to report success, even if it changes its own local bytes.
These checks enforce a winner; they do not choose one or determine which writes committed. The coordination system needs a proven election and commit protocol that selects one replacement containing the committed history, such as Raft. A timeout, a larger number or two arbitrary responses alone does not provide that protocol.
Peer writes: multiple entry points still coordinate replicas
Follow the numbered arrows
- Client X → Replica Asubmit X’s record version
- Client Y → Replica Csubmit Y’s competing record version
- Replica A → Replica Bask B to save and acknowledge the update
- Replica A → Replica Cask C to save and acknowledge the update
- Replica C → Replica Bask B to save Y’s update
Trace the diagram in words
- Client X sends its update to A, while client Y can send a competing update to C.
- The coordinating node requests the configured acknowledgments from replicas before success.
- For separate comments with unique IDs, merging can retain both. Two conflicting claims to the same last seat cannot both be promised first and repaired into one valid original sale later.
03Worked example: a three-replica register
First follow an asynchronously copied write
Psaves order version 8 locally and tells the client it succeeded.AandBstill hold 7.P's only saved copy is permanently lost before replication.- Promoting
Arecovers only what survives onA; it cannot invent version 8's missing bytes. - For a one-machine-loss promise, a concrete protocol can wait until
Pand another machineAhave durably stored the committed version 8 before replying. IfPfails,Astill has 8. A replacement cannot simply useB's old 7; recover the required committed state from a surviving copy under the chosen commit/promotion protocol. - Enforce the new writer permission before accepting conflicting old-primary writes.
Read after saving version 8
- The caller carries the required version 8. A selected follower reports applied/query-visible version 7, so it cannot answer that request yet.
- Wait for that follower to apply 8, route to the valid current primary, or fail under the stated policy.
- This minimum-version check provides read-your-writes, not global linearizability alone. For example, a Raft-style primary first commits an entry for its leadership period. For a read, it obtains confirmation from a majority that it is still leader, then waits until its query state has applied through the committed position established for that read. Without those confirmations or applied progress it waits or fails; a stale health check is insufficient. This is a database protocol, not an extra client-side version comparison.
Now examine one value on a fixed replica set A, B and C without one fixed write-entry primary
N=3 means three copies exist.W=2 means a successful write waits for two acknowledgments.R=2 means a read consults two replicas.
A write saved on A/B and a read consulting B/C overlap at B. Since R + W > N, any such completed participant sets intersect. With R=1, a read choosing only C can miss the write.
Intersection alone does not prove linearizability. The read must choose or reconcile versions correctly, and the protocol must handle concurrent writes, failed attempts, changed replica membership and repairs. Timestamp-based last-write-wins can converge while discarding a valid change. Also keep historical backups: live replication can copy an accidental deletion to every current replica.
Primary-replica versus peer replication — decision flow
Follow the numbered arrows
- Order client → Current primaryrequest a write at the current primary
- Current primary → Replica Asend saved update records
- Current primary → Replica Bsend saved update records
- Choose a replacement → Current primaryend the old primary’s write permission
- Choose a replacement → Replica Achoose a replica with required saved writes
Trace the diagram in words
- For the one-machine-loss promise, wait for P and A to durably save version 8 before telling the client success.
- Followers receive, save and apply the update as separate progress steps.
- A replacement must recover committed version 8; write-accepting participants then record permission 13 and reject P’s former permission 12.
04Pick authority before drawing copies
Choose the write behavior before drawing copies:
- One current primary per shard can simplify enforcing record rules in one ordered path.
- Peer write entry can fit operations that still make sense when accepted separately, such as different users adding distinct comments. Two disconnected nodes cannot independently promise the same last seat merely because they can later choose one record.
- Acknowledgment must name received, durable or applied progress, and independent copy placement.
- Failover must select sufficiently current state and block the former writer at the write-accepting participants.
- Reads must establish the freshness required by that operation. A live follower can still be too old.
- Unavailable participants require an explicit wait/reject or weaker-operation policy. Waiting for a remote durable copy can add delay or reduce availability during its outage.
Trace a process crash separately from a still-running but network-isolated primary. Those situations show why copying data, finding a responsive server and granting write permission are different responsibilities.
DRAW. CALCULATE. EXPLAIN.
Specify one-machine-loss recovery
20 minutes · guided practice
An order store has P, A, B and must not lose acknowledged orders after one machine loss. P can become network-isolated without stopping. Define acknowledgment, successor eligibility, old-writer rejection and read routing.
Compare the worked solution
- Choose an acknowledgment protocol requiring independent durable copies sufficient to survive the stated one-machine loss. Name those copies and failure assumptions.
- Select a successor whose saved state includes the required committed orders; the first responding replica is not automatically eligible.
- Record the new writer permission number at participants that accept writes. They reject the former primary's older number, including after it reconnects.
- For a request requiring version 8, inspect the selected follower’s applied version. If it is 7, wait, route to the valid current primary or fail. For a Raft-style linearizable read, also require majority leader confirmation and query state applied through its established read position. A minimum-version token alone only guarantees that this session sees its own saved version.
- Wait or reject when required acknowledgments cannot be obtained, and keep tested backups for accidental deletion or corruption.
The checkboxes record your own assessment. The worked solution gives you a way to check the reasoning.
RETRIEVE BEFORE YOU REVEAL
Keep the important ideas.
Say the answer aloud, then turn the card over.
CARD 01Where do primary and peer writes enter?
- A primary orders writes for a defined set of records.
- Peer designs can accept or coordinate at several nodes.
- Both need explicit acknowledgment, conflicting-write and recovery rules.
CARD 02What are replica progress stages?
- Receive update bytes.
- Save them durably.
- Apply them to query results.
State which step success waits for and what a replacement must contain; these stages are not interchangeable.
CARD 03Why is quorum overlap insufficient alone?
- With fixed N copies, R + W > N makes read/write participant sets overlap.
- The protocol must still choose versions and handle concurrent or failed writes, changed membership and repair.
CHECK THE REASONING
Can you explain the choice?
Answer every question, then finish for an immediate score and an explanation for each answer. Aim for 80%, then revisit any gaps.
CLOSE THE LOOP
Explain it without the page.
Summarise the mechanism, one trade-off, and one failure mode. Then mark this chapter complete.