Leader and follower replication

01One writer orders a replicated history

Two shoppers try to reserve inventory. A three-machine group A, B, and C sends changes through its current leader A, which puts proposed commands in an order. The followers copy that history. The arrangement is leader–follower replication for this group’s records, such as one shard.

Follow what a saved command means before assuming a query can see it:

1. A node appends a command to its log: an ordered list of proposed changes saved at that node.
2. The replication protocol decides when the command is committed, meaning it is accepted into the group’s history under the protocol’s safety rules.
3. A node applies the command by changing the records its ordinary queries read.

These steps can occur at different times. A follower can have a command saved in its log while its queries still see earlier records. A committed command also need not have been applied by every follower.

The leader coordinates order, but can fail or become isolated. A consensus protocol is the complete set of rules by which the group agrees on accepted history and changes leaders safely. Raft is one example. Calling a server “primary” does not automatically give it those rules or specify its acknowledgment settings.

02Trace a command and a freshness-sensitive read

Suppose a client wants to reserve seat S7 under reservation R12. These names identify the particular seat and reservation in this trace. The proposed command receives log position 22.

1. The client sends its stable request identifier to leader A. A retry of this same intended reservation retains that identifier.
2. A checks the request and appends the command at position 22.
3. A sends log entries to followers. The group follows the protocol’s required saved-progress and commitment rules.
4. Once the protocol permits commitment, apply the command by first checking the group’s saved request results. If the same request already completed, reuse its saved result without claiming another seat. Otherwise apply the inventory change and save its request identifier and result together as one state-machine step. Reject an identifier reused for a different intended request.
5. Return the saved result only after the required commitment and application have completed. A lost response can then be recovered through the same request identifier.

Now a read asks follower C for state at least as new as position 22. C has saved the log through 22 but applied only through 21:

1. Check C’s applied position, not just its saved-log position.
2. Wait until it applies the required state, use an eligible source that has applied it, or return the documented failure.
3. Do not successfully return state 21 for a request explicitly requiring 22.

A current-value read needs another check: is its source still allowed to act as leader? A can remain running while isolated from B and C, which may establish a replacement through the valid protocol. A’s local leader flag is no longer sufficient.

In a quorum-based strong-read procedure, the leader asks the required followers to confirm its leadership, then waits until its own records include the committed position chosen for that read. The complete protocol supplies the voting, history and read-position rules; ordinary health replies do not. Another supported approach uses a lease: permission that expires at a stated time under specific clock and delay assumptions. If confirming authority or applying commands through that read position cannot finish before the caller’s deadline, redirect to an eligible source or return unavailable instead of waiting indefinitely.

03Change leaders without inventing history

A safe replacement must preserve accepted history as well as establish permission to lead. In Raft, an increasing term number identifies a newer leadership attempt or authority period; it is not wall-clock time. Voting also checks candidate logs under the full protocol so required accepted history is preserved.

Separate the takeover tasks:

1. Suspect the current leader when required contact is missing. Silence starts investigation or an election attempt; it does not prove the old process stopped.
2. Establish a replacement through the implementation’s voting and history-preservation rules. A timeout does not make the candidate’s own uncommitted entries accepted.
3. Make a returning old leader recognize newer authority, stop leading, and reconcile its history through the protocol.
4. Direct clients to the current authority. Recover uncertain requests by looking up their retained identifiers and results, rather than inventing another reservation.

An ordered command still does not atomically control an outside service. For a payment:

1. The group can agree to execute an instruction to charge.
2. The payment provider may charge before its response is lost.
3. A retry must reuse the provider’s supported payment identifier or query its retained outcome.
4. A newly generated payment identity could charge again. Agreement on the instruction did not include the provider inside the database commit.

Measure saved-log progress separately from applied progress, commitment waiting, election frequency, and rejection of obsolete authority. Test a lagging follower, an isolated leader, a lost response, and the old leader’s return. Use a proven consensus implementation; a few majority counts do not reproduce its complete safety rules.