Heartbeats and failure suspicion

01Silence is an observation, not a death certificate

Worker A builds an export made of several output files. A manifest lists the completed files that readers should use. A coordinator watches A’s progress and may start a replacement if work stops.

1. A sends a contact or progress message every second.
2. The coordinator records when it last received one.
3. After three seconds without contact, the coordinator suspects a problem.
4. It does not yet know whether A crashed, paused, lost its network path, or is waiting behind an overloaded receiver.

That periodic signal is a heartbeat. Its contents define what it tells the receiver:

• An “alive” signal says the signal-sending part of the process responded.
• A readiness signal can report whether required requests can be served.
• A progress signal can report the last completed export chunk or applied log position.

A control thread can keep sending “alive” while the export is stuck. Monitor useful progress as well as contact: how long since the last completed chunk, for example?

Without a known bound on message delivery, silence cannot identify the cause with certainty. Recovery must stay safe even when the old worker is still running and later resumes.

02Trade detection speed against unnecessary takeover

Choose the suspicion threshold from measured delays and the cost of a mistaken takeover.

• A shorter threshold finds some real failures sooner, but treats more ordinary pauses or delays as failures.
• A longer threshold avoids some unnecessary takeovers, but leaves genuine failures waiting longer.

Work the idealized timing separately from total recovery:

1. Heartbeats arrive every second, with no scheduling or network delay in this simplified example.
2. A crash just after contact is suspected roughly three seconds later.
3. A crash just before the next expected contact is suspected roughly two seconds after the crash, because almost one second has already elapsed since the last contact.
4. Real scheduling, network, and receiver delays change those times.
5. After suspicion, selecting an authorized replacement, recovering saved state, preventing old writes, and redirecting work take additional time.

Other detector improvements have limited purposes:

• Adapting a threshold to observed arrival times can reduce unnecessary suspicions under its assumptions.
• Randomizing election timers can reduce repeated simultaneous election attempts.
• Measuring elapsed waiting with an appropriate local timer avoids treating a wall-clock adjustment as elapsed time.

None makes silence proof of death. Test long process pauses and one-way message loss as well as killing a process. Different participants can receive different signals and hold different beliefs while the original worker still acts.

03Prevent a resumed old owner from making stale changes

A dangerous sequence is: A pauses, the coordinator starts B, then A resumes and overwrites B’s manifest. Missing contact did not stop A. Protect publication at the storage operation itself.

Use a saved permission number that increases with legitimate takeovers:

1. A receives permission number 41 from the chosen ownership protocol.
2. After a valid takeover, B receives the newer number 42. A timer by itself must not invent legitimate ownership.
3. B submits its manifest with 42. The protected resource checks the permission number and saves both 42 and the accepted result together.
4. A resumes and submits its old manifest with 41.
5. The resource sees that it has already accepted 42 and rejects 41. A cannot overwrite the newer accepted result just by arriving later.

The increasing permission number is a fencing token. The check and result update must be indivisible at the resource, and the saved number must survive the failures covered by the contract. A client-side check can pause before writing, so it does not provide the same protection. A random request identifier does not order ownership changes.

If the file store cannot perform this check:

1. Have workers write separate output objects instead of overwriting one shared object.
2. Publish the chosen object through a protected database record that names the accepted manifest.
3. Update that record only while checking and saving the permission number together.
4. Readers use the object named by the accepted record. Old bytes may exist, but cannot become the chosen result without accepted publication.
5. Clean abandoned objects only after checking references and the retention policy.

Keep four tasks distinct: detect missing progress, authorize a replacement, reject obsolete writes, and recover unfinished work. Heartbeats mainly help with the first. Test A resuming after B publishes; safety must survive a wrong suspicion.