Multi-region architecture and disaster recovery

01Multi-region architecture, high availability, and disaster recovery

Multi-region architecture runs a service across geographically separate deployment regions. Disaster recovery is the planned restoration of usable service and data after a major disruption. A region is a geographical deployment area whose infrastructure can share risks such as a regional network failure; an availability zone is a separate failure domain within a region under the provider’s isolation model. Putting servers in two locations does not provide regional recovery if both still depend on the same regional database, credential service, or network.

High availability keeps the service operating through expected component failures. Disaster recovery restores a useful service after a larger disruption. Backups preserve earlier recoverable states. These capabilities overlap, but a replica that immediately copies an accidental deletion is not a substitute for a backup that can restore yesterday's data.

Specify allowed data loss and recovery time before choosing a regional topology. An East-primary/West-asynchronous-replica example illustrates the tradeoff: an acknowledged order O17 can be absent from West when East fails. Whether that loss is acceptable, and whether writes may pause during recovery, determines the required coordination and cost.

02Recovery point objective (RPO) and recovery time objective (RTO)

The recovery point objective, RPO, is the target maximum amount of data loss measured as a time window. An RPO of 30 seconds means the recovery plan targets a recoverable state no more than 30 seconds behind the disruption. The recovery time objective, RTO, is the target time to restore the agreed service after disruption. Neither is a guarantee merely because it appears in a diagram.

03Active-passive, active-active, and write ownership

A regional topology defines where the service runs and which regions may serve each operation. Compare write ownership separately from replication timing: a region may serve reads while another owns writes, and a write may wait for remote durability before success. These choices determine both normal latency and what remains possible after a region is lost.

Start with a primary region and a standby. East owns writes. West receives the ordered change stream. Reads may use West only under a stated staleness policy. This is easier to reason about than allowing both regions to update the same inventory row independently.

Active-active means more than two copies of a web server. If both regions accept writes, specify ownership or conflict handling. Assigning each tenant a home region gives one authority per tenant. Globally coordinating a row can preserve stricter guarantees but adds cross-region latency. Accepting concurrent updates and merging them requires business-compatible semantics; “last timestamp wins” can silently erase an order or inventory reservation.

Read replicas, immutable assets, and regional caches can reduce geographic read latency without making all writes multi-primary. Choose the narrowest distributed-write requirement the product actually needs.

A different design puts one voting, data-bearing replica in each of three regions and commits through a proven majority protocol. Every acknowledged write is durable in two regions. After any one region is lost, the two survivors can elect according to the protocol and recover the committed history; a lagging survivor cannot simply ignore the protocol's election restrictions. This is a constructed quorum example, not a claim that every three-region product uses this layout. It costs cross-region commit latency and still depends on surviving network and service capacity.

04Regional failover: detection, fencing, promotion, and routing

Assume East acknowledged O16 at 12:00:00 and West durably applied it. East acknowledged O17 at 12:00:04, but its log entry has not reached West. Connectivity fails at 12:00:05.

1. At 12:00:10 monitoring detects failure. It cannot infer whether East is dead or merely unreachable from West.
2. A promotion procedure establishes that the old writer cannot continue accepted writes under the ownership protocol. A fencing epoch is an increasing ownership-generation number. Resources that check the current epoch can reject an old writer’s operations; changing a number without an enforcing resource does not stop the old process.
3. West is promoted from its last safe durable position. In this example O17 may be absent, despite its prior acknowledgement. The observed loss window is five seconds; the missing record was accepted one second before disruption.
4. Routing moves eligible traffic. DNS caches, connection pools, and clients may keep using old endpoints, so routing changes alone do not fence the old writer.
5. The team validates order creation and payment reconciliation before declaring recovery complete. If that happens at 12:07:05, service recovery took seven minutes.

The client retries O17 using its original operation identity. A payment might have succeeded outside the lost database state. The recovery path queries the payment attempt or reconciles provider events rather than charging blindly. The write and payment contracts must survive the disaster plan together.

The payment recovery identity must also survive. Store the original client operation ID and provider attempt/resource reference in recoverable state, or ensure the provider can recover the mapping from a durable business reference. If both the mapping and the acknowledged order are lost, the client retry alone does not prove whether a charge exists. Hold new charging attempts while reconciliation reconstructs that fact.

05Standby capacity and restore-time estimates

A warm standby has some running resources and scales up during recovery. A hot standby keeps more capacity ready. Backup-and-restore starts from stored snapshots/logs and generally has more work on the recovery path. These are cost and recovery-time choices, not universal time guarantees.

Suppose peak traffic is 10,000 requests/s and West is provisioned for 2,000. Promotion without a capacity plan creates a second outage. Reserve or validate capacity, warm critical caches carefully, and use admission control while recovering. Include database connections, queue throughput, key management, identity providers, configuration, and secrets distribution in the dependency inventory.

For backup transfer alone, restoring 6 TB over a sustained 1 GB/s path takes approximately 6,000 seconds, or 100 minutes, before replay, indexing, startup, and validation. That cannot support a ten-minute RTO without another recovery mechanism. Use measured restore throughput, not a network-interface headline rate.

06Backups, point-in-time recovery, and restore validation

Point-in-time recovery restores a backup and replays retained changes only up to a selected moment. Choosing a point before a destructive update can recover data that live replicas have already deleted. The backup, required log history and decryption keys must all be available for that selected point.

Replication can faithfully copy corruption, deletion, or an application bug. Preserve point-in-time recovery logs and backups under access and retention policies that reduce correlated loss. Test restoration into an isolated environment, validate application-level invariants, and measure the entire process.

Retention has a business and security cost. Keep enough history to detect and recover from plausible mistakes while applying deletion and regulatory obligations deliberately. A disaster-recovery copy remains sensitive production data.

07Failback and disaster-recovery exercises

When East returns, it may have different data from West. Keep West in charge of new writes. Rebuild or reconcile East from West, verify replication, then plan the transfer back. Choose a clear switch point and prevent the former writer from continuing afterward. Old clients and running jobs must be rejected if they use an obsolete ownership version.

Run exercises that fail a database, sever regional connectivity, remove a dependency, and restore a backup. Record detection time, last recoverable write, promotion time, routing convergence, and usable capacity. The interview answer becomes credible when it identifies which promise the exercise validates and what would prevent declaring success.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.