Design a payment system and ledger

01Problem and scope

A merchant payment platform coordinates authorization, capture, status and refunds through an external processor while maintaining an internal accounting record. Each payment operation must have one logical effect, each journal must balance within its currency, and refunds must stay within captured funds. A timeout is an unknown external outcome. A USD 25.00 capture is the example. Merchant checkout, stored-value transfers and processor accounting are distinct products and should not be conflated.

Candidate: “I will design merchant checkout through one external processor, plus the internal payment history and ledger. Does a timeout permit us to show pending while we find the result?” Interviewer: “Yes, but a retry must not charge the customer again, and merchants need partial refunds.” The service can show pending while it recovers the result instead of guessing whether the charge succeeded.

A payment intent records the workflow: what the customer wants and which steps have completed. A ledger records financial movements. In double-entry bookkeeping, each journal has debit and credit entries whose totals balance within its currency. Workflow state may change as new evidence arrives; a posted journal is corrected with a new journal, not erased.

We support authorization, one full capture per payment, status and full or partial refunds. Partial or incremental capture is a separate extension requiring its own reserved-capacity model. Authorization reserves spending capacity under the provider’s contract; capture requests the financial movement. We exclude lending, foreign-exchange conversion and a complete dispute platform. The account names below form a simplified platform example, not a claim about a particular company’s accounting system. Every volume, latency and retention value is an explicit interview assumption.

02Functional requirements

1. Create a payment intent. A merchant creates a payment intent for order o81 and a server-validated price of USD 25.00 (2500 minor units). A retry with the same identity returns the same pay81, even if it reaches another application instance.
2. Collect provider authorization. The customer supplies a processor-issued payment token and completes any required customer action through the provider-supported flow. Our service never stores raw card numbers in its own intent rows.
3. Authorize and capture. The service requests authorization and capture, separately when required. The merchant sees captured only after authoritative processor evidence has been durably applied locally. Accepted-for-processing is a distinct state.
4. Refund captured funds. An authorized merchant agent requests a full or partial refund. The total of successful refunds plus unresolved reserved refunds cannot exceed the captured refundable amount.
5. Read status and history. The customer and the merchant retrieve status and paginated history. An unknown processor outcome appears as processing or reconciliation-required, with a stable resource to check.
6. Reconcile financial facts. Operators compare processor and settlement facts with internal journals and resolve discrepancies through auditable actions.

Acceptance boundaries

A browser redirect or a client message saying “payment succeeded” is not authoritative evidence. Likewise, sending an email is not part of the financial commit: it follows a durable outcome event. A customer action, a processor result and a local database commit prove different things; the API reports each separately.

03Non-functional requirements

1. Intent latency. Target p95 below 200 ms, excluding customer/provider steps; reject before acceptance when durable capacity is unavailable.
2. Status latency and availability. Target p95 below 150 ms and 99.95% eligible-request availability. Return known pending state rather than fabricated success.
3. Capture completion. Target p95 within 3 seconds when the provider is healthy. External uncertainty can remain pending beyond this objective.
4. Durability. Acknowledged intents and postings survive one zone loss through durable quorum in the selected regional design.
5. Journal retention. Use seven-year illustrative retention; confirm the actual business policy before deployment.
6. Isolation and authorization. Enforce customer authorization and merchant isolation on every status, refund and export path.

Workload and money representation

Assume ten million new payment intents/day and a twentyfold peak over the daily average, with merchants across several currencies. Each payment uses one currency and integer minor units; not every currency has two decimal places. The amount comes from a verified order/quote, not an editable browser total.

Financial invariants

A financial invariant is a rule every committed operation must preserve, even if doing so means declining work during an outage. Here the rules protect accounting integrity and prevent the same captured funds from being promised to overlapping refunds.

Regional disaster boundary

A total regional loss is separate from one-zone durability. Zero regional data loss requires the service to wait for the necessary durable copies outside the region before acknowledging writes, with the corresponding latency and availability costs. Preserve processor references for reconciliation, test restore, and describe the recovery gap.

04Capacity estimates

Assume ten million new payment intents/day and a twentyfold peak over the daily average.

Check the assumptions

A page polling ten times per second breaks the four-reads-per-intent estimate; backoff and push notifications affect capacity. Do not multiply the retained-record estimate again as though journal entries were omitted.

Provision the transaction and recovery paths

Benchmark the real transactional workload and merchant partitioning plan before concluding that one database cannot work. Reserve reconciliation capacity. Dependency limits and financial correctness dominate raw provider bandwidth, and backlog drain requires actual spare completion capacity.

These rates count new logical payments, not every HTTP or provider retry. The four-phase local estimate follows this chapter’s separate authorization and capture APIs. Automatically claiming capture in the authorization-outcome transaction could combine two phases, but that is a different flow and must be stated explicitly.

05APIs and contracts

A payment identifies the purchase workflow; an operation identifies one authorization, capture, cancellation or refund within it. Keeping those identities separate lets a caller retry one step or inspect its unresolved outcome without creating another purchase.

Scope keys to the authenticated merchant and operation type. Store a canonical payload fingerprint; changing amount under an existing key is a conflict, not a new attempt. A 202 never means captured. A provider timeout returns a durable pending resource, while validation or permission errors do not create processor work.

The capture API describes our service’s operation. The adapter maps it to the chosen provider’s documented object/state flow, and stores the provider resource ID as soon as known. Provider idempotency retention is finite and implementation-specific. After its safe retry window, an old local key alone cannot make a repeated remote call safe. Status queries and reconciliation replace blind retries.

The capture command locks the payment and claims its single capture slot in the same transaction as inserting the operation and outbox. A different idempotency key cannot create a second capture for that payment: return the existing capture operation or a conflict. The slot remains claimed while its outcome is unknown. This product permits the validated full amount once; supporting partial captures would require atomically reserving the remaining authorized amount, just as refunds reserve captured capacity.

Authorization, capture and cancellation

These are illustrative service states, not literal names shared by every processor. This exercise uses a payment method that supports separate authorization and capture. The adapter verifies provider evidence before advancing financial state.

Capture and cancellation lock the same payment row, so the database decides which claim comes first. Once capture may be in flight, cancellation cannot report a successful void merely from local intent: reconcile that operation, then cancel a still-unused authorization or refund a confirmed capture as appropriate. Read the authoritative database clock after acquiring the row lock and leave a provider-dependent processing margin before the capture deadline. A valid local check does not stop authorization expiring before the remote call completes; a definitive provider rejection or unknown response still follows the established outcome-recovery path. Do not hard-code a universal seven-day hold: methods and networks have different validity rules. Stripe authorization and capture.

06Data model and access patterns

Payment and operation records track workflow progress. Journals and their entry lines record financial movements; a projection is a derived view, such as a balance total, maintained for convenient reads. The journal remains the accounting source from which that view can be checked or rebuilt.

The inbox stores received provider events durably before applying them. The outbox stores commands or notifications in the same transaction as the local state change that requires them, so a later worker can publish them without losing the handoff.

Initially, partition by merchant so its payment and journal records can commit in one transaction. A directory maps merchant ranges to database shards. Large merchants may eventually need an explicit subledger design; casually hashing individual entry IDs would scatter a balanced journal across independent commits.

Indexes support merchant/payment lookup, (state,nextAttemptAt,operationId) recovery scans and (merchantId,createdAt,paymentId) history. A balance projection is updated with journal posting or reconstructed from entries. It is not allowed to override the journal. Direct entry writes are denied to ordinary services; an authorized posting routine validates currency and totals and commits the entire journal. A row-level CHECK alone cannot enforce an arbitrary multi-row journal sum.

07Basic working design

A useful baseline is a payment API, one transactional database and a small background worker calling one processor. The API creates pay81, its authorization operation and an outbox command in one transaction, then returns accepted. After verified authorization and any customer action, a separate short transaction validates the authorized amount/deadline and claims the capture slot as op81 with its own outbox command. The worker scans pending commands directly; an external broker is unnecessary at this size. The processor lies outside the database transaction.

In this example, processor receivable records money owed to the platform by the processor, and merchant payable records money the platform owes the merchant. Processor settlement transfers funds owed to the platform; paying the merchant is a separate movement that reduces merchant payable. These later movements need their own journals. These account roles explain the two sides of the capture journal below.

The API first replies after the payment intent commits. If the service dies after that commit, another worker can find the pending operation. Authorization and capture use distinct stable identities; an authorization alone creates no capture journal. When the processor later reports ch81 captured for USD 25.00 (2500 minor units), a local transaction records that fact and journal j81. In the simplified platform account model, j81 debits processor receivable by 2500 and credits merchant payable by 2500. The two totals match within USD. Settlement later changes receivable/cash/fee accounts through additional journals.

Status reads use this same database and return its current known state. The single region simplifies authority: there is one place to decide whether op81 was applied. The baseline already has durable request identity because a low-traffic service can still lose a response. It deliberately does not yet have many shards, a cache or multiple providers. We benchmark the real posting transaction before deciding which scaling change solves an observed limit.

08Find the baseline flaws

Suppose the baseline database sustains an assumed 2,500 local transactions/s at the target tail latency. The estimated 9,260 peak phases/s exceed that measured capacity before recovery work is added. Increasing API replicas does not fix the shared posting bottleneck. Status reads at 9,260/s may compete for the same I/O and buffer pool, so history reads that tolerate lag should run separately from decisions that require current balances.

Now examine a correctness counterexample. Worker W1 sends capture op81. The processor commits ch81, but its response disappears. W1 marks the request failed and a replacement worker invents op82. Both captures can succeed. No amount of database replication repairs this duplicated external effect. Preserve op81, record uncertainty, and reconcile the same remote operation instead.

A second race concerns refunds. Agents A and B both read captured=2500 and refunded=0, then each request 2000. If the read and reservation are separate, both may send externally, totaling 4000. A unique refund ID does not help because these are two different valid IDs. Both decisions must lock or atomically compare the same captured-capacity record.

These failures drive different changes: distribute independent merchant workloads for capacity; use stable identities and a protected refund balance for correctness. A queue alone prevents neither duplicate charges nor excessive refunds.

09Improve the design, step by step

First, protect and parallelize provider work. The trigger is the 600,000-item outage backlog and provider concurrency limits. A transactional outbox relay publishes operation IDs to a durable queue; bounded workers claim work and use the already-stored provider key. A separate recovery pool handles old unknown outcomes. The API can quickly save accepted work, while provider workers scale separately. The cost is queue storage, extra handoffs and pending states. Duplicate delivery is the new risk, handled by operation state and stable provider identity. Direct database polling is the rejected alternative only once scans or scheduling fairness become expensive; it remains simpler at modest load.

Second, split independent merchant authorities. The trigger is a measured 2,500-transactions/s shard versus roughly 9,260/s peak workload. Six comparable shards offer 15,000 transactions/s of assumed measured capacity, about 62% utilization before the omitted recovery and refund work. This is an initial sizing candidate; benchmark merchant skew and failure reserve rather than dividing blindly. Merchant routing sends pay81 and its journal to one shard. A synchronous replica set protects each shard against the stated zone failure. Each shard holds a smaller active dataset and posts independently; operators must manage routing, migrations and more database groups. A stale directory is a new risk, so owners validate routing epochs. A larger single database is a reasonable alternative when it meets the target with less operational work. Cross-merchant financial transfers remain outside this partition-local scope until a deliberate transaction design is added.

Third, isolate reads without weakening decisions. Status-history and reporting traffic now competes with posting. Serve explicitly stale-tolerant history from read replicas or a derived reporting view, while immediate pay81 status after submission carries a minimum committed version or goes to its authority. A refund checks current balances in the primary database transaction; it never trusts the reporting copy. The benefit is predictable posting capacity; the cost is replication traffic and read routing. Replica lag is the new risk. Keeping every read authoritative is preferable when the read volume fits or the product requires current answers everywhere.

Fourth, close the evidence gap. Missing callbacks and lost responses motivate a durable webhook inbox and scheduled settlement reconciler. Both feed the same outcome applier, which validates facts and invokes the unique journal-posting transaction. This recovers effects the request path missed. Costs include provider queries, settlement ingestion and explicit unresolved cases. Badly matched imported facts can create incorrect postings, so match processor reference, merchant, amount, currency and operation kind. Relying only on signed webhooks is simpler but cannot independently detect a missed event or accounting mismatch; retain that alternative only when a weaker recovery contract is acceptable.

Each change addresses a measured bottleneck or a specific failure. None makes a remote call part of our SQL transaction. Their success is measured by backlog drain, posting latency and reconciled outcomes, not the number of new components.

10Detailed architecture

Merchant-owned transactions

The public payment API authenticates the merchant, validates the order/quote, and routes by merchant through a versioned shard directory. Each shard owns intents, operations, capture capacity, journals, entries and local inbox/outbox state. Its replicas provide the selected durability protocol; the diagram’s replication edge is not a second independent writer.

External execution and evidence

An outbox relay exports committed commands into an operation queue. Provider workers consume operation IDs and read/claim their durable state at the owner. The queue is a wake-up and scheduling mechanism, not the sole record of what money should move. Workers call the external processor with stable keys, and submit verified outcomes to the posting service. A webhook receiver verifies the provider signature, persists an inbox record, then returns promptly; applying the callback to payment and ledger state happens asynchronously.

One posting authority

Both callback processing and settlement reconciliation use the same merchant database and posting routine. They do not append independent journals into separate databases. A merchant-event relay publishes captured/refunded facts after the journal commit. Order fulfillment consumes these idempotently; email and analytics cannot delay the financial transaction.

Read paths and implementation

History replicas answer only reads whose freshness contract permits them. Immediate status reads and commands that change available funds reach the authoritative database. Region-local synchronous work is therefore short: authorize the request, route, execute one database transaction and respond. Workers handle provider calls, callbacks, reconciliation and merchant notifications later. Saved operation IDs let them resume after a crash.

A practical starting stack is PostgreSQL for the merchant-local transaction, a provider SDK for documented authentication and request semantics, and a bounded worker using the database outbox. Add a broker for measured scheduling or isolation needs. With PostgreSQL synchronous replication, explicitly select the durability policy and synchronous standbys across the intended failure domains; ordinary asynchronous replication does not satisfy the zone-loss acknowledgment claim by itself. Neither the SDK nor a unique SQL key makes a remote processor part of the local transaction.

11Write path and acknowledgement

Local intent acceptance and external money movement are separate commit boundaries. Persist operation identity before the processor call and retain uncertainty until evidence resolves it.

1. The customer submits checkout-81. The API derives merchant M7 from the verified checkout context, validates the order amount and token reference, and routes to M7’s current shard.
2. One transaction inserts the unique create identity, pay81 and the initial outgoing intent. Repeated keys compare the canonical payload and return the saved resource. A commit failure returns no acceptance promise.
3. The authorization operation persists its identity, resolves any required customer action and records verified authorization amount, currency and capture deadline. A capture command then locks pay81, checks that authorization is eligible, unexpired and not being canceled, claims its only capture slot and records op81 with immutable amountMinor=2500, currency=USD and provider attempt key cap-op81. A unique per-payment capture claim rejects a second command even if its idempotency key differs. The outbox insertion shares this commit.
4. A worker claims the operation, checks that it still requires a remote call, and invokes the provider outside database locks. A delivery retry reuses cap-op81; it does not manufacture another payment attempt.
5. A definitive capture result identifies ch81. A timeout changes the local state to unknown and schedules status recovery. The provider may already have succeeded, so the refund capacity is not inferred from an HTTP error.
6. The outcome applier validates merchant, currency, amount and allowed state, locks op81 and posts its unique capture journal together with captured status, balance projection and an order-update outbox event.
7. Only that successful local commit allows a captured response or event. If its acknowledgment is lost, retrying the same outcome finds the journal and returns the stored result. The worker acknowledges its queue item after a recoverable local state has been saved.

A successful remote effect with failed local posting is repaired by replaying the outcome, not by calling capture again. The diagram’s lost response interval is exactly where that distinction matters.

12Read and delivery path

Payment status exposes the authoritative known outcome. Derived history or caches cannot turn an unresolved processor call into a definitive failure.

1. The customer requests pay81 under authenticated checkout ownership. The API verifies access before exposing status; knowing a payment ID is insufficient.
2. Routing uses M7’s current shard. For a status read immediately following submission, use the authoritative owner or a replica proven to have applied at least the acknowledged version. A stale replica must not turn a recorded capture back into “not found.”
3. The service loads the payment and its operation summary. It returns captured only after verified processor evidence has committed locally, otherwise a precise pending, action-required, failed or reconciliation state. These values describe known facts, not an attempt to guess the provider’s current state from elapsed time.
4. A refund screen may display an explanatory total, but the subsequent refund command rechecks capacity atomically. A displayed available amount is never a reservation.
5. Merchant history uses (createdAt,paymentId) keyset pagination, capped at fifty records. Stale-tolerant history can use a read replica and report its freshness. A cursor is a position, not a fixed snapshot unless an explicit versioned export contract says so.

Cache static payment-method metadata if useful, but do not put mutable refund capacity in an unversioned shared cache. Large exports run against an auditable snapshot or a declared reporting cutoff so scanning seven years of history does not consume the database resources needed for live posting.

13Correctness deep dive

A unique webhook event ID suppresses repeated delivery of that event. It does not alone prevent two different notifications about the same capture from posting twice. The financial operation has its own uniqueness key: (M7, op81, capture). The posting routine serializes all outcomes for op81 and either returns its existing journal or commits the new one.

applyCapture(op81, fact ch81):
  begin transaction
  lock Operation(op81)
  verify fact merchant, currency, amount and provider reference
  if Journal(M7, op81, capture) exists: return existing result
  if fact conflicts: abort; record durable review case; return
  construct debit receivable 2500 / credit payable 2500 in USD minor units
  assert sum(debits) == sum(credits) and account/currency validity
  insert unique Journal and all Entries
  update Payment, CaptureBalance and balance projection
  insert captured event in Outbox
  commit

The review case is recorded after an aborted conflicting attempt, without posting money. Restrict direct writes so every posting uses this transaction. Worker W1 and callback applier W2 may reach it together. One holds the operation lock; the other waits and then observes the existing journal. If W1 aborts, it leaves no half-journal. If W1 commits and loses its reply, W2 returns j81. The unique constraint remains a second guard against an implementation race.

Refund requests lock the same capture balance before reserving funds:

This protects the limit even though provider calls occur outside the lock. A definitive failed refund can release its reservation transactionally. A refund confirmation appends reversing movements; it never edits j81. Partial refunds retain the remaining capacity, and reconciliation resolves unknown outcomes before capacity is reused.

14Failure and recovery

Failures can occur before the processor acts, after it acts but before we learn the result, or after our local posting commits. The recovery action depends on which boundary was crossed; treating every failure as permission to issue another capture would duplicate effects.

15Operations, security, and cost

Use processor tokens, least-privileged merchant roles and a separate audited refund permission. Verify webhook signatures over the provider-required payload representation, keep replay/deduplication records, and restrict outgoing provider credentials to adapters. Logs contain operation IDs and safe state transitions, not payment tokens or raw credentials. The merchant boundary must also appear in history queries, cache keys, settlement matching and operator tools.

Measure p95 intent latency and successful status availability, but alert first on unknown-operation age and unmatched amounts by currency. A five-dollar discrepancy is not equivalent to five failed HTTP calls. Track duplicate posting suppression, rejected refund reservations, provider throttling and reconciliation backlog. The posting routine’s balance assertion should reject a malformed journal and create an actionable audit signal.

The retention estimate gives a useful cost calculation without inventing cloud prices. Keeping one year hot and six years in a verified archive reduces hot logical record payload from about 255.5 TB to 36.5 TB, roughly sevenfold, before different index and replication policies. It introduces archive retrieval delay and restore complexity; archiving must keep the request identities and history needed to safely process current commands available. Processor operation fees and reconciliation calls may dominate storage savings, so report them as separate cost terms.

Roll out a new posting schema with old/new writer compatibility, shadow reconciliation and a small merchant cohort. Test crashes after remote success, after local commit and before queue acknowledgment; send reordered callbacks; race two refunds; restore a backup and replay captured events. The restored service must preserve operation/journal uniqueness. Passing a balance-total check alone cannot prove every real processor effect was recorded exactly once.

16Decision ledger and limitations

The design uses a local transaction to protect journal integrity and a separate recovery protocol to learn external processor outcomes. The decisions below explain what each mechanism protects, what it costs and when its assumptions would need to change.

We have not designed cross-currency transfers, chargeback adjudication or unrestricted multi-region multi-primary posting. Each would require new account rules and coordination, beyond simply adding a status value.

17Interview closing

“I start with a durable payment intent and a stable identity for each external operation. The API commits intent before acceptance. Provider workers run outside the database transaction; callbacks and reconciliation return evidence to one outcome applier.

“Each merchant routes to a posting authority where a unique operation and a balanced journal commit together. Concurrent outcomes cannot create a second journal. Refund requests reserve capacity under the capture’s lock before calling the processor, so unknown refunds retain their claim. Status can remain pending while we learn an external outcome; an error response never proves no money moved.

“The assumed peak leads me to benchmark about nine thousand local transaction phases per second before callbacks, recovery and refunds, isolate status reads and split independent merchant authorities only when needed. My costs are pending states, extra records, provider recovery work and shard operations. The next measurements are the heaviest merchant’s posting latency and the provider-limited backlog drain rate.”

Interviewer: “Now require zero loss after a region disappears.” Candidate: “I change the acknowledgment boundary to durable survival outside that region and validate recovery of operation identity as well as journals. I would price cross-region latency and reduced write availability during partitions. Merely placing an asynchronous copy in another region would not meet the new promise.”

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.