Design a webhook delivery platform

01Define acceptance before promising delivery

A webhook is an HTTP event sent to a customer-controlled endpoint. The sender records a business fact; the receiver decides what to do with it. In this design, a receiver’s successful response means it durably accepted the event, not that every downstream business action completed. An unreachable endpoint can exhaust a bounded retry contract.

Ask what the receiver’s success response must mean: durable acceptance or completed business work, and whether events need ordering. This design chooses durable acceptance and best-effort order, with bounded retries and inspectable failures.

Use shipment event E402: order O901 changed to shipped at object version 7. It matches endpoint EP 9 and creates logical delivery D22. The first HTTP attempt is A1. If the receiver commits E402 to storage but its 202 reply disappears, the sender does not know whether acceptance happened. Attempt A2 must preserve E402 and D22 while carrying a new attempt identity.

The smallest complete flow is shipment transaction plus pending event → delivery worker → signed HTTP request → receiver durable receipt → acknowledgment → later business processing. A pending sender record is an outbox. A receiver’s record of accepted events is an inbox. These solve opposite sides of the handoff, and no local transaction spans both companies’ databases.

02Functional requirements

Agree on what the service must do before choosing its components.

1. Manage destinations. Support endpoint registration, subscriptions, configuration changes and signing-key rotation. Pin queued work to an endpoint/configuration version so a URL edit does not silently rewrite past attempts.
2. Deliver business events. Send the recorded event to each matching customer endpoint and record durable receiver acceptance separately from completion of its downstream work.
3. Inspect and control delivery. Expose status, bounded attempt history, next retry and terminal reason; support authorized pause and cancellation.
4. Redrive retained events. Allow an authorized request to send a retained historical event again. Preserve its event identity and record the reason; today’s business state cannot replace expired historical bytes.

03Non-functional requirements

Use these hypothetical requirements for the worked interview. Confirm the assumptions with the interviewer; the numerical targets require testing and are not measured results.

1. Workload and initial delay. Assume ten million events/day with three matching endpoints/event and a tenfold peak. Start healthy first attempts within five seconds for at least 95% of deliveries.
2. Bounded retry and retention. Retain shared event payloads for seven days. For this exercise, allow at most ten automatic attempts in the initial run. An authorized redrive starts a separately audited run with the same ten-attempt ceiling; it preserves event/delivery identity and cannot extend the original seven-day payload lifetime. Cap total HTTP attempt duration at five seconds and capture at most 16 KiB of response data. Retain sanitized attempt/audit metadata for 30 days. A permanently offline receiver ends in an inspectable exhausted state.
3. Durability and effects. A committed business change must not lose its pending event. Retries preserve event/delivery identity; the receiver must durably accept before acknowledging. One local business effect depends on the receiver’s own transaction and retained identity, not a sender-only exactly-once promise.
4. Ordering. Use best-effort order. Consumers rely on object versions or reconciliation. Optional per-object serial dispatch can block later events behind a failure and does not establish global order across customers.
5. Isolation and fair capacity. A slow endpoint must not occupy every socket or worker. Bound endpoint concurrency, tenant shares and global transport work; receiver replay protection must cover the supported redrive horizon.
6. Authenticity and safe delivery. Authenticate tenant operations, sign documented bytes, enforce timestamp replay policy and validate the actual outbound destination on every attempt. Signature validity is distinct from schema validation, authorization and business idempotency.

04Build one durable sender and one receiver inbox

One periodic worker scans due deliveries. In a short database transaction it claims D22, records attempt A1 and assigns a lease token. A lease gives temporary ownership of local processing; its token lets the database recognize which attempt may update current state. Commit and release locks before making the remote HTTP call.

The worker loads the pinned event/configuration, validates the destination and signs a fresh request. On 202 it records accepted only if its token and expected state are still current. On a timeout it records an uncertain outcome and schedules a bounded retry. It cannot roll back the shipment or infer that the event never arrived.

The receiver verifies signature and schema, then atomically inserts a uniquely scoped inbox E402 and durable processing work. It returns 202 only after this commits. Its worker later applies the local business mutation and marks the inbox done in one local transaction. A process crash after acceptance is then recoverable internally.

This baseline supports a small service. Its one blocking worker at 0.5 seconds/request handles only two attempts/s, so throughput is the first scaling limit. Its lost-response behavior is already the core correctness problem; more workers will not remove that uncertainty.

05Size attempts and sockets rather than events alone

Ten million events with three destinations create 30 million logical deliveries/day, about 347/s average. At a tenfold peak that is roughly 3,470 first attempts/s. If each logical delivery averages 1.2 attempts, peak transport work becomes approximately 4,167/s. Retry rate is a capacity assumption to measure, not an unlimited allowance.

At an extra 1,000 attempts/s, that backlog takes about 3.5 hours to drain, if receivers permit the rate. More workers cannot responsibly exceed a customer’s accepted concurrency. Separate endpoint limits, tenant fairness and global socket budgets so a backlog cannot turn into a recovery stampede.

06Give every level a stable identity

The business transaction updates O901 and inserts immutable outbox event E402 together. The planner creates D22 with this unique delivery key:

(tenant, eventId, endpointId, configVersion)

Including endpoint ID matters: two different endpoints can both have configuration version 3 and both need the event.

Sender records

A due-time index finds pending work without scanning every historical attempt.

Event envelope fields

Per-attempt headers

Define exactly which raw bytes and fields are signed. Re-parsing and reserializing JSON can change bytes, so verification operates on the received representation before trusting its content.

Inspection returns tenant-scoped state and a bounded history cursor. Redrive requires authorization and retained original bytes. An expired payload cannot be reconstructed from today’s order state and called the same historical event. Store secret references rather than signing keys in ordinary delivery records or logs.

Record a scoped redrive request ID and its run ID atomically before making exhausted work eligible again. Repeating that request returns the same run rather than resetting its budget repeatedly. Track automatic-attempt count per run; do not reset the event identity or original payload-expiry time.

07Classify failures and schedule them fairly

Retry eligibility asks whether another attempt could help; timing asks when. Network errors, timeouts and many 5xx responses receive bounded retries. A 429 indicates receiver load control and may supply retry guidance that must be validated and capped. Document permanent endpoint/configuration errors rather than assuming every 4xx has identical meaning.

Exponential backoff increases delay after repeated failures. Jitter varies that delay across deliveries to avoid synchronized retries. Stop when the current run reaches ten automatic attempts or the original retained event reaches its seven-day expiry, whichever comes first. An authorized redrive is a new audited run, not an automatic counter reset. Expose next retry and terminal reason so “we retry” is an observable contract.

Add bounded parallel workers, with per-endpoint concurrency, per-tenant scheduling shares and a global socket ceiling. Limits cannot simply be independent per-worker counters, or adding workers multiplies the advertised endpoint cap. Expensive retries must not starve first attempts or other tenants.

When database polling becomes costly, use a ready queue containing delivery IDs and a durable due-time schedule for delayed work. The database remains authoritative. Workers recheck state after dequeue, and periodic due scans repair a lost enqueue. Queue duplication is expected and must not create a second logical delivery. This adds scheduling machinery while preserving the simple baseline’s stored obligations.

08Follow the lost acknowledgment through both databases

A1 sends E402/D22. NorthHarbor’s endpoint verifies the raw signed bytes and commits inbox E402 plus its processing job. Its 202 response is lost. The sender eventually records timeout and keeps the logical delivery eligible for retry; it has no observation distinguishing this history from a request that never arrived.

A2 receives a new attempt ID, timestamp, signature and lease token, while event/body identity remains stable. The receiver’s unique inbox lookup finds the already accepted fingerprint and returns 202 again. It does not create another processing job. D22 becomes accepted under A2’s valid token. The receiver’s separate processor may already have completed or may still be queued.

Now let A1’s old worker resume and report retrying after A2 succeeded. A guarded sender update requires its token to match the current delivery token and its expected state. The stale update fails instead of undoing accepted. Attempt history can still record a correctly associated late observation without replacing delivery truth.

The lease-token check stops A1 from overwriting newer sender state. The receiver’s saved event identity stops repeated arrivals from creating another local effect. Both are needed: rejecting A1’s late database write cannot undo an HTTP request already sent.

09Prove the receiver’s effect and explain ordering

Use a trusted (sender, tenant, eventId) inbox key and immutable payload fingerprint. A receipt transaction inserts the inbox and job under that unique identity; a duplicate verifies the same fingerprint. Different bytes under the same identity conflict rather than silently replacing the event. Identity comes from the verified credential mapping, not an unsigned tenant header.

For a local database effect, use one processing transaction:

1. Lock the inbox row. Concurrent processors serialize through that row.
2. Check completion. Return without repeating the effect if the inbox is already done.
3. Apply the effect. Make the local business change and mark the inbox done.
4. Commit both changes. A crash before commit preserves neither; a retry after commit sees done.

This proves one local effect under the stated database boundary despite multiple HTTP attempts.

An external charge or email is outside that transaction. Persist a stable outgoing action identity and use the destination’s idempotency/status contract. Represent uncertainty and reconcile before inventing another action. An inbox alone cannot make an arbitrary remote API atomic.

For full object snapshots, atomically install only a newer object version. For changes such as “subtract one unit,” an older unapplied event still matters. Detect missing events and replay them or fetch complete state. Serial sending through each 202 orders receipt, while the receiver must separately preserve processing order if business actions require it.

10Treat outbound URLs as a trust boundary

A customer-controlled URL can trick the service into requesting internal or metadata endpoints: this is server-side request forgery. Restrict schemes, validate resolved IPv4/IPv6 destinations at connection time, and either disable redirects or validate every hop. Registration-time DNS checks cannot establish where a later connection will go.

The HTTP client must connect to the address that passed validation. Resolve, choose an allowed address and connect to that address while retaining the original hostname for TLS certificate verification. Do not validate one resolution and let the HTTP client perform a separate unchecked lookup. Add network egress restrictions as another boundary and repeat checks on every attempt.

Sign exact body bytes under a documented timestamp/key-ID scheme. The receiver verifies authenticity and rejects excessive timestamp age under a replay policy. Fresh retry signatures remain compatible with stable event identity. A valid signature does not remove schema checks or business authorization.

Rotate keys with a bounded overlap and explicit retirement rules. Limit body size, decompressed size, response bytes, connection time and total attempt duration. Sanitize inspection and logs because a receiver can return secrets in an error body. Pause/delete policies must be rechecked before sending even when a stale ready-queue entry exists.

11Operate a bounded promise through outages

Monitor first-attempt delay, eventual acceptance, oldest pending age, attempts per delivery, endpoint sockets, lease expiry, backlog age and payload-retention margin. Successful eventual delivery can hide hours of delay; separate it from the five-second healthy-first-attempt objective.

During an endpoint outage, back off that endpoint while other tenants continue. Reconnection does not authorize an unbounded drain. Keep failed deliveries inspectable and offer redrive only within the payload and destination-ownership contract. Receiver inbox retention must cover the allowed replay/redrive horizon, or the service must disclose a weaker duplicate-effect guarantee for older work.

Test sender death after POST, receiver death after inbox commit, local processor death around its transaction, stale lease updates, duplicate redrive, key rotation and DNS changes between attempts. Test one noisy tenant alongside healthy ones, not merely a global throughput benchmark.

Shared payloads reduce repeated storage, while attempt history can become a major cost under persistent failure. Bound retention and response capture. If the product requires proof of eventual business completion, add a separate authenticated status protocol; changing the meaning of HTTP accepted in marketing does not implement that requirement.

12Check the design against its requirements

Before closing, check the final design against the agreed requirements. FR means functional requirement and NFR means non-functional requirement; the numbers refer to the lists above. These are proposed validation checks, not test results.

13Rapid revision

Rehearse the numbered functional requirements and non-functional targets first. Use this table to recall the mechanisms, then close with the requirements check above.

Remember: Lost reply → same event, new attempt.

Close with: “I preserve the shipment event before sending, schedule bounded fair attempts, and keep event identity stable through uncertain transport. The receiver commits an inbox before acknowledgment and protects its own business boundary. I can guarantee durable attempts and explain outcomes, but cannot force an unreachable customer to accept or make an arbitrary external action exactly once.”

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.