Synchronous versus asynchronous communication

01Who waits for the outcome?

A report takes 45 seconds to produce, but the product wants to acknowledge the request within 300 ms. Decide what that first response promises.

1. Synchronous completion: the caller waits for the report operation's final result. The response claims completion or reports a failure/uncertain outcome.
2. Asynchronous acceptance: the service saves a job, returns its identifier quickly and completes it later. The caller checks a status endpoint for the final result.

Accepted must not mean “a function was called in memory.” Under our promise it means enough work was saved to survive a worker restart. It also does not mean the report file already exists.

Nonblocking I/O is a separate programming technique: a thread does other work while disk or network activity waits. The API can still be synchronous if the caller waits for completion. Likewise, a queue does not remove a business dependency: shipping must still wait for required payment confirmation.

02Decoupling creates a lifecycle

Give the report a saved job record containing its owner, parameters, state and output reference. Follow that record:

1. The API checks permission and parameters. It saves the job and recoverable pending work before replying.
2. The API returns accepted with job ID 42 and a status URL. The caller now has a way to distinguish pending, running, complete and failed.
3. A worker first checks for completed work. If the job already records completion, reuse its published output reference. Otherwise, one job-store update records a new claim number and deadline. Other workers cannot take the incomplete job before that deadline; afterward one can claim it with a new number. Expiry does not stop a paused old process.
4. Each claim writes a separate immutable output object. For job 42, claim 7 writes reports/42/claims/7; a replacement claim 8 writes a different object. Use an object-store write that fails if that key already exists; a retry can inspect that object instead of overwriting it. In one conditional job-store update, publish the output reference and completed state only if the recorded claim number still matches. If claim 8 has taken over, claim 7 cannot publish its reference or overwrite claim 8’s bytes. A separate earlier claim check would leave a race before publication.
5. The caller checks the authenticated status API. It downloads only a completed result it may access.
6. Repeated failure reaches a finite repair path. After limited attempts, mark failed or move work for inspection instead of wasting workers forever.

A repeated delivery keeps the same logical job ID and returns its published result. Cleanup may remove abandoned, unpublished objects, but must retain the object referenced by a completed job for its documented result-retention period.

An event says something happened; a queued command asks a worker to do something. Both need saved identity and retry rules.

03Worked example: report generation

If 60 reports/minute arrive and 50 complete, unfinished work grows by ten/minute. After 15 minutes it adds 150 jobs. A fast acceptance response can hide that growing delay; measure oldest pending age, add verified capacity or accept less work.

Now follow an order and its receipt email

1. The database saves the order and a pending receipt record together. This pending record is an outbox entry. If the transaction fails, neither is committed; if it succeeds, a crash cannot make the system forget the receipt work.
2. A relay reads pending entries and sends them to the message system. It marks progress after a successful response, and retries unfinished entries.
3. The message system accepts a send but its reply is lost. The relay may send the same event again. The outbox prevents forgotten work, not all repeated delivery.
4. A mail worker sends the receipt. If the provider recognizes a stable send identifier, a retry can resolve the earlier send instead of sending again.
5. Without that provider support, a gap remains. Writing a local “sent” flag before sending risks no email if the worker crashes. Writing it afterward permits a duplicate if the provider accepted the email before the crash.

Explain that external boundary honestly. A database transaction cannot unsend an accepted email, and a durable queue does not give every external action exactly-once behavior.

04Keep the critical path purposeful

Separate operations by what the user needs now:

• Finish purchase checks before returning purchase success. Verify the buyer’s permission and save the stock reservation; a queued intention to check them later is not an accepted purchase.
• Run suitable reports, email and analytics later when delayed completion fits the product and saved jobs expose the result.
• Define cancellation by progress. For a pending job, save cancelled state and make workers skip it. For running work, report cancellation requested until the worker confirms a stop. A completed charge needs a separate refund action; an accepted email cannot be unsent by changing job state.
• Keep job history long enough for a client to resolve an old identifier and for operators to repair failed work.
• Bound arrivals and retries. A queue absorbs a finite burst, not an unlimited sustained completion promise.

In the interview, stop the worker before its effect, after its effect and after acknowledgment. At each point, show the saved job/result and what another worker does next. That is the mechanism behind reliable delayed work.