Design durable agent workflows

01Design a business process that uses a model

An agent workflow is a process that uses a model to interpret information or propose a next step. The application still decides which tools exist, which actions are permitted and what counts as completion. A conversational transcript alone is not a reliable record of a purchase, approval or retry.

Ask which actions the model may propose, which need human approval, and what should happen if the vendor receives an order but its reply is lost. This design permits a fixed purchasing process, approval of exact terms and an explicit uncertain-submission outcome.

Our example buys ten laptops from one of three approved vendors. The model compares quotes and drafts a proposal. A human approves an exact proposal, and only then may the service submit the order. The workflow may wait thirty minutes for approval, survive worker restarts and report the final order identity. Payment settlement and arbitrary browsing are outside this scope.

The complete flow is save task → research quotes → save proposal → wait for approval → submit approved order → save result. This fixed sequence is intentionally enough for the interview. The model helps with research and explanation; it does not invent new privileges or freely change the workflow. A durable workflow means that progress survives a process crash and execution resumes from recorded state rather than starting the business action again.

02Functional requirements

Agree on what the service must do before choosing its components.

1. Create and inspect tasks. Create a tenant/requester-owned purchase task, return its durable identity and show its phase, current proposal and any external order result.
2. Research an allowed purchase. Compare quotes from the three approved vendors and draft a proposal for ten laptops. The model supplies bounded interpretation; trusted code checks commercial terms and policy.
3. Review and approve exact terms. Let an authorized human review and approve the exact current proposal, with a documented expiry. Replacing the proposal or changing its terms invalidates earlier approval.
4. Submit and audit an order. Submit only the authorized proposal, preserve the external action identity, record the provider’s result and expose reconciliation when the result is uncertain.
5. Cancel at a defined boundary. Before submission admission, cancellation prevents the action from starting. After the provider may have accepted, any cancellation is a separate business operation. Payment settlement and arbitrary browsing are outside scope.

03Non-functional requirements

Use these hypothetical requirements for the worked interview. Confirm the assumptions with the interviewer; the numerical targets require testing and are not measured results. For latency, p95 and p99 mean that 95% and 99% of measured delays, respectively, are no greater than the reported value.

1. Workload. Assume 100,000 starts/day, ten model calls and fifteen ordinary tool calls per task, with a thirty-minute average approval wait. Account separately for bursts, retries and vendor limits.
2. Responsive control. Target task creation and status p95 below 300 ms. Measure research, approval wait and submission completion separately; an approval that may take hours cannot share a tiny end-to-end latency target.
3. Durable progress. Saved task state and activity results must survive the selected database-node failure. Waiting tasks consume durable state and timers rather than blocked workers. Retain activity results, approvals and provider reconciliation records through the declared recovery window, and keep unresolved action evidence available until reconciled.
4. Exact authorization. Submission must use the exact currently approved proposal, a stable action identity and current permission/budget checks. Tenant isolation, allowed vendors, bounded quantities/spend and approval/quote expiry apply even when the model recommends otherwise.
5. Honest external outcomes. Provider idempotency and lookup determine recovery after a lost response. Report SubmissionUnknown when evidence is insufficient; do not claim canceled, completed or exactly-once remote execution merely because local workflow history is durable.
6. Bounded execution and privacy. Limit steps, tokens, retries, elapsed time and provider concurrency per tenant/task. Separate model calls, vendor reads and submission capacity; keep credentials and unrestricted network access outside model control.

04Save each completed step before moving forward

Begin with an API, a relational database and one background worker. Creating task T81 commits its owner, inputs and initial state before returning its ID. The worker claims ready work, loads T81 and performs the next activity. An activity is one bounded operation, such as fetching a vendor quote or asking the model to compare saved quotes.

Save each successful activity result and advance the workflow state in a transaction. If the worker restarts after that commit, it reads the saved result instead of making the same model call again. If it crashes before recording a harmless quote read, that read may run again. Re-execution is acceptable only when the operation’s effects permit it.

After research, save proposal P8 and enter WaitingApproval. No worker remains blocked for thirty minutes; the database records the wait. An approval request validates the actor and proposal version, then makes the submission step eligible. A worker submits it using a stable external action key and records the provider’s order ID.

The API returns task status from durable state. A browser disconnect does not cancel or erase the purchase process. This baseline already explains asynchronous execution, restart recovery and the difference between a waiting task and a running worker.

05Separate running work from waiting tasks

Assume 100,000 task starts per day: approximately 1.16 starts/s on average. At ten model calls and fifteen ordinary tool calls per task, the averages are about 11.6 model calls/s and 17.4 tool calls/s. Peak bursts, retries and vendor rate limits require separate headroom.

With 3,000 input tokens and 500 output tokens per model call, average model demand is roughly 34,700 input tokens/s and 5,800 output tokens/s. Token volume and the number of active requests both matter when choosing provider quotas and worker concurrency. A fixed request count hides large prompts and long responses.

If approval takes thirty minutes on average, about 1.16 × 1,800 ≈ 2,083 tasks are waiting at any instant. They need durable rows and timers, not 2,083 blocked worker processes. The worker pool should scale with ready activities and external call latency.

At 200 saved events of 2 KB each, one task adds about 400 KB of history and 100,000 tasks add 40 GB/day before indexes and replication. Store large documents separately and reference immutable versions from history. Retention and redaction decisions should preserve the evidence needed for recovery without keeping every sensitive prompt forever.

06Give task, proposal and action separate identities

Create a task

POST /tasks

Task control operations

Durable workflow records

Proposal fields

External action identity in the example

Task:        T81
Proposal:    P8
Action key:  T81-P8-submit

The key identifies one intended purchase attempt, not a worker process. A replacement worker must reuse it when recovering that action.

Use expected task versions or a database lock to prevent two workers from advancing the same state concurrently. This protects local records. It does not undo an external effect already performed by a worker whose lease expired, so external action identity remains necessary even with careful scheduling.

07Keep model output inside an explicit workflow

Research first gathers quotes through permitted read-only tools. The model receives those saved facts and returns a structured comparison. Validate required fields and business constraints before constructing P8. Model output is a proposal; trusted application code calculates totals, checks approved vendors and determines whether the next step is allowed.

Waiting for approval is a durable state with an expiry timer. The approval endpoint checks the caller’s authority and the exact current proposal. A sentence saying “approved” in a document or chat message is not an authenticated approval. If a vendor changes the price or quote validity expires, produce a new proposal and request new approval.

Before contacting the provider

1. Recheck permission and terms. Validate applicable permission, budget and proposal validity.
2. Save intent. Persist the exact outgoing action and its stable key.
3. Submit. Only then contact the provider.

This creates a recoverable account of what the worker intended to send even if the response never arrives.

The model may explain why one quote is preferable, but it does not receive raw production credentials or construct unrestricted network requests. Application code checks permission and saves progress; the model compares the supplied facts and explains its proposal. Each responsibility can be tested separately.

08Add workers without losing the saved process

A durable work queue separates task admission from execution. Publish ready work through a transactional outbox or an equivalent database-backed queue so a committed task cannot disappear between saving it and enqueueing it. An outbox is a table of pending dispatch records written in the same transaction as the state change; a relay forwards them and may retry.

Workers claim activities with bounded leases and renew while running. On expiry, another worker can recover the activity from durable state. Local version checks reject stale state updates. A duplicate queue message first inspects the recorded step outcome instead of assuming another complete execution is required.

Use separate concurrency limits for model calls, vendor reads and order submission. One slow vendor should not consume the entire pool. Apply tenant quotas and bound retry counts, token budgets and elapsed time per task. Waiting approval tasks consume neither provider concurrency nor busy workers.

A workflow engine can supply timers, history and retry scheduling once those requirements justify it. It does not automatically solve provider idempotency or business approval semantics. Keep the same saved-state and external-action contracts when adopting an engine, and pin workflow versions so a code deployment does not reinterpret an old task unpredictably.

09Recover the order whose response was lost

Lost-response trace for approved proposal P8

1. Save the action. T81’s worker records T81-P8-submit.
2. Send the purchase. The worker calls the vendor.
3. The vendor accepts. It creates order O902.
4. The response is lost. The connection breaks before the worker receives O902.

The saved action records what the worker was authorized to submit. It does not show whether the request reached the vendor or created an order.

If the provider supports durable idempotent submission, retry the same key and identical payload within its documented retention window. It should return the existing outcome instead of creating another order. If lookup by action key or client reference is available, query it and save the confirmed order identity. Provider guarantees must be verified; no generic HTTP retry supplies them.

If the provider cannot resolve the action safely, leave it SubmissionUnknown and reconcile through a controlled operator process. Do not invent a fresh key just to get a successful response. That would convert one uncertain purchase into a possible duplicate purchase.

This is the central recovery story. Saving workflow history prevents forgotten progress, but cannot make a remote side effect part of a local database transaction. The application needs provider cooperation or an honest uncertain state. “Exactly once” without that boundary obscures the business risk instead of solving it.

10Handle changes while a task is in progress

Before the service authorizes submission, canceling T81 saves a terminal state that blocks the purchase. Submission checks the current task version. If cancellation and submission race, the database accepts one state change first; that order determines whether submission was authorized, and the UI reports the result.

Once submission may have reached the provider, cancellation becomes a request to stop or reverse an order, subject to the vendor’s API and business rules. A successful local cancel button cannot prove an in-flight remote purchase vanished. Resolve the original action first, then perform any supported cancellation under its own tracked identity.

Changing quantity, vendor or destination creates a new proposal version. Approval for P8 does not authorize P9 even if both belong to T81. Similarly, an expired quote may require research and approval again rather than silently purchasing at a new price.

Retry read-only activities with bounded backoff, but classify failures. Invalid input is not repaired by ten retries, and an uncertain purchase is not equivalent to a failed quote read. Escalation, compensation and human review are explicit workflow outcomes. The simplest safe design makes those states visible rather than hiding them behind a generic retry loop.

11Test business boundaries and recovery evidence

Trace task, activity, proposal and action IDs across logs while redacting credentials and unnecessary personal data. Measure queue age, ready versus waiting tasks, activity latency, provider error rate, token/spend budgets, approval age and unresolved submissions. A high completion rate can conceal dangerous duplicates or stale approvals, so inspect those outcomes separately.

Test a worker crash before saving a quote, after saving a model result, while waiting for approval and immediately after a provider accepts an order. The expected recovery differs at each point. Also test duplicate queue messages, changed proposals, revoked approvers, expired quotes and exhausted provider idempotency retention.

Treat retrieved content and vendor descriptions as untrusted input. Instructions embedded in them cannot authorize a tool or change the allowlist. Tool gateways validate schemas and permissions; secrets remain in trusted service integrations. Sandboxed computations may help comparison, but a sandbox is not a substitute for authorizing the eventual purchase.

Deploy workflow changes with compatibility for existing histories or explicitly migrate them. Practice operational reconciliation using realistic provider evidence. The strongest final design is one the operator can explain after a timeout: what was approved, what may have been sent, what was confirmed, and which action is still safe to take.

12Check the design against its requirements

Before closing, check the final design against the agreed requirements. FR means functional requirement and NFR means non-functional requirement; the numbers refer to the lists above. These are proposed validation checks, not test results.

13Rapid revision

Rehearse the numbered functional requirements and non-functional targets first. Use this table to recall the mechanisms, then close with the requirements check above.

Remember: Lost purchase reply → recover the same action.

Close with: “I use a fixed durable workflow around the model. Saved proposals and authenticated approvals authorize exact actions. Workers can restart, but external effects recover through stable identities and provider evidence. When that evidence is insufficient, the workflow reports uncertainty instead of blindly buying again.”

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.