Durable Agent Workflows

A worker is disposable. A durable workflow can be reconstructed from persisted history after a crash, deployment, eviction, or long human pause. The record needs a stable workflow identity, a definition version, typed inputs, ordered commands, activity outcomes, signals, approvals, timers, cancellation, and terminal status. Model context is derived working material; it is neither the authoritative history nor a safe place for credentials.

For a remote write, the caller can atomically control neither the remote mutation nor its own outcome record. If the worker crashes after the remote service commits but before history records success, recovery sees a scheduled action with no terminal result. That is an ambiguous outcome. Blind retry can duplicate the effect; assuming success can skip work that never happened.

1. 01Persist intentRecord the exact operation, authorization context, input digest, idempotency key, deadline, and attempt before dispatch.
2. 02Execute through a narrow adapterPass the same caller-chosen key on every retry and require the service to reject key reuse with different parameters.
3. 03Persist the typed outcomeRecord success, retryable failure, permanent failure, or cancellation with an external receipt when available.
4. 04Reconcile ambiguityAfter a missing acknowledgement, query authoritative state using the original identity before issuing another mutation.

An idempotency key is not a random retry label. Scope it to the authenticated caller and stable business intent, store the first accepted parameters, and return a semantically equivalent outcome for a duplicate when the service contract supports that behavior. The same key with a different amount, recipient, tenant, or payload must fail as an intent mismatch rather than being silently deduplicated.

1def audit_recovery(contract: WorkflowContract, snapshot: RecoverySnapshot) -> RecoveryReport:
2    """Replay declarations and reconcile ambiguous effects without executing a tool."""
3    contract = validate_record(contract, WorkflowContract)
4    snapshot = validate_record(snapshot, RecoverySnapshot)
5    if snapshot.scope != contract.scope or snapshot.contract_content_id != contract.content_id:
6        raise ValueError("snapshot belongs to another workflow contract")
7    states = _validate_history(snapshot.plan, snapshot.events, contract)

example=illustrative_only
status=RESUMABLE
completed=reserve-stock,charge-card
next=send-receipt
decision=reconcile-before-retry
claim=LOCAL_RECOVERY_AUDIT_NOT_EXACTLY_ONCE_GUARANTEE

The fixture records a completed inventory reservation, an approved payment that was scheduled, and an authoritative payment receipt observed after the schedule event but before a local success event. Recovery counts the payment once and advances to the receipt step. The implementation reconstructs every frozen record, copies supplied collections, verifies digests and scope, rejects noncontiguous plan/history positions, enforces a sequential transition grammar, caps attempts, and rejects boolean integers, non-finite timeout values, string subclasses, and constructor bypass.

This result is illustrative only. The script does not run an activity, persist an event, authenticate a receipt, lock a resource, emulate a workflow service, or prove exactly-once delivery. Its receipt is declared authoritative by fixture input; a real adapter must establish that fact from authenticated external evidence.

A human pause is a persisted workflow state, not a blocked process. Store the proposed action and digest shown to the reviewer, authenticated reviewer, decision, policy version, time, and expiry. On resume, re-check tenant and resource authorization, approval freshness, and mutable preconditions. If the payload changes, request a new approval. If external state changed, branch through an explicit replan or intervention event rather than pretending the old plan still applies.

• Cancellation is cooperative for work already dispatched. Record the request and the observed cancellation outcome; reconcile any effect that may already have committed.
• Use logical workflow timers rather than worker sleep. Persist deadlines and define what late signals mean.
• Version workflow definitions with replay compatibility. A code deployment must still interpret open histories or migrate them through an explicit marker.
• Keep secrets and bulky model context out of history. Store bounded references with retention, encryption, and deletion policy appropriate to the data.

Test recovery by killing workers at every persistence boundary, not merely by throwing an exception inside an activity. Re-deliver events, reorder allowed signals, deploy old and new workflow code against captured histories, and inject timeouts after an external service has committed. A release gate should prove deterministic replay for representative histories and show which effect classes have genuine reconciliation support.