3.1 KiB
Durable Workflow Runtime Recovery
Workflow Engine uses Core's recovery ledger and distributed leases for every consequential module-action invocation and every worker-owned instance, trigger delivery, or timer transition. The ledger is evidence for an effect boundary; it is not a replacement for the Workflow instance history.
Module Actions
An action provider declares a recovery mode and concrete verification checks in
its ActionDefinition. Before dispatch, Workflow Engine persists the pinned
definition hash, canonical action-input and preview hashes, current access
context hash, provider idempotency-key hash, and an execution fence. Raw action
input and credentials are not copied into recovery evidence.
A conclusive provider result commits the Workflow projection and verified terminal checkpoint together. Pending and retryable provider results remain visible handoffs, but each later provider poll gets a new ledger operation while retaining the provider's stable action idempotency key.
An exception, invalid result, or unannounced effect after a non-atomic dispatch
is not an ordinary failure. The step becomes outcome_unknown, continuation is
blocked, and Retry is unavailable. An operator must inspect the provider and
record durable evidence for one of two decisions:
- Effect confirmed records that the provider effect occurred and advances the Workflow without invoking the provider again.
- Effect absent records forward recovery and enables a deliberate retry under a new invocation ledger key and the same provider idempotency key.
Atomic actions are different: provider database changes and the Workflow projection share one transaction. An exception rolls that transaction back, so the step can become retryable without claiming an unknown external outcome.
Snapshot-restore and irreversible actions are rejected before dispatch unless the Workflow node pins the required backup or approval reference.
Workers, Triggers, And Timers
Runtime workers acquire a distributed fence for each instance. Trigger deliveries and wait states use their own fences. Database row locks remain a local optimization; the fence is the authority proof across hosts. A second runtime cannot advance an actively owned item. After lease expiry, a new runtime receives a higher fence number and may resume from durable state.
Linked Dataflow runs retain their own recovery boundary. If Dataflow reports an unknown publication outcome, Workflow displays a recovery handoff and only allows cancellation. It polls again after the Dataflow outcome is reconciled and advances only from a conclusive run descriptor.
Process Failure Versus Effect Uncertainty
A process failure before dispatch is retryable after current authority and inputs are revalidated. A failure after non-atomic dispatch is an unresolved effect even when no provider response was received. Operators must not infer absence from a timeout, worker restart, HTTP error, or empty local result.
Checkpoint evidence is hash-chained. A broken chain prevents a success commit; it must be investigated as an integrity incident rather than bypassed.