60 lines
3.1 KiB
Markdown
60 lines
3.1 KiB
Markdown
# Durable Workflow Runtime Recovery
|
|
|
|
Workflow Engine uses Core's recovery ledger and distributed leases for every
|
|
consequential module-action invocation and every worker-owned instance, trigger
|
|
delivery, or timer transition. The ledger is evidence for an effect boundary;
|
|
it is not a replacement for the Workflow instance history.
|
|
|
|
## Module Actions
|
|
|
|
An action provider declares a recovery mode and concrete verification checks in
|
|
its `ActionDefinition`. Before dispatch, Workflow Engine persists the pinned
|
|
definition hash, canonical action-input and preview hashes, current access
|
|
context hash, provider idempotency-key hash, and an execution fence. Raw action
|
|
input and credentials are not copied into recovery evidence.
|
|
|
|
A conclusive provider result commits the Workflow projection and verified
|
|
terminal checkpoint together. Pending and retryable provider results remain
|
|
visible handoffs, but each later provider poll gets a new ledger operation while
|
|
retaining the provider's stable action idempotency key.
|
|
|
|
An exception, invalid result, or unannounced effect after a non-atomic dispatch
|
|
is not an ordinary failure. The step becomes `outcome_unknown`, continuation is
|
|
blocked, and Retry is unavailable. An operator must inspect the provider and
|
|
record durable evidence for one of two decisions:
|
|
|
|
- **Effect confirmed** records that the provider effect occurred and advances
|
|
the Workflow without invoking the provider again.
|
|
- **Effect absent** records forward recovery and enables a deliberate retry
|
|
under a new invocation ledger key and the same provider idempotency key.
|
|
|
|
Atomic actions are different: provider database changes and the Workflow
|
|
projection share one transaction. An exception rolls that transaction back, so
|
|
the step can become retryable without claiming an unknown external outcome.
|
|
|
|
Snapshot-restore and irreversible actions are rejected before dispatch unless
|
|
the Workflow node pins the required backup or approval reference.
|
|
|
|
## Workers, Triggers, And Timers
|
|
|
|
Runtime workers acquire a distributed fence for each instance. Trigger
|
|
deliveries and wait states use their own fences. Database row locks remain a
|
|
local optimization; the fence is the authority proof across hosts. A second
|
|
runtime cannot advance an actively owned item. After lease expiry, a new
|
|
runtime receives a higher fence number and may resume from durable state.
|
|
|
|
Linked Dataflow runs retain their own recovery boundary. If Dataflow reports an
|
|
unknown publication outcome, Workflow displays a recovery handoff and only
|
|
allows cancellation. It polls again after the Dataflow outcome is reconciled
|
|
and advances only from a conclusive run descriptor.
|
|
|
|
## Process Failure Versus Effect Uncertainty
|
|
|
|
A process failure before dispatch is retryable after current authority and
|
|
inputs are revalidated. A failure after non-atomic dispatch is an unresolved
|
|
effect even when no provider response was received. Operators must not infer
|
|
absence from a timeout, worker restart, HTTP error, or empty local result.
|
|
|
|
Checkpoint evidence is hash-chained. A broken chain prevents a success commit;
|
|
it must be investigated as an integrity incident rather than bypassed.
|