# Durable Workflow Runtime Recovery

Workflow Engine uses Core's recovery ledger and distributed leases for every
consequential module-action invocation and every worker-owned instance, trigger
delivery, or timer transition. The ledger is evidence for an effect boundary;
it is not a replacement for the Workflow instance history.

## Module Actions

An action provider declares a recovery mode and concrete verification checks in
its `ActionDefinition`. Before dispatch, Workflow Engine persists the pinned
definition hash, canonical action-input and preview hashes, current access
context hash, provider idempotency-key hash, and an execution fence. Raw action
input and credentials are not copied into recovery evidence.

A conclusive provider result commits the Workflow projection and verified
terminal checkpoint together. Pending and retryable provider results remain
visible handoffs, but each later provider poll gets a new ledger operation while
retaining the provider's stable action idempotency key.

An exception, invalid result, or unannounced effect after a non-atomic dispatch
is not an ordinary failure. The step becomes `outcome_unknown`, continuation is
blocked, and Retry is unavailable. An operator must inspect the provider and
record durable evidence for one of two decisions:

- **Effect confirmed** records that the provider effect occurred and advances
  the Workflow without invoking the provider again.
- **Effect absent** records forward recovery and enables a deliberate retry
  under a new invocation ledger key and the same provider idempotency key.

Atomic actions are different: provider database changes and the Workflow
projection share one transaction. An exception rolls that transaction back, so
the step can become retryable without claiming an unknown external outcome.

Snapshot-restore and irreversible actions are rejected before dispatch unless
the Workflow node pins the required backup or approval reference.

## Workers, Triggers, And Timers

Runtime workers acquire a distributed fence for each instance. Trigger
deliveries and wait states use their own fences. Database row locks remain a
local optimization; the fence is the authority proof across hosts. A second
runtime cannot advance an actively owned item. After lease expiry, a new
runtime receives a higher fence number and may resume from durable state.

Linked Dataflow runs retain their own recovery boundary. If Dataflow reports an
unknown publication outcome, Workflow displays a recovery handoff and only
allows cancellation. It polls again after the Dataflow outcome is reconciled
and advances only from a conclusive run descriptor.

## Process Failure Versus Effect Uncertainty

A process failure before dispatch is retryable after current authority and
inputs are revalidated. A failure after non-atomic dispatch is an unresolved
effect even when no provider response was received. Operators must not infer
absence from a timeout, worker restart, HTTP error, or empty local result.

Checkpoint evidence is hash-chained. A broken chain prevents a success commit;
it must be investigated as an integrity incident rather than bypassed.