Fence and reconcile workflow runtime effects

docs/DURABLE_RUNTIME_RECOVERY.md

@@ -0,0 +1,59 @@
+# Durable Workflow Runtime Recovery
+
+Workflow Engine uses Core's recovery ledger and distributed leases for every
+consequential module-action invocation and every worker-owned instance, trigger
+delivery, or timer transition. The ledger is evidence for an effect boundary;
+it is not a replacement for the Workflow instance history.
+
+## Module Actions
+
+An action provider declares a recovery mode and concrete verification checks in
+its `ActionDefinition`. Before dispatch, Workflow Engine persists the pinned
+definition hash, canonical action-input and preview hashes, current access
+context hash, provider idempotency-key hash, and an execution fence. Raw action
+input and credentials are not copied into recovery evidence.
+
+A conclusive provider result commits the Workflow projection and verified
+terminal checkpoint together. Pending and retryable provider results remain
+visible handoffs, but each later provider poll gets a new ledger operation while
+retaining the provider's stable action idempotency key.
+
+An exception, invalid result, or unannounced effect after a non-atomic dispatch
+is not an ordinary failure. The step becomes `outcome_unknown`, continuation is
+blocked, and Retry is unavailable. An operator must inspect the provider and
+record durable evidence for one of two decisions:
+
+- **Effect confirmed** records that the provider effect occurred and advances
+  the Workflow without invoking the provider again.
+- **Effect absent** records forward recovery and enables a deliberate retry
+  under a new invocation ledger key and the same provider idempotency key.
+
+Atomic actions are different: provider database changes and the Workflow
+projection share one transaction. An exception rolls that transaction back, so
+the step can become retryable without claiming an unknown external outcome.
+
+Snapshot-restore and irreversible actions are rejected before dispatch unless
+the Workflow node pins the required backup or approval reference.
+
+## Workers, Triggers, And Timers
+
+Runtime workers acquire a distributed fence for each instance. Trigger
+deliveries and wait states use their own fences. Database row locks remain a
+local optimization; the fence is the authority proof across hosts. A second
+runtime cannot advance an actively owned item. After lease expiry, a new
+runtime receives a higher fence number and may resume from durable state.
+
+Linked Dataflow runs retain their own recovery boundary. If Dataflow reports an
+unknown publication outcome, Workflow displays a recovery handoff and only
+allows cancellation. It polls again after the Dataflow outcome is reconciled
+and advances only from a conclusive run descriptor.
+
+## Process Failure Versus Effect Uncertainty
+
+A process failure before dispatch is retryable after current authority and
+inputs are revalidated. A failure after non-atomic dispatch is an unresolved
+effect even when no provider response was received. Operators must not infer
+absence from a timeout, worker restart, HTTP error, or empty local result.
+
+Checkpoint evidence is hash-chained. A broken chain prevents a success commit;
+it must be investigated as an integrity incident rather than bypassed.