Fence and reconcile Dataflow runs

2026-08-03 06:09:53 +02:00
parent e1c092ece7
commit 3fa7a29f48
13 changed files with 1314 additions and 44 deletions
@@ -0,0 +1,48 @@
+# Durable Dataflow Run Recovery
+
+Dataflow uses Core's recovery ledger and distributed leases for every complete
+run. The run ID, attempt number, pinned definition hash, canonical request hash,
+invocation provenance, and access-context digest are recorded before execution.
+Secrets and resolved credentials are never recovery evidence.
+
+## Database-Only Runs
+
+Runs without output publication use the `atomic` mode. Source resolution and
+typed execution are recomputable and intermediate rows remain ephemeral. The
+terminal `DataflowRun` projection and verified recovery checkpoint commit in
+one database transaction. A stale transaction is failed by Core's fencing
+contract and may be retried under the next attempt number.
+
+## Published Output
+
+Runs with a `datasources.publication` target use `forward_recovery`. Before
+dispatch, Dataflow commits the bounded calculation result and records an output
+digest, source-fingerprint digest, row count, and publication-idempotency digest.
+The actual sink effect is not claimed to be atomic.
+
+A conclusive provider response supplies publication, datasource, and
+materialization references. Those references, the local `succeeded` run state,
+and the verified recovery checkpoint then commit together. If dispatch may have
+started but no conclusive response exists, the run and ledger become
+`outcome_unknown`; automatic retry is prohibited.
+
+## Crash And Retry Rules
+
+- Before output dispatch, an expired forward-recovery attempt is proven absent,
+  closed as recovered, and may be retried with a new fenced attempt.
+- At or after output dispatch, an expired attempt is not retried. The sink must
+  be inspected using the stable idempotency key and recorded output digest.
+- A failed evidence-chain verification prevents the local success projection
+  from committing.
+- Replaying the same run idempotency key returns the existing run, including an
+  unresolved run, and never performs the publication again.
+- Missing optional Datasources publication capability fails before dispatch and
+  records that no external output was created.
+
+## Operator View
+
+The Dataflow run history shows the Core recovery status and an explicit warning
+for unresolved runs. Ops provides the platform-wide ledger projection, but sink
+reconciliation belongs to Dataflow and the owning Datasources/provider adapter.
+An operator must not infer absence from a timeout, process crash, or missing
+local output reference.