49 lines
2.4 KiB
Markdown
49 lines
2.4 KiB
Markdown
# Durable Dataflow Run Recovery
|
|
|
|
Dataflow uses Core's recovery ledger and distributed leases for every complete
|
|
run. The run ID, attempt number, pinned definition hash, canonical request hash,
|
|
invocation provenance, and access-context digest are recorded before execution.
|
|
Secrets and resolved credentials are never recovery evidence.
|
|
|
|
## Database-Only Runs
|
|
|
|
Runs without output publication use the `atomic` mode. Source resolution and
|
|
typed execution are recomputable and intermediate rows remain ephemeral. The
|
|
terminal `DataflowRun` projection and verified recovery checkpoint commit in
|
|
one database transaction. A stale transaction is failed by Core's fencing
|
|
contract and may be retried under the next attempt number.
|
|
|
|
## Published Output
|
|
|
|
Runs with a `datasources.publication` target use `forward_recovery`. Before
|
|
dispatch, Dataflow commits the bounded calculation result and records an output
|
|
digest, source-fingerprint digest, row count, and publication-idempotency digest.
|
|
The actual sink effect is not claimed to be atomic.
|
|
|
|
A conclusive provider response supplies publication, datasource, and
|
|
materialization references. Those references, the local `succeeded` run state,
|
|
and the verified recovery checkpoint then commit together. If dispatch may have
|
|
started but no conclusive response exists, the run and ledger become
|
|
`outcome_unknown`; automatic retry is prohibited.
|
|
|
|
## Crash And Retry Rules
|
|
|
|
- Before output dispatch, an expired forward-recovery attempt is proven absent,
|
|
closed as recovered, and may be retried with a new fenced attempt.
|
|
- At or after output dispatch, an expired attempt is not retried. The sink must
|
|
be inspected using the stable idempotency key and recorded output digest.
|
|
- A failed evidence-chain verification prevents the local success projection
|
|
from committing.
|
|
- Replaying the same run idempotency key returns the existing run, including an
|
|
unresolved run, and never performs the publication again.
|
|
- Missing optional Datasources publication capability fails before dispatch and
|
|
records that no external output was created.
|
|
|
|
## Operator View
|
|
|
|
The Dataflow run history shows the Core recovery status and an explicit warning
|
|
for unresolved runs. Ops provides the platform-wide ledger projection, but sink
|
|
reconciliation belongs to Dataflow and the owning Datasources/provider adapter.
|
|
An operator must not infer absence from a timeout, process crash, or missing
|
|
local output reference.
|