# Durable Dataflow Run Recovery

Dataflow uses Core's recovery ledger and distributed leases for every complete
run. The run ID, attempt number, pinned definition hash, canonical request hash,
invocation provenance, and access-context digest are recorded before execution.
Secrets and resolved credentials are never recovery evidence.

## Database-Only Runs

Runs without output publication use the `atomic` mode. Source resolution and
typed execution are recomputable and intermediate rows remain ephemeral. The
terminal `DataflowRun` projection and verified recovery checkpoint commit in
one database transaction. A stale transaction is failed by Core's fencing
contract and may be retried under the next attempt number.

## Published Output

Runs with a `datasources.publication` target use `forward_recovery`. Before
dispatch, Dataflow commits the bounded calculation result and records an output
digest, source-fingerprint digest, row count, and publication-idempotency digest.
The actual sink effect is not claimed to be atomic.

A conclusive provider response supplies publication, datasource, and
materialization references. Those references, the local `succeeded` run state,
and the verified recovery checkpoint then commit together. If dispatch may have
started but no conclusive response exists, the run and ledger become
`outcome_unknown`; automatic retry is prohibited.

## Crash And Retry Rules

- Before output dispatch, an expired forward-recovery attempt is proven absent,
  closed as recovered, and may be retried with a new fenced attempt.
- At or after output dispatch, an expired attempt is not retried. The sink must
  be inspected using the stable idempotency key and recorded output digest.
- A failed evidence-chain verification prevents the local success projection
  from committing.
- Replaying the same run idempotency key returns the existing run, including an
  unresolved run, and never performs the publication again.
- Missing optional Datasources publication capability fails before dispatch and
  records that no external output was created.

## Operator View

The Dataflow run history shows the Core recovery status and an explicit warning
for unresolved runs. Ops provides the platform-wide ledger projection, but sink
reconciliation belongs to Dataflow and the owning Datasources/provider adapter.
An operator must not infer absence from a timeout, process crash, or missing
local output reference.