govoplan/docs/operations/RECOVERY_LEDGER_ADOPTION.md

Recovery Ledger Adoption

The Core recovery ledger is a platform primitive, not automatic protection for module-owned effects. The canonical, machine-checked inventory is recovery-operation-inventory.json.

Classification Rules

• Use atomic only when every mutation commits in one database transaction and no external effect occurs.
• Use compensation when every completed effect has a bounded, verifiable inverse action. A best-effort delete is not proof of compensation.
• Use snapshot_restore only with fresh, signed backup evidence that covers all affected state services at one recovery point.
• Use forward_recovery for provider acceptance, queue publication, cursor advancement, and other effects that may be resumable but cannot safely be undone.
• Use irreversible for approved purge or destruction where no automated recovery is claimed.

One feature may cross more than one boundary. Module installation is compensatable before schema migration, forward-only after migration starts, and snapshot-restorable for an approved destructive retirement. Mail submission is forward recovery because losing the response after provider acceptance must not cause an automatic resend.

Adoption Order

1. Campaign build is the reference implementation for a database plus object storage operation. Its operation reserves a build-specific object prefix, persists request and precondition evidence before writes, records the final object manifest, and verifies database/object state before success.
2. Campaign delivery and Mail provider effects adopt outcome-unknown semantics without weakening their existing provider-specific idempotency records.
3. Files applies the same contract to uploads, purge, integrity reconciliation, and writable connector synchronization.
4. Connectors, Dataflow, and Workflow Engine consume the contract at their registry/capability boundaries so optional providers remain optional.
5. Core module lifecycle uses the ledger in addition to, not instead of, signed deployment and backup evidence.

Every fenced operation uses a process incarnation and distributed lease. A stale process cannot append a checkpoint or report success. An expired operation is claimed for recovery through an explicit takeover that preserves the prior fence in the checkpoint chain; it is never resumed as a normal retry.

Connectors read-only sanctions and feed acquisitions are adopted: source revision/cursor and dry-run evidence are recorded before provider I/O, while the immutable snapshot and terminal checkpoint commit atomically. The generic external-mutation contract is conformance-tested but remains planned until a production connector actually publishes, updates, or deletes provider state.

Dataflow runs are adopted. Database-only execution uses one atomic terminal commit for the run projection and recovery checkpoint. Output publication uses forward recovery: source and output digests are checkpointed before dispatch, a conclusive provider result commits with the run projection, and an expired or failed attempt after dispatch becomes outcome_unknown. A stale attempt may be retried only when its durable boundary proves dispatch had not started.

Workflow Engine is adopted at both declared boundaries. Instance workers, trigger deliveries, and timer resumptions use process-bound distributed fences. Every module-action invocation records the pinned definition, input, preview, authority, provider-idempotency, and action-contract hashes before dispatch. Conclusive results commit with the Workflow projection. A lost acknowledgement, invalid result, or unannounced non-atomic effect becomes outcome_unknown and cannot be retried until evidence confirms either that the effect occurred or is absent. Linked Dataflow uncertainty blocks the Workflow without duplicating Dataflow's recovery authority.

Core module lifecycle is adopted at four boundaries. Installer recovery is prepared before snapshots so a full database restore preserves the attempted operation. Pre-migration package changes use compensation, migrated changes use forward recovery, destructive retirement requires a hashed and restore-checked snapshot, and live graph changes restore the prior registry when no migration ran. A deployment-wide database fence serializes these effects; any unresolved predecessor blocks a differently keyed retry until explicit reconciliation. Supervised installs become successful only after restart and health evidence is recorded.

Operator Contract

Ops lists non-terminal and manual-intervention operations. Operators must verify the checkpoint chain before trusting evidence, distinguish outcome_unknown from rejection, and use the owning module's documented reconciliation action. No evidence payload may contain credentials or resolved secrets.

The parent adoption issue remains open until all inventory rows are adopted and the module matrix proves crash, retry, stale-fence, tamper, and optional-module behavior for each consequential path.