90 lines
5.0 KiB
Markdown
90 lines
5.0 KiB
Markdown
# Recovery Ledger Adoption
|
|
|
|
The Core recovery ledger is a platform primitive, not automatic protection for
|
|
module-owned effects. The canonical, machine-checked inventory is
|
|
[`recovery-operation-inventory.json`](../recovery-operation-inventory.json).
|
|
|
|
## Classification Rules
|
|
|
|
- Use `atomic` only when every mutation commits in one database transaction and
|
|
no external effect occurs.
|
|
- Use `compensation` when every completed effect has a bounded, verifiable
|
|
inverse action. A best-effort delete is not proof of compensation.
|
|
- Use `snapshot_restore` only with fresh, signed backup evidence that covers all
|
|
affected state services at one recovery point.
|
|
- Use `forward_recovery` for provider acceptance, queue publication, cursor
|
|
advancement, and other effects that may be resumable but cannot safely be
|
|
undone.
|
|
- Use `irreversible` for approved purge or destruction where no automated
|
|
recovery is claimed.
|
|
|
|
One feature may cross more than one boundary. Module installation is
|
|
compensatable before schema migration, forward-only after migration starts, and
|
|
snapshot-restorable for an approved destructive retirement. Mail submission is
|
|
forward recovery because losing the response after provider acceptance must not
|
|
cause an automatic resend.
|
|
|
|
## Adoption Order
|
|
|
|
1. Campaign build is the reference implementation for a database plus object
|
|
storage operation. Its operation reserves a build-specific object prefix,
|
|
persists request and precondition evidence before writes, records the final
|
|
object manifest, and verifies database/object state before success.
|
|
2. Campaign delivery and Mail provider effects adopt outcome-unknown semantics
|
|
without weakening their existing provider-specific idempotency records.
|
|
3. Files applies the same contract to uploads, purge, integrity reconciliation,
|
|
and writable connector synchronization.
|
|
4. Connectors, Dataflow, and Workflow Engine consume the contract at their
|
|
registry/capability boundaries so optional providers remain optional.
|
|
5. Core module lifecycle uses the ledger in addition to, not instead of, signed
|
|
deployment and backup evidence.
|
|
|
|
Every fenced operation uses a process incarnation and distributed lease. A
|
|
stale process cannot append a checkpoint or report success. An expired operation
|
|
is claimed for recovery through an explicit takeover that preserves the prior
|
|
fence in the checkpoint chain; it is never resumed as a normal retry.
|
|
|
|
Connectors read-only sanctions and feed acquisitions are adopted: source
|
|
revision/cursor and dry-run evidence are recorded before provider I/O, while
|
|
the immutable snapshot and terminal checkpoint commit atomically. The generic
|
|
external-mutation contract is conformance-tested but remains `planned` until a
|
|
production connector actually publishes, updates, or deletes provider state.
|
|
|
|
Dataflow runs are adopted. Database-only execution uses one atomic terminal
|
|
commit for the run projection and recovery checkpoint. Output publication uses
|
|
forward recovery: source and output digests are checkpointed before dispatch,
|
|
a conclusive provider result commits with the run projection, and an expired
|
|
or failed attempt after dispatch becomes `outcome_unknown`. A stale attempt may
|
|
be retried only when its durable boundary proves dispatch had not started.
|
|
|
|
Workflow Engine is adopted at both declared boundaries. Instance workers,
|
|
trigger deliveries, and timer resumptions use process-bound distributed fences.
|
|
Every module-action invocation records the pinned definition, input, preview,
|
|
authority, provider-idempotency, and action-contract hashes before dispatch.
|
|
Conclusive results commit with the Workflow projection. A lost acknowledgement,
|
|
invalid result, or unannounced non-atomic effect becomes `outcome_unknown` and
|
|
cannot be retried until evidence confirms either that the effect occurred or is
|
|
absent. Linked Dataflow uncertainty blocks the Workflow without duplicating
|
|
Dataflow's recovery authority.
|
|
|
|
Core module lifecycle is adopted at four boundaries. Installer recovery is
|
|
prepared before snapshots so a full database restore preserves the attempted
|
|
operation. Pre-migration package changes use compensation, migrated changes use
|
|
forward recovery, destructive retirement requires a hashed and restore-checked
|
|
snapshot, and live graph changes restore the prior registry when no migration
|
|
ran. A deployment-wide database fence serializes these effects; any unresolved
|
|
predecessor blocks a differently keyed retry until explicit reconciliation.
|
|
Supervised installs become successful only after restart and health evidence is
|
|
recorded.
|
|
|
|
## Operator Contract
|
|
|
|
Ops lists non-terminal and manual-intervention operations. Operators must verify
|
|
the checkpoint chain before trusting evidence, distinguish `outcome_unknown`
|
|
from rejection, and use the owning module's documented reconciliation action.
|
|
No evidence payload may contain credentials or resolved secrets.
|
|
|
|
The parent adoption issue remains open until all inventory rows are adopted and
|
|
the module matrix proves crash, retry, stale-fence, tamper, and optional-module
|
|
behavior for each consequential path.
|