3.1 KiB
Module Lifecycle Recovery
Package changes and live module-graph changes use Core's durable recovery
ledger. The local install.lock still prevents duplicate work in one runtime
directory; the database lease core:module-lifecycle:deployment is the
deployment-wide authority across API, installer, worker, and scheduler nodes.
Declared Boundaries
| Operation | Recovery mode | Completion condition |
|---|---|---|
module-lifecycle.pre-migration |
compensation | package, WebUI, manifest, and desired-graph evidence match |
module-lifecycle.post-migration |
forward recovery | migration tasks, manifests, desired graph, restart, and health are verified |
module-retirement.destroy-data |
snapshot restore | a hashed, restore-checked backup exists and retirement state is verified |
module-runtime.apply-graph |
compensation | hooks, capability contexts, active graph, and workflow contributions match |
The installer prepares the recovery operation before it captures the database snapshot. A full database restore therefore retains the prepared operation and its fence instead of erasing the fact that a mutation was attempted. Backup artifacts are hashed and sized before any package, migration, or retirement effect starts.
Every command boundary records the command source and canonical hashes of the redacted command/result records. Credentials, database URLs, command output, and package-registry secrets are never copied into recovery evidence.
Failure And Retry Rules
- A conclusive failure before effects is terminal
failed. - A command or compensatable effect that started but did not complete is
recovery_required. - A lost or unexpected outcome after a migration/external boundary is
outcome_unknown. - A verified package/database rollback becomes
recovered. - A supervised install becomes
succeededonly after restart and all configured health probes succeed.
An unresolved lifecycle operation blocks every later lifecycle mutation on the same deployment fence, even after its execution lease is released. Operators must inspect the checkpoint chain and run record, restore or complete the declared recovery path, and explicitly reconcile the operation. A new install must not be used as an implicit retry.
Live graph changes use the same fence. A non-migrating hook or registry failure restores the prior in-process graph and records verified compensation. A failure after migrations begin remains unresolved because restoring the process-local registry does not reverse database schema effects.
Operator Evidence
The installer run record contains the recovery operation id, mode, plan hash, and current lifecycle status. The Ops recovery view is authoritative for the durable state and evidence-chain result. Keep both the run directory and the state-service backup evidence until the operation is terminal and the normal retention policy permits removal.
Run the module installer rollback drill and recovery-runtime test matrix before enabling lifecycle mutation in a new deployment. Shared-state deployments must still use immutable release images; the ledger does not make in-place package mutation across replicas safe.