# Module Lifecycle Recovery

Package changes and live module-graph changes use Core's durable recovery
ledger. The local `install.lock` still prevents duplicate work in one runtime
directory; the database lease `core:module-lifecycle:deployment` is the
deployment-wide authority across API, installer, worker, and scheduler nodes.

## Declared Boundaries

| Operation | Recovery mode | Completion condition |
| --- | --- | --- |
| `module-lifecycle.pre-migration` | compensation | package, WebUI, manifest, and desired-graph evidence match |
| `module-lifecycle.post-migration` | forward recovery | migration tasks, manifests, desired graph, restart, and health are verified |
| `module-retirement.destroy-data` | snapshot restore | a hashed, restore-checked backup exists and retirement state is verified |
| `module-runtime.apply-graph` | compensation | hooks, capability contexts, active graph, and workflow contributions match |

The installer prepares the recovery operation before it captures the database
snapshot. A full database restore therefore retains the prepared operation and
its fence instead of erasing the fact that a mutation was attempted. Backup
artifacts are hashed and sized before any package, migration, or retirement
effect starts.

Every command boundary records the command source and canonical hashes of the
redacted command/result records. Credentials, database URLs, command output,
and package-registry secrets are never copied into recovery evidence.

## Failure And Retry Rules

- A conclusive failure before effects is terminal `failed`.
- A command or compensatable effect that started but did not complete is
  `recovery_required`.
- A lost or unexpected outcome after a migration/external boundary is
  `outcome_unknown`.
- A verified package/database rollback becomes `recovered`.
- A supervised install becomes `succeeded` only after restart and all configured
  health probes succeed.

An unresolved lifecycle operation blocks every later lifecycle mutation on the
same deployment fence, even after its execution lease is released. Operators
must inspect the checkpoint chain and run record, restore or complete the
declared recovery path, and explicitly reconcile the operation. A new install
must not be used as an implicit retry.

Live graph changes use the same fence. A non-migrating hook or registry failure
restores the prior in-process graph and records verified compensation. A failure
after migrations begin remains unresolved because restoring the process-local
registry does not reverse database schema effects.

## Operator Evidence

The installer run record contains the recovery operation id, mode, plan hash,
and current lifecycle status. The Ops recovery view is authoritative for the
durable state and evidence-chain result. Keep both the run directory and the
state-service backup evidence until the operation is terminal and the normal
retention policy permits removal.

Run the module installer rollback drill and recovery-runtime test matrix before
enabling lifecycle mutation in a new deployment. Shared-state deployments must
still use immutable release images; the ledger does not make in-place package
mutation across replicas safe.