4.0 KiB
Module Lifecycle Recovery
Migration Revision Namespace
All enabled module migration directories are assembled into one Alembic graph. Revision IDs are therefore global across Core and every module even though each module owns a separate migrations/versions directory. Core validates literal revision declarations before constructing the graph and rejects duplicates with both file paths. A module must assign a new globally unique revision ID; reusing another module's ID can otherwise make Alembic treat an unrelated schema change as already applied or report an ancestor/head overlap.
When correcting a collision that has already reached a database, first verify the schema objects that identify which migration actually ran. Rename the unapplied migration, or transactionally translate the corresponding alembic_version row when the applied owner is unambiguous. Never add both colliding IDs as heads or blindly stamp the database.
Package changes and live module-graph changes use Core's durable recovery
ledger. The local install.lock still prevents duplicate work in one runtime
directory; the database lease core:module-lifecycle:deployment is the
deployment-wide authority across API, installer, worker, and scheduler nodes.
Declared Boundaries
| Operation | Recovery mode | Completion condition |
|---|---|---|
module-lifecycle.pre-migration |
compensation | package, WebUI, manifest, and desired-graph evidence match |
module-lifecycle.post-migration |
forward recovery | migration tasks, manifests, desired graph, restart, and health are verified |
module-retirement.destroy-data |
snapshot restore | a hashed, restore-checked backup exists and retirement state is verified |
module-runtime.apply-graph |
compensation | hooks, capability contexts, active graph, and workflow contributions match |
The installer prepares the recovery operation before it captures the database snapshot. A full database restore therefore retains the prepared operation and its fence instead of erasing the fact that a mutation was attempted. Backup artifacts are hashed and sized before any package, migration, or retirement effect starts.
Every command boundary records the command source and canonical hashes of the redacted command/result records. Credentials, database URLs, command output, and package-registry secrets are never copied into recovery evidence.
Failure And Retry Rules
- A conclusive failure before effects is terminal
failed. - A command or compensatable effect that started but did not complete is
recovery_required. - A lost or unexpected outcome after a migration/external boundary is
outcome_unknown. - A verified package/database rollback becomes
recovered. - A supervised install becomes
succeededonly after restart and all configured health probes succeed.
An unresolved lifecycle operation blocks every later lifecycle mutation on the same deployment fence, even after its execution lease is released. Operators must inspect the checkpoint chain and run record, restore or complete the declared recovery path, and explicitly reconcile the operation. A new install must not be used as an implicit retry.
Live graph changes use the same fence. A non-migrating hook or registry failure restores the prior in-process graph and records verified compensation. A failure after migrations begin remains unresolved because restoring the process-local registry does not reverse database schema effects.
Operator Evidence
The installer run record contains the recovery operation id, mode, plan hash, and current lifecycle status. The Ops recovery view is authoritative for the durable state and evidence-chain result. Keep both the run directory and the state-service backup evidence until the operation is terminal and the normal retention policy permits removal.
Run the module installer rollback drill and recovery-runtime test matrix before enabling lifecycle mutation in a new deployment. Shared-state deployments must still use immutable release images; the ledger does not make in-place package mutation across replicas safe.