67 lines
4.0 KiB
Markdown
67 lines
4.0 KiB
Markdown
# Module Lifecycle Recovery
|
|
|
|
## Migration Revision Namespace
|
|
|
|
All enabled module migration directories are assembled into one Alembic graph. Revision IDs are therefore global across Core and every module even though each module owns a separate `migrations/versions` directory. Core validates literal revision declarations before constructing the graph and rejects duplicates with both file paths. A module must assign a new globally unique revision ID; reusing another module's ID can otherwise make Alembic treat an unrelated schema change as already applied or report an ancestor/head overlap.
|
|
|
|
When correcting a collision that has already reached a database, first verify the schema objects that identify which migration actually ran. Rename the unapplied migration, or transactionally translate the corresponding `alembic_version` row when the applied owner is unambiguous. Never add both colliding IDs as heads or blindly stamp the database.
|
|
|
|
Package changes and live module-graph changes use Core's durable recovery
|
|
ledger. The local `install.lock` still prevents duplicate work in one runtime
|
|
directory; the database lease `core:module-lifecycle:deployment` is the
|
|
deployment-wide authority across API, installer, worker, and scheduler nodes.
|
|
|
|
## Declared Boundaries
|
|
|
|
| Operation | Recovery mode | Completion condition |
|
|
| --- | --- | --- |
|
|
| `module-lifecycle.pre-migration` | compensation | package, WebUI, manifest, and desired-graph evidence match |
|
|
| `module-lifecycle.post-migration` | forward recovery | migration tasks, manifests, desired graph, restart, and health are verified |
|
|
| `module-retirement.destroy-data` | snapshot restore | a hashed, restore-checked backup exists and retirement state is verified |
|
|
| `module-runtime.apply-graph` | compensation | hooks, capability contexts, active graph, and workflow contributions match |
|
|
|
|
The installer prepares the recovery operation before it captures the database
|
|
snapshot. A full database restore therefore retains the prepared operation and
|
|
its fence instead of erasing the fact that a mutation was attempted. Backup
|
|
artifacts are hashed and sized before any package, migration, or retirement
|
|
effect starts.
|
|
|
|
Every command boundary records the command source and canonical hashes of the
|
|
redacted command/result records. Credentials, database URLs, command output,
|
|
and package-registry secrets are never copied into recovery evidence.
|
|
|
|
## Failure And Retry Rules
|
|
|
|
- A conclusive failure before effects is terminal `failed`.
|
|
- A command or compensatable effect that started but did not complete is
|
|
`recovery_required`.
|
|
- A lost or unexpected outcome after a migration/external boundary is
|
|
`outcome_unknown`.
|
|
- A verified package/database rollback becomes `recovered`.
|
|
- A supervised install becomes `succeeded` only after restart and all configured
|
|
health probes succeed.
|
|
|
|
An unresolved lifecycle operation blocks every later lifecycle mutation on the
|
|
same deployment fence, even after its execution lease is released. Operators
|
|
must inspect the checkpoint chain and run record, restore or complete the
|
|
declared recovery path, and explicitly reconcile the operation. A new install
|
|
must not be used as an implicit retry.
|
|
|
|
Live graph changes use the same fence. A non-migrating hook or registry failure
|
|
restores the prior in-process graph and records verified compensation. A failure
|
|
after migrations begin remains unresolved because restoring the process-local
|
|
registry does not reverse database schema effects.
|
|
|
|
## Operator Evidence
|
|
|
|
The installer run record contains the recovery operation id, mode, plan hash,
|
|
and current lifecycle status. The Ops recovery view is authoritative for the
|
|
durable state and evidence-chain result. Keep both the run directory and the
|
|
state-service backup evidence until the operation is terminal and the normal
|
|
retention policy permits removal.
|
|
|
|
Run the module installer rollback drill and recovery-runtime test matrix before
|
|
enabling lifecycle mutation in a new deployment. Shared-state deployments must
|
|
still use immutable release images; the ledger does not make in-place package
|
|
mutation across replicas safe.
|