Fence and reconcile module lifecycle effects

2026-08-03 07:02:07 +02:00
parent b962f6756e
commit 729b84d3af
12 changed files with 1166 additions and 34 deletions
@@ -0,0 +1,60 @@
+# Module Lifecycle Recovery
+
+Package changes and live module-graph changes use Core's durable recovery
+ledger. The local `install.lock` still prevents duplicate work in one runtime
+directory; the database lease `core:module-lifecycle:deployment` is the
+deployment-wide authority across API, installer, worker, and scheduler nodes.
+
+## Declared Boundaries
+
+| Operation | Recovery mode | Completion condition |
+| --- | --- | --- |
+| `module-lifecycle.pre-migration` | compensation | package, WebUI, manifest, and desired-graph evidence match |
+| `module-lifecycle.post-migration` | forward recovery | migration tasks, manifests, desired graph, restart, and health are verified |
+| `module-retirement.destroy-data` | snapshot restore | a hashed, restore-checked backup exists and retirement state is verified |
+| `module-runtime.apply-graph` | compensation | hooks, capability contexts, active graph, and workflow contributions match |
+
+The installer prepares the recovery operation before it captures the database
+snapshot. A full database restore therefore retains the prepared operation and
+its fence instead of erasing the fact that a mutation was attempted. Backup
+artifacts are hashed and sized before any package, migration, or retirement
+effect starts.
+
+Every command boundary records the command source and canonical hashes of the
+redacted command/result records. Credentials, database URLs, command output,
+and package-registry secrets are never copied into recovery evidence.
+
+## Failure And Retry Rules
+
+- A conclusive failure before effects is terminal `failed`.
+- A command or compensatable effect that started but did not complete is
+  `recovery_required`.
+- A lost or unexpected outcome after a migration/external boundary is
+  `outcome_unknown`.
+- A verified package/database rollback becomes `recovered`.
+- A supervised install becomes `succeeded` only after restart and all configured
+  health probes succeed.
+
+An unresolved lifecycle operation blocks every later lifecycle mutation on the
+same deployment fence, even after its execution lease is released. Operators
+must inspect the checkpoint chain and run record, restore or complete the
+declared recovery path, and explicitly reconcile the operation. A new install
+must not be used as an implicit retry.
+
+Live graph changes use the same fence. A non-migrating hook or registry failure
+restores the prior in-process graph and records verified compensation. A failure
+after migrations begin remains unresolved because restoring the process-local
+registry does not reverse database schema effects.
+
+## Operator Evidence
+
+The installer run record contains the recovery operation id, mode, plan hash,
+and current lifecycle status. The Ops recovery view is authoritative for the
+durable state and evidence-chain result. Keep both the run directory and the
+state-service backup evidence until the operation is terminal and the normal
+retention policy permits removal.
+
+Run the module installer rollback drill and recovery-runtime test matrix before
+enabling lifecycle mutation in a new deployment. Shared-state deployments must
+still use immutable release images; the ledger does not make in-place package
+mutation across replicas safe.