148 lines
6.5 KiB
Markdown
148 lines
6.5 KiB
Markdown
# Recovery And Rollback Guarantees
|
|
|
|
## Principle
|
|
|
|
GovOPlaN must prove recovery claims with durable state recorded before and
|
|
after side effects. A failed operation is not automatically rolled back merely
|
|
because the previous application image still exists. Database schema and
|
|
external effects may make release rollback unsafe.
|
|
|
|
Core therefore distinguishes five recovery modes:
|
|
|
|
| Mode | Meaning |
|
|
| --- | --- |
|
|
| `atomic` | One database transaction either commits or rolls back. No external effect is claimed. |
|
|
| `compensation` | Durable evidence identifies explicit inverse actions for completed effects. |
|
|
| `snapshot_restore` | A separately verified backup reference and restore procedure exist. |
|
|
| `forward_recovery` | Repair or resume the current version; reverting code/configuration is not claimed safe. |
|
|
| `irreversible` | No automated recovery is claimed and an approval reference is mandatory. |
|
|
|
|
An operation plan must include verification steps. Compensation requires named
|
|
compensation steps, snapshot restore requires a verified backup reference,
|
|
forward recovery requires repair steps, and irreversible work requires explicit
|
|
approval.
|
|
|
|
## Core Recovery Ledger
|
|
|
|
Core stores recovery operations and append-only, hash-chained checkpoints in
|
|
PostgreSQL. The contract provides:
|
|
|
|
- installation/module/resource identity;
|
|
- an idempotency key bound to a canonical request hash;
|
|
- recovery mode, preconditions, verification steps, and references;
|
|
- optional runtime lease holder and fencing token;
|
|
- explicit planned, prepared, running, recovery-required, recovering,
|
|
succeeded, recovered, failed, outcome-unknown, and manual-intervention states;
|
|
- an evidence-chain head and sequence count;
|
|
- rejection of plaintext secrets in metadata or evidence.
|
|
|
|
Preparation cannot succeed without durable precondition evidence. A non-atomic
|
|
operation cannot hide a partial effect by transitioning directly from running
|
|
to failed. Success and recovery require explicit verification evidence with at
|
|
least one check. The ledger verifies its hash chain before evidence is trusted.
|
|
|
|
This is a platform contract, not an assertion that every existing module
|
|
operation has adopted it. Module operations with external or multi-resource
|
|
effects must be migrated to the ledger before claiming these guarantees.
|
|
|
|
## Deployment Journal
|
|
|
|
Every `govoplan-deploy apply` begins an operation journal before it pulls images
|
|
or mutates runtime state. The private installation directory records:
|
|
|
|
```text
|
|
operations/<operation-id>/operation.json
|
|
operations/<operation-id>/before/
|
|
applied-state/
|
|
```
|
|
|
|
Each stage is hash-chained. The previous applied bundle is copied with per-file
|
|
SHA-256 evidence. Applied state is replaced atomically after health verification;
|
|
an interrupted replacement restores its previous directory.
|
|
New journals also bind the complete desired deployment plan, snapshot
|
|
availability, failure summary, recovery mode, and terminal status to the
|
|
evidence chain. Recovery verifies every snapshot entry and checksum before it
|
|
changes any live bundle file, then replaces each live file atomically. A crash
|
|
between file replacements is recoverable by rerunning the same idempotent
|
|
recovery command under the deployment lock.
|
|
|
|
Inspect operations:
|
|
|
|
```sh
|
|
python tools/deployment/govoplan-deploy.py operations \
|
|
--directory /srv/govoplan/default
|
|
```
|
|
|
|
Recover the latest failed operation, or provide its identifier:
|
|
|
|
```sh
|
|
python tools/deployment/govoplan-deploy.py recover \
|
|
--directory /srv/govoplan/default \
|
|
--operation-id 20260801T120000Z-1234abcd
|
|
```
|
|
|
|
Add `--apply` only after reviewing the reported action.
|
|
|
|
## Migration Boundary
|
|
|
|
Before database migration starts, a failed deployment with a verified prior
|
|
applied snapshot may restore its prior release/configuration bundle and
|
|
reconcile that desired state.
|
|
|
|
As soon as migration starts, the journal permanently changes to
|
|
`forward_recovery`. It will not restore old application configuration because
|
|
old code may not understand the new schema. Recovery then means one of:
|
|
|
|
1. fix and re-run the current release;
|
|
2. deploy a newer compatible repair release;
|
|
3. restore a separately verified, coordinated database/object/key backup and
|
|
then deploy the matching release.
|
|
|
|
The deployment tool does not create or validate that database backup. A
|
|
`backup-required` annotation on the Kubernetes migration Job is an operator
|
|
gate, not backup evidence. Production automation must provide a backup hook or
|
|
external backup controller whose artifact, timestamp, scope, encryption key,
|
|
and restore test can be referenced from the recovery record.
|
|
|
|
## Scaled Nodes
|
|
|
|
Recovery actions must be safe across replicas:
|
|
|
|
- drain affected API and worker nodes before incompatible changes;
|
|
- use the deployment-wide PostgreSQL advisory lock for schema migration;
|
|
- use distributed leases and fencing tokens for singleton or externally
|
|
visible effects;
|
|
- use idempotency keys for retried commands and jobs;
|
|
- retain shared object keys and database references until deletion succeeds;
|
|
- classify uncertain external outcomes instead of retrying blindly;
|
|
- verify the exact software/module composition after replacement.
|
|
|
|
Campaign generated-message objects now follow this model: object writes are
|
|
compensated when a build fails before database commit, workers verify stored
|
|
size and digest before delivery, and retention keeps the database reference
|
|
when storage deletion fails. A hard process loss between object creation and
|
|
database commit can still leave an orphan object; an inventory reconciler is a
|
|
separate operational slice and must use the build-specific object prefix.
|
|
|
|
## Required Drills
|
|
|
|
Record evidence for at least these scenarios before production acceptance:
|
|
|
|
1. Kill an API replica and verify traffic continues without session loss.
|
|
2. Drain and replace a worker while work is queued and while one job is active.
|
|
3. Start two migration jobs and verify only one mutates schema.
|
|
4. Kill the fenced scheduler and verify one replacement acquires a higher
|
|
fencing token.
|
|
5. Fail deployment before migration and restore the prior applied bundle.
|
|
6. Fail deployment after migration and verify old configuration is not
|
|
restored.
|
|
7. Restore PostgreSQL, object storage, and encryption keys to one coordinated
|
|
recovery point and verify representative object hashes.
|
|
8. Interrupt object storage during Campaign build and retention and verify
|
|
compensation/reference-preservation behavior.
|
|
9. Tamper with a deployment or Core recovery checkpoint and verify chain
|
|
validation rejects it.
|
|
|
|
No runbook, status badge, or green health endpoint substitutes for a dated,
|
|
repeatable restore drill against the actual deployment topology.
|