134 lines
6.4 KiB
Markdown
134 lines
6.4 KiB
Markdown
# Backup And Restore Evidence
|
|
|
|
## Boundary
|
|
|
|
`govoplan-deploy` verifies backup and restore evidence; it does not receive
|
|
database, object-store, KMS, or orchestrator administration credentials and it
|
|
does not create the backup. A provider-owned backup controller creates one
|
|
coordinated recovery point, a separate drill runner restores it into an
|
|
isolated target, and an evidence authority signs the resulting receipt.
|
|
|
|
The application containers receive only a sanitized projection: evidence,
|
|
recovery-point and drill identifiers, hashes, timestamps, component count, and
|
|
measured RPO/RTO. Artifact locations, provider credentials, encryption-key
|
|
references, the public trust keyring, and private signing keys remain in the
|
|
deployment/evidence boundary.
|
|
|
|
The machine-readable contracts are:
|
|
|
|
- [`backup-evidence.schema.json`](../backup-evidence.schema.json);
|
|
- [`backup-evidence-keyring.schema.json`](../backup-evidence-keyring.schema.json).
|
|
|
|
One evidence document is bound to the installation id, deployment profile,
|
|
topology subject, exact signed release manifest, image digests, and composition
|
|
digest. It covers PostgreSQL, objects, protected configuration, and recoverable
|
|
key custody at one recovery point. It contains references, never key material.
|
|
|
|
## Production Sequence
|
|
|
|
1. Establish the provider snapshot, application quiesce, or transaction
|
|
boundary and retain a hash of its fencing token.
|
|
2. Capture PostgreSQL, object storage, protected deployment configuration, and
|
|
key-custody state within five minutes of that recovery point.
|
|
3. Restore all four components into a target isolated from production write
|
|
endpoints and production queues.
|
|
4. Start the exact immutable release named in the evidence, verify migration
|
|
heads, verify a deterministic manifest of representative object hashes, and
|
|
execute the documented semantic journey checks.
|
|
5. Record actual data loss and elapsed recovery as measured RPO and RTO. A
|
|
measured RPO above the declared objective invalidates the evidence.
|
|
6. Sign the canonical receipt using an evidence-authority Ed25519 key held
|
|
outside the application and deployment host. During key rotation, include
|
|
both accepted signatures.
|
|
7. Transfer the evidence SHA-256 through an independent approved channel, then
|
|
verify and adopt it on the deployment host.
|
|
|
|
Provider automation can sign and validate an unsigned receipt with:
|
|
|
|
```sh
|
|
python tools/deployment/sign-backup-evidence.py \
|
|
--input unsigned-backup-evidence.json \
|
|
--output backup-evidence.json \
|
|
--trusted-keyring backup-evidence-keyring.json \
|
|
--signing-key backup-authority-2026=/run/keys/backup-authority.pem
|
|
```
|
|
|
|
The private key file must be owner-only. The tool refuses an unexpected key
|
|
type, an inactive/untrusted signer, malformed or partial evidence, stale
|
|
recovery points, failed drill checks, mismatched releases, and non-canonical
|
|
output.
|
|
|
|
Adopt the result using the independently obtained digest:
|
|
|
|
```sh
|
|
python3 govoplan-deploy.pyz verify-backup \
|
|
--directory /srv/govoplan/default \
|
|
--evidence ./backup-evidence.json \
|
|
--evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \
|
|
--trusted-keyring ./backup-evidence-keyring.json \
|
|
--adopt
|
|
```
|
|
|
|
Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every
|
|
self-hosted release identity change is conservatively treated as a migration
|
|
boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when
|
|
fresh evidence for the previously applied immutable release is unavailable.
|
|
Compose verifies once before changing runtime state and again after API/worker
|
|
quiescing immediately before migration. The exported Kubernetes migration Job
|
|
is generated only after verification and is annotated with the sanitized
|
|
evidence digest, recovery-point id, and drill id.
|
|
|
|
## Provider Runbooks
|
|
|
|
### PostgreSQL
|
|
|
|
Use a managed transaction-consistent snapshot or a base backup plus retained
|
|
WAL sufficient to reconstruct the declared point. Record the provider,
|
|
protected artifact reference and digest, snapshot identity, and PostgreSQL LSN.
|
|
The restore drill must connect only to the isolated database and must compare
|
|
the resulting migration-head digest with the release expectation.
|
|
|
|
### Object Storage
|
|
|
|
Use provider snapshots/versioning or an immutable object copy. Build a sorted
|
|
manifest containing object key, version, size, and content digest, then record
|
|
its digest, object count, total bytes, provider version identity, and protected
|
|
artifact reference. Verify representative objects from every owning module
|
|
after restore. Single-node managed Garage is persistent but not highly
|
|
available; copy its coordinated recovery material to an independent failure
|
|
domain.
|
|
|
|
### Configuration And Key Custody
|
|
|
|
Back up the private installation bundle and external secret-manager bindings as
|
|
an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault
|
|
state, record the provider keyset reference, version, and a successful
|
|
recoverability assertion. Never put a key, recovery share, token, password, or
|
|
credential-bearing URL in evidence. The isolated drill must prove that the
|
|
restored release can decrypt representative protected content without
|
|
exporting the key material into the report.
|
|
|
|
## Ownership And Retention
|
|
|
|
The deployment owner approves the RPO/RTO objectives. State-service owners
|
|
operate backup capture and restoration. Module owners define representative
|
|
objects and semantic checks. Security owns evidence-authority keys and
|
|
revocation. Operations schedules drills and retains sanitized status.
|
|
|
|
Retain backup artifacts for the approved legal/operational period and at least
|
|
through the release's rollback window. Retain signed evidence, drill reports,
|
|
and deletion receipts for the audit period. Disposal must remove every backup
|
|
copy and provider version according to policy, then revoke or retire references
|
|
without deleting the audit receipt. Cryptographic erasure is valid only when
|
|
key-destruction evidence and provider-copy coverage are independently proven.
|
|
|
|
## Failure Handling
|
|
|
|
Missing components, component-time skew, stale or expired evidence, revocation,
|
|
signature/key mismatch, changed stored files, release mismatch, failed semantic
|
|
checks, or an RPO breach block migration. The deployment journal records the
|
|
rejection without private provider details. If migration has not started, the
|
|
operator may supply fresh evidence and retry. Once migration starts, recovery
|
|
is explicitly forward-only until the verified coordinated recovery point is
|
|
restored with its matching release.
|