docs: organize cross-product documentation

docs/operations/BACKUP_AND_RESTORE_EVIDENCE.md

@@ -0,0 +1,133 @@
+# Backup And Restore Evidence
+
+## Boundary
+
+`govoplan-deploy` verifies backup and restore evidence; it does not receive
+database, object-store, KMS, or orchestrator administration credentials and it
+does not create the backup. A provider-owned backup controller creates one
+coordinated recovery point, a separate drill runner restores it into an
+isolated target, and an evidence authority signs the resulting receipt.
+
+The application containers receive only a sanitized projection: evidence,
+recovery-point and drill identifiers, hashes, timestamps, component count, and
+measured RPO/RTO. Artifact locations, provider credentials, encryption-key
+references, the public trust keyring, and private signing keys remain in the
+deployment/evidence boundary.
+
+The machine-readable contracts are:
+
+- [`backup-evidence.schema.json`](../backup-evidence.schema.json);
+- [`backup-evidence-keyring.schema.json`](../backup-evidence-keyring.schema.json).
+
+One evidence document is bound to the installation id, deployment profile,
+topology subject, exact signed release manifest, image digests, and composition
+digest. It covers PostgreSQL, objects, protected configuration, and recoverable
+key custody at one recovery point. It contains references, never key material.
+
+## Production Sequence
+
+1. Establish the provider snapshot, application quiesce, or transaction
+   boundary and retain a hash of its fencing token.
+2. Capture PostgreSQL, object storage, protected deployment configuration, and
+   key-custody state within five minutes of that recovery point.
+3. Restore all four components into a target isolated from production write
+   endpoints and production queues.
+4. Start the exact immutable release named in the evidence, verify migration
+   heads, verify a deterministic manifest of representative object hashes, and
+   execute the documented semantic journey checks.
+5. Record actual data loss and elapsed recovery as measured RPO and RTO. A
+   measured RPO above the declared objective invalidates the evidence.
+6. Sign the canonical receipt using an evidence-authority Ed25519 key held
+   outside the application and deployment host. During key rotation, include
+   both accepted signatures.
+7. Transfer the evidence SHA-256 through an independent approved channel, then
+   verify and adopt it on the deployment host.
+
+Provider automation can sign and validate an unsigned receipt with:
+
+```sh
+python tools/deployment/sign-backup-evidence.py \
+  --input unsigned-backup-evidence.json \
+  --output backup-evidence.json \
+  --trusted-keyring backup-evidence-keyring.json \
+  --signing-key backup-authority-2026=/run/keys/backup-authority.pem
+```
+
+The private key file must be owner-only. The tool refuses an unexpected key
+type, an inactive/untrusted signer, malformed or partial evidence, stale
+recovery points, failed drill checks, mismatched releases, and non-canonical
+output.
+
+Adopt the result using the independently obtained digest:
+
+```sh
+python3 govoplan-deploy.pyz verify-backup \
+  --directory /srv/govoplan/default \
+  --evidence ./backup-evidence.json \
+  --evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \
+  --trusted-keyring ./backup-evidence-keyring.json \
+  --adopt
+```
+
+Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every
+self-hosted release identity change is conservatively treated as a migration
+boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when
+fresh evidence for the previously applied immutable release is unavailable.
+Compose verifies once before changing runtime state and again after API/worker
+quiescing immediately before migration. The exported Kubernetes migration Job
+is generated only after verification and is annotated with the sanitized
+evidence digest, recovery-point id, and drill id.
+
+## Provider Runbooks
+
+### PostgreSQL
+
+Use a managed transaction-consistent snapshot or a base backup plus retained
+WAL sufficient to reconstruct the declared point. Record the provider,
+protected artifact reference and digest, snapshot identity, and PostgreSQL LSN.
+The restore drill must connect only to the isolated database and must compare
+the resulting migration-head digest with the release expectation.
+
+### Object Storage
+
+Use provider snapshots/versioning or an immutable object copy. Build a sorted
+manifest containing object key, version, size, and content digest, then record
+its digest, object count, total bytes, provider version identity, and protected
+artifact reference. Verify representative objects from every owning module
+after restore. Single-node managed Garage is persistent but not highly
+available; copy its coordinated recovery material to an independent failure
+domain.
+
+### Configuration And Key Custody
+
+Back up the private installation bundle and external secret-manager bindings as
+an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault
+state, record the provider keyset reference, version, and a successful
+recoverability assertion. Never put a key, recovery share, token, password, or
+credential-bearing URL in evidence. The isolated drill must prove that the
+restored release can decrypt representative protected content without
+exporting the key material into the report.
+
+## Ownership And Retention
+
+The deployment owner approves the RPO/RTO objectives. State-service owners
+operate backup capture and restoration. Module owners define representative
+objects and semantic checks. Security owns evidence-authority keys and
+revocation. Operations schedules drills and retains sanitized status.
+
+Retain backup artifacts for the approved legal/operational period and at least
+through the release's rollback window. Retain signed evidence, drill reports,
+and deletion receipts for the audit period. Disposal must remove every backup
+copy and provider version according to policy, then revoke or retire references
+without deleting the audit receipt. Cryptographic erasure is valid only when
+key-destruction evidence and provider-copy coverage are independently proven.
+
+## Failure Handling
+
+Missing components, component-time skew, stale or expired evidence, revocation,
+signature/key mismatch, changed stored files, release mismatch, failed semantic
+checks, or an RPO breach block migration. The deployment journal records the
+rejection without private provider details. If migration has not started, the
+operator may supply fresh evidence and retry. Once migration starts, recovery
+is explicitly forward-only until the verified coordinated recovery point is
+restored with its matching release.