# Backup And Restore Evidence ## Boundary `govoplan-deploy` verifies backup and restore evidence; it does not receive database, object-store, KMS, or orchestrator administration credentials and it does not create the backup. A provider-owned backup controller creates one coordinated recovery point, a separate drill runner restores it into an isolated target, and an evidence authority signs the resulting receipt. The application containers receive only a sanitized projection: evidence, recovery-point and drill identifiers, hashes, timestamps, component count, and measured RPO/RTO. Artifact locations, provider credentials, encryption-key references, the public trust keyring, and private signing keys remain in the deployment/evidence boundary. The machine-readable contracts are: - [`backup-evidence.schema.json`](../backup-evidence.schema.json); - [`backup-evidence-keyring.schema.json`](../backup-evidence-keyring.schema.json). One evidence document is bound to the installation id, deployment profile, topology subject, exact signed release manifest, image digests, and composition digest. It covers PostgreSQL, objects, protected configuration, and recoverable key custody at one recovery point. It contains references, never key material. ## Production Sequence 1. Establish the provider snapshot, application quiesce, or transaction boundary and retain a hash of its fencing token. 2. Capture PostgreSQL, object storage, protected deployment configuration, and key-custody state within five minutes of that recovery point. 3. Restore all four components into a target isolated from production write endpoints and production queues. 4. Start the exact immutable release named in the evidence, verify migration heads, verify a deterministic manifest of representative object hashes, and execute the documented semantic journey checks. 5. Record actual data loss and elapsed recovery as measured RPO and RTO. A measured RPO above the declared objective invalidates the evidence. 6. Sign the canonical receipt using an evidence-authority Ed25519 key held outside the application and deployment host. During key rotation, include both accepted signatures. 7. Transfer the evidence SHA-256 through an independent approved channel, then verify and adopt it on the deployment host. Provider automation can sign and validate an unsigned receipt with: ```sh python tools/deployment/sign-backup-evidence.py \ --input unsigned-backup-evidence.json \ --output backup-evidence.json \ --trusted-keyring backup-evidence-keyring.json \ --signing-key backup-authority-2026=/run/keys/backup-authority.pem ``` The private key file must be owner-only. The tool refuses an unexpected key type, an inactive/untrusted signer, malformed or partial evidence, stale recovery points, failed drill checks, mismatched releases, and non-canonical output. Adopt the result using the independently obtained digest: ```sh python3 govoplan-deploy.pyz verify-backup \ --directory /srv/govoplan/default \ --evidence ./backup-evidence.json \ --evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \ --trusted-keyring ./backup-evidence-keyring.json \ --adopt ``` Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every self-hosted release identity change is conservatively treated as a migration boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when fresh evidence for the previously applied immutable release is unavailable. Compose verifies once before changing runtime state and again after API/worker quiescing immediately before migration. The exported Kubernetes migration Job is generated only after verification and is annotated with the sanitized evidence digest, recovery-point id, and drill id. ## Provider Runbooks ### PostgreSQL Use a managed transaction-consistent snapshot or a base backup plus retained WAL sufficient to reconstruct the declared point. Record the provider, protected artifact reference and digest, snapshot identity, and PostgreSQL LSN. The restore drill must connect only to the isolated database and must compare the resulting migration-head digest with the release expectation. ### Object Storage Use provider snapshots/versioning or an immutable object copy. Build a sorted manifest containing object key, version, size, and content digest, then record its digest, object count, total bytes, provider version identity, and protected artifact reference. Verify representative objects from every owning module after restore. Single-node managed Garage is persistent but not highly available; copy its coordinated recovery material to an independent failure domain. ### Configuration And Key Custody Back up the private installation bundle and external secret-manager bindings as an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault state, record the provider keyset reference, version, and a successful recoverability assertion. Never put a key, recovery share, token, password, or credential-bearing URL in evidence. The isolated drill must prove that the restored release can decrypt representative protected content without exporting the key material into the report. ## Ownership And Retention The deployment owner approves the RPO/RTO objectives. State-service owners operate backup capture and restoration. Module owners define representative objects and semantic checks. Security owns evidence-authority keys and revocation. Operations schedules drills and retains sanitized status. Retain backup artifacts for the approved legal/operational period and at least through the release's rollback window. Retain signed evidence, drill reports, and deletion receipts for the audit period. Disposal must remove every backup copy and provider version according to policy, then revoke or retire references without deleting the audit receipt. Cryptographic erasure is valid only when key-destruction evidence and provider-copy coverage are independently proven. ## Failure Handling Missing components, component-time skew, stale or expired evidence, revocation, signature/key mismatch, changed stored files, release mismatch, failed semantic checks, or an RPO breach block migration. The deployment journal records the rejection without private provider details. If migration has not started, the operator may supply fresh evidence and retry. Once migration starts, recovery is explicitly forward-only until the verified coordinated recovery point is restored with its matching release.