Enforce signed backup evidence before migrations

2026-08-03 02:01:13 +02:00
parent 2c515f73c2
commit cbbe08d912
16 changed files with 2157 additions and 52 deletions
@@ -0,0 +1,133 @@
+# Backup And Restore Evidence
+
+## Boundary
+
+`govoplan-deploy` verifies backup and restore evidence; it does not receive
+database, object-store, KMS, or orchestrator administration credentials and it
+does not create the backup. A provider-owned backup controller creates one
+coordinated recovery point, a separate drill runner restores it into an
+isolated target, and an evidence authority signs the resulting receipt.
+
+The application containers receive only a sanitized projection: evidence,
+recovery-point and drill identifiers, hashes, timestamps, component count, and
+measured RPO/RTO. Artifact locations, provider credentials, encryption-key
+references, the public trust keyring, and private signing keys remain in the
+deployment/evidence boundary.
+
+The machine-readable contracts are:
+
+- [`backup-evidence.schema.json`](backup-evidence.schema.json);
+- [`backup-evidence-keyring.schema.json`](backup-evidence-keyring.schema.json).
+
+One evidence document is bound to the installation id, deployment profile,
+topology subject, exact signed release manifest, image digests, and composition
+digest. It covers PostgreSQL, objects, protected configuration, and recoverable
+key custody at one recovery point. It contains references, never key material.
+
+## Production Sequence
+
+1. Establish the provider snapshot, application quiesce, or transaction
+   boundary and retain a hash of its fencing token.
+2. Capture PostgreSQL, object storage, protected deployment configuration, and
+   key-custody state within five minutes of that recovery point.
+3. Restore all four components into a target isolated from production write
+   endpoints and production queues.
+4. Start the exact immutable release named in the evidence, verify migration
+   heads, verify a deterministic manifest of representative object hashes, and
+   execute the documented semantic journey checks.
+5. Record actual data loss and elapsed recovery as measured RPO and RTO. A
+   measured RPO above the declared objective invalidates the evidence.
+6. Sign the canonical receipt using an evidence-authority Ed25519 key held
+   outside the application and deployment host. During key rotation, include
+   both accepted signatures.
+7. Transfer the evidence SHA-256 through an independent approved channel, then
+   verify and adopt it on the deployment host.
+
+Provider automation can sign and validate an unsigned receipt with:
+
+```sh
+python tools/deployment/sign-backup-evidence.py \
+  --input unsigned-backup-evidence.json \
+  --output backup-evidence.json \
+  --trusted-keyring backup-evidence-keyring.json \
+  --signing-key backup-authority-2026=/run/keys/backup-authority.pem
+```
+
+The private key file must be owner-only. The tool refuses an unexpected key
+type, an inactive/untrusted signer, malformed or partial evidence, stale
+recovery points, failed drill checks, mismatched releases, and non-canonical
+output.
+
+Adopt the result using the independently obtained digest:
+
+```sh
+python3 govoplan-deploy.pyz verify-backup \
+  --directory /srv/govoplan/default \
+  --evidence ./backup-evidence.json \
+  --evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \
+  --trusted-keyring ./backup-evidence-keyring.json \
+  --adopt
+```
+
+Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every
+self-hosted release identity change is conservatively treated as a migration
+boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when
+fresh evidence for the previously applied immutable release is unavailable.
+Compose verifies once before changing runtime state and again after API/worker
+quiescing immediately before migration. The exported Kubernetes migration Job
+is generated only after verification and is annotated with the sanitized
+evidence digest, recovery-point id, and drill id.
+
+## Provider Runbooks
+
+### PostgreSQL
+
+Use a managed transaction-consistent snapshot or a base backup plus retained
+WAL sufficient to reconstruct the declared point. Record the provider,
+protected artifact reference and digest, snapshot identity, and PostgreSQL LSN.
+The restore drill must connect only to the isolated database and must compare
+the resulting migration-head digest with the release expectation.
+
+### Object Storage
+
+Use provider snapshots/versioning or an immutable object copy. Build a sorted
+manifest containing object key, version, size, and content digest, then record
+its digest, object count, total bytes, provider version identity, and protected
+artifact reference. Verify representative objects from every owning module
+after restore. Single-node managed Garage is persistent but not highly
+available; copy its coordinated recovery material to an independent failure
+domain.
+
+### Configuration And Key Custody
+
+Back up the private installation bundle and external secret-manager bindings as
+an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault
+state, record the provider keyset reference, version, and a successful
+recoverability assertion. Never put a key, recovery share, token, password, or
+credential-bearing URL in evidence. The isolated drill must prove that the
+restored release can decrypt representative protected content without
+exporting the key material into the report.
+
+## Ownership And Retention
+
+The deployment owner approves the RPO/RTO objectives. State-service owners
+operate backup capture and restoration. Module owners define representative
+objects and semantic checks. Security owns evidence-authority keys and
+revocation. Operations schedules drills and retains sanitized status.
+
+Retain backup artifacts for the approved legal/operational period and at least
+through the release's rollback window. Retain signed evidence, drill reports,
+and deletion receipts for the audit period. Disposal must remove every backup
+copy and provider version according to policy, then revoke or retire references
+without deleting the audit receipt. Cryptographic erasure is valid only when
+key-destruction evidence and provider-copy coverage are independently proven.
+
+## Failure Handling
+
+Missing components, component-time skew, stale or expired evidence, revocation,
+signature/key mismatch, changed stored files, release mismatch, failed semantic
+checks, or an RPO breach block migration. The deployment journal records the
+rejection without private provider details. If migration has not started, the
+operator may supply fresh evidence and retry. Once migration starts, recovery
+is explicitly forward-only until the verified coordinated recovery point is
+restored with its matching release.
@@ -97,6 +97,9 @@ The private installation directory contains:
 | `receipt.json` | Last successfully applied immutable identities |
 | `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
 | `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
+| `backup-evidence.json` | Signed provider-neutral coordinated backup and isolated-restore receipt |
+| `backup-keyring.json` | Explicit public trust anchor for backup evidence authorities |
+| `backup-verification.json` | Sanitized local verification/adoption receipt |
 | `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
 | `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
 | `kubernetes.json` | Optional stateless multi-host Kubernetes export |
@@ -369,10 +372,12 @@ starts, recovery is forward-only unless an independently verified database
 backup is restored. See
 [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).

-Production updates still need an operator-provided database backup/restore
-gate, database compatibility declaration, image signature verification, and
-deployment-specific drain policy. The deployment journal proves its own
-actions; it does not manufacture backup evidence.
+Production updates still need operator/provider-created coordinated backup and
+restore evidence, a database compatibility declaration, and a
+deployment-specific drain policy. The deployer now verifies and enforces the
+signed evidence before migration, but does not manufacture backups or receive
+provider administration credentials. See
+[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md).

 ## Stateless Kubernetes Runtime

@@ -391,7 +396,9 @@ The output includes a release-specific migration Job, database-head wait init
 containers, API readiness/liveness probes, rolling Deployments, Services, Pod
 disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
 the named Secret through the cluster's secret manager and review ingress proxy
-CIDRs before deployment. Detailed rollout and scaling rules live in
+CIDRs before deployment. A release-changing export requires adopted backup
+evidence and carries only its sanitized digest and identifiers as Job
+annotations. Detailed rollout and scaling rules live in
 [Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).

 ## Recovery Commands
@@ -98,11 +98,15 @@ old code may not understand the new schema. Recovery then means one of:
 3. restore a separately verified, coordinated database/object/key backup and
   then deploy the matching release.

-The deployment tool does not create or validate that database backup. A
-`backup-required` annotation on the Kubernetes migration Job is an operator
-gate, not backup evidence. Production automation must provide a backup hook or
-external backup controller whose artifact, timestamp, scope, encryption key,
-and restore test can be referenced from the recovery record.
+The deployment tool does not create that backup. It does verify an externally
+produced, signed evidence contract covering PostgreSQL, objects, protected
+configuration, and key custody at one recovery point plus an isolated restore
+drill. A self-hosted release change cannot reach the migration command or be
+exported as a Kubernetes migration Job until fresh evidence bound to the
+previous immutable release has been adopted. Compose verifies it again after
+runtime quiescing. See
+[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md) for the contract,
+provider runbooks, RPO/RTO ownership, retention, and disposal rules.

 ## Scaled Nodes

@@ -190,14 +190,16 @@ access, node visibility, drain controls, migration serialization, and scheduler
 fencing. It does not by itself provide:

 - a highly available PostgreSQL, Redis, or object-store deployment;
- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
+- automatic PostgreSQL/object backup creation or point-in-time recovery;
 - autoscaling policy;
 - central logs, metrics, traces, or alert routing;
 - certificate portability between independently managed ingress providers;
 - automatic reconciliation of every possible module side effect;
 - a service-level availability guarantee.

-Those are deployment and module-adoption requirements. Before claiming high
+The deployer verifies and gates migrations on signed coordinated backup and
+isolated-restore evidence, but backup capture and restoration remain owned by
+the selected state-service providers. Before claiming high
 availability, drill replica loss, rolling replacement, session continuity, job
 redelivery, scheduler failover, migration exclusion, object-store outage, and a
 coordinated database/object/key restore. Recovery rules and evidence are
@@ -0,0 +1,37 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://govoplan.add-ideas.de/schemas/backup-evidence-keyring-v1.json",
+  "title": "GovOPlaN backup evidence trust keyring",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schema_version", "purpose", "keys"],
+  "properties": {
+    "schema_version": { "const": "1" },
+    "purpose": { "const": "govoplan-backup-evidence" },
+    "keys": {
+      "type": "array",
+      "minItems": 1,
+      "maxItems": 64,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": [
+          "key_id",
+          "algorithm",
+          "status",
+          "public_key_pem",
+          "not_before",
+          "expires_at"
+        ],
+        "properties": {
+          "key_id": { "type": "string", "minLength": 1, "maxLength": 128 },
+          "algorithm": { "const": "ed25519" },
+          "status": { "enum": ["active", "retired", "revoked"] },
+          "public_key_pem": { "type": "string", "minLength": 1, "maxLength": 8192 },
+          "not_before": { "type": "string", "format": "date-time" },
+          "expires_at": { "type": "string", "format": "date-time" }
+        }
+      }
+    }
+  }
+}
@@ -0,0 +1,255 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://govoplan.add-ideas.de/schemas/backup-evidence-v1.json",
+  "title": "GovOPlaN coordinated backup and restore evidence",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "schema_version",
+    "evidence_id",
+    "installation_id",
+    "deployment_subject",
+    "release",
+    "recovery_point",
+    "components",
+    "restore_drill",
+    "issued_at",
+    "expires_at",
+    "revoked",
+    "signatures"
+  ],
+  "properties": {
+    "schema_version": { "const": "1" },
+    "evidence_id": { "$ref": "#/$defs/token" },
+    "installation_id": { "$ref": "#/$defs/token" },
+    "deployment_subject": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["profile", "topology", "subject_ref"],
+      "properties": {
+        "profile": { "enum": ["evaluation", "self-hosted"] },
+        "topology": { "$ref": "#/$defs/token" },
+        "subject_ref": { "$ref": "#/$defs/reference" }
+      }
+    },
+    "release": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "channel",
+        "version",
+        "manifest_sha256",
+        "composition_sha256",
+        "api_image",
+        "web_image"
+      ],
+      "properties": {
+        "channel": { "$ref": "#/$defs/token" },
+        "version": { "$ref": "#/$defs/token" },
+        "manifest_sha256": { "$ref": "#/$defs/sha256" },
+        "composition_sha256": { "$ref": "#/$defs/sha256" },
+        "api_image": { "$ref": "#/$defs/digest_image" },
+        "web_image": { "$ref": "#/$defs/digest_image" }
+      }
+    },
+    "recovery_point": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["id", "captured_at", "consistency", "rpo_seconds", "write_fence"],
+      "properties": {
+        "id": { "$ref": "#/$defs/token" },
+        "captured_at": { "type": "string", "format": "date-time" },
+        "consistency": {
+          "enum": ["provider-atomic", "application-quiesced", "transaction-consistent"]
+        },
+        "rpo_seconds": { "$ref": "#/$defs/duration" },
+        "write_fence": {
+          "type": "object",
+          "additionalProperties": false,
+          "required": ["mode", "token_sha256", "established_at"],
+          "properties": {
+            "mode": {
+              "enum": ["provider-snapshot", "application-quiesce", "transaction-boundary"]
+            },
+            "token_sha256": { "$ref": "#/$defs/sha256" },
+            "established_at": { "type": "string", "format": "date-time" }
+          }
+        }
+      }
+    },
+    "components": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["database", "objects", "configuration", "key_custody"],
+      "properties": {
+        "database": { "$ref": "#/$defs/database" },
+        "objects": { "$ref": "#/$defs/objects" },
+        "configuration": { "$ref": "#/$defs/configuration" },
+        "key_custody": { "$ref": "#/$defs/key_custody" }
+      }
+    },
+    "restore_drill": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "drill_id",
+        "recovery_point_id",
+        "started_at",
+        "completed_at",
+        "isolated_target_ref",
+        "release_manifest_sha256",
+        "migration_heads_sha256",
+        "representative_object_manifest_sha256",
+        "database_verified",
+        "objects_verified",
+        "configuration_verified",
+        "key_custody_verified",
+        "semantic_checks",
+        "measured_rpo_seconds",
+        "measured_rto_seconds",
+        "evidence_ref"
+      ],
+      "properties": {
+        "drill_id": { "$ref": "#/$defs/token" },
+        "recovery_point_id": { "$ref": "#/$defs/token" },
+        "started_at": { "type": "string", "format": "date-time" },
+        "completed_at": { "type": "string", "format": "date-time" },
+        "isolated_target_ref": { "$ref": "#/$defs/reference" },
+        "release_manifest_sha256": { "$ref": "#/$defs/sha256" },
+        "migration_heads_sha256": { "$ref": "#/$defs/sha256" },
+        "representative_object_manifest_sha256": { "$ref": "#/$defs/sha256" },
+        "database_verified": { "const": true },
+        "objects_verified": { "const": true },
+        "configuration_verified": { "const": true },
+        "key_custody_verified": { "const": true },
+        "semantic_checks": {
+          "type": "array",
+          "minItems": 1,
+          "maxItems": 128,
+          "items": {
+            "type": "object",
+            "additionalProperties": false,
+            "required": ["id", "status", "evidence_ref"],
+            "properties": {
+              "id": { "$ref": "#/$defs/token" },
+              "status": { "const": "passed" },
+              "evidence_ref": { "$ref": "#/$defs/reference" }
+            }
+          }
+        },
+        "measured_rpo_seconds": { "$ref": "#/$defs/duration" },
+        "measured_rto_seconds": { "$ref": "#/$defs/duration" },
+        "evidence_ref": { "$ref": "#/$defs/reference" }
+      }
+    },
+    "issued_at": { "type": "string", "format": "date-time" },
+    "expires_at": { "type": "string", "format": "date-time" },
+    "revoked": { "const": false },
+    "signatures": {
+      "type": "array",
+      "minItems": 1,
+      "maxItems": 16,
+      "items": { "$ref": "#/$defs/signature" }
+    }
+  },
+  "$defs": {
+    "token": {
+      "type": "string",
+      "minLength": 1,
+      "maxLength": 128,
+      "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$"
+    },
+    "sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
+    "digest_image": {
+      "type": "string",
+      "maxLength": 300,
+      "pattern": "^[^@\\s]+@sha256:[0-9a-f]{64}$"
+    },
+    "reference": {
+      "type": "string",
+      "minLength": 3,
+      "maxLength": 2048,
+      "pattern": "^[A-Za-z][A-Za-z0-9+.-]*:[^\\s]+$"
+    },
+    "duration": { "type": "integer", "minimum": 0, "maximum": 2592000 },
+    "protected_key": {
+      "type": "object",
+      "properties": {
+        "protected": { "const": true },
+        "encryption_key_ref": { "$ref": "#/$defs/reference" },
+        "captured_at": { "type": "string", "format": "date-time" }
+      }
+    },
+    "database": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "provider", "artifact_ref", "artifact_sha256", "snapshot_id", "lsn",
+        "protected", "encryption_key_ref", "captured_at"
+      ],
+      "properties": {
+        "provider": { "$ref": "#/$defs/token" },
+        "artifact_ref": { "$ref": "#/$defs/reference" },
+        "artifact_sha256": { "$ref": "#/$defs/sha256" },
+        "snapshot_id": { "$ref": "#/$defs/token" },
+        "lsn": { "type": "string", "minLength": 1, "maxLength": 256 },
+        "protected": { "const": true },
+        "encryption_key_ref": { "$ref": "#/$defs/reference" },
+        "captured_at": { "type": "string", "format": "date-time" }
+      }
+    },
+    "objects": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "provider", "artifact_ref", "manifest_sha256", "version_id",
+        "object_count", "total_bytes", "protected", "encryption_key_ref", "captured_at"
+      ],
+      "properties": {
+        "provider": { "$ref": "#/$defs/token" },
+        "artifact_ref": { "$ref": "#/$defs/reference" },
+        "manifest_sha256": { "$ref": "#/$defs/sha256" },
+        "version_id": { "$ref": "#/$defs/token" },
+        "object_count": { "type": "integer", "minimum": 0 },
+        "total_bytes": { "type": "integer", "minimum": 0 },
+        "protected": { "const": true },
+        "encryption_key_ref": { "$ref": "#/$defs/reference" },
+        "captured_at": { "type": "string", "format": "date-time" }
+      }
+    },
+    "configuration": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["artifact_ref", "sha256", "protected", "encryption_key_ref", "captured_at"],
+      "properties": {
+        "artifact_ref": { "$ref": "#/$defs/reference" },
+        "sha256": { "$ref": "#/$defs/sha256" },
+        "protected": { "const": true },
+        "encryption_key_ref": { "$ref": "#/$defs/reference" },
+        "captured_at": { "type": "string", "format": "date-time" }
+      }
+    },
+    "key_custody": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["provider", "keyset_ref", "keyset_version", "recoverable", "captured_at"],
+      "properties": {
+        "provider": { "$ref": "#/$defs/token" },
+        "keyset_ref": { "$ref": "#/$defs/reference" },
+        "keyset_version": { "$ref": "#/$defs/token" },
+        "recoverable": { "const": true },
+        "captured_at": { "type": "string", "format": "date-time" }
+      }
+    },
+    "signature": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["key_id", "algorithm", "value"],
+      "properties": {
+        "key_id": { "$ref": "#/$defs/token" },
+        "algorithm": { "const": "ed25519" },
+        "value": { "type": "string", "minLength": 1, "maxLength": 256 }
+      }
+    }
+  }
+}