Enforce signed backup evidence before migrations
This commit is contained in:
@@ -0,0 +1,133 @@
|
|||||||
|
# Backup And Restore Evidence
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
`govoplan-deploy` verifies backup and restore evidence; it does not receive
|
||||||
|
database, object-store, KMS, or orchestrator administration credentials and it
|
||||||
|
does not create the backup. A provider-owned backup controller creates one
|
||||||
|
coordinated recovery point, a separate drill runner restores it into an
|
||||||
|
isolated target, and an evidence authority signs the resulting receipt.
|
||||||
|
|
||||||
|
The application containers receive only a sanitized projection: evidence,
|
||||||
|
recovery-point and drill identifiers, hashes, timestamps, component count, and
|
||||||
|
measured RPO/RTO. Artifact locations, provider credentials, encryption-key
|
||||||
|
references, the public trust keyring, and private signing keys remain in the
|
||||||
|
deployment/evidence boundary.
|
||||||
|
|
||||||
|
The machine-readable contracts are:
|
||||||
|
|
||||||
|
- [`backup-evidence.schema.json`](backup-evidence.schema.json);
|
||||||
|
- [`backup-evidence-keyring.schema.json`](backup-evidence-keyring.schema.json).
|
||||||
|
|
||||||
|
One evidence document is bound to the installation id, deployment profile,
|
||||||
|
topology subject, exact signed release manifest, image digests, and composition
|
||||||
|
digest. It covers PostgreSQL, objects, protected configuration, and recoverable
|
||||||
|
key custody at one recovery point. It contains references, never key material.
|
||||||
|
|
||||||
|
## Production Sequence
|
||||||
|
|
||||||
|
1. Establish the provider snapshot, application quiesce, or transaction
|
||||||
|
boundary and retain a hash of its fencing token.
|
||||||
|
2. Capture PostgreSQL, object storage, protected deployment configuration, and
|
||||||
|
key-custody state within five minutes of that recovery point.
|
||||||
|
3. Restore all four components into a target isolated from production write
|
||||||
|
endpoints and production queues.
|
||||||
|
4. Start the exact immutable release named in the evidence, verify migration
|
||||||
|
heads, verify a deterministic manifest of representative object hashes, and
|
||||||
|
execute the documented semantic journey checks.
|
||||||
|
5. Record actual data loss and elapsed recovery as measured RPO and RTO. A
|
||||||
|
measured RPO above the declared objective invalidates the evidence.
|
||||||
|
6. Sign the canonical receipt using an evidence-authority Ed25519 key held
|
||||||
|
outside the application and deployment host. During key rotation, include
|
||||||
|
both accepted signatures.
|
||||||
|
7. Transfer the evidence SHA-256 through an independent approved channel, then
|
||||||
|
verify and adopt it on the deployment host.
|
||||||
|
|
||||||
|
Provider automation can sign and validate an unsigned receipt with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python tools/deployment/sign-backup-evidence.py \
|
||||||
|
--input unsigned-backup-evidence.json \
|
||||||
|
--output backup-evidence.json \
|
||||||
|
--trusted-keyring backup-evidence-keyring.json \
|
||||||
|
--signing-key backup-authority-2026=/run/keys/backup-authority.pem
|
||||||
|
```
|
||||||
|
|
||||||
|
The private key file must be owner-only. The tool refuses an unexpected key
|
||||||
|
type, an inactive/untrusted signer, malformed or partial evidence, stale
|
||||||
|
recovery points, failed drill checks, mismatched releases, and non-canonical
|
||||||
|
output.
|
||||||
|
|
||||||
|
Adopt the result using the independently obtained digest:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python3 govoplan-deploy.pyz verify-backup \
|
||||||
|
--directory /srv/govoplan/default \
|
||||||
|
--evidence ./backup-evidence.json \
|
||||||
|
--evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \
|
||||||
|
--trusted-keyring ./backup-evidence-keyring.json \
|
||||||
|
--adopt
|
||||||
|
```
|
||||||
|
|
||||||
|
Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every
|
||||||
|
self-hosted release identity change is conservatively treated as a migration
|
||||||
|
boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when
|
||||||
|
fresh evidence for the previously applied immutable release is unavailable.
|
||||||
|
Compose verifies once before changing runtime state and again after API/worker
|
||||||
|
quiescing immediately before migration. The exported Kubernetes migration Job
|
||||||
|
is generated only after verification and is annotated with the sanitized
|
||||||
|
evidence digest, recovery-point id, and drill id.
|
||||||
|
|
||||||
|
## Provider Runbooks
|
||||||
|
|
||||||
|
### PostgreSQL
|
||||||
|
|
||||||
|
Use a managed transaction-consistent snapshot or a base backup plus retained
|
||||||
|
WAL sufficient to reconstruct the declared point. Record the provider,
|
||||||
|
protected artifact reference and digest, snapshot identity, and PostgreSQL LSN.
|
||||||
|
The restore drill must connect only to the isolated database and must compare
|
||||||
|
the resulting migration-head digest with the release expectation.
|
||||||
|
|
||||||
|
### Object Storage
|
||||||
|
|
||||||
|
Use provider snapshots/versioning or an immutable object copy. Build a sorted
|
||||||
|
manifest containing object key, version, size, and content digest, then record
|
||||||
|
its digest, object count, total bytes, provider version identity, and protected
|
||||||
|
artifact reference. Verify representative objects from every owning module
|
||||||
|
after restore. Single-node managed Garage is persistent but not highly
|
||||||
|
available; copy its coordinated recovery material to an independent failure
|
||||||
|
domain.
|
||||||
|
|
||||||
|
### Configuration And Key Custody
|
||||||
|
|
||||||
|
Back up the private installation bundle and external secret-manager bindings as
|
||||||
|
an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault
|
||||||
|
state, record the provider keyset reference, version, and a successful
|
||||||
|
recoverability assertion. Never put a key, recovery share, token, password, or
|
||||||
|
credential-bearing URL in evidence. The isolated drill must prove that the
|
||||||
|
restored release can decrypt representative protected content without
|
||||||
|
exporting the key material into the report.
|
||||||
|
|
||||||
|
## Ownership And Retention
|
||||||
|
|
||||||
|
The deployment owner approves the RPO/RTO objectives. State-service owners
|
||||||
|
operate backup capture and restoration. Module owners define representative
|
||||||
|
objects and semantic checks. Security owns evidence-authority keys and
|
||||||
|
revocation. Operations schedules drills and retains sanitized status.
|
||||||
|
|
||||||
|
Retain backup artifacts for the approved legal/operational period and at least
|
||||||
|
through the release's rollback window. Retain signed evidence, drill reports,
|
||||||
|
and deletion receipts for the audit period. Disposal must remove every backup
|
||||||
|
copy and provider version according to policy, then revoke or retire references
|
||||||
|
without deleting the audit receipt. Cryptographic erasure is valid only when
|
||||||
|
key-destruction evidence and provider-copy coverage are independently proven.
|
||||||
|
|
||||||
|
## Failure Handling
|
||||||
|
|
||||||
|
Missing components, component-time skew, stale or expired evidence, revocation,
|
||||||
|
signature/key mismatch, changed stored files, release mismatch, failed semantic
|
||||||
|
checks, or an RPO breach block migration. The deployment journal records the
|
||||||
|
rejection without private provider details. If migration has not started, the
|
||||||
|
operator may supply fresh evidence and retry. Once migration starts, recovery
|
||||||
|
is explicitly forward-only until the verified coordinated recovery point is
|
||||||
|
restored with its matching release.
|
||||||
@@ -97,6 +97,9 @@ The private installation directory contains:
|
|||||||
| `receipt.json` | Last successfully applied immutable identities |
|
| `receipt.json` | Last successfully applied immutable identities |
|
||||||
| `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
|
| `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
|
||||||
| `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
|
| `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
|
||||||
|
| `backup-evidence.json` | Signed provider-neutral coordinated backup and isolated-restore receipt |
|
||||||
|
| `backup-keyring.json` | Explicit public trust anchor for backup evidence authorities |
|
||||||
|
| `backup-verification.json` | Sanitized local verification/adoption receipt |
|
||||||
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
|
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
|
||||||
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
|
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
|
||||||
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
|
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
|
||||||
@@ -369,10 +372,12 @@ starts, recovery is forward-only unless an independently verified database
|
|||||||
backup is restored. See
|
backup is restored. See
|
||||||
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
|
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
|
||||||
|
|
||||||
Production updates still need an operator-provided database backup/restore
|
Production updates still need operator/provider-created coordinated backup and
|
||||||
gate, database compatibility declaration, image signature verification, and
|
restore evidence, a database compatibility declaration, and a
|
||||||
deployment-specific drain policy. The deployment journal proves its own
|
deployment-specific drain policy. The deployer now verifies and enforces the
|
||||||
actions; it does not manufacture backup evidence.
|
signed evidence before migration, but does not manufacture backups or receive
|
||||||
|
provider administration credentials. See
|
||||||
|
[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md).
|
||||||
|
|
||||||
## Stateless Kubernetes Runtime
|
## Stateless Kubernetes Runtime
|
||||||
|
|
||||||
@@ -391,7 +396,9 @@ The output includes a release-specific migration Job, database-head wait init
|
|||||||
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
|
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
|
||||||
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
|
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
|
||||||
the named Secret through the cluster's secret manager and review ingress proxy
|
the named Secret through the cluster's secret manager and review ingress proxy
|
||||||
CIDRs before deployment. Detailed rollout and scaling rules live in
|
CIDRs before deployment. A release-changing export requires adopted backup
|
||||||
|
evidence and carries only its sanitized digest and identifiers as Job
|
||||||
|
annotations. Detailed rollout and scaling rules live in
|
||||||
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
|
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
|
||||||
|
|
||||||
## Recovery Commands
|
## Recovery Commands
|
||||||
|
|||||||
@@ -98,11 +98,15 @@ old code may not understand the new schema. Recovery then means one of:
|
|||||||
3. restore a separately verified, coordinated database/object/key backup and
|
3. restore a separately verified, coordinated database/object/key backup and
|
||||||
then deploy the matching release.
|
then deploy the matching release.
|
||||||
|
|
||||||
The deployment tool does not create or validate that database backup. A
|
The deployment tool does not create that backup. It does verify an externally
|
||||||
`backup-required` annotation on the Kubernetes migration Job is an operator
|
produced, signed evidence contract covering PostgreSQL, objects, protected
|
||||||
gate, not backup evidence. Production automation must provide a backup hook or
|
configuration, and key custody at one recovery point plus an isolated restore
|
||||||
external backup controller whose artifact, timestamp, scope, encryption key,
|
drill. A self-hosted release change cannot reach the migration command or be
|
||||||
and restore test can be referenced from the recovery record.
|
exported as a Kubernetes migration Job until fresh evidence bound to the
|
||||||
|
previous immutable release has been adopted. Compose verifies it again after
|
||||||
|
runtime quiescing. See
|
||||||
|
[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md) for the contract,
|
||||||
|
provider runbooks, RPO/RTO ownership, retention, and disposal rules.
|
||||||
|
|
||||||
## Scaled Nodes
|
## Scaled Nodes
|
||||||
|
|
||||||
|
|||||||
@@ -190,14 +190,16 @@ access, node visibility, drain controls, migration serialization, and scheduler
|
|||||||
fencing. It does not by itself provide:
|
fencing. It does not by itself provide:
|
||||||
|
|
||||||
- a highly available PostgreSQL, Redis, or object-store deployment;
|
- a highly available PostgreSQL, Redis, or object-store deployment;
|
||||||
- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
|
- automatic PostgreSQL/object backup creation or point-in-time recovery;
|
||||||
- autoscaling policy;
|
- autoscaling policy;
|
||||||
- central logs, metrics, traces, or alert routing;
|
- central logs, metrics, traces, or alert routing;
|
||||||
- certificate portability between independently managed ingress providers;
|
- certificate portability between independently managed ingress providers;
|
||||||
- automatic reconciliation of every possible module side effect;
|
- automatic reconciliation of every possible module side effect;
|
||||||
- a service-level availability guarantee.
|
- a service-level availability guarantee.
|
||||||
|
|
||||||
Those are deployment and module-adoption requirements. Before claiming high
|
The deployer verifies and gates migrations on signed coordinated backup and
|
||||||
|
isolated-restore evidence, but backup capture and restoration remain owned by
|
||||||
|
the selected state-service providers. Before claiming high
|
||||||
availability, drill replica loss, rolling replacement, session continuity, job
|
availability, drill replica loss, rolling replacement, session continuity, job
|
||||||
redelivery, scheduler failover, migration exclusion, object-store outage, and a
|
redelivery, scheduler failover, migration exclusion, object-store outage, and a
|
||||||
coordinated database/object/key restore. Recovery rules and evidence are
|
coordinated database/object/key restore. Recovery rules and evidence are
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://govoplan.add-ideas.de/schemas/backup-evidence-keyring-v1.json",
|
||||||
|
"title": "GovOPlaN backup evidence trust keyring",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["schema_version", "purpose", "keys"],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": { "const": "1" },
|
||||||
|
"purpose": { "const": "govoplan-backup-evidence" },
|
||||||
|
"keys": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"maxItems": 64,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"key_id",
|
||||||
|
"algorithm",
|
||||||
|
"status",
|
||||||
|
"public_key_pem",
|
||||||
|
"not_before",
|
||||||
|
"expires_at"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"key_id": { "type": "string", "minLength": 1, "maxLength": 128 },
|
||||||
|
"algorithm": { "const": "ed25519" },
|
||||||
|
"status": { "enum": ["active", "retired", "revoked"] },
|
||||||
|
"public_key_pem": { "type": "string", "minLength": 1, "maxLength": 8192 },
|
||||||
|
"not_before": { "type": "string", "format": "date-time" },
|
||||||
|
"expires_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,255 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://govoplan.add-ideas.de/schemas/backup-evidence-v1.json",
|
||||||
|
"title": "GovOPlaN coordinated backup and restore evidence",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"schema_version",
|
||||||
|
"evidence_id",
|
||||||
|
"installation_id",
|
||||||
|
"deployment_subject",
|
||||||
|
"release",
|
||||||
|
"recovery_point",
|
||||||
|
"components",
|
||||||
|
"restore_drill",
|
||||||
|
"issued_at",
|
||||||
|
"expires_at",
|
||||||
|
"revoked",
|
||||||
|
"signatures"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": { "const": "1" },
|
||||||
|
"evidence_id": { "$ref": "#/$defs/token" },
|
||||||
|
"installation_id": { "$ref": "#/$defs/token" },
|
||||||
|
"deployment_subject": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["profile", "topology", "subject_ref"],
|
||||||
|
"properties": {
|
||||||
|
"profile": { "enum": ["evaluation", "self-hosted"] },
|
||||||
|
"topology": { "$ref": "#/$defs/token" },
|
||||||
|
"subject_ref": { "$ref": "#/$defs/reference" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"release": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"channel",
|
||||||
|
"version",
|
||||||
|
"manifest_sha256",
|
||||||
|
"composition_sha256",
|
||||||
|
"api_image",
|
||||||
|
"web_image"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"channel": { "$ref": "#/$defs/token" },
|
||||||
|
"version": { "$ref": "#/$defs/token" },
|
||||||
|
"manifest_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"composition_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"api_image": { "$ref": "#/$defs/digest_image" },
|
||||||
|
"web_image": { "$ref": "#/$defs/digest_image" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"recovery_point": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["id", "captured_at", "consistency", "rpo_seconds", "write_fence"],
|
||||||
|
"properties": {
|
||||||
|
"id": { "$ref": "#/$defs/token" },
|
||||||
|
"captured_at": { "type": "string", "format": "date-time" },
|
||||||
|
"consistency": {
|
||||||
|
"enum": ["provider-atomic", "application-quiesced", "transaction-consistent"]
|
||||||
|
},
|
||||||
|
"rpo_seconds": { "$ref": "#/$defs/duration" },
|
||||||
|
"write_fence": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["mode", "token_sha256", "established_at"],
|
||||||
|
"properties": {
|
||||||
|
"mode": {
|
||||||
|
"enum": ["provider-snapshot", "application-quiesce", "transaction-boundary"]
|
||||||
|
},
|
||||||
|
"token_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"established_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"components": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["database", "objects", "configuration", "key_custody"],
|
||||||
|
"properties": {
|
||||||
|
"database": { "$ref": "#/$defs/database" },
|
||||||
|
"objects": { "$ref": "#/$defs/objects" },
|
||||||
|
"configuration": { "$ref": "#/$defs/configuration" },
|
||||||
|
"key_custody": { "$ref": "#/$defs/key_custody" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"restore_drill": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"drill_id",
|
||||||
|
"recovery_point_id",
|
||||||
|
"started_at",
|
||||||
|
"completed_at",
|
||||||
|
"isolated_target_ref",
|
||||||
|
"release_manifest_sha256",
|
||||||
|
"migration_heads_sha256",
|
||||||
|
"representative_object_manifest_sha256",
|
||||||
|
"database_verified",
|
||||||
|
"objects_verified",
|
||||||
|
"configuration_verified",
|
||||||
|
"key_custody_verified",
|
||||||
|
"semantic_checks",
|
||||||
|
"measured_rpo_seconds",
|
||||||
|
"measured_rto_seconds",
|
||||||
|
"evidence_ref"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"drill_id": { "$ref": "#/$defs/token" },
|
||||||
|
"recovery_point_id": { "$ref": "#/$defs/token" },
|
||||||
|
"started_at": { "type": "string", "format": "date-time" },
|
||||||
|
"completed_at": { "type": "string", "format": "date-time" },
|
||||||
|
"isolated_target_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"release_manifest_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"migration_heads_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"representative_object_manifest_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"database_verified": { "const": true },
|
||||||
|
"objects_verified": { "const": true },
|
||||||
|
"configuration_verified": { "const": true },
|
||||||
|
"key_custody_verified": { "const": true },
|
||||||
|
"semantic_checks": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"maxItems": 128,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["id", "status", "evidence_ref"],
|
||||||
|
"properties": {
|
||||||
|
"id": { "$ref": "#/$defs/token" },
|
||||||
|
"status": { "const": "passed" },
|
||||||
|
"evidence_ref": { "$ref": "#/$defs/reference" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"measured_rpo_seconds": { "$ref": "#/$defs/duration" },
|
||||||
|
"measured_rto_seconds": { "$ref": "#/$defs/duration" },
|
||||||
|
"evidence_ref": { "$ref": "#/$defs/reference" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"issued_at": { "type": "string", "format": "date-time" },
|
||||||
|
"expires_at": { "type": "string", "format": "date-time" },
|
||||||
|
"revoked": { "const": false },
|
||||||
|
"signatures": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"maxItems": 16,
|
||||||
|
"items": { "$ref": "#/$defs/signature" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"$defs": {
|
||||||
|
"token": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1,
|
||||||
|
"maxLength": 128,
|
||||||
|
"pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$"
|
||||||
|
},
|
||||||
|
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
|
||||||
|
"digest_image": {
|
||||||
|
"type": "string",
|
||||||
|
"maxLength": 300,
|
||||||
|
"pattern": "^[^@\\s]+@sha256:[0-9a-f]{64}$"
|
||||||
|
},
|
||||||
|
"reference": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 3,
|
||||||
|
"maxLength": 2048,
|
||||||
|
"pattern": "^[A-Za-z][A-Za-z0-9+.-]*:[^\\s]+$"
|
||||||
|
},
|
||||||
|
"duration": { "type": "integer", "minimum": 0, "maximum": 2592000 },
|
||||||
|
"protected_key": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"protected": { "const": true },
|
||||||
|
"encryption_key_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"captured_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"database": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"provider", "artifact_ref", "artifact_sha256", "snapshot_id", "lsn",
|
||||||
|
"protected", "encryption_key_ref", "captured_at"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"provider": { "$ref": "#/$defs/token" },
|
||||||
|
"artifact_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"artifact_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"snapshot_id": { "$ref": "#/$defs/token" },
|
||||||
|
"lsn": { "type": "string", "minLength": 1, "maxLength": 256 },
|
||||||
|
"protected": { "const": true },
|
||||||
|
"encryption_key_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"captured_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"objects": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"provider", "artifact_ref", "manifest_sha256", "version_id",
|
||||||
|
"object_count", "total_bytes", "protected", "encryption_key_ref", "captured_at"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"provider": { "$ref": "#/$defs/token" },
|
||||||
|
"artifact_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"manifest_sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"version_id": { "$ref": "#/$defs/token" },
|
||||||
|
"object_count": { "type": "integer", "minimum": 0 },
|
||||||
|
"total_bytes": { "type": "integer", "minimum": 0 },
|
||||||
|
"protected": { "const": true },
|
||||||
|
"encryption_key_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"captured_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"configuration": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["artifact_ref", "sha256", "protected", "encryption_key_ref", "captured_at"],
|
||||||
|
"properties": {
|
||||||
|
"artifact_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"sha256": { "$ref": "#/$defs/sha256" },
|
||||||
|
"protected": { "const": true },
|
||||||
|
"encryption_key_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"captured_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"key_custody": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["provider", "keyset_ref", "keyset_version", "recoverable", "captured_at"],
|
||||||
|
"properties": {
|
||||||
|
"provider": { "$ref": "#/$defs/token" },
|
||||||
|
"keyset_ref": { "$ref": "#/$defs/reference" },
|
||||||
|
"keyset_version": { "$ref": "#/$defs/token" },
|
||||||
|
"recoverable": { "const": true },
|
||||||
|
"captured_at": { "type": "string", "format": "date-time" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"signature": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["key_id", "algorithm", "value"],
|
||||||
|
"properties": {
|
||||||
|
"key_id": { "$ref": "#/$defs/token" },
|
||||||
|
"algorithm": { "const": "ed25519" },
|
||||||
|
"value": { "type": "string", "minLength": 1, "maxLength": 256 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,430 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
from contextlib import redirect_stderr, redirect_stdout
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
from cryptography.hazmat.primitives import serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||||
|
|
||||||
|
|
||||||
|
META_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(META_ROOT / "tools" / "deployment"))
|
||||||
|
|
||||||
|
from govoplan_deploy.backup_evidence import verify_backup_evidence # noqa: E402
|
||||||
|
from govoplan_deploy.bundle import ( # noqa: E402
|
||||||
|
atomic_write,
|
||||||
|
bundle_paths,
|
||||||
|
canonical_json,
|
||||||
|
read_env,
|
||||||
|
)
|
||||||
|
from govoplan_deploy.cli import main as deploy_main # noqa: E402
|
||||||
|
from govoplan_deploy.distribution import ( # noqa: E402
|
||||||
|
DistributionError,
|
||||||
|
canonical_signed_payload,
|
||||||
|
canonical_json as canonical_distribution_json,
|
||||||
|
)
|
||||||
|
from govoplan_deploy.model import default_spec, parse_spec # noqa: E402
|
||||||
|
from govoplan_deploy.planning import ( # noqa: E402
|
||||||
|
release_change_requires_backup,
|
||||||
|
verify_stored_backup_evidence,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class BackupEvidenceTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.now = datetime(2026, 8, 3, 12, tzinfo=UTC)
|
||||||
|
self.private = Ed25519PrivateKey.generate()
|
||||||
|
public = (
|
||||||
|
self.private.public_key()
|
||||||
|
.public_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PublicFormat.SubjectPublicKeyInfo,
|
||||||
|
)
|
||||||
|
.decode("ascii")
|
||||||
|
)
|
||||||
|
self.keyring = {
|
||||||
|
"schema_version": "1",
|
||||||
|
"purpose": "govoplan-backup-evidence",
|
||||||
|
"keys": [
|
||||||
|
{
|
||||||
|
"key_id": "backup-controller-1",
|
||||||
|
"algorithm": "ed25519",
|
||||||
|
"status": "active",
|
||||||
|
"public_key_pem": public,
|
||||||
|
"not_before": (self.now - timedelta(days=1)).isoformat(),
|
||||||
|
"expires_at": (self.now + timedelta(days=365)).isoformat(),
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
self.release = {
|
||||||
|
"channel": "stable",
|
||||||
|
"version": "1.2.3",
|
||||||
|
"manifest_sha256": "a" * 64,
|
||||||
|
"composition_sha256": "b" * 64,
|
||||||
|
"api_image": "registry.example/api@sha256:" + "c" * 64,
|
||||||
|
"web_image": "registry.example/web@sha256:" + "d" * 64,
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_verifies_coordinated_restore_drill_and_release_binding(self) -> None:
|
||||||
|
summary = verify_backup_evidence(
|
||||||
|
self._evidence(),
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual("recovery-1", summary["recovery_point_id"])
|
||||||
|
self.assertEqual("restore-1", summary["restore_drill_id"])
|
||||||
|
self.assertEqual("backup-controller-1", summary["signature_key_id"])
|
||||||
|
|
||||||
|
def test_tampering_staleness_and_partial_restore_fail_closed(self) -> None:
|
||||||
|
tampered = self._evidence()
|
||||||
|
tampered["components"]["objects"]["object_count"] = 999
|
||||||
|
with self.assertRaisesRegex(DistributionError, "signature verification"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
tampered,
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
stale = self._evidence(captured=self.now - timedelta(days=2))
|
||||||
|
with self.assertRaisesRegex(DistributionError, "stale"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
stale,
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
partial = self._evidence()
|
||||||
|
partial["restore_drill"]["objects_verified"] = False
|
||||||
|
partial["signatures"] = [self._signature(partial)]
|
||||||
|
with self.assertRaisesRegex(DistributionError, "objects_verified"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
partial,
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_wrong_release_key_purpose_and_component_skew_fail_closed(self) -> None:
|
||||||
|
wrong_release = dict(self.release)
|
||||||
|
wrong_release["version"] = "1.2.4"
|
||||||
|
with self.assertRaisesRegex(DistributionError, "release field"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
self._evidence(),
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=wrong_release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
wrong_keyring = dict(self.keyring)
|
||||||
|
wrong_keyring["purpose"] = "govoplan-runtime-distribution"
|
||||||
|
with self.assertRaisesRegex(DistributionError, "wrong purpose"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
self._evidence(),
|
||||||
|
wrong_keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
skewed = self._evidence()
|
||||||
|
skewed["components"]["database"]["captured_at"] = (
|
||||||
|
self.now - timedelta(hours=1)
|
||||||
|
).isoformat()
|
||||||
|
skewed["signatures"] = [self._signature(skewed)]
|
||||||
|
with self.assertRaisesRegex(DistributionError, "one recovery point"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
skewed,
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
false_rto = self._evidence()
|
||||||
|
false_rto["restore_drill"]["measured_rto_seconds"] = 1
|
||||||
|
false_rto["signatures"] = [self._signature(false_rto)]
|
||||||
|
with self.assertRaisesRegex(DistributionError, "RTO"):
|
||||||
|
verify_backup_evidence(
|
||||||
|
false_rto,
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
now=self.now,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_provider_signing_tool_emits_canonical_verified_evidence(self) -> None:
|
||||||
|
self.now = datetime.now(UTC)
|
||||||
|
self.keyring["keys"][0]["not_before"] = (
|
||||||
|
self.now - timedelta(days=1)
|
||||||
|
).isoformat()
|
||||||
|
self.keyring["keys"][0]["expires_at"] = (
|
||||||
|
self.now + timedelta(days=365)
|
||||||
|
).isoformat()
|
||||||
|
evidence = self._evidence()
|
||||||
|
evidence["signatures"] = []
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory(prefix="govoplan-backup-signer-") as value:
|
||||||
|
root = Path(value)
|
||||||
|
source = root / "unsigned.json"
|
||||||
|
output = root / "signed.json"
|
||||||
|
keyring = root / "keyring.json"
|
||||||
|
private_key = root / "private.pem"
|
||||||
|
atomic_write(source, canonical_json(evidence), mode=0o600)
|
||||||
|
atomic_write(keyring, canonical_json(self.keyring), mode=0o600)
|
||||||
|
atomic_write(
|
||||||
|
private_key,
|
||||||
|
self.private.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.PKCS8,
|
||||||
|
serialization.NoEncryption(),
|
||||||
|
),
|
||||||
|
mode=0o600,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(META_ROOT / "tools/deployment/sign-backup-evidence.py"),
|
||||||
|
"--input",
|
||||||
|
str(source),
|
||||||
|
"--output",
|
||||||
|
str(output),
|
||||||
|
"--trusted-keyring",
|
||||||
|
str(keyring),
|
||||||
|
"--signing-key",
|
||||||
|
f"backup-controller-1={private_key}",
|
||||||
|
],
|
||||||
|
cwd=META_ROOT,
|
||||||
|
check=False,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(0, result.returncode, result.stderr)
|
||||||
|
encoded = output.read_bytes()
|
||||||
|
signed = json.loads(encoded)
|
||||||
|
self.assertEqual(canonical_distribution_json(signed), encoded)
|
||||||
|
summary = verify_backup_evidence(
|
||||||
|
signed,
|
||||||
|
self.keyring,
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
release=self.release,
|
||||||
|
)
|
||||||
|
self.assertEqual("backup-controller-1", summary["signature_key_id"])
|
||||||
|
|
||||||
|
def test_cli_adoption_gates_the_next_release_against_previous_receipt(self) -> None:
|
||||||
|
self.now = datetime.now(UTC)
|
||||||
|
self.keyring["keys"][0]["not_before"] = (
|
||||||
|
self.now - timedelta(days=1)
|
||||||
|
).isoformat()
|
||||||
|
self.keyring["keys"][0]["expires_at"] = (
|
||||||
|
self.now + timedelta(days=365)
|
||||||
|
).isoformat()
|
||||||
|
evidence = self._evidence()
|
||||||
|
encoded_evidence = canonical_distribution_json(evidence)
|
||||||
|
encoded_keyring = canonical_distribution_json(self.keyring)
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory(prefix="govoplan-backup-evidence-") as value:
|
||||||
|
paths = bundle_paths(Path(value))
|
||||||
|
paths.root.chmod(0o700)
|
||||||
|
raw = default_spec(
|
||||||
|
installation_id="govoplan-test",
|
||||||
|
profile="self-hosted",
|
||||||
|
public_url="https://govoplan.example.test",
|
||||||
|
ingress_mode="existing-proxy",
|
||||||
|
trusted_proxy_cidrs=("127.0.0.1/32",),
|
||||||
|
).to_dict()
|
||||||
|
raw["release"] = {
|
||||||
|
**raw["release"],
|
||||||
|
**self.release,
|
||||||
|
}
|
||||||
|
current = parse_spec(raw)
|
||||||
|
atomic_write(paths.spec, canonical_json(current.to_dict()), mode=0o600)
|
||||||
|
source_evidence = paths.root / "source-backup.json"
|
||||||
|
source_keyring = paths.root / "source-keyring.json"
|
||||||
|
atomic_write(source_evidence, encoded_evidence, mode=0o600)
|
||||||
|
atomic_write(source_keyring, encoded_keyring, mode=0o600)
|
||||||
|
|
||||||
|
output = io.StringIO()
|
||||||
|
with redirect_stdout(output), redirect_stderr(output):
|
||||||
|
result = deploy_main(
|
||||||
|
[
|
||||||
|
"verify-backup",
|
||||||
|
"--directory",
|
||||||
|
str(paths.root),
|
||||||
|
"--evidence",
|
||||||
|
str(source_evidence),
|
||||||
|
"--evidence-sha256",
|
||||||
|
hashlib.sha256(encoded_evidence).hexdigest(),
|
||||||
|
"--trusted-keyring",
|
||||||
|
str(source_keyring),
|
||||||
|
"--adopt",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
self.assertEqual(0, result, output.getvalue())
|
||||||
|
runtime_environment = read_env(paths.env)
|
||||||
|
self.assertEqual(
|
||||||
|
"verified",
|
||||||
|
runtime_environment["GOVOPLAN_BACKUP_EVIDENCE_STATE"],
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
"recovery-1",
|
||||||
|
runtime_environment["GOVOPLAN_BACKUP_RECOVERY_POINT_ID"],
|
||||||
|
)
|
||||||
|
self.assertNotIn("snapshot:postgres", str(runtime_environment))
|
||||||
|
self.assertNotIn("urn:kms", str(runtime_environment))
|
||||||
|
|
||||||
|
receipt = {
|
||||||
|
"installation_id": current.installation_id,
|
||||||
|
"profile": current.profile,
|
||||||
|
"release": dict(self.release),
|
||||||
|
}
|
||||||
|
atomic_write(paths.receipt, canonical_json(receipt), mode=0o600)
|
||||||
|
target_raw = current.to_dict()
|
||||||
|
target_raw["release"]["version"] = "1.2.4"
|
||||||
|
target_raw["release"]["manifest_sha256"] = "9" * 64
|
||||||
|
target = parse_spec(target_raw)
|
||||||
|
|
||||||
|
self.assertTrue(release_change_requires_backup(target, receipt))
|
||||||
|
summary = verify_stored_backup_evidence(
|
||||||
|
target,
|
||||||
|
paths,
|
||||||
|
receipt=receipt,
|
||||||
|
)
|
||||||
|
self.assertEqual("recovery-1", summary["recovery_point_id"])
|
||||||
|
|
||||||
|
def _evidence(self, *, captured: datetime | None = None) -> dict[str, object]:
|
||||||
|
captured = captured or self.now - timedelta(hours=2)
|
||||||
|
started = captured + timedelta(minutes=15)
|
||||||
|
completed = captured + timedelta(minutes=30)
|
||||||
|
issued = completed + timedelta(minutes=10)
|
||||||
|
artifact_time = captured.isoformat()
|
||||||
|
payload: dict[str, object] = {
|
||||||
|
"schema_version": "1",
|
||||||
|
"evidence_id": "backup-1",
|
||||||
|
"installation_id": "govoplan-test",
|
||||||
|
"deployment_subject": {
|
||||||
|
"profile": "self-hosted",
|
||||||
|
"topology": "compose",
|
||||||
|
"subject_ref": "urn:govoplan:installation:govoplan-test",
|
||||||
|
},
|
||||||
|
"release": dict(self.release),
|
||||||
|
"recovery_point": {
|
||||||
|
"id": "recovery-1",
|
||||||
|
"captured_at": captured.isoformat(),
|
||||||
|
"consistency": "application-quiesced",
|
||||||
|
"rpo_seconds": 300,
|
||||||
|
"write_fence": {
|
||||||
|
"mode": "application-quiesce",
|
||||||
|
"token_sha256": "e" * 64,
|
||||||
|
"established_at": captured.isoformat(),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"components": {
|
||||||
|
"database": {
|
||||||
|
"provider": "postgres",
|
||||||
|
"artifact_ref": "snapshot:postgres:backup-1",
|
||||||
|
"artifact_sha256": "1" * 64,
|
||||||
|
"snapshot_id": "pg-snapshot-1",
|
||||||
|
"lsn": "0/16B6C50",
|
||||||
|
"protected": True,
|
||||||
|
"encryption_key_ref": "urn:kms:key:database-backup",
|
||||||
|
"captured_at": artifact_time,
|
||||||
|
},
|
||||||
|
"objects": {
|
||||||
|
"provider": "s3",
|
||||||
|
"artifact_ref": "s3://backup/govoplan-test/recovery-1",
|
||||||
|
"manifest_sha256": "2" * 64,
|
||||||
|
"version_id": "object-snapshot-1",
|
||||||
|
"object_count": 4,
|
||||||
|
"total_bytes": 1024,
|
||||||
|
"protected": True,
|
||||||
|
"encryption_key_ref": "urn:kms:key:object-backup",
|
||||||
|
"captured_at": artifact_time,
|
||||||
|
},
|
||||||
|
"configuration": {
|
||||||
|
"artifact_ref": "backup:configuration:recovery-1",
|
||||||
|
"sha256": "3" * 64,
|
||||||
|
"protected": True,
|
||||||
|
"encryption_key_ref": "urn:kms:key:configuration-backup",
|
||||||
|
"captured_at": artifact_time,
|
||||||
|
},
|
||||||
|
"key_custody": {
|
||||||
|
"provider": "kms",
|
||||||
|
"keyset_ref": "urn:kms:keyset:govoplan-test",
|
||||||
|
"keyset_version": "version-4",
|
||||||
|
"recoverable": True,
|
||||||
|
"captured_at": artifact_time,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"restore_drill": {
|
||||||
|
"drill_id": "restore-1",
|
||||||
|
"recovery_point_id": "recovery-1",
|
||||||
|
"started_at": started.isoformat(),
|
||||||
|
"completed_at": completed.isoformat(),
|
||||||
|
"isolated_target_ref": "urn:govoplan:restore-target:restore-1",
|
||||||
|
"release_manifest_sha256": self.release["manifest_sha256"],
|
||||||
|
"migration_heads_sha256": "4" * 64,
|
||||||
|
"representative_object_manifest_sha256": "2" * 64,
|
||||||
|
"database_verified": True,
|
||||||
|
"objects_verified": True,
|
||||||
|
"configuration_verified": True,
|
||||||
|
"key_custody_verified": True,
|
||||||
|
"semantic_checks": [
|
||||||
|
{
|
||||||
|
"id": "institutional-journey",
|
||||||
|
"status": "passed",
|
||||||
|
"evidence_ref": "evidence:journey:institutional-1",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"measured_rpo_seconds": 120,
|
||||||
|
"measured_rto_seconds": 900,
|
||||||
|
"evidence_ref": "evidence:restore:restore-1",
|
||||||
|
},
|
||||||
|
"issued_at": issued.isoformat(),
|
||||||
|
"expires_at": (self.now + timedelta(days=7)).isoformat(),
|
||||||
|
"revoked": False,
|
||||||
|
"signatures": [],
|
||||||
|
}
|
||||||
|
payload["signatures"] = [self._signature(payload)]
|
||||||
|
return payload
|
||||||
|
|
||||||
|
def _signature(self, payload: dict[str, object]) -> dict[str, str]:
|
||||||
|
return {
|
||||||
|
"key_id": "backup-controller-1",
|
||||||
|
"algorithm": "ed25519",
|
||||||
|
"value": base64.b64encode(
|
||||||
|
self.private.sign(canonical_signed_payload(payload))
|
||||||
|
).decode("ascii"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -318,7 +318,16 @@ class DeploymentInstallerTests(unittest.TestCase):
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
manifest = render_kubernetes(spec, environment)
|
manifest = render_kubernetes(
|
||||||
|
spec,
|
||||||
|
environment,
|
||||||
|
backup_required=True,
|
||||||
|
backup_evidence={
|
||||||
|
"evidence_sha256": "c" * 64,
|
||||||
|
"recovery_point_id": "recovery-1",
|
||||||
|
"restore_drill_id": "drill-1",
|
||||||
|
},
|
||||||
|
)
|
||||||
rendered = json.dumps(manifest, sort_keys=True)
|
rendered = json.dumps(manifest, sort_keys=True)
|
||||||
kinds = [item["kind"] for item in manifest["items"]]
|
kinds = [item["kind"] for item in manifest["items"]]
|
||||||
deployments = {
|
deployments = {
|
||||||
@@ -351,6 +360,18 @@ class DeploymentInstallerTests(unittest.TestCase):
|
|||||||
"forward-recovery",
|
"forward-recovery",
|
||||||
migration["metadata"]["annotations"]["govoplan.add-ideas.de/recovery-mode"],
|
migration["metadata"]["annotations"]["govoplan.add-ideas.de/recovery-mode"],
|
||||||
)
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
"c" * 64,
|
||||||
|
migration["metadata"]["annotations"][
|
||||||
|
"govoplan.add-ideas.de/backup-evidence-sha256"
|
||||||
|
],
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
"recovery-1",
|
||||||
|
migration["metadata"]["annotations"][
|
||||||
|
"govoplan.add-ideas.de/recovery-point"
|
||||||
|
],
|
||||||
|
)
|
||||||
config = next(item for item in manifest["items"] if item["kind"] == "ConfigMap")
|
config = next(item for item in manifest["items"] if item["kind"] == "ConfigMap")
|
||||||
self.assertEqual("shared", config["data"]["GOVOPLAN_STATE_PROFILE"])
|
self.assertEqual("shared", config["data"]["GOVOPLAN_STATE_PROFILE"])
|
||||||
self.assertEqual("3", config["data"]["GOVOPLAN_EXPECTED_API_REPLICAS"])
|
self.assertEqual("3", config["data"]["GOVOPLAN_EXPECTED_API_REPLICAS"])
|
||||||
|
|||||||
@@ -0,0 +1,500 @@
|
|||||||
|
"""Signed, provider-neutral backup and isolated-restore evidence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
from typing import Any, Mapping
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from .distribution import (
|
||||||
|
DistributionError,
|
||||||
|
load_bounded_json,
|
||||||
|
verify_signed_document,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
MAX_BACKUP_EVIDENCE_BYTES = 1024 * 1024
|
||||||
|
MAX_BACKUP_KEYRING_BYTES = 1024 * 1024
|
||||||
|
DEFAULT_MAX_BACKUP_AGE_SECONDS = 24 * 60 * 60
|
||||||
|
MAX_COORDINATION_SKEW_SECONDS = 5 * 60
|
||||||
|
SHA256 = re.compile(r"^[0-9a-f]{64}$")
|
||||||
|
TOKEN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
|
||||||
|
IMAGE = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$")
|
||||||
|
REFERENCE = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*:[^\s]{1,2040}$")
|
||||||
|
|
||||||
|
|
||||||
|
def load_backup_evidence(path: Path) -> dict[str, Any]:
|
||||||
|
return load_bounded_json(path, maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES)
|
||||||
|
|
||||||
|
|
||||||
|
def load_backup_keyring(path: Path) -> dict[str, Any]:
|
||||||
|
return load_bounded_json(path, maximum_bytes=MAX_BACKUP_KEYRING_BYTES)
|
||||||
|
|
||||||
|
|
||||||
|
def verify_backup_evidence(
|
||||||
|
payload: Mapping[str, Any],
|
||||||
|
keyring: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
installation_id: str,
|
||||||
|
profile: str,
|
||||||
|
release: Mapping[str, object],
|
||||||
|
now: datetime | None = None,
|
||||||
|
max_age_seconds: int = DEFAULT_MAX_BACKUP_AGE_SECONDS,
|
||||||
|
openssl: str = "openssl",
|
||||||
|
) -> dict[str, object]:
|
||||||
|
current = (now or datetime.now(UTC)).astimezone(UTC)
|
||||||
|
values = _validate_payload(payload, now=current, max_age_seconds=max_age_seconds)
|
||||||
|
if payload.get("installation_id") != installation_id:
|
||||||
|
raise DistributionError("backup evidence belongs to another installation")
|
||||||
|
subject = _object(payload.get("deployment_subject"), "deployment_subject")
|
||||||
|
if subject.get("profile") != profile:
|
||||||
|
raise DistributionError("backup evidence belongs to another deployment profile")
|
||||||
|
evidence_release = _object(payload.get("release"), "release")
|
||||||
|
for field in (
|
||||||
|
"channel",
|
||||||
|
"version",
|
||||||
|
"manifest_sha256",
|
||||||
|
"composition_sha256",
|
||||||
|
"api_image",
|
||||||
|
"web_image",
|
||||||
|
):
|
||||||
|
if evidence_release.get(field) != release.get(field):
|
||||||
|
raise DistributionError(
|
||||||
|
f"backup evidence does not match release field {field!r}"
|
||||||
|
)
|
||||||
|
key_id = verify_signed_document(
|
||||||
|
payload,
|
||||||
|
keyring,
|
||||||
|
purpose="govoplan-backup-evidence",
|
||||||
|
label="backup evidence",
|
||||||
|
now=current,
|
||||||
|
openssl=openssl,
|
||||||
|
)
|
||||||
|
recovery_point = _object(payload.get("recovery_point"), "recovery_point")
|
||||||
|
restore = _object(payload.get("restore_drill"), "restore_drill")
|
||||||
|
return {
|
||||||
|
"evidence_id": payload["evidence_id"],
|
||||||
|
"recovery_point_id": recovery_point["id"],
|
||||||
|
"captured_at": recovery_point["captured_at"],
|
||||||
|
"expires_at": payload["expires_at"],
|
||||||
|
"restore_drill_id": restore["drill_id"],
|
||||||
|
"restore_started_at": restore["started_at"],
|
||||||
|
"restore_completed_at": restore["completed_at"],
|
||||||
|
"measured_rpo_seconds": restore["measured_rpo_seconds"],
|
||||||
|
"measured_rto_seconds": restore["measured_rto_seconds"],
|
||||||
|
"signature_key_id": key_id,
|
||||||
|
"component_count": values["component_count"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_payload(
|
||||||
|
payload: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
now: datetime,
|
||||||
|
max_age_seconds: int,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
if max_age_seconds < 60 or max_age_seconds > 30 * 24 * 60 * 60:
|
||||||
|
raise DistributionError("backup maximum age is out of bounds")
|
||||||
|
_exact_keys(
|
||||||
|
payload,
|
||||||
|
{
|
||||||
|
"schema_version",
|
||||||
|
"evidence_id",
|
||||||
|
"installation_id",
|
||||||
|
"deployment_subject",
|
||||||
|
"release",
|
||||||
|
"recovery_point",
|
||||||
|
"components",
|
||||||
|
"restore_drill",
|
||||||
|
"issued_at",
|
||||||
|
"expires_at",
|
||||||
|
"revoked",
|
||||||
|
"signatures",
|
||||||
|
},
|
||||||
|
"backup evidence",
|
||||||
|
)
|
||||||
|
if payload.get("schema_version") != "1":
|
||||||
|
raise DistributionError("unsupported backup evidence schema_version")
|
||||||
|
_token(payload.get("evidence_id"), "evidence_id")
|
||||||
|
_token(payload.get("installation_id"), "installation_id")
|
||||||
|
issued = _timestamp(payload.get("issued_at"), "issued_at")
|
||||||
|
expires = _timestamp(payload.get("expires_at"), "expires_at")
|
||||||
|
if issued > now or expires <= issued or expires <= now:
|
||||||
|
raise DistributionError("backup evidence is not currently valid")
|
||||||
|
if payload.get("revoked") is not False:
|
||||||
|
raise DistributionError("backup evidence is revoked")
|
||||||
|
|
||||||
|
subject = _object(payload.get("deployment_subject"), "deployment_subject")
|
||||||
|
_exact_keys(subject, {"profile", "topology", "subject_ref"}, "deployment_subject")
|
||||||
|
if subject.get("profile") not in {"evaluation", "self-hosted"}:
|
||||||
|
raise DistributionError("deployment_subject.profile is invalid")
|
||||||
|
_token(subject.get("topology"), "deployment_subject.topology")
|
||||||
|
_reference(subject.get("subject_ref"), "deployment_subject.subject_ref")
|
||||||
|
|
||||||
|
release = _object(payload.get("release"), "release")
|
||||||
|
_exact_keys(
|
||||||
|
release,
|
||||||
|
{
|
||||||
|
"channel",
|
||||||
|
"version",
|
||||||
|
"manifest_sha256",
|
||||||
|
"composition_sha256",
|
||||||
|
"api_image",
|
||||||
|
"web_image",
|
||||||
|
},
|
||||||
|
"release",
|
||||||
|
)
|
||||||
|
_token(release.get("channel"), "release.channel")
|
||||||
|
_token(release.get("version"), "release.version")
|
||||||
|
_sha256(release.get("manifest_sha256"), "release.manifest_sha256")
|
||||||
|
_sha256(release.get("composition_sha256"), "release.composition_sha256")
|
||||||
|
_image(release.get("api_image"), "release.api_image")
|
||||||
|
_image(release.get("web_image"), "release.web_image")
|
||||||
|
|
||||||
|
recovery = _object(payload.get("recovery_point"), "recovery_point")
|
||||||
|
_exact_keys(
|
||||||
|
recovery,
|
||||||
|
{"id", "captured_at", "consistency", "rpo_seconds", "write_fence"},
|
||||||
|
"recovery_point",
|
||||||
|
)
|
||||||
|
recovery_id = _token(recovery.get("id"), "recovery_point.id")
|
||||||
|
captured = _timestamp(recovery.get("captured_at"), "recovery_point.captured_at")
|
||||||
|
age = (now - captured).total_seconds()
|
||||||
|
if age < 0 or age > max_age_seconds:
|
||||||
|
raise DistributionError("backup recovery point is stale or in the future")
|
||||||
|
if recovery.get("consistency") not in {
|
||||||
|
"provider-atomic",
|
||||||
|
"application-quiesced",
|
||||||
|
"transaction-consistent",
|
||||||
|
}:
|
||||||
|
raise DistributionError("recovery_point.consistency is invalid")
|
||||||
|
declared_rpo = _bounded_integer(
|
||||||
|
recovery.get("rpo_seconds"),
|
||||||
|
"recovery_point.rpo_seconds",
|
||||||
|
maximum=30 * 24 * 60 * 60,
|
||||||
|
)
|
||||||
|
fence = _object(recovery.get("write_fence"), "recovery_point.write_fence")
|
||||||
|
_exact_keys(
|
||||||
|
fence,
|
||||||
|
{"mode", "token_sha256", "established_at"},
|
||||||
|
"recovery_point.write_fence",
|
||||||
|
)
|
||||||
|
if fence.get("mode") not in {
|
||||||
|
"provider-snapshot",
|
||||||
|
"application-quiesce",
|
||||||
|
"transaction-boundary",
|
||||||
|
}:
|
||||||
|
raise DistributionError("recovery_point.write_fence.mode is invalid")
|
||||||
|
_sha256(fence.get("token_sha256"), "recovery_point.write_fence.token_sha256")
|
||||||
|
established = _timestamp(
|
||||||
|
fence.get("established_at"),
|
||||||
|
"recovery_point.write_fence.established_at",
|
||||||
|
)
|
||||||
|
if abs((captured - established).total_seconds()) > MAX_COORDINATION_SKEW_SECONDS:
|
||||||
|
raise DistributionError(
|
||||||
|
"backup write fence is not coordinated with recovery point"
|
||||||
|
)
|
||||||
|
|
||||||
|
components = _object(payload.get("components"), "components")
|
||||||
|
_exact_keys(
|
||||||
|
components,
|
||||||
|
{"database", "objects", "configuration", "key_custody"},
|
||||||
|
"components",
|
||||||
|
)
|
||||||
|
captured_components = [
|
||||||
|
_database_component(components.get("database")),
|
||||||
|
_objects_component(components.get("objects")),
|
||||||
|
_configuration_component(components.get("configuration")),
|
||||||
|
_key_custody_component(components.get("key_custody")),
|
||||||
|
]
|
||||||
|
if any(
|
||||||
|
abs((component_time - captured).total_seconds()) > MAX_COORDINATION_SKEW_SECONDS
|
||||||
|
for component_time in captured_components
|
||||||
|
):
|
||||||
|
raise DistributionError("backup components do not share one recovery point")
|
||||||
|
|
||||||
|
restore = _object(payload.get("restore_drill"), "restore_drill")
|
||||||
|
_exact_keys(
|
||||||
|
restore,
|
||||||
|
{
|
||||||
|
"drill_id",
|
||||||
|
"recovery_point_id",
|
||||||
|
"started_at",
|
||||||
|
"completed_at",
|
||||||
|
"isolated_target_ref",
|
||||||
|
"release_manifest_sha256",
|
||||||
|
"migration_heads_sha256",
|
||||||
|
"representative_object_manifest_sha256",
|
||||||
|
"database_verified",
|
||||||
|
"objects_verified",
|
||||||
|
"configuration_verified",
|
||||||
|
"key_custody_verified",
|
||||||
|
"semantic_checks",
|
||||||
|
"measured_rpo_seconds",
|
||||||
|
"measured_rto_seconds",
|
||||||
|
"evidence_ref",
|
||||||
|
},
|
||||||
|
"restore_drill",
|
||||||
|
)
|
||||||
|
_token(restore.get("drill_id"), "restore_drill.drill_id")
|
||||||
|
if restore.get("recovery_point_id") != recovery_id:
|
||||||
|
raise DistributionError("restore drill used another recovery point")
|
||||||
|
started = _timestamp(restore.get("started_at"), "restore_drill.started_at")
|
||||||
|
completed = _timestamp(restore.get("completed_at"), "restore_drill.completed_at")
|
||||||
|
if started < captured or completed < started or completed > issued:
|
||||||
|
raise DistributionError(
|
||||||
|
"restore drill completion is outside evidence chronology"
|
||||||
|
)
|
||||||
|
_reference(restore.get("isolated_target_ref"), "restore_drill.isolated_target_ref")
|
||||||
|
_reference(restore.get("evidence_ref"), "restore_drill.evidence_ref")
|
||||||
|
for field in (
|
||||||
|
"release_manifest_sha256",
|
||||||
|
"migration_heads_sha256",
|
||||||
|
"representative_object_manifest_sha256",
|
||||||
|
):
|
||||||
|
_sha256(restore.get(field), f"restore_drill.{field}")
|
||||||
|
if restore.get("release_manifest_sha256") != release.get("manifest_sha256"):
|
||||||
|
raise DistributionError("restore drill used another immutable release")
|
||||||
|
for field in (
|
||||||
|
"database_verified",
|
||||||
|
"objects_verified",
|
||||||
|
"configuration_verified",
|
||||||
|
"key_custody_verified",
|
||||||
|
):
|
||||||
|
if restore.get(field) is not True:
|
||||||
|
raise DistributionError(f"restore_drill.{field} must be true")
|
||||||
|
semantic = restore.get("semantic_checks")
|
||||||
|
if not isinstance(semantic, list) or not semantic or len(semantic) > 128:
|
||||||
|
raise DistributionError("restore_drill.semantic_checks must not be empty")
|
||||||
|
seen_checks: set[str] = set()
|
||||||
|
for index, raw in enumerate(semantic):
|
||||||
|
check = _object(raw, f"restore_drill.semantic_checks[{index}]")
|
||||||
|
_exact_keys(
|
||||||
|
check,
|
||||||
|
{"id", "status", "evidence_ref"},
|
||||||
|
f"restore_drill.semantic_checks[{index}]",
|
||||||
|
)
|
||||||
|
check_id = _token(check.get("id"), f"semantic_checks[{index}].id")
|
||||||
|
if check_id in seen_checks or check.get("status") != "passed":
|
||||||
|
raise DistributionError("restore drill semantic checks are invalid")
|
||||||
|
seen_checks.add(check_id)
|
||||||
|
_reference(check.get("evidence_ref"), f"semantic_checks[{index}].evidence_ref")
|
||||||
|
measured_rpo = _bounded_integer(
|
||||||
|
restore.get("measured_rpo_seconds"),
|
||||||
|
"restore_drill.measured_rpo_seconds",
|
||||||
|
maximum=30 * 24 * 60 * 60,
|
||||||
|
)
|
||||||
|
measured_rto = _bounded_integer(
|
||||||
|
restore.get("measured_rto_seconds"),
|
||||||
|
"restore_drill.measured_rto_seconds",
|
||||||
|
maximum=30 * 24 * 60 * 60,
|
||||||
|
)
|
||||||
|
if abs((completed - started).total_seconds() - measured_rto) > 5:
|
||||||
|
raise DistributionError("restore drill RTO does not match its timestamps")
|
||||||
|
if measured_rpo > declared_rpo:
|
||||||
|
raise DistributionError(
|
||||||
|
"restore drill exceeds the declared recovery point objective"
|
||||||
|
)
|
||||||
|
_validate_signatures(payload.get("signatures"))
|
||||||
|
return {"component_count": len(captured_components)}
|
||||||
|
|
||||||
|
|
||||||
|
def _database_component(raw: object) -> datetime:
|
||||||
|
value = _object(raw, "components.database")
|
||||||
|
_exact_keys(
|
||||||
|
value,
|
||||||
|
{
|
||||||
|
"provider",
|
||||||
|
"artifact_ref",
|
||||||
|
"artifact_sha256",
|
||||||
|
"snapshot_id",
|
||||||
|
"lsn",
|
||||||
|
"protected",
|
||||||
|
"encryption_key_ref",
|
||||||
|
"captured_at",
|
||||||
|
},
|
||||||
|
"components.database",
|
||||||
|
)
|
||||||
|
_common_artifact(value, "components.database")
|
||||||
|
_token(value.get("snapshot_id"), "components.database.snapshot_id")
|
||||||
|
_bounded_text(value.get("lsn"), "components.database.lsn", maximum=256)
|
||||||
|
return _timestamp(value.get("captured_at"), "components.database.captured_at")
|
||||||
|
|
||||||
|
|
||||||
|
def _objects_component(raw: object) -> datetime:
|
||||||
|
value = _object(raw, "components.objects")
|
||||||
|
_exact_keys(
|
||||||
|
value,
|
||||||
|
{
|
||||||
|
"provider",
|
||||||
|
"artifact_ref",
|
||||||
|
"manifest_sha256",
|
||||||
|
"version_id",
|
||||||
|
"object_count",
|
||||||
|
"total_bytes",
|
||||||
|
"protected",
|
||||||
|
"encryption_key_ref",
|
||||||
|
"captured_at",
|
||||||
|
},
|
||||||
|
"components.objects",
|
||||||
|
)
|
||||||
|
_token(value.get("provider"), "components.objects.provider")
|
||||||
|
_reference(value.get("artifact_ref"), "components.objects.artifact_ref")
|
||||||
|
_sha256(value.get("manifest_sha256"), "components.objects.manifest_sha256")
|
||||||
|
_token(value.get("version_id"), "components.objects.version_id")
|
||||||
|
_bounded_integer(value.get("object_count"), "components.objects.object_count")
|
||||||
|
_bounded_integer(value.get("total_bytes"), "components.objects.total_bytes")
|
||||||
|
_protected_key_reference(value, "components.objects")
|
||||||
|
return _timestamp(value.get("captured_at"), "components.objects.captured_at")
|
||||||
|
|
||||||
|
|
||||||
|
def _configuration_component(raw: object) -> datetime:
|
||||||
|
value = _object(raw, "components.configuration")
|
||||||
|
_exact_keys(
|
||||||
|
value,
|
||||||
|
{
|
||||||
|
"artifact_ref",
|
||||||
|
"sha256",
|
||||||
|
"protected",
|
||||||
|
"encryption_key_ref",
|
||||||
|
"captured_at",
|
||||||
|
},
|
||||||
|
"components.configuration",
|
||||||
|
)
|
||||||
|
_reference(value.get("artifact_ref"), "components.configuration.artifact_ref")
|
||||||
|
_sha256(value.get("sha256"), "components.configuration.sha256")
|
||||||
|
_protected_key_reference(value, "components.configuration")
|
||||||
|
return _timestamp(value.get("captured_at"), "components.configuration.captured_at")
|
||||||
|
|
||||||
|
|
||||||
|
def _key_custody_component(raw: object) -> datetime:
|
||||||
|
value = _object(raw, "components.key_custody")
|
||||||
|
_exact_keys(
|
||||||
|
value,
|
||||||
|
{"provider", "keyset_ref", "keyset_version", "recoverable", "captured_at"},
|
||||||
|
"components.key_custody",
|
||||||
|
)
|
||||||
|
_token(value.get("provider"), "components.key_custody.provider")
|
||||||
|
_reference(value.get("keyset_ref"), "components.key_custody.keyset_ref")
|
||||||
|
_token(value.get("keyset_version"), "components.key_custody.keyset_version")
|
||||||
|
if value.get("recoverable") is not True:
|
||||||
|
raise DistributionError("components.key_custody.recoverable must be true")
|
||||||
|
return _timestamp(value.get("captured_at"), "components.key_custody.captured_at")
|
||||||
|
|
||||||
|
|
||||||
|
def _common_artifact(value: Mapping[str, Any], label: str) -> None:
|
||||||
|
_token(value.get("provider"), f"{label}.provider")
|
||||||
|
_reference(value.get("artifact_ref"), f"{label}.artifact_ref")
|
||||||
|
_sha256(value.get("artifact_sha256"), f"{label}.artifact_sha256")
|
||||||
|
_protected_key_reference(value, label)
|
||||||
|
|
||||||
|
|
||||||
|
def _protected_key_reference(value: Mapping[str, Any], label: str) -> None:
|
||||||
|
if value.get("protected") is not True:
|
||||||
|
raise DistributionError(f"{label}.protected must be true")
|
||||||
|
_reference(value.get("encryption_key_ref"), f"{label}.encryption_key_ref")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_signatures(raw: object) -> None:
|
||||||
|
if not isinstance(raw, list) or not raw or len(raw) > 16:
|
||||||
|
raise DistributionError("backup evidence signatures must not be empty")
|
||||||
|
seen: set[str] = set()
|
||||||
|
for index, item in enumerate(raw):
|
||||||
|
signature = _object(item, f"signatures[{index}]")
|
||||||
|
_exact_keys(signature, {"key_id", "algorithm", "value"}, f"signatures[{index}]")
|
||||||
|
key_id = _token(signature.get("key_id"), f"signatures[{index}].key_id")
|
||||||
|
if key_id in seen or signature.get("algorithm") != "ed25519":
|
||||||
|
raise DistributionError("backup evidence signatures are invalid")
|
||||||
|
seen.add(key_id)
|
||||||
|
encoded = signature.get("value")
|
||||||
|
if not isinstance(encoded, str) or len(encoded) > 256:
|
||||||
|
raise DistributionError("backup evidence signature value is invalid")
|
||||||
|
|
||||||
|
|
||||||
|
def _reference(raw: object, label: str) -> str:
|
||||||
|
value = _bounded_text(raw, label, maximum=2048)
|
||||||
|
if REFERENCE.fullmatch(value) is None or "BEGIN " in value.upper():
|
||||||
|
raise DistributionError(f"{label} must be an opaque provider reference")
|
||||||
|
parsed = urlsplit(value)
|
||||||
|
if parsed.username or parsed.password or parsed.query or parsed.fragment:
|
||||||
|
raise DistributionError(f"{label} must not contain credentials or query data")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _object(raw: object, label: str) -> dict[str, Any]:
|
||||||
|
if not isinstance(raw, dict) or not all(isinstance(key, str) for key in raw):
|
||||||
|
raise DistributionError(f"{label} must be an object")
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def _exact_keys(value: Mapping[str, Any], keys: set[str], label: str) -> None:
|
||||||
|
if set(value) != keys:
|
||||||
|
missing = sorted(keys - set(value))
|
||||||
|
extra = sorted(set(value) - keys)
|
||||||
|
detail = []
|
||||||
|
if missing:
|
||||||
|
detail.append("missing " + ", ".join(missing))
|
||||||
|
if extra:
|
||||||
|
detail.append("unknown " + ", ".join(extra))
|
||||||
|
raise DistributionError(f"{label} has invalid fields: {'; '.join(detail)}")
|
||||||
|
|
||||||
|
|
||||||
|
def _timestamp(raw: object, label: str) -> datetime:
|
||||||
|
value = _bounded_text(raw, label, maximum=64)
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise DistributionError(f"{label} must be an RFC3339 timestamp") from exc
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
raise DistributionError(f"{label} must include a timezone")
|
||||||
|
return parsed.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _token(raw: object, label: str) -> str:
|
||||||
|
value = _bounded_text(raw, label, maximum=128)
|
||||||
|
if TOKEN.fullmatch(value) is None:
|
||||||
|
raise DistributionError(f"{label} is invalid")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(raw: object, label: str) -> str:
|
||||||
|
value = _bounded_text(raw, label, maximum=64)
|
||||||
|
if SHA256.fullmatch(value) is None:
|
||||||
|
raise DistributionError(f"{label} must be a lowercase SHA-256 digest")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _image(raw: object, label: str) -> str:
|
||||||
|
value = _bounded_text(raw, label, maximum=300)
|
||||||
|
if IMAGE.fullmatch(value) is None:
|
||||||
|
raise DistributionError(f"{label} must be an OCI image pinned by sha256")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_text(raw: object, label: str, *, maximum: int) -> str:
|
||||||
|
if not isinstance(raw, str) or not raw or len(raw) > maximum or "\n" in raw:
|
||||||
|
raise DistributionError(f"{label} is invalid")
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_integer(
|
||||||
|
raw: object,
|
||||||
|
label: str,
|
||||||
|
*,
|
||||||
|
maximum: int = 2**63 - 1,
|
||||||
|
) -> int:
|
||||||
|
if isinstance(raw, bool) or not isinstance(raw, int) or raw < 0 or raw > maximum:
|
||||||
|
raise DistributionError(f"{label} is out of bounds")
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"DEFAULT_MAX_BACKUP_AGE_SECONDS",
|
||||||
|
"MAX_BACKUP_EVIDENCE_BYTES",
|
||||||
|
"MAX_BACKUP_KEYRING_BYTES",
|
||||||
|
"load_backup_evidence",
|
||||||
|
"load_backup_keyring",
|
||||||
|
"verify_backup_evidence",
|
||||||
|
]
|
||||||
@@ -28,7 +28,26 @@ PLAN_FILENAME = "plan.json"
|
|||||||
RECEIPT_FILENAME = "receipt.json"
|
RECEIPT_FILENAME = "receipt.json"
|
||||||
MANIFEST_FILENAME = "distribution-manifest.json"
|
MANIFEST_FILENAME = "distribution-manifest.json"
|
||||||
KEYRING_FILENAME = "distribution-keyring.json"
|
KEYRING_FILENAME = "distribution-keyring.json"
|
||||||
|
BACKUP_EVIDENCE_FILENAME = "backup-evidence.json"
|
||||||
|
BACKUP_KEYRING_FILENAME = "backup-keyring.json"
|
||||||
|
BACKUP_VERIFICATION_FILENAME = "backup-verification.json"
|
||||||
LOCK_FILENAME = ".deployment.lock"
|
LOCK_FILENAME = ".deployment.lock"
|
||||||
|
BACKUP_RUNTIME_ENV_KEYS = (
|
||||||
|
"GOVOPLAN_BACKUP_EVIDENCE_STATE",
|
||||||
|
"GOVOPLAN_BACKUP_EVIDENCE_ID",
|
||||||
|
"GOVOPLAN_BACKUP_RECOVERY_POINT_ID",
|
||||||
|
"GOVOPLAN_BACKUP_RESTORE_DRILL_ID",
|
||||||
|
"GOVOPLAN_BACKUP_EVIDENCE_SHA256",
|
||||||
|
"GOVOPLAN_BACKUP_RELEASE_MANIFEST_SHA256",
|
||||||
|
"GOVOPLAN_BACKUP_CAPTURED_AT",
|
||||||
|
"GOVOPLAN_BACKUP_EXPIRES_AT",
|
||||||
|
"GOVOPLAN_BACKUP_RESTORE_STARTED_AT",
|
||||||
|
"GOVOPLAN_BACKUP_RESTORE_COMPLETED_AT",
|
||||||
|
"GOVOPLAN_BACKUP_VERIFIED_AT",
|
||||||
|
"GOVOPLAN_BACKUP_MEASURED_RPO_SECONDS",
|
||||||
|
"GOVOPLAN_BACKUP_MEASURED_RTO_SECONDS",
|
||||||
|
"GOVOPLAN_BACKUP_COMPONENT_COUNT",
|
||||||
|
)
|
||||||
RUNTIME_ENV_KEYS = (
|
RUNTIME_ENV_KEYS = (
|
||||||
"APP_ENV",
|
"APP_ENV",
|
||||||
"GOVOPLAN_INSTALL_PROFILE",
|
"GOVOPLAN_INSTALL_PROFILE",
|
||||||
@@ -78,6 +97,7 @@ RUNTIME_ENV_KEYS = (
|
|||||||
"FILE_STORAGE_S3_BUCKET",
|
"FILE_STORAGE_S3_BUCKET",
|
||||||
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
|
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
|
||||||
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
|
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
|
||||||
|
*BACKUP_RUNTIME_ENV_KEYS,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -95,6 +115,9 @@ class BundlePaths:
|
|||||||
receipt: Path
|
receipt: Path
|
||||||
manifest: Path
|
manifest: Path
|
||||||
keyring: Path
|
keyring: Path
|
||||||
|
backup_evidence: Path
|
||||||
|
backup_keyring: Path
|
||||||
|
backup_verification: Path
|
||||||
lock: Path
|
lock: Path
|
||||||
|
|
||||||
|
|
||||||
@@ -116,6 +139,9 @@ def bundle_paths(root: Path) -> BundlePaths:
|
|||||||
receipt=resolved / RECEIPT_FILENAME,
|
receipt=resolved / RECEIPT_FILENAME,
|
||||||
manifest=resolved / MANIFEST_FILENAME,
|
manifest=resolved / MANIFEST_FILENAME,
|
||||||
keyring=resolved / KEYRING_FILENAME,
|
keyring=resolved / KEYRING_FILENAME,
|
||||||
|
backup_evidence=resolved / BACKUP_EVIDENCE_FILENAME,
|
||||||
|
backup_keyring=resolved / BACKUP_KEYRING_FILENAME,
|
||||||
|
backup_verification=resolved / BACKUP_VERIFICATION_FILENAME,
|
||||||
lock=resolved / LOCK_FILENAME,
|
lock=resolved / LOCK_FILENAME,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -20,7 +20,14 @@ from typing import Iterator, Mapping, Sequence
|
|||||||
from urllib.error import URLError
|
from urllib.error import URLError
|
||||||
from urllib.request import urlopen
|
from urllib.request import urlopen
|
||||||
|
|
||||||
|
from .backup_evidence import (
|
||||||
|
DEFAULT_MAX_BACKUP_AGE_SECONDS,
|
||||||
|
MAX_BACKUP_EVIDENCE_BYTES,
|
||||||
|
MAX_BACKUP_KEYRING_BYTES,
|
||||||
|
verify_backup_evidence,
|
||||||
|
)
|
||||||
from .bundle import (
|
from .bundle import (
|
||||||
|
BACKUP_RUNTIME_ENV_KEYS,
|
||||||
atomic_write,
|
atomic_write,
|
||||||
bundle_paths,
|
bundle_paths,
|
||||||
canonical_json,
|
canonical_json,
|
||||||
@@ -72,7 +79,12 @@ from .kubernetes import (
|
|||||||
render_kubernetes,
|
render_kubernetes,
|
||||||
write_secret_creation_hint,
|
write_secret_creation_hint,
|
||||||
)
|
)
|
||||||
from .planning import DeploymentPlan, build_plan
|
from .planning import (
|
||||||
|
DeploymentPlan,
|
||||||
|
build_plan,
|
||||||
|
release_change_requires_backup,
|
||||||
|
verify_stored_backup_evidence,
|
||||||
|
)
|
||||||
from .recovery import (
|
from .recovery import (
|
||||||
DeploymentOperationJournal,
|
DeploymentOperationJournal,
|
||||||
list_operations,
|
list_operations,
|
||||||
@@ -183,6 +195,22 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
help="Load verified archives into Docker using fixed image-load commands.",
|
help="Load verified archives into Docker using fixed image-load commands.",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
verify_backup = subparsers.add_parser(
|
||||||
|
"verify-backup",
|
||||||
|
help="Verify and optionally adopt signed coordinated backup evidence.",
|
||||||
|
)
|
||||||
|
_directory_argument(verify_backup)
|
||||||
|
evidence_source = verify_backup.add_mutually_exclusive_group(required=True)
|
||||||
|
evidence_source.add_argument("--evidence", type=Path)
|
||||||
|
evidence_source.add_argument("--evidence-url")
|
||||||
|
verify_backup.add_argument("--evidence-sha256", required=True)
|
||||||
|
verify_backup.add_argument("--trusted-keyring", type=Path, required=True)
|
||||||
|
verify_backup.add_argument(
|
||||||
|
"--allow-private-evidence-host",
|
||||||
|
action="store_true",
|
||||||
|
)
|
||||||
|
verify_backup.add_argument("--adopt", action="store_true")
|
||||||
|
|
||||||
kubernetes = subparsers.add_parser(
|
kubernetes = subparsers.add_parser(
|
||||||
"render-kubernetes",
|
"render-kubernetes",
|
||||||
help="Export the stateless multi-host runtime for Kubernetes.",
|
help="Export the stateless multi-host runtime for Kubernetes.",
|
||||||
@@ -392,6 +420,8 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
return _verify_release(args)
|
return _verify_release(args)
|
||||||
if args.command == "verify-offline-images":
|
if args.command == "verify-offline-images":
|
||||||
return _verify_offline_images(args)
|
return _verify_offline_images(args)
|
||||||
|
if args.command == "verify-backup":
|
||||||
|
return _verify_backup(args)
|
||||||
if args.command == "render-kubernetes":
|
if args.command == "render-kubernetes":
|
||||||
return _render_kubernetes(args)
|
return _render_kubernetes(args)
|
||||||
if args.command == "verify-kubernetes":
|
if args.command == "verify-kubernetes":
|
||||||
@@ -534,15 +564,17 @@ def _render_or_doctor(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
def _apply(args: argparse.Namespace) -> int:
|
def _apply(args: argparse.Namespace) -> int:
|
||||||
paths = bundle_paths(args.directory)
|
paths = bundle_paths(args.directory)
|
||||||
spec = load_spec(paths.spec)
|
|
||||||
if args.allow_unverified_images and spec.profile != "evaluation":
|
|
||||||
raise ValueError(
|
|
||||||
"--allow-unverified-images is restricted to evaluation installations"
|
|
||||||
)
|
|
||||||
ensure_private_directory(paths.root)
|
ensure_private_directory(paths.root)
|
||||||
with _deployment_lock(paths.lock):
|
with _deployment_lock(paths.lock):
|
||||||
|
spec = load_spec(paths.spec)
|
||||||
|
previous_receipt = _read_json_object(paths.receipt)
|
||||||
|
backup_required = release_change_requires_backup(spec, previous_receipt)
|
||||||
|
if args.allow_unverified_images and spec.profile != "evaluation":
|
||||||
|
raise ValueError(
|
||||||
|
"--allow-unverified-images is restricted to evaluation installations"
|
||||||
|
)
|
||||||
secrets = reconcile_runtime_environment(spec, read_env(paths.env))
|
secrets = reconcile_runtime_environment(spec, read_env(paths.env))
|
||||||
_write_bundle(spec, paths, secrets)
|
secrets = _write_bundle(spec, paths, secrets)
|
||||||
plan = build_plan(spec, paths, include_host_checks=True)
|
plan = build_plan(spec, paths, include_host_checks=True)
|
||||||
_write_plan(paths.plan, plan)
|
_write_plan(paths.plan, plan)
|
||||||
effective_errors = [
|
effective_errors = [
|
||||||
@@ -577,6 +609,36 @@ def _apply(args: argparse.Namespace) -> int:
|
|||||||
paths,
|
paths,
|
||||||
plan=plan.to_dict(),
|
plan=plan.to_dict(),
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
|
if backup_required:
|
||||||
|
backup_summary = verify_stored_backup_evidence(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
receipt=previous_receipt,
|
||||||
|
)
|
||||||
|
journal.record(
|
||||||
|
"backup-evidence-verified",
|
||||||
|
"succeeded",
|
||||||
|
dict(backup_summary),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
journal.record(
|
||||||
|
"backup-evidence-not-required",
|
||||||
|
"succeeded",
|
||||||
|
{"release_change": False},
|
||||||
|
)
|
||||||
|
except BaseException as exc:
|
||||||
|
journal.record(
|
||||||
|
"backup-evidence-rejected",
|
||||||
|
"blocked",
|
||||||
|
{
|
||||||
|
"phase": "preflight",
|
||||||
|
"exception_type": type(exc).__name__,
|
||||||
|
"migration_started": False,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
journal.failed(exc)
|
||||||
|
raise
|
||||||
compose = [
|
compose = [
|
||||||
docker,
|
docker,
|
||||||
"compose",
|
"compose",
|
||||||
@@ -624,6 +686,29 @@ def _apply(args: argparse.Namespace) -> int:
|
|||||||
"succeeded",
|
"succeeded",
|
||||||
{"services": mutable_runtime_services, "timeout_seconds": 120},
|
{"services": mutable_runtime_services, "timeout_seconds": 120},
|
||||||
)
|
)
|
||||||
|
if backup_required:
|
||||||
|
try:
|
||||||
|
backup_summary = verify_stored_backup_evidence(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
receipt=previous_receipt,
|
||||||
|
)
|
||||||
|
except BaseException as exc:
|
||||||
|
journal.record(
|
||||||
|
"backup-evidence-rejected",
|
||||||
|
"blocked",
|
||||||
|
{
|
||||||
|
"phase": "migration-boundary",
|
||||||
|
"exception_type": type(exc).__name__,
|
||||||
|
"migration_started": False,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
journal.record(
|
||||||
|
"backup-evidence-reverified",
|
||||||
|
"succeeded",
|
||||||
|
dict(backup_summary),
|
||||||
|
)
|
||||||
journal.migration_started()
|
journal.migration_started()
|
||||||
_run([*compose, "run", "--rm", "migrate"], cwd=paths.root)
|
_run([*compose, "run", "--rm", "migrate"], cwd=paths.root)
|
||||||
journal.migration_completed()
|
journal.migration_completed()
|
||||||
@@ -898,6 +983,108 @@ def _verify_offline_images(args: argparse.Namespace) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_backup(args: argparse.Namespace) -> int:
|
||||||
|
paths = bundle_paths(args.directory)
|
||||||
|
ensure_private_directory(paths.root)
|
||||||
|
with _deployment_lock(paths.lock):
|
||||||
|
return _verify_backup_locked(args, paths)
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_backup_locked(args: argparse.Namespace, paths) -> int:
|
||||||
|
spec = load_spec(paths.spec)
|
||||||
|
expected_digest = str(args.evidence_sha256 or "").strip().lower()
|
||||||
|
if len(expected_digest) != 64 or any(
|
||||||
|
character not in "0123456789abcdef" for character in expected_digest
|
||||||
|
):
|
||||||
|
raise ValueError("--evidence-sha256 must be a lowercase SHA-256 digest")
|
||||||
|
if args.evidence_url:
|
||||||
|
encoded_evidence = fetch_bounded_https(
|
||||||
|
str(args.evidence_url),
|
||||||
|
maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES,
|
||||||
|
allow_private_host=args.allow_private_evidence_host,
|
||||||
|
)
|
||||||
|
evidence = decode_json_bytes(encoded_evidence, label="backup evidence")
|
||||||
|
else:
|
||||||
|
evidence_path = args.evidence.expanduser().resolve()
|
||||||
|
encoded_evidence = read_bounded_bytes(
|
||||||
|
evidence_path,
|
||||||
|
maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES,
|
||||||
|
)
|
||||||
|
evidence = decode_json_bytes(encoded_evidence, label="backup evidence")
|
||||||
|
if encoded_evidence != canonical_distribution_json(evidence):
|
||||||
|
raise DistributionError("backup evidence is not canonical JSON")
|
||||||
|
if hashlib.sha256(encoded_evidence).hexdigest() != expected_digest:
|
||||||
|
raise DistributionError("backup evidence SHA-256 does not match")
|
||||||
|
keyring_path = args.trusted_keyring.expanduser().resolve()
|
||||||
|
keyring = load_bounded_json(
|
||||||
|
keyring_path,
|
||||||
|
maximum_bytes=MAX_BACKUP_KEYRING_BYTES,
|
||||||
|
)
|
||||||
|
encoded_keyring = canonical_distribution_json(keyring)
|
||||||
|
receipt = _read_json_object(paths.receipt)
|
||||||
|
previous_release = receipt.get("release") if receipt else None
|
||||||
|
release: Mapping[str, object] = (
|
||||||
|
previous_release
|
||||||
|
if isinstance(previous_release, Mapping)
|
||||||
|
else {
|
||||||
|
"channel": spec.release.channel,
|
||||||
|
"version": spec.release.version,
|
||||||
|
"manifest_sha256": spec.release.manifest_sha256,
|
||||||
|
"composition_sha256": spec.release.composition_sha256,
|
||||||
|
"api_image": spec.release.api_image,
|
||||||
|
"web_image": spec.release.web_image,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
summary = verify_backup_evidence(
|
||||||
|
evidence,
|
||||||
|
keyring,
|
||||||
|
installation_id=spec.installation_id,
|
||||||
|
profile=spec.profile,
|
||||||
|
release=release,
|
||||||
|
max_age_seconds=DEFAULT_MAX_BACKUP_AGE_SECONDS,
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
"Verified coordinated recovery point "
|
||||||
|
f"{summary['recovery_point_id']} with restore drill "
|
||||||
|
f"{summary['restore_drill_id']} and trusted key "
|
||||||
|
f"{summary['signature_key_id']}."
|
||||||
|
)
|
||||||
|
if not args.adopt:
|
||||||
|
return 0
|
||||||
|
verification = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"evidence_sha256": expected_digest,
|
||||||
|
"keyring_sha256": hashlib.sha256(encoded_keyring).hexdigest(),
|
||||||
|
"signature_key_id": summary["signature_key_id"],
|
||||||
|
"verified_at": _now(),
|
||||||
|
"evidence_id": summary["evidence_id"],
|
||||||
|
"recovery_point_id": summary["recovery_point_id"],
|
||||||
|
"restore_drill_id": summary["restore_drill_id"],
|
||||||
|
"release_manifest_sha256": release.get("manifest_sha256"),
|
||||||
|
"captured_at": summary["captured_at"],
|
||||||
|
"expires_at": summary["expires_at"],
|
||||||
|
"restore_started_at": summary["restore_started_at"],
|
||||||
|
"restore_completed_at": summary["restore_completed_at"],
|
||||||
|
"measured_rpo_seconds": summary["measured_rpo_seconds"],
|
||||||
|
"measured_rto_seconds": summary["measured_rto_seconds"],
|
||||||
|
"component_count": summary["component_count"],
|
||||||
|
}
|
||||||
|
atomic_write(paths.backup_evidence, encoded_evidence, mode=0o600)
|
||||||
|
atomic_write(paths.backup_keyring, encoded_keyring, mode=0o600)
|
||||||
|
atomic_write(
|
||||||
|
paths.backup_verification,
|
||||||
|
canonical_json(verification),
|
||||||
|
mode=0o600,
|
||||||
|
)
|
||||||
|
_write_bundle(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
reconcile_runtime_environment(spec, read_env(paths.env)),
|
||||||
|
)
|
||||||
|
print(f"Adopted signed backup evidence in {paths.root}.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def _selected_dependency_images(
|
def _selected_dependency_images(
|
||||||
spec: InstallationSpec,
|
spec: InstallationSpec,
|
||||||
*,
|
*,
|
||||||
@@ -929,6 +1116,30 @@ def _render_kubernetes(args: argparse.Namespace) -> int:
|
|||||||
paths = bundle_paths(args.directory)
|
paths = bundle_paths(args.directory)
|
||||||
spec = load_spec(paths.spec)
|
spec = load_spec(paths.spec)
|
||||||
environment = reconcile_runtime_environment(spec, read_env(paths.env))
|
environment = reconcile_runtime_environment(spec, read_env(paths.env))
|
||||||
|
environment.update(_backup_runtime_environment(spec, paths))
|
||||||
|
receipt = _read_json_object(paths.receipt)
|
||||||
|
backup_required = release_change_requires_backup(spec, receipt)
|
||||||
|
backup_summary: Mapping[str, object] | None = None
|
||||||
|
evidence_files = (
|
||||||
|
paths.backup_evidence,
|
||||||
|
paths.backup_keyring,
|
||||||
|
paths.backup_verification,
|
||||||
|
)
|
||||||
|
if backup_required:
|
||||||
|
backup_summary = verify_stored_backup_evidence(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
receipt=receipt,
|
||||||
|
)
|
||||||
|
elif all(path.is_file() for path in evidence_files):
|
||||||
|
try:
|
||||||
|
backup_summary = verify_stored_backup_evidence(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
receipt=receipt,
|
||||||
|
)
|
||||||
|
except (DistributionError, OSError):
|
||||||
|
backup_summary = None
|
||||||
manifest = render_kubernetes(
|
manifest = render_kubernetes(
|
||||||
spec,
|
spec,
|
||||||
environment,
|
environment,
|
||||||
@@ -936,6 +1147,8 @@ def _render_kubernetes(args: argparse.Namespace) -> int:
|
|||||||
secret_name=args.secret_name,
|
secret_name=args.secret_name,
|
||||||
tls_secret_name=args.tls_secret_name,
|
tls_secret_name=args.tls_secret_name,
|
||||||
ingress_class_name=args.ingress_class_name,
|
ingress_class_name=args.ingress_class_name,
|
||||||
|
backup_required=backup_required,
|
||||||
|
backup_evidence=backup_summary,
|
||||||
)
|
)
|
||||||
output = (args.output or (paths.root / "kubernetes.json")).expanduser().resolve()
|
output = (args.output or (paths.root / "kubernetes.json")).expanduser().resolve()
|
||||||
atomic_write(output, canonical_json(manifest), mode=0o600)
|
atomic_write(output, canonical_json(manifest), mode=0o600)
|
||||||
@@ -1027,6 +1240,7 @@ def _deployment_receipt(
|
|||||||
return {
|
return {
|
||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
"installation_id": spec.installation_id,
|
"installation_id": spec.installation_id,
|
||||||
|
"profile": spec.profile,
|
||||||
"applied_at": _now(),
|
"applied_at": _now(),
|
||||||
"spec_sha256": digest_json(spec.to_dict()),
|
"spec_sha256": digest_json(spec.to_dict()),
|
||||||
"compose_sha256": digest_json(render_compose(spec)),
|
"compose_sha256": digest_json(render_compose(spec)),
|
||||||
@@ -1064,6 +1278,16 @@ def _deployment_receipt(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _read_json_object(path: Path) -> dict[str, object]:
|
||||||
|
if not path.is_file():
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
value = load_bounded_json(path, maximum_bytes=64 * 1024)
|
||||||
|
except (DistributionError, OSError):
|
||||||
|
return {}
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
def _updated_spec(
|
def _updated_spec(
|
||||||
current: InstallationSpec, args: argparse.Namespace
|
current: InstallationSpec, args: argparse.Namespace
|
||||||
) -> InstallationSpec:
|
) -> InstallationSpec:
|
||||||
@@ -1222,10 +1446,12 @@ def _write_bundle(
|
|||||||
spec: InstallationSpec,
|
spec: InstallationSpec,
|
||||||
paths,
|
paths,
|
||||||
secrets: Mapping[str, str],
|
secrets: Mapping[str, str],
|
||||||
) -> None:
|
) -> dict[str, str]:
|
||||||
ensure_private_directory(paths.root)
|
ensure_private_directory(paths.root)
|
||||||
|
runtime_environment = dict(secrets)
|
||||||
|
runtime_environment.update(_backup_runtime_environment(spec, paths))
|
||||||
atomic_write(paths.spec, canonical_json(spec.to_dict()), mode=0o600)
|
atomic_write(paths.spec, canonical_json(spec.to_dict()), mode=0o600)
|
||||||
write_env(paths.env, secrets)
|
write_env(paths.env, runtime_environment)
|
||||||
atomic_write(paths.compose, canonical_json(render_compose(spec)), mode=0o600)
|
atomic_write(paths.compose, canonical_json(render_compose(spec)), mode=0o600)
|
||||||
atomic_write(
|
atomic_write(
|
||||||
paths.load_balancer_config,
|
paths.load_balancer_config,
|
||||||
@@ -1247,6 +1473,62 @@ def _write_bundle(
|
|||||||
render_garage_config().encode("utf-8"),
|
render_garage_config().encode("utf-8"),
|
||||||
mode=0o644,
|
mode=0o644,
|
||||||
)
|
)
|
||||||
|
return dict(sorted(runtime_environment.items()))
|
||||||
|
|
||||||
|
|
||||||
|
def _backup_runtime_environment(
|
||||||
|
spec: InstallationSpec,
|
||||||
|
paths,
|
||||||
|
) -> dict[str, str]:
|
||||||
|
values = {key: "" for key in BACKUP_RUNTIME_ENV_KEYS}
|
||||||
|
evidence_files = (
|
||||||
|
paths.backup_evidence,
|
||||||
|
paths.backup_keyring,
|
||||||
|
paths.backup_verification,
|
||||||
|
)
|
||||||
|
if not any(path.is_file() for path in evidence_files):
|
||||||
|
values["GOVOPLAN_BACKUP_EVIDENCE_STATE"] = "absent"
|
||||||
|
return values
|
||||||
|
if not all(path.is_file() for path in evidence_files):
|
||||||
|
values["GOVOPLAN_BACKUP_EVIDENCE_STATE"] = "invalid"
|
||||||
|
return values
|
||||||
|
verification = _read_json_object(paths.backup_verification)
|
||||||
|
try:
|
||||||
|
summary = verify_stored_backup_evidence(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
receipt=_read_json_object(paths.receipt),
|
||||||
|
)
|
||||||
|
except (DistributionError, OSError):
|
||||||
|
values["GOVOPLAN_BACKUP_EVIDENCE_STATE"] = "invalid"
|
||||||
|
return values
|
||||||
|
values.update(
|
||||||
|
{
|
||||||
|
"GOVOPLAN_BACKUP_EVIDENCE_STATE": "verified",
|
||||||
|
"GOVOPLAN_BACKUP_EVIDENCE_ID": str(summary["evidence_id"]),
|
||||||
|
"GOVOPLAN_BACKUP_RECOVERY_POINT_ID": str(summary["recovery_point_id"]),
|
||||||
|
"GOVOPLAN_BACKUP_RESTORE_DRILL_ID": str(summary["restore_drill_id"]),
|
||||||
|
"GOVOPLAN_BACKUP_EVIDENCE_SHA256": str(summary["evidence_sha256"]),
|
||||||
|
"GOVOPLAN_BACKUP_RELEASE_MANIFEST_SHA256": str(
|
||||||
|
verification["release_manifest_sha256"]
|
||||||
|
),
|
||||||
|
"GOVOPLAN_BACKUP_CAPTURED_AT": str(summary["captured_at"]),
|
||||||
|
"GOVOPLAN_BACKUP_EXPIRES_AT": str(summary["expires_at"]),
|
||||||
|
"GOVOPLAN_BACKUP_RESTORE_STARTED_AT": str(summary["restore_started_at"]),
|
||||||
|
"GOVOPLAN_BACKUP_RESTORE_COMPLETED_AT": str(
|
||||||
|
summary["restore_completed_at"]
|
||||||
|
),
|
||||||
|
"GOVOPLAN_BACKUP_VERIFIED_AT": str(verification["verified_at"]),
|
||||||
|
"GOVOPLAN_BACKUP_MEASURED_RPO_SECONDS": str(
|
||||||
|
summary["measured_rpo_seconds"]
|
||||||
|
),
|
||||||
|
"GOVOPLAN_BACKUP_MEASURED_RTO_SECONDS": str(
|
||||||
|
summary["measured_rto_seconds"]
|
||||||
|
),
|
||||||
|
"GOVOPLAN_BACKUP_COMPONENT_COUNT": str(summary["component_count"]),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
def _write_plan(path: Path, plan: DeploymentPlan) -> None:
|
def _write_plan(path: Path, plan: DeploymentPlan) -> None:
|
||||||
|
|||||||
@@ -74,7 +74,9 @@ def read_bounded_bytes(path: Path, *, maximum_bytes: int) -> bytes:
|
|||||||
try:
|
try:
|
||||||
opened = os.fstat(descriptor)
|
opened = os.fstat(descriptor)
|
||||||
if not stat.S_ISREG(opened.st_mode) or opened.st_size > maximum_bytes:
|
if not stat.S_ISREG(opened.st_mode) or opened.st_size > maximum_bytes:
|
||||||
raise DistributionError(f"trusted JSON file is invalid or too large: {path}")
|
raise DistributionError(
|
||||||
|
f"trusted JSON file is invalid or too large: {path}"
|
||||||
|
)
|
||||||
chunks: list[bytes] = []
|
chunks: list[bytes] = []
|
||||||
total = 0
|
total = 0
|
||||||
while True:
|
while True:
|
||||||
@@ -109,7 +111,9 @@ def fetch_bounded_https(
|
|||||||
if parsed.scheme != "https" or not parsed.hostname:
|
if parsed.scheme != "https" or not parsed.hostname:
|
||||||
raise DistributionError("distribution downloads require an absolute HTTPS URL")
|
raise DistributionError("distribution downloads require an absolute HTTPS URL")
|
||||||
if parsed.username or parsed.password or parsed.fragment:
|
if parsed.username or parsed.password or parsed.fragment:
|
||||||
raise DistributionError("distribution URL must not contain credentials or a fragment")
|
raise DistributionError(
|
||||||
|
"distribution URL must not contain credentials or a fragment"
|
||||||
|
)
|
||||||
if not allow_private_host:
|
if not allow_private_host:
|
||||||
_require_public_host(parsed.hostname)
|
_require_public_host(parsed.hostname)
|
||||||
request = Request(url, headers={"Accept": "application/json"})
|
request = Request(url, headers={"Accept": "application/json"})
|
||||||
@@ -172,9 +176,11 @@ def validate_manifest(
|
|||||||
raise DistributionError(
|
raise DistributionError(
|
||||||
f"distribution channel is {channel!r}, expected {expected_channel!r}"
|
f"distribution channel is {channel!r}, expected {expected_channel!r}"
|
||||||
)
|
)
|
||||||
if isinstance(payload.get("sequence"), bool) or not isinstance(
|
if (
|
||||||
payload.get("sequence"), int
|
isinstance(payload.get("sequence"), bool)
|
||||||
) or int(payload["sequence"]) < 1:
|
or not isinstance(payload.get("sequence"), int)
|
||||||
|
or int(payload["sequence"]) < 1
|
||||||
|
):
|
||||||
raise DistributionError("distribution sequence must be a positive integer")
|
raise DistributionError("distribution sequence must be a positive integer")
|
||||||
_token(payload.get("version"), "version", maximum=128, pattern=TOKEN)
|
_token(payload.get("version"), "version", maximum=128, pattern=TOKEN)
|
||||||
issued = _datetime(payload.get("issued_at"), "issued_at")
|
issued = _datetime(payload.get("issued_at"), "issued_at")
|
||||||
@@ -283,11 +289,41 @@ def verify_manifest(
|
|||||||
) -> str:
|
) -> str:
|
||||||
current = (now or datetime.now(UTC)).astimezone(UTC)
|
current = (now or datetime.now(UTC)).astimezone(UTC)
|
||||||
validate_manifest(payload, expected_channel=expected_channel, now=current)
|
validate_manifest(payload, expected_channel=expected_channel, now=current)
|
||||||
keys = _trusted_keys(keyring, now=current)
|
return verify_signed_document(
|
||||||
|
payload,
|
||||||
|
keyring,
|
||||||
|
purpose="govoplan-runtime-distribution",
|
||||||
|
label="distribution",
|
||||||
|
now=current,
|
||||||
|
openssl=openssl,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def verify_signed_document(
|
||||||
|
payload: Mapping[str, Any],
|
||||||
|
keyring: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
purpose: str,
|
||||||
|
label: str,
|
||||||
|
now: datetime,
|
||||||
|
openssl: str = "openssl",
|
||||||
|
) -> str:
|
||||||
|
keys = _trusted_keys(keyring, now=now, purpose=purpose, label=label)
|
||||||
signed = canonical_signed_payload(payload)
|
signed = canonical_signed_payload(payload)
|
||||||
failures: list[str] = []
|
failures: list[str] = []
|
||||||
for item in payload["signatures"]:
|
signatures = payload.get("signatures")
|
||||||
|
if not isinstance(signatures, list) or not signatures:
|
||||||
|
raise DistributionError(f"{label} has no signatures")
|
||||||
|
for index, raw in enumerate(signatures):
|
||||||
|
item = _object(raw, f"{label}.signatures[{index}]")
|
||||||
|
_exact_keys(
|
||||||
|
item,
|
||||||
|
required={"key_id", "algorithm", "value"},
|
||||||
|
label=f"{label}.signatures[{index}]",
|
||||||
|
)
|
||||||
key_id = str(item["key_id"])
|
key_id = str(item["key_id"])
|
||||||
|
if KEY_ID.fullmatch(key_id) is None or item.get("algorithm") != "ed25519":
|
||||||
|
raise DistributionError(f"{label} signature is invalid")
|
||||||
public_key = keys.get(key_id)
|
public_key = keys.get(key_id)
|
||||||
if public_key is None:
|
if public_key is None:
|
||||||
continue
|
continue
|
||||||
@@ -303,8 +339,10 @@ def verify_manifest(
|
|||||||
failures.append(f"{key_id}: {exc}")
|
failures.append(f"{key_id}: {exc}")
|
||||||
continue
|
continue
|
||||||
return key_id
|
return key_id
|
||||||
detail = "; ".join(failures) if failures else "no signature used an active trusted key"
|
detail = (
|
||||||
raise DistributionError(f"distribution signature verification failed: {detail}")
|
"; ".join(failures) if failures else "no signature used an active trusted key"
|
||||||
|
)
|
||||||
|
raise DistributionError(f"{label} signature verification failed: {detail}")
|
||||||
|
|
||||||
|
|
||||||
def verify_manifest_binding(
|
def verify_manifest_binding(
|
||||||
@@ -319,7 +357,9 @@ def verify_manifest_binding(
|
|||||||
dependencies: Mapping[str, str],
|
dependencies: Mapping[str, str],
|
||||||
) -> None:
|
) -> None:
|
||||||
if payload.get("channel") != channel or payload.get("version") != version:
|
if payload.get("channel") != channel or payload.get("version") != version:
|
||||||
raise DistributionError("stored manifest does not match release channel/version")
|
raise DistributionError(
|
||||||
|
"stored manifest does not match release channel/version"
|
||||||
|
)
|
||||||
images = _object(payload.get("images"), "images")
|
images = _object(payload.get("images"), "images")
|
||||||
if _object(images.get("api"), "images.api").get("index") != api_image:
|
if _object(images.get("api"), "images.api").get("index") != api_image:
|
||||||
raise DistributionError("stored manifest does not match API image")
|
raise DistributionError("stored manifest does not match API image")
|
||||||
@@ -372,9 +412,9 @@ def verify_offline_image_index(
|
|||||||
archive = root / archive_relative
|
archive = root / archive_relative
|
||||||
if reference in references:
|
if reference in references:
|
||||||
raise DistributionError("offline image index contains duplicate references")
|
raise DistributionError("offline image index contains duplicate references")
|
||||||
if _sha256_regular_file(archive, maximum_bytes=MAX_OFFLINE_IMAGE_BYTES) != _sha256(
|
if _sha256_regular_file(
|
||||||
value.get("sha256"), "offline image sha256"
|
archive, maximum_bytes=MAX_OFFLINE_IMAGE_BYTES
|
||||||
):
|
) != _sha256(value.get("sha256"), "offline image sha256"):
|
||||||
raise DistributionError(f"offline image archive digest mismatch: {archive}")
|
raise DistributionError(f"offline image archive digest mismatch: {archive}")
|
||||||
references[reference] = archive
|
references[reference] = archive
|
||||||
missing = sorted(set(expected_references) - set(references))
|
missing = sorted(set(expected_references) - set(references))
|
||||||
@@ -418,19 +458,21 @@ def _trusted_keys(
|
|||||||
keyring: Mapping[str, Any],
|
keyring: Mapping[str, Any],
|
||||||
*,
|
*,
|
||||||
now: datetime,
|
now: datetime,
|
||||||
|
purpose: str,
|
||||||
|
label: str,
|
||||||
) -> dict[str, str]:
|
) -> dict[str, str]:
|
||||||
_exact_keys(
|
_exact_keys(
|
||||||
keyring,
|
keyring,
|
||||||
required={"schema_version", "purpose", "keys"},
|
required={"schema_version", "purpose", "keys"},
|
||||||
label="distribution keyring",
|
label=f"{label} keyring",
|
||||||
)
|
)
|
||||||
if keyring.get("schema_version") != "1":
|
if keyring.get("schema_version") != "1":
|
||||||
raise DistributionError("unsupported distribution keyring schema_version")
|
raise DistributionError(f"unsupported {label} keyring schema_version")
|
||||||
if keyring.get("purpose") != "govoplan-runtime-distribution":
|
if keyring.get("purpose") != purpose:
|
||||||
raise DistributionError("distribution keyring has the wrong purpose")
|
raise DistributionError(f"{label} keyring has the wrong purpose")
|
||||||
values = keyring.get("keys")
|
values = keyring.get("keys")
|
||||||
if not isinstance(values, list) or not values:
|
if not isinstance(values, list) or not values:
|
||||||
raise DistributionError("distribution keyring contains no keys")
|
raise DistributionError(f"{label} keyring contains no keys")
|
||||||
trusted: dict[str, str] = {}
|
trusted: dict[str, str] = {}
|
||||||
for index, item in enumerate(values):
|
for index, item in enumerate(values):
|
||||||
key = _object(item, f"keyring.keys[{index}]")
|
key = _object(item, f"keyring.keys[{index}]")
|
||||||
@@ -453,11 +495,11 @@ def _trusted_keys(
|
|||||||
pattern=KEY_ID,
|
pattern=KEY_ID,
|
||||||
)
|
)
|
||||||
if key_id in trusted:
|
if key_id in trusted:
|
||||||
raise DistributionError("distribution keyring contains duplicate key ids")
|
raise DistributionError(f"{label} keyring contains duplicate key ids")
|
||||||
if key.get("algorithm") != "ed25519":
|
if key.get("algorithm") != "ed25519":
|
||||||
raise DistributionError("distribution key must use ed25519")
|
raise DistributionError(f"{label} key must use ed25519")
|
||||||
if key.get("status") not in {"active", "retired", "revoked"}:
|
if key.get("status") not in {"active", "retired", "revoked"}:
|
||||||
raise DistributionError("distribution key has an invalid status")
|
raise DistributionError(f"{label} key has an invalid status")
|
||||||
not_before = _datetime(key.get("not_before"), "key.not_before")
|
not_before = _datetime(key.get("not_before"), "key.not_before")
|
||||||
expires = _datetime(key.get("expires_at"), "key.expires_at")
|
expires = _datetime(key.get("expires_at"), "key.expires_at")
|
||||||
public_key = key.get("public_key_pem")
|
public_key = key.get("public_key_pem")
|
||||||
@@ -466,11 +508,11 @@ def _trusted_keys(
|
|||||||
or len(public_key.encode("utf-8")) > 8192
|
or len(public_key.encode("utf-8")) > 8192
|
||||||
or "BEGIN PUBLIC KEY" not in public_key
|
or "BEGIN PUBLIC KEY" not in public_key
|
||||||
):
|
):
|
||||||
raise DistributionError("distribution key has an invalid public key")
|
raise DistributionError(f"{label} key has an invalid public key")
|
||||||
if key.get("status") == "active" and not_before <= now < expires:
|
if key.get("status") == "active" and not_before <= now < expires:
|
||||||
trusted[key_id] = public_key
|
trusted[key_id] = public_key
|
||||||
if not trusted:
|
if not trusted:
|
||||||
raise DistributionError("distribution keyring has no currently active keys")
|
raise DistributionError(f"{label} keyring has no currently active keys")
|
||||||
return trusted
|
return trusted
|
||||||
|
|
||||||
|
|
||||||
@@ -534,13 +576,17 @@ def _require_public_host(hostname: str) -> None:
|
|||||||
for value in socket.getaddrinfo(hostname, 443, type=socket.SOCK_STREAM)
|
for value in socket.getaddrinfo(hostname, 443, type=socket.SOCK_STREAM)
|
||||||
}
|
}
|
||||||
except OSError as exc:
|
except OSError as exc:
|
||||||
raise DistributionError(f"distribution host cannot be resolved: {hostname}") from exc
|
raise DistributionError(
|
||||||
|
f"distribution host cannot be resolved: {hostname}"
|
||||||
|
) from exc
|
||||||
if not addresses:
|
if not addresses:
|
||||||
raise DistributionError("distribution host resolved to no addresses")
|
raise DistributionError("distribution host resolved to no addresses")
|
||||||
for value in addresses:
|
for value in addresses:
|
||||||
address = ipaddress.ip_address(value)
|
address = ipaddress.ip_address(value)
|
||||||
if not address.is_global:
|
if not address.is_global:
|
||||||
raise DistributionError("distribution host resolves to a non-public address")
|
raise DistributionError(
|
||||||
|
"distribution host resolves to a non-public address"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _sha256_regular_file(path: Path, *, maximum_bytes: int) -> str:
|
def _sha256_regular_file(path: Path, *, maximum_bytes: int) -> str:
|
||||||
@@ -553,7 +599,9 @@ def _sha256_regular_file(path: Path, *, maximum_bytes: int) -> str:
|
|||||||
try:
|
try:
|
||||||
opened = os.fstat(descriptor)
|
opened = os.fstat(descriptor)
|
||||||
if not stat.S_ISREG(opened.st_mode) or opened.st_size > maximum_bytes:
|
if not stat.S_ISREG(opened.st_mode) or opened.st_size > maximum_bytes:
|
||||||
raise DistributionError(f"immutable artifact is invalid or too large: {path}")
|
raise DistributionError(
|
||||||
|
f"immutable artifact is invalid or too large: {path}"
|
||||||
|
)
|
||||||
while True:
|
while True:
|
||||||
chunk = os.read(descriptor, 1024 * 1024)
|
chunk = os.read(descriptor, 1024 * 1024)
|
||||||
if not chunk:
|
if not chunk:
|
||||||
@@ -602,7 +650,11 @@ def _token(
|
|||||||
maximum: int,
|
maximum: int,
|
||||||
pattern: re.Pattern[str],
|
pattern: re.Pattern[str],
|
||||||
) -> str:
|
) -> str:
|
||||||
if not isinstance(value, str) or len(value) > maximum or pattern.fullmatch(value) is None:
|
if (
|
||||||
|
not isinstance(value, str)
|
||||||
|
or len(value) > maximum
|
||||||
|
or pattern.fullmatch(value) is None
|
||||||
|
):
|
||||||
raise DistributionError(f"{label} is invalid")
|
raise DistributionError(f"{label} is invalid")
|
||||||
return value
|
return value
|
||||||
|
|
||||||
@@ -626,7 +678,11 @@ def _sha256(value: object, label: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _digest_image(value: object, label: str) -> str:
|
def _digest_image(value: object, label: str) -> str:
|
||||||
if not isinstance(value, str) or len(value) > 300 or DIGEST_IMAGE.fullmatch(value) is None:
|
if (
|
||||||
|
not isinstance(value, str)
|
||||||
|
or len(value) > 300
|
||||||
|
or DIGEST_IMAGE.fullmatch(value) is None
|
||||||
|
):
|
||||||
raise DistributionError(f"{label} must be an OCI image pinned by sha256")
|
raise DistributionError(f"{label} must be an OCI image pinned by sha256")
|
||||||
return value
|
return value
|
||||||
|
|
||||||
@@ -635,7 +691,12 @@ def _https_url(value: object, label: str) -> str:
|
|||||||
if not isinstance(value, str) or len(value) > 2048:
|
if not isinstance(value, str) or len(value) > 2048:
|
||||||
raise DistributionError(f"{label} must be an HTTPS URL")
|
raise DistributionError(f"{label} must be an HTTPS URL")
|
||||||
parsed = urlsplit(value)
|
parsed = urlsplit(value)
|
||||||
if parsed.scheme != "https" or not parsed.netloc or parsed.username or parsed.password:
|
if (
|
||||||
|
parsed.scheme != "https"
|
||||||
|
or not parsed.netloc
|
||||||
|
or parsed.username
|
||||||
|
or parsed.password
|
||||||
|
):
|
||||||
raise DistributionError(f"{label} must be an HTTPS URL without credentials")
|
raise DistributionError(f"{label} must be an HTTPS URL without credentials")
|
||||||
return value
|
return value
|
||||||
|
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import re
|
|||||||
from typing import Any, Mapping
|
from typing import Any, Mapping
|
||||||
from urllib.parse import urlsplit
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from .bundle import BACKUP_RUNTIME_ENV_KEYS
|
||||||
from .model import InstallationSpec, image_is_digest_pinned
|
from .model import InstallationSpec, image_is_digest_pinned
|
||||||
|
|
||||||
|
|
||||||
@@ -55,6 +56,7 @@ _CONFIG_KEYS = (
|
|||||||
"FILE_STORAGE_S3_BUCKET",
|
"FILE_STORAGE_S3_BUCKET",
|
||||||
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
|
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
|
||||||
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
|
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
|
||||||
|
*BACKUP_RUNTIME_ENV_KEYS,
|
||||||
)
|
)
|
||||||
_QUEUE_NAME = re.compile(r"^[a-z][a-z0-9_.-]{0,63}$")
|
_QUEUE_NAME = re.compile(r"^[a-z][a-z0-9_.-]{0,63}$")
|
||||||
|
|
||||||
@@ -75,6 +77,8 @@ def render_kubernetes(
|
|||||||
secret_name: str = "govoplan-runtime",
|
secret_name: str = "govoplan-runtime",
|
||||||
tls_secret_name: str = "govoplan-tls",
|
tls_secret_name: str = "govoplan-tls",
|
||||||
ingress_class_name: str | None = None,
|
ingress_class_name: str | None = None,
|
||||||
|
backup_required: bool = True,
|
||||||
|
backup_evidence: Mapping[str, object] | None = None,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Render runtime roles only; shared state services stay externally managed."""
|
"""Render runtime roles only; shared state services stay externally managed."""
|
||||||
|
|
||||||
@@ -199,6 +203,8 @@ def render_kubernetes(
|
|||||||
environment,
|
environment,
|
||||||
"MIGRATION",
|
"MIGRATION",
|
||||||
),
|
),
|
||||||
|
backup_required=backup_required,
|
||||||
|
backup_evidence=backup_evidence,
|
||||||
),
|
),
|
||||||
]
|
]
|
||||||
for pool in worker_pools:
|
for pool in worker_pools:
|
||||||
@@ -765,9 +771,28 @@ def _migration_job(
|
|||||||
secret_name: str,
|
secret_name: str,
|
||||||
service_account: str,
|
service_account: str,
|
||||||
database_environment: Mapping[str, str],
|
database_environment: Mapping[str, str],
|
||||||
|
backup_required: bool,
|
||||||
|
backup_evidence: Mapping[str, object] | None,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
job_labels = {**labels, "app.kubernetes.io/component": "migration"}
|
job_labels = {**labels, "app.kubernetes.io/component": "migration"}
|
||||||
job_name = _name_with_suffix(name, f"migrate-{release_key}")
|
job_name = _name_with_suffix(name, f"migrate-{release_key}")
|
||||||
|
backup_annotations = {
|
||||||
|
"govoplan.add-ideas.de/backup-required": str(backup_required).lower(),
|
||||||
|
}
|
||||||
|
if backup_evidence is not None:
|
||||||
|
backup_annotations.update(
|
||||||
|
{
|
||||||
|
"govoplan.add-ideas.de/backup-evidence-sha256": str(
|
||||||
|
backup_evidence["evidence_sha256"]
|
||||||
|
),
|
||||||
|
"govoplan.add-ideas.de/recovery-point": str(
|
||||||
|
backup_evidence["recovery_point_id"]
|
||||||
|
),
|
||||||
|
"govoplan.add-ideas.de/restore-drill": str(
|
||||||
|
backup_evidence["restore_drill_id"]
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
return {
|
return {
|
||||||
"apiVersion": "batch/v1",
|
"apiVersion": "batch/v1",
|
||||||
"kind": "Job",
|
"kind": "Job",
|
||||||
@@ -777,7 +802,7 @@ def _migration_job(
|
|||||||
"labels": job_labels,
|
"labels": job_labels,
|
||||||
"annotations": {
|
"annotations": {
|
||||||
"govoplan.add-ideas.de/recovery-mode": "forward-recovery",
|
"govoplan.add-ideas.de/recovery-mode": "forward-recovery",
|
||||||
"govoplan.add-ideas.de/backup-required": "true",
|
**backup_annotations,
|
||||||
"argocd.argoproj.io/sync-wave": "-1",
|
"argocd.argoproj.io/sync-wave": "-1",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from dataclasses import asdict, dataclass
|
from dataclasses import asdict, dataclass
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -18,6 +19,11 @@ from urllib.error import HTTPError, URLError
|
|||||||
from urllib.parse import urlsplit
|
from urllib.parse import urlsplit
|
||||||
from urllib.request import Request, urlopen
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
|
from .backup_evidence import (
|
||||||
|
MAX_BACKUP_EVIDENCE_BYTES,
|
||||||
|
MAX_BACKUP_KEYRING_BYTES,
|
||||||
|
verify_backup_evidence,
|
||||||
|
)
|
||||||
from .bundle import (
|
from .bundle import (
|
||||||
BundlePaths,
|
BundlePaths,
|
||||||
canonical_json,
|
canonical_json,
|
||||||
@@ -33,8 +39,11 @@ from .distribution import (
|
|||||||
MAX_KEYRING_BYTES,
|
MAX_KEYRING_BYTES,
|
||||||
MAX_MANIFEST_BYTES,
|
MAX_MANIFEST_BYTES,
|
||||||
DistributionError,
|
DistributionError,
|
||||||
|
canonical_json as canonical_distribution_json,
|
||||||
|
decode_json_bytes,
|
||||||
file_sha256,
|
file_sha256,
|
||||||
load_bounded_json,
|
load_bounded_json,
|
||||||
|
read_bounded_bytes,
|
||||||
verify_manifest,
|
verify_manifest,
|
||||||
verify_manifest_binding,
|
verify_manifest_binding,
|
||||||
)
|
)
|
||||||
@@ -232,6 +241,9 @@ def static_checks(spec: InstallationSpec, paths: BundlePaths) -> tuple[Check, ..
|
|||||||
)
|
)
|
||||||
|
|
||||||
checks.extend(_distribution_checks(spec, paths))
|
checks.extend(_distribution_checks(spec, paths))
|
||||||
|
checks.extend(
|
||||||
|
_backup_evidence_checks(spec, paths, receipt=_read_receipt(paths.receipt))
|
||||||
|
)
|
||||||
|
|
||||||
values = read_env(paths.env)
|
values = read_env(paths.env)
|
||||||
required = {"MASTER_KEY_B64", "DATABASE_URL"}
|
required = {"MASTER_KEY_B64", "DATABASE_URL"}
|
||||||
@@ -350,6 +362,189 @@ def static_checks(spec: InstallationSpec, paths: BundlePaths) -> tuple[Check, ..
|
|||||||
return tuple(checks)
|
return tuple(checks)
|
||||||
|
|
||||||
|
|
||||||
|
def release_change_requires_backup(
|
||||||
|
spec: InstallationSpec,
|
||||||
|
receipt: Mapping[str, object],
|
||||||
|
) -> bool:
|
||||||
|
if spec.profile != "self-hosted" or not receipt:
|
||||||
|
return False
|
||||||
|
previous = receipt.get("release")
|
||||||
|
if not isinstance(previous, Mapping):
|
||||||
|
return True
|
||||||
|
desired = {
|
||||||
|
"channel": spec.release.channel,
|
||||||
|
"version": spec.release.version,
|
||||||
|
"manifest_sha256": spec.release.manifest_sha256,
|
||||||
|
"composition_sha256": spec.release.composition_sha256,
|
||||||
|
"api_image": spec.release.api_image,
|
||||||
|
"web_image": spec.release.web_image,
|
||||||
|
}
|
||||||
|
return any(previous.get(key) != value for key, value in desired.items())
|
||||||
|
|
||||||
|
|
||||||
|
def verify_stored_backup_evidence(
|
||||||
|
spec: InstallationSpec,
|
||||||
|
paths: BundlePaths,
|
||||||
|
*,
|
||||||
|
receipt: Mapping[str, object],
|
||||||
|
) -> dict[str, object]:
|
||||||
|
verification = load_bounded_json(
|
||||||
|
paths.backup_verification,
|
||||||
|
maximum_bytes=64 * 1024,
|
||||||
|
)
|
||||||
|
expected_fields = {
|
||||||
|
"schema_version",
|
||||||
|
"evidence_sha256",
|
||||||
|
"keyring_sha256",
|
||||||
|
"signature_key_id",
|
||||||
|
"verified_at",
|
||||||
|
"evidence_id",
|
||||||
|
"recovery_point_id",
|
||||||
|
"restore_drill_id",
|
||||||
|
"release_manifest_sha256",
|
||||||
|
"captured_at",
|
||||||
|
"expires_at",
|
||||||
|
"restore_started_at",
|
||||||
|
"restore_completed_at",
|
||||||
|
"measured_rpo_seconds",
|
||||||
|
"measured_rto_seconds",
|
||||||
|
"component_count",
|
||||||
|
}
|
||||||
|
if set(verification) != expected_fields or verification.get("schema_version") != 1:
|
||||||
|
raise DistributionError("backup verification receipt is malformed")
|
||||||
|
encoded_evidence = read_bounded_bytes(
|
||||||
|
paths.backup_evidence,
|
||||||
|
maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES,
|
||||||
|
)
|
||||||
|
encoded_keyring = read_bounded_bytes(
|
||||||
|
paths.backup_keyring,
|
||||||
|
maximum_bytes=MAX_BACKUP_KEYRING_BYTES,
|
||||||
|
)
|
||||||
|
evidence = decode_json_bytes(encoded_evidence, label="backup evidence")
|
||||||
|
keyring = decode_json_bytes(encoded_keyring, label="backup keyring")
|
||||||
|
if encoded_evidence != canonical_distribution_json(evidence):
|
||||||
|
raise DistributionError("stored backup evidence is not canonical JSON")
|
||||||
|
if encoded_keyring != canonical_distribution_json(keyring):
|
||||||
|
raise DistributionError("stored backup keyring is not canonical JSON")
|
||||||
|
evidence_digest = hashlib.sha256(encoded_evidence).hexdigest()
|
||||||
|
keyring_digest = hashlib.sha256(encoded_keyring).hexdigest()
|
||||||
|
if evidence_digest != verification.get("evidence_sha256"):
|
||||||
|
raise DistributionError("stored backup evidence digest has changed")
|
||||||
|
if keyring_digest != verification.get("keyring_sha256"):
|
||||||
|
raise DistributionError("stored backup keyring digest has changed")
|
||||||
|
previous_release = receipt.get("release") if receipt else None
|
||||||
|
expected_release: Mapping[str, object] = (
|
||||||
|
previous_release
|
||||||
|
if isinstance(previous_release, Mapping)
|
||||||
|
else {
|
||||||
|
"channel": spec.release.channel,
|
||||||
|
"version": spec.release.version,
|
||||||
|
"manifest_sha256": spec.release.manifest_sha256,
|
||||||
|
"composition_sha256": spec.release.composition_sha256,
|
||||||
|
"api_image": spec.release.api_image,
|
||||||
|
"web_image": spec.release.web_image,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
summary = verify_backup_evidence(
|
||||||
|
evidence,
|
||||||
|
keyring,
|
||||||
|
installation_id=spec.installation_id,
|
||||||
|
profile=spec.profile,
|
||||||
|
release=expected_release,
|
||||||
|
)
|
||||||
|
expected_summary = {
|
||||||
|
"signature_key_id": verification.get("signature_key_id"),
|
||||||
|
"evidence_id": verification.get("evidence_id"),
|
||||||
|
"recovery_point_id": verification.get("recovery_point_id"),
|
||||||
|
"restore_drill_id": verification.get("restore_drill_id"),
|
||||||
|
"captured_at": verification.get("captured_at"),
|
||||||
|
"expires_at": verification.get("expires_at"),
|
||||||
|
"restore_started_at": verification.get("restore_started_at"),
|
||||||
|
"restore_completed_at": verification.get("restore_completed_at"),
|
||||||
|
"measured_rpo_seconds": verification.get("measured_rpo_seconds"),
|
||||||
|
"measured_rto_seconds": verification.get("measured_rto_seconds"),
|
||||||
|
"component_count": verification.get("component_count"),
|
||||||
|
}
|
||||||
|
for field, expected in expected_summary.items():
|
||||||
|
if summary.get(field) != expected:
|
||||||
|
raise DistributionError(
|
||||||
|
f"backup verification receipt does not match {field!r}"
|
||||||
|
)
|
||||||
|
if expected_release.get("manifest_sha256") != verification.get(
|
||||||
|
"release_manifest_sha256"
|
||||||
|
):
|
||||||
|
raise DistributionError("backup verification receipt has another release")
|
||||||
|
return {
|
||||||
|
**summary,
|
||||||
|
"evidence_sha256": evidence_digest,
|
||||||
|
"keyring_sha256": keyring_digest,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _backup_evidence_checks(
|
||||||
|
spec: InstallationSpec,
|
||||||
|
paths: BundlePaths,
|
||||||
|
*,
|
||||||
|
receipt: Mapping[str, object],
|
||||||
|
) -> tuple[Check, ...]:
|
||||||
|
required = release_change_requires_backup(spec, receipt)
|
||||||
|
available = all(
|
||||||
|
path.is_file()
|
||||||
|
for path in (
|
||||||
|
paths.backup_evidence,
|
||||||
|
paths.backup_keyring,
|
||||||
|
paths.backup_verification,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if not available:
|
||||||
|
return (
|
||||||
|
Check(
|
||||||
|
"backup.migration_gate",
|
||||||
|
"error"
|
||||||
|
if required
|
||||||
|
else "warning"
|
||||||
|
if spec.profile == "self-hosted"
|
||||||
|
else "ok",
|
||||||
|
(
|
||||||
|
"A release-changing migration has no verified coordinated backup evidence."
|
||||||
|
if required
|
||||||
|
else "No current coordinated backup evidence is adopted."
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"Run verify-backup --adopt after an isolated restore drill."
|
||||||
|
if spec.profile == "self-hosted"
|
||||||
|
else ""
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
summary = verify_stored_backup_evidence(
|
||||||
|
spec,
|
||||||
|
paths,
|
||||||
|
receipt=receipt,
|
||||||
|
)
|
||||||
|
except (DistributionError, OSError) as exc:
|
||||||
|
return (
|
||||||
|
Check(
|
||||||
|
"backup.migration_gate",
|
||||||
|
"error" if required else "warning",
|
||||||
|
f"Coordinated backup evidence is invalid: {exc}",
|
||||||
|
"Adopt fresh signed evidence for the currently applied release.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
Check(
|
||||||
|
"backup.migration_gate",
|
||||||
|
"ok",
|
||||||
|
(
|
||||||
|
"Release migration is backed by recovery point "
|
||||||
|
f"{summary['recovery_point_id']} and restore drill "
|
||||||
|
f"{summary['restore_drill_id']}."
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _ingress_configuration_checks(
|
def _ingress_configuration_checks(
|
||||||
spec: InstallationSpec,
|
spec: InstallationSpec,
|
||||||
paths: BundlePaths,
|
paths: BundlePaths,
|
||||||
|
|||||||
@@ -29,6 +29,9 @@ _BUNDLE_FILES = (
|
|||||||
"existing-proxy.json",
|
"existing-proxy.json",
|
||||||
"distribution-manifest.json",
|
"distribution-manifest.json",
|
||||||
"distribution-keyring.json",
|
"distribution-keyring.json",
|
||||||
|
"backup-evidence.json",
|
||||||
|
"backup-keyring.json",
|
||||||
|
"backup-verification.json",
|
||||||
"receipt.json",
|
"receipt.json",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,124 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Sign and validate provider-produced GovOPlaN backup evidence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import base64
|
||||||
|
import hashlib
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import stat
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from cryptography.hazmat.primitives import serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||||
|
|
||||||
|
from govoplan_deploy.backup_evidence import (
|
||||||
|
MAX_BACKUP_EVIDENCE_BYTES,
|
||||||
|
load_backup_keyring,
|
||||||
|
verify_backup_evidence,
|
||||||
|
)
|
||||||
|
from govoplan_deploy.bundle import atomic_write
|
||||||
|
from govoplan_deploy.distribution import (
|
||||||
|
canonical_json,
|
||||||
|
canonical_signed_payload,
|
||||||
|
load_bounded_json,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
KEY_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description=(
|
||||||
|
"Sign a provider-produced backup/restore evidence document and "
|
||||||
|
"validate it against an independently managed public keyring."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
parser.add_argument("--input", type=Path, required=True)
|
||||||
|
parser.add_argument("--output", type=Path, required=True)
|
||||||
|
parser.add_argument("--trusted-keyring", type=Path, required=True)
|
||||||
|
parser.add_argument(
|
||||||
|
"--signing-key",
|
||||||
|
action="append",
|
||||||
|
required=True,
|
||||||
|
metavar="KEY_ID=PRIVATE_PEM",
|
||||||
|
help="Ed25519 signer; may be repeated during key rotation.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--replace-signatures",
|
||||||
|
action="store_true",
|
||||||
|
help="Replace existing signatures instead of rejecting the input.",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
source = args.input.expanduser().resolve()
|
||||||
|
payload = load_bounded_json(source, maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES)
|
||||||
|
existing = payload.get("signatures")
|
||||||
|
if existing not in (None, []) and not args.replace_signatures:
|
||||||
|
raise SystemExit("input already contains signatures; use --replace-signatures")
|
||||||
|
|
||||||
|
signers = [_load_signer(value) for value in args.signing_key]
|
||||||
|
if len({key_id for key_id, _ in signers}) != len(signers):
|
||||||
|
raise SystemExit("duplicate signing key id")
|
||||||
|
payload["signatures"] = []
|
||||||
|
signed = canonical_signed_payload(payload)
|
||||||
|
payload["signatures"] = [
|
||||||
|
{
|
||||||
|
"key_id": key_id,
|
||||||
|
"algorithm": "ed25519",
|
||||||
|
"value": base64.b64encode(private_key.sign(signed)).decode("ascii"),
|
||||||
|
}
|
||||||
|
for key_id, private_key in signers
|
||||||
|
]
|
||||||
|
|
||||||
|
keyring = load_backup_keyring(args.trusted_keyring.expanduser().resolve())
|
||||||
|
release = payload.get("release")
|
||||||
|
if not isinstance(release, dict):
|
||||||
|
raise SystemExit("input release must be an object")
|
||||||
|
verify_backup_evidence(
|
||||||
|
payload,
|
||||||
|
keyring,
|
||||||
|
installation_id=str(payload.get("installation_id") or ""),
|
||||||
|
profile=str(
|
||||||
|
_object(payload.get("deployment_subject"), "deployment_subject").get(
|
||||||
|
"profile"
|
||||||
|
)
|
||||||
|
or ""
|
||||||
|
),
|
||||||
|
release=release,
|
||||||
|
)
|
||||||
|
encoded = canonical_json(payload)
|
||||||
|
output = args.output.expanduser().resolve()
|
||||||
|
atomic_write(output, encoded, mode=0o600)
|
||||||
|
print(f"Wrote {output}")
|
||||||
|
print(f"SHA256 {hashlib.sha256(encoded).hexdigest()}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _load_signer(value: str) -> tuple[str, Ed25519PrivateKey]:
|
||||||
|
key_id, separator, raw_path = value.partition("=")
|
||||||
|
if not separator or KEY_ID.fullmatch(key_id) is None or not raw_path:
|
||||||
|
raise SystemExit("--signing-key must use KEY_ID=/path/to/private.pem")
|
||||||
|
path = Path(raw_path).expanduser().resolve()
|
||||||
|
mode = stat.S_IMODE(path.stat().st_mode)
|
||||||
|
if mode & 0o077:
|
||||||
|
raise SystemExit(
|
||||||
|
f"private signing key must not be group/world accessible: {path}"
|
||||||
|
)
|
||||||
|
private_key = serialization.load_pem_private_key(path.read_bytes(), password=None)
|
||||||
|
if not isinstance(private_key, Ed25519PrivateKey):
|
||||||
|
raise SystemExit(f"signing key is not Ed25519: {path}")
|
||||||
|
return key_id, private_key
|
||||||
|
|
||||||
|
|
||||||
|
def _object(value: object, label: str) -> dict[str, Any]:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise SystemExit(f"input {label} must be an object")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Reference in New Issue
Block a user