183 lines
8.2 KiB
Markdown
183 lines
8.2 KiB
Markdown
# Production Target And Independent Evidence Handoff
|
|
|
|
This runbook identifies the external inputs needed to finish
|
|
[GovOPlaN #27](https://git.add-ideas.de/GovOPlaN/govoplan/issues/27) and
|
|
[GovOPlaN #37](https://git.add-ideas.de/GovOPlaN/govoplan/issues/37). The
|
|
repository can render, inspect and sign evidence for a target, but it cannot
|
|
manufacture an independent failure domain or an independent approval authority.
|
|
|
|
## GovOPlaN #27: real two-node target
|
|
|
|
The bounded acceptance target is two independently schedulable worker nodes.
|
|
The API and WebUI must each have ready replicas on both nodes, all Deployments
|
|
must be available, the active module composition and software versions must be
|
|
consistent, every configured queue must have a worker, and the database
|
|
connection budget must pass. The validation then deletes one ready API pod and
|
|
requires replacement without an observed readiness outage.
|
|
|
|
Two virtual machines on different physical hosts or availability zones meet the
|
|
failure-domain intent. Two containers, VMs or Kubernetes nodes on one physical
|
|
host are useful development targets but do not close #27. A two-worker cluster
|
|
also does not prove control-plane high availability. For a self-managed
|
|
production cluster, use three control-plane nodes plus at least two workers; a
|
|
managed control plane plus two workers is the shorter path.
|
|
|
|
### What the target owner must provide
|
|
|
|
Provide these through a secure handoff, not an issue, chat message or Git:
|
|
|
|
1. A kubeconfig path with access to the target, for example
|
|
`~/.config/govoplan/targets/<target>.kubeconfig`, mode `0600`.
|
|
2. A stable installation ID, public HTTPS hostname, namespace, ingress class and
|
|
TLS-secret or certificate-manager arrangement.
|
|
3. Two independently schedulable workers and permission to place API and WebUI
|
|
replicas on both.
|
|
4. External, logically shared PostgreSQL, Redis and S3 endpoints with trusted
|
|
CA material and network reachability from every worker. Do not co-locate the
|
|
only copies of these services on the two workers used for the failure drill.
|
|
5. The six runtime secret values required by the generated manifest:
|
|
`MASTER_KEY_B64`, `DATABASE_URL`, `GOVOPLAN_DATABASE_URL_PGTOOLS`,
|
|
`REDIS_URL`, `FILE_STORAGE_S3_ACCESS_KEY_ID` and
|
|
`FILE_STORAGE_S3_SECRET_ACCESS_KEY`.
|
|
6. A short-lived GovOPlaN API key limited to `ops:operations:read`, supplied in
|
|
`GOVOPLAN_OPS_API_KEY` only for evidence collection.
|
|
7. An approved drill window and permission to delete one API pod.
|
|
|
|
If no Kubernetes target exists, provide hostnames/IP addresses for the machines,
|
|
an SSH user and key path, the internal/external DNS plan, and the permitted
|
|
firewall ports. Those inputs are sufficient to provision a k3s target. They are
|
|
not sufficient to claim control-plane HA unless three control-plane failure
|
|
domains are present.
|
|
|
|
### Separate deployment and evidence authorities
|
|
|
|
The deployment identity may create and update the namespace, Secret,
|
|
ConfigMap, Deployments, Services, Jobs, PodDisruptionBudgets and Ingress. The
|
|
evidence collector only needs:
|
|
|
|
- cluster scope: `get` and `list` for `nodes`;
|
|
- target namespace: `get` and `list` for `pods` and `deployments`;
|
|
- target namespace during the approved drill: `delete` for `pods`.
|
|
|
|
Use separate kubeconfig contexts or service accounts when the same person does
|
|
not hold both roles.
|
|
|
|
### Render, apply and verify
|
|
|
|
Use the signed, digest-pinned installation bundle selected for the target:
|
|
|
|
```bash
|
|
export KUBECONFIG="$HOME/.config/govoplan/targets/<target>.kubeconfig"
|
|
|
|
python tools/deployment/govoplan-deploy.py render-kubernetes \
|
|
--directory /srv/govoplan/<installation-id> \
|
|
--namespace govoplan \
|
|
--secret-name govoplan-runtime \
|
|
--tls-secret-name govoplan-tls \
|
|
--ingress-class-name nginx \
|
|
--output /srv/govoplan/<installation-id>/kubernetes.json
|
|
|
|
kubectl apply -f /srv/govoplan/<installation-id>/kubernetes.json
|
|
kubectl -n govoplan wait --for=condition=available deployment --all --timeout=10m
|
|
|
|
export GOVOPLAN_OPS_API_KEY="$(cat /run/secrets/govoplan-ops-evidence-key)"
|
|
python tools/deployment/govoplan-deploy.py verify-kubernetes \
|
|
--directory /srv/govoplan/<installation-id> \
|
|
--namespace govoplan \
|
|
--exercise-api-pod-loss \
|
|
--output /srv/govoplan/<installation-id>/evidence/kubernetes-multi-host.json
|
|
unset GOVOPLAN_OPS_API_KEY
|
|
```
|
|
|
|
The verifier emits sanitized JSON and exits nonzero if the topology, runtime,
|
|
queue, connection-budget or pod-loss checks fail. Preserve the private cluster
|
|
logs and manifest alongside the sanitized result in the controlled evidence
|
|
store.
|
|
|
|
## GovOPlaN #37: controlled signed target evidence
|
|
|
|
Yes, collection, review and signing can run in containers. A container provides
|
|
repeatability and process isolation; it does not create independent authority.
|
|
The production approver must control a different private key from the target
|
|
operator/assessor and must review the evidence before signing the
|
|
`production_approval` scope.
|
|
|
|
Use at least these three key boundaries:
|
|
|
|
1. **Installer authority:** signs installed-release-origin receipts only.
|
|
2. **Target assessment authority:** signs the permitted target, accessibility,
|
|
privacy, security, operations and recovery scopes.
|
|
3. **Production approval authority:** independently signs only
|
|
`production_approval` after reviewing the other evidence.
|
|
|
|
Do not reuse release-catalog keys for any of these roles. Keep private Ed25519
|
|
keys outside Git, Gitea, GovOPlaN application storage and chat. Publish only the
|
|
public keyrings. The proof issuer already rejects key reuse across release,
|
|
installer and proof trust domains.
|
|
|
|
### Generate independently held keys
|
|
|
|
Each authority runs this command in its own `0700` directory. The generator
|
|
refuses existing output paths and writes both files as `0600`:
|
|
|
|
```bash
|
|
install -d -m 0700 "$HOME/.config/govoplan/authority-keys"
|
|
|
|
python tools/assessments/generate-authority-keypair.py \
|
|
--purpose proof \
|
|
--key-id authority:target-2026 \
|
|
--scope target_environment \
|
|
--scope accessibility \
|
|
--scope privacy \
|
|
--scope security \
|
|
--scope operations \
|
|
--scope recovery \
|
|
--private-key "$HOME/.config/govoplan/authority-keys/target-2026.pem" \
|
|
--keyring "$HOME/.config/govoplan/authority-keys/target-2026-public.json"
|
|
```
|
|
|
|
The independent production approver generates another key with only
|
|
`--scope production_approval`. An installer authority uses `--purpose installer`
|
|
and no `--scope`. Merge public key entries into the separately controlled
|
|
keyrings only after the responsible authorities verify fingerprints out of
|
|
band.
|
|
|
|
### Container boundary
|
|
|
|
Use two one-shot jobs or containers:
|
|
|
|
- **Collector/assessor:** network access, read-only source and trust mounts,
|
|
read/write private evidence output, and the narrowly scoped kubeconfig. It
|
|
must not receive the production-approval private key.
|
|
- **Production approver:** `--network none`, read-only assessment/evidence/trust
|
|
mounts, a read-only secret mount containing only the approval key, and a
|
|
separate output mount. It must not receive deployment credentials.
|
|
|
|
Build or select the assessment image by digest and record that digest in the
|
|
evidence log. A representative runtime shape is:
|
|
|
|
```bash
|
|
docker run --rm --network none --read-only --tmpfs /tmp \
|
|
--user "$(id -u):$(id -g)" \
|
|
--mount type=bind,src="$PWD/evidence",dst=/evidence,readonly \
|
|
--mount type=bind,src="$PWD/trust",dst=/trust,readonly \
|
|
--mount type=bind,src="$HOME/.config/govoplan/authority-keys",dst=/run/keys,readonly \
|
|
--mount type=bind,src="$PWD/approved",dst=/output \
|
|
<assessment-image>@sha256:<digest> \
|
|
<assessment command>
|
|
```
|
|
|
|
The current evidence commands and required scopes are documented in
|
|
[`TARGET_MATURITY_EVIDENCE_RUNBOOK.md`](TARGET_MATURITY_EVIDENCE_RUNBOOK.md).
|
|
The final proof must cover `target_environment`, `accessibility`, `privacy`,
|
|
`security`, `operations`, `recovery` and independent `production_approval`, and
|
|
must bind to the verified installed composition and installer receipt.
|
|
|
|
## Completion boundary
|
|
|
|
#27 can close after the real target produces a passing pod-loss result. #37 can
|
|
close after an independently approved, schema-valid proof is generated for that
|
|
same installed composition and the public authority keyrings, proof and private
|
|
evidence custody references are recorded. Neither issue should close from a
|
|
single-host simulation or a self-approved signature.
|