govoplan/docs/operations/PRODUCTION_TARGET_HANDOFF.md

Production Target And Independent Evidence Handoff

This runbook identifies the external inputs needed to finish GovOPlaN #27 and GovOPlaN #37. The repository can render, inspect and sign evidence for a target, but it cannot manufacture an independent failure domain or an independent approval authority.

GovOPlaN #27: real two-node target

The bounded acceptance target is two independently schedulable worker nodes. The API and WebUI must each have ready replicas on both nodes, all Deployments must be available, the active module composition and software versions must be consistent, every configured queue must have a worker, and the database connection budget must pass. The validation then deletes one ready API pod and requires replacement without an observed readiness outage.

Two virtual machines on different physical hosts or availability zones meet the failure-domain intent. Two containers, VMs or Kubernetes nodes on one physical host are useful development targets but do not close #27. A two-worker cluster also does not prove control-plane high availability. For a self-managed production cluster, use three control-plane nodes plus at least two workers; a managed control plane plus two workers is the shorter path.

What the target owner must provide

Provide these through a secure handoff, not an issue, chat message or Git:

1. A kubeconfig path with access to the target, for example ~/.config/govoplan/targets/<target>.kubeconfig, mode 0600.
2. A stable installation ID, public HTTPS hostname, namespace, ingress class and TLS-secret or certificate-manager arrangement.
3. Two independently schedulable workers and permission to place API and WebUI replicas on both.
4. External, logically shared PostgreSQL, Redis and S3 endpoints with trusted CA material and network reachability from every worker. Do not co-locate the only copies of these services on the two workers used for the failure drill.
5. The six runtime secret values required by the generated manifest: MASTER_KEY_B64, DATABASE_URL, GOVOPLAN_DATABASE_URL_PGTOOLS, REDIS_URL, FILE_STORAGE_S3_ACCESS_KEY_ID and FILE_STORAGE_S3_SECRET_ACCESS_KEY.
6. A short-lived GovOPlaN API key limited to ops:operations:read, supplied in GOVOPLAN_OPS_API_KEY only for evidence collection.
7. An approved drill window and permission to delete one API pod.

If no Kubernetes target exists, provide hostnames/IP addresses for the machines, an SSH user and key path, the internal/external DNS plan, and the permitted firewall ports. Those inputs are sufficient to provision a k3s target. They are not sufficient to claim control-plane HA unless three control-plane failure domains are present.

The repository now supplies the strict libvirt/K3s lifecycle and example inventories for this handoff in KUBERNETES_TEST_LAB.md. Its acceptance mode rejects a declared topology unless the workers and shared-state fixture occupy different hypervisor and failure-domain identifiers. Reviewers must still verify that those identifiers correspond to genuinely independent hosts.

Separate deployment and evidence authorities

The deployment identity may create and update the namespace, Secret, ConfigMap, Deployments, Services, Jobs, PodDisruptionBudgets and Ingress. The evidence collector only needs:

• cluster scope: get and list for nodes;
• target namespace: get and list for pods and deployments;
• target namespace during the approved drill: delete for pods.

Use separate kubeconfig contexts or service accounts when the same person does not hold both roles.

Render, apply and verify

Use the signed, digest-pinned installation bundle selected for the target:

export KUBECONFIG="$HOME/.config/govoplan/targets/<target>.kubeconfig"

python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/<installation-id> \
  --namespace govoplan \
  --secret-name govoplan-runtime \
  --tls-secret-name govoplan-tls \
  --s3-ca-secret-name govoplan-s3-ca \
  --ingress-class-name nginx \
  --output /srv/govoplan/<installation-id>/kubernetes.json

kubectl apply -f /srv/govoplan/<installation-id>/kubernetes.json
kubectl -n govoplan wait --for=condition=available deployment --all --timeout=10m

export GOVOPLAN_OPS_API_KEY="$(cat /run/secrets/govoplan-ops-evidence-key)"
python tools/deployment/govoplan-deploy.py verify-kubernetes \
  --directory /srv/govoplan/<installation-id> \
  --namespace govoplan \
  --exercise-api-pod-loss \
  --output /srv/govoplan/<installation-id>/evidence/kubernetes-multi-host.json
unset GOVOPLAN_OPS_API_KEY

The verifier emits sanitized JSON and exits nonzero if the topology, runtime, queue, connection-budget or pod-loss checks fail. Preserve the private cluster logs and manifest alongside the sanitized result in the controlled evidence store.

GovOPlaN #37: controlled signed target evidence

Yes, collection, review and signing can run in containers. A container provides repeatability and process isolation; it does not create independent authority. The production approver must control a different private key from the target operator/assessor and must review the evidence before signing the production_approval scope.

Use at least these three key boundaries:

1. Installer authority: signs installed-release-origin receipts only.
2. Target assessment authority: signs the permitted target, accessibility, privacy, security, operations and recovery scopes.
3. Production approval authority: independently signs only production_approval after reviewing the other evidence.

Do not reuse release-catalog keys for any of these roles. Keep private Ed25519 keys outside Git, Gitea, GovOPlaN application storage and chat. Publish only the public keyrings. The proof issuer already rejects key reuse across release, installer and proof trust domains.

Generate independently held keys

Each authority runs this command in its own 0700 directory. The generator refuses existing output paths and writes both files as 0600:

install -d -m 0700 "$HOME/.config/govoplan/authority-keys"

python tools/assessments/generate-authority-keypair.py \
  --purpose proof \
  --key-id authority:target-2026 \
  --scope target_environment \
  --scope accessibility \
  --scope privacy \
  --scope security \
  --scope operations \
  --scope recovery \
  --private-key "$HOME/.config/govoplan/authority-keys/target-2026.pem" \
  --keyring "$HOME/.config/govoplan/authority-keys/target-2026-public.json"

The independent production approver generates another key with only --scope production_approval. An installer authority uses --purpose installer and no --scope. Merge public key entries into the separately controlled keyrings only after the responsible authorities verify fingerprints out of band.

Container boundary

Use two one-shot jobs or containers:

• Collector/assessor: network access, read-only source and trust mounts, read/write private evidence output, and the narrowly scoped kubeconfig. It must not receive the production-approval private key.
• Production approver: --network none, read-only assessment/evidence/trust mounts, a read-only secret mount containing only the approval key, and a separate output mount. It must not receive deployment credentials.

Build or select the assessment image by digest and record that digest in the evidence log. A representative runtime shape is:

docker run --rm --network none --read-only --tmpfs /tmp \
  --user "$(id -u):$(id -g)" \
  --mount type=bind,src="$PWD/evidence",dst=/evidence,readonly \
  --mount type=bind,src="$PWD/trust",dst=/trust,readonly \
  --mount type=bind,src="$HOME/.config/govoplan/authority-keys",dst=/run/keys,readonly \
  --mount type=bind,src="$PWD/approved",dst=/output \
  <assessment-image>@sha256:<digest> \
  <assessment command>

The current evidence commands and required scopes are documented in TARGET_MATURITY_EVIDENCE_RUNBOOK.md. The final proof must cover target_environment, accessibility, privacy, security, operations, recovery and independent production_approval, and must bind to the verified installed composition and installer receipt.

Completion boundary

#27 can close after the real target produces a passing pod-loss result. #37 can close after an independently approved, schema-valid proof is generated for that same installed composition and the public authority keyrings, proof and private evidence custody references are recorded. Neither issue should close from a single-host simulation or a self-approved signature.