docs: organize cross-product documentation

docs/operations/KUBERNETES_TEST_LAB.md

@@ -0,0 +1,343 @@
+# Kubernetes VM Test Lab
+
+`tools/lab/govoplan-lab.py` creates and operates an amd64 Ubuntu/K3s test
+environment on local or SSH-accessible libvirt hypervisors. It provides the
+commands requested for the complete VM lifecycle:
+
+| Command | Effect |
+| --- | --- |
+| `doctor` | Validate the strict inventory and, with `--online`, every hypervisor. |
+| `create --apply` | Download checksum-pinned cloud images, create VM overlays and boot the declared VMs. |
+| `deploy --apply` | Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN. |
+| `update --apply` | Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release. |
+| `status` | Show libvirt VM state, Kubernetes nodes and GovOPlaN pods. |
+| `pause --apply` | Gracefully shut down workers, control planes and shared state while retaining disks. |
+| `resume --apply` | Start the retained environment in dependency order and wait for readiness. |
+| `verify` | Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill. |
+| `destroy --apply --confirm <lab>` | Delete only the lab-owned domains and overlays; local evidence is retained by default. |
+
+Every mutating command is a dry run unless `--apply` is present. Destruction
+also requires the exact lab name. Generated credentials, CA keys, manifests and
+evidence are written below the configured `state_directory` with owner-only
+permissions. Keep that directory outside the repository and include it in the
+workstation backup policy. Existing domains are reused or removed only when
+their GovOPlaN ownership description and both expected lab disk paths match.
+
+## What The Lab Proves
+
+The supplied inventories describe two different assurance levels:
+
+- `tools/lab/govoplan-lab.example.toml` creates four VMs on one libvirt host.
+  It is suitable for development, deployment rehearsal, migration testing,
+  application-pod replacement and recovery-tool exercises. It cannot close
+  GovOPlaN #27 because one physical host remains one failure domain.
+- `tools/lab/govoplan-lab.acceptance.example.toml` places the two workers on
+  different hypervisors and puts the control and state VMs on a third. It can
+  produce the bounded stateless application-tier evidence required by #27 when
+  the declared hypervisors are genuinely independent physical failure domains.
+
+Both examples use one control-plane VM and one state VM. This keeps the bounded
+#27 target economical, but it does not prove control-plane or state-service
+high availability. For control-plane failover, declare exactly three control
+nodes on independent hosts. PostgreSQL, Redis and object-storage failover must
+be tested against independently operated HA services; the lab's single state
+VM is intentionally a replaceable integration fixture.
+
+Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM
+and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs
+additional capacity. Do not overcommit memory on an acceptance target.
+
+## 1. Prepare The Hypervisors
+
+On each Ubuntu/Debian libvirt host:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y \
+  qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl
+sudo systemctl enable --now libvirtd
+```
+
+Use a dedicated lab-administration account. Remote hypervisors are managed over
+SSH and the lifecycle invokes `sudo -n` there, so that account needs bounded
+non-interactive permission for libvirt, image and cloud-init operations.
+`NOPASSWD: ALL` is acceptable only on isolated lab hypervisors.
+
+On a local hypervisor, put the workstation account in the `libvirt` group and
+point `vm_image_directory` at a directory writable by that account and
+traversable by `libvirt-qemu`. The lifecycle connects explicitly to
+`qemu:///system` and does not require passwordless local sudo. Log out and back
+in after a new group assignment before running `doctor --online`. Create the
+configured image directory before running the doctor; it deliberately rejects
+a missing or non-writable storage root instead of silently falling back to a
+different filesystem.
+
+Create a dedicated SSH key on the management workstation:
+
+```bash
+ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \
+  -C "GovOPlaN Kubernetes lab"
+```
+
+Install its public key for every remote hypervisor account. The same public key
+is injected into the VMs. The lifecycle keeps its own `ssh_known_hosts` file,
+uses `accept-new` for first contact, and rejects changed host keys until a
+lab-owned VM is deliberately recreated.
+
+### Network contract
+
+The configured `bridge` must exist on every selected hypervisor. All VM
+addresses are static. Reserve them outside DHCP allocation and ensure that the
+management workstation can route directly to every VM address; the lifecycle
+does not tunnel VM traffic through the hypervisor SSH connection.
+
+Permit only these flows inside the lab network:
+
+| Port | Source and destination | Purpose |
+| --- | --- | --- |
+| TCP 22 | management workstation to every VM/hypervisor | Provisioning and evidence collection |
+| TCP 6443 | all K3s nodes and management path to controls | Kubernetes API |
+| UDP 8472 | K3s node to K3s node | Default Flannel VXLAN; never expose publicly |
+| TCP 10250 | K3s node to K3s node | Kubelet metrics and API |
+| TCP 2379-2380 | control to control, only with three controls | Embedded etcd |
+| TCP 80/443 | test clients to K3s nodes | Traefik/ServiceLB ingress |
+| TCP 5432/6379/9443 | K3s nodes to the state VM | PostgreSQL, Redis and TLS-protected Garage S3 |
+| TCP 3025/3143 | approved test clients/workers to the state VM | GreenMail SMTP/IMAP test endpoints |
+
+The official
+[K3s networking requirements](https://docs.k3s.io/installation/requirements#networking)
+remain authoritative. Restrict state ports to the lab network even though the
+generated integration stack binds them on the state VM.
+
+## 2. Create The Inventory
+
+Start with the one-host rehearsal:
+
+```bash
+install -d -m 0700 "$HOME/.config/govoplan/labs"
+cp tools/lab/govoplan-lab.example.toml \
+  "$HOME/.config/govoplan/labs/development.toml"
+chmod 0600 "$HOME/.config/govoplan/labs/development.toml"
+```
+
+Edit at least the bridge, network, static addresses and SSH key paths. For a
+multi-host run, copy the acceptance example and replace every example hostname,
+failure-domain declaration and network value. Strict parsing rejects unknown
+keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an
+acceptance inventory that collapses workers onto one declared hypervisor or
+failure domain.
+
+Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus
+SHA-256 pairs. Updating means changing those reviewed pins and then running the
+`update` command; the tool deliberately does not follow `latest` aliases.
+
+The one-host example uses the dedicated `govoplan-lab` NAT network. Its DHCP
+pool ends at `192.168.123.99`; the static lab addresses start at
+`192.168.123.201`. Define and start it once on the local hypervisor:
+
+```bash
+virsh --connect qemu:///system net-define \
+  tools/lab/libvirt/govoplan-lab-network.xml
+virsh --connect qemu:///system net-autostart govoplan-lab
+virsh --connect qemu:///system net-start govoplan-lab
+```
+
+Re-running those commands is unnecessary when `virsh net-info govoplan-lab`
+already reports an active, persistent network. The lab destroy command leaves
+this reusable network in place.
+
+## 3. Validate And Create The VMs
+
+```bash
+LAB="$HOME/.config/govoplan/labs/development.toml"
+PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python"
+
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply
+```
+
+The preview is safe to run repeatedly. Creation reuses a domain whose exact
+lab-owned name already exists and otherwise creates a thin qcow2 overlay under
+`vm_image_directory/<lab>/<node>`.
+
+## 4. Deploy GovOPlaN
+
+If `git.add-ideas.de` requires authentication for release images, export a
+read-only package/container-registry identity for this shell. A Gitea package
+token can be used as the password:
+
+```bash
+export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader'
+read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD
+export GOVOPLAN_LAB_REGISTRY_PASSWORD
+```
+
+Then preview and apply:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status
+unset GOVOPLAN_LAB_REGISTRY_PASSWORD
+```
+
+Deployment verifies the downloaded release manifest and keyring by pinned
+digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL,
+Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers
+and scheduler run in K3s from digest-pinned release images. A private lab CA
+protects both ingress and S3; backend pods receive only the CA Secret and keep
+TLS verification enabled. The CA profile carries critical `CA:TRUE` and
+`keyCertSign,cRLSign` constraints. `deploy` and `update` rotate older lab CAs
+that do not satisfy that profile and reissue the ingress/S3 certificate.
+
+The final output identifies two local files below `state_directory`:
+
+- `hosts` maps the public GovOPlaN and S3 test names to their VM addresses;
+- `pki/ca.crt` is the private lab CA certificate.
+
+Add the host mappings to the test client's resolver and trust the CA only on
+devices used for this lab. On Debian/Ubuntu:
+
+```bash
+STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab"
+cat "$STATE/hosts"
+sudo install -m 0644 "$STATE/pki/ca.crt" \
+  /usr/local/share/ca-certificates/govoplan-k8s-lab.crt
+sudo update-ca-certificates
+```
+
+Review mappings before adding them to `/etc/hosts`; the lifecycle does not edit
+the workstation's trust or resolver configuration. Reinstall `pki/ca.crt` in
+the client trust store after an automatic CA rotation.
+
+### Enroll the first administrator
+
+The production runtime does not create a default password. Issue one expiring,
+single-use first-administrator credential inside an API pod and copy its
+owner-only artifact out immediately:
+
+```bash
+KUBECTL="$STATE/bin/kubectl"
+POD="$($KUBECTL -n govoplan get pods \
+  -l app.kubernetes.io/component=api \
+  -o jsonpath='{.items[0].metadata.name}')"
+ARTIFACT="$STATE/first-admin-enrollment.json"
+umask 077
+
+$KUBECTL -n govoplan exec "$POD" -- \
+  python -m govoplan_core.commands.first_admin issue \
+  --reason 'initial Kubernetes lab enrollment' \
+  --output /tmp/first-admin-enrollment.json
+$KUBECTL -n govoplan exec "$POD" -- \
+  cat /tmp/first-admin-enrollment.json > "$ARTIFACT"
+$KUBECTL -n govoplan exec "$POD" -- \
+  rm -f /tmp/first-admin-enrollment.json
+chmod 0600 "$ARTIFACT"
+```
+
+Submit the token from that artifact once to
+`/api/v1/bootstrap/first-admin` with the administrator email, display name,
+password, tenant slug and tenant name. The password must contain at least 12
+characters. The lab command performs that exchange without placing either the
+token or password in process arguments, rejects redirects, and removes the
+artifact only after HTTP 201:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
+  --email 'owner@example.org' \
+  --display-name 'System Owner' \
+  --tenant-slug default \
+  --tenant-name 'Default Tenant'
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
+  --email 'owner@example.org' \
+  --display-name 'System Owner' \
+  --tenant-slug default \
+  --tenant-name 'Default Tenant' \
+  --apply
+```
+
+The public lab hostname must already resolve on the management workstation;
+the command verifies TLS through the generated private CA directly.
+
+## 5. Collect #27 Evidence
+
+Create a short-lived API key authorized to read the Ops status endpoint. In the
+current Access administration UI, open **Tenant API keys** and select only
+**View tenant settings** (`admin:settings:read`); the Ops endpoint explicitly
+accepts that compatibility scope. A dedicated operator credential may instead
+use `ops:operations:read`. Then run:
+
+```bash
+export GOVOPLAN_OPS_API_KEY='short-lived-value'
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \
+  --exercise-api-pod-loss
+unset GOVOPLAN_OPS_API_KEY
+```
+
+The verifier requires ready API and WebUI pods across at least two Kubernetes
+nodes, all Deployments available, consistent runtime composition, queue
+coverage and a valid database-connection budget. During the optional drill it
+deletes one ready API pod, probes public readiness and waits for replacement.
+It writes sanitized output to
+`state_directory/evidence/kubernetes-multi-host.json` and never stores the API
+key. A rehearsal inventory prints an explicit warning that its result is not
+independent-failure-domain evidence.
+
+Retain these private artifacts together for review:
+
+1. `inventory.json` and the reviewed inventory TOML;
+2. the adopted release manifest/keyring and installation receipt;
+3. `kubernetes.json`;
+4. the Kubernetes verifier output;
+5. private cluster logs for the approved drill window;
+6. the operator's out-of-band evidence that the worker hypervisors are
+   independent physical hosts or availability zones.
+
+GovOPlaN #37 additionally requires independent assessment and production
+approval keys. Running its evidence jobs in containers is supported, but a
+container does not create an independent authority. Follow
+`TARGET_MATURITY_EVIDENCE_RUNBOOK.md` after the #27 drill passes.
+
+## 6. Update, Pause, Resume And Remove
+
+After reviewing and changing pinned image/K3s/release values in the inventory:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply
+```
+
+Workers are cordoned, drained, updated and uncordoned one at a time. K3s
+controls are reconciled serially. The release-specific migration Job remains
+subject to GovOPlaN's signed backup-evidence gate. The lab update command is not
+a substitute for creating recovery evidence before a destructive state-schema
+change.
+
+To stop compute use without deleting disks:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply
+```
+
+To remove VM resources while preserving local evidence:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \
+  --apply --confirm govoplan-k8s-lab
+```
+
+Add `--purge-local-state` only after evidence and recovery material have been
+retained elsewhere. That option deletes the generated local CA, secrets,
+manifests and evidence as well as the VMs.
+
+## Acceptance Boundary
+
+This tool supplies reproducible infrastructure and executes the bounded
+stateless-node drill. It does not certify the truth of operator-entered failure
+domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence,
+or approve its own results. Those boundaries are deliberate: #27 can close
+after a passing run on independently controlled hosts; broader production
+maturity remains governed by #35, #37 and the target evidence runbook.