govoplan/docs/KUBERNETES_TEST_LAB.md

Kubernetes VM Test Lab

tools/lab/govoplan-lab.py creates and operates an amd64 Ubuntu/K3s test environment on local or SSH-accessible libvirt hypervisors. It provides the commands requested for the complete VM lifecycle:

Command	Effect
doctor	Validate the strict inventory and, with --online, every hypervisor.
create --apply	Download checksum-pinned cloud images, create VM overlays and boot the declared VMs.
deploy --apply	Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN.
update --apply	Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release.
status	Show libvirt VM state, Kubernetes nodes and GovOPlaN pods.
pause --apply	Gracefully shut down workers, control planes and shared state while retaining disks.
resume --apply	Start the retained environment in dependency order and wait for readiness.
verify	Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill.
destroy --apply --confirm <lab>	Delete only the lab-owned domains and overlays; local evidence is retained by default.

Every mutating command is a dry run unless --apply is present. Destruction also requires the exact lab name. Generated credentials, CA keys, manifests and evidence are written below the configured state_directory with owner-only permissions. Keep that directory outside the repository and include it in the workstation backup policy. Existing domains are reused or removed only when their GovOPlaN ownership description and both expected lab disk paths match.

What The Lab Proves

The supplied inventories describe two different assurance levels:

• tools/lab/govoplan-lab.example.toml creates four VMs on one libvirt host. It is suitable for development, deployment rehearsal, migration testing, application-pod replacement and recovery-tool exercises. It cannot close GovOPlaN #27 because one physical host remains one failure domain.
• tools/lab/govoplan-lab.acceptance.example.toml places the two workers on different hypervisors and puts the control and state VMs on a third. It can produce the bounded stateless application-tier evidence required by #27 when the declared hypervisors are genuinely independent physical failure domains.

Both examples use one control-plane VM and one state VM. This keeps the bounded #27 target economical, but it does not prove control-plane or state-service high availability. For control-plane failover, declare exactly three control nodes on independent hosts. PostgreSQL, Redis and object-storage failover must be tested against independently operated HA services; the lab's single state VM is intentionally a replaceable integration fixture.

Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs additional capacity. Do not overcommit memory on an acceptance target.

1. Prepare The Hypervisors

On each Ubuntu/Debian libvirt host:

sudo apt-get update
sudo apt-get install -y \
  qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl
sudo systemctl enable --now libvirtd

Use a dedicated lab-administration account. Remote hypervisors are managed over SSH and the lifecycle invokes sudo -n there, so that account needs bounded non-interactive permission for libvirt, image and cloud-init operations. NOPASSWD: ALL is acceptable only on isolated lab hypervisors.

On a local hypervisor, put the workstation account in the libvirt group and point vm_image_directory at a directory writable by that account and traversable by libvirt-qemu. The lifecycle connects explicitly to qemu:///system and does not require passwordless local sudo. Log out and back in after a new group assignment before running doctor --online. Create the configured image directory before running the doctor; it deliberately rejects a missing or non-writable storage root instead of silently falling back to a different filesystem.

Create a dedicated SSH key on the management workstation:

ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \
  -C "GovOPlaN Kubernetes lab"

Install its public key for every remote hypervisor account. The same public key is injected into the VMs. The lifecycle keeps its own ssh_known_hosts file, uses accept-new for first contact, and rejects changed host keys until a lab-owned VM is deliberately recreated.

Network contract

The configured bridge must exist on every selected hypervisor. All VM addresses are static. Reserve them outside DHCP allocation and ensure that the management workstation can route directly to every VM address; the lifecycle does not tunnel VM traffic through the hypervisor SSH connection.

Permit only these flows inside the lab network:

Port	Source and destination	Purpose
TCP 22	management workstation to every VM/hypervisor	Provisioning and evidence collection
TCP 6443	all K3s nodes and management path to controls	Kubernetes API
UDP 8472	K3s node to K3s node	Default Flannel VXLAN; never expose publicly
TCP 10250	K3s node to K3s node	Kubelet metrics and API
TCP 2379-2380	control to control, only with three controls	Embedded etcd
TCP 80/443	test clients to K3s nodes	Traefik/ServiceLB ingress
TCP 5432/6379/9443	K3s nodes to the state VM	PostgreSQL, Redis and TLS-protected Garage S3
TCP 3025/3143	approved test clients/workers to the state VM	GreenMail SMTP/IMAP test endpoints

The official K3s networking requirements remain authoritative. Restrict state ports to the lab network even though the generated integration stack binds them on the state VM.

2. Create The Inventory

Start with the one-host rehearsal:

install -d -m 0700 "$HOME/.config/govoplan/labs"
cp tools/lab/govoplan-lab.example.toml \
  "$HOME/.config/govoplan/labs/development.toml"
chmod 0600 "$HOME/.config/govoplan/labs/development.toml"

Edit at least the bridge, network, static addresses and SSH key paths. For a multi-host run, copy the acceptance example and replace every example hostname, failure-domain declaration and network value. Strict parsing rejects unknown keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an acceptance inventory that collapses workers onto one declared hypervisor or failure domain.

Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus SHA-256 pairs. Updating means changing those reviewed pins and then running the update command; the tool deliberately does not follow latest aliases.

The one-host example uses the dedicated govoplan-lab NAT network. Its DHCP pool ends at 192.168.123.99; the static lab addresses start at 192.168.123.201. Define and start it once on the local hypervisor:

virsh --connect qemu:///system net-define \
  tools/lab/libvirt/govoplan-lab-network.xml
virsh --connect qemu:///system net-autostart govoplan-lab
virsh --connect qemu:///system net-start govoplan-lab

Re-running those commands is unnecessary when virsh net-info govoplan-lab already reports an active, persistent network. The lab destroy command leaves this reusable network in place.

3. Validate And Create The VMs

LAB="$HOME/.config/govoplan/labs/development.toml"
PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python"

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply

The preview is safe to run repeatedly. Creation reuses a domain whose exact lab-owned name already exists and otherwise creates a thin qcow2 overlay under vm_image_directory/<lab>/<node>.

4. Deploy GovOPlaN

If git.add-ideas.de requires authentication for release images, export a read-only package/container-registry identity for this shell. A Gitea package token can be used as the password:

export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader'
read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD
export GOVOPLAN_LAB_REGISTRY_PASSWORD

Then preview and apply:

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status
unset GOVOPLAN_LAB_REGISTRY_PASSWORD

Deployment verifies the downloaded release manifest and keyring by pinned digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL, Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers and scheduler run in K3s from digest-pinned release images. A private lab CA protects both ingress and S3; backend pods receive only the CA Secret and keep TLS verification enabled.

The final output identifies two local files below state_directory:

• hosts maps the public GovOPlaN and S3 test names to their VM addresses;
• pki/ca.crt is the private lab CA certificate.

Add the host mappings to the test client's resolver and trust the CA only on devices used for this lab. On Debian/Ubuntu:

STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab"
cat "$STATE/hosts"
sudo install -m 0644 "$STATE/pki/ca.crt" \
  /usr/local/share/ca-certificates/govoplan-k8s-lab.crt
sudo update-ca-certificates

Review mappings before adding them to /etc/hosts; the lifecycle does not edit the workstation's trust or resolver configuration.

Enroll the first administrator

The production runtime does not create a default password. Issue one expiring, single-use first-administrator credential inside an API pod and copy its owner-only artifact out immediately:

KUBECTL="$STATE/bin/kubectl"
POD="$($KUBECTL -n govoplan get pods \
  -l app.kubernetes.io/component=api \
  -o jsonpath='{.items[0].metadata.name}')"
ARTIFACT="$STATE/first-admin-enrollment.json"
umask 077

$KUBECTL -n govoplan exec "$POD" -- \
  python -m govoplan_core.commands.first_admin issue \
  --reason 'initial Kubernetes lab enrollment' \
  --output /tmp/first-admin-enrollment.json
$KUBECTL -n govoplan exec "$POD" -- \
  cat /tmp/first-admin-enrollment.json > "$ARTIFACT"
$KUBECTL -n govoplan exec "$POD" -- \
  rm -f /tmp/first-admin-enrollment.json
chmod 0600 "$ARTIFACT"

Submit the token from that artifact once to /api/v1/bootstrap/first-admin with the administrator email, display name, password, tenant slug and tenant name. The password must contain at least 12 characters. The lab command performs that exchange without placing either the token or password in process arguments, rejects redirects, and removes the artifact only after HTTP 201:

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
  --email 'owner@example.org' \
  --display-name 'System Owner' \
  --tenant-slug default \
  --tenant-name 'Default Tenant'
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
  --email 'owner@example.org' \
  --display-name 'System Owner' \
  --tenant-slug default \
  --tenant-name 'Default Tenant' \
  --apply

The public lab hostname must already resolve on the management workstation; the command verifies TLS through the generated private CA directly.

5. Collect #27 Evidence

Create a short-lived API key authorized to read the Ops status endpoint. In the current Access administration UI, open Tenant API keys and select only View tenant settings (admin:settings:read); the Ops endpoint explicitly accepts that compatibility scope. A dedicated operator credential may instead use ops:operations:read. Then run:

export GOVOPLAN_OPS_API_KEY='short-lived-value'
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \
  --exercise-api-pod-loss
unset GOVOPLAN_OPS_API_KEY

The verifier requires ready API and WebUI pods across at least two Kubernetes nodes, all Deployments available, consistent runtime composition, queue coverage and a valid database-connection budget. During the optional drill it deletes one ready API pod, probes public readiness and waits for replacement. It writes sanitized output to state_directory/evidence/kubernetes-multi-host.json and never stores the API key. A rehearsal inventory prints an explicit warning that its result is not independent-failure-domain evidence.

Retain these private artifacts together for review:

1. inventory.json and the reviewed inventory TOML;
2. the adopted release manifest/keyring and installation receipt;
3. kubernetes.json;
4. the Kubernetes verifier output;
5. private cluster logs for the approved drill window;
6. the operator's out-of-band evidence that the worker hypervisors are independent physical hosts or availability zones.

GovOPlaN #37 additionally requires independent assessment and production approval keys. Running its evidence jobs in containers is supported, but a container does not create an independent authority. Follow TARGET_MATURITY_EVIDENCE_RUNBOOK.md after the #27 drill passes.

6. Update, Pause, Resume And Remove

After reviewing and changing pinned image/K3s/release values in the inventory:

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply

Workers are cordoned, drained, updated and uncordoned one at a time. K3s controls are reconciled serially. The release-specific migration Job remains subject to GovOPlaN's signed backup-evidence gate. The lab update command is not a substitute for creating recovery evidence before a destructive state-schema change.

To stop compute use without deleting disks:

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply

To remove VM resources while preserving local evidence:

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \
  --apply --confirm govoplan-k8s-lab

Add --purge-local-state only after evidence and recovery material have been retained elsewhere. That option deletes the generated local CA, secrets, manifests and evidence as well as the VMs.

Acceptance Boundary

This tool supplies reproducible infrastructure and executes the bounded stateless-node drill. It does not certify the truth of operator-entered failure domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence, or approve its own results. Those boundaries are deliberate: #27 can close after a passing run on independently controlled hosts; broader production maturity remains governed by #35, #37 and the target evidence runbook.