govoplan/docs/operations/KUBERNETES_TEST_LAB.md

# Kubernetes VM Test Lab

`tools/lab/govoplan-lab.py` creates and operates an amd64 Ubuntu/K3s test
environment on local or SSH-accessible libvirt hypervisors. It provides the
commands requested for the complete VM lifecycle:

| Command | Effect |
| --- | --- |
| `doctor` | Validate the strict inventory and, with `--online`, every hypervisor. |
| `create --apply` | Download checksum-pinned cloud images, create VM overlays and boot the declared VMs. |
| `deploy --apply` | Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN. |
| `update --apply` | Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release. |
| `status` | Show libvirt VM state, Kubernetes nodes and GovOPlaN pods. |
| `pause --apply` | Gracefully shut down workers, control planes and shared state while retaining disks. |
| `resume --apply` | Start the retained environment in dependency order and wait for readiness. |
| `verify` | Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill. |
| `destroy --apply --confirm <lab>` | Delete only the lab-owned domains and overlays; local evidence is retained by default. |

Every mutating command is a dry run unless `--apply` is present. Destruction
also requires the exact lab name. Generated credentials, CA keys, manifests and
evidence are written below the configured `state_directory` with owner-only
permissions. Keep that directory outside the repository and include it in the
workstation backup policy. Existing domains are reused or removed only when
their GovOPlaN ownership description and both expected lab disk paths match.

## What The Lab Proves

The supplied inventories describe two different assurance levels:

- `tools/lab/govoplan-lab.example.toml` creates four VMs on one libvirt host.
  It is suitable for development, deployment rehearsal, migration testing,
  application-pod replacement and recovery-tool exercises. It cannot close
  GovOPlaN #27 because one physical host remains one failure domain.
- `tools/lab/govoplan-lab.acceptance.example.toml` places the two workers on
  different hypervisors and puts the control and state VMs on a third. It can
  produce the bounded stateless application-tier evidence required by #27 when
  the declared hypervisors are genuinely independent physical failure domains.

Both examples use one control-plane VM and one state VM. This keeps the bounded
#27 target economical, but it does not prove control-plane or state-service
high availability. For control-plane failover, declare exactly three control
nodes on independent hosts. PostgreSQL, Redis and object-storage failover must
be tested against independently operated HA services; the lab's single state
VM is intentionally a replaceable integration fixture.

Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM
and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs
additional capacity. Do not overcommit memory on an acceptance target.

## 1. Prepare The Hypervisors

On each Ubuntu/Debian libvirt host:

```bash
sudo apt-get update
sudo apt-get install -y \
  qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl
sudo systemctl enable --now libvirtd
```

Use a dedicated lab-administration account. Remote hypervisors are managed over
SSH and the lifecycle invokes `sudo -n` there, so that account needs bounded
non-interactive permission for libvirt, image and cloud-init operations.
`NOPASSWD: ALL` is acceptable only on isolated lab hypervisors.

On a local hypervisor, put the workstation account in the `libvirt` group and
point `vm_image_directory` at a directory writable by that account and
traversable by `libvirt-qemu`. The lifecycle connects explicitly to
`qemu:///system` and does not require passwordless local sudo. Log out and back
in after a new group assignment before running `doctor --online`. Create the
configured image directory before running the doctor; it deliberately rejects
a missing or non-writable storage root instead of silently falling back to a
different filesystem.

Create a dedicated SSH key on the management workstation:

```bash
ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \
  -C "GovOPlaN Kubernetes lab"
```

Install its public key for every remote hypervisor account. The same public key
is injected into the VMs. The lifecycle keeps its own `ssh_known_hosts` file,
uses `accept-new` for first contact, and rejects changed host keys until a
lab-owned VM is deliberately recreated.

### Network contract

The configured `bridge` must exist on every selected hypervisor. All VM
addresses are static. Reserve them outside DHCP allocation and ensure that the
management workstation can route directly to every VM address; the lifecycle
does not tunnel VM traffic through the hypervisor SSH connection.

Permit only these flows inside the lab network:

| Port | Source and destination | Purpose |
| --- | --- | --- |
| TCP 22 | management workstation to every VM/hypervisor | Provisioning and evidence collection |
| TCP 6443 | all K3s nodes and management path to controls | Kubernetes API |
| UDP 8472 | K3s node to K3s node | Default Flannel VXLAN; never expose publicly |
| TCP 10250 | K3s node to K3s node | Kubelet metrics and API |
| TCP 2379-2380 | control to control, only with three controls | Embedded etcd |
| TCP 80/443 | test clients to K3s nodes | Traefik/ServiceLB ingress |
| TCP 5432/6379/9443 | K3s nodes to the state VM | PostgreSQL, Redis and TLS-protected Garage S3 |
| TCP 3025/3143 | approved test clients/workers to the state VM | GreenMail SMTP/IMAP test endpoints |

The official
[K3s networking requirements](https://docs.k3s.io/installation/requirements#networking)
remain authoritative. Restrict state ports to the lab network even though the
generated integration stack binds them on the state VM.

## 2. Create The Inventory

Start with the one-host rehearsal:

```bash
install -d -m 0700 "$HOME/.config/govoplan/labs"
cp tools/lab/govoplan-lab.example.toml \
  "$HOME/.config/govoplan/labs/development.toml"
chmod 0600 "$HOME/.config/govoplan/labs/development.toml"
```

Edit at least the bridge, network, static addresses and SSH key paths. For a
multi-host run, copy the acceptance example and replace every example hostname,
failure-domain declaration and network value. Strict parsing rejects unknown
keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an
acceptance inventory that collapses workers onto one declared hypervisor or
failure domain.

Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus
SHA-256 pairs. Updating means changing those reviewed pins and then running the
`update` command; the tool deliberately does not follow `latest` aliases.

The one-host example uses the dedicated `govoplan-lab` NAT network. Its DHCP
pool ends at `192.168.123.99`; the static lab addresses start at
`192.168.123.201`. Define and start it once on the local hypervisor:

```bash
virsh --connect qemu:///system net-define \
  tools/lab/libvirt/govoplan-lab-network.xml
virsh --connect qemu:///system net-autostart govoplan-lab
virsh --connect qemu:///system net-start govoplan-lab
```

Re-running those commands is unnecessary when `virsh net-info govoplan-lab`
already reports an active, persistent network. The lab destroy command leaves
this reusable network in place.

## 3. Validate And Create The VMs

```bash
LAB="$HOME/.config/govoplan/labs/development.toml"
PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python"

"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply
```

The preview is safe to run repeatedly. Creation reuses a domain whose exact
lab-owned name already exists and otherwise creates a thin qcow2 overlay under
`vm_image_directory/<lab>/<node>`.

## 4. Deploy GovOPlaN

If `git.add-ideas.de` requires authentication for release images, export a
read-only package/container-registry identity for this shell. A Gitea package
token can be used as the password:

```bash
export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader'
read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD
export GOVOPLAN_LAB_REGISTRY_PASSWORD
```

Then preview and apply:

```bash
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status
unset GOVOPLAN_LAB_REGISTRY_PASSWORD
```

Deployment verifies the downloaded release manifest and keyring by pinned
digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL,
Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers
and scheduler run in K3s from digest-pinned release images. A private lab CA
protects both ingress and S3; backend pods receive only the CA Secret and keep
TLS verification enabled. The CA profile carries critical `CA:TRUE` and
`keyCertSign,cRLSign` constraints. `deploy` and `update` rotate older lab CAs
that do not satisfy that profile and reissue the ingress/S3 certificate.

The final output identifies two local files below `state_directory`:

- `hosts` maps the public GovOPlaN and S3 test names to their VM addresses;
- `pki/ca.crt` is the private lab CA certificate.

Add the host mappings to the test client's resolver and trust the CA only on
devices used for this lab. On Debian/Ubuntu:

```bash
STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab"
cat "$STATE/hosts"
sudo install -m 0644 "$STATE/pki/ca.crt" \
  /usr/local/share/ca-certificates/govoplan-k8s-lab.crt
sudo update-ca-certificates
```

Review mappings before adding them to `/etc/hosts`; the lifecycle does not edit
the workstation's trust or resolver configuration. Reinstall `pki/ca.crt` in
the client trust store after an automatic CA rotation.

### Enroll the first administrator

The production runtime does not create a default password. Issue one expiring,
single-use first-administrator credential inside an API pod and copy its
owner-only artifact out immediately:

```bash
KUBECTL="$STATE/bin/kubectl"
POD="$($KUBECTL -n govoplan get pods \
  -l app.kubernetes.io/component=api \
  -o jsonpath='{.items[0].metadata.name}')"
ARTIFACT="$STATE/first-admin-enrollment.json"
umask 077

$KUBECTL -n govoplan exec "$POD" -- \
  python -m govoplan_core.commands.first_admin issue \
  --reason 'initial Kubernetes lab enrollment' \
  --output /tmp/first-admin-enrollment.json
$KUBECTL -n govoplan exec "$POD" -- \
  cat /tmp/first-admin-enrollment.json > "$ARTIFACT"
$KUBECTL -n govoplan exec "$POD" -- \
  rm -f /tmp/first-admin-enrollment.json
chmod 0600 "$ARTIFACT"
```

Submit the token from that artifact once to
`/api/v1/bootstrap/first-admin` with the administrator email, display name,
password, tenant slug and tenant name. The password must contain at least 12
characters. The lab command performs that exchange without placing either the
token or password in process arguments, rejects redirects, and removes the
artifact only after HTTP 201:

```bash
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
  --email 'owner@example.org' \
  --display-name 'System Owner' \
  --tenant-slug default \
  --tenant-name 'Default Tenant'
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
  --email 'owner@example.org' \
  --display-name 'System Owner' \
  --tenant-slug default \
  --tenant-name 'Default Tenant' \
  --apply
```

The public lab hostname must already resolve on the management workstation;
the command verifies TLS through the generated private CA directly.

## 5. Collect #27 Evidence

Create a short-lived API key authorized to read the Ops status endpoint. In the
current Access administration UI, open **Tenant API keys** and select only
**View tenant settings** (`admin:settings:read`); the Ops endpoint explicitly
accepts that compatibility scope. A dedicated operator credential may instead
use `ops:operations:read`. Then run:

```bash
export GOVOPLAN_OPS_API_KEY='short-lived-value'
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \
  --exercise-api-pod-loss
unset GOVOPLAN_OPS_API_KEY
```

The verifier requires ready API and WebUI pods across at least two Kubernetes
nodes, all Deployments available, consistent runtime composition, queue
coverage and a valid database-connection budget. During the optional drill it
deletes one ready API pod, probes public readiness and waits for replacement.
It writes sanitized output to
`state_directory/evidence/kubernetes-multi-host.json` and never stores the API
key. A rehearsal inventory prints an explicit warning that its result is not
independent-failure-domain evidence.

Retain these private artifacts together for review:

1. `inventory.json` and the reviewed inventory TOML;
2. the adopted release manifest/keyring and installation receipt;
3. `kubernetes.json`;
4. the Kubernetes verifier output;
5. private cluster logs for the approved drill window;
6. the operator's out-of-band evidence that the worker hypervisors are
   independent physical hosts or availability zones.

GovOPlaN #37 additionally requires independent assessment and production
approval keys. Running its evidence jobs in containers is supported, but a
container does not create an independent authority. Follow
`TARGET_MATURITY_EVIDENCE_RUNBOOK.md` after the #27 drill passes.

## 6. Update, Pause, Resume And Remove

After reviewing and changing pinned image/K3s/release values in the inventory:

```bash
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply
```

Workers are cordoned, drained, updated and uncordoned one at a time. K3s
controls are reconciled serially. The release-specific migration Job remains
subject to GovOPlaN's signed backup-evidence gate. The lab update command is not
a substitute for creating recovery evidence before a destructive state-schema
change.

To stop compute use without deleting disks:

```bash
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply
```

To remove VM resources while preserving local evidence:

```bash
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \
  --apply --confirm govoplan-k8s-lab
```

Add `--purge-local-state` only after evidence and recovery material have been
retained elsewhere. That option deletes the generated local CA, secrets,
manifests and evidence as well as the VMs.

## Acceptance Boundary

This tool supplies reproducible infrastructure and executes the bounded
stateless-node drill. It does not certify the truth of operator-entered failure
domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence,
or approve its own results. Those boundaries are deliberate: #27 can close
after a passing run on independently controlled hosts; broader production
maturity remains governed by #35, #37 and the target evidence runbook.