|
|
|
@@ -0,0 +1,299 @@
|
|
|
|
|
<!-- codex-wiki-sync:6c6bab4e7696c71be8a2bbad -->
|
|
|
|
|
|
|
|
|
|
> Mirrored from `/mnt/DATA/git/govoplan/docs/KUBERNETES_TEST_LAB.md`.
|
|
|
|
|
> Origin: `repository`.
|
|
|
|
|
> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
# Kubernetes VM Test Lab
|
|
|
|
|
|
|
|
|
|
`tools/lab/govoplan-lab.py` creates and operates an amd64 Ubuntu/K3s test
|
|
|
|
|
environment on local or SSH-accessible libvirt hypervisors. It provides the
|
|
|
|
|
commands requested for the complete VM lifecycle:
|
|
|
|
|
|
|
|
|
|
| Command | Effect |
|
|
|
|
|
| --- | --- |
|
|
|
|
|
| `doctor` | Validate the strict inventory and, with `--online`, every hypervisor. |
|
|
|
|
|
| `create --apply` | Download checksum-pinned cloud images, create VM overlays and boot the declared VMs. |
|
|
|
|
|
| `deploy --apply` | Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN. |
|
|
|
|
|
| `update --apply` | Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release. |
|
|
|
|
|
| `status` | Show libvirt VM state, Kubernetes nodes and GovOPlaN pods. |
|
|
|
|
|
| `pause --apply` | Gracefully shut down workers, control planes and shared state while retaining disks. |
|
|
|
|
|
| `resume --apply` | Start the retained environment in dependency order and wait for readiness. |
|
|
|
|
|
| `verify` | Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill. |
|
|
|
|
|
| `destroy --apply --confirm <lab>` | Delete only the lab-owned domains and overlays; local evidence is retained by default. |
|
|
|
|
|
|
|
|
|
|
Every mutating command is a dry run unless `--apply` is present. Destruction
|
|
|
|
|
also requires the exact lab name. Generated credentials, CA keys, manifests and
|
|
|
|
|
evidence are written below the configured `state_directory` with owner-only
|
|
|
|
|
permissions. Keep that directory outside the repository and include it in the
|
|
|
|
|
workstation backup policy.
|
|
|
|
|
|
|
|
|
|
## What The Lab Proves
|
|
|
|
|
|
|
|
|
|
The supplied inventories describe two different assurance levels:
|
|
|
|
|
|
|
|
|
|
- `tools/lab/govoplan-lab.example.toml` creates four VMs on one libvirt host.
|
|
|
|
|
It is suitable for development, deployment rehearsal, migration testing,
|
|
|
|
|
application-pod replacement and recovery-tool exercises. It cannot close
|
|
|
|
|
GovOPlaN #27 because one physical host remains one failure domain.
|
|
|
|
|
- `tools/lab/govoplan-lab.acceptance.example.toml` places the two workers on
|
|
|
|
|
different hypervisors and puts the control and state VMs on a third. It can
|
|
|
|
|
produce the bounded stateless application-tier evidence required by #27 when
|
|
|
|
|
the declared hypervisors are genuinely independent physical failure domains.
|
|
|
|
|
|
|
|
|
|
Both examples use one control-plane VM and one state VM. This keeps the bounded
|
|
|
|
|
#27 target economical, but it does not prove control-plane or state-service
|
|
|
|
|
high availability. For control-plane failover, declare exactly three control
|
|
|
|
|
nodes on independent hosts. PostgreSQL, Redis and object-storage failover must
|
|
|
|
|
be tested against independently operated HA services; the lab's single state
|
|
|
|
|
VM is intentionally a replaceable integration fixture.
|
|
|
|
|
|
|
|
|
|
Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM
|
|
|
|
|
and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs
|
|
|
|
|
additional capacity. Do not overcommit memory on an acceptance target.
|
|
|
|
|
|
|
|
|
|
## 1. Prepare The Hypervisors
|
|
|
|
|
|
|
|
|
|
On each Ubuntu/Debian libvirt host:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
sudo apt-get update
|
|
|
|
|
sudo apt-get install -y \
|
|
|
|
|
qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl
|
|
|
|
|
sudo systemctl enable --now libvirtd
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use a dedicated lab-administration account. The lifecycle invokes `sudo -n`, so
|
|
|
|
|
that account needs non-interactive permission for libvirt, image and cloud-init
|
|
|
|
|
operations. `NOPASSWD: ALL` is acceptable only on isolated lab hypervisors; use
|
|
|
|
|
a bounded sudo policy on shared infrastructure.
|
|
|
|
|
|
|
|
|
|
Create a dedicated SSH key on the management workstation:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \
|
|
|
|
|
-C "GovOPlaN Kubernetes lab"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Install its public key for every remote hypervisor account. The same public key
|
|
|
|
|
is injected into the VMs. The lifecycle keeps its own `ssh_known_hosts` file,
|
|
|
|
|
uses `accept-new` for first contact, and rejects changed host keys until a
|
|
|
|
|
lab-owned VM is deliberately recreated.
|
|
|
|
|
|
|
|
|
|
### Network contract
|
|
|
|
|
|
|
|
|
|
The configured `bridge` must exist on every selected hypervisor. All VM
|
|
|
|
|
addresses are static. Reserve them outside DHCP allocation and ensure that the
|
|
|
|
|
management workstation can route directly to every VM address; the lifecycle
|
|
|
|
|
does not tunnel VM traffic through the hypervisor SSH connection.
|
|
|
|
|
|
|
|
|
|
Permit only these flows inside the lab network:
|
|
|
|
|
|
|
|
|
|
| Port | Source and destination | Purpose |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| TCP 22 | management workstation to every VM/hypervisor | Provisioning and evidence collection |
|
|
|
|
|
| TCP 6443 | all K3s nodes and management path to controls | Kubernetes API |
|
|
|
|
|
| UDP 8472 | K3s node to K3s node | Default Flannel VXLAN; never expose publicly |
|
|
|
|
|
| TCP 10250 | K3s node to K3s node | Kubelet metrics and API |
|
|
|
|
|
| TCP 2379-2380 | control to control, only with three controls | Embedded etcd |
|
|
|
|
|
| TCP 80/443 | test clients to K3s nodes | Traefik/ServiceLB ingress |
|
|
|
|
|
| TCP 5432/6379/9443 | K3s nodes to the state VM | PostgreSQL, Redis and TLS-protected Garage S3 |
|
|
|
|
|
| TCP 3025/3143 | approved test clients/workers to the state VM | GreenMail SMTP/IMAP test endpoints |
|
|
|
|
|
|
|
|
|
|
The official
|
|
|
|
|
[K3s networking requirements](https://docs.k3s.io/installation/requirements#networking)
|
|
|
|
|
remain authoritative. Restrict state ports to the lab network even though the
|
|
|
|
|
generated integration stack binds them on the state VM.
|
|
|
|
|
|
|
|
|
|
## 2. Create The Inventory
|
|
|
|
|
|
|
|
|
|
Start with the one-host rehearsal:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
install -d -m 0700 "$HOME/.config/govoplan/labs"
|
|
|
|
|
cp tools/lab/govoplan-lab.example.toml \
|
|
|
|
|
"$HOME/.config/govoplan/labs/development.toml"
|
|
|
|
|
chmod 0600 "$HOME/.config/govoplan/labs/development.toml"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Edit at least the bridge, network, static addresses and SSH key paths. For a
|
|
|
|
|
multi-host run, copy the acceptance example and replace every example hostname,
|
|
|
|
|
failure-domain declaration and network value. Strict parsing rejects unknown
|
|
|
|
|
keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an
|
|
|
|
|
acceptance inventory that collapses workers onto one declared hypervisor or
|
|
|
|
|
failure domain.
|
|
|
|
|
|
|
|
|
|
Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus
|
|
|
|
|
SHA-256 pairs. Updating means changing those reviewed pins and then running the
|
|
|
|
|
`update` command; the tool deliberately does not follow `latest` aliases.
|
|
|
|
|
|
|
|
|
|
## 3. Validate And Create The VMs
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
LAB="$HOME/.config/govoplan/labs/development.toml"
|
|
|
|
|
PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python"
|
|
|
|
|
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The preview is safe to run repeatedly. Creation reuses a domain whose exact
|
|
|
|
|
lab-owned name already exists and otherwise creates a thin qcow2 overlay under
|
|
|
|
|
`vm_image_directory/<lab>/<node>`.
|
|
|
|
|
|
|
|
|
|
## 4. Deploy GovOPlaN
|
|
|
|
|
|
|
|
|
|
If `git.add-ideas.de` requires authentication for release images, export a
|
|
|
|
|
read-only package/container-registry identity for this shell. A Gitea package
|
|
|
|
|
token can be used as the password:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader'
|
|
|
|
|
read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD
|
|
|
|
|
export GOVOPLAN_LAB_REGISTRY_PASSWORD
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Then preview and apply:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status
|
|
|
|
|
unset GOVOPLAN_LAB_REGISTRY_PASSWORD
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Deployment verifies the downloaded release manifest and keyring by pinned
|
|
|
|
|
digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL,
|
|
|
|
|
Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers
|
|
|
|
|
and scheduler run in K3s from digest-pinned release images. A private lab CA
|
|
|
|
|
protects both ingress and S3; backend pods receive only the CA Secret and keep
|
|
|
|
|
TLS verification enabled.
|
|
|
|
|
|
|
|
|
|
The final output identifies two local files below `state_directory`:
|
|
|
|
|
|
|
|
|
|
- `hosts` maps the public GovOPlaN and S3 test names to their VM addresses;
|
|
|
|
|
- `pki/ca.crt` is the private lab CA certificate.
|
|
|
|
|
|
|
|
|
|
Add the host mappings to the test client's resolver and trust the CA only on
|
|
|
|
|
devices used for this lab. On Debian/Ubuntu:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab"
|
|
|
|
|
cat "$STATE/hosts"
|
|
|
|
|
sudo install -m 0644 "$STATE/pki/ca.crt" \
|
|
|
|
|
/usr/local/share/ca-certificates/govoplan-k8s-lab.crt
|
|
|
|
|
sudo update-ca-certificates
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Review mappings before adding them to `/etc/hosts`; the lifecycle does not edit
|
|
|
|
|
the workstation's trust or resolver configuration.
|
|
|
|
|
|
|
|
|
|
### Enroll the first administrator
|
|
|
|
|
|
|
|
|
|
The production runtime does not create a default password. Issue one expiring,
|
|
|
|
|
single-use first-administrator credential inside an API pod and copy its
|
|
|
|
|
owner-only artifact out immediately:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
KUBECTL="$STATE/bin/kubectl"
|
|
|
|
|
POD="$($KUBECTL -n govoplan get pods \
|
|
|
|
|
-l app.kubernetes.io/component=api \
|
|
|
|
|
-o jsonpath='{.items[0].metadata.name}')"
|
|
|
|
|
ARTIFACT="$STATE/first-admin-enrollment.json"
|
|
|
|
|
umask 077
|
|
|
|
|
|
|
|
|
|
$KUBECTL -n govoplan exec "$POD" -- \
|
|
|
|
|
python -m govoplan_core.commands.first_admin issue \
|
|
|
|
|
--reason 'initial Kubernetes lab enrollment' \
|
|
|
|
|
--output /tmp/first-admin-enrollment.json
|
|
|
|
|
$KUBECTL -n govoplan exec "$POD" -- \
|
|
|
|
|
cat /tmp/first-admin-enrollment.json > "$ARTIFACT"
|
|
|
|
|
$KUBECTL -n govoplan exec "$POD" -- \
|
|
|
|
|
rm -f /tmp/first-admin-enrollment.json
|
|
|
|
|
chmod 0600 "$ARTIFACT"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Submit the token from that artifact once to
|
|
|
|
|
`/api/v1/bootstrap/first-admin` with the administrator email, display name,
|
|
|
|
|
password, tenant slug and tenant name. The password must contain at least 12
|
|
|
|
|
characters. Delete the local artifact after successful enrollment.
|
|
|
|
|
|
|
|
|
|
## 5. Collect #27 Evidence
|
|
|
|
|
|
|
|
|
|
Create a short-lived API key with only `ops:operations:read`, then run:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
export GOVOPLAN_OPS_API_KEY='short-lived-value'
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \
|
|
|
|
|
--exercise-api-pod-loss
|
|
|
|
|
unset GOVOPLAN_OPS_API_KEY
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The verifier requires ready API and WebUI pods across at least two Kubernetes
|
|
|
|
|
nodes, all Deployments available, consistent runtime composition, queue
|
|
|
|
|
coverage and a valid database-connection budget. During the optional drill it
|
|
|
|
|
deletes one ready API pod, probes public readiness and waits for replacement.
|
|
|
|
|
It writes sanitized output to
|
|
|
|
|
`state_directory/evidence/kubernetes-multi-host.json` and never stores the API
|
|
|
|
|
key. A rehearsal inventory prints an explicit warning that its result is not
|
|
|
|
|
independent-failure-domain evidence.
|
|
|
|
|
|
|
|
|
|
Retain these private artifacts together for review:
|
|
|
|
|
|
|
|
|
|
1. `inventory.json` and the reviewed inventory TOML;
|
|
|
|
|
2. the adopted release manifest/keyring and installation receipt;
|
|
|
|
|
3. `kubernetes.json`;
|
|
|
|
|
4. the Kubernetes verifier output;
|
|
|
|
|
5. private cluster logs for the approved drill window;
|
|
|
|
|
6. the operator's out-of-band evidence that the worker hypervisors are
|
|
|
|
|
independent physical hosts or availability zones.
|
|
|
|
|
|
|
|
|
|
GovOPlaN #37 additionally requires independent assessment and production
|
|
|
|
|
approval keys. Running its evidence jobs in containers is supported, but a
|
|
|
|
|
container does not create an independent authority. Follow
|
|
|
|
|
`TARGET_MATURITY_EVIDENCE_RUNBOOK.md` after the #27 drill passes.
|
|
|
|
|
|
|
|
|
|
## 6. Update, Pause, Resume And Remove
|
|
|
|
|
|
|
|
|
|
After reviewing and changing pinned image/K3s/release values in the inventory:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Workers are cordoned, drained, updated and uncordoned one at a time. K3s
|
|
|
|
|
controls are reconciled serially. The release-specific migration Job remains
|
|
|
|
|
subject to GovOPlaN's signed backup-evidence gate. The lab update command is not
|
|
|
|
|
a substitute for creating recovery evidence before a destructive state-schema
|
|
|
|
|
change.
|
|
|
|
|
|
|
|
|
|
To stop compute use without deleting disks:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
To remove VM resources while preserving local evidence:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy
|
|
|
|
|
"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \
|
|
|
|
|
--apply --confirm govoplan-k8s-lab
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Add `--purge-local-state` only after evidence and recovery material have been
|
|
|
|
|
retained elsewhere. That option deletes the generated local CA, secrets,
|
|
|
|
|
manifests and evidence as well as the VMs.
|
|
|
|
|
|
|
|
|
|
## Acceptance Boundary
|
|
|
|
|
|
|
|
|
|
This tool supplies reproducible infrastructure and executes the bounded
|
|
|
|
|
stateless-node drill. It does not certify the truth of operator-entered failure
|
|
|
|
|
domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence,
|
|
|
|
|
or approve its own results. Those boundaries are deliberate: #27 can close
|
|
|
|
|
after a passing run on independently controlled hosts; broader production
|
|
|
|
|
maturity remains governed by #35, #37 and the target evidence runbook.
|
