Sync wiki from project files

Codex-Project-Index.md

@@ -18,6 +18,7 @@ This page is generated from repository and product-directory project files.
 - [Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE](Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE) - `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md`
 - [Repo-docs-INTERFACE-PATTERN-LANGUAGE](Repo-docs-INTERFACE-PATTERN-LANGUAGE) - `/mnt/DATA/git/govoplan/docs/INTERFACE_PATTERN_LANGUAGE.md`
 - [Repo-docs-INTERFACE-SURFACE-INVENTORY](Repo-docs-INTERFACE-SURFACE-INVENTORY) - `/mnt/DATA/git/govoplan/docs/INTERFACE_SURFACE_INVENTORY.md`
+- [Repo-docs-KUBERNETES-TEST-LAB](Repo-docs-KUBERNETES-TEST-LAB) - `/mnt/DATA/git/govoplan/docs/KUBERNETES_TEST_LAB.md`
 - [Repo-docs-META-REPO-SCAN](Repo-docs-META-REPO-SCAN) - `/mnt/DATA/git/govoplan/docs/META_REPO_SCAN.md`
 - [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md`
 - [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md`

Repo-README.md

@@ -1,4 +1,4 @@
-<!-- codex-wiki-sync:30b3fe88cdb697b19a734447 -->
+<!-- codex-wiki-sync:7c8af531fbcb498e2147a78e -->
 
 > Mirrored from `/mnt/DATA/git/govoplan/README.md`.
 > Origin: `repository`.
@@ -180,6 +180,9 @@ The current executable slice and remaining production gates are documented in
 [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
 Same-host replica balancing and the multi-host promotion boundary are documented
 in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
+Create, update, pause, resume, verify and remove a local or multi-hypervisor K3s
+VM target with the guarded lifecycle documented in
+[Kubernetes VM Test Lab](docs/KUBERNETES_TEST_LAB.md).
 The recovery state machine, migration rollback boundary, and required restore
 drills are documented in
 [Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md).

Repo-docs-KUBERNETES-TEST-LAB.md

@@ -0,0 +1,299 @@
+<!-- codex-wiki-sync:6c6bab4e7696c71be8a2bbad -->
+
+> Mirrored from `/mnt/DATA/git/govoplan/docs/KUBERNETES_TEST_LAB.md`.
+> Origin: `repository`.
+> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
+
+---
+# Kubernetes VM Test Lab
+
+`tools/lab/govoplan-lab.py` creates and operates an amd64 Ubuntu/K3s test
+environment on local or SSH-accessible libvirt hypervisors. It provides the
+commands requested for the complete VM lifecycle:
+
+| Command | Effect |
+| --- | --- |
+| `doctor` | Validate the strict inventory and, with `--online`, every hypervisor. |
+| `create --apply` | Download checksum-pinned cloud images, create VM overlays and boot the declared VMs. |
+| `deploy --apply` | Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN. |
+| `update --apply` | Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release. |
+| `status` | Show libvirt VM state, Kubernetes nodes and GovOPlaN pods. |
+| `pause --apply` | Gracefully shut down workers, control planes and shared state while retaining disks. |
+| `resume --apply` | Start the retained environment in dependency order and wait for readiness. |
+| `verify` | Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill. |
+| `destroy --apply --confirm <lab>` | Delete only the lab-owned domains and overlays; local evidence is retained by default. |
+
+Every mutating command is a dry run unless `--apply` is present. Destruction
+also requires the exact lab name. Generated credentials, CA keys, manifests and
+evidence are written below the configured `state_directory` with owner-only
+permissions. Keep that directory outside the repository and include it in the
+workstation backup policy.
+
+## What The Lab Proves
+
+The supplied inventories describe two different assurance levels:
+
+- `tools/lab/govoplan-lab.example.toml` creates four VMs on one libvirt host.
+  It is suitable for development, deployment rehearsal, migration testing,
+  application-pod replacement and recovery-tool exercises. It cannot close
+  GovOPlaN #27 because one physical host remains one failure domain.
+- `tools/lab/govoplan-lab.acceptance.example.toml` places the two workers on
+  different hypervisors and puts the control and state VMs on a third. It can
+  produce the bounded stateless application-tier evidence required by #27 when
+  the declared hypervisors are genuinely independent physical failure domains.
+
+Both examples use one control-plane VM and one state VM. This keeps the bounded
+#27 target economical, but it does not prove control-plane or state-service
+high availability. For control-plane failover, declare exactly three control
+nodes on independent hosts. PostgreSQL, Redis and object-storage failover must
+be tested against independently operated HA services; the lab's single state
+VM is intentionally a replaceable integration fixture.
+
+Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM
+and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs
+additional capacity. Do not overcommit memory on an acceptance target.
+
+## 1. Prepare The Hypervisors
+
+On each Ubuntu/Debian libvirt host:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y \
+  qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl
+sudo systemctl enable --now libvirtd
+```
+
+Use a dedicated lab-administration account. The lifecycle invokes `sudo -n`, so
+that account needs non-interactive permission for libvirt, image and cloud-init
+operations. `NOPASSWD: ALL` is acceptable only on isolated lab hypervisors; use
+a bounded sudo policy on shared infrastructure.
+
+Create a dedicated SSH key on the management workstation:
+
+```bash
+ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \
+  -C "GovOPlaN Kubernetes lab"
+```
+
+Install its public key for every remote hypervisor account. The same public key
+is injected into the VMs. The lifecycle keeps its own `ssh_known_hosts` file,
+uses `accept-new` for first contact, and rejects changed host keys until a
+lab-owned VM is deliberately recreated.
+
+### Network contract
+
+The configured `bridge` must exist on every selected hypervisor. All VM
+addresses are static. Reserve them outside DHCP allocation and ensure that the
+management workstation can route directly to every VM address; the lifecycle
+does not tunnel VM traffic through the hypervisor SSH connection.
+
+Permit only these flows inside the lab network:
+
+| Port | Source and destination | Purpose |
+| --- | --- | --- |
+| TCP 22 | management workstation to every VM/hypervisor | Provisioning and evidence collection |
+| TCP 6443 | all K3s nodes and management path to controls | Kubernetes API |
+| UDP 8472 | K3s node to K3s node | Default Flannel VXLAN; never expose publicly |
+| TCP 10250 | K3s node to K3s node | Kubelet metrics and API |
+| TCP 2379-2380 | control to control, only with three controls | Embedded etcd |
+| TCP 80/443 | test clients to K3s nodes | Traefik/ServiceLB ingress |
+| TCP 5432/6379/9443 | K3s nodes to the state VM | PostgreSQL, Redis and TLS-protected Garage S3 |
+| TCP 3025/3143 | approved test clients/workers to the state VM | GreenMail SMTP/IMAP test endpoints |
+
+The official
+[K3s networking requirements](https://docs.k3s.io/installation/requirements#networking)
+remain authoritative. Restrict state ports to the lab network even though the
+generated integration stack binds them on the state VM.
+
+## 2. Create The Inventory
+
+Start with the one-host rehearsal:
+
+```bash
+install -d -m 0700 "$HOME/.config/govoplan/labs"
+cp tools/lab/govoplan-lab.example.toml \
+  "$HOME/.config/govoplan/labs/development.toml"
+chmod 0600 "$HOME/.config/govoplan/labs/development.toml"
+```
+
+Edit at least the bridge, network, static addresses and SSH key paths. For a
+multi-host run, copy the acceptance example and replace every example hostname,
+failure-domain declaration and network value. Strict parsing rejects unknown
+keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an
+acceptance inventory that collapses workers onto one declared hypervisor or
+failure domain.
+
+Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus
+SHA-256 pairs. Updating means changing those reviewed pins and then running the
+`update` command; the tool deliberately does not follow `latest` aliases.
+
+## 3. Validate And Create The VMs
+
+```bash
+LAB="$HOME/.config/govoplan/labs/development.toml"
+PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python"
+
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply
+```
+
+The preview is safe to run repeatedly. Creation reuses a domain whose exact
+lab-owned name already exists and otherwise creates a thin qcow2 overlay under
+`vm_image_directory/<lab>/<node>`.
+
+## 4. Deploy GovOPlaN
+
+If `git.add-ideas.de` requires authentication for release images, export a
+read-only package/container-registry identity for this shell. A Gitea package
+token can be used as the password:
+
+```bash
+export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader'
+read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD
+export GOVOPLAN_LAB_REGISTRY_PASSWORD
+```
+
+Then preview and apply:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status
+unset GOVOPLAN_LAB_REGISTRY_PASSWORD
+```
+
+Deployment verifies the downloaded release manifest and keyring by pinned
+digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL,
+Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers
+and scheduler run in K3s from digest-pinned release images. A private lab CA
+protects both ingress and S3; backend pods receive only the CA Secret and keep
+TLS verification enabled.
+
+The final output identifies two local files below `state_directory`:
+
+- `hosts` maps the public GovOPlaN and S3 test names to their VM addresses;
+- `pki/ca.crt` is the private lab CA certificate.
+
+Add the host mappings to the test client's resolver and trust the CA only on
+devices used for this lab. On Debian/Ubuntu:
+
+```bash
+STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab"
+cat "$STATE/hosts"
+sudo install -m 0644 "$STATE/pki/ca.crt" \
+  /usr/local/share/ca-certificates/govoplan-k8s-lab.crt
+sudo update-ca-certificates
+```
+
+Review mappings before adding them to `/etc/hosts`; the lifecycle does not edit
+the workstation's trust or resolver configuration.
+
+### Enroll the first administrator
+
+The production runtime does not create a default password. Issue one expiring,
+single-use first-administrator credential inside an API pod and copy its
+owner-only artifact out immediately:
+
+```bash
+KUBECTL="$STATE/bin/kubectl"
+POD="$($KUBECTL -n govoplan get pods \
+  -l app.kubernetes.io/component=api \
+  -o jsonpath='{.items[0].metadata.name}')"
+ARTIFACT="$STATE/first-admin-enrollment.json"
+umask 077
+
+$KUBECTL -n govoplan exec "$POD" -- \
+  python -m govoplan_core.commands.first_admin issue \
+  --reason 'initial Kubernetes lab enrollment' \
+  --output /tmp/first-admin-enrollment.json
+$KUBECTL -n govoplan exec "$POD" -- \
+  cat /tmp/first-admin-enrollment.json > "$ARTIFACT"
+$KUBECTL -n govoplan exec "$POD" -- \
+  rm -f /tmp/first-admin-enrollment.json
+chmod 0600 "$ARTIFACT"
+```
+
+Submit the token from that artifact once to
+`/api/v1/bootstrap/first-admin` with the administrator email, display name,
+password, tenant slug and tenant name. The password must contain at least 12
+characters. Delete the local artifact after successful enrollment.
+
+## 5. Collect #27 Evidence
+
+Create a short-lived API key with only `ops:operations:read`, then run:
+
+```bash
+export GOVOPLAN_OPS_API_KEY='short-lived-value'
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \
+  --exercise-api-pod-loss
+unset GOVOPLAN_OPS_API_KEY
+```
+
+The verifier requires ready API and WebUI pods across at least two Kubernetes
+nodes, all Deployments available, consistent runtime composition, queue
+coverage and a valid database-connection budget. During the optional drill it
+deletes one ready API pod, probes public readiness and waits for replacement.
+It writes sanitized output to
+`state_directory/evidence/kubernetes-multi-host.json` and never stores the API
+key. A rehearsal inventory prints an explicit warning that its result is not
+independent-failure-domain evidence.
+
+Retain these private artifacts together for review:
+
+1. `inventory.json` and the reviewed inventory TOML;
+2. the adopted release manifest/keyring and installation receipt;
+3. `kubernetes.json`;
+4. the Kubernetes verifier output;
+5. private cluster logs for the approved drill window;
+6. the operator's out-of-band evidence that the worker hypervisors are
+   independent physical hosts or availability zones.
+
+GovOPlaN #37 additionally requires independent assessment and production
+approval keys. Running its evidence jobs in containers is supported, but a
+container does not create an independent authority. Follow
+`TARGET_MATURITY_EVIDENCE_RUNBOOK.md` after the #27 drill passes.
+
+## 6. Update, Pause, Resume And Remove
+
+After reviewing and changing pinned image/K3s/release values in the inventory:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply
+```
+
+Workers are cordoned, drained, updated and uncordoned one at a time. K3s
+controls are reconciled serially. The release-specific migration Job remains
+subject to GovOPlaN's signed backup-evidence gate. The lab update command is not
+a substitute for creating recovery evidence before a destructive state-schema
+change.
+
+To stop compute use without deleting disks:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply
+```
+
+To remove VM resources while preserving local evidence:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \
+  --apply --confirm govoplan-k8s-lab
+```
+
+Add `--purge-local-state` only after evidence and recovery material have been
+retained elsewhere. That option deletes the generated local CA, secrets,
+manifests and evidence as well as the VMs.
+
+## Acceptance Boundary
+
+This tool supplies reproducible infrastructure and executes the bounded
+stateless-node drill. It does not certify the truth of operator-entered failure
+domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence,
+or approve its own results. Those boundaries are deliberate: #27 can close
+after a passing run on independently controlled hosts; broader production
+maturity remains governed by #35, #37 and the target evidence runbook.

Repo-docs-PRODUCTION-TARGET-HANDOFF.md

@@ -1,4 +1,4 @@
-<!-- codex-wiki-sync:f300a18700ddb6305b431a33 -->
+<!-- codex-wiki-sync:e36fc00d5c1106c77ccee6a5 -->
 
 > Mirrored from `/mnt/DATA/git/govoplan/docs/PRODUCTION_TARGET_HANDOFF.md`.
 > Origin: `repository`.
@@ -56,6 +56,13 @@ firewall ports. Those inputs are sufficient to provision a k3s target. They are
 not sufficient to claim control-plane HA unless three control-plane failure
 domains are present.
 
+The repository now supplies the strict libvirt/K3s lifecycle and example
+inventories for this handoff in
+[`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md). Its `acceptance` mode
+rejects a declared topology unless the workers and shared-state fixture occupy
+different hypervisor and failure-domain identifiers. Reviewers must still
+verify that those identifiers correspond to genuinely independent hosts.
+
 ### Separate deployment and evidence authorities
 
 The deployment identity may create and update the namespace, Secret,
@@ -81,6 +88,7 @@ python tools/deployment/govoplan-deploy.py render-kubernetes \
   --namespace govoplan \
   --secret-name govoplan-runtime \
   --tls-secret-name govoplan-tls \
+  --s3-ca-secret-name govoplan-s3-ca \
   --ingress-class-name nginx \
   --output /srv/govoplan/<installation-id>/kubernetes.json
 

Repo-docs-README.md

@@ -1,4 +1,4 @@
-<!-- codex-wiki-sync:b3a66bfea806aebd00371934 -->
+<!-- codex-wiki-sync:e59c13a76eecd85ff5105327 -->
 
 > Mirrored from `/mnt/DATA/git/govoplan/docs/README.md`.
 > Origin: `repository`.
@@ -45,6 +45,7 @@ second live status page.
 - [Module Contracts and Installs](MODULE_CONTRACTS_AND_INSTALLS.md)
 - [Platform Control Plane](PLATFORM_CONTROL_PLANE.md)
 - [Installation and Deployment Architecture](INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md)
+- [Kubernetes VM Test Lab](KUBERNETES_TEST_LAB.md)
 - [Scaling and Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md)
 - [Recovery and Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md)
 - [Recovery Ledger Adoption](RECOVERY_LEDGER_ADOPTION.md)

Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md

@@ -1,4 +1,4 @@
-<!-- codex-wiki-sync:26d7e61532af09dda11e554f -->
+<!-- codex-wiki-sync:672fe8294424be9b05593892 -->
 
 > Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`.
 > Origin: `repository`.
@@ -10,6 +10,8 @@
 For the exact external handoff, least-privilege collector permissions and live
 two-node acceptance procedure, see
 [`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
+For a reproducible local or multi-hypervisor libvirt/K3s target, use
+[`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md).
 
 ## Implemented Contract
 
@@ -81,6 +83,7 @@ python tools/deployment/govoplan-deploy.py render-kubernetes \
   --namespace govoplan \
   --secret-name govoplan-runtime \
   --tls-secret-name govoplan-tls \
+  --s3-ca-secret-name govoplan-s3-ca \
   --ingress-class-name nginx \
   --output /srv/govoplan/default/kubernetes.json
 ```
@@ -97,6 +100,12 @@ command prints the exact required key contract. Review the generated
 `FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network
 before production use.
 
+When an external S3 endpoint is signed by a private CA, create the optional CA
+Secret with a `ca.crt` key and pass `--s3-ca-secret-name`. The renderer mounts
+that Secret read-only and sets `AWS_CA_BUNDLE` for API, worker, scheduler,
+migration and database-wait containers. It does not disable certificate
+verification or replace the WebUI trust store.
+
 The generated containers run as non-root with a read-only root filesystem and
 an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database
 migration heads before starting. The API exposes `/health/ready`, which fails