diff --git a/Codex-Project-Index.md b/Codex-Project-Index.md index 0d3142a..7539daf 100644 --- a/Codex-Project-Index.md +++ b/Codex-Project-Index.md @@ -18,6 +18,7 @@ This page is generated from repository and product-directory project files. - [Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE](Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE) - `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md` - [Repo-docs-INTERFACE-PATTERN-LANGUAGE](Repo-docs-INTERFACE-PATTERN-LANGUAGE) - `/mnt/DATA/git/govoplan/docs/INTERFACE_PATTERN_LANGUAGE.md` - [Repo-docs-INTERFACE-SURFACE-INVENTORY](Repo-docs-INTERFACE-SURFACE-INVENTORY) - `/mnt/DATA/git/govoplan/docs/INTERFACE_SURFACE_INVENTORY.md` +- [Repo-docs-KUBERNETES-TEST-LAB](Repo-docs-KUBERNETES-TEST-LAB) - `/mnt/DATA/git/govoplan/docs/KUBERNETES_TEST_LAB.md` - [Repo-docs-META-REPO-SCAN](Repo-docs-META-REPO-SCAN) - `/mnt/DATA/git/govoplan/docs/META_REPO_SCAN.md` - [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md` - [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md` diff --git a/Repo-README.md b/Repo-README.md index 390279e..3770bcc 100644 --- a/Repo-README.md +++ b/Repo-README.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/README.md`. > Origin: `repository`. @@ -180,6 +180,9 @@ The current executable slice and remaining production gates are documented in [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). Same-host replica balancing and the multi-host promotion boundary are documented in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). +Create, update, pause, resume, verify and remove a local or multi-hypervisor K3s +VM target with the guarded lifecycle documented in +[Kubernetes VM Test Lab](docs/KUBERNETES_TEST_LAB.md). The recovery state machine, migration rollback boundary, and required restore drills are documented in [Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md). diff --git a/Repo-docs-KUBERNETES-TEST-LAB.md b/Repo-docs-KUBERNETES-TEST-LAB.md new file mode 100644 index 0000000..95c2f41 --- /dev/null +++ b/Repo-docs-KUBERNETES-TEST-LAB.md @@ -0,0 +1,299 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan/docs/KUBERNETES_TEST_LAB.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Kubernetes VM Test Lab + +`tools/lab/govoplan-lab.py` creates and operates an amd64 Ubuntu/K3s test +environment on local or SSH-accessible libvirt hypervisors. It provides the +commands requested for the complete VM lifecycle: + +| Command | Effect | +| --- | --- | +| `doctor` | Validate the strict inventory and, with `--online`, every hypervisor. | +| `create --apply` | Download checksum-pinned cloud images, create VM overlays and boot the declared VMs. | +| `deploy --apply` | Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN. | +| `update --apply` | Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release. | +| `status` | Show libvirt VM state, Kubernetes nodes and GovOPlaN pods. | +| `pause --apply` | Gracefully shut down workers, control planes and shared state while retaining disks. | +| `resume --apply` | Start the retained environment in dependency order and wait for readiness. | +| `verify` | Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill. | +| `destroy --apply --confirm ` | Delete only the lab-owned domains and overlays; local evidence is retained by default. | + +Every mutating command is a dry run unless `--apply` is present. Destruction +also requires the exact lab name. Generated credentials, CA keys, manifests and +evidence are written below the configured `state_directory` with owner-only +permissions. Keep that directory outside the repository and include it in the +workstation backup policy. + +## What The Lab Proves + +The supplied inventories describe two different assurance levels: + +- `tools/lab/govoplan-lab.example.toml` creates four VMs on one libvirt host. + It is suitable for development, deployment rehearsal, migration testing, + application-pod replacement and recovery-tool exercises. It cannot close + GovOPlaN #27 because one physical host remains one failure domain. +- `tools/lab/govoplan-lab.acceptance.example.toml` places the two workers on + different hypervisors and puts the control and state VMs on a third. It can + produce the bounded stateless application-tier evidence required by #27 when + the declared hypervisors are genuinely independent physical failure domains. + +Both examples use one control-plane VM and one state VM. This keeps the bounded +#27 target economical, but it does not prove control-plane or state-service +high availability. For control-plane failover, declare exactly three control +nodes on independent hosts. PostgreSQL, Redis and object-storage failover must +be tested against independently operated HA services; the lab's single state +VM is intentionally a replaceable integration fixture. + +Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM +and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs +additional capacity. Do not overcommit memory on an acceptance target. + +## 1. Prepare The Hypervisors + +On each Ubuntu/Debian libvirt host: + +```bash +sudo apt-get update +sudo apt-get install -y \ + qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl +sudo systemctl enable --now libvirtd +``` + +Use a dedicated lab-administration account. The lifecycle invokes `sudo -n`, so +that account needs non-interactive permission for libvirt, image and cloud-init +operations. `NOPASSWD: ALL` is acceptable only on isolated lab hypervisors; use +a bounded sudo policy on shared infrastructure. + +Create a dedicated SSH key on the management workstation: + +```bash +ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \ + -C "GovOPlaN Kubernetes lab" +``` + +Install its public key for every remote hypervisor account. The same public key +is injected into the VMs. The lifecycle keeps its own `ssh_known_hosts` file, +uses `accept-new` for first contact, and rejects changed host keys until a +lab-owned VM is deliberately recreated. + +### Network contract + +The configured `bridge` must exist on every selected hypervisor. All VM +addresses are static. Reserve them outside DHCP allocation and ensure that the +management workstation can route directly to every VM address; the lifecycle +does not tunnel VM traffic through the hypervisor SSH connection. + +Permit only these flows inside the lab network: + +| Port | Source and destination | Purpose | +| --- | --- | --- | +| TCP 22 | management workstation to every VM/hypervisor | Provisioning and evidence collection | +| TCP 6443 | all K3s nodes and management path to controls | Kubernetes API | +| UDP 8472 | K3s node to K3s node | Default Flannel VXLAN; never expose publicly | +| TCP 10250 | K3s node to K3s node | Kubelet metrics and API | +| TCP 2379-2380 | control to control, only with three controls | Embedded etcd | +| TCP 80/443 | test clients to K3s nodes | Traefik/ServiceLB ingress | +| TCP 5432/6379/9443 | K3s nodes to the state VM | PostgreSQL, Redis and TLS-protected Garage S3 | +| TCP 3025/3143 | approved test clients/workers to the state VM | GreenMail SMTP/IMAP test endpoints | + +The official +[K3s networking requirements](https://docs.k3s.io/installation/requirements#networking) +remain authoritative. Restrict state ports to the lab network even though the +generated integration stack binds them on the state VM. + +## 2. Create The Inventory + +Start with the one-host rehearsal: + +```bash +install -d -m 0700 "$HOME/.config/govoplan/labs" +cp tools/lab/govoplan-lab.example.toml \ + "$HOME/.config/govoplan/labs/development.toml" +chmod 0600 "$HOME/.config/govoplan/labs/development.toml" +``` + +Edit at least the bridge, network, static addresses and SSH key paths. For a +multi-host run, copy the acceptance example and replace every example hostname, +failure-domain declaration and network value. Strict parsing rejects unknown +keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an +acceptance inventory that collapses workers onto one declared hypervisor or +failure domain. + +Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus +SHA-256 pairs. Updating means changing those reviewed pins and then running the +`update` command; the tool deliberately does not follow `latest` aliases. + +## 3. Validate And Create The VMs + +```bash +LAB="$HOME/.config/govoplan/labs/development.toml" +PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python" + +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply +``` + +The preview is safe to run repeatedly. Creation reuses a domain whose exact +lab-owned name already exists and otherwise creates a thin qcow2 overlay under +`vm_image_directory//`. + +## 4. Deploy GovOPlaN + +If `git.add-ideas.de` requires authentication for release images, export a +read-only package/container-registry identity for this shell. A Gitea package +token can be used as the password: + +```bash +export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader' +read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD +export GOVOPLAN_LAB_REGISTRY_PASSWORD +``` + +Then preview and apply: + +```bash +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status +unset GOVOPLAN_LAB_REGISTRY_PASSWORD +``` + +Deployment verifies the downloaded release manifest and keyring by pinned +digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL, +Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers +and scheduler run in K3s from digest-pinned release images. A private lab CA +protects both ingress and S3; backend pods receive only the CA Secret and keep +TLS verification enabled. + +The final output identifies two local files below `state_directory`: + +- `hosts` maps the public GovOPlaN and S3 test names to their VM addresses; +- `pki/ca.crt` is the private lab CA certificate. + +Add the host mappings to the test client's resolver and trust the CA only on +devices used for this lab. On Debian/Ubuntu: + +```bash +STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab" +cat "$STATE/hosts" +sudo install -m 0644 "$STATE/pki/ca.crt" \ + /usr/local/share/ca-certificates/govoplan-k8s-lab.crt +sudo update-ca-certificates +``` + +Review mappings before adding them to `/etc/hosts`; the lifecycle does not edit +the workstation's trust or resolver configuration. + +### Enroll the first administrator + +The production runtime does not create a default password. Issue one expiring, +single-use first-administrator credential inside an API pod and copy its +owner-only artifact out immediately: + +```bash +KUBECTL="$STATE/bin/kubectl" +POD="$($KUBECTL -n govoplan get pods \ + -l app.kubernetes.io/component=api \ + -o jsonpath='{.items[0].metadata.name}')" +ARTIFACT="$STATE/first-admin-enrollment.json" +umask 077 + +$KUBECTL -n govoplan exec "$POD" -- \ + python -m govoplan_core.commands.first_admin issue \ + --reason 'initial Kubernetes lab enrollment' \ + --output /tmp/first-admin-enrollment.json +$KUBECTL -n govoplan exec "$POD" -- \ + cat /tmp/first-admin-enrollment.json > "$ARTIFACT" +$KUBECTL -n govoplan exec "$POD" -- \ + rm -f /tmp/first-admin-enrollment.json +chmod 0600 "$ARTIFACT" +``` + +Submit the token from that artifact once to +`/api/v1/bootstrap/first-admin` with the administrator email, display name, +password, tenant slug and tenant name. The password must contain at least 12 +characters. Delete the local artifact after successful enrollment. + +## 5. Collect #27 Evidence + +Create a short-lived API key with only `ops:operations:read`, then run: + +```bash +export GOVOPLAN_OPS_API_KEY='short-lived-value' +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \ + --exercise-api-pod-loss +unset GOVOPLAN_OPS_API_KEY +``` + +The verifier requires ready API and WebUI pods across at least two Kubernetes +nodes, all Deployments available, consistent runtime composition, queue +coverage and a valid database-connection budget. During the optional drill it +deletes one ready API pod, probes public readiness and waits for replacement. +It writes sanitized output to +`state_directory/evidence/kubernetes-multi-host.json` and never stores the API +key. A rehearsal inventory prints an explicit warning that its result is not +independent-failure-domain evidence. + +Retain these private artifacts together for review: + +1. `inventory.json` and the reviewed inventory TOML; +2. the adopted release manifest/keyring and installation receipt; +3. `kubernetes.json`; +4. the Kubernetes verifier output; +5. private cluster logs for the approved drill window; +6. the operator's out-of-band evidence that the worker hypervisors are + independent physical hosts or availability zones. + +GovOPlaN #37 additionally requires independent assessment and production +approval keys. Running its evidence jobs in containers is supported, but a +container does not create an independent authority. Follow +`TARGET_MATURITY_EVIDENCE_RUNBOOK.md` after the #27 drill passes. + +## 6. Update, Pause, Resume And Remove + +After reviewing and changing pinned image/K3s/release values in the inventory: + +```bash +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply +``` + +Workers are cordoned, drained, updated and uncordoned one at a time. K3s +controls are reconciled serially. The release-specific migration Job remains +subject to GovOPlaN's signed backup-evidence gate. The lab update command is not +a substitute for creating recovery evidence before a destructive state-schema +change. + +To stop compute use without deleting disks: + +```bash +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply +``` + +To remove VM resources while preserving local evidence: + +```bash +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy +"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \ + --apply --confirm govoplan-k8s-lab +``` + +Add `--purge-local-state` only after evidence and recovery material have been +retained elsewhere. That option deletes the generated local CA, secrets, +manifests and evidence as well as the VMs. + +## Acceptance Boundary + +This tool supplies reproducible infrastructure and executes the bounded +stateless-node drill. It does not certify the truth of operator-entered failure +domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence, +or approve its own results. Those boundaries are deliberate: #27 can close +after a passing run on independently controlled hosts; broader production +maturity remains governed by #35, #37 and the target evidence runbook. diff --git a/Repo-docs-PRODUCTION-TARGET-HANDOFF.md b/Repo-docs-PRODUCTION-TARGET-HANDOFF.md index 599ea29..2158ef6 100644 --- a/Repo-docs-PRODUCTION-TARGET-HANDOFF.md +++ b/Repo-docs-PRODUCTION-TARGET-HANDOFF.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/PRODUCTION_TARGET_HANDOFF.md`. > Origin: `repository`. @@ -56,6 +56,13 @@ firewall ports. Those inputs are sufficient to provision a k3s target. They are not sufficient to claim control-plane HA unless three control-plane failure domains are present. +The repository now supplies the strict libvirt/K3s lifecycle and example +inventories for this handoff in +[`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md). Its `acceptance` mode +rejects a declared topology unless the workers and shared-state fixture occupy +different hypervisor and failure-domain identifiers. Reviewers must still +verify that those identifiers correspond to genuinely independent hosts. + ### Separate deployment and evidence authorities The deployment identity may create and update the namespace, Secret, @@ -81,6 +88,7 @@ python tools/deployment/govoplan-deploy.py render-kubernetes \ --namespace govoplan \ --secret-name govoplan-runtime \ --tls-secret-name govoplan-tls \ + --s3-ca-secret-name govoplan-s3-ca \ --ingress-class-name nginx \ --output /srv/govoplan//kubernetes.json diff --git a/Repo-docs-README.md b/Repo-docs-README.md index 1ca85ab..f907a97 100644 --- a/Repo-docs-README.md +++ b/Repo-docs-README.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/README.md`. > Origin: `repository`. @@ -45,6 +45,7 @@ second live status page. - [Module Contracts and Installs](MODULE_CONTRACTS_AND_INSTALLS.md) - [Platform Control Plane](PLATFORM_CONTROL_PLANE.md) - [Installation and Deployment Architecture](INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md) +- [Kubernetes VM Test Lab](KUBERNETES_TEST_LAB.md) - [Scaling and Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) - [Recovery and Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md) - [Recovery Ledger Adoption](RECOVERY_LEDGER_ADOPTION.md) diff --git a/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md b/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md index 4ea9189..2fea9af 100644 --- a/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md +++ b/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`. > Origin: `repository`. @@ -10,6 +10,8 @@ For the exact external handoff, least-privilege collector permissions and live two-node acceptance procedure, see [`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md). +For a reproducible local or multi-hypervisor libvirt/K3s target, use +[`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md). ## Implemented Contract @@ -81,6 +83,7 @@ python tools/deployment/govoplan-deploy.py render-kubernetes \ --namespace govoplan \ --secret-name govoplan-runtime \ --tls-secret-name govoplan-tls \ + --s3-ca-secret-name govoplan-s3-ca \ --ingress-class-name nginx \ --output /srv/govoplan/default/kubernetes.json ``` @@ -97,6 +100,12 @@ command prints the exact required key contract. Review the generated `FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network before production use. +When an external S3 endpoint is signed by a private CA, create the optional CA +Secret with a `ca.crt` key and pass `--s3-ca-secret-name`. The renderer mounts +that Secret read-only and sets `AWS_CA_BUNDLE` for API, worker, scheduler, +migration and database-wait containers. It does not disable certificate +verification or replace the WebUI trust store. + The generated containers run as non-root with a read-only root filesystem and an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database migration heads before starting. The API exposes `/health/ready`, which fails