docs: organize cross-product documentation

docs/operations/BACKUP_AND_RESTORE_EVIDENCE.md

@@ -0,0 +1,133 @@
+# Backup And Restore Evidence
+
+## Boundary
+
+`govoplan-deploy` verifies backup and restore evidence; it does not receive
+database, object-store, KMS, or orchestrator administration credentials and it
+does not create the backup. A provider-owned backup controller creates one
+coordinated recovery point, a separate drill runner restores it into an
+isolated target, and an evidence authority signs the resulting receipt.
+
+The application containers receive only a sanitized projection: evidence,
+recovery-point and drill identifiers, hashes, timestamps, component count, and
+measured RPO/RTO. Artifact locations, provider credentials, encryption-key
+references, the public trust keyring, and private signing keys remain in the
+deployment/evidence boundary.
+
+The machine-readable contracts are:
+
+- [`backup-evidence.schema.json`](../backup-evidence.schema.json);
+- [`backup-evidence-keyring.schema.json`](../backup-evidence-keyring.schema.json).
+
+One evidence document is bound to the installation id, deployment profile,
+topology subject, exact signed release manifest, image digests, and composition
+digest. It covers PostgreSQL, objects, protected configuration, and recoverable
+key custody at one recovery point. It contains references, never key material.
+
+## Production Sequence
+
+1. Establish the provider snapshot, application quiesce, or transaction
+   boundary and retain a hash of its fencing token.
+2. Capture PostgreSQL, object storage, protected deployment configuration, and
+   key-custody state within five minutes of that recovery point.
+3. Restore all four components into a target isolated from production write
+   endpoints and production queues.
+4. Start the exact immutable release named in the evidence, verify migration
+   heads, verify a deterministic manifest of representative object hashes, and
+   execute the documented semantic journey checks.
+5. Record actual data loss and elapsed recovery as measured RPO and RTO. A
+   measured RPO above the declared objective invalidates the evidence.
+6. Sign the canonical receipt using an evidence-authority Ed25519 key held
+   outside the application and deployment host. During key rotation, include
+   both accepted signatures.
+7. Transfer the evidence SHA-256 through an independent approved channel, then
+   verify and adopt it on the deployment host.
+
+Provider automation can sign and validate an unsigned receipt with:
+
+```sh
+python tools/deployment/sign-backup-evidence.py \
+  --input unsigned-backup-evidence.json \
+  --output backup-evidence.json \
+  --trusted-keyring backup-evidence-keyring.json \
+  --signing-key backup-authority-2026=/run/keys/backup-authority.pem
+```
+
+The private key file must be owner-only. The tool refuses an unexpected key
+type, an inactive/untrusted signer, malformed or partial evidence, stale
+recovery points, failed drill checks, mismatched releases, and non-canonical
+output.
+
+Adopt the result using the independently obtained digest:
+
+```sh
+python3 govoplan-deploy.pyz verify-backup \
+  --directory /srv/govoplan/default \
+  --evidence ./backup-evidence.json \
+  --evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \
+  --trusted-keyring ./backup-evidence-keyring.json \
+  --adopt
+```
+
+Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every
+self-hosted release identity change is conservatively treated as a migration
+boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when
+fresh evidence for the previously applied immutable release is unavailable.
+Compose verifies once before changing runtime state and again after API/worker
+quiescing immediately before migration. The exported Kubernetes migration Job
+is generated only after verification and is annotated with the sanitized
+evidence digest, recovery-point id, and drill id.
+
+## Provider Runbooks
+
+### PostgreSQL
+
+Use a managed transaction-consistent snapshot or a base backup plus retained
+WAL sufficient to reconstruct the declared point. Record the provider,
+protected artifact reference and digest, snapshot identity, and PostgreSQL LSN.
+The restore drill must connect only to the isolated database and must compare
+the resulting migration-head digest with the release expectation.
+
+### Object Storage
+
+Use provider snapshots/versioning or an immutable object copy. Build a sorted
+manifest containing object key, version, size, and content digest, then record
+its digest, object count, total bytes, provider version identity, and protected
+artifact reference. Verify representative objects from every owning module
+after restore. Single-node managed Garage is persistent but not highly
+available; copy its coordinated recovery material to an independent failure
+domain.
+
+### Configuration And Key Custody
+
+Back up the private installation bundle and external secret-manager bindings as
+an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault
+state, record the provider keyset reference, version, and a successful
+recoverability assertion. Never put a key, recovery share, token, password, or
+credential-bearing URL in evidence. The isolated drill must prove that the
+restored release can decrypt representative protected content without
+exporting the key material into the report.
+
+## Ownership And Retention
+
+The deployment owner approves the RPO/RTO objectives. State-service owners
+operate backup capture and restoration. Module owners define representative
+objects and semantic checks. Security owns evidence-authority keys and
+revocation. Operations schedules drills and retains sanitized status.
+
+Retain backup artifacts for the approved legal/operational period and at least
+through the release's rollback window. Retain signed evidence, drill reports,
+and deletion receipts for the audit period. Disposal must remove every backup
+copy and provider version according to policy, then revoke or retire references
+without deleting the audit receipt. Cryptographic erasure is valid only when
+key-destruction evidence and provider-copy coverage are independently proven.
+
+## Failure Handling
+
+Missing components, component-time skew, stale or expired evidence, revocation,
+signature/key mismatch, changed stored files, release mismatch, failed semantic
+checks, or an RPO breach block migration. The deployment journal records the
+rejection without private provider details. If migration has not started, the
+operator may supply fresh evidence and retry. Once migration starts, recovery
+is explicitly forward-only until the verified coordinated recovery point is
+restored with its matching release.

docs/operations/DEPLOYMENT_PROFILES.md

@@ -0,0 +1,120 @@
+# GovOPlaN Deployment Profiles
+
+## Purpose
+
+GovOPlaN distinguishes how code is executed, where it is placed, and how mature
+the target is. These are separate concerns:
+
+- **execution basis:** editable source trees or an immutable signed release;
+- **topology:** local processes, one-host containers, or a multi-host
+  orchestrator;
+- **component ownership:** installer-managed or externally supplied state and
+  infrastructure services; and
+- **assurance state:** development, rehearsal/acceptance, or approved
+  production.
+
+PostgreSQL, Redis, object storage, mail and ingress choices are component
+bindings inside a profile. They do not create a new application topology by
+themselves.
+
+## Canonical Profiles
+
+| Profile | Entry point | Application execution | State services | Intended use | Explicit boundary |
+| --- | --- | --- | --- | --- | --- |
+| Local source development | `tools/launch/launch-dev.sh` | Editable Uvicorn/Vite processes with reload | Local development bindings, optionally the shared PostgreSQL helper | Fast module and UI work | No production packaging, isolation, availability or capacity claim |
+| Split source integration | `tools/launch/launch-production-like-dev.sh` | Editable API, WebUI, worker and scheduler processes | Containerized PostgreSQL/Redis by default; environment bindings may point at developer-owned services | Queue, migration, Redis and split-role integration while retaining source reload | “Production-like” describes behavior, not immutable artifacts or a production security boundary |
+| Immutable single-host rehearsal | `govoplan-deploy init/apply --profile evaluation` | Signed API/WebUI images and generated Compose roles | Bounded managed components or explicit external bindings | Test the downloadable artifacts, installer, migrations, load balancer and component choices | All containers and managed services may share one host and failure domain; evaluation conveniences are not production controls |
+| Single-host production | `govoplan-deploy init/apply --profile self-hosted` | Signed API/WebUI images behind generated HAProxy and selected TLS ingress | Durable local/single-node managed services where accepted, or external services | Small and medium installations whose accepted availability boundary is one host | Multiple containers add capacity and rolling-process resilience, but do not survive host loss |
+| Multi-host Kubernetes production | `govoplan-deploy render-kubernetes` or the guarded K3s lab/acceptance workflow | Immutable API, WebUI and queue-specific worker Deployments across failure domains | External PostgreSQL, Redis and S3-compatible storage; external secret and ingress control | Institution-scale availability and horizontal application-tier capacity | Production claims require independent nodes, HA state services, load/capacity evidence and signed recovery evidence |
+
+Docker Compose services are containers or replicas, not Kubernetes pods. The
+immutable single-host rehearsal is the appropriate Dockerized whole-product
+test when source reload is not required.
+
+The K3s VM lab has two modes over the same Kubernetes profile:
+
+- `rehearsal` may place VMs on one physical hypervisor and proves bounded
+  orchestration behavior;
+- `acceptance` requires independently controlled worker failure domains and can
+  contribute target evidence.
+
+## Module composition and availability
+
+Official immutable API and WebUI images carry the verified `full` package
+profile. This is package availability, not runtime activation and not a license
+or tenant entitlement. The signed distribution manifest records the complete
+package composition; the desired module graph selects which installed modules
+are active; tenant module policy applies unavailable/available/forced ceilings;
+and Views/Policy control group and user presentation.
+
+Local and single-host profiles may use the supervised installer to download a
+signed catalog artifact into a private digest cache and mutate the local package
+environment during maintenance. A multi-host/shared-state profile must never
+change one replica in place. Its Admin install plan is a composition request:
+publish and roll out a new signed image whose package lock contains the target,
+then activate the module graph after all replicas report the same composition.
+
+## Component Choices
+
+The installer may manage a component where its bounded profile is appropriate,
+or consume an operator-provided service:
+
+| Component | Managed boundary | External/BYO boundary |
+| --- | --- | --- |
+| PostgreSQL | Single-host Compose database | Stable primary-aware endpoint supplied by a PostgreSQL provider/operator |
+| Redis | Single-host persistent Redis | Tested HA Redis endpoint compatible with queues, throttling and coordination |
+| File/object storage | Durable local storage or single-node Garage | Shared, redundant S3-compatible storage |
+| Mail | Development GreenMail only | Institution/provider SMTP and IMAP services |
+| Ingress/TLS | Generated Caddy on one host | Existing reverse proxy or Kubernetes ingress and secret management |
+
+Switching to an external component changes ownership and evidence requirements;
+it does not remove GovOPlaN's health, capacity, backup and recovery checks.
+
+## Scaling Responsibilities
+
+GovOPlaN scales application roles, while the infrastructure control plane owns
+machines and state-service replication:
+
+| Concern | Scaling model | Owner |
+| --- | --- | --- |
+| API and WebUI | Increase replicas behind health-aware Services/Ingress | GovOPlaN deployment desired state, reconciled by Compose or Kubernetes |
+| Background work | Add queue-specific worker replicas and bounded concurrency | GovOPlaN deployment desired state and worker-pool configuration |
+| Scheduler, migrations and module lifecycle | Singleton execution protected by database leases/fencing | GovOPlaN; these roles are never scaled as unfenced active-active workers |
+| Kubernetes worker/control nodes | Add, drain, replace and upgrade machines; optionally use a cluster autoscaler | Kubernetes/platform operator, not the GovOPlaN application |
+| PostgreSQL | Replication, failover, backups, connection pooling and stable writer endpoint | Database operator/provider; GovOPlaN currently consumes the stable endpoint and does not route arbitrary reads to replicas |
+| Redis | Replication/failover, persistence, eviction and TLS/authentication | Redis operator/provider |
+| S3-compatible storage | Placement, replication, repair and capacity | Storage operator/provider |
+
+Administrators should eventually be able to review and change permitted
+application replica and worker-pool desired state through the Ops surface.
+Creating physical machines, database replicas or storage members remains an
+orchestrator/provider action. GovOPlaN must observe their health and block unsafe
+changes rather than becoming a second infrastructure scheduler.
+
+Every scale change must recalculate the database connection budget, preserve
+queue coverage, verify software/module-composition consistency and respect
+drain and fencing state.
+
+## What Has Been Proven
+
+The current implementation and the signed `v0.1.18` rehearsal prove that the
+application tier can run as stateless API, WebUI and worker replicas against
+logically shared state. Two Kubernetes worker VMs hosted API and WebUI replicas,
+and an API pod was replaced without an observed public-readiness failure.
+
+This is not yet proof of general “large organization fit.” That claim also
+requires:
+
+- representative concurrent-user, dataset, report and background-job load
+  tests with latency and saturation budgets;
+- independent physical failure domains and real ingress/network behavior;
+- HA PostgreSQL, Redis and object storage with failover drills;
+- session, accepted-job and provider-effect continuity under node and service
+  loss;
+- coordinated backup/isolated restore, measured RTO/RPO and semantic recovery;
+- observability, alerting, capacity forecasting and sustained soak evidence.
+
+The profile therefore proves the architecture is horizontally deployable. A
+specific institution is production-fit only after its target topology and load
+envelope have produced the governed evidence described in
+`TARGET_MATURITY_EVIDENCE_RUNBOOK.md`.

docs/operations/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

@@ -0,0 +1,584 @@
+# Installation And Deployment Architecture
+
+## Goal
+
+A supported GovOPlaN installation starts with one downloaded, verified
+bootstrap artifact. The administrator answers a bounded set of questions and
+receives a working base system. Re-running the same tool repairs or
+reconfigures that installation instead of creating unrelated state.
+
+The canonical product journey remains
+[System Administrator Lifecycle User Story](../strategy/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md).
+This document defines the deployer boundary and the first executable slice.
+The execution, topology, component-ownership and assurance modes are defined
+canonically in [Deployment Profiles](DEPLOYMENT_PROFILES.md). In particular,
+the editable production-like developer launcher is distinct from both an
+immutable Compose rehearsal and a supported one-host production deployment.
+
+## First Executable Slice
+
+`tools/deployment/govoplan-deploy.py` is a standard-library-only deployment
+compiler and reconciler. It can be tested without installing GovOPlaN itself.
+
+It currently supports:
+
+- evaluation and self-hosted profiles;
+- managed or external PostgreSQL;
+- managed, external, or evaluation-only disabled Redis;
+- disabled mail, an external relay declaration, or an evaluation-only
+  GreenMail service;
+- durable local file storage, managed single-node Garage S3, or external
+  S3-compatible storage;
+- an explicit HAProxy service that load-balances configured WebUI and API
+  replicas without access to the Docker socket;
+- declarative API, WebUI, and worker replica counts while keeping migrations
+  and the scheduler singleton;
+- Core, base, or full initial module selections;
+- deterministic Compose JSON accepted by Compose v2;
+- generated secrets stored in a private `0600` file;
+- service-specific environment allowlists so infrastructure containers do not
+  receive unrelated application credentials;
+- plan, render, doctor, status, apply, Kubernetes export, operation history,
+  and bounded recovery commands;
+- an installation lock, migration-before-start ordering, readiness polling,
+  and an applied-state receipt;
+- a durable hash-chained deployment journal captured before runtime mutation;
+- PostgreSQL advisory serialization for Core and module migrations;
+- runtime initialization that waits for exact configured migration heads
+  without mutating schema;
+- runtime node registration, heartbeats, drain state, and a fenced scheduler;
+- idempotent reconfiguration that preserves generated secrets;
+- a keyed environment fingerprint that detects private binding changes without
+  writing secret values to plans or receipts;
+- host CPU, memory, disk, entropy, architecture, Docker daemon, Compose,
+  listen-port, and external endpoint preflight checks;
+- service removal without implicit data-volume deletion.
+
+Create a local evaluation bundle:
+
+```sh
+./.venv/bin/python tools/deployment/govoplan-deploy.py init \
+  --directory /tmp/govoplan-evaluation \
+  --profile evaluation \
+  --postgres managed \
+  --redis managed \
+  --storage garage \
+  --mail test-mail \
+  --api-replicas 2 \
+  --web-replicas 2 \
+  --worker-replicas 2 \
+  --module-set base
+```
+
+Inspect the generated intent and host requirements:
+
+```sh
+./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
+  --directory /tmp/govoplan-evaluation
+```
+
+Change a component without rotating existing generated secrets:
+
+```sh
+./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
+  --directory /tmp/govoplan-evaluation \
+  --redis external \
+  --redis-url 'rediss://:password@redis.example.org:6379/0'
+```
+
+The private installation directory contains:
+
+| File | Purpose |
+| --- | --- |
+| `installation.json` | Versioned, non-secret desired state |
+| `secrets.env` | Deployment-local secrets and external service bindings |
+| `compose.json` | Deterministic generated Compose definition |
+| `garage.toml` | Non-secret managed Garage server configuration |
+| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
+| `Caddyfile` | Non-secret managed-ingress route and ACME policy |
+| `existing-proxy.json` | Exact upstream, trusted-source, header, and health contract for an operator-owned proxy |
+| `plan.json` | Latest desired-state diff and readiness findings |
+| `receipt.json` | Last successfully applied immutable identities |
+| `infrastructure-capabilities.json` | Deterministic non-secret capability states, endpoint metadata, secret references, consumers, and resumable post-install tasks |
+| `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
+| `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
+| `backup-evidence.json` | Signed provider-neutral coordinated backup and isolated-restore receipt |
+| `backup-keyring.json` | Explicit public trust anchor for backup evidence authorities |
+| `backup-verification.json` | Sanitized local verification/adoption receipt |
+| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
+| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
+| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
+| `.deployment.lock` | Same-host operation exclusion |
+
+The specification contract is
+[`installation-spec.schema.json`](../installation-spec.schema.json).
+
+The API, workers, scheduler, and Ops read the capability receipt through the
+same bounded Core validator. Configuration-package providers receive that typed
+receipt in preflight context. Mail uses `mail.smtp` to offer an idempotent SMTP
+profile plan and accepts only an existing credential-envelope reference; Files
+uses `files.storage` to prove that the deployment-owned local/S3 runtime binding
+already matches. Files deliberately blocks drift instead of rewriting process
+environment or initiating an implicit object migration. Invalid receipts fail
+closed, while a deployment without a mounted receipt continues to run but
+cannot apply receipt-bound configuration fragments.
+
+Build the same dependency-free tool as one downloadable artifact:
+
+```sh
+./.venv/bin/python tools/deployment/build-deployer-zipapp.py \
+  --output /tmp/govoplan-deploy.pyz
+python /tmp/govoplan-deploy.pyz --help
+```
+
+To exercise reconciliation with locally available evaluation images:
+
+```sh
+python /tmp/govoplan-deploy.pyz init \
+  --non-interactive \
+  --directory /tmp/govoplan-evaluation \
+  --profile evaluation \
+  --api-image local/govoplan-api:test \
+  --web-image local/govoplan-web:test
+python /tmp/govoplan-deploy.pyz apply \
+  --directory /tmp/govoplan-evaluation \
+  --allow-unverified-images \
+  --skip-pull
+```
+
+Those images must already contain the selected module set. The override exists
+only to exercise local orchestration before release artifacts exist; it is
+rejected for `self-hosted`.
+
+## Runtime Distribution Boundary
+
+The protected `Runtime Distribution` workflow builds GovOPlaN wheels first,
+resolves architecture-specific third-party wheels into offline wheelhouses, and
+then assembles the API images with `pip --no-index`. The target host never
+clones Git repositories and neither runtime image performs network package
+installation. Separate amd64/arm64 API and WebUI images are joined into OCI
+indexes and run as non-root identities. The release assets include CycloneDX
+application SBOMs, SLSA-style provenance, exact composition evidence, the
+single-file deployer, its detached Ed25519 signature, and a signed, expiring
+distribution manifest. Evidence generation and signing run through the
+workflow's isolated release Python environment so their cryptographic tooling
+is explicit and independent of packages preinstalled in the Actions runner.
+The API image points Core at the migration scripts installed from the verified
+wheel under `/opt/govoplan/runtime/govoplan_core_runtime`; migrations therefore
+do not depend on a source checkout or the build host's Python installation
+scheme.
+Before publication, the exact amd64 and arm64 image manifests each run release
+migrations against the pinned PostgreSQL image, reach API and WebUI readiness
+as non-root/read-only processes, and complete a task through the pinned Redis
+image and packaged worker. Sanitized per-platform smoke receipts are retained
+as immutable release assets.
+PostgreSQL and Redis indexes are resolved to untagged platform-child digests
+before each smoke run. This keeps the evidence architecture-specific and
+avoids retargeting one local Docker tag between incompatible platforms.
+The CI host registers arm64 execution with an explicitly supplied,
+digest-pinned `tonistiigi/binfmt` image immediately before the smoke. This
+privileged helper is confined to the release runner and is never part of a
+GovOPlaN target deployment or its runtime image set.
+Because QEMU user-mode execution triggers Redis's arm64 host-kernel COW guard,
+the arm64 smoke suppresses only `ARM64-COW-BUG` while persistence, snapshots,
+and append-only files are disabled. Target Redis services never inherit this
+test-only option.
+The smoke also proves a bounded post-migration table contract and aborts as
+soon as a required container exits, rather than allowing a dead process to
+consume the full readiness timeout.
+
+Ingress acceptance streams generated configuration into Docker-managed
+volumes before starting the read-only containers. It therefore also works when
+an Actions job reaches a host or remote Docker daemon through a mounted socket;
+the drill never assumes that a job-container path is visible to that daemon.
+The drill allocates explicit loopback-only host ports and verifies Docker's
+host binding configuration, avoiding daemon-specific random-port shorthand
+behavior. Because an Actions job and deployment containers may be Docker
+siblings, functional HTTP/TLS checks run from the digest-pinned API image on
+the deployment network instead of assuming the Docker host is job-local.
+The dispatch-only `Runtime Ingress Drill` workflow exposes the same bounded
+check independently so ingress changes can be diagnosed before an immutable
+runtime publication; it accepts only digest-pinned Caddy, HAProxy, and API
+images and has no push trigger.
+The official Caddy binary carries the `NET_BIND_SERVICE` file capability. The
+managed-ingress container therefore drops every capability and adds back only
+`NET_BIND_SERVICE`; otherwise Linux rejects the binary at `execve` before its
+high-port configuration can start. `no-new-privileges`, a read-only root
+filesystem, and non-privileged container ports remain enforced.
+The bounded setup helper writes only generated public configuration as root so
+it can initialize a new volume; the actual HAProxy process retains the image's
+non-root identity and runs read-only with all capabilities dropped.
+
+The manifest contract is
+[`runtime-distribution-manifest.schema.json`](../runtime-distribution-manifest.schema.json),
+and its separately distributed trust-anchor contract is
+[`runtime-distribution-keyring.schema.json`](../runtime-distribution-keyring.schema.json).
+Publication is immutable: an existing Gitea release asset must have the same
+size and SHA-256 digest or publication fails.
+
+Adopt a downloaded or prefetched release only after obtaining the manifest
+digest and trusted keyring through the documented independent channel:
+
+```sh
+python3 govoplan-deploy.pyz verify-release \
+  --directory /srv/govoplan/installation \
+  --manifest ./distribution-manifest.json \
+  --manifest-sha256 "$(cut -d' ' -f1 distribution-manifest.json.sha256)" \
+  --trusted-keyring ./distribution-keyring.json \
+  --adopt
+```
+
+`doctor` and `apply` rehash both stored files, re-run OpenSSL Ed25519
+verification, enforce channel/expiry/revocation, compare every selected image,
+and prove that all enabled module ids occur in the signed image composition.
+An offline image index can bind prefetched OCI archives to the same exact image
+references and archive hashes; mutable tags or incomplete bundles are rejected.
+
+## Current Production Gates
+
+The first immutable production-distribution baseline is published as
+[`v0.1.14`](https://git.add-ideas.de/GovOPlaN/govoplan/releases/tag/v0.1.14)
+from source commit `1f039dd39c1ce2672f4978c8abc6dff862ef1445`. Runtime
+Distribution [run #459](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/459)
+proved migrations, schema compatibility, non-root API/Web readiness, and worker
+delivery/shutdown on both `linux/amd64` and `linux/arm64`. Its signed manifest
+has SHA-256
+`d703267e01855dee63200cb20921c91c3f95fbff550c8ca76e9a35cba3f69109`
+and pins these runtime indexes:
+
+- API: `git.add-ideas.de/govoplan/runtime-api@sha256:197ed01790986f2bc927eaa5d8348fa118702e5d2dc05feb851fc2643c23764a`
+- WebUI: `git.add-ideas.de/govoplan/runtime-web@sha256:e936cca124f1fad29a067834cf17627d4c236410fdc3fa129e0ccb26b8193812`
+
+The signed bootstrap has SHA-256
+`1ff946fba82b0895d153b23352d06e30fe18388450dfd37fed6fb9912310efc5`
+and key id `runtime-distribution-2026-01`. The managed-ingress boundary passed
+the same publication run and the independently dispatchable Runtime Ingress
+Drill [run #458](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/458).
+Every later release must renew this evidence; the following target-specific
+gates remain:
+
+1. **First administrator.** Production needs a one-time, restricted enrollment
+   identity. The development bootstrap must not be enabled in production.
+2. **Image/module composition.** The deployer enforces the signed
+   composition. A selected module not shipped by that release cannot be
+   enabled.
+3. **Deployment agent.** Web updates need a separate privileged reconciler with
+   a typed command allowlist. The API and browser must never receive the Docker
+   socket or arbitrary shell access.
+4. **Target reachability evidence.** Managed Caddy ingress and the
+   existing-proxy contract are implemented. A production claim still requires
+   running `doctor` from the target host after public DNS/firewall changes and
+   retaining public TLS/readiness evidence for that deployment.
+
+`apply --allow-unverified-images` is therefore restricted to the evaluation
+profile. It explicitly acknowledges both mutable image identities and
+unverified image/module composition. It is a local test escape hatch, not a
+production setting.
+
+## Component Choices
+
+### PostgreSQL
+
+`managed` creates a persistent PostgreSQL container and private generated
+credentials. `external` requires an explicit `DATABASE_URL`; switching from
+managed to external cannot reuse the old `postgres` Docker hostname
+accidentally.
+
+Interactive entry hides external URLs because they commonly contain
+credentials. For unattended automation, provide them through a protected
+operator mechanism and avoid storing secret-bearing flags in shell history.
+
+Production policy should support external managed databases and local managed
+PostgreSQL equally at the application boundary. Backup, point-in-time recovery,
+high availability, and major-version upgrades remain deployment properties.
+
+### Redis
+
+`managed` creates an authenticated, append-only Redis container. `external`
+requires an explicit `REDIS_URL`. `disabled` is evaluation-only and disables
+workers while recording the single-process login-throttle risk acknowledgement.
+
+`doctor` performs a bounded TCP connection check for external PostgreSQL,
+Redis, and S3 endpoints. This verifies DNS, routing, and that the port accepts a
+connection; it is not an authentication or semantic health check.
+
+Production base installations include Redis because durable queues, distributed
+throttling, notifications, scheduled work, and transactional event delivery
+must survive API restarts.
+
+### Mail
+
+The first slice distinguishes:
+
+- `disabled`;
+- `external-relay`, which records the infrastructure decision but leaves Mail
+  server/credential creation as a visible post-install task;
+- `test-mail`, an evaluation-only GreenMail service.
+
+A bundled production mail server is intentionally not a default. Operating one
+requires DNS, reverse DNS, TLS, DKIM, SPF, DMARC, reputation, abuse handling,
+queue monitoring, and upgrade policy. A later profile may support an
+operator-selected MTA/relay, but it must expose these requirements rather than
+presenting a container as a complete mail service.
+
+### File Storage
+
+`local` uses a durable Compose volume and is appropriate for one-host
+installations. `garage` provisions Garage 2.3 in its supported single-node
+bootstrap mode, generates a private application key and bucket, and connects
+the Files S3 backend to the exact installer-owned internal endpoint. The
+managed trust marker cannot authorize another S3 host.
+Garage metadata and object data use separate persistent volumes. `s3` requires
+an external endpoint, region, access key, secret key, and bucket values.
+Self-hosted external S3 endpoints must be clean HTTPS origins. The generated
+runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that
+operator-selected endpoint. The trust flag is not accepted for local storage
+and cannot be combined with installer-managed Garage trust.
+
+Local storage must be included in backup and restore drills. Horizontal API or
+worker scale-out requires shared/object storage. The managed Garage profile is
+persistent but has no data redundancy; availability-sensitive installations
+must use a tested multi-node Garage cluster or another external S3 service.
+
+### Load Balancing And Replicas
+
+The generated Compose topology publishes only `load-balancer` for local or
+existing-proxy profiles. With managed ingress, only Caddy publishes host ports
+and HAProxy remains private. HAProxy uses
+Docker DNS service discovery to distribute public traffic across WebUI replicas
+and WebUI API proxy traffic across API replicas. The WebUI and API services do
+not publish host ports. HAProxy has no Docker socket and discovers only the
+bounded replica slots rendered into `load-balancer.cfg`.
+
+Replica counts are desired state:
+
+```sh
+./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
+  --directory /tmp/govoplan-evaluation \
+  --api-replicas 3 \
+  --web-replicas 2 \
+  --worker-replicas 4
+./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
+  --directory /tmp/govoplan-evaluation
+```
+
+Workers are queue consumers, so they are scaled through Redis rather than put
+behind an HTTP load balancer. Migrations are serialized with a deployment-wide
+PostgreSQL advisory lock. The Celery scheduler is run under a renewable,
+fencing-token lease. Multiple API replicas are rejected when Redis is disabled
+because distributed throttling and queued work cannot then be shared correctly.
+
+### Public Ingress And TLS
+
+A self-hosted installation is fail-closed until one of these boundaries is
+selected:
+
+- `existing-proxy` publishes HAProxy at `listen.address:listen.port` and emits
+  `existing-proxy.json`. The operator-owned proxy must use the recorded host,
+  upstream, and health paths. Only the exact CIDRs listed with repeated
+  `--trusted-proxy-cidr` values may supply `X-Forwarded-*` headers. Public
+  proxy addresses must be `/32` or `/128`; private ranges are limited to `/24`
+  or narrower for IPv4 and `/64` or narrower for IPv6.
+- `managed` publishes Caddy on the selected HTTP/HTTPS ports, redirects HTTP to
+  HTTPS, obtains and renews certificates through ACME, and keeps certificate
+  material exclusively in the private `caddy-data` and `caddy-config` volumes.
+  The application containers receive no ACME account or TLS private keys.
+
+Example existing-proxy configuration:
+
+```sh
+python govoplan-deploy.py configure \
+  --directory /srv/govoplan \
+  --ingress existing-proxy \
+  --trusted-proxy-cidr 172.20.0.7/32
+```
+
+Example managed configuration:
+
+```sh
+python govoplan-deploy.py configure \
+  --directory /srv/govoplan \
+  --ingress managed \
+  --acme-email operator@example.org
+```
+
+Before managed ingress starts, public A/AAAA records must resolve to the target
+and inbound TCP 80/443 must reach it. Existing-proxy mode additionally requires
+the public proxy and valid certificate to be reachable before apply. After a
+successful receipt, `doctor` reports DNS resolution, certificate validity and
+remaining lifetime, public `/health/ready`, and the private HAProxy/WebUI path
+as separate checks. Reconfiguration retains the certificate volumes; bundle
+rollback never deletes or exposes their contents. Include both Caddy volumes
+in coordinated backup and restore evidence.
+
+This is same-host scaling. Docker Compose uses a bridge network and does not
+place containers on another machine. See
+[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) for
+the supported topology and promotion path.
+
+## Reconfiguration Semantics
+
+`installation.json` is desired state. `receipt.json` is the last successfully
+applied state. `plan` compares their canonical hashes, service sets, and
+infrastructure capability projections.
+
+- Adding a managed component creates its service and persistent volume.
+- Removing a component removes its service container on apply.
+- Replacing or removing a capability adds a review action that names the prior
+  and desired state/source plus declared module consumers. This does not claim
+  that the deployer can inspect module-owned database configuration; the
+  operator must review that inventory before apply.
+- Volumes are retained by default; deleting data requires a separate,
+  deliberately destructive workflow.
+- Existing generated credentials are retained unless an explicit future rotate
+  operation is requested.
+- Private configuration changes are represented by a keyed fingerprint in the
+  plan and receipt; plaintext values are never copied there.
+- Capability documents contain sanitized scheme/host/port metadata and stable
+  `env:` references only. Credential values and secret-bearing URLs remain in
+  `secrets.env` or module-owned credential envelopes.
+- Managed-to-external transitions require the new endpoint in the same
+  operation.
+- Migrations run as a one-shot service before API/worker replacement.
+- API, worker, and scheduler start commands wait for exact configured migration
+  heads; only the migration command is permitted to change schema.
+- API and worker replicas register their software/module composition and
+  heartbeat in PostgreSQL. Ops can request and cancel a node drain.
+- The first upgrade from a direct WebUI host port stops that legacy WebUI
+  container immediately before HAProxy claims the same endpoint.
+- Health must recover before a new receipt and applied-state snapshot are
+  committed.
+
+Compose mounts the capability document read-only into API and worker runtime
+containers. The Kubernetes export projects the same document through a
+dedicated ConfigMap and read-only file mount. Ops validates the bounded schema
+before displaying configured, externally supplied, available-unconfigured, or
+unavailable states and any pending post-install tasks.
+
+Every apply operation is journalled before image pulls or runtime mutation. A
+failure before migration may restore a verified previous bundle. Once migration
+starts, recovery is forward-only unless an independently verified database
+backup is restored. See
+[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
+
+Production updates still need operator/provider-created coordinated backup and
+restore evidence, a database compatibility declaration, and a
+deployment-specific drain policy. The deployer now verifies and enforces the
+signed evidence before migration, but does not manufacture backups or receive
+provider administration credentials. See
+[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md).
+
+## Stateless Kubernetes Runtime
+
+`render-kubernetes` exports the application tier for a standard orchestrator.
+It requires external PostgreSQL, Redis, and S3 and emits no stateful service or
+secret value:
+
+```sh
+python tools/deployment/govoplan-deploy.py render-kubernetes \
+  --directory /srv/govoplan/default \
+  --namespace govoplan \
+  --secret-name govoplan-runtime
+```
+
+The output includes a release-specific migration Job, database-head wait init
+containers, API readiness/liveness probes, rolling Deployments, Services, Pod
+disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
+the named Secret through the cluster's secret manager and review ingress proxy
+CIDRs before deployment. A release-changing export requires adopted backup
+evidence and carries only its sanitized digest and identifiers as Job
+annotations. Detailed rollout and scaling rules live in
+[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
+
+## Recovery Commands
+
+List durable deployment operations:
+
+```sh
+python tools/deployment/govoplan-deploy.py operations \
+  --directory /srv/govoplan/default
+```
+
+Recover a selected failed operation after reviewing its stage evidence:
+
+```sh
+python tools/deployment/govoplan-deploy.py recover \
+  --directory /srv/govoplan/default \
+  --operation-id <operation-id>
+```
+
+The command reports whether it restored the pre-migration applied bundle,
+requires forward recovery, or needs manual intervention. Add `--apply` only
+after that decision has been reviewed.
+
+## Web Update Boundary
+
+The intended update path is:
+
+1. Ops reads the non-secret installation receipt and reports management mode,
+   current release, component health, and update availability.
+2. An authorized administrator asks Core to create a typed deployment request,
+   for example `reconcile_release` or `rollback_release`.
+3. Core persists the reviewed immutable plan, actor, expected current receipt,
+   and idempotency key.
+4. A separately deployed, narrow deployment agent claims the request.
+5. The agent verifies signatures/digests, acquires a fenced deployment lock,
+   backs up, pulls, migrates, reconciles, probes health, and writes evidence.
+6. Ops presents durable progress and the resulting receipt.
+
+The agent owns container-runtime access. It accepts no command strings from the
+browser and has no domain-data permissions. Installations managed by Kubernetes,
+systemd, or another external orchestrator expose read-only status and an export
+of the reviewed update recipe instead of a non-functional update button.
+
+## Distribution Workflow
+
+The downloadable entry point is a reproducible release asset: sorted source
+paths, fixed ZIP metadata, fixed compression settings, and identical source
+bytes produce an identical zipapp regardless of checkout timestamps. Obtain the
+zipapp, detached signature, checksum, and trusted public keyring through
+independently authenticated paths before execution:
+
+```sh
+curl --proto '=https' --tlsv1.2 --fail --location \
+  https://git.add-ideas.de/GovOPlaN/govoplan/releases/download/vX.Y.Z/govoplan-deploy.pyz \
+  --output govoplan-deploy.pyz
+sha256sum --check govoplan-deploy.pyz.sha256
+python3 - <<'PY'
+import json
+from pathlib import Path
+
+keyring = json.loads(Path("distribution-keyring.json").read_text())
+active = [key for key in keyring["keys"] if key["status"] == "active"]
+if len(active) != 1:
+    raise SystemExit("expected exactly one active runtime release key")
+Path("runtime-release-public.pem").write_text(active[0]["public_key_pem"])
+PY
+openssl pkeyutl -verify -pubin -inkey runtime-release-public.pem -rawin \
+  -in govoplan-deploy.pyz -sigfile govoplan-deploy.pyz.sig
+python3 govoplan-deploy.pyz init
+```
+
+The zipapp has no GovOPlaN package dependency. It accepts a bounded HTTPS
+manifest or a prefetched file, requires an independently supplied SHA-256
+digest and explicit trusted keyring, and executes OpenSSL with a fixed argument
+vector for Ed25519 verification. It never evaluates downloaded shell text or
+accepts an arbitrary command string.
+
+## Verification
+
+Run the focused tests:
+
+```sh
+./.venv/bin/python -m unittest -v tests.test_deployment_installer
+```
+
+The tests cover signed release adoption, tamper/expiry/revocation/unknown-key
+rejection, architecture composition, offline image integrity, profile
+restrictions, secret persistence, external endpoint
+requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
+discovery configuration, Compose service selection, secret non-disclosure,
+service-specific environment isolation, private file modes, external endpoint
+preflight, first-plan generation, apply ordering, receipt idempotency,
+hash-chained recovery journals, migration recovery boundaries, and stateless
+Kubernetes rendering.

docs/operations/KUBERNETES_TEST_LAB.md

@@ -0,0 +1,343 @@
+# Kubernetes VM Test Lab
+
+`tools/lab/govoplan-lab.py` creates and operates an amd64 Ubuntu/K3s test
+environment on local or SSH-accessible libvirt hypervisors. It provides the
+commands requested for the complete VM lifecycle:
+
+| Command | Effect |
+| --- | --- |
+| `doctor` | Validate the strict inventory and, with `--online`, every hypervisor. |
+| `create --apply` | Download checksum-pinned cloud images, create VM overlays and boot the declared VMs. |
+| `deploy --apply` | Verify the signed GovOPlaN release, deploy shared state, install pinned K3s and apply GovOPlaN. |
+| `update --apply` | Pull newly pinned state images, update K3s serially and roll the selected GovOPlaN release. |
+| `status` | Show libvirt VM state, Kubernetes nodes and GovOPlaN pods. |
+| `pause --apply` | Gracefully shut down workers, control planes and shared state while retaining disks. |
+| `resume --apply` | Start the retained environment in dependency order and wait for readiness. |
+| `verify` | Collect sanitized live-cluster evidence and optionally perform the API-pod-loss drill. |
+| `destroy --apply --confirm <lab>` | Delete only the lab-owned domains and overlays; local evidence is retained by default. |
+
+Every mutating command is a dry run unless `--apply` is present. Destruction
+also requires the exact lab name. Generated credentials, CA keys, manifests and
+evidence are written below the configured `state_directory` with owner-only
+permissions. Keep that directory outside the repository and include it in the
+workstation backup policy. Existing domains are reused or removed only when
+their GovOPlaN ownership description and both expected lab disk paths match.
+
+## What The Lab Proves
+
+The supplied inventories describe two different assurance levels:
+
+- `tools/lab/govoplan-lab.example.toml` creates four VMs on one libvirt host.
+  It is suitable for development, deployment rehearsal, migration testing,
+  application-pod replacement and recovery-tool exercises. It cannot close
+  GovOPlaN #27 because one physical host remains one failure domain.
+- `tools/lab/govoplan-lab.acceptance.example.toml` places the two workers on
+  different hypervisors and puts the control and state VMs on a third. It can
+  produce the bounded stateless application-tier evidence required by #27 when
+  the declared hypervisors are genuinely independent physical failure domains.
+
+Both examples use one control-plane VM and one state VM. This keeps the bounded
+#27 target economical, but it does not prove control-plane or state-service
+high availability. For control-plane failover, declare exactly three control
+nodes on independent hosts. PostgreSQL, Redis and object-storage failover must
+be tested against independently operated HA services; the lab's single state
+VM is intentionally a replaceable integration fixture.
+
+Approximate minimum capacity for the four-VM profile is 10 vCPUs, 16 GiB RAM
+and 192 GiB of thin-provisioned disk. A six-VM profile with three controls needs
+additional capacity. Do not overcommit memory on an acceptance target.
+
+## 1. Prepare The Hypervisors
+
+On each Ubuntu/Debian libvirt host:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y \
+  qemu-kvm libvirt-daemon-system libvirt-clients virtinst cloud-image-utils curl
+sudo systemctl enable --now libvirtd
+```
+
+Use a dedicated lab-administration account. Remote hypervisors are managed over
+SSH and the lifecycle invokes `sudo -n` there, so that account needs bounded
+non-interactive permission for libvirt, image and cloud-init operations.
+`NOPASSWD: ALL` is acceptable only on isolated lab hypervisors.
+
+On a local hypervisor, put the workstation account in the `libvirt` group and
+point `vm_image_directory` at a directory writable by that account and
+traversable by `libvirt-qemu`. The lifecycle connects explicitly to
+`qemu:///system` and does not require passwordless local sudo. Log out and back
+in after a new group assignment before running `doctor --online`. Create the
+configured image directory before running the doctor; it deliberately rejects
+a missing or non-writable storage root instead of silently falling back to a
+different filesystem.
+
+Create a dedicated SSH key on the management workstation:
+
+```bash
+ssh-keygen -t ed25519 -f "$HOME/.ssh/govoplan-lab" \
+  -C "GovOPlaN Kubernetes lab"
+```
+
+Install its public key for every remote hypervisor account. The same public key
+is injected into the VMs. The lifecycle keeps its own `ssh_known_hosts` file,
+uses `accept-new` for first contact, and rejects changed host keys until a
+lab-owned VM is deliberately recreated.
+
+### Network contract
+
+The configured `bridge` must exist on every selected hypervisor. All VM
+addresses are static. Reserve them outside DHCP allocation and ensure that the
+management workstation can route directly to every VM address; the lifecycle
+does not tunnel VM traffic through the hypervisor SSH connection.
+
+Permit only these flows inside the lab network:
+
+| Port | Source and destination | Purpose |
+| --- | --- | --- |
+| TCP 22 | management workstation to every VM/hypervisor | Provisioning and evidence collection |
+| TCP 6443 | all K3s nodes and management path to controls | Kubernetes API |
+| UDP 8472 | K3s node to K3s node | Default Flannel VXLAN; never expose publicly |
+| TCP 10250 | K3s node to K3s node | Kubelet metrics and API |
+| TCP 2379-2380 | control to control, only with three controls | Embedded etcd |
+| TCP 80/443 | test clients to K3s nodes | Traefik/ServiceLB ingress |
+| TCP 5432/6379/9443 | K3s nodes to the state VM | PostgreSQL, Redis and TLS-protected Garage S3 |
+| TCP 3025/3143 | approved test clients/workers to the state VM | GreenMail SMTP/IMAP test endpoints |
+
+The official
+[K3s networking requirements](https://docs.k3s.io/installation/requirements#networking)
+remain authoritative. Restrict state ports to the lab network even though the
+generated integration stack binds them on the state VM.
+
+## 2. Create The Inventory
+
+Start with the one-host rehearsal:
+
+```bash
+install -d -m 0700 "$HOME/.config/govoplan/labs"
+cp tools/lab/govoplan-lab.example.toml \
+  "$HOME/.config/govoplan/labs/development.toml"
+chmod 0600 "$HOME/.config/govoplan/labs/development.toml"
+```
+
+Edit at least the bridge, network, static addresses and SSH key paths. For a
+multi-host run, copy the acceptance example and replace every example hostname,
+failure-domain declaration and network value. Strict parsing rejects unknown
+keys, mutable HTTP inputs, malformed checksums, duplicate addresses/MACs and an
+acceptance inventory that collapses workers onto one declared hypervisor or
+failure domain.
+
+Cloud image, K3s binary, K3s installer and GovOPlaN release inputs are URL plus
+SHA-256 pairs. Updating means changing those reviewed pins and then running the
+`update` command; the tool deliberately does not follow `latest` aliases.
+
+The one-host example uses the dedicated `govoplan-lab` NAT network. Its DHCP
+pool ends at `192.168.123.99`; the static lab addresses start at
+`192.168.123.201`. Define and start it once on the local hypervisor:
+
+```bash
+virsh --connect qemu:///system net-define \
+  tools/lab/libvirt/govoplan-lab-network.xml
+virsh --connect qemu:///system net-autostart govoplan-lab
+virsh --connect qemu:///system net-start govoplan-lab
+```
+
+Re-running those commands is unnecessary when `virsh net-info govoplan-lab`
+already reports an active, persistent network. The lab destroy command leaves
+this reusable network in place.
+
+## 3. Validate And Create The VMs
+
+```bash
+LAB="$HOME/.config/govoplan/labs/development.toml"
+PYTHON="/mnt/DATA/git/govoplan/.venv/bin/python"
+
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" doctor --online
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" create --apply
+```
+
+The preview is safe to run repeatedly. Creation reuses a domain whose exact
+lab-owned name already exists and otherwise creates a thin qcow2 overlay under
+`vm_image_directory/<lab>/<node>`.
+
+## 4. Deploy GovOPlaN
+
+If `git.add-ideas.de` requires authentication for release images, export a
+read-only package/container-registry identity for this shell. A Gitea package
+token can be used as the password:
+
+```bash
+export GOVOPLAN_LAB_REGISTRY_USERNAME='package-reader'
+read -r -s GOVOPLAN_LAB_REGISTRY_PASSWORD
+export GOVOPLAN_LAB_REGISTRY_PASSWORD
+```
+
+Then preview and apply:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" deploy --apply
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" status
+unset GOVOPLAN_LAB_REGISTRY_PASSWORD
+```
+
+Deployment verifies the downloaded release manifest and keyring by pinned
+digest and by the existing GovOPlaN signature policy. It deploys PostgreSQL,
+Redis, single-node Garage and GreenMail on the state VM. The API, WebUI, workers
+and scheduler run in K3s from digest-pinned release images. A private lab CA
+protects both ingress and S3; backend pods receive only the CA Secret and keep
+TLS verification enabled. The CA profile carries critical `CA:TRUE` and
+`keyCertSign,cRLSign` constraints. `deploy` and `update` rotate older lab CAs
+that do not satisfy that profile and reissue the ingress/S3 certificate.
+
+The final output identifies two local files below `state_directory`:
+
+- `hosts` maps the public GovOPlaN and S3 test names to their VM addresses;
+- `pki/ca.crt` is the private lab CA certificate.
+
+Add the host mappings to the test client's resolver and trust the CA only on
+devices used for this lab. On Debian/Ubuntu:
+
+```bash
+STATE="$HOME/.local/share/govoplan/labs/govoplan-k8s-lab"
+cat "$STATE/hosts"
+sudo install -m 0644 "$STATE/pki/ca.crt" \
+  /usr/local/share/ca-certificates/govoplan-k8s-lab.crt
+sudo update-ca-certificates
+```
+
+Review mappings before adding them to `/etc/hosts`; the lifecycle does not edit
+the workstation's trust or resolver configuration. Reinstall `pki/ca.crt` in
+the client trust store after an automatic CA rotation.
+
+### Enroll the first administrator
+
+The production runtime does not create a default password. Issue one expiring,
+single-use first-administrator credential inside an API pod and copy its
+owner-only artifact out immediately:
+
+```bash
+KUBECTL="$STATE/bin/kubectl"
+POD="$($KUBECTL -n govoplan get pods \
+  -l app.kubernetes.io/component=api \
+  -o jsonpath='{.items[0].metadata.name}')"
+ARTIFACT="$STATE/first-admin-enrollment.json"
+umask 077
+
+$KUBECTL -n govoplan exec "$POD" -- \
+  python -m govoplan_core.commands.first_admin issue \
+  --reason 'initial Kubernetes lab enrollment' \
+  --output /tmp/first-admin-enrollment.json
+$KUBECTL -n govoplan exec "$POD" -- \
+  cat /tmp/first-admin-enrollment.json > "$ARTIFACT"
+$KUBECTL -n govoplan exec "$POD" -- \
+  rm -f /tmp/first-admin-enrollment.json
+chmod 0600 "$ARTIFACT"
+```
+
+Submit the token from that artifact once to
+`/api/v1/bootstrap/first-admin` with the administrator email, display name,
+password, tenant slug and tenant name. The password must contain at least 12
+characters. The lab command performs that exchange without placing either the
+token or password in process arguments, rejects redirects, and removes the
+artifact only after HTTP 201:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
+  --email 'owner@example.org' \
+  --display-name 'System Owner' \
+  --tenant-slug default \
+  --tenant-name 'Default Tenant'
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" enroll-admin \
+  --email 'owner@example.org' \
+  --display-name 'System Owner' \
+  --tenant-slug default \
+  --tenant-name 'Default Tenant' \
+  --apply
+```
+
+The public lab hostname must already resolve on the management workstation;
+the command verifies TLS through the generated private CA directly.
+
+## 5. Collect #27 Evidence
+
+Create a short-lived API key authorized to read the Ops status endpoint. In the
+current Access administration UI, open **Tenant API keys** and select only
+**View tenant settings** (`admin:settings:read`); the Ops endpoint explicitly
+accepts that compatibility scope. A dedicated operator credential may instead
+use `ops:operations:read`. Then run:
+
+```bash
+export GOVOPLAN_OPS_API_KEY='short-lived-value'
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" verify \
+  --exercise-api-pod-loss
+unset GOVOPLAN_OPS_API_KEY
+```
+
+The verifier requires ready API and WebUI pods across at least two Kubernetes
+nodes, all Deployments available, consistent runtime composition, queue
+coverage and a valid database-connection budget. During the optional drill it
+deletes one ready API pod, probes public readiness and waits for replacement.
+It writes sanitized output to
+`state_directory/evidence/kubernetes-multi-host.json` and never stores the API
+key. A rehearsal inventory prints an explicit warning that its result is not
+independent-failure-domain evidence.
+
+Retain these private artifacts together for review:
+
+1. `inventory.json` and the reviewed inventory TOML;
+2. the adopted release manifest/keyring and installation receipt;
+3. `kubernetes.json`;
+4. the Kubernetes verifier output;
+5. private cluster logs for the approved drill window;
+6. the operator's out-of-band evidence that the worker hypervisors are
+   independent physical hosts or availability zones.
+
+GovOPlaN #37 additionally requires independent assessment and production
+approval keys. Running its evidence jobs in containers is supported, but a
+container does not create an independent authority. Follow
+`TARGET_MATURITY_EVIDENCE_RUNBOOK.md` after the #27 drill passes.
+
+## 6. Update, Pause, Resume And Remove
+
+After reviewing and changing pinned image/K3s/release values in the inventory:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" update --apply
+```
+
+Workers are cordoned, drained, updated and uncordoned one at a time. K3s
+controls are reconciled serially. The release-specific migration Job remains
+subject to GovOPlaN's signed backup-evidence gate. The lab update command is not
+a substitute for creating recovery evidence before a destructive state-schema
+change.
+
+To stop compute use without deleting disks:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" pause --apply
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" resume --apply
+```
+
+To remove VM resources while preserving local evidence:
+
+```bash
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy
+"$PYTHON" tools/lab/govoplan-lab.py --config "$LAB" destroy \
+  --apply --confirm govoplan-k8s-lab
+```
+
+Add `--purge-local-state` only after evidence and recovery material have been
+retained elsewhere. That option deletes the generated local CA, secrets,
+manifests and evidence as well as the VMs.
+
+## Acceptance Boundary
+
+This tool supplies reproducible infrastructure and executes the bounded
+stateless-node drill. It does not certify the truth of operator-entered failure
+domains, provide HA PostgreSQL/Redis/Garage, create production backup evidence,
+or approve its own results. Those boundaries are deliberate: #27 can close
+after a passing run on independently controlled hosts; broader production
+maturity remains governed by #35, #37 and the target evidence runbook.

docs/operations/MODULE_CONTRACTS_AND_INSTALLS.md

@@ -0,0 +1,123 @@
+# Module Contracts and Install Boundaries
+
+## Why Some Changes Require `pip install`
+
+GovOPlaN discovers runtime modules through Python package entry points in the
+`govoplan.modules` group. Core reads those entry points with
+`importlib.metadata.entry_points()`, imports the configured manifest factory, and
+then builds the module registry from the returned `ModuleManifest`.
+
+That means the Python environment must know that a package exists before core can
+discover it.
+
+In development, `requirements-dev.txt` installs modules with `-e
+../govoplan-module`. With editable installs:
+
+- normal Python source changes are picked up after the process reloads;
+- manifest code changes are picked up after the process reloads;
+- new imports inside an already installed package are picked up after reload.
+
+`pip install` is still needed when package metadata changes:
+
+- a repository was not installed in the environment before;
+- a `pyproject.toml` entry point is added, renamed, or removed;
+- package dependencies change;
+- optional extras change;
+- package names or import roots change;
+- console scripts or other installed metadata change;
+- release installs need a different tag, wheel, or source ref.
+
+The same principle applies to WebUI packages: source changes are local during
+development, but `package.json` dependency or export changes require an install
+step so the consuming app sees the correct package metadata.
+
+## Current Contract Mechanism
+
+Modules already announce contracts through `ModuleManifest`:
+
+- `dependencies`
+- `optional_dependencies`
+- `required_capabilities`
+- `optional_capabilities`
+- `provides_interfaces`
+- `requires_interfaces`
+- `capability_factories`
+- permissions and role templates
+- route factories, migrations, docs, lifecycle hooks, and frontend metadata
+
+Core validates the active manifest graph when it builds the registry. Release
+tooling can inspect those manifests to calculate compatibility and migration
+impact.
+
+## What Core Can and Cannot Pick Up Automatically
+
+Core can pick up contract changes automatically after reload when the changed
+module package is already installed and importable.
+
+Core cannot discover a new module or new entry point that has not been installed
+into the environment, because there is no distribution metadata to enumerate.
+
+Core also cannot notify every module about a contract change at edit time by
+itself. The runtime registry is built from installed/importable packages. The
+right place for cross-module announcement is the meta repo tooling and CI:
+
+- scan manifests across all repositories;
+- build an impact graph from provided/required interfaces and capabilities;
+- run affected module tests;
+- post Gitea issue/release notes for affected modules;
+- block releases when a required interface is missing or incompatible.
+
+## Mitigation Strategy
+
+Use three layers:
+
+1. Editable development environment.
+   Keep `requirements-dev.txt` in the meta repo as the one workspace installer.
+   Source edits then need process reloads, not repeated full installs.
+
+2. Versioned runtime contracts.
+   Keep adding and tightening `provides_interfaces`, `requires_interfaces`, and
+   capability protocols. Treat interface names and versions as public module
+   contracts.
+
+3. Meta-level contract audit.
+   The meta repo statically reads `src/**/backend/manifest.py` files across all
+   repositories and validates provided/required interface ranges without
+   importing the packages. CI blocks missing or incompatible required
+   interfaces before release installs are attempted.
+
+Run the static graph check with:
+
+```sh
+./tools/checks/check-contracts.sh
+```
+
+Use `--json` when the release console or another automation needs structured
+provider/consumer impact data.
+
+## Practical Rule
+
+Do not run `pip install` for every code edit. Run it when package metadata or the
+set of installed packages changes.
+
+For normal development:
+
+```sh
+./.venv/bin/python tools/repo/sync-python-environment.py --requirements requirements-dev.txt --python ./.venv/bin/python
+```
+
+Then restart the server when Python code or manifests change. The development
+launcher runs this helper automatically by default; set
+`GOVOPLAN_AUTO_SYNC_PYTHON=0` to disable that preflight.
+
+For release validation:
+
+```sh
+./.venv/bin/python tools/repo/sync-python-environment.py --requirements requirements-release.txt --python ./.venv/bin/python
+./.venv/bin/python -m pip install -r requirements-release-tests.txt
+```
+
+That install is necessary because the release environment intentionally resolves
+tagged package refs, not local editable source trees. The second requirements
+file contains only the harness needed to execute tests from those immutable
+source tags; it is not part of the deployable release dependency set.

docs/operations/PACKAGE_REGISTRY_RELEASES.md

@@ -0,0 +1,263 @@
+# Package Registry Releases
+
+GovOPlaN publishes reusable module artifacts through Gitea's native PyPI and
+npm registries. These packages improve developer installation, release
+resolution, cacheability, and artifact inspection. They do not replace the
+signed runtime distribution: the signed manifest and digest-pinned OCI images
+remain the production deployment authority.
+
+## Publication boundary
+
+Every repository with a `pyproject.toml` contains
+`.gitea/workflows/module-package-release.yml`. The meta repository owns the
+canonical template and installs it with:
+
+```bash
+python tools/repo/sync-module-package-workflows.py --write
+python tools/repo/sync-module-package-workflows.py --check
+```
+
+The workflow runs for `v*` tags and may be dispatched manually for an existing
+tag. The organization preflight verifies that every package repository protects
+the `v*` namespace. Before building, the workflow itself verifies that:
+
+- the tagged commit is contained in `main`;
+- the tag, Python project version, and optional WebUI package version agree;
+- package names remain in the `govoplan-*` and `@govoplan/*-webui` namespaces.
+
+The workflow binds the repository explicitly from the Gitea Actions context.
+Do not rely on GitHub-compatible environment variables being injected by the
+runner image; Gitea runners may expose only the context values. Gitea 1.24 job
+tokens cannot read repository tag-protection settings, so package jobs must not
+receive a broad administrator token merely to repeat the organization preflight.
+Run the following before the first publication and after repository or tag-rule
+changes:
+
+```bash
+python tools/gitea/gitea-configure-package-releases.py
+```
+
+Preview and dispatch the exact wheel/WebUI versions selected by the developer
+meta-package with:
+
+```bash
+python tools/gitea/gitea-dispatch-package-set.py \
+  --env-file ~/.config/gitea/gitea.env
+python tools/gitea/gitea-dispatch-package-set.py \
+  --env-file ~/.config/gitea/gitea.env \
+  --apply
+```
+
+The dispatcher reads exact versions from `packages/govoplan-meta/pyproject.toml`,
+inspects the selected tag to determine whether a WebUI package is expected,
+skips complete registry pairs and does not duplicate an active workflow. Use
+`--repository govoplan-core` for a bounded dispatch or `--verify-existing` to
+rebuild and hash-verify versions already present in both registries.
+
+For coordinated lockstep tags, `push-release-tag.sh` pushes module tags first,
+Core next, and the meta tag last. This is a dependency guarantee for a
+single-capacity Actions runner: the developer package cannot run before its
+exact Core and module versions have entered the queue.
+
+The same release entry point first validates the migration graph, then records
+the reviewed current Alembic heads under the target release version and reruns
+the strict migration audit before it changes package versions, commits, or
+tags. The default preflight intentionally does not require those heads to exist
+in the previous release baseline. A failed candidate-baseline check therefore
+cannot produce a protected package release.
+
+The source gate validates `pyproject.toml`, the module version declaration
+(`MODULE_VERSION` or the top-level `ModuleManifest.version`), public package
+`__version__`, and WebUI metadata before creating tags. Release-tag artifact
+checks run only after the candidate tags and immutable WebUI lock have been
+created locally.
+
+Release-lock regeneration resolves a fresh immutable lock from the reviewed
+candidate manifests; it does not seed resolution from the previous release
+lock. This prevents removed transitive packages and stale peer metadata from
+blocking or contaminating the new release. Candidate resolution also uses an
+isolated temporary npm cache, so a locally replaced tag cannot reuse metadata
+from a failed, unpushed release attempt.
+
+Modules that retain the same WebUI package identity in both a root publish
+manifest and `webui/package.json` use the WebUI manifest as the canonical peer
+contract. The coordinated release synchronizes `peerDependencies` and
+`peerDependenciesMeta` into the publish manifest before creating the module
+tag, then synchronizes each lockfile root from the final package metadata. A
+distinct root package remains independent.
+
+It builds one wheel and, where applicable, one npm tarball. The workflow records
+the source tag, source commit, filename, size, and SHA-256 in
+`package-artifacts.json` before publishing. Gitea rejects a second upload of the
+same package version, so correction requires a new version rather than artifact
+replacement.
+
+A retry after partial publication is safe. Before upload, the workflow reads the
+native package registry file record and compares its SHA-256 with the artifact
+rebuilt from the protected tag. An exact existing artifact is skipped; a
+same-version artifact with another digest or an unexpected file set fails
+closed. This permits a failed npm publication to resume without weakening
+package immutability or accepting `--skip-existing` blindly.
+
+The npm tarball is always published through an explicit local `./dist/...`
+path. Without that prefix, npm may interpret a relative tarball name as a Git
+package shorthand before it ever contacts the configured registry.
+
+Published WebUI packages contain registry-compatible dependencies only. The
+workflow converts an internal dependency pinned to a protected `vX.Y.Z` Git tag
+into the exact `X.Y.Z` registry version and rejects unresolved `file:` or Git
+dependencies. Repository development metadata may therefore keep local or Git
+references without leaking them into the published package contract.
+Historical `add-ideas` and current `GovOPlaN` organization URLs are accepted
+for immutable tagged releases; both normalize to the same exact registry
+dependency and no branch or unversioned Git reference is accepted.
+
+## One-time Gitea setup
+
+Protect `v*` tags in every package repository and the meta repository. Allow
+only the `Owners` team to create or delete those tags.
+
+```bash
+set -a
+. ~/.config/gitea/gitea.env
+set +a
+python tools/gitea/gitea-configure-package-releases.py --apply
+```
+
+Create a dedicated personal access token with only `write:package` scope and
+store these organization-level Actions secrets on `GovOPlaN`:
+
+- `GOVOPLAN_PACKAGE_USERNAME`: account owning the package token;
+- `GOVOPLAN_PACKAGE_TOKEN`: dedicated package-write token.
+
+Do not use an administrator or general release token. Gitea 1.24 does not grant
+package publication to the automatic Actions job token. Organization secrets
+allow the same least-privilege credential to serve every module workflow.
+
+## Exact release consumption
+
+`tools/release/generate-release-package-set.py` supports two explicit package
+profiles. `base` translates the reviewed roots in `requirements-release.txt`;
+`full` reads the exact `govoplan[full]` dependency set from the developer
+meta-package. Both profiles resolve every version tag to its commit and verify
+the package metadata from that exact Git tree. The official module directory
+and immutable runtime distribution use `full`, so every publicly released
+module can be discovered without rebuilding the application image.
+
+`tools/release/resolve-package-artifacts.py` then downloads exactly those wheel
+and WebUI versions from Gitea. It reads the identity embedded in every wheel and
+npm tarball, rejects missing, duplicate, unexpected, or oversized artifacts,
+and writes `package-artifacts.lock.json` with credential-free HTTPS download
+URLs, SHA-256 values, and npm registry integrity values. The resolver verifies
+that the bytes downloaded by `npm pack` match the registry's own integrity
+record. Credentials are accepted only through environment variables and are
+never written to the lock. Python resolution ignores ambient pip configuration
+and extra indexes for GovOPlaN roots, preventing an internal package name from
+being selected from an undeclared registry.
+
+The runtime distribution workflow uses the verified full-profile wheelhouse
+directly and installs every selected module WebUI tarball only after matching
+it to the lock. It publishes the package set, package lock, and hash-locked
+requirements as release assets.
+The WebUI installer receives the absolute runtime-build interpreter path so its
+directory changes cannot escape the isolated release environment.
+Gitea 1.24 dispatches this workflow from a branch, but that branch is only the
+workflow implementation. The job fetches and peels the protected `v<version>`
+tag explicitly and materializes both `requirements-release.txt` and the
+developer meta-package from that Git tree. It then binds the signed distribution
+source and Gitea release assets to the same exact commit. A post-tag workflow
+repair can therefore retry publication without changing the released package
+composition or relabelling the later branch commit as released source.
+The package-lock SHA-256 is part of the signed distribution manifest. Runtime
+finalization also requires the lock's package versions and hashes to match the
+wheel composition embedded in the images. OCI assembly remains network-free
+after package and third-party dependency resolution.
+
+The source refs remain in the module catalog for source provenance and release
+planning. Production installation consumes the signed runtime images rather
+than invoking `pip`, `npm`, or Git on the target host.
+
+## Public module directory
+
+`tools/release/publish-release-catalog.sh` resolves the selected package set and
+registry lock before it creates a catalog. Catalog entries are synthesized from
+the exact tagged module manifests, never from a hand-maintained module list or
+the current workspace. Each entry binds its Python wheel and optional WebUI
+tarball to the registry URL, filename, size, SHA-256, package identity, source
+tag, and source commit before the complete catalog is signed.
+
+The same publication transaction regenerates and prunes the browsable static
+directory under `public/catalogs/v1/modules/`. It writes a global
+`modules/index.json`, one `<module>/index.json`, and one
+`<module>/<version>/manifest.json` for every entry in the signed channel.
+These files are derived from that exact signed payload and keyring; stale JSON
+from an older partial catalog is removed while unrelated static assets are left
+untouched. The signed channel remains the trust anchor, while the module
+directory provides stable discovery URLs for browsers and external tooling.
+
+Official GovOPlaN modules are open-source directory entries and do not require
+license entitlements. The generic `license_features` contract remains available
+for third-party package directories, support/configuration packages, or future
+deployment-specific presets. A catalog entry is gated only when that entry
+explicitly declares such features.
+
+Core carries the public stable catalog URL and its independently pinned trust
+anchor. In the absence of an operator-configured catalog, Admin discovers the
+official directory automatically. Selecting an entry creates a reviewed
+install/update plan; the trusted installer downloads the exact signed artifacts
+into a private digest cache, verifies size and hash, and installs only from that
+cache. A saved plan is rejected if any package ref, artifact identity, catalog
+channel, sequence, or signing-key identity differs from the currently validated
+catalog.
+
+The Admin directory can be searched by module, package, repository, or tag and
+filtered by available, installed, update, and blocked/withdrawn states. It
+shows the source revision, artifact digest, release notes, and configuration
+requirements. Missing dependency/interface providers and unsupported update
+windows are surfaced before an operator adds the entry to a plan; installer
+preflight remains authoritative.
+
+Catalog entries also carry the permission definitions declared by the tagged
+module manifest. Admin groups and exposes their scopes before an install or
+update is planned. This is disclosure only: installing a module does not grant
+its permissions to an account, role, group, tenant, or service account.
+
+Package lifecycle and availability are intentionally separate:
+
+- install, update, and uninstall change the instance-wide package composition;
+- enable and disable change the active instance runtime graph;
+- tenant module entitlements define unavailable, available, and forced modules;
+- group/user presentation is governed through Views and Policy; and
+- enabling a capability module does not opt data into that capability.
+
+Single-process or single-host installations may execute a supervised package
+plan locally. Shared-state and Kubernetes profiles reject node-local package
+mutation: operators compose and roll out a new signed full-profile runtime image
+instead. This prevents replicas from drifting while retaining the same Admin
+catalog and preflight experience.
+
+## Developer meta-package
+
+`packages/govoplan-meta` builds the optional `govoplan` package. Its default
+dependencies mirror the reviewed runtime roots; `govoplan[full]` adds all
+currently packageable workspace modules. Regenerate it after changing release
+requirements or package versions:
+
+```bash
+python tools/release/generate-developer-meta-package.py
+python tools/release/generate-developer-meta-package.py --check
+```
+
+`push-release-tag.sh` performs this synchronization before release commits and
+tags. The meta-package is for editable/developer setup and composition tests. It
+does not enable modules, apply migrations, provision services, or establish
+backup and recovery evidence.
+
+If the tag-triggered developer meta-package job fails before publication, rerun
+`publish-developer-meta-package.yml` with the existing protected version. The
+manual path validates that tag against `main`, checks out its exact commit, and
+publishes only when the registry does not already contain the same wheel hash.
+
+Generic Packages are intentionally not used. Add that transport only when a
+consumer needs an artifact format unsupported by PyPI, npm, Gitea Releases, or
+the OCI registry.

docs/operations/PRODUCTION_TARGET_HANDOFF.md

@@ -0,0 +1,190 @@
+# Production Target And Independent Evidence Handoff
+
+This runbook identifies the external inputs needed to finish
+[GovOPlaN #27](https://git.add-ideas.de/GovOPlaN/govoplan/issues/27) and
+[GovOPlaN #37](https://git.add-ideas.de/GovOPlaN/govoplan/issues/37). The
+repository can render, inspect and sign evidence for a target, but it cannot
+manufacture an independent failure domain or an independent approval authority.
+
+## GovOPlaN #27: real two-node target
+
+The bounded acceptance target is two independently schedulable worker nodes.
+The API and WebUI must each have ready replicas on both nodes, all Deployments
+must be available, the active module composition and software versions must be
+consistent, every configured queue must have a worker, and the database
+connection budget must pass. The validation then deletes one ready API pod and
+requires replacement without an observed readiness outage.
+
+Two virtual machines on different physical hosts or availability zones meet the
+failure-domain intent. Two containers, VMs or Kubernetes nodes on one physical
+host are useful development targets but do not close #27. A two-worker cluster
+also does not prove control-plane high availability. For a self-managed
+production cluster, use three control-plane nodes plus at least two workers; a
+managed control plane plus two workers is the shorter path.
+
+### What the target owner must provide
+
+Provide these through a secure handoff, not an issue, chat message or Git:
+
+1. A kubeconfig path with access to the target, for example
+   `~/.config/govoplan/targets/<target>.kubeconfig`, mode `0600`.
+2. A stable installation ID, public HTTPS hostname, namespace, ingress class and
+   TLS-secret or certificate-manager arrangement.
+3. Two independently schedulable workers and permission to place API and WebUI
+   replicas on both.
+4. External, logically shared PostgreSQL, Redis and S3 endpoints with trusted
+   CA material and network reachability from every worker. Do not co-locate the
+   only copies of these services on the two workers used for the failure drill.
+5. The six runtime secret values required by the generated manifest:
+   `MASTER_KEY_B64`, `DATABASE_URL`, `GOVOPLAN_DATABASE_URL_PGTOOLS`,
+   `REDIS_URL`, `FILE_STORAGE_S3_ACCESS_KEY_ID` and
+   `FILE_STORAGE_S3_SECRET_ACCESS_KEY`.
+6. A short-lived GovOPlaN API key limited to `ops:operations:read`, supplied in
+   `GOVOPLAN_OPS_API_KEY` only for evidence collection.
+7. An approved drill window and permission to delete one API pod.
+
+If no Kubernetes target exists, provide hostnames/IP addresses for the machines,
+an SSH user and key path, the internal/external DNS plan, and the permitted
+firewall ports. Those inputs are sufficient to provision a k3s target. They are
+not sufficient to claim control-plane HA unless three control-plane failure
+domains are present.
+
+The repository now supplies the strict libvirt/K3s lifecycle and example
+inventories for this handoff in
+[`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md). Its `acceptance` mode
+rejects a declared topology unless the workers and shared-state fixture occupy
+different hypervisor and failure-domain identifiers. Reviewers must still
+verify that those identifiers correspond to genuinely independent hosts.
+
+### Separate deployment and evidence authorities
+
+The deployment identity may create and update the namespace, Secret,
+ConfigMap, Deployments, Services, Jobs, PodDisruptionBudgets and Ingress. The
+evidence collector only needs:
+
+- cluster scope: `get` and `list` for `nodes`;
+- target namespace: `get` and `list` for `pods` and `deployments`;
+- target namespace during the approved drill: `delete` for `pods`.
+
+Use separate kubeconfig contexts or service accounts when the same person does
+not hold both roles.
+
+### Render, apply and verify
+
+Use the signed, digest-pinned installation bundle selected for the target:
+
+```bash
+export KUBECONFIG="$HOME/.config/govoplan/targets/<target>.kubeconfig"
+
+python tools/deployment/govoplan-deploy.py render-kubernetes \
+  --directory /srv/govoplan/<installation-id> \
+  --namespace govoplan \
+  --secret-name govoplan-runtime \
+  --tls-secret-name govoplan-tls \
+  --s3-ca-secret-name govoplan-s3-ca \
+  --ingress-class-name nginx \
+  --output /srv/govoplan/<installation-id>/kubernetes.json
+
+kubectl apply -f /srv/govoplan/<installation-id>/kubernetes.json
+kubectl -n govoplan wait --for=condition=available deployment --all --timeout=10m
+
+export GOVOPLAN_OPS_API_KEY="$(cat /run/secrets/govoplan-ops-evidence-key)"
+python tools/deployment/govoplan-deploy.py verify-kubernetes \
+  --directory /srv/govoplan/<installation-id> \
+  --namespace govoplan \
+  --exercise-api-pod-loss \
+  --output /srv/govoplan/<installation-id>/evidence/kubernetes-multi-host.json
+unset GOVOPLAN_OPS_API_KEY
+```
+
+The verifier emits sanitized JSON and exits nonzero if the topology, runtime,
+queue, connection-budget or pod-loss checks fail. Preserve the private cluster
+logs and manifest alongside the sanitized result in the controlled evidence
+store.
+
+## GovOPlaN #37: controlled signed target evidence
+
+Yes, collection, review and signing can run in containers. A container provides
+repeatability and process isolation; it does not create independent authority.
+The production approver must control a different private key from the target
+operator/assessor and must review the evidence before signing the
+`production_approval` scope.
+
+Use at least these three key boundaries:
+
+1. **Installer authority:** signs installed-release-origin receipts only.
+2. **Target assessment authority:** signs the permitted target, accessibility,
+   privacy, security, operations and recovery scopes.
+3. **Production approval authority:** independently signs only
+   `production_approval` after reviewing the other evidence.
+
+Do not reuse release-catalog keys for any of these roles. Keep private Ed25519
+keys outside Git, Gitea, GovOPlaN application storage and chat. Publish only the
+public keyrings. The proof issuer already rejects key reuse across release,
+installer and proof trust domains.
+
+### Generate independently held keys
+
+Each authority runs this command in its own `0700` directory. The generator
+refuses existing output paths and writes both files as `0600`:
+
+```bash
+install -d -m 0700 "$HOME/.config/govoplan/authority-keys"
+
+python tools/assessments/generate-authority-keypair.py \
+  --purpose proof \
+  --key-id authority:target-2026 \
+  --scope target_environment \
+  --scope accessibility \
+  --scope privacy \
+  --scope security \
+  --scope operations \
+  --scope recovery \
+  --private-key "$HOME/.config/govoplan/authority-keys/target-2026.pem" \
+  --keyring "$HOME/.config/govoplan/authority-keys/target-2026-public.json"
+```
+
+The independent production approver generates another key with only
+`--scope production_approval`. An installer authority uses `--purpose installer`
+and no `--scope`. Merge public key entries into the separately controlled
+keyrings only after the responsible authorities verify fingerprints out of
+band.
+
+### Container boundary
+
+Use two one-shot jobs or containers:
+
+- **Collector/assessor:** network access, read-only source and trust mounts,
+  read/write private evidence output, and the narrowly scoped kubeconfig. It
+  must not receive the production-approval private key.
+- **Production approver:** `--network none`, read-only assessment/evidence/trust
+  mounts, a read-only secret mount containing only the approval key, and a
+  separate output mount. It must not receive deployment credentials.
+
+Build or select the assessment image by digest and record that digest in the
+evidence log. A representative runtime shape is:
+
+```bash
+docker run --rm --network none --read-only --tmpfs /tmp \
+  --user "$(id -u):$(id -g)" \
+  --mount type=bind,src="$PWD/evidence",dst=/evidence,readonly \
+  --mount type=bind,src="$PWD/trust",dst=/trust,readonly \
+  --mount type=bind,src="$HOME/.config/govoplan/authority-keys",dst=/run/keys,readonly \
+  --mount type=bind,src="$PWD/approved",dst=/output \
+  <assessment-image>@sha256:<digest> \
+  <assessment command>
+```
+
+The current evidence commands and required scopes are documented in
+[`TARGET_MATURITY_EVIDENCE_RUNBOOK.md`](TARGET_MATURITY_EVIDENCE_RUNBOOK.md).
+The final proof must cover `target_environment`, `accessibility`, `privacy`,
+`security`, `operations`, `recovery` and independent `production_approval`, and
+must bind to the verified installed composition and installer receipt.
+
+## Completion boundary
+
+#27 can close after the real target produces a passing pod-loss result. #37 can
+close after an independently approved, schema-valid proof is generated for that
+same installed composition and the public authority keyrings, proof and private
+evidence custody references are recorded. Neither issue should close from a
+single-host simulation or a self-approved signature.

docs/operations/RECOVERY_AND_ROLLBACK_GUARANTEES.md

@@ -0,0 +1,155 @@
+# Recovery And Rollback Guarantees
+
+## Principle
+
+GovOPlaN must prove recovery claims with durable state recorded before and
+after side effects. A failed operation is not automatically rolled back merely
+because the previous application image still exists. Database schema and
+external effects may make release rollback unsafe.
+
+Core therefore distinguishes five recovery modes:
+
+| Mode | Meaning |
+| --- | --- |
+| `atomic` | One database transaction either commits or rolls back. No external effect is claimed. |
+| `compensation` | Durable evidence identifies explicit inverse actions for completed effects. |
+| `snapshot_restore` | A separately verified backup reference and restore procedure exist. |
+| `forward_recovery` | Repair or resume the current version; reverting code/configuration is not claimed safe. |
+| `irreversible` | No automated recovery is claimed and an approval reference is mandatory. |
+
+An operation plan must include verification steps. Compensation requires named
+compensation steps, snapshot restore requires a verified backup reference,
+forward recovery requires repair steps, and irreversible work requires explicit
+approval.
+
+## Core Recovery Ledger
+
+Core stores recovery operations and append-only, hash-chained checkpoints in
+PostgreSQL. The contract provides:
+
+- installation/module/resource identity;
+- an idempotency key bound to a canonical request hash;
+- recovery mode, preconditions, verification steps, and references;
+- optional runtime lease holder and fencing token;
+- explicit planned, prepared, running, recovery-required, recovering,
+  succeeded, recovered, failed, outcome-unknown, and manual-intervention states;
+- an evidence-chain head and sequence count;
+- rejection of plaintext secrets in metadata or evidence.
+
+Preparation cannot succeed without durable precondition evidence. A non-atomic
+operation cannot hide a partial effect by transitioning directly from running
+to failed. Success and recovery require explicit verification evidence with at
+least one check. The ledger verifies its hash chain before evidence is trusted.
+
+This is a platform contract, not an assertion that every existing module
+operation has adopted it. Module operations with external or multi-resource
+effects must be migrated to the ledger before claiming these guarantees.
+The owning-module inventory and adoption state are maintained in
+[Recovery Ledger Adoption](RECOVERY_LEDGER_ADOPTION.md); CI validates the
+machine-readable inventory so newly identified boundaries cannot disappear from
+the backlog silently.
+
+## Deployment Journal
+
+Every `govoplan-deploy apply` begins an operation journal before it pulls images
+or mutates runtime state. The private installation directory records:
+
+```text
+operations/<operation-id>/operation.json
+operations/<operation-id>/before/
+applied-state/
+```
+
+Each stage is hash-chained. The previous applied bundle is copied with per-file
+SHA-256 evidence. Applied state is replaced atomically after health verification;
+an interrupted replacement restores its previous directory.
+New journals also bind the complete desired deployment plan, snapshot
+availability, failure summary, recovery mode, and terminal status to the
+evidence chain. Recovery verifies every snapshot entry and checksum before it
+changes any live bundle file, then replaces each live file atomically. A crash
+between file replacements is recoverable by rerunning the same idempotent
+recovery command under the deployment lock.
+
+Inspect operations:
+
+```sh
+python tools/deployment/govoplan-deploy.py operations \
+  --directory /srv/govoplan/default
+```
+
+Recover the latest failed operation, or provide its identifier:
+
+```sh
+python tools/deployment/govoplan-deploy.py recover \
+  --directory /srv/govoplan/default \
+  --operation-id 20260801T120000Z-1234abcd
+```
+
+Add `--apply` only after reviewing the reported action.
+
+## Migration Boundary
+
+Before database migration starts, a failed deployment with a verified prior
+applied snapshot may restore its prior release/configuration bundle and
+reconcile that desired state.
+
+As soon as migration starts, the journal permanently changes to
+`forward_recovery`. It will not restore old application configuration because
+old code may not understand the new schema. Recovery then means one of:
+
+1. fix and re-run the current release;
+2. deploy a newer compatible repair release;
+3. restore a separately verified, coordinated database/object/key backup and
+   then deploy the matching release.
+
+The deployment tool does not create that backup. It does verify an externally
+produced, signed evidence contract covering PostgreSQL, objects, protected
+configuration, and key custody at one recovery point plus an isolated restore
+drill. A self-hosted release change cannot reach the migration command or be
+exported as a Kubernetes migration Job until fresh evidence bound to the
+previous immutable release has been adopted. Compose verifies it again after
+runtime quiescing. See
+[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md) for the contract,
+provider runbooks, RPO/RTO ownership, retention, and disposal rules.
+
+## Scaled Nodes
+
+Recovery actions must be safe across replicas:
+
+- drain affected API and worker nodes before incompatible changes;
+- use the deployment-wide PostgreSQL advisory lock for schema migration;
+- use distributed leases and fencing tokens for singleton or externally
+  visible effects;
+- use idempotency keys for retried commands and jobs;
+- retain shared object keys and database references until deletion succeeds;
+- classify uncertain external outcomes instead of retrying blindly;
+- verify the exact software/module composition after replacement.
+
+Campaign generated-message objects now follow this model: object writes are
+compensated when a build fails before database commit, workers verify stored
+size and digest before delivery, and retention keeps the database reference
+when storage deletion fails. A hard process loss between object creation and
+database commit can still leave an orphan object; an inventory reconciler is a
+separate operational slice and must use the build-specific object prefix.
+
+## Required Drills
+
+Record evidence for at least these scenarios before production acceptance:
+
+1. Kill an API replica and verify traffic continues without session loss.
+2. Drain and replace a worker while work is queued and while one job is active.
+3. Start two migration jobs and verify only one mutates schema.
+4. Kill the fenced scheduler and verify one replacement acquires a higher
+   fencing token.
+5. Fail deployment before migration and restore the prior applied bundle.
+6. Fail deployment after migration and verify old configuration is not
+   restored.
+7. Restore PostgreSQL, object storage, and encryption keys to one coordinated
+   recovery point and verify representative object hashes.
+8. Interrupt object storage during Campaign build and retention and verify
+   compensation/reference-preservation behavior.
+9. Tamper with a deployment or Core recovery checkpoint and verify chain
+   validation rejects it.
+
+No runbook, status badge, or green health endpoint substitutes for a dated,
+repeatable restore drill against the actual deployment topology.

docs/operations/RECOVERY_LEDGER_ADOPTION.md

@@ -0,0 +1,89 @@
+# Recovery Ledger Adoption
+
+The Core recovery ledger is a platform primitive, not automatic protection for
+module-owned effects. The canonical, machine-checked inventory is
+[`recovery-operation-inventory.json`](../recovery-operation-inventory.json).
+
+## Classification Rules
+
+- Use `atomic` only when every mutation commits in one database transaction and
+  no external effect occurs.
+- Use `compensation` when every completed effect has a bounded, verifiable
+  inverse action. A best-effort delete is not proof of compensation.
+- Use `snapshot_restore` only with fresh, signed backup evidence that covers all
+  affected state services at one recovery point.
+- Use `forward_recovery` for provider acceptance, queue publication, cursor
+  advancement, and other effects that may be resumable but cannot safely be
+  undone.
+- Use `irreversible` for approved purge or destruction where no automated
+  recovery is claimed.
+
+One feature may cross more than one boundary. Module installation is
+compensatable before schema migration, forward-only after migration starts, and
+snapshot-restorable for an approved destructive retirement. Mail submission is
+forward recovery because losing the response after provider acceptance must not
+cause an automatic resend.
+
+## Adoption Order
+
+1. Campaign build is the reference implementation for a database plus object
+   storage operation. Its operation reserves a build-specific object prefix,
+   persists request and precondition evidence before writes, records the final
+   object manifest, and verifies database/object state before success.
+2. Campaign delivery and Mail provider effects adopt outcome-unknown semantics
+   without weakening their existing provider-specific idempotency records.
+3. Files applies the same contract to uploads, purge, integrity reconciliation,
+   and writable connector synchronization.
+4. Connectors, Dataflow, and Workflow Engine consume the contract at their
+   registry/capability boundaries so optional providers remain optional.
+5. Core module lifecycle uses the ledger in addition to, not instead of, signed
+   deployment and backup evidence.
+
+Every fenced operation uses a process incarnation and distributed lease. A
+stale process cannot append a checkpoint or report success. An expired operation
+is claimed for recovery through an explicit takeover that preserves the prior
+fence in the checkpoint chain; it is never resumed as a normal retry.
+
+Connectors read-only sanctions and feed acquisitions are adopted: source
+revision/cursor and dry-run evidence are recorded before provider I/O, while
+the immutable snapshot and terminal checkpoint commit atomically. The generic
+external-mutation contract is conformance-tested but remains `planned` until a
+production connector actually publishes, updates, or deletes provider state.
+
+Dataflow runs are adopted. Database-only execution uses one atomic terminal
+commit for the run projection and recovery checkpoint. Output publication uses
+forward recovery: source and output digests are checkpointed before dispatch,
+a conclusive provider result commits with the run projection, and an expired
+or failed attempt after dispatch becomes `outcome_unknown`. A stale attempt may
+be retried only when its durable boundary proves dispatch had not started.
+
+Workflow Engine is adopted at both declared boundaries. Instance workers,
+trigger deliveries, and timer resumptions use process-bound distributed fences.
+Every module-action invocation records the pinned definition, input, preview,
+authority, provider-idempotency, and action-contract hashes before dispatch.
+Conclusive results commit with the Workflow projection. A lost acknowledgement,
+invalid result, or unannounced non-atomic effect becomes `outcome_unknown` and
+cannot be retried until evidence confirms either that the effect occurred or is
+absent. Linked Dataflow uncertainty blocks the Workflow without duplicating
+Dataflow's recovery authority.
+
+Core module lifecycle is adopted at four boundaries. Installer recovery is
+prepared before snapshots so a full database restore preserves the attempted
+operation. Pre-migration package changes use compensation, migrated changes use
+forward recovery, destructive retirement requires a hashed and restore-checked
+snapshot, and live graph changes restore the prior registry when no migration
+ran. A deployment-wide database fence serializes these effects; any unresolved
+predecessor blocks a differently keyed retry until explicit reconciliation.
+Supervised installs become successful only after restart and health evidence is
+recorded.
+
+## Operator Contract
+
+Ops lists non-terminal and manual-intervention operations. Operators must verify
+the checkpoint chain before trusting evidence, distinguish `outcome_unknown`
+from rejection, and use the owning module's documented reconciliation action.
+No evidence payload may contain credentials or resolved secrets.
+
+The parent adoption issue remains open until all inventory rows are adopted and
+the module matrix proves crash, retry, stale-fence, tamper, and optional-module
+behavior for each consequential path.

docs/operations/RELEASE_CONSOLE.md

@@ -0,0 +1,587 @@
+# GovOPlaN Release Console
+
+The release console is a local operator tool for planning and executing
+GovOPlaN releases. It belongs to the `govoplan` meta repository because it works
+across all local checkouts, release scripts, module manifests, migration audits,
+catalog files, Git state, and signing keys.
+
+The current implementation has a read-only dashboard, preview-only legacy
+controls, and a bounded durable executor for release steps whose inputs and
+effects can be verified safely:
+
+- inspect repositories from `repositories.json`
+- show dirty, ahead, behind, missing, no-HEAD, and tag state
+- show local package catalog and keyring state
+- optionally run release/dev migration audits
+- propose next actions without executing them
+- compare local catalog/keyring JSON with the published channel and public
+  keyring when online checks are enabled
+- configure target versions per release unit in the web UI
+- select the repositories that should advance through checkboxes
+- generate dry-run selective release plans for independently versioned packages
+- freeze a selective plan as a durable, resumable local release run
+- durably preflight a creation-time-bound repository, create its annotated tag,
+  and publish its branch/tag pair atomically
+- deterministically update recognized package/manifest version declarations and
+  commit only the receipt-bound metadata paths
+- order selected module providers before consumers, create module tags before
+  Core, regenerate Core's selected WebUI release lock, and re-run alignment
+  before any remote push
+- build selected Python wheels and generate a private, signed, receipt-bound
+  catalog candidate
+- publish that exact candidate through a verified website commit and immutable
+  tag after explicit confirmation
+- install selected candidate wheels into a private no-network/no-dependency
+  target and verify their installed metadata against the frozen plan
+
+Start it from the meta repository:
+
+```sh
+./.venv/bin/python tools/release/release-console.py
+```
+
+The launcher accepts only a numeric loopback address, binds to `127.0.0.1` by
+default, always creates a fresh API token, and prints that token only in the URL
+fragment. Open that URL in a browser on the same machine. A deliberately
+tokenless embedded app is read-only: non-GET `/api/` requests fail with `403`.
+Non-loopback operation needs a separately deployed authenticated TLS boundary;
+the local launcher will not expose the mutation API that way.
+
+Run durable release mutation only against an operator-private workspace.
+Repository roots, every path ancestor, critical and nested Git metadata, and
+tracked worktree files must be owned by the console process UID (or root where
+appropriate) and must not be group/world writable. Symlink/special Git
+metadata, object alternates, and grafts are rejected. The console pins
+`/usr/bin/git`, `/usr/bin/ssh`, a fixed system `PATH`, disabled hooks, isolated
+Git configuration, and no replace objects. Shared or `nfsnobody`-owned
+checkouts remain usable for read-only planning, but every durable executor
+fails closed there; clone the registered origins into a private workspace
+before releasing.
+
+The runtime itself is part of the authority boundary. Durable run creation
+verifies the meta checkout, release/check tooling, repository registry, Python
+environment, loaded `govoplan_core` and cryptography packages, and Git/SSH
+executables. It then binds a clean meta-repository HEAD and named branch that
+exactly match the registered `origin`. Every execution and reconciliation
+rechecks that receipt. Permission checks cannot prove that code was safe before
+it entered a writable tree, so release from a fresh operator-private clone or a
+separately verified installed console artifact.
+
+The web UI starts with a repository table. Each repository can be checked
+independently and assigned its own target version. Repositories without version
+metadata remain visible so they can be planned as initial releases. `Build Plan`
+shows the dry-run commands for the selected rows, and `Generate Candidate`
+creates a signed catalog candidate that advances only selected repositories that
+already have a catalog entry.
+
+The full-width **Release Workflow** guide projects the server state into seven
+operator phases: Inspect, Targets, Validate, Source, Package, Publish, and
+Verify. It does not maintain a second workflow state. Completed, current,
+blocked, locked, and unavailable phases are derived from the dashboard,
+selective plan, and durable run record. The next-action panel opens the exact
+section or durable step that needs attention. Changing a channel, target
+version, repository selection, or release gate detaches the browser from the
+current run and invalidates the draft plan; the persisted run remains available
+from the saved-run selector. Problems in unselected repositories remain visible
+as workspace notices but do not lock an unrelated release; the selective plan
+is the authority for blockers in the selected repository set.
+
+Installation verification is an explicit durable step after catalog
+publication. It verifies the exact candidate receipt, installs every selected
+Python wheel into a temporary target with network and dependency resolution
+disabled, and compares installed names and versions with the frozen plan.
+Deployment startup, database upgrades, and module-combination smoke tests remain
+release-integration CI gates; the console does not represent its local wheel
+check as a production deployment.
+
+`Build Plan` also returns structured release-gate findings for each selected
+repository. The plan names the recommended next action and gives an explicit
+remediation for source-version, lockfile, Core WebUI composition, Git state, and
+worktree findings. A target version that differs from internally consistent
+source metadata becomes a bounded `UPDATE` step. Unsupported, missing, or
+internally inconsistent declarations remain a blocker with exact remediation.
+`source_preflight_ready` means that plan-visible source gates pass or have a
+bounded deterministic mutation; the non-mutating `Preview Tag + Publish`
+remains mandatory for remote, manifest, and immutable-tag checks.
+
+## Durable release runs
+
+The **Durable Run State** card turns the current repository/version selection
+into a versioned local run record. The server rebuilds the selective plan and
+requires the plan to resolve exactly the requested repositories and target
+versions; the browser cannot submit or replace the plan snapshot. The input and
+plan are then immutable and covered by a canonical SHA-256 integrity digest.
+Every executable repository step also carries its creation-time full HEAD,
+branch, target tag, a SHA-256 over both the fetch and push URLs of `origin`, and
+a SHA-256 over bounded dirty-path names and bytes. Both URLs must exactly equal
+the remote registered in `repositories.json`; a changed HEAD, branch, worktree,
+remote, push URL, or metadata byte requires a new run or explicit
+interrupted-step reconciliation rather than silently retargeting the frozen
+compatibility decision.
+The complete record also has a checksum so a valid-looking manual edit to its
+mutable state fails closed. File permissions remain the authority boundary;
+these digests detect accidental or manual corruption, not an attacker who can
+replace the private record and recompute its checksums. Changing a target,
+channel, or gate input requires a new run.
+
+Creation requires a caller-generated `request_id`. Its SHA-256 fingerprint is
+the private, workspace-scoped durable mapping to exactly one run; the raw
+identifier is never persisted. Repeating the same identifier with the same
+immutable inputs returns that run without rebuilding the plan while it remains
+inside the bounded local retention window, even if the live dashboard has
+since drifted. Reusing it with different inputs fails closed. The browser keeps
+an uncertain create identifier in session storage,
+replays it after reload, and selects the known run returned by the server. A
+successful create remains shown as saved if only the subsequent list refresh
+fails.
+
+Run records survive console restarts, but remain local operator state rather
+than a signed release artifact or the system audit log. The default location is
+`$XDG_STATE_HOME/govoplan/release-console/workspace-<sha256>/release-runs/`, or
+`~/.local/state/...` when `XDG_STATE_HOME` is unset or relative. It is outside
+the source checkout so filesystems without enforceable POSIX modes cannot
+silently weaken the journal. Newly created state directories use mode `0700`
+and records use `0600`. Writes use a cross-process lock, a same-directory
+temporary file, `fsync`, atomic replacement, and directory/parent `fsync`.
+Symbolic-link paths, untrusted owners or writable ancestry, overly broad record
+modes, malformed schemas, unknown fields, invalid state combinations,
+oversized files, and digest mismatches fail closed. A bad record is neither
+rewritten nor automatically quarantined.
+
+The store retains at most 512 workspace-scoped run records created through the
+console. On creation at that limit it removes only the oldest fully completed,
+integrity-verified record. Running, planned, attention, blocked, foreign, and
+unreadable records are never deleted implicitly; if no completed record is
+available, creation fails closed with a retention remediation. Unavailable
+records still consume the bound and remain visible in cursor-paginated lists
+so corruption cannot be hidden by normal turnover.
+
+The full resolved-workspace SHA-256 is part of both the private storage
+namespace and immutable input snapshot. Every list, read, and state transition
+checks it. An alternate workspace therefore cannot list or resume another
+workspace's runs. This remains true for an embedding/test `run_state_root`
+override: the server always appends
+`workspace-<full-sha256>/release-runs/` rather than treating the override as a
+shared record directory. Durable candidates use its private sibling
+`release-candidates/` directory, never a checkout-local runtime path. A corrupt
+record from one workspace therefore cannot leak even its identifier or an
+integrity error into another workspace.
+
+Each frozen plan step has an explicit `pending`, `running`, `succeeded`,
+`failed`, or `interrupted` state. Plan order remains a prerequisite: a later
+step is unavailable until earlier steps have succeeded. Exact attempt and
+resume/retry/reconciliation request identifiers are fingerprinted so delayed
+repetitions remain idempotent for the retained run's lifetime. These fingerprints are
+never evicted: the store fails closed before accepting more than 2,048 commands
+or 2,048 attempts and asks the operator to create a fresh run. The display
+record keeps at most 256 server-generated state events. Events contain only an
+enum event type, timestamp, step identifier, and bounded result code—never
+commands, process output, confirmation text, credentials, bearer tokens, or
+signing material.
+
+Before a step can enter `running`, the store reserves the two command-ledger
+slots needed for worst-case recovery. It also projects the serialized record
+through start, finish/failure, resume, and the required terminal reconciliation
+or read-only retry; the start is rejected unless every required atomic write
+fits the record-size bound. An explicit resume consumes one slot and is
+accepted only while a persisted attempt is actually running. A mutating attempt
+always retains its final `effect_absent` or `effect_succeeded` slot;
+`unresolved` may be recorded at most once for that attempt and only when an
+additional ledger slot and serialized terminal-write capacity are available.
+Read-only interruption and known failure retain the slot and byte capacity
+needed to prepare a retry. Capacity exhaustion is therefore detected before
+starting an effect rather than stranding an ambiguous attempt.
+
+An explicit resume after a process restart converts every persisted `running`
+step to `interrupted`; it never guesses whether an external effect happened.
+An interrupted read-only step can be prepared for retry. An interrupted
+mutating step remains unavailable until the operator independently reconciles
+local and remote state, selects `effect_absent`, `effect_succeeded`, or
+`unresolved`, and types `RECONCILE`. `effect_absent` prepares a safe new
+attempt, `effect_succeeded` advances the run without repeating the external
+effect, and `unresolved` keeps the run blocked. Each outcome emits a bounded,
+code-only state event. A known failed attempt can likewise be prepared for
+retry. The UI keeps unavailable controls visible and disabled.
+
+Supported executors durably claim the step before invoking an effect. Exact
+attempt replays return the recorded outcome and never invoke the executor a
+second time. Successful repository preflight, version, commit, Core-bundle,
+tag, and push steps persist a bounded repository-state receipt. Version
+reconciliation requires aligned declarations in the same commit; commit
+reconciliation requires the expected single-parent release commit and only
+recognized metadata paths. Tag reconciliation independently requires an
+annotated local tag at the frozen HEAD; push reconciliation additionally
+requires both the remote annotated tag object and remote branch to match.
+Catalog generation persists
+only its server-issued opaque candidate ID and canonical catalog SHA-256, then
+re-resolves and re-hashes that private candidate before publication.
+
+Repository capabilities are frozen into each plan unit (`python-package`,
+`webui-package`, `module-manifest`, `database-migrations`, `documentation`,
+`core-release-bundle`, and the universal `git-source`) and determine which
+steps appear. Internally aligned version changes are rendered deterministically
+from recognized TOML, JSON, lockfile, manifest, and package declarations.
+Pre-existing dirty worktrees remain visible but have no commit executor; the
+console never absorbs unrelated operator changes.
+
+For mixed releases, module interface providers are ordered before consumers
+and Core is tagged last. The durable sequence creates and commits module
+metadata, creates local module tags, updates Core's selected WebUI references,
+regenerates the release lock against those local tags, commits/tags Core, and
+runs a receipt-bound alignment gate before exposing any atomic branch/tag push.
+A failed step stops later steps while preserving prior receipts for explicit
+retry or reconciliation.
+
+The browser likewise retains the request identifier for an uncertain
+resume/retry/reconciliation response and replays it after reload. A successful
+replay selects the returned run state. Transport and server failures retain the
+identifier; a deterministic `4xx` rejection clears it so a stale command cannot
+poison a later attempt.
+
+The run API is covered by the same local console token middleware as every
+other `/api/` route:
+
+- `POST /api/release-runs` requires `request_id`, then idempotently rebuilds and
+  freezes a selective plan only when that creation is not already known.
+- `GET /api/release-runs` lists bounded summaries ordered by immutable
+  `created_at` and run identifier. `next_cursor` advances a stable descending
+  traversal even while older runs are updated, without offset duplicates or
+  skips. Unreadable entries
+  remain a deterministic final section and are therefore reachable through
+  pagination instead of displacing newer verified runs.
+- `GET /api/release-runs/{run_id}` reads and verifies one exact record.
+- `POST /api/release-runs/{run_id}/resume` records explicit recovery.
+- `POST /api/release-runs/{run_id}/steps/{step_id}/retry` prepares a failed or
+  read-only interrupted step for another attempt.
+- `POST /api/release-runs/{run_id}/steps/{step_id}/reconcile` records a
+  confirmed observed outcome for an interrupted mutating step.
+- `POST /api/release-runs/{run_id}/steps/{step_id}/execute` claims and invokes
+  only the narrow executor declared by the immutable plan step. Durable release
+  execution accepts only the registered `origin`, never a caller-selected
+  remote.
+- `POST /api/release-runs/{run_id}/steps/{step_id}/preview` provides the
+  non-mutating preview for receipt-bound catalog publication.
+
+Run-storage errors are confined to the Durable Release Run section; dashboard
+and release-preview collection continue and the workflow guide points to the
+bounded storage remediation.
+
+The run record is execution evidence only for a supported step whose durable
+claim and bounded result receipt were persisted. The console never infers
+success from a button click or process exit alone. An executor exception or a
+lost result write leaves the attempt interrupted and non-retriable until
+explicit recovery. Catalog publication persists the candidate and keyring
+hashes, exact website commit, annotated tag object and peeled commit, branch,
+tag name, and registered-origin digest. A successful reconciliation revalidates
+the candidate against the trust anchor in the frozen website parent, requires
+the exact deterministic catalog/keyring/module blobs and full commit delta, a
+sole frozen parent, and matching local and remote branch/tag identities. If any
+part cannot be proved, the run remains interrupted and may only be recorded as
+`unresolved`.
+
+Dashboard collection is also fail closed. Unreadable Core version metadata or a
+malformed module contract is returned as a bounded `collection_errors` entry
+with a remediation, marks the dashboard blocked, and becomes a structured
+blocker in both full and selective release plans. The console does not silently
+omit a contract or turn these source errors into an HTTP 500.
+
+The release-control area above the repository table is read-only and is meant
+to become the central release cockpit. It shows:
+
+- local and published channel health
+- catalog/keyring drift
+- signature and trusted-key status
+- published module versions and refs
+- local checkout version drift against the catalog
+- catalog-declared interface compatibility
+
+The target-version control can generate the next major, minor, or subversion
+from the current base version. Manual target input accepts explicit versions
+such as `0.2.0` or `0.2.0-alpha1`, but requires the first three version numbers
+to move forward.
+
+Plain repository pushes are separate from catalog publication. `Preview Push`
+shows the selected repository push commands. `Push Selected` requires `PUSH` in
+the repository push confirmation field.
+
+The source release panel retains `Preview Tag + Publish` as a non-mutating
+inspection. Its legacy `Create Tags` and `Publish Tags` controls stay visible
+but disabled; the corresponding mutation endpoint rejects apply requests.
+Creation and atomic branch/tag publication use the `TAG` and `PUBLISH`
+confirmations on the durable run steps. The gate requires an aligned target
+version, a clean named branch with a HEAD, and a checkout that is not behind. Existing
+local or remote tags must resolve to the selected HEAD and are never moved.
+The local and remote annotated tag objects must also be identical, not merely
+point at the same commit. Before any source tag is created, the console loads
+the cross-repository module registry. This release gate also rejects every
+user-facing workflow documentation topic that has no scope condition, or has
+an alternative condition without `required_scopes` or `any_scopes`.
+
+The catalog workflow panel can also operate on the same selected rows for
+generation and preview. Its legacy apply/push controls stay disabled; durable
+publication consumes only the candidate receipt recorded by that run:
+
+- `Generate` creates a signed candidate in the operator's private XDG state
+  directory and records only an opaque candidate ID plus the canonical catalog
+  SHA-256 in durable run state.
+- `Preview` validates a candidate and shows what would be copied into the
+  website repository.
+- `Apply + Website Tag` remains visible but disabled outside a durable run.
+- `Push Website Release` remains visible but disabled outside a durable run.
+
+Source release tags belong to Core or module repositories. Website catalog
+publication creates a separate catalog tag in the website repository; the UI
+names these independently to avoid confusing the two immutable references.
+
+Candidate signing is fail-closed: every Core and module source ref in the
+complete post-update catalog must already have the requested annotated tag both
+locally and on the configured source remote. Both tags must be the same
+annotation object with version-aligned tagged package metadata. Selected tags
+must additionally resolve to the selected clean, version-aligned HEAD, and the
+signed `selected_units` record captures the peeled commit and tag-object
+identifiers. Create and publish the source tags before using `Generate`.
+
+Catalog publication repeats that check for selected units and also verifies
+every Core and module source ref in the complete candidate, including entries
+preserved from the previous catalog. Historical refs need not be at the current
+worktree HEAD, but their local and remote annotated tags must match and their
+tagged installable package and module-manifest metadata must declare the catalog
+version. New selected releases additionally pass the stricter current-source
+alignment gate, including public runtime version declarations. A candidate
+cannot be applied or published while any referenced source tag is absent or
+inconsistent; the preview and API response list each repository and missing or
+invalid ref so the operator can repair the exact source releases first.
+
+Every selected Python release must also supply its exact built wheel. Durable
+generation first verifies that the receipt-bound annotated tag is the same
+object locally and on the registered origin. It clones an isolated checkout at
+the receipt's exact commit and builds from that checkout, never from the mutable
+live worktree. Every authenticated base-catalog source is likewise cloned from
+its registered origin at an annotated release tag; only selected repositories
+contribute synthesized fields. The base catalog and keyring are read as one
+authenticated, exact private snapshot before synthesis.
+
+Python wheels are built by a required Bubblewrap worker with no network,
+private temporary/home directories, a read-only exact source mount, no host
+home or file keys, cleared environment, fixed system tools, and CPU/address
+space/process/file-size limits. If a trusted Bubblewrap launcher is unavailable,
+generation fails closed. The worker must return one bounded regular wheel; the
+console copies it through no-follow descriptors into a fresh private file,
+`fsync`s it, then validates its package identity. Generation computes the
+archive SHA-256 and an install-stable
+payload identity from one bounded, regular-file descriptor and signs those
+values into `release.artifacts`. Updating a Python version without a matching
+wheel removes the stale identity. A source-only preview can still explain the
+gap, but apply, commit, tag, and push fail closed until every selected Python
+unit has a matching built-artifact identity. Repositories selected only for a
+meta/source tag do not need a Python artifact.
+
+Candidate directories and files are operator-owned `0700`/`0600` state. Before
+any later release step consumes one, the executor re-resolves the opaque handle
+below the configured root and re-hashes the signed catalog against its persisted
+receipt. Shared roots, symlinks, channel path fragments, altered candidates, and
+unowned files are rejected.
+
+The current same-host worker is a containment baseline, not the final hostile
+build-service boundary. `RLIMIT_FSIZE` is per file, and the worker does not yet
+have a size-limited filesystem/cgroup quota, a fresh kernel keyring, or a
+seccomp profile denying keyctl/request-key/ptrace/mount operations. A malicious
+backend could therefore exhaust scratch disk/inodes or target same-UID kernel
+facilities. Closing that denial-of-service/isolation gap requires a dedicated
+quota/cgroup worker with a fresh keyring and seccomp policy.
+
+The server-owned wheel builds remain under the private candidate's `artifacts/`
+directory. This slice signs their identities but does **not** upload the wheel or
+add a download URL to the public catalog; the existing Git `python_ref` is source
+provenance and rebuilding it is not an equivalent artifact. Keep the private
+candidate until the exact wheels have been transferred through an approved
+deployment channel, verified against `archive_sha256`, consumed by the installer,
+and covered by its signed receipt. Public artifact transport and receipt-aware
+candidate cleanup remain separate release-lifecycle work.
+
+Read-only provenance checks derive a same-host HTTPS Git URL from conventional
+SSH remotes when possible, with a non-interactive check of the configured remote
+as fallback. Source tag creation and atomic publication always use the explicit
+configured Git remote.
+
+The default signing key is
+`$HOME/.config/govoplan/release-keys/release-key-1.pem` when no signing key is
+entered in the UI. Signing-key files must be regular, owned by the operator, and
+inaccessible to group/other users. The browser sends a configured key path only
+on the initial execution request; it never persists signing material in session
+storage, and request-ID recovery replays no key path.
+
+Generate a selective release plan from the terminal:
+
+```sh
+./.venv/bin/python tools/release/release-plan.py \
+  --repo govoplan-files \
+  --target-version 0.1.9 \
+  --channel stable \
+  --online
+```
+
+`--online` compares the local catalog/keyring with the published channel and
+published keyring. Use `--remote-tags` only when the plan also needs to check
+Git remotes for tag existence; that can be slower across the full repository
+set.
+
+Build a signed selective catalog candidate:
+
+```sh
+KEY_DIR="$HOME/.config/govoplan/release-keys"
+ARTIFACT_DIR="$(mktemp -d)"
+
+../govoplan-files/.venv/bin/python -m pip wheel \
+  --no-deps \
+  --no-build-isolation \
+  --wheel-dir "$ARTIFACT_DIR" \
+  ../govoplan-files
+
+./.venv/bin/python tools/release/release-catalog.py selective \
+  --repo-version govoplan-files=0.1.9 \
+  --python-artifact \
+    "govoplan-files=$ARTIFACT_DIR/govoplan_files-0.1.9-py3-none-any.whl" \
+  --channel stable \
+  --catalog-signing-key "release-key-1=$KEY_DIR/release-key-1.pem"
+```
+
+This writes candidate catalog/keyring files below
+`$XDG_STATE_HOME/govoplan/release-candidates/` (or
+`~/.local/state/govoplan/release-candidates/`), generates the browsable module
+directory below `modules/`, validates the signed candidate with the module
+installer validator, and reports whether the candidate catalog/keyring match the
+currently published channel. It does not publish to the website repository.
+
+Regenerate the browsable module directory from an existing catalog/keyring:
+
+```sh
+./.venv/bin/python tools/release/release-catalog.py module-directory \
+  --catalog ../addideas-govoplan-website/public/catalogs/v1/channels/stable.json \
+  --keyring ../addideas-govoplan-website/public/catalogs/v1/keyring.json \
+  --output-dir runtime/module-directory-preview \
+  --channel stable
+```
+
+Preview publication of a reviewed candidate:
+
+```sh
+CANDIDATE_ROOT="${XDG_STATE_HOME:-$HOME/.local/state}/govoplan/release-candidates"
+
+./.venv/bin/python tools/release/release-catalog.py publish-candidate \
+  --candidate-dir "$CANDIDATE_ROOT/stable-YYYYMMDD-HHMMSS" \
+  --channel stable
+```
+
+Apply the reviewed candidate into the website repository without pushing:
+
+```sh
+./.venv/bin/python tools/release/release-catalog.py publish-candidate \
+  --candidate-dir "$CANDIDATE_ROOT/stable-YYYYMMDD-HHMMSS" \
+  --channel stable \
+  --apply \
+  --commit \
+  --tag
+```
+
+Push is a separate explicit flag:
+
+```sh
+./.venv/bin/python tools/release/release-catalog.py publish-candidate \
+  --candidate-dir "$CANDIDATE_ROOT/stable-YYYYMMDD-HHMMSS" \
+  --channel stable \
+  --apply \
+  --commit \
+  --tag \
+  --push
+```
+
+Publication validates the same in-memory catalog object that it writes; it does
+not copy a path that can change after validation. On commit, the console reads
+the catalog, keyring, and complete module directory back from the immutable Git
+tree and requires byte-for-byte equality with those validated objects. Tags and
+remote branch updates then reference that exact commit SHA rather than the
+mutable worktree `HEAD`.
+
+Published channels are expected below the public catalog base URL:
+
+- `https://govoplan.add-ideas.de/catalogs/v1/channels/stable.json`
+- `https://govoplan.add-ideas.de/catalogs/v1/keyring.json`
+
+The console treats channels as release artifacts. A package can advance without
+forcing every repository to the same tag, but channel publication must preserve
+the unchanged package versions, validate interface compatibility, sign the
+updated catalog, and keep the published keyring healthy.
+
+When a selected module exposes a WebUI package, its requested version must also
+match Core's `webui/package.release.json` input and the resolved
+`package-lock.release.json` entry. The source-tag preflight, selective plan, and
+catalog-candidate writer all enforce this composition boundary. Pins for modules
+that are not part of the selective release remain unchanged.
+
+Release integration also enforces repository and composition version alignment
+and generates a CycloneDX SBOM from the resolved Python environment and the
+release WebUI lockfile. Catalog publication should attach that immutable SBOM
+and its digest to the corresponding Core/composition release.
+
+Addresses and Notifications have current WebUI source contributions but are
+not part of the pinned v0.1.8 WebUI release composition because their v0.1.8
+tags predate those packages. They re-enter the release composition only through
+new, immutable module tags whose backend, manifest, frontend, and lock metadata
+pass the alignment gate.
+
+Candidate publication verifies signatures against the already published
+keyring, not against keys supplied only by the candidate. A changed keyring
+must have its canonical SHA-256 embedded in the signed catalog. The public
+module-directory files are regenerated from that verified catalog and keyring
+at publication time; candidate-supplied directory files are never copied as
+authoritative provenance.
+
+## Published Module Directory
+
+The target release repository is an online, browsable module directory. The
+catalog remains the machine-readable channel entry point, but the published
+space should also expose a tree that operators and installations can inspect:
+
+- `/catalogs/v1/channels/<channel>.json` describes the active channel state.
+- `/catalogs/v1/keyring.json` publishes trusted release signing keys.
+- `/catalogs/v1/modules/<module>/<version>/manifest.json` describes one
+  published module version, including repository refs, package refs, contracts,
+  compatibility windows, signatures, and available artifacts.
+- `/catalogs/v1/modules/<module>/index.json` lists available versions for
+  one module.
+- `/catalogs/v1/modules/index.json` lists all published modules.
+
+Gitea tags/releases remain the source release anchors. The public GovOPlaN
+catalog directory becomes the installation-facing release repository that points
+to those anchors and carries structured compatibility information.
+
+Selective catalog candidates now include this directory tree. Publishing a
+candidate copies the channel catalog, keyring, and `modules/` tree into the
+website repository together.
+
+Selective catalog generation can add a module that is not yet represented in
+the source channel. Initial entries are synthesized from the selected
+repository's `[project]` metadata, `govoplan.modules` entry point, and runtime
+`ModuleManifest`; there is no separate hand-maintained initial-entry list. The
+release gate requires the entry point to resolve inside the selected checkout,
+exact package/manifest/frontend version agreement, verified source-tag
+provenance, a non-conflicting module id, and complete required dependency and
+interface closure in the resulting candidate. Classification beyond the
+manifest-derived `official` tag remains an explicit later catalog curation
+step rather than an inferred business/service label.
+
+## Direction
+
+The console should grow in slices:
+
+1. Read-only dashboard and next-action suggestions.
+2. Release plan builder that writes explicit JSON plans.
+3. Dry-run executor that shows exact commands and expected file changes.
+4. Apply executor for commit, tag, push, artifact build, signing, and catalog
+   publication.
+5. Compatibility planner for manifest contracts and version ranges.
+6. Install/test workflow that validates a release from tags or catalog entries.
+
+All mutation must stay explicit: plan first, dry run second, apply only after a
+clear confirmation.

docs/operations/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

@@ -0,0 +1,280 @@
+# Scaling And Multi-Host Deployment
+
+For the exact external handoff, least-privilege collector permissions and live
+two-node acceptance procedure, see
+[`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
+For a reproducible local or multi-hypervisor libvirt/K3s target, use
+[`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md).
+
+## Implemented Contract
+
+GovOPlaN now supports a stateless application tier backed by logically shared
+state services. The runtime roles are independently replaceable API, WebUI,
+worker, and scheduler processes. Every replica in one installation must use the
+same immutable release composition and the same:
+
+- `GOVOPLAN_INSTALLATION_ID`;
+- PostgreSQL database;
+- Redis broker and coordination service;
+- `MASTER_KEY_B64` and deployment secret references;
+- enabled-module graph;
+- S3-compatible object-storage namespace.
+
+The application tier must not use node-local durable business data in a
+multi-host deployment. Files owns managed file metadata while Core provides the
+storage-backend contract. Campaign build artifacts are stored under opaque
+object keys and workers read those objects from the shared backend. Temporary
+build and materialization directories may remain node-local because they are
+discardable.
+
+Core validates three explicit state profiles:
+
+| Profile | Supported shape | Storage rule |
+| --- | --- | --- |
+| `local` | One API and one worker process for development | Local filesystem permitted. |
+| `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. |
+| `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. |
+
+`shared` also requires a stable installation identifier. Module package
+mutation is blocked in this profile: build and verify a new immutable release,
+then roll the complete cluster to it.
+
+## Same-Host Compose
+
+The generated Compose bundle provides:
+
+```text
+client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas
+
+API/worker/scheduler -> PostgreSQL
+                     -> Redis
+                     -> local volume, managed Garage, or external S3
+```
+
+HAProxy discovers Compose replicas through Docker DNS and performs health-aware
+balancing without mounting the Docker socket. This improves concurrency and
+permits process replacement, but the Docker host and installer-managed stateful
+services remain single failure domains. Generated Compose therefore declares
+the `host-shared` state profile even when its shared storage happens to be an
+external S3 service. Its API backend checks `/health/ready`, so drain or
+coordination loss removes a replica from rotation. Container, load-balancer,
+and Kubernetes probes send the configured public host explicitly, keeping
+readiness compatible with strict trusted-host validation.
+
+Managed Garage is a convenient single-node S3-compatible service. It is not a
+multi-host storage cluster. Use an independently operated Garage cluster or
+another S3-compatible service for the `shared` profile.
+
+## Kubernetes Export
+
+The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL,
+Redis, and S3 are all external:
+
+```sh
+python tools/deployment/govoplan-deploy.py render-kubernetes \
+  --directory /srv/govoplan/default \
+  --namespace govoplan \
+  --secret-name govoplan-runtime \
+  --tls-secret-name govoplan-tls \
+  --s3-ca-secret-name govoplan-s3-ca \
+  --ingress-class-name nginx \
+  --output /srv/govoplan/default/kubernetes.json
+```
+
+The export contains a Namespace, tokenless ServiceAccount, non-secret
+ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption
+budgets, Ingress, and a release-specific migration Job. It deliberately emits
+no Secret values, persistent volume, PostgreSQL, Redis, or object-store
+deployment. Export is rejected unless both release images use immutable
+`image@sha256:...` references.
+
+Create the named Secret through the cluster's secret-management path. The
+command prints the exact required key contract. Review the generated
+`FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network
+before production use.
+
+When an external S3 endpoint is signed by a private CA, create the optional CA
+Secret with a `ca.crt` key and pass `--s3-ca-secret-name`. The renderer mounts
+that Secret read-only and sets `AWS_CA_BUNDLE` for API, worker, scheduler,
+migration and database-wait containers. It does not disable certificate
+verification or replace the WebUI trust store.
+
+The generated containers run as non-root with a read-only root filesystem and
+an ephemeral `/tmp`. Celery Beat keeps its replaceable schedule database there;
+durable schedule definitions remain in shared state. The WebUI resolves its
+configured API Service when the container starts, so Kubernetes deployments do
+not inherit the Compose-only `load-balancer` hostname. Runtime Deployments wait
+for the exact dependency-resolved database migration heads before starting.
+The API exposes `/health/ready`, which fails while that API node is draining or
+cannot prove its runtime-coordination heartbeat.
+
+Replicated API, WebUI, and worker Deployments use a hard hostname-spread
+constraint scoped to the current pod-template hash. A rollout therefore keeps
+each replica set distributed across independently schedulable nodes instead of
+allowing all replacement pods to settle on one node after the old set exits.
+
+## Runtime Coordination
+
+Each API and worker incarnation registers in PostgreSQL with its role, software
+version, module-composition hash, queue set, and heartbeat. Ops shows active,
+draining, stopped, and stale nodes and compares active counts with configured
+replica expectations.
+
+An operator may request or cancel drain from Ops:
+
+- API readiness becomes unavailable on the next heartbeat so the load balancer
+  stops assigning new requests.
+- A worker stops consuming its configured queues and may finish work already
+  claimed by that process.
+- A stale process incarnation cannot overwrite a replacement incarnation's
+  heartbeat.
+- A coordination outage removes API readiness and cancels worker consumers;
+  the existing incarnation must heartbeat successfully before either resumes.
+
+Singleton work uses PostgreSQL-backed leases with monotonically increasing
+fencing tokens. The generated scheduler runs Celery beat through
+`govoplan_core.commands.fenced_run`; loss of its lease terminates the child and
+returns a distinct failure code. A fenced business operation must validate the
+same lease token immediately before committing its effect.
+
+## Release Ordering
+
+Use this order for every multi-replica rollout:
+
+1. Verify immutable image identities, module composition, external state
+   reachability, backup evidence, and the generated plan.
+2. Drain application replicas when the migration compatibility declaration
+   requires it.
+3. Run the release-specific migration Job exactly once. PostgreSQL advisory
+   locking serializes all Core and module migration tasks across competing
+   deployment jobs.
+4. Let runtime init containers run `wait_for_database`. They wait for exact
+   configured Alembic heads and never mutate schema.
+5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
+6. Verify runtime composition, expected replica counts, queue consumers,
+   object-storage round trips, and recovery status in Ops.
+
+Applying the complete generated manifest is fail-closed: runtime pods remain in
+their init phase until the migration Job reaches the expected heads. A second
+release may be submitted concurrently, but advisory locking prevents concurrent
+schema mutation and each release has a distinct migration Job name.
+
+## Storage Trust Boundary
+
+Installer-managed Garage uses its exact generated endpoint. An arbitrary
+external S3 endpoint is accepted only when the deployment explicitly sets
+`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS
+origin without embedded credentials, query, fragment, or path. This is an
+operator trust declaration, not a user-controlled connector bypass. Operators
+remain responsible for DNS, certificate, network-egress, bucket-policy,
+versioning, and lifecycle controls.
+
+## Capacity
+
+- Set `GOVOPLAN_DB_CONNECTION_LIMIT` to the PostgreSQL role's effective
+  connection limit. The Kubernetes export reserves
+  `GOVOPLAN_DB_CONNECTION_RESERVE` connections and rejects a topology whose
+  calculated rolling-update peak would exceed the remainder. The calculation
+  includes API pools, every Celery parent and prefork child, the scheduler,
+  migration, and one surge replica per deployment. Role-specific pool and
+  overflow values are emitted into each workload rather than inherited from one
+  unconstrained global default.
+- Scale workers by queue, with upper bounds based on external provider limits.
+  `GOVOPLAN_WORKER_POOLS` may contain a JSON list of exact queue owners, for
+  example:
+
+  ```json
+  [
+    {"name":"delivery","queues":["send_email","append_sent"],"replicas":2,"concurrency":2},
+    {"name":"platform","queues":["events","workflow","default"],"replicas":2,"concurrency":2}
+  ]
+  ```
+
+  Pool replica totals must equal `replicas.worker`, and the pools must cover
+  `CELERY_QUEUES` exactly without duplicate ownership. Each pool receives its
+  own Deployment, disruption budget, topology-spread selector, runtime identity,
+  and declared concurrency.
+- Keep one fenced scheduler rather than load-balancing schedulers.
+- Increase WebUI replicas for asset/proxy capacity.
+- Measure request latency, database query time and locks, active connections,
+  queue age, retry rate, storage latency, and provider throttling before adding
+  replicas.
+
+Workers compete for Redis-backed work and are not placed behind a load balancer.
+SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often
+hit external-system limits before host CPU is exhausted.
+
+## What This Does Not Claim
+
+The implemented contract provides stateless runtime placement, shared artifact
+access, node visibility, drain controls, migration serialization, and scheduler
+fencing. It does not by itself provide:
+
+- a highly available PostgreSQL, Redis, or object-store deployment;
+- automatic PostgreSQL/object backup creation or point-in-time recovery;
+- autoscaling policy;
+- central logs, metrics, traces, or alert routing;
+- certificate portability between independently managed ingress providers;
+- automatic reconciliation of every possible module side effect;
+- a service-level availability guarantee.
+
+The deployer verifies and gates migrations on signed coordinated backup and
+isolated-restore evidence, but backup capture and restoration remain owned by
+the selected state-service providers. Before claiming high
+availability, drill replica loss, rolling replacement, session continuity, job
+redelivery, scheduler failover, migration exclusion, object-store outage, and a
+coordinated database/object/key restore. Recovery rules and evidence are
+defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
+
+## Worker Delivery Evidence
+
+The module-matrix workflow runs `tools/checks/worker-runtime-drill.py` against a
+real isolated Redis database. The drill starts supervised Celery worker
+processes and records four guarantees without accessing tenant data:
+
+1. a task published through the broker is consumed exactly once;
+2. an application retry is delivered again and completes;
+3. warm `SIGTERM` lets an in-flight late-ack task complete before shutdown; and
+4. loss of a worker after task start causes the unacknowledged task to be
+   redelivered after the configured visibility timeout.
+
+Run the same drill with the release Python environment and target Redis before
+promoting a worker composition. Use a dedicated Redis database, retain the JSON
+evidence, and set `CELERY_VISIBILITY_TIMEOUT_SECONDS` above the longest supported
+business-task duration. The short visibility timeout used by CI is an isolated
+test setting, not a production recommendation.
+
+```bash
+GOVOPLAN_WORKER_DRILL_REDIS_URL=redis://redis.example.test:6379/15 \
+  .venv/bin/python tools/checks/worker-runtime-drill.py \
+  --output evidence/worker-runtime.json
+```
+
+## Live Multi-Host Evidence
+
+After deploying a pinned release on at least two Kubernetes nodes, create an API
+key with Ops read scope and run:
+
+```bash
+export GOVOPLAN_OPS_API_KEY='...'
+python tools/deployment/govoplan-deploy.py verify-kubernetes \
+  --directory /srv/govoplan/installation \
+  --namespace govoplan
+```
+
+The command fails unless API and WebUI pods are ready on at least two nodes,
+all rendered deployments are available, Ops reports a consistent release and
+module composition, every declared worker queue is served, and the calculated
+database peak remains below its budget. It writes a private, sanitized JSON
+record under the installation evidence directory and never retains the API key.
+
+Use `--exercise-api-pod-loss` in an approved drill window to delete one API pod,
+observe the public readiness path continuously, and record its replacement.
+Generated API workloads use a ten-second pre-stop drain so Kubernetes can remove
+the terminating endpoint from ingress and service routing before Uvicorn exits.
+Do not remove or shorten this drain without repeating the public-path pod-loss
+test against the target ingress controller and network implementation.
+This proves the bounded stateless-node-loss slice only. Session continuity,
+accepted-job redelivery, state-service failover, and coordinated restore remain
+separate target exercises whose signed evidence is governed by
+`docs/operations/TARGET_MATURITY_EVIDENCE_RUNBOOK.md` and GovOPlaN #37.

docs/operations/SECURITY_AUDIT.md

@@ -0,0 +1,218 @@
+# Security Audit Toolchain
+
+GovOPlaN uses a free/open-source-first audit toolchain that can run locally,
+inside a container, and in Gitea Actions.
+
+## Tools
+
+- Semgrep: multi-language SAST, with GovOPlaN-specific local rules plus
+  explicit public registry rulesets in CI/full runs.
+- Bandit: Python AST security checks.
+- Ruff `S` rules: fast flake8-bandit-compatible Python security linting.
+- Gitleaks: committed-secret scanning.
+- Trivy: filesystem dependency, secret, and misconfiguration scanning.
+- pip-audit and npm audit: package vulnerability scanning from dependency
+  manifests/locks.
+- OSV-Scanner: recursive dependency vulnerability scan in full mode.
+- jscpd: duplicated-code reports in full mode.
+- Radon/Xenon: Python complexity reports and thresholds in full mode.
+
+## Local Usage
+
+Build and run the toolbox from this repository:
+
+```bash
+cd /mnt/DATA/git/govoplan
+tools/checks/security-audit/run.sh --mode ci --scope current
+```
+
+Scan all sibling GovOPlaN repositories under `/mnt/DATA/git`:
+
+```bash
+cd /mnt/DATA/git/govoplan
+tools/checks/security-audit/run.sh --mode full --scope govoplan
+```
+
+When invoked from a Flatpak development environment without a sandbox-local
+Docker CLI, the wrapper automatically uses `flatpak-spawn --host docker`. The
+host account must still be allowed to open the Docker daemon socket. For a
+conventional rootful installation this commonly means membership in the
+`docker` group followed by a complete logout/login; that membership is
+root-equivalent, so rootless Docker is preferable where the deployment policy
+requires a smaller privilege boundary.
+
+Reports are written to `audit-reports/`, which is intentionally ignored by git.
+Each run records tool versions, report checksums, and start/end repository
+revision plus worktree fingerprints. A repository change during scanning makes
+the run fail so a mixed code snapshot cannot be reported as a valid audit.
+Step exit codes are recorded separately from findings: report-only mode may
+accept findings, but scanner execution errors and malformed JSON/SARIF reports
+always fail the run. The manifest lists the expected, present, and missing
+reports for that invocation. Validation and checksums use that explicit set, so
+reusing a report directory cannot make stale output look like part of a new run.
+It also contains `coverage_status` and structured `scanner_coverage` entries.
+Every required scanner is recorded as `no-findings`, `findings`,
+`scanner-failure`, or `skipped`; this makes an incomplete local run visible
+without treating it as a clean audit.
+
+The wrapper tags the toolbox image by a fingerprint of the Dockerfile,
+`requirements-audit.txt`, and the Semgrep smoke-test inputs. If those inputs have
+not changed, subsequent runs reuse the existing local image instead of
+reinstalling all tools. The stable alias is `govoplan/security-audit:local`
+unless `SECURITY_AUDIT_IMAGE` is set.
+
+Semgrep is installed separately in the toolbox image because current Semgrep
+packages pin affected Click and MCP versions. The image upgrades both after
+Semgrep installation, runs an actual local-rule scan, and audits the final
+toolbox Python environment during the image build. The compatibility releases
+are pinned exactly to keep the tested override reproducible. Remove either
+compatibility override only after Semgrep's own dependency bounds include a
+fixed version.
+
+Force a cached rebuild:
+
+```bash
+tools/checks/security-audit/run.sh --mode ci --scope current --rebuild
+```
+
+Refresh from upstream base images and package ranges:
+
+```bash
+tools/checks/security-audit/run.sh --mode ci --scope current --update
+```
+
+Build or refresh the toolbox without running an audit:
+
+```bash
+tools/checks/security-audit/run.sh --mode quick --scope current --build-only
+tools/checks/security-audit/run.sh --mode quick --scope current --update --build-only
+```
+
+## Modes
+
+- `quick`: local Semgrep rules, Bandit, Ruff security rules, Gitleaks.
+- `ci`: quick plus Semgrep public registry rulesets, Trivy, pip-audit, npm audit.
+- `full`: ci plus OSV-Scanner, jscpd, Radon, and Xenon.
+
+The Gitea workflow uses `full` mode so its coverage contract includes every
+scanner above. Missing scanners fail strict runs and all Actions runs, even
+while actual findings remain report-only. Local report-only runs may finish
+with missing tools for diagnostics, but their manifest is marked
+`coverage_status: incomplete`.
+
+Semgrep and Trivy are invoked with finding-sensitive exit codes. Their exit 1
+is therefore a finding under the wrapper contract; higher exit codes, missing
+output, invalid JSON/SARIF, and scanner error payloads are execution failures.
+
+Bandit and Ruff security reports are split by source kind. Production code under
+`src/` is written to `bandit.json` and `ruff-security.json` and controls strict
+mode. Test code under `tests/` is still scanned for visibility, but its findings
+are written separately to `bandit-tests.json` and `ruff-security-tests.json` so
+fixture passwords, assertions, and temporary paths do not hide the production
+baseline.
+
+## Gating
+
+The Gitea workflow currently runs findings in report-only mode:
+
+```bash
+SECURITY_AUDIT_FAIL_ON_FINDINGS=0
+```
+
+This avoids blocking every push while the first baseline is reviewed. After the
+baseline is clean, switch the workflow to:
+
+```bash
+SECURITY_AUDIT_FAIL_ON_FINDINGS=1
+```
+
+or run locally with:
+
+```bash
+tools/checks/security-audit/run.sh --mode full --scope govoplan --strict
+```
+
+## Audit Burndown Workflow
+
+Treat Gitea issues as the active audit state. A full GovOPlaN audit should
+produce one tracker issue in `GovOPlaN/govoplan` and child issues in the
+repository that owns each fix.
+
+Use the tracker issue for:
+
+- report path, timestamp, mode, and scope
+- scanner counts by category
+- clean scanners and resolved findings
+- links to child issues
+- the next audit run target
+
+Use child issues for concrete code or configuration changes. Apply
+`source/security-audit` to every issue created from a report, then add the
+most specific audit label:
+
+- `audit/quick-fix`: narrow direct remediation
+- `audit/structural`: behavior or architecture needs review
+- `audit/complexity`: Radon/Xenon maintainability finding
+- `audit/duplication`: jscpd duplication finding
+- `audit/false-positive`: reviewed narrow false positive or accepted risk
+- `audit/needs-design`: human decision needed before implementation
+
+Keep active implementation status in issues instead of committing generated
+audit reports. `audit-reports/` is ignored; quote the report directory and the
+important scanner counts in the tracker issue.
+
+The jscpd step is intentionally scoped to application and test source. It
+excludes documentation snippets, package manifests, generated translations,
+public SVG assets and catalog output, workflow YAML, declarative backend schema
+JSON, the generated migration baseline, and mirrored development migration
+directories because those reports produce metadata or generated-source
+repetition rather than actionable source duplication. Keep exclusions narrow
+and create child issues for source-code clusters that cross module ownership or
+make behavior harder to change safely.
+
+The 2026-08-02 full-workspace baseline covered 64 repositories and reported
+1.82% duplicated lines before those generated-source exclusions. The reviewed
+high-value clusters were catalog acceptance persistence, release publication
+result assembly, and local WebUI JSON mutation wrappers. Similar Dataflow and
+Workflow graph/governance code remains independently owned until its shared
+contract is stable enough for Core; a raw similarity score is not grounds for a
+module-to-module dependency.
+
+## Image Freshness
+
+The regular `Security Audit` workflow reuses the fingerprinted toolbox image
+when the Docker daemon is persistent, which is the normal case for the
+self-hosted Gitea runner using the host Docker socket. Trusted push, schedule,
+and manual runs scan all registered repositories; authenticated SSH is used
+only for the private website repository. The wrapper inspects the Actions job
+mount table and forwards only the narrowest writable mount covering the audit
+scope; it never inherits the job's Docker socket or unrelated runner mounts.
+Pull-request audit runs stay disabled while the audit runner exposes its host
+Docker socket: PR-controlled audit code must run on a disposable or rootless
+runner without host-socket access. The separate
+`Security Audit Toolbox Update` workflow runs weekly with
+`SECURITY_AUDIT_UPDATE=1`; it pulls current base images and re-resolves the
+allowed tool version ranges into a refreshed local image.
+
+## Direct Host Usage
+
+The container is the recommended path. For direct host usage, install the Python
+tools first:
+
+```bash
+cd /mnt/DATA/git/govoplan
+python -m venv .venv
+./.venv/bin/python -m pip install -r requirements-audit.txt
+./.venv/bin/python -m pip install 'semgrep>=1.140,<2'
+./.venv/bin/python -m pip install --upgrade --no-deps 'click==8.3.3'
+./.venv/bin/python -m pip install --upgrade --no-deps 'mcp==1.28.1'
+./.venv/bin/semgrep scan --metrics=off --config tools/checks/security-audit/semgrep-govoplan.yml tools/checks/check-version-alignment.py
+./.venv/bin/pip-audit --progress-spinner off
+```
+
+Then install the non-Python tools (`gitleaks`, `trivy`, `osv-scanner`, `jscpd`)
+through the host package manager or vendor instructions and run:
+
+```bash
+tools/checks/check-security-audit.sh --mode quick --scope current
+```

docs/operations/TARGET_MATURITY_EVIDENCE_RUNBOOK.md

@@ -0,0 +1,157 @@
+# Target Maturity Evidence Runbook
+
+For authority-key generation, container isolation and the concrete inputs that
+must be supplied by the target owner and independent production approver, see
+[`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
+
+This runbook turns retained target-environment results into a sanitized,
+signed GovOPlaN capability-fit proof. It does not make a deployment suitable,
+certified, supported, or production-approved by itself. The proof records what
+independent authorities assessed against one exact installed release.
+
+## Roles and custody
+
+Use separate trust domains for release signing, installation receipts,
+boundary assessment, and production approval. A private proof key must be
+provisioned outside the assessed application and its matching public key must
+already exist in a separately managed
+`capability-fit-proof-authority-keyring.schema.json` document. Do not store
+private keys, raw reports, credentials, personal data, backup material, or
+target endpoints in Git.
+
+Each authority key lists only the scopes that role may attest. At least one
+supplied signing key must cover every claim, and the issuer rejects a key that:
+
+- is absent, inactive, expired, or revoked in the authority keyring;
+- expires before the proof;
+- does not match its independently provisioned public key;
+- reuses release-catalog or installer-authority key material.
+
+## Target run
+
+Install one pinned catalog release and issue its installed-composition receipt
+with `tools/assessments/installer-receipt.py`. Exercise the actual target
+topology, including:
+
+- PostgreSQL and Redis as shared state services;
+- shared S3-compatible object storage;
+- at least two stateless API replicas and the intended worker topology;
+- fenced singleton work, ingress, certificates, proxy headers, and the real
+  network/trust boundary;
+- provider health and freshness for every provider required by the product;
+- monitoring, alerting, failure response, accessibility, privacy, and security
+  controls;
+- backup, isolated restore, failed-deployment rollback, and forward recovery.
+
+For the recovery claim, retain the observed recovery point, measured RPO and
+RTO, database/object-store consistency result, and semantic reconstruction of
+the institutional and Service/Form reference journeys. Measure RPO from the
+last acknowledged durable effect that survives recovery and RTO until service
+health plus semantic reconstruction pass. Failed runs are evidence too and
+must use a negative result.
+
+The private reports stay in the approved evidence store. Give each report an
+opaque artifact ID and each evaluated control a versioned opaque control ID.
+
+## Private claim manifest
+
+Create a private manifest conforming to
+`capability-fit-boundary-run.schema.json`. Relative artifact paths resolve from
+the manifest directory. Paths are read and hashed by the issuer and are never
+copied into the signed output.
+
+```json
+{
+  "$schema": "./capability-fit-boundary-run.schema.json",
+  "schema_version": "0.1.0",
+  "evidence_kind": "govoplan.capability-fit-boundary-run",
+  "proof_id": "target:production:20260802",
+  "expires_at": "2026-09-01T00:00:00Z",
+  "claims": [
+    {
+      "scope": "target_environment",
+      "result": "passed",
+      "control_ids": ["topology:shared-state-v1"],
+      "artifacts": [
+        {"artifact_id": "target:run-20260802", "path": "private/target.json"}
+      ]
+    },
+    {
+      "scope": "recovery",
+      "result": "passed",
+      "control_ids": ["recovery:restore-rollback-v1"],
+      "artifacts": [
+        {"artifact_id": "recovery:run-20260802", "path": "private/recovery.json"}
+      ]
+    }
+  ]
+}
+```
+
+Reference readiness needs positive `target_environment`, `accessibility`,
+`privacy`, `security`, `operations`, and `recovery` claims. Provider acceptance
+and production approval are separate scopes. A production-approval authority
+must not approve its own unreviewed target run.
+
+## Issue and verify
+
+Issue only while the signed installer observation is current. Repeat
+`--signing-key` when multiple independent roles are needed. The command first
+verifies the catalog, independent catalog trust root, exact installed payload,
+installer receipt, and installer authority. It then hashes artifacts, signs the
+sanitized proof, verifies it immediately, and writes both proof and review with
+atomic private-file permissions.
+
+```bash
+./.venv/bin/python tools/assessments/boundary-evidence.py \
+  --assessment /srv/govoplan/assessment.json \
+  --catalog /srv/govoplan/catalogs/stable.json \
+  --keyring /srv/govoplan/catalogs/keyring.json \
+  --trusted-keyring /srv/govoplan/trust/catalog-keyring.json \
+  --installed-evidence /srv/govoplan/evidence/installed.json \
+  --installer-receipt /srv/govoplan/evidence/installer-receipt.json \
+  --installer-authority-keyring /srv/govoplan/trust/installer-authorities.json \
+  --claims /srv/govoplan/evidence/private/target-run.json \
+  --authority-keyring /srv/govoplan/trust/proof-authorities.json \
+  --signing-key authority-target=/run/secrets/target-proof-ed25519.pem \
+  --output /srv/govoplan/evidence/target-proof.json \
+  --review-output /srv/govoplan/evidence/target-proof-review.json
+```
+
+Add `--expected-external-provider-subject provider-production` when the claim
+manifest contains `external_providers`. This value is an opaque deployment ID,
+not a URL or credential.
+
+## Promotion gate
+
+The general verifier can now be made admission-enforcing. These switches return
+a blocking exit status when a required claim is absent, expired, negative,
+revoked, or bound to another assessment, release, installation, or subject:
+
+```bash
+./.venv/bin/python tools/assessments/capability-fit.py \
+  --catalog /srv/govoplan/catalogs/stable.json \
+  --keyring /srv/govoplan/catalogs/keyring.json \
+  --trusted-keyring /srv/govoplan/trust/catalog-keyring.json \
+  --installed-evidence /srv/govoplan/evidence/installed.json \
+  --installer-receipt /srv/govoplan/evidence/installer-receipt.json \
+  --installer-authority-keyring /srv/govoplan/trust/installer-authorities.json \
+  --boundary-evidence /srv/govoplan/evidence/target-proof.json \
+  --boundary-authority-keyring /srv/govoplan/trust/proof-authorities.json \
+  --require-reference-readiness \
+  --require-production-approval \
+  --output /srv/govoplan/evidence/admission-review.json
+```
+
+Use `--require-external-provider-proof` as well when the promoted product
+requires an external provider. Live admission must not use
+`--verification-time`; that switch is only for clearly labelled historical
+review.
+
+## Renewal and failure
+
+Renew evidence after release, installed composition, deployment, control, or
+provider changes and before expiry. Revoke an authority key immediately after
+custody loss and rerun the affected assessment with a new independent key.
+Never copy a previous positive claim to a new release. Preserve negative and
+superseded receipts according to the approved evidence-retention policy.