# Scaling And Multi-Host Deployment For the exact external handoff, least-privilege collector permissions and live two-node acceptance procedure, see [`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md). For a reproducible local or multi-hypervisor libvirt/K3s target, use [`KUBERNETES_TEST_LAB.md`](KUBERNETES_TEST_LAB.md). ## Implemented Contract GovOPlaN now supports a stateless application tier backed by logically shared state services. The runtime roles are independently replaceable API, WebUI, worker, and scheduler processes. Every replica in one installation must use the same immutable release composition and the same: - `GOVOPLAN_INSTALLATION_ID`; - PostgreSQL database; - Redis broker and coordination service; - `MASTER_KEY_B64` and deployment secret references; - enabled-module graph; - S3-compatible object-storage namespace. The application tier must not use node-local durable business data in a multi-host deployment. Files owns managed file metadata while Core provides the storage-backend contract. Campaign build artifacts are stored under opaque object keys and workers read those objects from the shared backend. Temporary build and materialization directories may remain node-local because they are discardable. Core validates three explicit state profiles: | Profile | Supported shape | Storage rule | | --- | --- | --- | | `local` | One API and one worker process for development | Local filesystem permitted. | | `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. | | `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. | `shared` also requires a stable installation identifier. Module package mutation is blocked in this profile: build and verify a new immutable release, then roll the complete cluster to it. ## Same-Host Compose The generated Compose bundle provides: ```text client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas API/worker/scheduler -> PostgreSQL -> Redis -> local volume, managed Garage, or external S3 ``` HAProxy discovers Compose replicas through Docker DNS and performs health-aware balancing without mounting the Docker socket. This improves concurrency and permits process replacement, but the Docker host and installer-managed stateful services remain single failure domains. Generated Compose therefore declares the `host-shared` state profile even when its shared storage happens to be an external S3 service. Its API backend checks `/health/ready`, so drain or coordination loss removes a replica from rotation. Container, load-balancer, and Kubernetes probes send the configured public host explicitly, keeping readiness compatible with strict trusted-host validation. Managed Garage is a convenient single-node S3-compatible service. It is not a multi-host storage cluster. Use an independently operated Garage cluster or another S3-compatible service for the `shared` profile. ## Kubernetes Export The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL, Redis, and S3 are all external: ```sh python tools/deployment/govoplan-deploy.py render-kubernetes \ --directory /srv/govoplan/default \ --namespace govoplan \ --secret-name govoplan-runtime \ --tls-secret-name govoplan-tls \ --s3-ca-secret-name govoplan-s3-ca \ --ingress-class-name nginx \ --output /srv/govoplan/default/kubernetes.json ``` The export contains a Namespace, tokenless ServiceAccount, non-secret ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption budgets, Ingress, and a release-specific migration Job. It deliberately emits no Secret values, persistent volume, PostgreSQL, Redis, or object-store deployment. Export is rejected unless both release images use immutable `image@sha256:...` references. Create the named Secret through the cluster's secret-management path. The command prints the exact required key contract. Review the generated `FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network before production use. When an external S3 endpoint is signed by a private CA, create the optional CA Secret with a `ca.crt` key and pass `--s3-ca-secret-name`. The renderer mounts that Secret read-only and sets `AWS_CA_BUNDLE` for API, worker, scheduler, migration and database-wait containers. It does not disable certificate verification or replace the WebUI trust store. The generated containers run as non-root with a read-only root filesystem and an ephemeral `/tmp`. Celery Beat keeps its replaceable schedule database there; durable schedule definitions remain in shared state. The WebUI resolves its configured API Service when the container starts, so Kubernetes deployments do not inherit the Compose-only `load-balancer` hostname. Runtime Deployments wait for the exact dependency-resolved database migration heads before starting. The API exposes `/health/ready`, which fails while that API node is draining or cannot prove its runtime-coordination heartbeat. Replicated API, WebUI, and worker Deployments use a hard hostname-spread constraint scoped to the current pod-template hash. A rollout therefore keeps each replica set distributed across independently schedulable nodes instead of allowing all replacement pods to settle on one node after the old set exits. ## Runtime Coordination Each API and worker incarnation registers in PostgreSQL with its role, software version, module-composition hash, queue set, and heartbeat. Ops shows active, draining, stopped, and stale nodes and compares active counts with configured replica expectations. An operator may request or cancel drain from Ops: - API readiness becomes unavailable on the next heartbeat so the load balancer stops assigning new requests. - A worker stops consuming its configured queues and may finish work already claimed by that process. - A stale process incarnation cannot overwrite a replacement incarnation's heartbeat. - A coordination outage removes API readiness and cancels worker consumers; the existing incarnation must heartbeat successfully before either resumes. Singleton work uses PostgreSQL-backed leases with monotonically increasing fencing tokens. The generated scheduler runs Celery beat through `govoplan_core.commands.fenced_run`; loss of its lease terminates the child and returns a distinct failure code. A fenced business operation must validate the same lease token immediately before committing its effect. ## Release Ordering Use this order for every multi-replica rollout: 1. Verify immutable image identities, module composition, external state reachability, backup evidence, and the generated plan. 2. Drain application replicas when the migration compatibility declaration requires it. 3. Run the release-specific migration Job exactly once. PostgreSQL advisory locking serializes all Core and module migration tasks across competing deployment jobs. 4. Let runtime init containers run `wait_for_database`. They wait for exact configured Alembic heads and never mutate schema. 5. Roll API, workers, scheduler, and WebUI using health-aware replacement. 6. Verify runtime composition, expected replica counts, queue consumers, object-storage round trips, and recovery status in Ops. Applying the complete generated manifest is fail-closed: runtime pods remain in their init phase until the migration Job reaches the expected heads. A second release may be submitted concurrently, but advisory locking prevents concurrent schema mutation and each release has a distinct migration Job name. ## Storage Trust Boundary Installer-managed Garage uses its exact generated endpoint. An arbitrary external S3 endpoint is accepted only when the deployment explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS origin without embedded credentials, query, fragment, or path. This is an operator trust declaration, not a user-controlled connector bypass. Operators remain responsible for DNS, certificate, network-egress, bucket-policy, versioning, and lifecycle controls. ## Capacity - Set `GOVOPLAN_DB_CONNECTION_LIMIT` to the PostgreSQL role's effective connection limit. The Kubernetes export reserves `GOVOPLAN_DB_CONNECTION_RESERVE` connections and rejects a topology whose calculated rolling-update peak would exceed the remainder. The calculation includes API pools, every Celery parent and prefork child, the scheduler, migration, and one surge replica per deployment. Role-specific pool and overflow values are emitted into each workload rather than inherited from one unconstrained global default. - Scale workers by queue, with upper bounds based on external provider limits. `GOVOPLAN_WORKER_POOLS` may contain a JSON list of exact queue owners, for example: ```json [ {"name":"delivery","queues":["send_email","append_sent"],"replicas":2,"concurrency":2}, {"name":"platform","queues":["events","workflow","default"],"replicas":2,"concurrency":2} ] ``` Pool replica totals must equal `replicas.worker`, and the pools must cover `CELERY_QUEUES` exactly without duplicate ownership. Each pool receives its own Deployment, disruption budget, topology-spread selector, runtime identity, and declared concurrency. - Keep one fenced scheduler rather than load-balancing schedulers. - Increase WebUI replicas for asset/proxy capacity. - Measure request latency, database query time and locks, active connections, queue age, retry rate, storage latency, and provider throttling before adding replicas. Workers compete for Redis-backed work and are not placed behind a load balancer. SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often hit external-system limits before host CPU is exhausted. ## What This Does Not Claim The implemented contract provides stateless runtime placement, shared artifact access, node visibility, drain controls, migration serialization, and scheduler fencing. It does not by itself provide: - a highly available PostgreSQL, Redis, or object-store deployment; - automatic PostgreSQL/object backup creation or point-in-time recovery; - autoscaling policy; - central logs, metrics, traces, or alert routing; - certificate portability between independently managed ingress providers; - automatic reconciliation of every possible module side effect; - a service-level availability guarantee. The deployer verifies and gates migrations on signed coordinated backup and isolated-restore evidence, but backup capture and restoration remain owned by the selected state-service providers. Before claiming high availability, drill replica loss, rolling replacement, session continuity, job redelivery, scheduler failover, migration exclusion, object-store outage, and a coordinated database/object/key restore. Recovery rules and evidence are defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md). ## Worker Delivery Evidence The module-matrix workflow runs `tools/checks/worker-runtime-drill.py` against a real isolated Redis database. The drill starts supervised Celery worker processes and records four guarantees without accessing tenant data: 1. a task published through the broker is consumed exactly once; 2. an application retry is delivered again and completes; 3. warm `SIGTERM` lets an in-flight late-ack task complete before shutdown; and 4. loss of a worker after task start causes the unacknowledged task to be redelivered after the configured visibility timeout. Run the same drill with the release Python environment and target Redis before promoting a worker composition. Use a dedicated Redis database, retain the JSON evidence, and set `CELERY_VISIBILITY_TIMEOUT_SECONDS` above the longest supported business-task duration. The short visibility timeout used by CI is an isolated test setting, not a production recommendation. ```bash GOVOPLAN_WORKER_DRILL_REDIS_URL=redis://redis.example.test:6379/15 \ .venv/bin/python tools/checks/worker-runtime-drill.py \ --output evidence/worker-runtime.json ``` ## Live Multi-Host Evidence After deploying a pinned release on at least two Kubernetes nodes, create an API key with Ops read scope and run: ```bash export GOVOPLAN_OPS_API_KEY='...' python tools/deployment/govoplan-deploy.py verify-kubernetes \ --directory /srv/govoplan/installation \ --namespace govoplan ``` The command fails unless API and WebUI pods are ready on at least two nodes, all rendered deployments are available, Ops reports a consistent release and module composition, every declared worker queue is served, and the calculated database peak remains below its budget. It writes a private, sanitized JSON record under the installation evidence directory and never retains the API key. Use `--exercise-api-pod-loss` in an approved drill window to delete one API pod, observe the public readiness path continuously, and record its replacement. Generated API workloads use a ten-second pre-stop drain so Kubernetes can remove the terminating endpoint from ingress and service routing before Uvicorn exits. Do not remove or shorten this drain without repeating the public-path pod-loss test against the target ingress controller and network implementation. This proves the bounded stateless-node-loss slice only. Session continuity, accepted-job redelivery, state-service failover, and coordinated restore remain separate target exercises whose signed evidence is governed by `docs/operations/TARGET_MATURITY_EVIDENCE_RUNBOOK.md` and GovOPlaN #37.