# Scaling And Multi-Host Deployment ## Implemented Contract GovOPlaN now supports a stateless application tier backed by logically shared state services. The runtime roles are independently replaceable API, WebUI, worker, and scheduler processes. Every replica in one installation must use the same immutable release composition and the same: - `GOVOPLAN_INSTALLATION_ID`; - PostgreSQL database; - Redis broker and coordination service; - `MASTER_KEY_B64` and deployment secret references; - enabled-module graph; - S3-compatible object-storage namespace. The application tier must not use node-local durable business data in a multi-host deployment. Files owns managed file metadata while Core provides the storage-backend contract. Campaign build artifacts are stored under opaque object keys and workers read those objects from the shared backend. Temporary build and materialization directories may remain node-local because they are discardable. Core validates three explicit state profiles: | Profile | Supported shape | Storage rule | | --- | --- | --- | | `local` | One API and one worker process for development | Local filesystem permitted. | | `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. | | `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. | `shared` also requires a stable installation identifier. Module package mutation is blocked in this profile: build and verify a new immutable release, then roll the complete cluster to it. ## Same-Host Compose The generated Compose bundle provides: ```text client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas API/worker/scheduler -> PostgreSQL -> Redis -> local volume, managed Garage, or external S3 ``` HAProxy discovers Compose replicas through Docker DNS and performs health-aware balancing without mounting the Docker socket. This improves concurrency and permits process replacement, but the Docker host and installer-managed stateful services remain single failure domains. Generated Compose therefore declares the `host-shared` state profile even when its shared storage happens to be an external S3 service. Its API backend checks `/health/ready`, so drain or coordination loss removes a replica from rotation. Container, load-balancer, and Kubernetes probes send the configured public host explicitly, keeping readiness compatible with strict trusted-host validation. Managed Garage is a convenient single-node S3-compatible service. It is not a multi-host storage cluster. Use an independently operated Garage cluster or another S3-compatible service for the `shared` profile. ## Kubernetes Export The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL, Redis, and S3 are all external: ```sh python tools/deployment/govoplan-deploy.py render-kubernetes \ --directory /srv/govoplan/default \ --namespace govoplan \ --secret-name govoplan-runtime \ --tls-secret-name govoplan-tls \ --ingress-class-name nginx \ --output /srv/govoplan/default/kubernetes.json ``` The export contains a Namespace, tokenless ServiceAccount, non-secret ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption budgets, Ingress, and a release-specific migration Job. It deliberately emits no Secret values, persistent volume, PostgreSQL, Redis, or object-store deployment. Export is rejected unless both release images use immutable `image@sha256:...` references. Create the named Secret through the cluster's secret-management path. The command prints the exact required key contract. Review the generated `FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network before production use. The generated containers run as non-root with a read-only root filesystem and an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database migration heads before starting. The API exposes `/health/ready`, which fails while that API node is draining or cannot prove its runtime-coordination heartbeat. ## Runtime Coordination Each API and worker incarnation registers in PostgreSQL with its role, software version, module-composition hash, queue set, and heartbeat. Ops shows active, draining, stopped, and stale nodes and compares active counts with configured replica expectations. An operator may request or cancel drain from Ops: - API readiness becomes unavailable on the next heartbeat so the load balancer stops assigning new requests. - A worker stops consuming its configured queues and may finish work already claimed by that process. - A stale process incarnation cannot overwrite a replacement incarnation's heartbeat. - A coordination outage removes API readiness and cancels worker consumers; the existing incarnation must heartbeat successfully before either resumes. Singleton work uses PostgreSQL-backed leases with monotonically increasing fencing tokens. The generated scheduler runs Celery beat through `govoplan_core.commands.fenced_run`; loss of its lease terminates the child and returns a distinct failure code. A fenced business operation must validate the same lease token immediately before committing its effect. ## Release Ordering Use this order for every multi-replica rollout: 1. Verify immutable image identities, module composition, external state reachability, backup evidence, and the generated plan. 2. Drain application replicas when the migration compatibility declaration requires it. 3. Run the release-specific migration Job exactly once. PostgreSQL advisory locking serializes all Core and module migration tasks across competing deployment jobs. 4. Let runtime init containers run `wait_for_database`. They wait for exact configured Alembic heads and never mutate schema. 5. Roll API, workers, scheduler, and WebUI using health-aware replacement. 6. Verify runtime composition, expected replica counts, queue consumers, object-storage round trips, and recovery status in Ops. Applying the complete generated manifest is fail-closed: runtime pods remain in their init phase until the migration Job reaches the expected heads. A second release may be submitted concurrently, but advisory locking prevents concurrent schema mutation and each release has a distinct migration Job name. ## Storage Trust Boundary Installer-managed Garage uses its exact generated endpoint. An arbitrary external S3 endpoint is accepted only when the deployment explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS origin without embedded credentials, query, fragment, or path. This is an operator trust declaration, not a user-controlled connector bypass. Operators remain responsible for DNS, certificate, network-egress, bucket-policy, versioning, and lifecycle controls. ## Capacity - Scale API replicas only within the PostgreSQL connection budget. - Scale workers by queue, with upper bounds based on external provider limits. - Keep one fenced scheduler rather than load-balancing schedulers. - Increase WebUI replicas for asset/proxy capacity. - Measure request latency, database query time and locks, active connections, queue age, retry rate, storage latency, and provider throttling before adding replicas. Workers compete for Redis-backed work and are not placed behind a load balancer. SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often hit external-system limits before host CPU is exhausted. ## What This Does Not Claim The implemented contract provides stateless runtime placement, shared artifact access, node visibility, drain controls, migration serialization, and scheduler fencing. It does not by itself provide: - a highly available PostgreSQL, Redis, or object-store deployment; - automatic PostgreSQL backup, point-in-time recovery, or restore verification; - autoscaling policy; - central logs, metrics, traces, or alert routing; - managed ingress certificates; - automatic reconciliation of every possible module side effect; - a service-level availability guarantee. Those are deployment and module-adoption requirements. Before claiming high availability, drill replica loss, rolling replacement, session continuity, job redelivery, scheduler failover, migration exclusion, object-store outage, and a coordinated database/object/key restore. Recovery rules and evidence are defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).