govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

Scaling And Multi-Host Deployment

Implemented Contract

GovOPlaN now supports a stateless application tier backed by logically shared state services. The runtime roles are independently replaceable API, WebUI, worker, and scheduler processes. Every replica in one installation must use the same immutable release composition and the same:

• GOVOPLAN_INSTALLATION_ID;
• PostgreSQL database;
• Redis broker and coordination service;
• MASTER_KEY_B64 and deployment secret references;
• enabled-module graph;
• S3-compatible object-storage namespace.

The application tier must not use node-local durable business data in a multi-host deployment. Files owns managed file metadata while Core provides the storage-backend contract. Campaign build artifacts are stored under opaque object keys and workers read those objects from the shared backend. Temporary build and materialization directories may remain node-local because they are discardable.

Core validates three explicit state profiles:

Profile	Supported shape	Storage rule
local	One API and one worker process for development	Local filesystem permitted.
host-shared	Multiple processes on one Docker host	A shared host volume is permitted; PostgreSQL and Redis are required.
shared	Multiple independent hosts	PostgreSQL, Redis, and S3-compatible object storage are required.

shared also requires a stable installation identifier. Module package mutation is blocked in this profile: build and verify a new immutable release, then roll the complete cluster to it.

Same-Host Compose

The generated Compose bundle provides:

client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas

API/worker/scheduler -> PostgreSQL
                     -> Redis
                     -> local volume, managed Garage, or external S3

HAProxy discovers Compose replicas through Docker DNS and performs health-aware balancing without mounting the Docker socket. This improves concurrency and permits process replacement, but the Docker host and installer-managed stateful services remain single failure domains. Generated Compose therefore declares the host-shared state profile even when its shared storage happens to be an external S3 service. Its API backend checks /health/ready, so drain or coordination loss removes a replica from rotation. Container, load-balancer, and Kubernetes probes send the configured public host explicitly, keeping readiness compatible with strict trusted-host validation.

Managed Garage is a convenient single-node S3-compatible service. It is not a multi-host storage cluster. Use an independently operated Garage cluster or another S3-compatible service for the shared profile.

Kubernetes Export

The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL, Redis, and S3 are all external:

python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime \
  --tls-secret-name govoplan-tls \
  --ingress-class-name nginx \
  --output /srv/govoplan/default/kubernetes.json

The export contains a Namespace, tokenless ServiceAccount, non-secret ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption budgets, Ingress, and a release-specific migration Job. It deliberately emits no Secret values, persistent volume, PostgreSQL, Redis, or object-store deployment. Export is rejected unless both release images use immutable image@sha256:... references.

Create the named Secret through the cluster's secret-management path. The command prints the exact required key contract. Review the generated FORWARDED_ALLOW_IPS value and replace it with the exact ingress-proxy network before production use.

The generated containers run as non-root with a read-only root filesystem and an ephemeral /tmp. Runtime Deployments wait for the exact configured database migration heads before starting. The API exposes /health/ready, which fails while that API node is draining or cannot prove its runtime-coordination heartbeat.

Runtime Coordination

Each API and worker incarnation registers in PostgreSQL with its role, software version, module-composition hash, queue set, and heartbeat. Ops shows active, draining, stopped, and stale nodes and compares active counts with configured replica expectations.

An operator may request or cancel drain from Ops:

• API readiness becomes unavailable on the next heartbeat so the load balancer stops assigning new requests.
• A worker stops consuming its configured queues and may finish work already claimed by that process.
• A stale process incarnation cannot overwrite a replacement incarnation's heartbeat.
• A coordination outage removes API readiness and cancels worker consumers; the existing incarnation must heartbeat successfully before either resumes.

Singleton work uses PostgreSQL-backed leases with monotonically increasing fencing tokens. The generated scheduler runs Celery beat through govoplan_core.commands.fenced_run; loss of its lease terminates the child and returns a distinct failure code. A fenced business operation must validate the same lease token immediately before committing its effect.

Release Ordering

Use this order for every multi-replica rollout:

1. Verify immutable image identities, module composition, external state reachability, backup evidence, and the generated plan.
2. Drain application replicas when the migration compatibility declaration requires it.
3. Run the release-specific migration Job exactly once. PostgreSQL advisory locking serializes all Core and module migration tasks across competing deployment jobs.
4. Let runtime init containers run wait_for_database. They wait for exact configured Alembic heads and never mutate schema.
5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
6. Verify runtime composition, expected replica counts, queue consumers, object-storage round trips, and recovery status in Ops.

Applying the complete generated manifest is fail-closed: runtime pods remain in their init phase until the migration Job reaches the expected heads. A second release may be submitted concurrently, but advisory locking prevents concurrent schema mutation and each release has a distinct migration Job name.

Storage Trust Boundary

Installer-managed Garage uses its exact generated endpoint. An arbitrary external S3 endpoint is accepted only when the deployment explicitly sets FILE_STORAGE_S3_ENDPOINT_TRUSTED=true; that endpoint must be a clean HTTPS origin without embedded credentials, query, fragment, or path. This is an operator trust declaration, not a user-controlled connector bypass. Operators remain responsible for DNS, certificate, network-egress, bucket-policy, versioning, and lifecycle controls.

Capacity

• Scale API replicas only within the PostgreSQL connection budget.
• Scale workers by queue, with upper bounds based on external provider limits.
• Keep one fenced scheduler rather than load-balancing schedulers.
• Increase WebUI replicas for asset/proxy capacity.
• Measure request latency, database query time and locks, active connections, queue age, retry rate, storage latency, and provider throttling before adding replicas.

Workers compete for Redis-backed work and are not placed behind a load balancer. SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often hit external-system limits before host CPU is exhausted.

What This Does Not Claim

The implemented contract provides stateless runtime placement, shared artifact access, node visibility, drain controls, migration serialization, and scheduler fencing. It does not by itself provide:

• a highly available PostgreSQL, Redis, or object-store deployment;
• automatic PostgreSQL backup, point-in-time recovery, or restore verification;
• autoscaling policy;
• central logs, metrics, traces, or alert routing;
• managed ingress certificates;
• automatic reconciliation of every possible module side effect;
• a service-level availability guarantee.

Those are deployment and module-adoption requirements. Before claiming high availability, drill replica loss, rolling replacement, session continuity, job redelivery, scheduler failover, migration exclusion, object-store outage, and a coordinated database/object/key restore. Recovery rules and evidence are defined in Recovery And Rollback Guarantees.