govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

Scaling And Multi-Host Deployment

For the exact external handoff, least-privilege collector permissions and live two-node acceptance procedure, see PRODUCTION_TARGET_HANDOFF.md. For a reproducible local or multi-hypervisor libvirt/K3s target, use KUBERNETES_TEST_LAB.md.

Implemented Contract

GovOPlaN now supports a stateless application tier backed by logically shared state services. The runtime roles are independently replaceable API, WebUI, worker, and scheduler processes. Every replica in one installation must use the same immutable release composition and the same:

• GOVOPLAN_INSTALLATION_ID;
• PostgreSQL database;
• Redis broker and coordination service;
• MASTER_KEY_B64 and deployment secret references;
• enabled-module graph;
• S3-compatible object-storage namespace.

The application tier must not use node-local durable business data in a multi-host deployment. Files owns managed file metadata while Core provides the storage-backend contract. Campaign build artifacts are stored under opaque object keys and workers read those objects from the shared backend. Temporary build and materialization directories may remain node-local because they are discardable.

Core validates three explicit state profiles:

Profile	Supported shape	Storage rule
local	One API and one worker process for development	Local filesystem permitted.
host-shared	Multiple processes on one Docker host	A shared host volume is permitted; PostgreSQL and Redis are required.
shared	Multiple independent hosts	PostgreSQL, Redis, and S3-compatible object storage are required.

shared also requires a stable installation identifier. Module package mutation is blocked in this profile: build and verify a new immutable release, then roll the complete cluster to it.

Same-Host Compose

The generated Compose bundle provides:

client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas

API/worker/scheduler -> PostgreSQL
                     -> Redis
                     -> local volume, managed Garage, or external S3

HAProxy discovers Compose replicas through Docker DNS and performs health-aware balancing without mounting the Docker socket. This improves concurrency and permits process replacement, but the Docker host and installer-managed stateful services remain single failure domains. Generated Compose therefore declares the host-shared state profile even when its shared storage happens to be an external S3 service. Its API backend checks /health/ready, so drain or coordination loss removes a replica from rotation. Container, load-balancer, and Kubernetes probes send the configured public host explicitly, keeping readiness compatible with strict trusted-host validation.

Managed Garage is a convenient single-node S3-compatible service. It is not a multi-host storage cluster. Use an independently operated Garage cluster or another S3-compatible service for the shared profile.

Kubernetes Export

The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL, Redis, and S3 are all external:

python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime \
  --tls-secret-name govoplan-tls \
  --s3-ca-secret-name govoplan-s3-ca \
  --ingress-class-name nginx \
  --output /srv/govoplan/default/kubernetes.json

The export contains a Namespace, tokenless ServiceAccount, non-secret ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption budgets, Ingress, and a release-specific migration Job. It deliberately emits no Secret values, persistent volume, PostgreSQL, Redis, or object-store deployment. Export is rejected unless both release images use immutable image@sha256:... references.

Create the named Secret through the cluster's secret-management path. The command prints the exact required key contract. Review the generated FORWARDED_ALLOW_IPS value and replace it with the exact ingress-proxy network before production use.

When an external S3 endpoint is signed by a private CA, create the optional CA Secret with a ca.crt key and pass --s3-ca-secret-name. The renderer mounts that Secret read-only and sets AWS_CA_BUNDLE for API, worker, scheduler, migration and database-wait containers. It does not disable certificate verification or replace the WebUI trust store.

The generated containers run as non-root with a read-only root filesystem and an ephemeral /tmp. Celery Beat keeps its replaceable schedule database there; durable schedule definitions remain in shared state. The WebUI resolves its configured API Service when the container starts, so Kubernetes deployments do not inherit the Compose-only load-balancer hostname. Runtime Deployments wait for the exact dependency-resolved database migration heads before starting. The API exposes /health/ready, which fails while that API node is draining or cannot prove its runtime-coordination heartbeat.

Replicated API, WebUI, and worker Deployments use a hard hostname-spread constraint scoped to the current pod-template hash. A rollout therefore keeps each replica set distributed across independently schedulable nodes instead of allowing all replacement pods to settle on one node after the old set exits.

Runtime Coordination

Each API and worker incarnation registers in PostgreSQL with its role, software version, module-composition hash, queue set, and heartbeat. Ops shows active, draining, stopped, and stale nodes and compares active counts with configured replica expectations.

An operator may request or cancel drain from Ops:

• API readiness becomes unavailable on the next heartbeat so the load balancer stops assigning new requests.
• A worker stops consuming its configured queues and may finish work already claimed by that process.
• A stale process incarnation cannot overwrite a replacement incarnation's heartbeat.
• A coordination outage removes API readiness and cancels worker consumers; the existing incarnation must heartbeat successfully before either resumes.

Singleton work uses PostgreSQL-backed leases with monotonically increasing fencing tokens. The generated scheduler runs Celery beat through govoplan_core.commands.fenced_run; loss of its lease terminates the child and returns a distinct failure code. A fenced business operation must validate the same lease token immediately before committing its effect.

Release Ordering

Use this order for every multi-replica rollout:

1. Verify immutable image identities, module composition, external state reachability, backup evidence, and the generated plan.
2. Drain application replicas when the migration compatibility declaration requires it.
3. Run the release-specific migration Job exactly once. PostgreSQL advisory locking serializes all Core and module migration tasks across competing deployment jobs.
4. Let runtime init containers run wait_for_database. They wait for exact configured Alembic heads and never mutate schema.
5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
6. Verify runtime composition, expected replica counts, queue consumers, object-storage round trips, and recovery status in Ops.

Applying the complete generated manifest is fail-closed: runtime pods remain in their init phase until the migration Job reaches the expected heads. A second release may be submitted concurrently, but advisory locking prevents concurrent schema mutation and each release has a distinct migration Job name.

Storage Trust Boundary

Installer-managed Garage uses its exact generated endpoint. An arbitrary external S3 endpoint is accepted only when the deployment explicitly sets FILE_STORAGE_S3_ENDPOINT_TRUSTED=true; that endpoint must be a clean HTTPS origin without embedded credentials, query, fragment, or path. This is an operator trust declaration, not a user-controlled connector bypass. Operators remain responsible for DNS, certificate, network-egress, bucket-policy, versioning, and lifecycle controls.

Capacity

• Set GOVOPLAN_DB_CONNECTION_LIMIT to the PostgreSQL role's effective connection limit. The Kubernetes export reserves GOVOPLAN_DB_CONNECTION_RESERVE connections and rejects a topology whose calculated rolling-update peak would exceed the remainder. The calculation includes API pools, every Celery parent and prefork child, the scheduler, migration, and one surge replica per deployment. Role-specific pool and overflow values are emitted into each workload rather than inherited from one unconstrained global default.

• Scale workers by queue, with upper bounds based on external provider limits. GOVOPLAN_WORKER_POOLS may contain a JSON list of exact queue owners, for example:

  [
    {"name":"delivery","queues":["send_email","append_sent"],"replicas":2,"concurrency":2},
    {"name":"platform","queues":["events","workflow","default"],"replicas":2,"concurrency":2}
  ]
  

  Pool replica totals must equal replicas.worker, and the pools must cover CELERY_QUEUES exactly without duplicate ownership. Each pool receives its own Deployment, disruption budget, topology-spread selector, runtime identity, and declared concurrency.

• Keep one fenced scheduler rather than load-balancing schedulers.

• Increase WebUI replicas for asset/proxy capacity.

• Measure request latency, database query time and locks, active connections, queue age, retry rate, storage latency, and provider throttling before adding replicas.

Workers compete for Redis-backed work and are not placed behind a load balancer. SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often hit external-system limits before host CPU is exhausted.

What This Does Not Claim

The implemented contract provides stateless runtime placement, shared artifact access, node visibility, drain controls, migration serialization, and scheduler fencing. It does not by itself provide:

• a highly available PostgreSQL, Redis, or object-store deployment;
• automatic PostgreSQL/object backup creation or point-in-time recovery;
• autoscaling policy;
• central logs, metrics, traces, or alert routing;
• certificate portability between independently managed ingress providers;
• automatic reconciliation of every possible module side effect;
• a service-level availability guarantee.

The deployer verifies and gates migrations on signed coordinated backup and isolated-restore evidence, but backup capture and restoration remain owned by the selected state-service providers. Before claiming high availability, drill replica loss, rolling replacement, session continuity, job redelivery, scheduler failover, migration exclusion, object-store outage, and a coordinated database/object/key restore. Recovery rules and evidence are defined in Recovery And Rollback Guarantees.

Worker Delivery Evidence

The module-matrix workflow runs tools/checks/worker-runtime-drill.py against a real isolated Redis database. The drill starts supervised Celery worker processes and records four guarantees without accessing tenant data:

1. a task published through the broker is consumed exactly once;
2. an application retry is delivered again and completes;
3. warm SIGTERM lets an in-flight late-ack task complete before shutdown; and
4. loss of a worker after task start causes the unacknowledged task to be redelivered after the configured visibility timeout.

Run the same drill with the release Python environment and target Redis before promoting a worker composition. Use a dedicated Redis database, retain the JSON evidence, and set CELERY_VISIBILITY_TIMEOUT_SECONDS above the longest supported business-task duration. The short visibility timeout used by CI is an isolated test setting, not a production recommendation.

GOVOPLAN_WORKER_DRILL_REDIS_URL=redis://redis.example.test:6379/15 \
  .venv/bin/python tools/checks/worker-runtime-drill.py \
  --output evidence/worker-runtime.json

Live Multi-Host Evidence

After deploying a pinned release on at least two Kubernetes nodes, create an API key with Ops read scope and run:

export GOVOPLAN_OPS_API_KEY='...'
python tools/deployment/govoplan-deploy.py verify-kubernetes \
  --directory /srv/govoplan/installation \
  --namespace govoplan

The command fails unless API and WebUI pods are ready on at least two nodes, all rendered deployments are available, Ops reports a consistent release and module composition, every declared worker queue is served, and the calculated database peak remains below its budget. It writes a private, sanitized JSON record under the installation evidence directory and never retains the API key.

Use --exercise-api-pod-loss in an approved drill window to delete one API pod, observe the public readiness path continuously, and record its replacement. Generated API workloads use a ten-second pre-stop drain so Kubernetes can remove the terminating endpoint from ingress and service routing before Uvicorn exits. Do not remove or shorten this drain without repeating the public-path pod-loss test against the target ingress controller and network implementation. This proves the bounded stateless-node-loss slice only. Session continuity, accepted-job redelivery, state-service failover, and coordinated restore remain separate target exercises whose signed evidence is governed by docs/TARGET_MATURITY_EVIDENCE_RUNBOOK.md and GovOPlaN #37.