govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

# Scaling And Multi-Host Deployment

## Implemented Contract

GovOPlaN now supports a stateless application tier backed by logically shared
state services. The runtime roles are independently replaceable API, WebUI,
worker, and scheduler processes. Every replica in one installation must use the
same immutable release composition and the same:

- `GOVOPLAN_INSTALLATION_ID`;
- PostgreSQL database;
- Redis broker and coordination service;
- `MASTER_KEY_B64` and deployment secret references;
- enabled-module graph;
- S3-compatible object-storage namespace.

The application tier must not use node-local durable business data in a
multi-host deployment. Files owns managed file metadata while Core provides the
storage-backend contract. Campaign build artifacts are stored under opaque
object keys and workers read those objects from the shared backend. Temporary
build and materialization directories may remain node-local because they are
discardable.

Core validates three explicit state profiles:

| Profile | Supported shape | Storage rule |
| --- | --- | --- |
| `local` | One API and one worker process for development | Local filesystem permitted. |
| `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. |
| `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. |

`shared` also requires a stable installation identifier. Module package
mutation is blocked in this profile: build and verify a new immutable release,
then roll the complete cluster to it.

## Same-Host Compose

The generated Compose bundle provides:

```text
client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas

API/worker/scheduler -> PostgreSQL
                     -> Redis
                     -> local volume, managed Garage, or external S3
```

HAProxy discovers Compose replicas through Docker DNS and performs health-aware
balancing without mounting the Docker socket. This improves concurrency and
permits process replacement, but the Docker host and installer-managed stateful
services remain single failure domains. Generated Compose therefore declares
the `host-shared` state profile even when its shared storage happens to be an
external S3 service. Its API backend checks `/health/ready`, so drain or
coordination loss removes a replica from rotation. Container, load-balancer,
and Kubernetes probes send the configured public host explicitly, keeping
readiness compatible with strict trusted-host validation.

Managed Garage is a convenient single-node S3-compatible service. It is not a
multi-host storage cluster. Use an independently operated Garage cluster or
another S3-compatible service for the `shared` profile.

## Kubernetes Export

The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL,
Redis, and S3 are all external:

```sh
python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime \
  --tls-secret-name govoplan-tls \
  --ingress-class-name nginx \
  --output /srv/govoplan/default/kubernetes.json
```

The export contains a Namespace, tokenless ServiceAccount, non-secret
ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption
budgets, Ingress, and a release-specific migration Job. It deliberately emits
no Secret values, persistent volume, PostgreSQL, Redis, or object-store
deployment. Export is rejected unless both release images use immutable
`image@sha256:...` references.

Create the named Secret through the cluster's secret-management path. The
command prints the exact required key contract. Review the generated
`FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network
before production use.

The generated containers run as non-root with a read-only root filesystem and
an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database
migration heads before starting. The API exposes `/health/ready`, which fails
while that API node is draining or cannot prove its runtime-coordination
heartbeat.

## Runtime Coordination

Each API and worker incarnation registers in PostgreSQL with its role, software
version, module-composition hash, queue set, and heartbeat. Ops shows active,
draining, stopped, and stale nodes and compares active counts with configured
replica expectations.

An operator may request or cancel drain from Ops:

- API readiness becomes unavailable on the next heartbeat so the load balancer
  stops assigning new requests.
- A worker stops consuming its configured queues and may finish work already
  claimed by that process.
- A stale process incarnation cannot overwrite a replacement incarnation's
  heartbeat.
- A coordination outage removes API readiness and cancels worker consumers;
  the existing incarnation must heartbeat successfully before either resumes.

Singleton work uses PostgreSQL-backed leases with monotonically increasing
fencing tokens. The generated scheduler runs Celery beat through
`govoplan_core.commands.fenced_run`; loss of its lease terminates the child and
returns a distinct failure code. A fenced business operation must validate the
same lease token immediately before committing its effect.

## Release Ordering

Use this order for every multi-replica rollout:

1. Verify immutable image identities, module composition, external state
   reachability, backup evidence, and the generated plan.
2. Drain application replicas when the migration compatibility declaration
   requires it.
3. Run the release-specific migration Job exactly once. PostgreSQL advisory
   locking serializes all Core and module migration tasks across competing
   deployment jobs.
4. Let runtime init containers run `wait_for_database`. They wait for exact
   configured Alembic heads and never mutate schema.
5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
6. Verify runtime composition, expected replica counts, queue consumers,
   object-storage round trips, and recovery status in Ops.

Applying the complete generated manifest is fail-closed: runtime pods remain in
their init phase until the migration Job reaches the expected heads. A second
release may be submitted concurrently, but advisory locking prevents concurrent
schema mutation and each release has a distinct migration Job name.

## Storage Trust Boundary

Installer-managed Garage uses its exact generated endpoint. An arbitrary
external S3 endpoint is accepted only when the deployment explicitly sets
`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS
origin without embedded credentials, query, fragment, or path. This is an
operator trust declaration, not a user-controlled connector bypass. Operators
remain responsible for DNS, certificate, network-egress, bucket-policy,
versioning, and lifecycle controls.

## Capacity

- Scale API replicas only within the PostgreSQL connection budget.
- Scale workers by queue, with upper bounds based on external provider limits.
- Keep one fenced scheduler rather than load-balancing schedulers.
- Increase WebUI replicas for asset/proxy capacity.
- Measure request latency, database query time and locks, active connections,
  queue age, retry rate, storage latency, and provider throttling before adding
  replicas.

Workers compete for Redis-backed work and are not placed behind a load balancer.
SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often
hit external-system limits before host CPU is exhausted.

## What This Does Not Claim

The implemented contract provides stateless runtime placement, shared artifact
access, node visibility, drain controls, migration serialization, and scheduler
fencing. It does not by itself provide:

- a highly available PostgreSQL, Redis, or object-store deployment;
- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
- autoscaling policy;
- central logs, metrics, traces, or alert routing;
- managed ingress certificates;
- automatic reconciliation of every possible module side effect;
- a service-level availability guarantee.

Those are deployment and module-adoption requirements. Before claiming high
availability, drill replica loss, rolling replacement, session continuity, job
redelivery, scheduler failover, migration exclusion, object-store outage, and a
coordinated database/object/key restore. Recovery rules and evidence are
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).