govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

# Scaling And Multi-Host Deployment

## What Is Implemented

The default installer now supports explicit same-host horizontal scaling:

- one HAProxy container accepts the published HTTP endpoint;
- one or more WebUI containers serve assets and proxy API requests;
- one or more API containers serve normal domain traffic;
- one or more Celery workers consume shared Redis queues;
- one migration job runs before replacement;
- exactly one Celery scheduler runs;
- PostgreSQL, Redis, and file storage are shared by every runtime replica.

HAProxy discovers Compose replicas through Docker's internal DNS and performs
health-checked round-robin WebUI balancing and least-connection API balancing.
It does not mount the Docker socket. Replica counts live in
`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on
the desired topology.

This is not multi-host scheduling. Docker Compose's bridge network belongs to
one Docker Engine. Adding a second machine requires an orchestrator or
deployment manager that can place equivalent role definitions on multiple
hosts.

## Recommended Topologies

### One Host

Use the generated Compose bundle:

```text
client
  -> external TLS proxy, when required
  -> managed HAProxy
       -> WebUI replica(s)
            -> managed HAProxy API listener
                 -> API replica(s)

API/worker/scheduler
  -> PostgreSQL
  -> Redis
  -> local volume, managed Garage, or external S3
```

This improves concurrency and permits rolling process replacement, but the host
and every managed stateful component remain single failure domains. Two API
containers on one failed host do not provide host-level availability.

### Multiple Hosts

Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not
extend the Compose installer into a proprietary scheduler. The target topology
is:

```text
public TLS ingress/load balancer
  -> WebUI replicas on at least two nodes
  -> internal API service/load balancer
       -> API replicas on at least two nodes

queue-specific worker pools on worker nodes
one fenced scheduler
one fenced migration/deployment job

shared PostgreSQL
shared Redis
multi-node Garage or external S3
shared secret/config provider
central logs, metrics, and health alerts
```

All API and worker nodes must receive the same immutable software composition,
`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object
storage binding. Node-local file storage is not valid in this topology.

## Adding Capacity

### API And WebUI

Increase API replicas for measured request concurrency, CPU saturation, or
latency after checking database load. Increase WebUI replicas for static asset
and proxy capacity. No sticky session should be required because durable
sessions and throttling use shared services, but this must remain covered by
multi-replica integration tests.

Every added API process also adds database connections. Set the application
pool size and the total replica ceiling against PostgreSQL's connection budget;
adding replicas can otherwise reduce throughput.

### Workers

Workers are not placed behind a load balancer. They compete for jobs on shared
Redis queues. Add workers by queue and cap each pool according to the external
system it calls. SMTP, IMAP, directory, and connector jobs often reach provider
rate limits before CPU limits.

Use distinct pools when load warrants it:

- mail send and Sent-folder append;
- notifications;
- calendar and connector synchronization;
- dataflow/workflow execution;
- reporting and export;
- platform events and default work.

Before stopping a worker, mark it draining, stop new claims, and let or safely
requeue active jobs. Worker registration and drain control remain part of the
larger platform scale-out story.

### Stateful Services

- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover
  design appropriate to the service level.
- Redis needs persistence and, for high availability, a supported failover
  topology. Queue loss is not equivalent to a harmless cache loss.
- Managed Garage from the Compose installer is single-node. For redundant
  storage, deploy a multi-node Garage cluster separately and select external
  `s3`, or use another compatible object store.
- The scheduler stays at one replica until distributed leader election or a
  fenced lease is implemented.
- Migrations and module lifecycle mutations remain one-at-a-time operations.

## Promotion Path

1. Measure concurrent users, request latency, database query time, connection
   use, queue age, worker saturation, storage latency, and external throttling.
2. Move files to managed Garage or external S3 before introducing independent
   runtime hosts.
3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy
   their reviewed HA operators.
4. Publish immutable API/WebUI images and one versioned configuration/secret
   contract.
5. Translate the generated role commands, environment allowlists, health
   checks, and singleton constraints into the chosen orchestrator.
6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
7. Add queue-specific workers with fixed upper bounds.
8. Prove replica loss, rolling replacement, job redelivery, session continuity,
   migration exclusion, backup restore, and storage-node loss before claiming
   high availability.

## Remaining Platform Work

The one-host installer does not yet provide:

- a Kubernetes, Nomad, or Swarm deployment export/profile;
- distributed deployment locks and fenced scheduler leadership;
- worker registration, composition-skew reporting, and drain controls;
- managed PostgreSQL, Redis, or multi-node Garage high availability;
- autoscaling policy;
- managed TLS certificate issuance and renewal;
- measured multi-replica and failover integration evidence.

These are separate production capabilities. The local load balancer and
replica model provide the contract they should implement, but they do not by
themselves make a cluster highly available.