govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

Scaling And Multi-Host Deployment

What Is Implemented

The default installer now supports explicit same-host horizontal scaling:

• one HAProxy container accepts the published HTTP endpoint;
• one or more WebUI containers serve assets and proxy API requests;
• one or more API containers serve normal domain traffic;
• one or more Celery workers consume shared Redis queues;
• one migration job runs before replacement;
• exactly one Celery scheduler runs;
• PostgreSQL, Redis, and file storage are shared by every runtime replica.

HAProxy discovers Compose replicas through Docker's internal DNS and performs health-checked round-robin WebUI balancing and least-connection API balancing. It does not mount the Docker socket. Replica counts live in installation.json, so configure, plan, apply, and the receipt agree on the desired topology.

This is not multi-host scheduling. Docker Compose's bridge network belongs to one Docker Engine. Adding a second machine requires an orchestrator or deployment manager that can place equivalent role definitions on multiple hosts.

Recommended Topologies

One Host

Use the generated Compose bundle:

client
  -> external TLS proxy, when required
  -> managed HAProxy
       -> WebUI replica(s)
            -> managed HAProxy API listener
                 -> API replica(s)

API/worker/scheduler
  -> PostgreSQL
  -> Redis
  -> local volume, managed Garage, or external S3

This improves concurrency and permits rolling process replacement, but the host and every managed stateful component remain single failure domains. Two API containers on one failed host do not provide host-level availability.

Multiple Hosts

Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not extend the Compose installer into a proprietary scheduler. The target topology is:

public TLS ingress/load balancer
  -> WebUI replicas on at least two nodes
  -> internal API service/load balancer
       -> API replicas on at least two nodes

queue-specific worker pools on worker nodes
one fenced scheduler
one fenced migration/deployment job

shared PostgreSQL
shared Redis
multi-node Garage or external S3
shared secret/config provider
central logs, metrics, and health alerts

All API and worker nodes must receive the same immutable software composition, MASTER_KEY_B64, database URL, Redis URL, enabled-module graph, and object storage binding. Node-local file storage is not valid in this topology.

Adding Capacity

API And WebUI

Increase API replicas for measured request concurrency, CPU saturation, or latency after checking database load. Increase WebUI replicas for static asset and proxy capacity. No sticky session should be required because durable sessions and throttling use shared services, but this must remain covered by multi-replica integration tests.

Every added API process also adds database connections. Set the application pool size and the total replica ceiling against PostgreSQL's connection budget; adding replicas can otherwise reduce throughput.

Workers

Workers are not placed behind a load balancer. They compete for jobs on shared Redis queues. Add workers by queue and cap each pool according to the external system it calls. SMTP, IMAP, directory, and connector jobs often reach provider rate limits before CPU limits.

Use distinct pools when load warrants it:

• mail send and Sent-folder append;
• notifications;
• calendar and connector synchronization;
• dataflow/workflow execution;
• reporting and export;
• platform events and default work.

Before stopping a worker, mark it draining, stop new claims, and let or safely requeue active jobs. Worker registration and drain control remain part of the larger platform scale-out story.

Stateful Services

• PostgreSQL needs backups, restore drills, connection limits, and an HA/failover design appropriate to the service level.
• Redis needs persistence and, for high availability, a supported failover topology. Queue loss is not equivalent to a harmless cache loss.
• Managed Garage from the Compose installer is single-node. For redundant storage, deploy a multi-node Garage cluster separately and select external s3, or use another compatible object store.
• The scheduler stays at one replica until distributed leader election or a fenced lease is implemented.
• Migrations and module lifecycle mutations remain one-at-a-time operations.

Promotion Path

1. Measure concurrent users, request latency, database query time, connection use, queue age, worker saturation, storage latency, and external throttling.
2. Move files to managed Garage or external S3 before introducing independent runtime hosts.
3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy their reviewed HA operators.
4. Publish immutable API/WebUI images and one versioned configuration/secret contract.
5. Translate the generated role commands, environment allowlists, health checks, and singleton constraints into the chosen orchestrator.
6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
7. Add queue-specific workers with fixed upper bounds.
8. Prove replica loss, rolling replacement, job redelivery, session continuity, migration exclusion, backup restore, and storage-node loss before claiming high availability.

Remaining Platform Work

The one-host installer does not yet provide:

• a Kubernetes, Nomad, or Swarm deployment export/profile;
• distributed deployment locks and fenced scheduler leadership;
• worker registration, composition-skew reporting, and drain controls;
• managed PostgreSQL, Redis, or multi-node Garage high availability;
• autoscaling policy;
• managed TLS certificate issuance and renewal;
• measured multi-replica and failover integration evidence.

These are separate production capabilities. The local load balancer and replica model provide the contract they should implement, but they do not by themselves make a cluster highly available.