Add scalable deployment planning and guided releases

docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

@@ -0,0 +1,156 @@
+# Scaling And Multi-Host Deployment
+
+## What Is Implemented
+
+The default installer now supports explicit same-host horizontal scaling:
+
+- one HAProxy container accepts the published HTTP endpoint;
+- one or more WebUI containers serve assets and proxy API requests;
+- one or more API containers serve normal domain traffic;
+- one or more Celery workers consume shared Redis queues;
+- one migration job runs before replacement;
+- exactly one Celery scheduler runs;
+- PostgreSQL, Redis, and file storage are shared by every runtime replica.
+
+HAProxy discovers Compose replicas through Docker's internal DNS and performs
+health-checked round-robin WebUI balancing and least-connection API balancing.
+It does not mount the Docker socket. Replica counts live in
+`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on
+the desired topology.
+
+This is not multi-host scheduling. Docker Compose's bridge network belongs to
+one Docker Engine. Adding a second machine requires an orchestrator or
+deployment manager that can place equivalent role definitions on multiple
+hosts.
+
+## Recommended Topologies
+
+### One Host
+
+Use the generated Compose bundle:
+
+```text
+client
+  -> external TLS proxy, when required
+  -> managed HAProxy
+       -> WebUI replica(s)
+            -> managed HAProxy API listener
+                 -> API replica(s)
+
+API/worker/scheduler
+  -> PostgreSQL
+  -> Redis
+  -> local volume, managed Garage, or external S3
+```
+
+This improves concurrency and permits rolling process replacement, but the host
+and every managed stateful component remain single failure domains. Two API
+containers on one failed host do not provide host-level availability.
+
+### Multiple Hosts
+
+Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not
+extend the Compose installer into a proprietary scheduler. The target topology
+is:
+
+```text
+public TLS ingress/load balancer
+  -> WebUI replicas on at least two nodes
+  -> internal API service/load balancer
+       -> API replicas on at least two nodes
+
+queue-specific worker pools on worker nodes
+one fenced scheduler
+one fenced migration/deployment job
+
+shared PostgreSQL
+shared Redis
+multi-node Garage or external S3
+shared secret/config provider
+central logs, metrics, and health alerts
+```
+
+All API and worker nodes must receive the same immutable software composition,
+`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object
+storage binding. Node-local file storage is not valid in this topology.
+
+## Adding Capacity
+
+### API And WebUI
+
+Increase API replicas for measured request concurrency, CPU saturation, or
+latency after checking database load. Increase WebUI replicas for static asset
+and proxy capacity. No sticky session should be required because durable
+sessions and throttling use shared services, but this must remain covered by
+multi-replica integration tests.
+
+Every added API process also adds database connections. Set the application
+pool size and the total replica ceiling against PostgreSQL's connection budget;
+adding replicas can otherwise reduce throughput.
+
+### Workers
+
+Workers are not placed behind a load balancer. They compete for jobs on shared
+Redis queues. Add workers by queue and cap each pool according to the external
+system it calls. SMTP, IMAP, directory, and connector jobs often reach provider
+rate limits before CPU limits.
+
+Use distinct pools when load warrants it:
+
+- mail send and Sent-folder append;
+- notifications;
+- calendar and connector synchronization;
+- dataflow/workflow execution;
+- reporting and export;
+- platform events and default work.
+
+Before stopping a worker, mark it draining, stop new claims, and let or safely
+requeue active jobs. Worker registration and drain control remain part of the
+larger platform scale-out story.
+
+### Stateful Services
+
+- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover
+  design appropriate to the service level.
+- Redis needs persistence and, for high availability, a supported failover
+  topology. Queue loss is not equivalent to a harmless cache loss.
+- Managed Garage from the Compose installer is single-node. For redundant
+  storage, deploy a multi-node Garage cluster separately and select external
+  `s3`, or use another compatible object store.
+- The scheduler stays at one replica until distributed leader election or a
+  fenced lease is implemented.
+- Migrations and module lifecycle mutations remain one-at-a-time operations.
+
+## Promotion Path
+
+1. Measure concurrent users, request latency, database query time, connection
+   use, queue age, worker saturation, storage latency, and external throttling.
+2. Move files to managed Garage or external S3 before introducing independent
+   runtime hosts.
+3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy
+   their reviewed HA operators.
+4. Publish immutable API/WebUI images and one versioned configuration/secret
+   contract.
+5. Translate the generated role commands, environment allowlists, health
+   checks, and singleton constraints into the chosen orchestrator.
+6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
+7. Add queue-specific workers with fixed upper bounds.
+8. Prove replica loss, rolling replacement, job redelivery, session continuity,
+   migration exclusion, backup restore, and storage-node loss before claiming
+   high availability.
+
+## Remaining Platform Work
+
+The one-host installer does not yet provide:
+
+- a Kubernetes, Nomad, or Swarm deployment export/profile;
+- distributed deployment locks and fenced scheduler leadership;
+- worker registration, composition-skew reporting, and drain controls;
+- managed PostgreSQL, Redis, or multi-node Garage high availability;
+- autoscaling policy;
+- managed TLS certificate issuance and renewal;
+- measured multi-replica and failover integration evidence.
+
+These are separate production capabilities. The local load balancer and
+replica model provide the contract they should implement, but they do not by
+themselves make a cluster highly available.