Add scalable deployment planning and guided releases

docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

@@ -23,7 +23,12 @@ It currently supports:
 - managed, external, or evaluation-only disabled Redis;
 - disabled mail, an external relay declaration, or an evaluation-only
   GreenMail service;
-- durable local file storage or external S3-compatible storage;
+- durable local file storage, managed single-node Garage S3, or external
+  S3-compatible storage;
+- an explicit HAProxy service that load-balances configured WebUI and API
+  replicas without access to the Docker socket;
+- declarative API, WebUI, and worker replica counts while keeping migrations
+  and the scheduler singleton;
 - Core, base, or full initial module selections;
 - deterministic Compose JSON accepted by Compose v2;
 - generated secrets stored in a private `0600` file;
@@ -47,7 +52,11 @@ Create a local evaluation bundle:
   --profile evaluation \
   --postgres managed \
   --redis managed \
+  --storage garage \
   --mail test-mail \
+  --api-replicas 2 \
+  --web-replicas 2 \
+  --worker-replicas 2 \
   --module-set base
 ```
 
@@ -74,6 +83,8 @@ The private installation directory contains:
 | `installation.json` | Versioned, non-secret desired state |
 | `secrets.env` | Deployment-local secrets and external service bindings |
 | `compose.json` | Deterministic generated Compose definition |
+| `garage.toml` | Non-secret managed Garage server configuration |
+| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
 | `plan.json` | Latest desired-state diff and readiness findings |
 | `receipt.json` | Last successfully applied immutable identities |
 | `.deployment.lock` | Same-host operation exclusion |
@@ -126,7 +137,9 @@ a production distribution:
 5. **Deployment agent.** Web updates need a separate privileged reconciler with
    a typed command allowlist. The API and browser must never receive the Docker
    socket or arbitrary shell access.
-6. **Ingress and certificates.** A self-hosted profile needs an explicit choice
+6. **Ingress and certificates.** The managed HAProxy service provides HTTP
+   load balancing inside the deployment boundary; it does not issue or renew
+   certificates. A self-hosted profile still needs an explicit choice
    between an existing reverse proxy and a supported managed ingress, including
    trusted-proxy boundaries, TLS certificate issuance, renewal, and health
    probing through the public route.
@@ -185,11 +198,49 @@ presenting a container as a complete mail service.
 ### File Storage
 
 `local` uses a durable Compose volume and is appropriate for one-host
-installations. `s3` requires endpoint, region, access key, secret key, and
-bucket values. Self-hosted S3 endpoints must use HTTPS.
+installations. `garage` provisions Garage 2.3 in its supported single-node
+bootstrap mode, generates a private application key and bucket, and connects
+the Files S3 backend to the exact installer-owned internal endpoint. The
+deployment-only trust marker cannot authorize another S3 host; arbitrary
+external SDK endpoints remain fail-closed until peer pinning is implemented.
+Garage metadata and object data use separate persistent volumes. `s3` requires
+an external endpoint, region, access key, secret key, and bucket values.
+Self-hosted external S3 endpoints must use HTTPS.
 
 Local storage must be included in backup and restore drills. Horizontal API or
-worker scale-out requires shared/object storage.
+worker scale-out requires shared/object storage. The managed Garage profile is
+persistent but has no data redundancy; availability-sensitive installations
+must use a tested multi-node Garage cluster or another external S3 service.
+
+### Load Balancing And Replicas
+
+The generated Compose topology publishes only `load-balancer`. HAProxy uses
+Docker DNS service discovery to distribute public traffic across WebUI replicas
+and WebUI API proxy traffic across API replicas. The WebUI and API services do
+not publish host ports. HAProxy has no Docker socket and discovers only the
+bounded replica slots rendered into `load-balancer.cfg`.
+
+Replica counts are desired state:
+
+```sh
+./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
+  --directory /tmp/govoplan-evaluation \
+  --api-replicas 3 \
+  --web-replicas 2 \
+  --worker-replicas 4
+./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
+  --directory /tmp/govoplan-evaluation
+```
+
+Workers are queue consumers, so they are scaled through Redis rather than put
+behind an HTTP load balancer. The migration runner and Celery scheduler remain
+singletons. Multiple API replicas are rejected when Redis is disabled because
+distributed throttling and queued work cannot then be shared correctly.
+
+This is same-host scaling. Docker Compose uses a bridge network and does not
+place containers on another machine. See
+[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) for
+the supported topology and promotion path.
 
 ## Reconfiguration Semantics
 
@@ -207,6 +258,8 @@ applied state. `plan` compares their canonical hashes and service sets.
 - Managed-to-external transitions require the new endpoint in the same
   operation.
 - Migrations run as a one-shot service before API/worker replacement.
+- The first upgrade from a direct WebUI host port stops that legacy WebUI
+  container immediately before HAProxy claims the same endpoint.
 - Health must recover before a new receipt is committed.
 
 This is sufficient for one-host reconciliation. Production updates additionally
@@ -261,7 +314,8 @@ Run the focused tests:
 ```
 
 The tests cover profile restrictions, secret persistence, external endpoint
-requirements, S3 policy, Compose service selection, secret non-disclosure,
+requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
+discovery configuration, Compose service selection, secret non-disclosure,
 service-specific environment isolation, private file modes, external endpoint
 preflight, first-plan generation, apply ordering, and receipt-based
 idempotency.

docs/RELEASE_CONSOLE.md

@@ -67,6 +67,24 @@ shows the dry-run commands for the selected rows, and `Generate Candidate`
 creates a signed catalog candidate that advances only selected repositories that
 already have a catalog entry.
 
+The full-width **Release Workflow** guide projects the server state into seven
+operator phases: Inspect, Targets, Validate, Source, Package, Publish, and
+Verify. It does not maintain a second workflow state. Completed, current,
+blocked, locked, and unavailable phases are derived from the dashboard,
+selective plan, and durable run record. The next-action panel opens the exact
+section or durable step that needs attention. Changing a channel, target
+version, repository selection, or release gate detaches the browser from the
+current run and invalidates the draft plan; the persisted run remains available
+from the saved-run selector. Problems in unselected repositories remain visible
+as workspace notices but do not lock an unrelated release; the selective plan
+is the authority for blockers in the selected repository set.
+
+Installation verification remains an explicit unavailable phase after a
+durable run completes. It is currently enforced by release-integration CI,
+rather than being presented as a successful local-console step. This prevents
+the guide from treating source publication as proof that the published package
+can be installed and started.
+
 `Build Plan` also returns structured release-gate findings for each selected
 repository. The plan names the recommended next action and gives an explicit
 remediation for source-version, lockfile, Core WebUI composition, Git state, and
@@ -226,8 +244,9 @@ other `/api/` route:
 - `POST /api/release-runs/{run_id}/steps/{step_id}/preview` provides the
   non-mutating preview for receipt-bound catalog publication.
 
-Run-storage errors are confined to the Durable Run State card; dashboard and
-release-preview collection continue and show the bounded storage remediation.
+Run-storage errors are confined to the Durable Release Run section; dashboard
+and release-preview collection continue and the workflow guide points to the
+bounded storage remediation.
 
 The run record is execution evidence only for a supported step whose durable
 claim and bounded result receipt were persisted. The console never infers

docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

@@ -0,0 +1,156 @@
+# Scaling And Multi-Host Deployment
+
+## What Is Implemented
+
+The default installer now supports explicit same-host horizontal scaling:
+
+- one HAProxy container accepts the published HTTP endpoint;
+- one or more WebUI containers serve assets and proxy API requests;
+- one or more API containers serve normal domain traffic;
+- one or more Celery workers consume shared Redis queues;
+- one migration job runs before replacement;
+- exactly one Celery scheduler runs;
+- PostgreSQL, Redis, and file storage are shared by every runtime replica.
+
+HAProxy discovers Compose replicas through Docker's internal DNS and performs
+health-checked round-robin WebUI balancing and least-connection API balancing.
+It does not mount the Docker socket. Replica counts live in
+`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on
+the desired topology.
+
+This is not multi-host scheduling. Docker Compose's bridge network belongs to
+one Docker Engine. Adding a second machine requires an orchestrator or
+deployment manager that can place equivalent role definitions on multiple
+hosts.
+
+## Recommended Topologies
+
+### One Host
+
+Use the generated Compose bundle:
+
+```text
+client
+  -> external TLS proxy, when required
+  -> managed HAProxy
+       -> WebUI replica(s)
+            -> managed HAProxy API listener
+                 -> API replica(s)
+
+API/worker/scheduler
+  -> PostgreSQL
+  -> Redis
+  -> local volume, managed Garage, or external S3
+```
+
+This improves concurrency and permits rolling process replacement, but the host
+and every managed stateful component remain single failure domains. Two API
+containers on one failed host do not provide host-level availability.
+
+### Multiple Hosts
+
+Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not
+extend the Compose installer into a proprietary scheduler. The target topology
+is:
+
+```text
+public TLS ingress/load balancer
+  -> WebUI replicas on at least two nodes
+  -> internal API service/load balancer
+       -> API replicas on at least two nodes
+
+queue-specific worker pools on worker nodes
+one fenced scheduler
+one fenced migration/deployment job
+
+shared PostgreSQL
+shared Redis
+multi-node Garage or external S3
+shared secret/config provider
+central logs, metrics, and health alerts
+```
+
+All API and worker nodes must receive the same immutable software composition,
+`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object
+storage binding. Node-local file storage is not valid in this topology.
+
+## Adding Capacity
+
+### API And WebUI
+
+Increase API replicas for measured request concurrency, CPU saturation, or
+latency after checking database load. Increase WebUI replicas for static asset
+and proxy capacity. No sticky session should be required because durable
+sessions and throttling use shared services, but this must remain covered by
+multi-replica integration tests.
+
+Every added API process also adds database connections. Set the application
+pool size and the total replica ceiling against PostgreSQL's connection budget;
+adding replicas can otherwise reduce throughput.
+
+### Workers
+
+Workers are not placed behind a load balancer. They compete for jobs on shared
+Redis queues. Add workers by queue and cap each pool according to the external
+system it calls. SMTP, IMAP, directory, and connector jobs often reach provider
+rate limits before CPU limits.
+
+Use distinct pools when load warrants it:
+
+- mail send and Sent-folder append;
+- notifications;
+- calendar and connector synchronization;
+- dataflow/workflow execution;
+- reporting and export;
+- platform events and default work.
+
+Before stopping a worker, mark it draining, stop new claims, and let or safely
+requeue active jobs. Worker registration and drain control remain part of the
+larger platform scale-out story.
+
+### Stateful Services
+
+- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover
+  design appropriate to the service level.
+- Redis needs persistence and, for high availability, a supported failover
+  topology. Queue loss is not equivalent to a harmless cache loss.
+- Managed Garage from the Compose installer is single-node. For redundant
+  storage, deploy a multi-node Garage cluster separately and select external
+  `s3`, or use another compatible object store.
+- The scheduler stays at one replica until distributed leader election or a
+  fenced lease is implemented.
+- Migrations and module lifecycle mutations remain one-at-a-time operations.
+
+## Promotion Path
+
+1. Measure concurrent users, request latency, database query time, connection
+   use, queue age, worker saturation, storage latency, and external throttling.
+2. Move files to managed Garage or external S3 before introducing independent
+   runtime hosts.
+3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy
+   their reviewed HA operators.
+4. Publish immutable API/WebUI images and one versioned configuration/secret
+   contract.
+5. Translate the generated role commands, environment allowlists, health
+   checks, and singleton constraints into the chosen orchestrator.
+6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
+7. Add queue-specific workers with fixed upper bounds.
+8. Prove replica loss, rolling replacement, job redelivery, session continuity,
+   migration exclusion, backup restore, and storage-node loss before claiming
+   high availability.
+
+## Remaining Platform Work
+
+The one-host installer does not yet provide:
+
+- a Kubernetes, Nomad, or Swarm deployment export/profile;
+- distributed deployment locks and fenced scheduler leadership;
+- worker registration, composition-skew reporting, and drain controls;
+- managed PostgreSQL, Redis, or multi-node Garage high availability;
+- autoscaling policy;
+- managed TLS certificate issuance and renewal;
+- measured multi-replica and failover integration evidence.
+
+These are separate production capabilities. The local load balancer and
+replica model provide the contract they should implement, but they do not by
+themselves make a cluster highly available.

docs/installation-spec.schema.json

@@ -124,10 +124,44 @@
             "mode": {
               "enum": [
                 "local",
+                "garage",
                 "s3"
               ]
+            },
+            "image": {
+              "type": "string",
+              "maxLength": 300
             }
           }
+        },
+        "load_balancer": {
+          "$ref": "#/$defs/load_balancer"
+        }
+      }
+    },
+    "replicas": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "api",
+        "web",
+        "worker"
+      ],
+      "properties": {
+        "api": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 64
+        },
+        "web": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 64
+        },
+        "worker": {
+          "type": "integer",
+          "minimum": 0,
+          "maximum": 128
         }
       }
     },
@@ -222,6 +256,23 @@
           }
         }
       ]
+    },
+    "load_balancer": {
+      "allOf": [
+        {
+          "$ref": "#/$defs/service"
+        },
+        {
+          "properties": {
+            "mode": {
+              "const": "managed"
+            },
+            "url_env": {
+              "const": ""
+            }
+          }
+        }
+      ]
     }
   },
   "allOf": [