feat: implement institutional governance and recovery architecture

docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

@@ -1,156 +1,183 @@
 # Scaling And Multi-Host Deployment
 
-## What Is Implemented
+## Implemented Contract
 
-The default installer now supports explicit same-host horizontal scaling:
+GovOPlaN now supports a stateless application tier backed by logically shared
+state services. The runtime roles are independently replaceable API, WebUI,
+worker, and scheduler processes. Every replica in one installation must use the
+same immutable release composition and the same:
 
-- one HAProxy container accepts the published HTTP endpoint;
-- one or more WebUI containers serve assets and proxy API requests;
-- one or more API containers serve normal domain traffic;
-- one or more Celery workers consume shared Redis queues;
-- one migration job runs before replacement;
-- exactly one Celery scheduler runs;
-- PostgreSQL, Redis, and file storage are shared by every runtime replica.
+- `GOVOPLAN_INSTALLATION_ID`;
+- PostgreSQL database;
+- Redis broker and coordination service;
+- `MASTER_KEY_B64` and deployment secret references;
+- enabled-module graph;
+- S3-compatible object-storage namespace.
 
-HAProxy discovers Compose replicas through Docker's internal DNS and performs
-health-checked round-robin WebUI balancing and least-connection API balancing.
-It does not mount the Docker socket. Replica counts live in
-`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on
-the desired topology.
+The application tier must not use node-local durable business data in a
+multi-host deployment. Files owns managed file metadata while Core provides the
+storage-backend contract. Campaign build artifacts are stored under opaque
+object keys and workers read those objects from the shared backend. Temporary
+build and materialization directories may remain node-local because they are
+discardable.
 
-This is not multi-host scheduling. Docker Compose's bridge network belongs to
-one Docker Engine. Adding a second machine requires an orchestrator or
-deployment manager that can place equivalent role definitions on multiple
-hosts.
+Core validates three explicit state profiles:
 
-## Recommended Topologies
+| Profile | Supported shape | Storage rule |
+| --- | --- | --- |
+| `local` | One API and one worker process for development | Local filesystem permitted. |
+| `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. |
+| `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. |
 
-### One Host
+`shared` also requires a stable installation identifier. Module package
+mutation is blocked in this profile: build and verify a new immutable release,
+then roll the complete cluster to it.
 
-Use the generated Compose bundle:
+## Same-Host Compose
+
+The generated Compose bundle provides:
 
 ```text
-client
-  -> external TLS proxy, when required
-  -> managed HAProxy
-       -> WebUI replica(s)
-            -> managed HAProxy API listener
-                 -> API replica(s)
+client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas
 
-API/worker/scheduler
-  -> PostgreSQL
-  -> Redis
-  -> local volume, managed Garage, or external S3
+API/worker/scheduler -> PostgreSQL
+                     -> Redis
+                     -> local volume, managed Garage, or external S3
 ```
 
-This improves concurrency and permits rolling process replacement, but the host
-and every managed stateful component remain single failure domains. Two API
-containers on one failed host do not provide host-level availability.
+HAProxy discovers Compose replicas through Docker DNS and performs health-aware
+balancing without mounting the Docker socket. This improves concurrency and
+permits process replacement, but the Docker host and installer-managed stateful
+services remain single failure domains. Generated Compose therefore declares
+the `host-shared` state profile even when its shared storage happens to be an
+external S3 service. Its API backend checks `/health/ready`, so drain or
+coordination loss removes a replica from rotation. Container, load-balancer,
+and Kubernetes probes send the configured public host explicitly, keeping
+readiness compatible with strict trusted-host validation.
 
-### Multiple Hosts
+Managed Garage is a convenient single-node S3-compatible service. It is not a
+multi-host storage cluster. Use an independently operated Garage cluster or
+another S3-compatible service for the `shared` profile.
 
-Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not
-extend the Compose installer into a proprietary scheduler. The target topology
-is:
+## Kubernetes Export
 
-```text
-public TLS ingress/load balancer
-  -> WebUI replicas on at least two nodes
-  -> internal API service/load balancer
-       -> API replicas on at least two nodes
+The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL,
+Redis, and S3 are all external:
 
-queue-specific worker pools on worker nodes
-one fenced scheduler
-one fenced migration/deployment job
-
-shared PostgreSQL
-shared Redis
-multi-node Garage or external S3
-shared secret/config provider
-central logs, metrics, and health alerts
+```sh
+python tools/deployment/govoplan-deploy.py render-kubernetes \
+  --directory /srv/govoplan/default \
+  --namespace govoplan \
+  --secret-name govoplan-runtime \
+  --tls-secret-name govoplan-tls \
+  --ingress-class-name nginx \
+  --output /srv/govoplan/default/kubernetes.json
 ```
 
-All API and worker nodes must receive the same immutable software composition,
-`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object
-storage binding. Node-local file storage is not valid in this topology.
+The export contains a Namespace, tokenless ServiceAccount, non-secret
+ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption
+budgets, Ingress, and a release-specific migration Job. It deliberately emits
+no Secret values, persistent volume, PostgreSQL, Redis, or object-store
+deployment. Export is rejected unless both release images use immutable
+`image@sha256:...` references.
 
-## Adding Capacity
+Create the named Secret through the cluster's secret-management path. The
+command prints the exact required key contract. Review the generated
+`FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network
+before production use.
 
-### API And WebUI
+The generated containers run as non-root with a read-only root filesystem and
+an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database
+migration heads before starting. The API exposes `/health/ready`, which fails
+while that API node is draining or cannot prove its runtime-coordination
+heartbeat.
 
-Increase API replicas for measured request concurrency, CPU saturation, or
-latency after checking database load. Increase WebUI replicas for static asset
-and proxy capacity. No sticky session should be required because durable
-sessions and throttling use shared services, but this must remain covered by
-multi-replica integration tests.
+## Runtime Coordination
 
-Every added API process also adds database connections. Set the application
-pool size and the total replica ceiling against PostgreSQL's connection budget;
-adding replicas can otherwise reduce throughput.
+Each API and worker incarnation registers in PostgreSQL with its role, software
+version, module-composition hash, queue set, and heartbeat. Ops shows active,
+draining, stopped, and stale nodes and compares active counts with configured
+replica expectations.
 
-### Workers
+An operator may request or cancel drain from Ops:
 
-Workers are not placed behind a load balancer. They compete for jobs on shared
-Redis queues. Add workers by queue and cap each pool according to the external
-system it calls. SMTP, IMAP, directory, and connector jobs often reach provider
-rate limits before CPU limits.
+- API readiness becomes unavailable on the next heartbeat so the load balancer
+  stops assigning new requests.
+- A worker stops consuming its configured queues and may finish work already
+  claimed by that process.
+- A stale process incarnation cannot overwrite a replacement incarnation's
+  heartbeat.
+- A coordination outage removes API readiness and cancels worker consumers;
+  the existing incarnation must heartbeat successfully before either resumes.
 
-Use distinct pools when load warrants it:
+Singleton work uses PostgreSQL-backed leases with monotonically increasing
+fencing tokens. The generated scheduler runs Celery beat through
+`govoplan_core.commands.fenced_run`; loss of its lease terminates the child and
+returns a distinct failure code. A fenced business operation must validate the
+same lease token immediately before committing its effect.
 
-- mail send and Sent-folder append;
-- notifications;
-- calendar and connector synchronization;
-- dataflow/workflow execution;
-- reporting and export;
-- platform events and default work.
+## Release Ordering
 
-Before stopping a worker, mark it draining, stop new claims, and let or safely
-requeue active jobs. Worker registration and drain control remain part of the
-larger platform scale-out story.
+Use this order for every multi-replica rollout:
 
-### Stateful Services
+1. Verify immutable image identities, module composition, external state
+   reachability, backup evidence, and the generated plan.
+2. Drain application replicas when the migration compatibility declaration
+   requires it.
+3. Run the release-specific migration Job exactly once. PostgreSQL advisory
+   locking serializes all Core and module migration tasks across competing
+   deployment jobs.
+4. Let runtime init containers run `wait_for_database`. They wait for exact
+   configured Alembic heads and never mutate schema.
+5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
+6. Verify runtime composition, expected replica counts, queue consumers,
+   object-storage round trips, and recovery status in Ops.
 
-- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover
-  design appropriate to the service level.
-- Redis needs persistence and, for high availability, a supported failover
-  topology. Queue loss is not equivalent to a harmless cache loss.
-- Managed Garage from the Compose installer is single-node. For redundant
-  storage, deploy a multi-node Garage cluster separately and select external
-  `s3`, or use another compatible object store.
-- The scheduler stays at one replica until distributed leader election or a
-  fenced lease is implemented.
-- Migrations and module lifecycle mutations remain one-at-a-time operations.
+Applying the complete generated manifest is fail-closed: runtime pods remain in
+their init phase until the migration Job reaches the expected heads. A second
+release may be submitted concurrently, but advisory locking prevents concurrent
+schema mutation and each release has a distinct migration Job name.
 
-## Promotion Path
+## Storage Trust Boundary
 
-1. Measure concurrent users, request latency, database query time, connection
-   use, queue age, worker saturation, storage latency, and external throttling.
-2. Move files to managed Garage or external S3 before introducing independent
-   runtime hosts.
-3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy
-   their reviewed HA operators.
-4. Publish immutable API/WebUI images and one versioned configuration/secret
-   contract.
-5. Translate the generated role commands, environment allowlists, health
-   checks, and singleton constraints into the chosen orchestrator.
-6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
-7. Add queue-specific workers with fixed upper bounds.
-8. Prove replica loss, rolling replacement, job redelivery, session continuity,
-   migration exclusion, backup restore, and storage-node loss before claiming
-   high availability.
+Installer-managed Garage uses its exact generated endpoint. An arbitrary
+external S3 endpoint is accepted only when the deployment explicitly sets
+`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS
+origin without embedded credentials, query, fragment, or path. This is an
+operator trust declaration, not a user-controlled connector bypass. Operators
+remain responsible for DNS, certificate, network-egress, bucket-policy,
+versioning, and lifecycle controls.
 
-## Remaining Platform Work
+## Capacity
 
-The one-host installer does not yet provide:
+- Scale API replicas only within the PostgreSQL connection budget.
+- Scale workers by queue, with upper bounds based on external provider limits.
+- Keep one fenced scheduler rather than load-balancing schedulers.
+- Increase WebUI replicas for asset/proxy capacity.
+- Measure request latency, database query time and locks, active connections,
+  queue age, retry rate, storage latency, and provider throttling before adding
+  replicas.
 
-- a Kubernetes, Nomad, or Swarm deployment export/profile;
-- distributed deployment locks and fenced scheduler leadership;
-- worker registration, composition-skew reporting, and drain controls;
-- managed PostgreSQL, Redis, or multi-node Garage high availability;
+Workers compete for Redis-backed work and are not placed behind a load balancer.
+SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often
+hit external-system limits before host CPU is exhausted.
+
+## What This Does Not Claim
+
+The implemented contract provides stateless runtime placement, shared artifact
+access, node visibility, drain controls, migration serialization, and scheduler
+fencing. It does not by itself provide:
+
+- a highly available PostgreSQL, Redis, or object-store deployment;
+- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
 - autoscaling policy;
-- managed TLS certificate issuance and renewal;
-- measured multi-replica and failover integration evidence.
+- central logs, metrics, traces, or alert routing;
+- managed ingress certificates;
+- automatic reconciliation of every possible module side effect;
+- a service-level availability guarantee.
 
-These are separate production capabilities. The local load balancer and
-replica model provide the contract they should implement, but they do not by
-themselves make a cluster highly available.
+Those are deployment and module-adoption requirements. Before claiming high
+availability, drill replica loss, rolling replacement, session continuity, job
+redelivery, scheduler failover, migration exclusion, object-store outage, and a
+coordinated database/object/key restore. Recovery rules and evidence are
+defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).