feat: implement institutional governance and recovery architecture

docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

@@ -34,9 +34,15 @@ It currently supports:
 - generated secrets stored in a private `0600` file;
 - service-specific environment allowlists so infrastructure containers do not
   receive unrelated application credentials;
-- plan, render, doctor, status, and apply commands;
+- plan, render, doctor, status, apply, Kubernetes export, operation history,
+  and bounded recovery commands;
 - an installation lock, migration-before-start ordering, readiness polling,
   and an applied-state receipt;
+- a durable hash-chained deployment journal captured before runtime mutation;
+- PostgreSQL advisory serialization for Core and module migrations;
+- runtime initialization that waits for exact configured migration heads
+  without mutating schema;
+- runtime node registration, heartbeats, drain state, and a fenced scheduler;
 - idempotent reconfiguration that preserves generated secrets;
 - a keyed environment fingerprint that detects private binding changes without
   writing secret values to plans or receipts;
@@ -87,6 +93,9 @@ The private installation directory contains:
 | `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
 | `plan.json` | Latest desired-state diff and readiness findings |
 | `receipt.json` | Last successfully applied immutable identities |
+| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
+| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
+| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
 | `.deployment.lock` | Same-host operation exclusion |
 
 The specification contract is
@@ -201,11 +210,13 @@ presenting a container as a complete mail service.
 installations. `garage` provisions Garage 2.3 in its supported single-node
 bootstrap mode, generates a private application key and bucket, and connects
 the Files S3 backend to the exact installer-owned internal endpoint. The
-deployment-only trust marker cannot authorize another S3 host; arbitrary
-external SDK endpoints remain fail-closed until peer pinning is implemented.
+managed trust marker cannot authorize another S3 host.
 Garage metadata and object data use separate persistent volumes. `s3` requires
 an external endpoint, region, access key, secret key, and bucket values.
-Self-hosted external S3 endpoints must use HTTPS.
+Self-hosted external S3 endpoints must be clean HTTPS origins. The generated
+runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that
+operator-selected endpoint. The trust flag is not accepted for local storage
+and cannot be combined with installer-managed Garage trust.
 
 Local storage must be included in backup and restore drills. Horizontal API or
 worker scale-out requires shared/object storage. The managed Garage profile is
@@ -233,9 +244,10 @@ Replica counts are desired state:
 ```
 
 Workers are queue consumers, so they are scaled through Redis rather than put
-behind an HTTP load balancer. The migration runner and Celery scheduler remain
-singletons. Multiple API replicas are rejected when Redis is disabled because
-distributed throttling and queued work cannot then be shared correctly.
+behind an HTTP load balancer. Migrations are serialized with a deployment-wide
+PostgreSQL advisory lock. The Celery scheduler is run under a renewable,
+fencing-token lease. Multiple API replicas are rejected when Redis is disabled
+because distributed throttling and queued work cannot then be shared correctly.
 
 This is same-host scaling. Docker Compose uses a bridge network and does not
 place containers on another machine. See
@@ -258,13 +270,66 @@ applied state. `plan` compares their canonical hashes and service sets.
 - Managed-to-external transitions require the new endpoint in the same
   operation.
 - Migrations run as a one-shot service before API/worker replacement.
+- API, worker, and scheduler start commands wait for exact configured migration
+  heads; only the migration command is permitted to change schema.
+- API and worker replicas register their software/module composition and
+  heartbeat in PostgreSQL. Ops can request and cancel a node drain.
 - The first upgrade from a direct WebUI host port stops that legacy WebUI
   container immediately before HAProxy claims the same endpoint.
-- Health must recover before a new receipt is committed.
+- Health must recover before a new receipt and applied-state snapshot are
+  committed.
 
-This is sufficient for one-host reconciliation. Production updates additionally
-need backup/restore gates, maintenance/drain state, database compatibility
-windows, image signature verification, and rollback/forward-recovery policy.
+Every apply operation is journalled before image pulls or runtime mutation. A
+failure before migration may restore a verified previous bundle. Once migration
+starts, recovery is forward-only unless an independently verified database
+backup is restored. See
+[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
+
+Production updates still need an operator-provided database backup/restore
+gate, database compatibility declaration, image signature verification, and
+deployment-specific drain policy. The deployment journal proves its own
+actions; it does not manufacture backup evidence.
+
+## Stateless Kubernetes Runtime
+
+`render-kubernetes` exports the application tier for a standard orchestrator.
+It requires external PostgreSQL, Redis, and S3 and emits no stateful service or
+secret value:
+
+```sh
+python tools/deployment/govoplan-deploy.py render-kubernetes \
+  --directory /srv/govoplan/default \
+  --namespace govoplan \
+  --secret-name govoplan-runtime
+```
+
+The output includes a release-specific migration Job, database-head wait init
+containers, API readiness/liveness probes, rolling Deployments, Services, Pod
+disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
+the named Secret through the cluster's secret manager and review ingress proxy
+CIDRs before deployment. Detailed rollout and scaling rules live in
+[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
+
+## Recovery Commands
+
+List durable deployment operations:
+
+```sh
+python tools/deployment/govoplan-deploy.py operations \
+  --directory /srv/govoplan/default
+```
+
+Recover a selected failed operation after reviewing its stage evidence:
+
+```sh
+python tools/deployment/govoplan-deploy.py recover \
+  --directory /srv/govoplan/default \
+  --operation-id <operation-id>
+```
+
+The command reports whether it restored the pre-migration applied bundle,
+requires forward recovery, or needs manual intervention. Add `--apply` only
+after that decision has been reviewed.
 
 ## Web Update Boundary
 
@@ -317,5 +382,6 @@ The tests cover profile restrictions, secret persistence, external endpoint
 requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
 discovery configuration, Compose service selection, secret non-disclosure,
 service-specific environment isolation, private file modes, external endpoint
-preflight, first-plan generation, apply ordering, and receipt-based
-idempotency.
+preflight, first-plan generation, apply ordering, receipt idempotency,
+hash-chained recovery journals, migration recovery boundaries, and stateless
+Kubernetes rendering.