govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

# Installation And Deployment Architecture

## Goal

A supported GovOPlaN installation starts with one downloaded, verified
bootstrap artifact. The administrator answers a bounded set of questions and
receives a working base system. Re-running the same tool repairs or
reconfigures that installation instead of creating unrelated state.

The canonical product journey remains
[System Administrator Lifecycle User Story](SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md).
This document defines the deployer boundary and the first executable slice.
The execution, topology, component-ownership and assurance modes are defined
canonically in [Deployment Profiles](DEPLOYMENT_PROFILES.md). In particular,
the editable production-like developer launcher is distinct from both an
immutable Compose rehearsal and a supported one-host production deployment.

## First Executable Slice

`tools/deployment/govoplan-deploy.py` is a standard-library-only deployment
compiler and reconciler. It can be tested without installing GovOPlaN itself.

It currently supports:

- evaluation and self-hosted profiles;
- managed or external PostgreSQL;
- managed, external, or evaluation-only disabled Redis;
- disabled mail, an external relay declaration, or an evaluation-only
  GreenMail service;
- durable local file storage, managed single-node Garage S3, or external
  S3-compatible storage;
- an explicit HAProxy service that load-balances configured WebUI and API
  replicas without access to the Docker socket;
- declarative API, WebUI, and worker replica counts while keeping migrations
  and the scheduler singleton;
- Core, base, or full initial module selections;
- deterministic Compose JSON accepted by Compose v2;
- generated secrets stored in a private `0600` file;
- service-specific environment allowlists so infrastructure containers do not
  receive unrelated application credentials;
- plan, render, doctor, status, apply, Kubernetes export, operation history,
  and bounded recovery commands;
- an installation lock, migration-before-start ordering, readiness polling,
  and an applied-state receipt;
- a durable hash-chained deployment journal captured before runtime mutation;
- PostgreSQL advisory serialization for Core and module migrations;
- runtime initialization that waits for exact configured migration heads
  without mutating schema;
- runtime node registration, heartbeats, drain state, and a fenced scheduler;
- idempotent reconfiguration that preserves generated secrets;
- a keyed environment fingerprint that detects private binding changes without
  writing secret values to plans or receipts;
- host CPU, memory, disk, entropy, architecture, Docker daemon, Compose,
  listen-port, and external endpoint preflight checks;
- service removal without implicit data-volume deletion.

Create a local evaluation bundle:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py init \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --postgres managed \
  --redis managed \
  --storage garage \
  --mail test-mail \
  --api-replicas 2 \
  --web-replicas 2 \
  --worker-replicas 2 \
  --module-set base
```

Inspect the generated intent and host requirements:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
  --directory /tmp/govoplan-evaluation
```

Change a component without rotating existing generated secrets:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --redis external \
  --redis-url 'rediss://:password@redis.example.org:6379/0'
```

The private installation directory contains:

| File | Purpose |
| --- | --- |
| `installation.json` | Versioned, non-secret desired state |
| `secrets.env` | Deployment-local secrets and external service bindings |
| `compose.json` | Deterministic generated Compose definition |
| `garage.toml` | Non-secret managed Garage server configuration |
| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
| `Caddyfile` | Non-secret managed-ingress route and ACME policy |
| `existing-proxy.json` | Exact upstream, trusted-source, header, and health contract for an operator-owned proxy |
| `plan.json` | Latest desired-state diff and readiness findings |
| `receipt.json` | Last successfully applied immutable identities |
| `infrastructure-capabilities.json` | Deterministic non-secret capability states, endpoint metadata, secret references, consumers, and resumable post-install tasks |
| `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
| `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
| `backup-evidence.json` | Signed provider-neutral coordinated backup and isolated-restore receipt |
| `backup-keyring.json` | Explicit public trust anchor for backup evidence authorities |
| `backup-verification.json` | Sanitized local verification/adoption receipt |
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
| `.deployment.lock` | Same-host operation exclusion |

The specification contract is
[`installation-spec.schema.json`](installation-spec.schema.json).

Build the same dependency-free tool as one downloadable artifact:

```sh
./.venv/bin/python tools/deployment/build-deployer-zipapp.py \
  --output /tmp/govoplan-deploy.pyz
python /tmp/govoplan-deploy.pyz --help
```

To exercise reconciliation with locally available evaluation images:

```sh
python /tmp/govoplan-deploy.pyz init \
  --non-interactive \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --api-image local/govoplan-api:test \
  --web-image local/govoplan-web:test
python /tmp/govoplan-deploy.pyz apply \
  --directory /tmp/govoplan-evaluation \
  --allow-unverified-images \
  --skip-pull
```

Those images must already contain the selected module set. The override exists
only to exercise local orchestration before release artifacts exist; it is
rejected for `self-hosted`.

## Runtime Distribution Boundary

The protected `Runtime Distribution` workflow builds GovOPlaN wheels first,
resolves architecture-specific third-party wheels into offline wheelhouses, and
then assembles the API images with `pip --no-index`. The target host never
clones Git repositories and neither runtime image performs network package
installation. Separate amd64/arm64 API and WebUI images are joined into OCI
indexes and run as non-root identities. The release assets include CycloneDX
application SBOMs, SLSA-style provenance, exact composition evidence, the
single-file deployer, its detached Ed25519 signature, and a signed, expiring
distribution manifest. Evidence generation and signing run through the
workflow's isolated release Python environment so their cryptographic tooling
is explicit and independent of packages preinstalled in the Actions runner.
The API image points Core at the migration scripts installed from the verified
wheel under `/opt/govoplan/runtime/govoplan_core_runtime`; migrations therefore
do not depend on a source checkout or the build host's Python installation
scheme.
Before publication, the exact amd64 and arm64 image manifests each run release
migrations against the pinned PostgreSQL image, reach API and WebUI readiness
as non-root/read-only processes, and complete a task through the pinned Redis
image and packaged worker. Sanitized per-platform smoke receipts are retained
as immutable release assets.
PostgreSQL and Redis indexes are resolved to untagged platform-child digests
before each smoke run. This keeps the evidence architecture-specific and
avoids retargeting one local Docker tag between incompatible platforms.
The CI host registers arm64 execution with an explicitly supplied,
digest-pinned `tonistiigi/binfmt` image immediately before the smoke. This
privileged helper is confined to the release runner and is never part of a
GovOPlaN target deployment or its runtime image set.
Because QEMU user-mode execution triggers Redis's arm64 host-kernel COW guard,
the arm64 smoke suppresses only `ARM64-COW-BUG` while persistence, snapshots,
and append-only files are disabled. Target Redis services never inherit this
test-only option.
The smoke also proves a bounded post-migration table contract and aborts as
soon as a required container exits, rather than allowing a dead process to
consume the full readiness timeout.

Ingress acceptance streams generated configuration into Docker-managed
volumes before starting the read-only containers. It therefore also works when
an Actions job reaches a host or remote Docker daemon through a mounted socket;
the drill never assumes that a job-container path is visible to that daemon.
The drill allocates explicit loopback-only host ports and verifies Docker's
host binding configuration, avoiding daemon-specific random-port shorthand
behavior. Because an Actions job and deployment containers may be Docker
siblings, functional HTTP/TLS checks run from the digest-pinned API image on
the deployment network instead of assuming the Docker host is job-local.
The dispatch-only `Runtime Ingress Drill` workflow exposes the same bounded
check independently so ingress changes can be diagnosed before an immutable
runtime publication; it accepts only digest-pinned Caddy, HAProxy, and API
images and has no push trigger.
The official Caddy binary carries the `NET_BIND_SERVICE` file capability. The
managed-ingress container therefore drops every capability and adds back only
`NET_BIND_SERVICE`; otherwise Linux rejects the binary at `execve` before its
high-port configuration can start. `no-new-privileges`, a read-only root
filesystem, and non-privileged container ports remain enforced.
The bounded setup helper writes only generated public configuration as root so
it can initialize a new volume; the actual HAProxy process retains the image's
non-root identity and runs read-only with all capabilities dropped.

The manifest contract is
[`runtime-distribution-manifest.schema.json`](runtime-distribution-manifest.schema.json),
and its separately distributed trust-anchor contract is
[`runtime-distribution-keyring.schema.json`](runtime-distribution-keyring.schema.json).
Publication is immutable: an existing Gitea release asset must have the same
size and SHA-256 digest or publication fails.

Adopt a downloaded or prefetched release only after obtaining the manifest
digest and trusted keyring through the documented independent channel:

```sh
python3 govoplan-deploy.pyz verify-release \
  --directory /srv/govoplan/installation \
  --manifest ./distribution-manifest.json \
  --manifest-sha256 "$(cut -d' ' -f1 distribution-manifest.json.sha256)" \
  --trusted-keyring ./distribution-keyring.json \
  --adopt
```

`doctor` and `apply` rehash both stored files, re-run OpenSSL Ed25519
verification, enforce channel/expiry/revocation, compare every selected image,
and prove that all enabled module ids occur in the signed image composition.
An offline image index can bind prefetched OCI archives to the same exact image
references and archive hashes; mutable tags or incomplete bundles are rejected.

## Current Production Gates

The first immutable production-distribution baseline is published as
[`v0.1.14`](https://git.add-ideas.de/GovOPlaN/govoplan/releases/tag/v0.1.14)
from source commit `1f039dd39c1ce2672f4978c8abc6dff862ef1445`. Runtime
Distribution [run #459](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/459)
proved migrations, schema compatibility, non-root API/Web readiness, and worker
delivery/shutdown on both `linux/amd64` and `linux/arm64`. Its signed manifest
has SHA-256
`d703267e01855dee63200cb20921c91c3f95fbff550c8ca76e9a35cba3f69109`
and pins these runtime indexes:

- API: `git.add-ideas.de/govoplan/runtime-api@sha256:197ed01790986f2bc927eaa5d8348fa118702e5d2dc05feb851fc2643c23764a`
- WebUI: `git.add-ideas.de/govoplan/runtime-web@sha256:e936cca124f1fad29a067834cf17627d4c236410fdc3fa129e0ccb26b8193812`

The signed bootstrap has SHA-256
`1ff946fba82b0895d153b23352d06e30fe18388450dfd37fed6fb9912310efc5`
and key id `runtime-distribution-2026-01`. The managed-ingress boundary passed
the same publication run and the independently dispatchable Runtime Ingress
Drill [run #458](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/458).
Every later release must renew this evidence; the following target-specific
gates remain:

1. **First administrator.** Production needs a one-time, restricted enrollment
   identity. The development bootstrap must not be enabled in production.
2. **Image/module composition.** The deployer enforces the signed
   composition. A selected module not shipped by that release cannot be
   enabled.
3. **Deployment agent.** Web updates need a separate privileged reconciler with
   a typed command allowlist. The API and browser must never receive the Docker
   socket or arbitrary shell access.
4. **Target reachability evidence.** Managed Caddy ingress and the
   existing-proxy contract are implemented. A production claim still requires
   running `doctor` from the target host after public DNS/firewall changes and
   retaining public TLS/readiness evidence for that deployment.

`apply --allow-unverified-images` is therefore restricted to the evaluation
profile. It explicitly acknowledges both mutable image identities and
unverified image/module composition. It is a local test escape hatch, not a
production setting.

## Component Choices

### PostgreSQL

`managed` creates a persistent PostgreSQL container and private generated
credentials. `external` requires an explicit `DATABASE_URL`; switching from
managed to external cannot reuse the old `postgres` Docker hostname
accidentally.

Interactive entry hides external URLs because they commonly contain
credentials. For unattended automation, provide them through a protected
operator mechanism and avoid storing secret-bearing flags in shell history.

Production policy should support external managed databases and local managed
PostgreSQL equally at the application boundary. Backup, point-in-time recovery,
high availability, and major-version upgrades remain deployment properties.

### Redis

`managed` creates an authenticated, append-only Redis container. `external`
requires an explicit `REDIS_URL`. `disabled` is evaluation-only and disables
workers while recording the single-process login-throttle risk acknowledgement.

`doctor` performs a bounded TCP connection check for external PostgreSQL,
Redis, and S3 endpoints. This verifies DNS, routing, and that the port accepts a
connection; it is not an authentication or semantic health check.

Production base installations include Redis because durable queues, distributed
throttling, notifications, scheduled work, and transactional event delivery
must survive API restarts.

### Mail

The first slice distinguishes:

- `disabled`;
- `external-relay`, which records the infrastructure decision but leaves Mail
  server/credential creation as a visible post-install task;
- `test-mail`, an evaluation-only GreenMail service.

A bundled production mail server is intentionally not a default. Operating one
requires DNS, reverse DNS, TLS, DKIM, SPF, DMARC, reputation, abuse handling,
queue monitoring, and upgrade policy. A later profile may support an
operator-selected MTA/relay, but it must expose these requirements rather than
presenting a container as a complete mail service.

### File Storage

`local` uses a durable Compose volume and is appropriate for one-host
installations. `garage` provisions Garage 2.3 in its supported single-node
bootstrap mode, generates a private application key and bucket, and connects
the Files S3 backend to the exact installer-owned internal endpoint. The
managed trust marker cannot authorize another S3 host.
Garage metadata and object data use separate persistent volumes. `s3` requires
an external endpoint, region, access key, secret key, and bucket values.
Self-hosted external S3 endpoints must be clean HTTPS origins. The generated
runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that
operator-selected endpoint. The trust flag is not accepted for local storage
and cannot be combined with installer-managed Garage trust.

Local storage must be included in backup and restore drills. Horizontal API or
worker scale-out requires shared/object storage. The managed Garage profile is
persistent but has no data redundancy; availability-sensitive installations
must use a tested multi-node Garage cluster or another external S3 service.

### Load Balancing And Replicas

The generated Compose topology publishes only `load-balancer` for local or
existing-proxy profiles. With managed ingress, only Caddy publishes host ports
and HAProxy remains private. HAProxy uses
Docker DNS service discovery to distribute public traffic across WebUI replicas
and WebUI API proxy traffic across API replicas. The WebUI and API services do
not publish host ports. HAProxy has no Docker socket and discovers only the
bounded replica slots rendered into `load-balancer.cfg`.

Replica counts are desired state:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --api-replicas 3 \
  --web-replicas 2 \
  --worker-replicas 4
./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
  --directory /tmp/govoplan-evaluation
```

Workers are queue consumers, so they are scaled through Redis rather than put
behind an HTTP load balancer. Migrations are serialized with a deployment-wide
PostgreSQL advisory lock. The Celery scheduler is run under a renewable,
fencing-token lease. Multiple API replicas are rejected when Redis is disabled
because distributed throttling and queued work cannot then be shared correctly.

### Public Ingress And TLS

A self-hosted installation is fail-closed until one of these boundaries is
selected:

- `existing-proxy` publishes HAProxy at `listen.address:listen.port` and emits
  `existing-proxy.json`. The operator-owned proxy must use the recorded host,
  upstream, and health paths. Only the exact CIDRs listed with repeated
  `--trusted-proxy-cidr` values may supply `X-Forwarded-*` headers. Public
  proxy addresses must be `/32` or `/128`; private ranges are limited to `/24`
  or narrower for IPv4 and `/64` or narrower for IPv6.
- `managed` publishes Caddy on the selected HTTP/HTTPS ports, redirects HTTP to
  HTTPS, obtains and renews certificates through ACME, and keeps certificate
  material exclusively in the private `caddy-data` and `caddy-config` volumes.
  The application containers receive no ACME account or TLS private keys.

Example existing-proxy configuration:

```sh
python govoplan-deploy.py configure \
  --directory /srv/govoplan \
  --ingress existing-proxy \
  --trusted-proxy-cidr 172.20.0.7/32
```

Example managed configuration:

```sh
python govoplan-deploy.py configure \
  --directory /srv/govoplan \
  --ingress managed \
  --acme-email operator@example.org
```

Before managed ingress starts, public A/AAAA records must resolve to the target
and inbound TCP 80/443 must reach it. Existing-proxy mode additionally requires
the public proxy and valid certificate to be reachable before apply. After a
successful receipt, `doctor` reports DNS resolution, certificate validity and
remaining lifetime, public `/health/ready`, and the private HAProxy/WebUI path
as separate checks. Reconfiguration retains the certificate volumes; bundle
rollback never deletes or exposes their contents. Include both Caddy volumes
in coordinated backup and restore evidence.

This is same-host scaling. Docker Compose uses a bridge network and does not
place containers on another machine. See
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) for
the supported topology and promotion path.

## Reconfiguration Semantics

`installation.json` is desired state. `receipt.json` is the last successfully
applied state. `plan` compares their canonical hashes, service sets, and
infrastructure capability projections.

- Adding a managed component creates its service and persistent volume.
- Removing a component removes its service container on apply.
- Replacing or removing a capability adds a review action that names the prior
  and desired state/source plus declared module consumers. This does not claim
  that the deployer can inspect module-owned database configuration; the
  operator must review that inventory before apply.
- Volumes are retained by default; deleting data requires a separate,
  deliberately destructive workflow.
- Existing generated credentials are retained unless an explicit future rotate
  operation is requested.
- Private configuration changes are represented by a keyed fingerprint in the
  plan and receipt; plaintext values are never copied there.
- Capability documents contain sanitized scheme/host/port metadata and stable
  `env:` references only. Credential values and secret-bearing URLs remain in
  `secrets.env` or module-owned credential envelopes.
- Managed-to-external transitions require the new endpoint in the same
  operation.
- Migrations run as a one-shot service before API/worker replacement.
- API, worker, and scheduler start commands wait for exact configured migration
  heads; only the migration command is permitted to change schema.
- API and worker replicas register their software/module composition and
  heartbeat in PostgreSQL. Ops can request and cancel a node drain.
- The first upgrade from a direct WebUI host port stops that legacy WebUI
  container immediately before HAProxy claims the same endpoint.
- Health must recover before a new receipt and applied-state snapshot are
  committed.

Compose mounts the capability document read-only into API and worker runtime
containers. The Kubernetes export projects the same document through a
dedicated ConfigMap and read-only file mount. Ops validates the bounded schema
before displaying configured, externally supplied, available-unconfigured, or
unavailable states and any pending post-install tasks.

Every apply operation is journalled before image pulls or runtime mutation. A
failure before migration may restore a verified previous bundle. Once migration
starts, recovery is forward-only unless an independently verified database
backup is restored. See
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).

Production updates still need operator/provider-created coordinated backup and
restore evidence, a database compatibility declaration, and a
deployment-specific drain policy. The deployer now verifies and enforces the
signed evidence before migration, but does not manufacture backups or receive
provider administration credentials. See
[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md).

## Stateless Kubernetes Runtime

`render-kubernetes` exports the application tier for a standard orchestrator.
It requires external PostgreSQL, Redis, and S3 and emits no stateful service or
secret value:

```sh
python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime
```

The output includes a release-specific migration Job, database-head wait init
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
the named Secret through the cluster's secret manager and review ingress proxy
CIDRs before deployment. A release-changing export requires adopted backup
evidence and carries only its sanitized digest and identifiers as Job
annotations. Detailed rollout and scaling rules live in
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).

## Recovery Commands

List durable deployment operations:

```sh
python tools/deployment/govoplan-deploy.py operations \
  --directory /srv/govoplan/default
```

Recover a selected failed operation after reviewing its stage evidence:

```sh
python tools/deployment/govoplan-deploy.py recover \
  --directory /srv/govoplan/default \
  --operation-id <operation-id>
```

The command reports whether it restored the pre-migration applied bundle,
requires forward recovery, or needs manual intervention. Add `--apply` only
after that decision has been reviewed.

## Web Update Boundary

The intended update path is:

1. Ops reads the non-secret installation receipt and reports management mode,
   current release, component health, and update availability.
2. An authorized administrator asks Core to create a typed deployment request,
   for example `reconcile_release` or `rollback_release`.
3. Core persists the reviewed immutable plan, actor, expected current receipt,
   and idempotency key.
4. A separately deployed, narrow deployment agent claims the request.
5. The agent verifies signatures/digests, acquires a fenced deployment lock,
   backs up, pulls, migrates, reconciles, probes health, and writes evidence.
6. Ops presents durable progress and the resulting receipt.

The agent owns container-runtime access. It accepts no command strings from the
browser and has no domain-data permissions. Installations managed by Kubernetes,
systemd, or another external orchestrator expose read-only status and an export
of the reviewed update recipe instead of a non-functional update button.

## Distribution Workflow

The downloadable entry point is a reproducible release asset: sorted source
paths, fixed ZIP metadata, fixed compression settings, and identical source
bytes produce an identical zipapp regardless of checkout timestamps. Obtain the
zipapp, detached signature, checksum, and trusted public keyring through
independently authenticated paths before execution:

```sh
curl --proto '=https' --tlsv1.2 --fail --location \
  https://git.add-ideas.de/GovOPlaN/govoplan/releases/download/vX.Y.Z/govoplan-deploy.pyz \
  --output govoplan-deploy.pyz
sha256sum --check govoplan-deploy.pyz.sha256
python3 - <<'PY'
import json
from pathlib import Path

keyring = json.loads(Path("distribution-keyring.json").read_text())
active = [key for key in keyring["keys"] if key["status"] == "active"]
if len(active) != 1:
    raise SystemExit("expected exactly one active runtime release key")
Path("runtime-release-public.pem").write_text(active[0]["public_key_pem"])
PY
openssl pkeyutl -verify -pubin -inkey runtime-release-public.pem -rawin \
  -in govoplan-deploy.pyz -sigfile govoplan-deploy.pyz.sig
python3 govoplan-deploy.pyz init
```

The zipapp has no GovOPlaN package dependency. It accepts a bounded HTTPS
manifest or a prefetched file, requires an independently supplied SHA-256
digest and explicit trusted keyring, and executes OpenSSL with a fixed argument
vector for Ed25519 verification. It never evaluates downloaded shell text or
accepts an arbitrary command string.

## Verification

Run the focused tests:

```sh
./.venv/bin/python -m unittest -v tests.test_deployment_installer
```

The tests cover signed release adoption, tamper/expiry/revocation/unknown-key
rejection, architecture composition, offline image integrity, profile
restrictions, secret persistence, external endpoint
requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
discovery configuration, Compose service selection, secret non-disclosure,
service-specific environment isolation, private file modes, external endpoint
preflight, first-plan generation, apply ordering, receipt idempotency,
hash-chained recovery journals, migration recovery boundaries, and stateless
Kubernetes rendering.