govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

# Installation And Deployment Architecture

## Goal

A supported GovOPlaN installation starts with one downloaded, verified
bootstrap artifact. The administrator answers a bounded set of questions and
receives a working base system. Re-running the same tool repairs or
reconfigures that installation instead of creating unrelated state.

The canonical product journey remains
[System Administrator Lifecycle User Story](SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md).
This document defines the deployer boundary and the first executable slice.

## First Executable Slice

`tools/deployment/govoplan-deploy.py` is a standard-library-only deployment
compiler and reconciler. It can be tested without installing GovOPlaN itself.

It currently supports:

- evaluation and self-hosted profiles;
- managed or external PostgreSQL;
- managed, external, or evaluation-only disabled Redis;
- disabled mail, an external relay declaration, or an evaluation-only
  GreenMail service;
- durable local file storage, managed single-node Garage S3, or external
  S3-compatible storage;
- an explicit HAProxy service that load-balances configured WebUI and API
  replicas without access to the Docker socket;
- declarative API, WebUI, and worker replica counts while keeping migrations
  and the scheduler singleton;
- Core, base, or full initial module selections;
- deterministic Compose JSON accepted by Compose v2;
- generated secrets stored in a private `0600` file;
- service-specific environment allowlists so infrastructure containers do not
  receive unrelated application credentials;
- plan, render, doctor, status, apply, Kubernetes export, operation history,
  and bounded recovery commands;
- an installation lock, migration-before-start ordering, readiness polling,
  and an applied-state receipt;
- a durable hash-chained deployment journal captured before runtime mutation;
- PostgreSQL advisory serialization for Core and module migrations;
- runtime initialization that waits for exact configured migration heads
  without mutating schema;
- runtime node registration, heartbeats, drain state, and a fenced scheduler;
- idempotent reconfiguration that preserves generated secrets;
- a keyed environment fingerprint that detects private binding changes without
  writing secret values to plans or receipts;
- host CPU, memory, disk, entropy, architecture, Docker daemon, Compose,
  listen-port, and external endpoint preflight checks;
- service removal without implicit data-volume deletion.

Create a local evaluation bundle:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py init \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --postgres managed \
  --redis managed \
  --storage garage \
  --mail test-mail \
  --api-replicas 2 \
  --web-replicas 2 \
  --worker-replicas 2 \
  --module-set base
```

Inspect the generated intent and host requirements:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
  --directory /tmp/govoplan-evaluation
```

Change a component without rotating existing generated secrets:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --redis external \
  --redis-url 'rediss://:password@redis.example.org:6379/0'
```

The private installation directory contains:

| File | Purpose |
| --- | --- |
| `installation.json` | Versioned, non-secret desired state |
| `secrets.env` | Deployment-local secrets and external service bindings |
| `compose.json` | Deterministic generated Compose definition |
| `garage.toml` | Non-secret managed Garage server configuration |
| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
| `plan.json` | Latest desired-state diff and readiness findings |
| `receipt.json` | Last successfully applied immutable identities |
| `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
| `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
| `.deployment.lock` | Same-host operation exclusion |

The specification contract is
[`installation-spec.schema.json`](installation-spec.schema.json).

Build the same dependency-free tool as one downloadable artifact:

```sh
./.venv/bin/python tools/deployment/build-deployer-zipapp.py \
  --output /tmp/govoplan-deploy.pyz
python /tmp/govoplan-deploy.pyz --help
```

To exercise reconciliation with locally available evaluation images:

```sh
python /tmp/govoplan-deploy.pyz init \
  --non-interactive \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --api-image local/govoplan-api:test \
  --web-image local/govoplan-web:test
python /tmp/govoplan-deploy.pyz apply \
  --directory /tmp/govoplan-evaluation \
  --allow-unverified-images \
  --skip-pull
```

Those images must already contain the selected module set. The override exists
only to exercise local orchestration before release artifacts exist; it is
rejected for `self-hosted`.

## Runtime Distribution Boundary

The protected `Runtime Distribution` workflow builds GovOPlaN wheels first,
resolves architecture-specific third-party wheels into offline wheelhouses, and
then assembles the API images with `pip --no-index`. The target host never
clones Git repositories and neither runtime image performs network package
installation. Separate amd64/arm64 API and WebUI images are joined into OCI
indexes and run as non-root identities. The release assets include CycloneDX
application SBOMs, SLSA-style provenance, exact composition evidence, the
single-file deployer, its detached Ed25519 signature, and a signed, expiring
distribution manifest.

The manifest contract is
[`runtime-distribution-manifest.schema.json`](runtime-distribution-manifest.schema.json),
and its separately distributed trust-anchor contract is
[`runtime-distribution-keyring.schema.json`](runtime-distribution-keyring.schema.json).
Publication is immutable: an existing Gitea release asset must have the same
size and SHA-256 digest or publication fails.

Adopt a downloaded or prefetched release only after obtaining the manifest
digest and trusted keyring through the documented independent channel:

```sh
python3 govoplan-deploy.pyz verify-release \
  --directory /srv/govoplan/installation \
  --manifest ./distribution-manifest.json \
  --manifest-sha256 "$(cut -d' ' -f1 distribution-manifest.json.sha256)" \
  --trusted-keyring ./distribution-keyring.json \
  --adopt
```

`doctor` and `apply` rehash both stored files, re-run OpenSSL Ed25519
verification, enforce channel/expiry/revocation, compare every selected image,
and prove that all enabled module ids occur in the signed image composition.
An offline image index can bind prefetched OCI archives to the same exact image
references and archive hashes; mutable tags or incomplete bundles are rejected.

## Current Production Gates

The tool deliberately reports blockers instead of pretending the source tree is
a production distribution:

1. **First publication.** The protected workflow and fail-closed artifact
   contracts are implemented, but a release operator must configure the Gitea
   registry/release tokens and runtime Ed25519 key, publish the first pinned
   release, and retain its amd64/arm64 readiness evidence.
3. **First administrator.** Production needs a one-time, restricted enrollment
   identity. The development bootstrap must not be enabled in production.
4. **Image/module composition.** The deployer now enforces the signed
   composition. A selected module not shipped by that release cannot be
   enabled.
5. **Deployment agent.** Web updates need a separate privileged reconciler with
   a typed command allowlist. The API and browser must never receive the Docker
   socket or arbitrary shell access.
6. **Ingress and certificates.** The managed HAProxy service provides HTTP
   load balancing inside the deployment boundary; it does not issue or renew
   certificates. A self-hosted profile still needs an explicit choice
   between an existing reverse proxy and a supported managed ingress, including
   trusted-proxy boundaries, TLS certificate issuance, renewal, and health
   probing through the public route.

`apply --allow-unverified-images` is therefore restricted to the evaluation
profile. It explicitly acknowledges both mutable image identities and
unverified image/module composition. It is a local test escape hatch, not a
production setting.

## Component Choices

### PostgreSQL

`managed` creates a persistent PostgreSQL container and private generated
credentials. `external` requires an explicit `DATABASE_URL`; switching from
managed to external cannot reuse the old `postgres` Docker hostname
accidentally.

Interactive entry hides external URLs because they commonly contain
credentials. For unattended automation, provide them through a protected
operator mechanism and avoid storing secret-bearing flags in shell history.

Production policy should support external managed databases and local managed
PostgreSQL equally at the application boundary. Backup, point-in-time recovery,
high availability, and major-version upgrades remain deployment properties.

### Redis

`managed` creates an authenticated, append-only Redis container. `external`
requires an explicit `REDIS_URL`. `disabled` is evaluation-only and disables
workers while recording the single-process login-throttle risk acknowledgement.

`doctor` performs a bounded TCP connection check for external PostgreSQL,
Redis, and S3 endpoints. This verifies DNS, routing, and that the port accepts a
connection; it is not an authentication or semantic health check.

Production base installations include Redis because durable queues, distributed
throttling, notifications, scheduled work, and transactional event delivery
must survive API restarts.

### Mail

The first slice distinguishes:

- `disabled`;
- `external-relay`, which records the infrastructure decision but leaves Mail
  server/credential creation as a visible post-install task;
- `test-mail`, an evaluation-only GreenMail service.

A bundled production mail server is intentionally not a default. Operating one
requires DNS, reverse DNS, TLS, DKIM, SPF, DMARC, reputation, abuse handling,
queue monitoring, and upgrade policy. A later profile may support an
operator-selected MTA/relay, but it must expose these requirements rather than
presenting a container as a complete mail service.

### File Storage

`local` uses a durable Compose volume and is appropriate for one-host
installations. `garage` provisions Garage 2.3 in its supported single-node
bootstrap mode, generates a private application key and bucket, and connects
the Files S3 backend to the exact installer-owned internal endpoint. The
managed trust marker cannot authorize another S3 host.
Garage metadata and object data use separate persistent volumes. `s3` requires
an external endpoint, region, access key, secret key, and bucket values.
Self-hosted external S3 endpoints must be clean HTTPS origins. The generated
runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that
operator-selected endpoint. The trust flag is not accepted for local storage
and cannot be combined with installer-managed Garage trust.

Local storage must be included in backup and restore drills. Horizontal API or
worker scale-out requires shared/object storage. The managed Garage profile is
persistent but has no data redundancy; availability-sensitive installations
must use a tested multi-node Garage cluster or another external S3 service.

### Load Balancing And Replicas

The generated Compose topology publishes only `load-balancer`. HAProxy uses
Docker DNS service discovery to distribute public traffic across WebUI replicas
and WebUI API proxy traffic across API replicas. The WebUI and API services do
not publish host ports. HAProxy has no Docker socket and discovers only the
bounded replica slots rendered into `load-balancer.cfg`.

Replica counts are desired state:

```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --api-replicas 3 \
  --web-replicas 2 \
  --worker-replicas 4
./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
  --directory /tmp/govoplan-evaluation
```

Workers are queue consumers, so they are scaled through Redis rather than put
behind an HTTP load balancer. Migrations are serialized with a deployment-wide
PostgreSQL advisory lock. The Celery scheduler is run under a renewable,
fencing-token lease. Multiple API replicas are rejected when Redis is disabled
because distributed throttling and queued work cannot then be shared correctly.

This is same-host scaling. Docker Compose uses a bridge network and does not
place containers on another machine. See
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) for
the supported topology and promotion path.

## Reconfiguration Semantics

`installation.json` is desired state. `receipt.json` is the last successfully
applied state. `plan` compares their canonical hashes and service sets.

- Adding a managed component creates its service and persistent volume.
- Removing a component removes its service container on apply.
- Volumes are retained by default; deleting data requires a separate,
  deliberately destructive workflow.
- Existing generated credentials are retained unless an explicit future rotate
  operation is requested.
- Private configuration changes are represented by a keyed fingerprint in the
  plan and receipt; plaintext values are never copied there.
- Managed-to-external transitions require the new endpoint in the same
  operation.
- Migrations run as a one-shot service before API/worker replacement.
- API, worker, and scheduler start commands wait for exact configured migration
  heads; only the migration command is permitted to change schema.
- API and worker replicas register their software/module composition and
  heartbeat in PostgreSQL. Ops can request and cancel a node drain.
- The first upgrade from a direct WebUI host port stops that legacy WebUI
  container immediately before HAProxy claims the same endpoint.
- Health must recover before a new receipt and applied-state snapshot are
  committed.

Every apply operation is journalled before image pulls or runtime mutation. A
failure before migration may restore a verified previous bundle. Once migration
starts, recovery is forward-only unless an independently verified database
backup is restored. See
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).

Production updates still need an operator-provided database backup/restore
gate, database compatibility declaration, image signature verification, and
deployment-specific drain policy. The deployment journal proves its own
actions; it does not manufacture backup evidence.

## Stateless Kubernetes Runtime

`render-kubernetes` exports the application tier for a standard orchestrator.
It requires external PostgreSQL, Redis, and S3 and emits no stateful service or
secret value:

```sh
python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime
```

The output includes a release-specific migration Job, database-head wait init
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
the named Secret through the cluster's secret manager and review ingress proxy
CIDRs before deployment. Detailed rollout and scaling rules live in
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).

## Recovery Commands

List durable deployment operations:

```sh
python tools/deployment/govoplan-deploy.py operations \
  --directory /srv/govoplan/default
```

Recover a selected failed operation after reviewing its stage evidence:

```sh
python tools/deployment/govoplan-deploy.py recover \
  --directory /srv/govoplan/default \
  --operation-id <operation-id>
```

The command reports whether it restored the pre-migration applied bundle,
requires forward recovery, or needs manual intervention. Add `--apply` only
after that decision has been reviewed.

## Web Update Boundary

The intended update path is:

1. Ops reads the non-secret installation receipt and reports management mode,
   current release, component health, and update availability.
2. An authorized administrator asks Core to create a typed deployment request,
   for example `reconcile_release` or `rollback_release`.
3. Core persists the reviewed immutable plan, actor, expected current receipt,
   and idempotency key.
4. A separately deployed, narrow deployment agent claims the request.
5. The agent verifies signatures/digests, acquires a fenced deployment lock,
   backs up, pulls, migrates, reconciles, probes health, and writes evidence.
6. Ops presents durable progress and the resulting receipt.

The agent owns container-runtime access. It accepts no command strings from the
browser and has no domain-data permissions. Installations managed by Kubernetes,
systemd, or another external orchestrator expose read-only status and an export
of the reviewed update recipe instead of a non-functional update button.

## Distribution Workflow

The downloadable entry point is a release asset. Obtain the zipapp, detached
signature, checksum, and trusted public keyring through independently
authenticated paths before execution:

```sh
curl --proto '=https' --tlsv1.2 --fail --location \
  https://git.add-ideas.de/GovOPlaN/govoplan/releases/download/vX.Y.Z/govoplan-deploy.pyz \
  --output govoplan-deploy.pyz
sha256sum --check govoplan-deploy.pyz.sha256
openssl pkeyutl -verify -pubin -inkey runtime-release-public.pem -rawin \
  -in govoplan-deploy.pyz -sigfile govoplan-deploy.pyz.sig
python3 govoplan-deploy.pyz init
```

The zipapp has no GovOPlaN package dependency. It accepts a bounded HTTPS
manifest or a prefetched file, requires an independently supplied SHA-256
digest and explicit trusted keyring, and executes OpenSSL with a fixed argument
vector for Ed25519 verification. It never evaluates downloaded shell text or
accepts an arbitrary command string.

## Verification

Run the focused tests:

```sh
./.venv/bin/python -m unittest -v tests.test_deployment_installer
```

The tests cover signed release adoption, tamper/expiry/revocation/unknown-key
rejection, architecture composition, offline image integrity, profile
restrictions, secret persistence, external endpoint
requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
discovery configuration, Compose service selection, secret non-disclosure,
service-specific environment isolation, private file modes, external endpoint
preflight, first-plan generation, apply ordering, receipt idempotency,
hash-chained recovery journals, migration recovery boundaries, and stateless
Kubernetes rendering.