govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

Installation And Deployment Architecture

Goal

A supported GovOPlaN installation starts with one downloaded, verified bootstrap artifact. The administrator answers a bounded set of questions and receives a working base system. Re-running the same tool repairs or reconfigures that installation instead of creating unrelated state.

The canonical product journey remains System Administrator Lifecycle User Story. This document defines the deployer boundary and the first executable slice.

First Executable Slice

tools/deployment/govoplan-deploy.py is a standard-library-only deployment compiler and reconciler. It can be tested without installing GovOPlaN itself.

It currently supports:

• evaluation and self-hosted profiles;
• managed or external PostgreSQL;
• managed, external, or evaluation-only disabled Redis;
• disabled mail, an external relay declaration, or an evaluation-only GreenMail service;
• durable local file storage, managed single-node Garage S3, or external S3-compatible storage;
• an explicit HAProxy service that load-balances configured WebUI and API replicas without access to the Docker socket;
• declarative API, WebUI, and worker replica counts while keeping migrations and the scheduler singleton;
• Core, base, or full initial module selections;
• deterministic Compose JSON accepted by Compose v2;
• generated secrets stored in a private 0600 file;
• service-specific environment allowlists so infrastructure containers do not receive unrelated application credentials;
• plan, render, doctor, status, apply, Kubernetes export, operation history, and bounded recovery commands;
• an installation lock, migration-before-start ordering, readiness polling, and an applied-state receipt;
• a durable hash-chained deployment journal captured before runtime mutation;
• PostgreSQL advisory serialization for Core and module migrations;
• runtime initialization that waits for exact configured migration heads without mutating schema;
• runtime node registration, heartbeats, drain state, and a fenced scheduler;
• idempotent reconfiguration that preserves generated secrets;
• a keyed environment fingerprint that detects private binding changes without writing secret values to plans or receipts;
• host CPU, memory, disk, entropy, architecture, Docker daemon, Compose, listen-port, and external endpoint preflight checks;
• service removal without implicit data-volume deletion.

Create a local evaluation bundle:

./.venv/bin/python tools/deployment/govoplan-deploy.py init \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --postgres managed \
  --redis managed \
  --storage garage \
  --mail test-mail \
  --api-replicas 2 \
  --web-replicas 2 \
  --worker-replicas 2 \
  --module-set base

Inspect the generated intent and host requirements:

./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
  --directory /tmp/govoplan-evaluation

Change a component without rotating existing generated secrets:

./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --redis external \
  --redis-url 'rediss://:password@redis.example.org:6379/0'

The private installation directory contains:

File	Purpose
installation.json	Versioned, non-secret desired state
secrets.env	Deployment-local secrets and external service bindings
compose.json	Deterministic generated Compose definition
garage.toml	Non-secret managed Garage server configuration
load-balancer.cfg	Non-secret HAProxy WebUI/API discovery configuration
Caddyfile	Non-secret managed-ingress route and ACME policy
existing-proxy.json	Exact upstream, trusted-source, header, and health contract for an operator-owned proxy
plan.json	Latest desired-state diff and readiness findings
receipt.json	Last successfully applied immutable identities
distribution-manifest.json	Canonical signed runtime/image selection adopted by the installer
distribution-keyring.json	Explicitly installed public trust anchor for runtime releases
backup-evidence.json	Signed provider-neutral coordinated backup and isolated-restore receipt
backup-keyring.json	Explicit public trust anchor for backup evidence authorities
backup-verification.json	Sanitized local verification/adoption receipt
applied-state/	Checksum-verified snapshot of the last healthy deployment bundle
operations/<id>/	Private hash-chained deployment progress and recovery evidence
kubernetes.json	Optional stateless multi-host Kubernetes export
.deployment.lock	Same-host operation exclusion

The specification contract is installation-spec.schema.json.

Build the same dependency-free tool as one downloadable artifact:

./.venv/bin/python tools/deployment/build-deployer-zipapp.py \
  --output /tmp/govoplan-deploy.pyz
python /tmp/govoplan-deploy.pyz --help

To exercise reconciliation with locally available evaluation images:

python /tmp/govoplan-deploy.pyz init \
  --non-interactive \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --api-image local/govoplan-api:test \
  --web-image local/govoplan-web:test
python /tmp/govoplan-deploy.pyz apply \
  --directory /tmp/govoplan-evaluation \
  --allow-unverified-images \
  --skip-pull

Those images must already contain the selected module set. The override exists only to exercise local orchestration before release artifacts exist; it is rejected for self-hosted.

Runtime Distribution Boundary

The protected Runtime Distribution workflow builds GovOPlaN wheels first, resolves architecture-specific third-party wheels into offline wheelhouses, and then assembles the API images with pip --no-index. The target host never clones Git repositories and neither runtime image performs network package installation. Separate amd64/arm64 API and WebUI images are joined into OCI indexes and run as non-root identities. The release assets include CycloneDX application SBOMs, SLSA-style provenance, exact composition evidence, the single-file deployer, its detached Ed25519 signature, and a signed, expiring distribution manifest. Evidence generation and signing run through the workflow's isolated release Python environment so their cryptographic tooling is explicit and independent of packages preinstalled in the Actions runner. The API image points Core at the migration scripts installed from the verified wheel under /opt/govoplan/runtime/govoplan_core_runtime; migrations therefore do not depend on a source checkout or the build host's Python installation scheme. Before publication, the exact amd64 and arm64 image manifests each run release migrations against the pinned PostgreSQL image, reach API and WebUI readiness as non-root/read-only processes, and complete a task through the pinned Redis image and packaged worker. Sanitized per-platform smoke receipts are retained as immutable release assets. PostgreSQL and Redis indexes are resolved to untagged platform-child digests before each smoke run. This keeps the evidence architecture-specific and avoids retargeting one local Docker tag between incompatible platforms. The CI host registers arm64 execution with an explicitly supplied, digest-pinned tonistiigi/binfmt image immediately before the smoke. This privileged helper is confined to the release runner and is never part of a GovOPlaN target deployment or its runtime image set. Because QEMU user-mode execution triggers Redis's arm64 host-kernel COW guard, the arm64 smoke suppresses only ARM64-COW-BUG while persistence, snapshots, and append-only files are disabled. Target Redis services never inherit this test-only option. The smoke also proves a bounded post-migration table contract and aborts as soon as a required container exits, rather than allowing a dead process to consume the full readiness timeout.

Ingress acceptance streams generated configuration into Docker-managed volumes before starting the read-only containers. It therefore also works when an Actions job reaches a host or remote Docker daemon through a mounted socket; the drill never assumes that a job-container path is visible to that daemon. The drill allocates explicit loopback-only host ports and verifies Docker's host binding configuration, avoiding daemon-specific random-port shorthand behavior. Because an Actions job and deployment containers may be Docker siblings, functional HTTP/TLS checks run from the digest-pinned API image on the deployment network instead of assuming the Docker host is job-local. The dispatch-only Runtime Ingress Drill workflow exposes the same bounded check independently so ingress changes can be diagnosed before an immutable runtime publication; it accepts only digest-pinned Caddy, HAProxy, and API images and has no push trigger. The official Caddy binary carries the NET_BIND_SERVICE file capability. The managed-ingress container therefore drops every capability and adds back only NET_BIND_SERVICE; otherwise Linux rejects the binary at execve before its high-port configuration can start. no-new-privileges, a read-only root filesystem, and non-privileged container ports remain enforced. The bounded setup helper writes only generated public configuration as root so it can initialize a new volume; the actual HAProxy process retains the image's non-root identity and runs read-only with all capabilities dropped.

The manifest contract is runtime-distribution-manifest.schema.json, and its separately distributed trust-anchor contract is runtime-distribution-keyring.schema.json. Publication is immutable: an existing Gitea release asset must have the same size and SHA-256 digest or publication fails.

Adopt a downloaded or prefetched release only after obtaining the manifest digest and trusted keyring through the documented independent channel:

python3 govoplan-deploy.pyz verify-release \
  --directory /srv/govoplan/installation \
  --manifest ./distribution-manifest.json \
  --manifest-sha256 "$(cut -d' ' -f1 distribution-manifest.json.sha256)" \
  --trusted-keyring ./distribution-keyring.json \
  --adopt

doctor and apply rehash both stored files, re-run OpenSSL Ed25519 verification, enforce channel/expiry/revocation, compare every selected image, and prove that all enabled module ids occur in the signed image composition. An offline image index can bind prefetched OCI archives to the same exact image references and archive hashes; mutable tags or incomplete bundles are rejected.

Current Production Gates

The tool deliberately reports blockers instead of pretending the source tree is a production distribution:

1. First publication. The protected workflow and fail-closed artifact contracts are implemented, but a release operator must configure the Gitea registry/release tokens and runtime Ed25519 key, publish the first pinned release, and retain its amd64/arm64 readiness evidence.
2. First administrator. Production needs a one-time, restricted enrollment identity. The development bootstrap must not be enabled in production.
3. Image/module composition. The deployer now enforces the signed composition. A selected module not shipped by that release cannot be enabled.
4. Deployment agent. Web updates need a separate privileged reconciler with a typed command allowlist. The API and browser must never receive the Docker socket or arbitrary shell access.
5. Ingress reachability evidence. Managed Caddy ingress and the existing-proxy contract are implemented. A production claim still requires running doctor from the target host after public DNS/firewall changes and retaining the first successful container drill and public TLS/readiness evidence.

apply --allow-unverified-images is therefore restricted to the evaluation profile. It explicitly acknowledges both mutable image identities and unverified image/module composition. It is a local test escape hatch, not a production setting.

Component Choices

PostgreSQL

managed creates a persistent PostgreSQL container and private generated credentials. external requires an explicit DATABASE_URL; switching from managed to external cannot reuse the old postgres Docker hostname accidentally.

Interactive entry hides external URLs because they commonly contain credentials. For unattended automation, provide them through a protected operator mechanism and avoid storing secret-bearing flags in shell history.

Production policy should support external managed databases and local managed PostgreSQL equally at the application boundary. Backup, point-in-time recovery, high availability, and major-version upgrades remain deployment properties.

Redis

managed creates an authenticated, append-only Redis container. external requires an explicit REDIS_URL. disabled is evaluation-only and disables workers while recording the single-process login-throttle risk acknowledgement.

doctor performs a bounded TCP connection check for external PostgreSQL, Redis, and S3 endpoints. This verifies DNS, routing, and that the port accepts a connection; it is not an authentication or semantic health check.

Production base installations include Redis because durable queues, distributed throttling, notifications, scheduled work, and transactional event delivery must survive API restarts.

Mail

The first slice distinguishes:

• disabled;
• external-relay, which records the infrastructure decision but leaves Mail server/credential creation as a visible post-install task;
• test-mail, an evaluation-only GreenMail service.

A bundled production mail server is intentionally not a default. Operating one requires DNS, reverse DNS, TLS, DKIM, SPF, DMARC, reputation, abuse handling, queue monitoring, and upgrade policy. A later profile may support an operator-selected MTA/relay, but it must expose these requirements rather than presenting a container as a complete mail service.

File Storage

local uses a durable Compose volume and is appropriate for one-host installations. garage provisions Garage 2.3 in its supported single-node bootstrap mode, generates a private application key and bucket, and connects the Files S3 backend to the exact installer-owned internal endpoint. The managed trust marker cannot authorize another S3 host. Garage metadata and object data use separate persistent volumes. s3 requires an external endpoint, region, access key, secret key, and bucket values. Self-hosted external S3 endpoints must be clean HTTPS origins. The generated runtime explicitly sets FILE_STORAGE_S3_ENDPOINT_TRUSTED=true for that operator-selected endpoint. The trust flag is not accepted for local storage and cannot be combined with installer-managed Garage trust.

Local storage must be included in backup and restore drills. Horizontal API or worker scale-out requires shared/object storage. The managed Garage profile is persistent but has no data redundancy; availability-sensitive installations must use a tested multi-node Garage cluster or another external S3 service.

Load Balancing And Replicas

The generated Compose topology publishes only load-balancer for local or existing-proxy profiles. With managed ingress, only Caddy publishes host ports and HAProxy remains private. HAProxy uses Docker DNS service discovery to distribute public traffic across WebUI replicas and WebUI API proxy traffic across API replicas. The WebUI and API services do not publish host ports. HAProxy has no Docker socket and discovers only the bounded replica slots rendered into load-balancer.cfg.

Replica counts are desired state:

./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --api-replicas 3 \
  --web-replicas 2 \
  --worker-replicas 4
./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
  --directory /tmp/govoplan-evaluation

Workers are queue consumers, so they are scaled through Redis rather than put behind an HTTP load balancer. Migrations are serialized with a deployment-wide PostgreSQL advisory lock. The Celery scheduler is run under a renewable, fencing-token lease. Multiple API replicas are rejected when Redis is disabled because distributed throttling and queued work cannot then be shared correctly.

Public Ingress And TLS

A self-hosted installation is fail-closed until one of these boundaries is selected:

• existing-proxy publishes HAProxy at listen.address:listen.port and emits existing-proxy.json. The operator-owned proxy must use the recorded host, upstream, and health paths. Only the exact CIDRs listed with repeated --trusted-proxy-cidr values may supply X-Forwarded-* headers. Public proxy addresses must be /32 or /128; private ranges are limited to /24 or narrower for IPv4 and /64 or narrower for IPv6.
• managed publishes Caddy on the selected HTTP/HTTPS ports, redirects HTTP to HTTPS, obtains and renews certificates through ACME, and keeps certificate material exclusively in the private caddy-data and caddy-config volumes. The application containers receive no ACME account or TLS private keys.

Example existing-proxy configuration:

python govoplan-deploy.py configure \
  --directory /srv/govoplan \
  --ingress existing-proxy \
  --trusted-proxy-cidr 172.20.0.7/32

Example managed configuration:

python govoplan-deploy.py configure \
  --directory /srv/govoplan \
  --ingress managed \
  --acme-email operator@example.org

Before managed ingress starts, public A/AAAA records must resolve to the target and inbound TCP 80/443 must reach it. Existing-proxy mode additionally requires the public proxy and valid certificate to be reachable before apply. After a successful receipt, doctor reports DNS resolution, certificate validity and remaining lifetime, public /health/ready, and the private HAProxy/WebUI path as separate checks. Reconfiguration retains the certificate volumes; bundle rollback never deletes or exposes their contents. Include both Caddy volumes in coordinated backup and restore evidence.

This is same-host scaling. Docker Compose uses a bridge network and does not place containers on another machine. See Scaling And Multi-Host Deployment for the supported topology and promotion path.

Reconfiguration Semantics

installation.json is desired state. receipt.json is the last successfully applied state. plan compares their canonical hashes and service sets.

• Adding a managed component creates its service and persistent volume.
• Removing a component removes its service container on apply.
• Volumes are retained by default; deleting data requires a separate, deliberately destructive workflow.
• Existing generated credentials are retained unless an explicit future rotate operation is requested.
• Private configuration changes are represented by a keyed fingerprint in the plan and receipt; plaintext values are never copied there.
• Managed-to-external transitions require the new endpoint in the same operation.
• Migrations run as a one-shot service before API/worker replacement.
• API, worker, and scheduler start commands wait for exact configured migration heads; only the migration command is permitted to change schema.
• API and worker replicas register their software/module composition and heartbeat in PostgreSQL. Ops can request and cancel a node drain.
• The first upgrade from a direct WebUI host port stops that legacy WebUI container immediately before HAProxy claims the same endpoint.
• Health must recover before a new receipt and applied-state snapshot are committed.

Every apply operation is journalled before image pulls or runtime mutation. A failure before migration may restore a verified previous bundle. Once migration starts, recovery is forward-only unless an independently verified database backup is restored. See Recovery And Rollback Guarantees.

Production updates still need operator/provider-created coordinated backup and restore evidence, a database compatibility declaration, and a deployment-specific drain policy. The deployer now verifies and enforces the signed evidence before migration, but does not manufacture backups or receive provider administration credentials. See Backup And Restore Evidence.

Stateless Kubernetes Runtime

render-kubernetes exports the application tier for a standard orchestrator. It requires external PostgreSQL, Redis, and S3 and emits no stateful service or secret value:

python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime

The output includes a release-specific migration Job, database-head wait init containers, API readiness/liveness probes, rolling Deployments, Services, Pod disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply the named Secret through the cluster's secret manager and review ingress proxy CIDRs before deployment. A release-changing export requires adopted backup evidence and carries only its sanitized digest and identifiers as Job annotations. Detailed rollout and scaling rules live in Scaling And Multi-Host Deployment.

Recovery Commands

List durable deployment operations:

python tools/deployment/govoplan-deploy.py operations \
  --directory /srv/govoplan/default

Recover a selected failed operation after reviewing its stage evidence:

python tools/deployment/govoplan-deploy.py recover \
  --directory /srv/govoplan/default \
  --operation-id <operation-id>

The command reports whether it restored the pre-migration applied bundle, requires forward recovery, or needs manual intervention. Add --apply only after that decision has been reviewed.

Web Update Boundary

The intended update path is:

1. Ops reads the non-secret installation receipt and reports management mode, current release, component health, and update availability.
2. An authorized administrator asks Core to create a typed deployment request, for example reconcile_release or rollback_release.
3. Core persists the reviewed immutable plan, actor, expected current receipt, and idempotency key.
4. A separately deployed, narrow deployment agent claims the request.
5. The agent verifies signatures/digests, acquires a fenced deployment lock, backs up, pulls, migrates, reconciles, probes health, and writes evidence.
6. Ops presents durable progress and the resulting receipt.

The agent owns container-runtime access. It accepts no command strings from the browser and has no domain-data permissions. Installations managed by Kubernetes, systemd, or another external orchestrator expose read-only status and an export of the reviewed update recipe instead of a non-functional update button.

Distribution Workflow

The downloadable entry point is a reproducible release asset: sorted source paths, fixed ZIP metadata, fixed compression settings, and identical source bytes produce an identical zipapp regardless of checkout timestamps. Obtain the zipapp, detached signature, checksum, and trusted public keyring through independently authenticated paths before execution:

curl --proto '=https' --tlsv1.2 --fail --location \
  https://git.add-ideas.de/GovOPlaN/govoplan/releases/download/vX.Y.Z/govoplan-deploy.pyz \
  --output govoplan-deploy.pyz
sha256sum --check govoplan-deploy.pyz.sha256
python3 - <<'PY'
import json
from pathlib import Path

keyring = json.loads(Path("distribution-keyring.json").read_text())
active = [key for key in keyring["keys"] if key["status"] == "active"]
if len(active) != 1:
    raise SystemExit("expected exactly one active runtime release key")
Path("runtime-release-public.pem").write_text(active[0]["public_key_pem"])
PY
openssl pkeyutl -verify -pubin -inkey runtime-release-public.pem -rawin \
  -in govoplan-deploy.pyz -sigfile govoplan-deploy.pyz.sig
python3 govoplan-deploy.pyz init

The zipapp has no GovOPlaN package dependency. It accepts a bounded HTTPS manifest or a prefetched file, requires an independently supplied SHA-256 digest and explicit trusted keyring, and executes OpenSSL with a fixed argument vector for Ed25519 verification. It never evaluates downloaded shell text or accepts an arbitrary command string.

Verification

Run the focused tests:

./.venv/bin/python -m unittest -v tests.test_deployment_installer

The tests cover signed release adoption, tamper/expiry/revocation/unknown-key rejection, architecture composition, offline image integrity, profile restrictions, secret persistence, external endpoint requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy discovery configuration, Compose service selection, secret non-disclosure, service-specific environment isolation, private file modes, external endpoint preflight, first-plan generation, apply ordering, receipt idempotency, hash-chained recovery journals, migration recovery boundaries, and stateless Kubernetes rendering.