govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md

Installation And Deployment Architecture

Goal

A supported GovOPlaN installation starts with one downloaded, verified bootstrap artifact. The administrator answers a bounded set of questions and receives a working base system. Re-running the same tool repairs or reconfigures that installation instead of creating unrelated state.

The canonical product journey remains System Administrator Lifecycle User Story. This document defines the deployer boundary and the first executable slice.

First Executable Slice

tools/deployment/govoplan-deploy.py is a standard-library-only deployment compiler and reconciler. It can be tested without installing GovOPlaN itself.

It currently supports:

• evaluation and self-hosted profiles;
• managed or external PostgreSQL;
• managed, external, or evaluation-only disabled Redis;
• disabled mail, an external relay declaration, or an evaluation-only GreenMail service;
• durable local file storage, managed single-node Garage S3, or external S3-compatible storage;
• an explicit HAProxy service that load-balances configured WebUI and API replicas without access to the Docker socket;
• declarative API, WebUI, and worker replica counts while keeping migrations and the scheduler singleton;
• Core, base, or full initial module selections;
• deterministic Compose JSON accepted by Compose v2;
• generated secrets stored in a private 0600 file;
• service-specific environment allowlists so infrastructure containers do not receive unrelated application credentials;
• plan, render, doctor, status, apply, Kubernetes export, operation history, and bounded recovery commands;
• an installation lock, migration-before-start ordering, readiness polling, and an applied-state receipt;
• a durable hash-chained deployment journal captured before runtime mutation;
• PostgreSQL advisory serialization for Core and module migrations;
• runtime initialization that waits for exact configured migration heads without mutating schema;
• runtime node registration, heartbeats, drain state, and a fenced scheduler;
• idempotent reconfiguration that preserves generated secrets;
• a keyed environment fingerprint that detects private binding changes without writing secret values to plans or receipts;
• host CPU, memory, disk, entropy, architecture, Docker daemon, Compose, listen-port, and external endpoint preflight checks;
• service removal without implicit data-volume deletion.

Create a local evaluation bundle:

./.venv/bin/python tools/deployment/govoplan-deploy.py init \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --postgres managed \
  --redis managed \
  --storage garage \
  --mail test-mail \
  --api-replicas 2 \
  --web-replicas 2 \
  --worker-replicas 2 \
  --module-set base

Inspect the generated intent and host requirements:

./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
  --directory /tmp/govoplan-evaluation

Change a component without rotating existing generated secrets:

./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --redis external \
  --redis-url 'rediss://:password@redis.example.org:6379/0'

The private installation directory contains:

File	Purpose
installation.json	Versioned, non-secret desired state
secrets.env	Deployment-local secrets and external service bindings
compose.json	Deterministic generated Compose definition
garage.toml	Non-secret managed Garage server configuration
load-balancer.cfg	Non-secret HAProxy WebUI/API discovery configuration
plan.json	Latest desired-state diff and readiness findings
receipt.json	Last successfully applied immutable identities
distribution-manifest.json	Canonical signed runtime/image selection adopted by the installer
distribution-keyring.json	Explicitly installed public trust anchor for runtime releases
applied-state/	Checksum-verified snapshot of the last healthy deployment bundle
operations/<id>/	Private hash-chained deployment progress and recovery evidence
kubernetes.json	Optional stateless multi-host Kubernetes export
.deployment.lock	Same-host operation exclusion

The specification contract is installation-spec.schema.json.

Build the same dependency-free tool as one downloadable artifact:

./.venv/bin/python tools/deployment/build-deployer-zipapp.py \
  --output /tmp/govoplan-deploy.pyz
python /tmp/govoplan-deploy.pyz --help

To exercise reconciliation with locally available evaluation images:

python /tmp/govoplan-deploy.pyz init \
  --non-interactive \
  --directory /tmp/govoplan-evaluation \
  --profile evaluation \
  --api-image local/govoplan-api:test \
  --web-image local/govoplan-web:test
python /tmp/govoplan-deploy.pyz apply \
  --directory /tmp/govoplan-evaluation \
  --allow-unverified-images \
  --skip-pull

Those images must already contain the selected module set. The override exists only to exercise local orchestration before release artifacts exist; it is rejected for self-hosted.

Runtime Distribution Boundary

The protected Runtime Distribution workflow builds GovOPlaN wheels first, resolves architecture-specific third-party wheels into offline wheelhouses, and then assembles the API images with pip --no-index. The target host never clones Git repositories and neither runtime image performs network package installation. Separate amd64/arm64 API and WebUI images are joined into OCI indexes and run as non-root identities. The release assets include CycloneDX application SBOMs, SLSA-style provenance, exact composition evidence, the single-file deployer, its detached Ed25519 signature, and a signed, expiring distribution manifest.

The manifest contract is runtime-distribution-manifest.schema.json, and its separately distributed trust-anchor contract is runtime-distribution-keyring.schema.json. Publication is immutable: an existing Gitea release asset must have the same size and SHA-256 digest or publication fails.

Adopt a downloaded or prefetched release only after obtaining the manifest digest and trusted keyring through the documented independent channel:

python3 govoplan-deploy.pyz verify-release \
  --directory /srv/govoplan/installation \
  --manifest ./distribution-manifest.json \
  --manifest-sha256 "$(cut -d' ' -f1 distribution-manifest.json.sha256)" \
  --trusted-keyring ./distribution-keyring.json \
  --adopt

doctor and apply rehash both stored files, re-run OpenSSL Ed25519 verification, enforce channel/expiry/revocation, compare every selected image, and prove that all enabled module ids occur in the signed image composition. An offline image index can bind prefetched OCI archives to the same exact image references and archive hashes; mutable tags or incomplete bundles are rejected.

Current Production Gates

The tool deliberately reports blockers instead of pretending the source tree is a production distribution:

1. First publication. The protected workflow and fail-closed artifact contracts are implemented, but a release operator must configure the Gitea registry/release tokens and runtime Ed25519 key, publish the first pinned release, and retain its amd64/arm64 readiness evidence.
2. First administrator. Production needs a one-time, restricted enrollment identity. The development bootstrap must not be enabled in production.
3. Image/module composition. The deployer now enforces the signed composition. A selected module not shipped by that release cannot be enabled.
4. Deployment agent. Web updates need a separate privileged reconciler with a typed command allowlist. The API and browser must never receive the Docker socket or arbitrary shell access.
5. Ingress and certificates. The managed HAProxy service provides HTTP load balancing inside the deployment boundary; it does not issue or renew certificates. A self-hosted profile still needs an explicit choice between an existing reverse proxy and a supported managed ingress, including trusted-proxy boundaries, TLS certificate issuance, renewal, and health probing through the public route.

apply --allow-unverified-images is therefore restricted to the evaluation profile. It explicitly acknowledges both mutable image identities and unverified image/module composition. It is a local test escape hatch, not a production setting.

Component Choices

PostgreSQL

managed creates a persistent PostgreSQL container and private generated credentials. external requires an explicit DATABASE_URL; switching from managed to external cannot reuse the old postgres Docker hostname accidentally.

Interactive entry hides external URLs because they commonly contain credentials. For unattended automation, provide them through a protected operator mechanism and avoid storing secret-bearing flags in shell history.

Production policy should support external managed databases and local managed PostgreSQL equally at the application boundary. Backup, point-in-time recovery, high availability, and major-version upgrades remain deployment properties.

Redis

managed creates an authenticated, append-only Redis container. external requires an explicit REDIS_URL. disabled is evaluation-only and disables workers while recording the single-process login-throttle risk acknowledgement.

doctor performs a bounded TCP connection check for external PostgreSQL, Redis, and S3 endpoints. This verifies DNS, routing, and that the port accepts a connection; it is not an authentication or semantic health check.

Production base installations include Redis because durable queues, distributed throttling, notifications, scheduled work, and transactional event delivery must survive API restarts.

Mail

The first slice distinguishes:

• disabled;
• external-relay, which records the infrastructure decision but leaves Mail server/credential creation as a visible post-install task;
• test-mail, an evaluation-only GreenMail service.

A bundled production mail server is intentionally not a default. Operating one requires DNS, reverse DNS, TLS, DKIM, SPF, DMARC, reputation, abuse handling, queue monitoring, and upgrade policy. A later profile may support an operator-selected MTA/relay, but it must expose these requirements rather than presenting a container as a complete mail service.

File Storage

local uses a durable Compose volume and is appropriate for one-host installations. garage provisions Garage 2.3 in its supported single-node bootstrap mode, generates a private application key and bucket, and connects the Files S3 backend to the exact installer-owned internal endpoint. The managed trust marker cannot authorize another S3 host. Garage metadata and object data use separate persistent volumes. s3 requires an external endpoint, region, access key, secret key, and bucket values. Self-hosted external S3 endpoints must be clean HTTPS origins. The generated runtime explicitly sets FILE_STORAGE_S3_ENDPOINT_TRUSTED=true for that operator-selected endpoint. The trust flag is not accepted for local storage and cannot be combined with installer-managed Garage trust.

Local storage must be included in backup and restore drills. Horizontal API or worker scale-out requires shared/object storage. The managed Garage profile is persistent but has no data redundancy; availability-sensitive installations must use a tested multi-node Garage cluster or another external S3 service.

Load Balancing And Replicas

The generated Compose topology publishes only load-balancer. HAProxy uses Docker DNS service discovery to distribute public traffic across WebUI replicas and WebUI API proxy traffic across API replicas. The WebUI and API services do not publish host ports. HAProxy has no Docker socket and discovers only the bounded replica slots rendered into load-balancer.cfg.

Replica counts are desired state:

./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
  --directory /tmp/govoplan-evaluation \
  --api-replicas 3 \
  --web-replicas 2 \
  --worker-replicas 4
./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
  --directory /tmp/govoplan-evaluation

Workers are queue consumers, so they are scaled through Redis rather than put behind an HTTP load balancer. Migrations are serialized with a deployment-wide PostgreSQL advisory lock. The Celery scheduler is run under a renewable, fencing-token lease. Multiple API replicas are rejected when Redis is disabled because distributed throttling and queued work cannot then be shared correctly.

This is same-host scaling. Docker Compose uses a bridge network and does not place containers on another machine. See Scaling And Multi-Host Deployment for the supported topology and promotion path.

Reconfiguration Semantics

installation.json is desired state. receipt.json is the last successfully applied state. plan compares their canonical hashes and service sets.

• Adding a managed component creates its service and persistent volume.
• Removing a component removes its service container on apply.
• Volumes are retained by default; deleting data requires a separate, deliberately destructive workflow.
• Existing generated credentials are retained unless an explicit future rotate operation is requested.
• Private configuration changes are represented by a keyed fingerprint in the plan and receipt; plaintext values are never copied there.
• Managed-to-external transitions require the new endpoint in the same operation.
• Migrations run as a one-shot service before API/worker replacement.
• API, worker, and scheduler start commands wait for exact configured migration heads; only the migration command is permitted to change schema.
• API and worker replicas register their software/module composition and heartbeat in PostgreSQL. Ops can request and cancel a node drain.
• The first upgrade from a direct WebUI host port stops that legacy WebUI container immediately before HAProxy claims the same endpoint.
• Health must recover before a new receipt and applied-state snapshot are committed.

Every apply operation is journalled before image pulls or runtime mutation. A failure before migration may restore a verified previous bundle. Once migration starts, recovery is forward-only unless an independently verified database backup is restored. See Recovery And Rollback Guarantees.

Production updates still need an operator-provided database backup/restore gate, database compatibility declaration, image signature verification, and deployment-specific drain policy. The deployment journal proves its own actions; it does not manufacture backup evidence.

Stateless Kubernetes Runtime

render-kubernetes exports the application tier for a standard orchestrator. It requires external PostgreSQL, Redis, and S3 and emits no stateful service or secret value:

python tools/deployment/govoplan-deploy.py render-kubernetes \
  --directory /srv/govoplan/default \
  --namespace govoplan \
  --secret-name govoplan-runtime

The output includes a release-specific migration Job, database-head wait init containers, API readiness/liveness probes, rolling Deployments, Services, Pod disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply the named Secret through the cluster's secret manager and review ingress proxy CIDRs before deployment. Detailed rollout and scaling rules live in Scaling And Multi-Host Deployment.

Recovery Commands

List durable deployment operations:

python tools/deployment/govoplan-deploy.py operations \
  --directory /srv/govoplan/default

Recover a selected failed operation after reviewing its stage evidence:

python tools/deployment/govoplan-deploy.py recover \
  --directory /srv/govoplan/default \
  --operation-id <operation-id>

The command reports whether it restored the pre-migration applied bundle, requires forward recovery, or needs manual intervention. Add --apply only after that decision has been reviewed.

Web Update Boundary

The intended update path is:

1. Ops reads the non-secret installation receipt and reports management mode, current release, component health, and update availability.
2. An authorized administrator asks Core to create a typed deployment request, for example reconcile_release or rollback_release.
3. Core persists the reviewed immutable plan, actor, expected current receipt, and idempotency key.
4. A separately deployed, narrow deployment agent claims the request.
5. The agent verifies signatures/digests, acquires a fenced deployment lock, backs up, pulls, migrates, reconciles, probes health, and writes evidence.
6. Ops presents durable progress and the resulting receipt.

The agent owns container-runtime access. It accepts no command strings from the browser and has no domain-data permissions. Installations managed by Kubernetes, systemd, or another external orchestrator expose read-only status and an export of the reviewed update recipe instead of a non-functional update button.

Distribution Workflow

The downloadable entry point is a release asset. Obtain the zipapp, detached signature, checksum, and trusted public keyring through independently authenticated paths before execution:

curl --proto '=https' --tlsv1.2 --fail --location \
  https://git.add-ideas.de/GovOPlaN/govoplan/releases/download/vX.Y.Z/govoplan-deploy.pyz \
  --output govoplan-deploy.pyz
sha256sum --check govoplan-deploy.pyz.sha256
openssl pkeyutl -verify -pubin -inkey runtime-release-public.pem -rawin \
  -in govoplan-deploy.pyz -sigfile govoplan-deploy.pyz.sig
python3 govoplan-deploy.pyz init

The zipapp has no GovOPlaN package dependency. It accepts a bounded HTTPS manifest or a prefetched file, requires an independently supplied SHA-256 digest and explicit trusted keyring, and executes OpenSSL with a fixed argument vector for Ed25519 verification. It never evaluates downloaded shell text or accepts an arbitrary command string.

Verification

Run the focused tests:

./.venv/bin/python -m unittest -v tests.test_deployment_installer

The tests cover signed release adoption, tamper/expiry/revocation/unknown-key rejection, architecture composition, offline image integrity, profile restrictions, secret persistence, external endpoint requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy discovery configuration, Compose service selection, secret non-disclosure, service-specific environment isolation, private file modes, external endpoint preflight, first-plan generation, apply ordering, receipt idempotency, hash-chained recovery journals, migration recovery boundaries, and stateless Kubernetes rendering.