govoplan-core/docs/DEPLOYMENT_OPERATOR_GUIDE.md

GovOPlaN Deployment Operator Guide

This guide defines the current install/runtime configuration contract and the operator flow for a production-realistic self-hosted deployment. Keep secrets in the deployment environment or a secret manager; do not commit populated .env files.

Runtime Configuration Contract

Self-hosted installability follows the staged approach documented in SELF_HOSTED_INSTALLABILITY.md: generate an explicit env template, validate it, run production-like rehearsal with Compose-backed dependencies, then use the installer CLI/daemon for package mutation under maintenance mode.

Generate a deployment-local template:

cd /mnt/DATA/git/govoplan-core
./.venv/bin/python -m govoplan_core.commands.config env-template \
  --profile self-hosted \
  --generate-secrets \
  --output .env.self-hosted

Validate the active shell environment before migration or startup:

set -a
. .env.self-hosted
set +a
./.venv/bin/python -m govoplan_core.commands.config validate --profile self-hosted

Required Runtime Identity

Setting	Required outside dev	Purpose
APP_ENV	yes	Runtime profile. Use prod, staging, or a deployment-specific value outside local development.
MASTER_KEY_B64	yes	Fernet key or base64 encoded 32-byte key used for encrypted module secrets. Rotate through an explicit operator plan.
DATABASE_URL	yes	SQLAlchemy database URL for core and installed modules. SQLite is supported for dev/small installs; PostgreSQL is the preferred production target.
ENABLED_MODULES	yes	Comma-separated startup module set. Keep tenancy,access enabled; keep admin enabled for operator UI.

Generate a local key for a new non-production environment:

python - <<'PY'
from cryptography.fernet import Fernet
print(Fernet.generate_key().decode())
PY

Database And Migrations

Setting	Default	Notes
DATABASE_URL	postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev	Local development and production-like profiles use PostgreSQL. Use GOVOPLAN_DEV_DATABASE_BACKEND=sqlite only for disposable SQLite runs.
DEV_AUTO_MIGRATE_ENABLED	true	Dev convenience only. Production should run migration commands explicitly during deployment.
DEV_BOOTSTRAP_ENABLED	false	Dev bootstrap only. govoplan_core.devserver and scripts/launch-dev.sh default it to true; use controlled first-admin creation outside dev.

Operator rule: take a database backup before applying migrations or destructive module retirement. For non-SQLite databases, configure deployment-specific backup/restore hooks for the module installer.

PostgreSQL Production Target

PostgreSQL is the primary development and production target. SQLite remains supported only for tiny disposable profiles and unit-test style smoke runs. Production/staging deployments should use a managed PostgreSQL database and explicit migration commands.

Install the server extra so the psycopg driver is available:

cd /mnt/DATA/git/govoplan-core
./.venv/bin/python -m pip install -r requirements-release.txt

Example runtime database URLs:

export DATABASE_URL='postgresql+psycopg://govoplan:change-me@db.example.internal:5432/govoplan'
export GOVOPLAN_DATABASE_URL_PGTOOLS='postgresql://govoplan:change-me@db.example.internal:5432/govoplan'

Use the SQLAlchemy URL for GovOPlaN. Use the pg-tools URL for pg_dump, pg_restore, and psql; these tools do not understand the postgresql+psycopg:// driver marker.

Bootstrap or upgrade the schema explicitly during deployment:

export APP_ENV=prod
export ENABLED_MODULES=tenancy,access,admin,policy,audit,campaigns,files,mail,calendar,docs,ops
./.venv/bin/python -m govoplan_core.commands.init_db \
  --database-url "$DATABASE_URL"

Backup and restore-check before migration-bearing package changes:

pg_dump --format=custom \
  --file "$PWD/runtime/govoplan-$(date +%Y%m%d%H%M%S).dump" \
  "$GOVOPLAN_DATABASE_URL_PGTOOLS"
pg_restore --list "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump" >/dev/null

Restore a checked backup to the target database:

pg_restore --clean --if-exists \
  --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" \
  "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump"

For local development, create the host database described in dev/postgres/README.md, then run:

cd /mnt/DATA/git/govoplan-core
./.venv/bin/python -m govoplan_core.commands.init_db \
  --database-url postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev \
  --with-dev-data
./.venv/bin/python -m govoplan_core.devserver --smoke --no-reload
scripts/launch-dev.sh

For disposable local validation against a throwaway PostgreSQL instance, use the bundled PostgreSQL testbed:

cd /mnt/DATA/git/govoplan-core/dev/postgres
cp .env.example .env
docker compose --env-file .env up -d

cd /mnt/DATA/git/govoplan-core
set -a
. dev/postgres/.env
set +a
./.venv/bin/python scripts/postgres-integration-check.py \
  --database-url "$GOVOPLAN_POSTGRES_DATABASE_URL" \
  --reset-schema

The integration check runs migrations and startup smoke checks across the standard module permutations. --reset-schema is destructive and belongs only on throwaway databases.

Broker And Workers

Setting	Default	Notes
REDIS_URL	redis://redis:6379/0	Celery broker/result backend when async workers are enabled.
CELERY_ENABLED	false	Local/dev can send synchronously. Production campaign delivery should run workers and set this to true.
CELERY_QUEUES	send_email,append_sent,default	Queue list expected by worker/process manager definitions.

Worker command:

python -m celery -A govoplan_core.celery_app:celery worker \
  --queues send_email,append_sent,default \
  --loglevel INFO

Storage

Setting	Default	Notes
FILE_STORAGE_BACKEND	local	Use local for dev/small deployments; use object storage when files must scale independently.
FILE_STORAGE_LOCAL_ROOT	runtime/files	Must live on durable storage and be backed up when FILE_STORAGE_BACKEND=local.
FILE_STORAGE_LOCAL_FALLBACK_ROOTS	empty	Read-only fallback roots for migrated local files.
FILE_STORAGE_S3_ENDPOINT_URL	empty	Object-store endpoint for the files module.
FILE_STORAGE_S3_REGION	empty	Object-store region.
FILE_STORAGE_S3_ACCESS_KEY_ID	empty	Secret; inject through deployment environment.
FILE_STORAGE_S3_SECRET_ACCESS_KEY	empty	Secret; inject through deployment environment.
FILE_STORAGE_S3_BUCKET	files	Managed-file object bucket.

Legacy S3_* settings remain for older storage paths but new deployments should prefer FILE_STORAGE_*.

HTTP, Cookies, And Base URLs

Setting	Default	Notes
CORS_ORIGINS	local dev origins	Set to the exact WebUI origins in staging/production.
AUTH_SESSION_COOKIE_NAME	configured default	Change only through a controlled rollout because it logs users out.
AUTH_CSRF_COOKIE_NAME	configured default	Must match WebUI/API deployment.
AUTH_COOKIE_SECURE	false	Set true behind HTTPS.
AUTH_COOKIE_SAMESITE	lax	Use a stricter value only after testing login and CSRF flows.
AUTH_COOKIE_DOMAIN	empty	Set only when the API and WebUI intentionally share a parent domain.

Public URLs are currently supplied by deployment/reverse-proxy configuration and module settings. Do not hardcode them in core; configuration packages should ask for portal, WebUI, postbox, and notification URLs when they become relevant.

Module Catalogs, Licenses, And Trust Roots

Setting	Purpose
GOVOPLAN_MODULE_PACKAGE_CATALOG_URL or GOVOPLAN_MODULE_PACKAGE_CATALOG	Module package catalog source.
GOVOPLAN_MODULE_PACKAGE_CATALOG_TRUSTED_KEYS_FILE	Preferred production keyring path.
GOVOPLAN_MODULE_PACKAGE_CATALOG_APPROVED_CHANNEL	Approved catalog channel, for example stable.
GOVOPLAN_LICENSE_TRUSTED_KEYS_FILE	Trusted license issuer keyring path.
GOVOPLAN_LICENSE_ENFORCEMENT	Enables license enforcement when set to true.

Trust roots are deployment-managed and should not be editable through the running WebUI.

Mail Test Credentials

Dedicated SMTP/IMAP test credentials belong to the mail/campaign test-bed configuration, not the core runtime contract. Store them in a local ignored .env file for the test bed or in CI secrets. Required values are:

• SMTP host, port, TLS mode, username, password, and envelope/from address.
• IMAP host, port, TLS mode, username, password, and append folder.
• At least one recipient mailbox that is safe for automated send tests.

First Deployment Flow

1. Create an environment file or secret set with the runtime contract above.
2. Install the tagged core and module packages from requirements-release.txt.
3. Build the WebUI from webui/package.release.json or deploy a prebuilt artifact from the same release tag.
4. Run database migrations with the target DATABASE_URL.
5. Create the first tenant and system owner through the controlled bootstrap or one-time admin command for the deployment.
6. Start the API service with govoplan_core.server.app:app.
7. Start workers when CELERY_ENABLED=true.
8. Start the WebUI/reverse proxy and verify CORS/cookie settings.
9. Open Admin > System > Modules, verify enabled modules, and save desired module state if it differs from ENABLED_MODULES.
10. Run health checks:

curl -fsS http://127.0.0.1:8000/health
curl -fsS http://127.0.0.1:8000/api/v1/platform/modules

Authenticated health details require system:settings:read:

curl -fsS -H "X-API-Key: $GOVOPLAN_HEALTH_API_KEY" \
  http://127.0.0.1:8000/health/details

Production-Like Dev Profile

Use this profile to verify deployment behavior without publishing packages or using real production credentials. The canonical launcher keeps API, worker, and WebUI code in the editable repositories while Docker provides PostgreSQL and Redis:

cd /mnt/DATA/git/govoplan-core
scripts/launch-production-like-dev.sh

The helper wrapper provides explicit lifecycle commands:

scripts/production-like-dev.sh validate-config
scripts/production-like-dev.sh seed
scripts/production-like-dev.sh start
scripts/production-like-dev.sh stop
scripts/production-like-dev.sh reset --yes

The launcher uses dev/production-like/.env when present, otherwise the checked in .env.example. It runs:

• PostgreSQL on 127.0.0.1:55433
• Redis on 127.0.0.1:56379
• explicit ENABLED_MODULES
• explicit migrations and --with-dev-data bootstrap
• API via the module-aware devserver
• a Celery worker for send_email,append_sent,default
• WebUI through the Vite dev server
• durable local files under runtime/production-like/files

This profile validates explicit migration execution, config loading, module discovery, route aggregation, local storage paths, Redis broker connectivity, worker heartbeats, and health/readiness startup without real production credentials. It does not replace a managed PostgreSQL/Redis/WebUI/worker deployment test.

To stop PostgreSQL and Redis when the launcher exits:

GOVOPLAN_STOP_PROFILE_DEPENDENCIES_ON_EXIT=1 scripts/launch-production-like-dev.sh

Module Install/Uninstall Operations

Use Admin > System > Modules for planning. The running API server validates and queues install plans; it does not run pip/npm or restart itself from an HTTP request. Package mutation belongs to the trusted installer CLI/daemon in an operator shell while maintenance mode is active.

Preflight from the server shell:

govoplan-module-installer --format shell

Apply a prepared plan directly from a controlled shell:

govoplan-module-installer --apply --build-webui

For production-like runs, prefer supervised mode with migrations, database backup/restore hooks, restart commands, and health checks:

govoplan-module-installer \
  --supervise \
  --migrate \
  --restart-command 'systemctl restart govoplan-api' \
  --restart-command 'systemctl restart govoplan-worker' \
  --health-url http://127.0.0.1:8000/health \
  --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \
  --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"'

To let the admin UI submit work without executing package managers inside the API process, run the daemon in a separate operator shell:

govoplan-module-installer \
  --daemon \
  --migrate \
  --build-webui \
  --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \
  --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --health-url http://127.0.0.1:8000/health \
  --restart-command '<restart govoplan server>'

The daemon claims one queued request at a time and writes request/run records below runtime/module-installer. For process-manager one-shot usage or tests, use --daemon-once. Check daemon status with:

govoplan-module-installer --daemon-status --format json

The installer uses a runtime lock, snapshots pip freeze plus WebUI package files, writes a run record, and marks planned rows as applied only after all commands succeed. With --migrate, SQLite databases are backed up through SQLite's backup API; non-SQLite databases require --database-backup-command, --database-restore-check-command, and --database-restore-command.

Database hook commands receive:

• GOVOPLAN_INSTALLER_RUN_DIR
• GOVOPLAN_DATABASE_URL
• GOVOPLAN_DATABASE_URL_PGTOOLS for PostgreSQL tools
• GOVOPLAN_DATABASE_BACKUP_PATH
• GOVOPLAN_DATABASE_BACKUP_METADATA

Avoid embedding secrets directly in commands; prefer environment variables, service credentials, or deployment-local secret injection.

Inspect installer history and lock state from the operator shell:

govoplan-module-installer --list-runs --format json
govoplan-module-installer --show-run <run-id> --format json
govoplan-module-installer --lock-status --format json
govoplan-module-installer --list-requests --format json
govoplan-module-installer --show-request <request-id> --format json
govoplan-module-installer --cancel-request <request-id> --format json
govoplan-module-installer --retry-request <request-id> --format json

Rollback uses the saved run snapshot:

govoplan-module-installer --rollback <run-id>
govoplan-module-installer --rollback <run-id> --database-restore-command '<override restore command>'

Uninstall is non-destructive by default. A planned uninstall row can set destroy_data: true to request destructive module retirement. The module must provide an automated retirement provider, and the installer snapshots the database before dropping module-owned tables.

Run the rollback drill before relying on installer automation in a new environment:

./.venv/bin/python scripts/module-installer-rollback-drill.py --format json

The drill uses temporary SQLite databases and simulated package commands. It does not install or uninstall real packages. It exercises:

• package command failure followed by supervised rollback;
• migration failure with a SQLite database snapshot;
• restart-command failure;
• health timeout after restart;
• destructive retirement executor failure with database rollback;
• PostgreSQL-style backup, restore-check, and restore hooks;
• daemon heartbeat, request queue claim/update, retry/cancel, and stale lock detection/removal.

See RELEASE_DEPENDENCIES.md for release package refs, migration baseline checks, catalog trust, signing, keyring, replay, and license operation.

Operator Checklist

• Runtime secrets are injected outside git.
• MASTER_KEY_B64 is set and backed up securely.
• Database backup and restore commands are tested.
• File/object storage is durable and backed up.
• CORS_ORIGINS and cookie settings match the deployed WebUI origin.
• Redis and workers are running before CELERY_ENABLED=true.
• Module catalog and license keyrings are pinned locally.
• Health endpoints are monitored.
• Test SMTP/IMAP credentials are non-production and isolated.
• Module installer rollback drill has passed in the deployment environment.