# GovOPlaN Deployment Operator Guide

This guide defines the current install/runtime configuration contract and the
operator flow for a production-realistic self-hosted deployment. Keep secrets in
the deployment environment or a secret manager; do not commit populated `.env`
files.

## Runtime Configuration Contract

Self-hosted installability follows the staged approach documented in
`SELF_HOSTED_INSTALLABILITY.md`: generate an explicit env template, validate it,
run production-like rehearsal with Compose-backed dependencies, then use the
installer CLI/daemon for package mutation under maintenance mode.

Generate a deployment-local template:

```bash
cd /mnt/DATA/git/govoplan
./.venv/bin/python -m govoplan_core.commands.config env-template \
  --profile self-hosted \
  --generate-secrets \
  --output .env.self-hosted
```

Validate the active shell environment before migration or startup:

```bash
set -a
. .env.self-hosted
set +a
./.venv/bin/python -m govoplan_core.commands.config validate --profile self-hosted
```

### Required Runtime Identity

| Setting | Required outside dev | Purpose |
| --- | --- | --- |
| `APP_ENV` | yes | Runtime profile. Use `prod`, `staging`, or a deployment-specific value outside local development. |
| `MASTER_KEY_B64` | yes | Fernet key or base64 encoded 32-byte deployment root used for encrypted module secrets and, when enabled, the Encryption module's local server-envelope provider. Rotate only through an explicit provider-aware migration plan. |
| `DATABASE_URL` | yes | SQLAlchemy database URL for core and installed modules. SQLite is supported for dev/small installs; PostgreSQL is the preferred production target. |
| `ENABLED_MODULES` | yes | Comma-separated startup module set. Keep `tenancy,access` enabled; keep `admin` enabled for operator UI. |

Generate a local key for a new non-production environment:

```bash
python - <<'PY'
from cryptography.fernet import Fernet
print(Fernet.generate_key().decode())
PY
```

### Database And Migrations

| Setting | Default | Notes |
| --- | --- | --- |
| `DATABASE_URL` | `postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev` | Local development and production-like profiles use PostgreSQL. Use `GOVOPLAN_DEV_DATABASE_BACKEND=sqlite` only for disposable SQLite runs. |
| `GOVOPLAN_MIGRATION_TRACK` | `release` | Use the release track for normal runtime and deployments. Use `dev` only for fresh/disposable databases that intentionally replay detailed development migrations. |
| `DEV_AUTO_MIGRATE_ENABLED` | `true` | Dev convenience only. Production should run migration commands explicitly during deployment. |
| `DEV_BOOTSTRAP_ENABLED` | `false` | Dev bootstrap only. `govoplan_core.devserver` and `govoplan/tools/launch/launch-dev.sh` default it to `true`; use controlled first-admin creation outside dev. |

Operator rule: take a database backup before applying migrations or destructive
module retirement. For non-SQLite databases, configure deployment-specific
backup/restore hooks for the module installer.

### PostgreSQL Production Target

PostgreSQL is the primary development and production target. SQLite remains
supported only for tiny disposable profiles and unit-test style smoke runs.
Production/staging deployments should use a managed PostgreSQL database and
explicit migration commands.

Install the full release profile from the meta repository so core, modules, and
the `psycopg` driver are available:

```bash
cd /mnt/DATA/git/govoplan
./.venv/bin/python -m pip install -r requirements-release.txt
```

Example runtime database URLs:

```bash
export DATABASE_URL='postgresql+psycopg://govoplan:change-me@db.example.internal:5432/govoplan'
export GOVOPLAN_DATABASE_URL_PGTOOLS='postgresql://govoplan:change-me@db.example.internal:5432/govoplan'
```

Use the SQLAlchemy URL for GovOPlaN. Use the pg-tools URL for `pg_dump`,
`pg_restore`, and `psql`; these tools do not understand the
`postgresql+psycopg://` driver marker.

Bootstrap or upgrade the schema explicitly during deployment:

```bash
export APP_ENV=prod
export ENABLED_MODULES=tenancy,access,admin,policy,audit,campaigns,files,mail,calendar,docs,ops
./.venv/bin/python -m govoplan_core.commands.init_db \
  --database-url "$DATABASE_URL"
```

Backup and restore-check before migration-bearing package changes:

```bash
pg_dump --format=custom \
  --file "$PWD/runtime/govoplan-$(date +%Y%m%d%H%M%S).dump" \
  "$GOVOPLAN_DATABASE_URL_PGTOOLS"
pg_restore --list "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump" >/dev/null
```

Restore a checked backup to the target database:

```bash
pg_restore --clean --if-exists \
  --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" \
  "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump"
```

For local development, create the host database described in
`/mnt/DATA/git/govoplan/dev/postgres/README.md`, then run:

```bash
cd /mnt/DATA/git/govoplan
./.venv/bin/python -m govoplan_core.commands.init_db \
  --database-url postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev \
  --with-dev-data
./.venv/bin/python -m govoplan_core.devserver --smoke --no-reload
tools/launch/launch-dev.sh
```

For disposable local validation against a throwaway PostgreSQL instance, use
the bundled PostgreSQL testbed:

```bash
cd /mnt/DATA/git/govoplan/dev/postgres
cp .env.example .env
docker compose --env-file .env up -d

cd /mnt/DATA/git/govoplan
set -a
. /mnt/DATA/git/govoplan/dev/postgres/.env
set +a
tools/checks/postgres-integration-check.py \
  --database-url "$GOVOPLAN_POSTGRES_DATABASE_URL" \
  --reset-schema
```

The integration check runs migrations and startup smoke checks across the
standard module permutations. It first requires the retirement atomicity proof,
using Files' real secret-owning provider and Audit's persistent recorder. That
proof uses only random, test-owned schemas and cleans them afterward; it does
not reset `public`. `--reset-schema` is destructive and belongs only on
throwaway databases. Do not pass `--skip-retirement-atomicity` when collecting
release evidence.

### Broker And Workers

| Setting | Default | Notes |
| --- | --- | --- |
| `REDIS_URL` | `redis://redis:6379/0` | Celery broker/result backend when async workers are enabled. |
| `CELERY_ENABLED` | `false` | Local/dev can send synchronously. Production campaign delivery should run workers and set this to `true`. |
| `CELERY_QUEUES` | `send_email,append_sent,notifications,mail,calendar,dataflow,workflow,postbox,events,idm,default` | Queue list expected by worker/process manager definitions. Keep this aligned with every enabled module task route; Ops reports missing worker queue consumers. |
| `PLATFORM_EVENT_OUTBOX_MAX_ATTEMPTS` | `8` | Failed durable consumer deliveries are quarantined after this many attempts. |
| `PLATFORM_EVENT_OUTBOX_TERMINAL_RETENTION_DAYS` | `90` | Successful event envelopes older than this are removed by the daily retention task. Quarantined evidence is retained. |

Worker command:

```bash
python -m celery -A govoplan_core.celery_app:celery worker \
  --queues send_email,append_sent,notifications,mail,calendar,dataflow,workflow,postbox,events,idm,default \
  --loglevel INFO
```

Run Celery beat as a separately supervised process. Its built-in one-minute
schedule recovers Calendar outbox rows left behind by broker failures, process
crashes, and expired worker leases:

```bash
python -m celery -A govoplan_core.celery_app:celery beat --loglevel INFO
```

### Storage

| Setting | Default | Notes |
| --- | --- | --- |
| `FILE_STORAGE_BACKEND` | `local` | Use `local` for dev/small deployments; use object storage when files must scale independently. |
| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Must live on durable storage and be backed up when `FILE_STORAGE_BACKEND=local`. |
| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Read-only fallback roots for migrated local files. |
| `FILE_STORAGE_S3_ENDPOINT_URL` | empty | Object-store endpoint for the files module. |
| `FILE_STORAGE_S3_REGION` | empty | Object-store region. |
| `FILE_STORAGE_S3_ACCESS_KEY_ID` | empty | Secret; inject through deployment environment. |
| `FILE_STORAGE_S3_SECRET_ACCESS_KEY` | empty | Secret; inject through deployment environment. |
| `FILE_STORAGE_S3_BUCKET` | `files` | Managed-file object bucket. |
| `FILE_STORAGE_S3_DEPLOYMENT_MANAGED` | `false` | Reserved for the installer-owned `http://garage:3900` service. It does not authorize arbitrary internal or external S3 endpoints. |
| `FILE_ARCHIVE_MAX_ENTRIES` | `10000` | Maximum declared entries accepted during archive preview and confirmation. |
| `FILE_ARCHIVE_MAX_EXPANDED_BYTES` | `2147483648` | Maximum actual expanded bytes across selected archive content. |
| `FILE_ARCHIVE_MAX_EXPANSION_RATIO` | `100` | Maximum expanded-to-compressed archive ratio. |
| `FILE_ARCHIVE_PREVIEW_TTL_SECONDS` | `1800` | Lifetime of the sealed actor/archive/destination-bound preview token. |

Legacy `S3_*` settings remain for older storage paths but new deployments should
prefer `FILE_STORAGE_*`.

### HTTP, Cookies, And Base URLs

| Setting | Default | Notes |
| --- | --- | --- |
| `CORS_ORIGINS` | local dev origins | Set to the exact WebUI origins in staging/production. |
| `GOVOPLAN_TRUSTED_HOSTS` | empty | Exact API host names accepted by the application. Production-like validation requires an explicit list; narrowly scoped `*.example.org` entries are supported. |
| `FORWARDED_ALLOW_IPS` | Uvicorn default | Address or network of the trusted reverse proxy. Never use `*` in production-like deployments. |
| `AUTH_SESSION_COOKIE_NAME` | configured default | Change only through a controlled rollout because it logs users out. |
| `AUTH_CSRF_COOKIE_NAME` | configured default | Must match WebUI/API deployment. |
| `AUTH_COOKIE_SECURE` | `false` | Set `true` behind HTTPS. |
| `AUTH_COOKIE_SAMESITE` | `lax` | Use a stricter value only after testing login and CSRF flows. |
| `AUTH_COOKIE_DOMAIN` | empty | Set only when the API and WebUI intentionally share a parent domain. |
| `GOVOPLAN_HTTP_HSTS_SECONDS` | `31536000` in production, otherwise `0` | Emitted only for HTTPS requests. Set `0` while rehearsing a deployment that is not yet HTTPS-only. |
| `GOVOPLAN_HTTP_MAX_REQUEST_BODY_BYTES` | `536870912` (512 MiB) | Deployment hard ceiling; file and module APIs apply their own lower limits where appropriate. |

Interactive password login is enabled with fixed-window limits of 10 failures
per normalized identity and 100 failures per direct client over 900 seconds.
`AUTH_LOGIN_THROTTLE_*` settings change those limits. Counters use `REDIS_URL`
when Redis is reachable so replicas share state. Production-like startup fails
when throttling is enabled without `REDIS_URL`. Set
`GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE=true` only as an explicit
single-process risk acceptance. A bounded process-local fallback keeps
development and temporary Redis outages functional, with per-process
enforcement until Redis recovers; monitor Redis because protection is weaker
during that fallback.

### Outbound Connector Egress

| Setting | Default | Notes |
| --- | --- | --- |
| `GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS` | `true` in dev/test, otherwise `false` | Deployment-wide decision. Set `true` only when pinned HTTP(S), DAV, SMTP, or IMAP transports must reach internal addresses. It does not enable an SDK transport that cannot pin every peer. |
| `GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` | `16777216` (16 MiB) | Maximum buffered JSON, XML, iCalendar, vCard, catalog, and connector error response. |
| `GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES` | `536870912` (512 MiB) | Hard upper bound for a single remote file; module upload limits may be lower. |
| `GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST` | empty | Comma-separated exact environment names usable by deployment-owned connector profiles. Tenant/API-managed profiles cannot select process variables, even when a name is listed. |
| `GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST` | empty | Comma-separated exact absolute CA bundle paths. Mount the same files at the same paths on every API and connector worker. |

Production-like configuration validation requires the private-network choice to
be explicit. HTTP connector downloads are streamed up to the configured bound,
and credential-bearing DAV redirects remain confined to their configured
origin.

The urllib, HTTPX/httpcore, SMTP, and IMAP transports resolve, validate, and
connect to the same approved address record while retaining the original host
for HTTP Host, TLS SNI, and certificate verification. Live SMB and S3 access
fails closed in both public-only and private-network deployments: the current
SDK transports cannot pin every initial and secondary peer or revalidate every
SDK-managed redirect/referral. An explicit IP endpoint does not bypass this
rule. Production deployments should still enforce the same decision at their
worker/container egress firewall or outbound proxy as a second boundary.

File connector TLS verification may be disabled only in dev/test. A custom CA
bundle must be an existing regular file whose resolved absolute path is listed
in `GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST`. Environment-backed file connector
credentials are supported only in deployment-owned connector JSON and require
their exact names in `GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; UI/API profiles
must use encrypted stored credentials or a scoped secret-provider reference.

Public URLs are currently supplied by deployment/reverse-proxy configuration and
module settings. Do not hardcode them in core; configuration packages should ask
for portal, WebUI, postbox, and notification URLs when they become relevant.
Uvicorn applies `X-Forwarded-*` only from `FORWARDED_ALLOW_IPS`; keep that value
aligned with the reverse proxy and do not expose the application server directly
through the same trusted address range.

### Module Catalogs, Licenses, And Trust Roots

| Setting | Purpose |
| --- | --- |
| `GOVOPLAN_MODULE_PACKAGE_CATALOG_URL` or `GOVOPLAN_MODULE_PACKAGE_CATALOG` | Module package catalog source. |
| `GOVOPLAN_MODULE_PACKAGE_CATALOG_TRUSTED_KEYS_FILE` | Preferred production keyring path. |
| `GOVOPLAN_MODULE_PACKAGE_CATALOG_APPROVED_CHANNEL` | Approved catalog channel, for example `stable`. |
| `GOVOPLAN_LICENSE_TRUSTED_KEYS_FILE` | Trusted license issuer keyring path. |
| `GOVOPLAN_LICENSE_ENFORCEMENT` | Enables license enforcement when set to `true`. |

Trust roots are deployment-managed and should not be editable through the
running WebUI.

### Mail Test Credentials

Dedicated SMTP/IMAP test credentials belong to the mail/campaign test-bed
configuration, not the core runtime contract. Store them in a local ignored
`.env` file for the test bed or in CI secrets. Required values are:

- SMTP host, port, TLS mode, username, password, and envelope/from address.
- IMAP host, port, TLS mode, username, password, and append folder.
- At least one recipient mailbox that is safe for automated send tests.

## First Deployment Flow

1. Create an environment file or secret set with the runtime contract above.
2. Install the tagged core and module packages from meta `requirements-release.txt`.
3. Build the WebUI from `webui/package.release.json` or deploy a prebuilt
   artifact from the same release tag.
4. Run database migrations with the target `DATABASE_URL`.
5. Create the first tenant and system owner through the controlled bootstrap or
   one-time admin command for the deployment.
6. Start the API service with `govoplan_core.server.app:app`.
7. Start workers when `CELERY_ENABLED=true`.
8. Start the WebUI/reverse proxy and verify CORS/cookie settings.
9. Open Admin > System > Modules, verify enabled modules, and save desired
   module state if it differs from `ENABLED_MODULES`.
10. Run health checks:

```bash
curl -fsS http://127.0.0.1:8000/health
curl -fsS http://127.0.0.1:8000/api/v1/platform/modules
```

Authenticated health details require `system:settings:read`:

```bash
curl -fsS -H "X-API-Key: $GOVOPLAN_HEALTH_API_KEY" \
  http://127.0.0.1:8000/health/details
```

## Production-Like Dev Profile

Use this profile to verify deployment behavior without publishing packages or
using real production credentials. The canonical launcher keeps API, worker, and
WebUI code in the editable repositories while Docker provides PostgreSQL and
Redis:

```bash
cd /mnt/DATA/git/govoplan
tools/launch/launch-production-like-dev.sh
```

The helper wrapper provides explicit lifecycle commands:

```bash
tools/launch/production-like-dev.sh validate-config
tools/launch/production-like-dev.sh seed
tools/launch/production-like-dev.sh start
tools/launch/production-like-dev.sh stop
tools/launch/production-like-dev.sh reset --yes
```

The launcher uses `govoplan/dev/production-like/.env` when present, otherwise
the checked in `.env.example`. It runs:

- PostgreSQL on `127.0.0.1:55433`
- Redis on `127.0.0.1:56379`
- explicit `ENABLED_MODULES`
- explicit migrations and `--with-dev-data` bootstrap
- API via the module-aware devserver
- a Celery worker for `send_email,append_sent,notifications,mail,calendar,dataflow,workflow,postbox,events,idm,default`
- WebUI through the Vite dev server
- durable local files under `runtime/production-like/files`

This profile validates explicit migration execution, config loading, module
discovery, route aggregation, local storage paths, Redis broker connectivity,
worker heartbeats, and health/readiness startup without real production
credentials. It does not replace a managed PostgreSQL/Redis/WebUI/worker
deployment test.

To stop PostgreSQL and Redis when the launcher exits:

```bash
GOVOPLAN_STOP_PROFILE_DEPENDENCIES_ON_EXIT=1 tools/launch/launch-production-like-dev.sh
```

## Module Install/Uninstall Operations

Use Admin > System > Modules for planning. The running API server validates and
queues install plans; it does not run pip/npm or restart itself from an HTTP
request. Package mutation belongs to the trusted installer CLI/daemon in an
operator shell while maintenance mode is active.

Preflight from the server shell:

```bash
govoplan-module-installer --format shell
```

Apply a prepared plan directly from a controlled shell:

```bash
govoplan-module-installer --apply --build-webui
```

For production-like runs, prefer supervised mode with migrations, database
backup/restore hooks, restart commands, and health checks:

```bash
govoplan-module-installer \
  --supervise \
  --migrate \
  --restart-command 'systemctl restart govoplan-api' \
  --restart-command 'systemctl restart govoplan-worker' \
  --health-url http://127.0.0.1:8000/health \
  --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \
  --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"'
```

To let the admin UI submit work without executing package managers inside the
API process, run the daemon in a separate operator shell:

```bash
govoplan-module-installer \
  --daemon \
  --migrate \
  --build-webui \
  --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \
  --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --health-url http://127.0.0.1:8000/health \
  --restart-command '<restart govoplan server>'
```

The daemon claims one queued request at a time and writes request/run records
below `runtime/module-installer`. For process-manager one-shot usage or tests,
use `--daemon-once`. Check daemon status with:

```bash
govoplan-module-installer --daemon-status --format json
```

The installer uses a runtime lock, snapshots `pip freeze` plus WebUI package
files, writes a run record, and marks planned rows as applied only after all
commands succeed. With `--migrate`, SQLite databases are backed up through
SQLite's backup API; non-SQLite databases require
`--database-backup-command`, `--database-restore-check-command`, and
`--database-restore-command`.

Database hook commands receive:

- `GOVOPLAN_INSTALLER_RUN_DIR`
- `GOVOPLAN_DATABASE_URL`
- `GOVOPLAN_DATABASE_URL_PGTOOLS` for PostgreSQL tools
- `GOVOPLAN_DATABASE_BACKUP_PATH`
- `GOVOPLAN_DATABASE_BACKUP_METADATA`

Avoid embedding secrets directly in commands; prefer environment variables,
service credentials, or deployment-local secret injection.

Inspect installer history and lock state from the operator shell:

```bash
govoplan-module-installer --list-runs --format json
govoplan-module-installer --show-run <run-id> --format json
govoplan-module-installer --lock-status --format json
govoplan-module-installer --list-requests --format json
govoplan-module-installer --show-request <request-id> --format json
govoplan-module-installer --cancel-request <request-id> --format json
govoplan-module-installer --retry-request <request-id> --format json
```

Rollback uses the saved run snapshot:

```bash
govoplan-module-installer --rollback <run-id>
govoplan-module-installer --rollback <run-id> --database-restore-command '<override restore command>'
```

Uninstall is non-destructive by default. A planned uninstall row can set
`destroy_data: true` to request destructive module retirement. The module must
provide an automated retirement provider, and the installer snapshots the
database before dropping module-owned tables.

Run the rollback drill before relying on installer automation in a new
environment:

```bash
/mnt/DATA/git/govoplan/tools/checks/module-installer-rollback-drill.py \
  --format json \
  --evidence-path runtime/module-installer/restore-drill-evidence.json
```

The drill uses temporary SQLite databases and simulated package commands. It
does not install or uninstall real packages. It exercises:

- package command failure followed by supervised rollback;
- migration failure with a SQLite database snapshot;
- restart-command failure;
- health timeout after restart;
- destructive retirement executor failure with database rollback;
- PostgreSQL-style backup, restore-check, and restore hooks;
- daemon heartbeat, request queue claim/update, retry/cancel, and stale lock
  detection/removal.

See `RELEASE_DEPENDENCIES.md` for release package refs, migration baseline
checks, catalog trust, signing, keyring, replay, and license operation.

## Operator Checklist

- Runtime secrets are injected outside git.
- `MASTER_KEY_B64` is set and backed up securely; restores of locally encrypted content fail closed without the exact matching key.
- Database backup and restore commands are tested.
- File/object storage is durable and backed up.
- `CORS_ORIGINS` and cookie settings match the deployed WebUI origin.
- Redis and workers are running before `CELERY_ENABLED=true`.
- Module catalog and license keyrings are pinned locally.
- Health endpoints are monitored.
- Test SMTP/IMAP credentials are non-production and isolated.
- Module installer rollback drill has passed in the deployment environment.
