govoplan-core/docs/DEPLOYMENT_OPERATOR_GUIDE.md

# GovOPlaN Deployment Operator Guide

This guide defines the current install/runtime configuration contract and the
operator flow for a production-realistic self-hosted deployment. Keep secrets in
the deployment environment or a secret manager; do not commit populated `.env`
files.

## Runtime Configuration Contract

### Required Runtime Identity

| Setting | Required outside dev | Purpose |
| --- | --- | --- |
| `APP_ENV` | yes | Runtime profile. Use `prod`, `staging`, or a deployment-specific value outside local development. |
| `MASTER_KEY_B64` | yes | Fernet key or base64 encoded 32-byte key used for encrypted module secrets. Rotate through an explicit operator plan. |
| `DATABASE_URL` | yes | SQLAlchemy database URL for core and installed modules. SQLite is supported for dev/small installs; PostgreSQL is the preferred production target. |
| `ENABLED_MODULES` | yes | Comma-separated startup module set. Keep `tenancy,access` enabled; keep `admin` enabled for operator UI. |

Generate a local key for a new non-production environment:

```bash
python - <<'PY'
from cryptography.fernet import Fernet
print(Fernet.generate_key().decode())
PY
```

### Database And Migrations

| Setting | Default | Notes |
| --- | --- | --- |
| `DATABASE_URL` | `postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev` | Local development and production-like profiles use PostgreSQL. Use `GOVOPLAN_DEV_DATABASE_BACKEND=sqlite` only for disposable SQLite runs. |
| `DEV_AUTO_MIGRATE_ENABLED` | `true` | Dev convenience only. Production should run migration commands explicitly during deployment. |
| `DEV_BOOTSTRAP_ENABLED` | `false` | Dev bootstrap only. `govoplan_core.devserver` and `scripts/launch-dev.sh` default it to `true`; use controlled first-admin creation outside dev. |

Operator rule: take a database backup before applying migrations or destructive
module retirement. For non-SQLite databases, configure deployment-specific
backup/restore hooks for the module installer.

### PostgreSQL Production Target

PostgreSQL is the primary development and production target. SQLite remains
supported only for tiny disposable profiles and unit-test style smoke runs.
Production/staging deployments should use a managed PostgreSQL database and
explicit migration commands.

Install the server extra so the `psycopg` driver is available:

```bash
cd /mnt/DATA/git/govoplan-core
./.venv/bin/python -m pip install -r requirements-release.txt
```

Example runtime database URLs:

```bash
export DATABASE_URL='postgresql+psycopg://govoplan:change-me@db.example.internal:5432/govoplan'
export GOVOPLAN_DATABASE_URL_PGTOOLS='postgresql://govoplan:change-me@db.example.internal:5432/govoplan'
```

Use the SQLAlchemy URL for GovOPlaN. Use the pg-tools URL for `pg_dump`,
`pg_restore`, and `psql`; these tools do not understand the
`postgresql+psycopg://` driver marker.

Bootstrap or upgrade the schema explicitly during deployment:

```bash
export APP_ENV=prod
export ENABLED_MODULES=tenancy,access,admin,policy,audit,campaigns,files,mail,calendar,docs,ops
./.venv/bin/python -m govoplan_core.commands.init_db \
  --database-url "$DATABASE_URL"
```

Backup and restore-check before migration-bearing package changes:

```bash
pg_dump --format=custom \
  --file "$PWD/runtime/govoplan-$(date +%Y%m%d%H%M%S).dump" \
  "$GOVOPLAN_DATABASE_URL_PGTOOLS"
pg_restore --list "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump" >/dev/null
```

Restore a checked backup to the target database:

```bash
pg_restore --clean --if-exists \
  --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" \
  "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump"
```

For local development, create the host database described in
`dev/postgres/README.md`, then run:

```bash
cd /mnt/DATA/git/govoplan-core
./.venv/bin/python -m govoplan_core.commands.init_db \
  --database-url postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev \
  --with-dev-data
./.venv/bin/python -m govoplan_core.devserver --smoke --no-reload
scripts/launch-dev.sh
```

For disposable local validation against a throwaway PostgreSQL instance, use
the bundled PostgreSQL testbed:

```bash
cd /mnt/DATA/git/govoplan-core/dev/postgres
cp .env.example .env
docker compose --env-file .env up -d

cd /mnt/DATA/git/govoplan-core
set -a
. dev/postgres/.env
set +a
./.venv/bin/python scripts/postgres-integration-check.py \
  --database-url "$GOVOPLAN_POSTGRES_DATABASE_URL" \
  --reset-schema
```

The integration check runs migrations and startup smoke checks across the
standard module permutations. `--reset-schema` is destructive and belongs only
on throwaway databases.

### Broker And Workers

| Setting | Default | Notes |
| --- | --- | --- |
| `REDIS_URL` | `redis://redis:6379/0` | Celery broker/result backend when async workers are enabled. |
| `CELERY_ENABLED` | `false` | Local/dev can send synchronously. Production campaign delivery should run workers and set this to `true`. |
| `CELERY_QUEUES` | `send_email,append_sent,default` | Queue list expected by worker/process manager definitions. |

Worker command:

```bash
python -m celery -A govoplan_core.celery_app:celery worker \
  --queues send_email,append_sent,default \
  --loglevel INFO
```

### Storage

| Setting | Default | Notes |
| --- | --- | --- |
| `FILE_STORAGE_BACKEND` | `local` | Use `local` for dev/small deployments; use object storage when files must scale independently. |
| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Must live on durable storage and be backed up when `FILE_STORAGE_BACKEND=local`. |
| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Read-only fallback roots for migrated local files. |
| `FILE_STORAGE_S3_ENDPOINT_URL` | empty | Object-store endpoint for the files module. |
| `FILE_STORAGE_S3_REGION` | empty | Object-store region. |
| `FILE_STORAGE_S3_ACCESS_KEY_ID` | empty | Secret; inject through deployment environment. |
| `FILE_STORAGE_S3_SECRET_ACCESS_KEY` | empty | Secret; inject through deployment environment. |
| `FILE_STORAGE_S3_BUCKET` | `files` | Managed-file object bucket. |

Legacy `S3_*` settings remain for older storage paths but new deployments should
prefer `FILE_STORAGE_*`.

### HTTP, Cookies, And Base URLs

| Setting | Default | Notes |
| --- | --- | --- |
| `CORS_ORIGINS` | local dev origins | Set to the exact WebUI origins in staging/production. |
| `AUTH_SESSION_COOKIE_NAME` | configured default | Change only through a controlled rollout because it logs users out. |
| `AUTH_CSRF_COOKIE_NAME` | configured default | Must match WebUI/API deployment. |
| `AUTH_COOKIE_SECURE` | `false` | Set `true` behind HTTPS. |
| `AUTH_COOKIE_SAMESITE` | `lax` | Use a stricter value only after testing login and CSRF flows. |
| `AUTH_COOKIE_DOMAIN` | empty | Set only when the API and WebUI intentionally share a parent domain. |

Public URLs are currently supplied by deployment/reverse-proxy configuration and
module settings. Do not hardcode them in core; configuration packages should ask
for portal, WebUI, postbox, and notification URLs when they become relevant.

### Module Catalogs, Licenses, And Trust Roots

| Setting | Purpose |
| --- | --- |
| `GOVOPLAN_MODULE_PACKAGE_CATALOG_URL` or `GOVOPLAN_MODULE_PACKAGE_CATALOG` | Module package catalog source. |
| `GOVOPLAN_MODULE_PACKAGE_CATALOG_TRUSTED_KEYS_FILE` | Preferred production keyring path. |
| `GOVOPLAN_MODULE_PACKAGE_CATALOG_APPROVED_CHANNEL` | Approved catalog channel, for example `stable`. |
| `GOVOPLAN_LICENSE_TRUSTED_KEYS_FILE` | Trusted license issuer keyring path. |
| `GOVOPLAN_LICENSE_ENFORCEMENT` | Enables license enforcement when set to `true`. |

Trust roots are deployment-managed and should not be editable through the
running WebUI.

### Mail Test Credentials

Dedicated SMTP/IMAP test credentials belong to the mail/campaign test-bed
configuration, not the core runtime contract. Store them in a local ignored
`.env` file for the test bed or in CI secrets. Required values are:

- SMTP host, port, TLS mode, username, password, and envelope/from address.
- IMAP host, port, TLS mode, username, password, and append folder.
- At least one recipient mailbox that is safe for automated send tests.

## First Deployment Flow

1. Create an environment file or secret set with the runtime contract above.
2. Install the tagged core and module packages from `requirements-release.txt`.
3. Build the WebUI from `webui/package.release.json` or deploy a prebuilt
   artifact from the same release tag.
4. Run database migrations with the target `DATABASE_URL`.
5. Create the first tenant and system owner through the controlled bootstrap or
   one-time admin command for the deployment.
6. Start the API service with `govoplan_core.server.app:app`.
7. Start workers when `CELERY_ENABLED=true`.
8. Start the WebUI/reverse proxy and verify CORS/cookie settings.
9. Open Admin > System > Modules, verify enabled modules, and save desired
   module state if it differs from `ENABLED_MODULES`.
10. Run health checks:

```bash
curl -fsS http://127.0.0.1:8000/health
curl -fsS http://127.0.0.1:8000/api/v1/platform/modules
```

Authenticated health details require `system:settings:read`:

```bash
curl -fsS -H "X-API-Key: $GOVOPLAN_HEALTH_API_KEY" \
  http://127.0.0.1:8000/health/details
```

## Production-Like Dev Profile

Use this profile to verify deployment behavior without publishing packages or
using real production credentials. The canonical launcher keeps API, worker, and
WebUI code in the editable repositories while Docker provides PostgreSQL and
Redis:

```bash
cd /mnt/DATA/git/govoplan-core
scripts/launch-production-like-dev.sh
```

The launcher uses `dev/production-like/.env` when present, otherwise the checked
in `.env.example`. It runs:

- PostgreSQL on `127.0.0.1:55433`
- Redis on `127.0.0.1:56379`
- explicit `ENABLED_MODULES`
- explicit migrations and `--with-dev-data` bootstrap
- API via the module-aware devserver
- a Celery worker for `send_email,append_sent,default`
- WebUI through the Vite dev server
- durable local files under `runtime/production-like/files`

This profile validates explicit migration execution, config loading, module
discovery, route aggregation, local storage paths, Redis broker connectivity,
worker heartbeats, and health/readiness startup without real production
credentials. It does not replace a managed PostgreSQL/Redis/WebUI/worker
deployment test.

To stop PostgreSQL and Redis when the launcher exits:

```bash
GOVOPLAN_STOP_PROFILE_DEPENDENCIES_ON_EXIT=1 scripts/launch-production-like-dev.sh
```

## Module Install/Uninstall Operations

Use Admin > System > Modules for planning. The running API server validates and
queues install plans; it does not run pip/npm or restart itself from an HTTP
request. Package mutation belongs to the trusted installer CLI/daemon in an
operator shell while maintenance mode is active.

Preflight from the server shell:

```bash
govoplan-module-installer --format shell
```

Apply a prepared plan directly from a controlled shell:

```bash
govoplan-module-installer --apply --build-webui
```

For production-like runs, prefer supervised mode with migrations, database
backup/restore hooks, restart commands, and health checks:

```bash
govoplan-module-installer \
  --supervise \
  --migrate \
  --restart-command 'systemctl restart govoplan-api' \
  --restart-command 'systemctl restart govoplan-worker' \
  --health-url http://127.0.0.1:8000/health \
  --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \
  --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"'
```

To let the admin UI submit work without executing package managers inside the
API process, run the daemon in a separate operator shell:

```bash
govoplan-module-installer \
  --daemon \
  --migrate \
  --build-webui \
  --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \
  --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \
  --health-url http://127.0.0.1:8000/health \
  --restart-command '<restart govoplan server>'
```

The daemon claims one queued request at a time and writes request/run records
below `runtime/module-installer`. For process-manager one-shot usage or tests,
use `--daemon-once`. Check daemon status with:

```bash
govoplan-module-installer --daemon-status --format json
```

The installer uses a runtime lock, snapshots `pip freeze` plus WebUI package
files, writes a run record, and marks planned rows as applied only after all
commands succeed. With `--migrate`, SQLite databases are backed up through
SQLite's backup API; non-SQLite databases require
`--database-backup-command`, `--database-restore-check-command`, and
`--database-restore-command`.

Database hook commands receive:

- `GOVOPLAN_INSTALLER_RUN_DIR`
- `GOVOPLAN_DATABASE_URL`
- `GOVOPLAN_DATABASE_URL_PGTOOLS` for PostgreSQL tools
- `GOVOPLAN_DATABASE_BACKUP_PATH`
- `GOVOPLAN_DATABASE_BACKUP_METADATA`

Avoid embedding secrets directly in commands; prefer environment variables,
service credentials, or deployment-local secret injection.

Inspect installer history and lock state from the operator shell:

```bash
govoplan-module-installer --list-runs --format json
govoplan-module-installer --show-run <run-id> --format json
govoplan-module-installer --lock-status --format json
govoplan-module-installer --list-requests --format json
govoplan-module-installer --show-request <request-id> --format json
govoplan-module-installer --cancel-request <request-id> --format json
govoplan-module-installer --retry-request <request-id> --format json
```

Rollback uses the saved run snapshot:

```bash
govoplan-module-installer --rollback <run-id>
govoplan-module-installer --rollback <run-id> --database-restore-command '<override restore command>'
```

Uninstall is non-destructive by default. A planned uninstall row can set
`destroy_data: true` to request destructive module retirement. The module must
provide an automated retirement provider, and the installer snapshots the
database before dropping module-owned tables.

Run the rollback drill before relying on installer automation in a new
environment:

```bash
./.venv/bin/python scripts/module-installer-rollback-drill.py --format json
```

The drill uses temporary SQLite databases and simulated package commands. It
does not install or uninstall real packages. It exercises:

- package command failure followed by supervised rollback;
- migration failure with a SQLite database snapshot;
- restart-command failure;
- health timeout after restart;
- destructive retirement executor failure with database rollback;
- PostgreSQL-style backup, restore-check, and restore hooks;
- daemon heartbeat, request queue claim/update, retry/cancel, and stale lock
  detection/removal.

See `RELEASE_DEPENDENCIES.md` for release package refs, migration baseline
checks, catalog trust, signing, keyring, replay, and license operation.

## Operator Checklist

- Runtime secrets are injected outside git.
- `MASTER_KEY_B64` is set and backed up securely.
- Database backup and restore commands are tested.
- File/object storage is durable and backed up.
- `CORS_ORIGINS` and cookie settings match the deployed WebUI origin.
- Redis and workers are running before `CELERY_ENABLED=true`.
- Module catalog and license keyrings are pinned locally.
- Health endpoints are monitored.
- Test SMTP/IMAP credentials are non-production and isolated.
- Module installer rollback drill has passed in the deployment environment.