From aa14cbfb567c44014ba237dfc0ab8a4de484afa6 Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Thu, 9 Jul 2026 11:42:41 +0200 Subject: [PATCH] Sync wiki from project files --- Codex-Project-Index.md | 13 + Repo-README.md | 24 +- Repo-docs-ACCESS-EXTRACTION-PLAN.md | 39 ++- Repo-docs-API-CONDITIONAL-DELTA.md | 71 ++++ Repo-docs-CATALOG-TRUST-AND-LICENSING.md | 94 +++++- Repo-docs-CONFIGURATION-PACKAGES.md | 43 ++- Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md | 310 ++++++++++++++++++ Repo-docs-EVENTS-AND-AUDIT.md | 53 ++- Repo-docs-GOVERNMENT-OPERATIONS-VISION.md | 14 +- Repo-docs-GOVOPLAN-MODULE-ROADMAP.md | 84 ++++- Repo-docs-MODULE-ARCHITECTURE.md | 39 ++- Repo-docs-MODULE-BOUNDARY-DECISIONS.md | 215 ++++++++++++ Repo-docs-POLICY-CONTRACTS.md | 112 +++++++ ...docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md | 283 ++++++++++++++++ Repo-docs-RELEASE-CATALOG-WORKFLOW.md | 16 +- Repo-docs-RELEASE-DEPENDENCIES.md | 225 ++++++++++++- Repo-docs-REMOTE-WEBUI-BUNDLES.md | 168 ++++++++++ Repo-docs-SCALABILITY-AND-SIZING.md | 204 ++++++++++++ 18 files changed, 1916 insertions(+), 91 deletions(-) create mode 100644 Repo-docs-API-CONDITIONAL-DELTA.md create mode 100644 Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md create mode 100644 Repo-docs-MODULE-BOUNDARY-DECISIONS.md create mode 100644 Repo-docs-POLICY-CONTRACTS.md create mode 100644 Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md create mode 100644 Repo-docs-REMOTE-WEBUI-BUNDLES.md create mode 100644 Repo-docs-SCALABILITY-AND-SIZING.md diff --git a/Codex-Project-Index.md b/Codex-Project-Index.md index 9626032..4be3618 100644 --- a/Codex-Project-Index.md +++ b/Codex-Project-Index.md @@ -7,10 +7,23 @@ This page is generated from repository and product-directory project files. - [Product govoplan-split-concept-action-plan](Product govoplan-split-concept-action-plan) - `/mnt/DATA/Nextcloud/ADD ideas UG/Products/govoplan/split-concept-action-plan.md` - [Repo-README](Repo-README) - `/mnt/DATA/git/govoplan-core/README.md` - [Repo-docs-ACCESS-EXTRACTION-PLAN](Repo-docs-ACCESS-EXTRACTION-PLAN) - `/mnt/DATA/git/govoplan-core/docs/ACCESS_EXTRACTION_PLAN.md` +- [Repo-docs-API-CONDITIONAL-DELTA](Repo-docs-API-CONDITIONAL-DELTA) - `/mnt/DATA/git/govoplan-core/docs/API_CONDITIONAL_DELTA.md` +- [Repo-docs-CATALOG-TRUST-AND-LICENSING](Repo-docs-CATALOG-TRUST-AND-LICENSING) - `/mnt/DATA/git/govoplan-core/docs/CATALOG_TRUST_AND_LICENSING.md` - [Repo-docs-CODEX-WORKFLOW](Repo-docs-CODEX-WORKFLOW) - `/mnt/DATA/git/govoplan-core/docs/CODEX_WORKFLOW.md` +- [Repo-docs-CONFIGURATION-PACKAGES](Repo-docs-CONFIGURATION-PACKAGES) - `/mnt/DATA/git/govoplan-core/docs/CONFIGURATION_PACKAGES.md` +- [Repo-docs-DEPLOYMENT-OPERATOR-GUIDE](Repo-docs-DEPLOYMENT-OPERATOR-GUIDE) - `/mnt/DATA/git/govoplan-core/docs/DEPLOYMENT_OPERATOR_GUIDE.md` +- [Repo-docs-EVENTS-AND-AUDIT](Repo-docs-EVENTS-AND-AUDIT) - `/mnt/DATA/git/govoplan-core/docs/EVENTS_AND_AUDIT.md` - [Repo-docs-GITEA-ISSUES](Repo-docs-GITEA-ISSUES) - `/mnt/DATA/git/govoplan-core/docs/GITEA_ISSUES.md` +- [Repo-docs-GOVERNMENT-OPERATIONS-VISION](Repo-docs-GOVERNMENT-OPERATIONS-VISION) - `/mnt/DATA/git/govoplan-core/docs/GOVERNMENT_OPERATIONS_VISION.md` +- [Repo-docs-GOVOPLAN-MASTER-ROADMAP](Repo-docs-GOVOPLAN-MASTER-ROADMAP) - `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MASTER_ROADMAP.md` - [Repo-docs-GOVOPLAN-MODULE-ROADMAP](Repo-docs-GOVOPLAN-MODULE-ROADMAP) - `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MODULE_ROADMAP.md` - [Repo-docs-MODULE-ARCHITECTURE](Repo-docs-MODULE-ARCHITECTURE) - `/mnt/DATA/git/govoplan-core/docs/MODULE_ARCHITECTURE.md` +- [Repo-docs-MODULE-BOUNDARY-DECISIONS](Repo-docs-MODULE-BOUNDARY-DECISIONS) - `/mnt/DATA/git/govoplan-core/docs/MODULE_BOUNDARY_DECISIONS.md` +- [Repo-docs-POLICY-CONTRACTS](Repo-docs-POLICY-CONTRACTS) - `/mnt/DATA/git/govoplan-core/docs/POLICY_CONTRACTS.md` +- [Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY](Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY) - `/mnt/DATA/git/govoplan-core/docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md` - [Repo-docs-RBAC-MANIFEST](Repo-docs-RBAC-MANIFEST) - `/mnt/DATA/git/govoplan-core/docs/RBAC_MANIFEST.md` +- [Repo-docs-RELEASE-CATALOG-WORKFLOW](Repo-docs-RELEASE-CATALOG-WORKFLOW) - `/mnt/DATA/git/govoplan-core/docs/RELEASE_CATALOG_WORKFLOW.md` - [Repo-docs-RELEASE-DEPENDENCIES](Repo-docs-RELEASE-DEPENDENCIES) - `/mnt/DATA/git/govoplan-core/docs/RELEASE_DEPENDENCIES.md` +- [Repo-docs-REMOTE-WEBUI-BUNDLES](Repo-docs-REMOTE-WEBUI-BUNDLES) - `/mnt/DATA/git/govoplan-core/docs/REMOTE_WEBUI_BUNDLES.md` +- [Repo-docs-SCALABILITY-AND-SIZING](Repo-docs-SCALABILITY-AND-SIZING) - `/mnt/DATA/git/govoplan-core/docs/SCALABILITY_AND_SIZING.md` - [Repo-docs-SYSTEM-GOVERNANCE-MANIFEST](Repo-docs-SYSTEM-GOVERNANCE-MANIFEST) - `/mnt/DATA/git/govoplan-core/docs/SYSTEM_GOVERNANCE_MANIFEST.md` diff --git a/Repo-README.md b/Repo-README.md index bebcb7c..cfae4fc 100644 --- a/Repo-README.md +++ b/Repo-README.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/README.md`. > Origin: `repository`. @@ -29,6 +29,7 @@ Canonical policy documents live in `docs/`: - [RBAC_MANIFEST.md](docs/RBAC_MANIFEST.md) - [SYSTEM_GOVERNANCE_MANIFEST.md](docs/SYSTEM_GOVERNANCE_MANIFEST.md) - [MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md) +- [DEPLOYMENT_OPERATOR_GUIDE.md](docs/DEPLOYMENT_OPERATOR_GUIDE.md) - [CODEX_WORKFLOW.md](docs/CODEX_WORKFLOW.md) Modules may define module-specific permissions and policy behavior, but the platform-level permission model and governance hierarchy belong here. @@ -42,7 +43,7 @@ cd /mnt/DATA/git/govoplan-core ./.venv/bin/python -m pip install -r requirements-dev.txt ``` -Run the platform server from core through the module-aware development runner. The default config reads `ENABLED_MODULES` and discovers installed module entry points. Local development defaults to `tenancy,access,admin,policy,audit,campaigns,files,mail`; set `ENABLED_MODULES` explicitly when testing a smaller module permutation. +Run the platform server from core through the module-aware development runner. The default config reads `ENABLED_MODULES` and discovers installed module entry points. Local development defaults to `tenancy,access,admin,policy,audit,campaigns,files,mail,calendar,docs,ops`; set `ENABLED_MODULES` explicitly when testing a smaller module permutation. ```bash cd /mnt/DATA/git/govoplan-core @@ -62,13 +63,24 @@ ENABLED_MODULES=access,campaigns ./.venv/bin/python -m govoplan_core.devserver \ The runner loads the same `GovoplanServerConfig` as `govoplan_core.server.app:app`, builds the platform registry, and passes core plus enabled module source roots to uvicorn as reload directories. After reinstalling the editable package, the same command is also available as `govoplan-devserver`. -The default development SQLite database lives at `runtime/multimailer-dev.db`, alongside other local runtime state. +The default development database is PostgreSQL at `postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev`. Store the password in `~/.pgpass`. To force the old disposable SQLite fallback, run with `GOVOPLAN_DEV_DATABASE_BACKEND=sqlite`; that database lives at `runtime/multimailer-dev.db`. Local devserver runs do not require Redis. `CELERY_ENABLED` defaults to `false`, so campaign queue actions update database state without publishing Celery tasks. Use the synchronous send flow for local send tests, or set `CELERY_ENABLED=true` only when a Redis broker and worker are running. -If the configured local SQLite database is missing or empty, `govoplan_core.devserver` enables the development bootstrap before loading settings. This creates the schema and the default development login on startup. Explicitly setting `DEV_BOOTSTRAP_ENABLED=false` disables this convenience. Production deployments should use migrations and managed database provisioning instead. +To run the production-like local profile with PostgreSQL, Redis, a Celery +worker, explicit module configuration, and persistent local file storage: -To verify the effective runtime paths and missing-SQLite bootstrap without starting uvicorn, run the smoke mode: +```bash +cd /mnt/DATA/git/govoplan-core +scripts/launch-production-like-dev.sh +``` + +See [dev/production-like/README.md](dev/production-like/README.md) for ports, +environment overrides, and cleanup commands. + +`govoplan_core.devserver` enables the development bootstrap before loading settings. In dev, startup migrations create or upgrade the schema and the bootstrap creates the default development login if needed. Explicitly setting `DEV_BOOTSTRAP_ENABLED=false` disables this convenience. Production deployments should use migrations and managed database provisioning instead. + +To verify the effective runtime paths and bootstrap behavior without starting uvicorn, run the smoke mode: ```bash cd /mnt/DATA/git/govoplan-core @@ -79,6 +91,8 @@ The smoke mode prints the effective config, runtime root, database URL, modules, `requirements-dev.txt` links local GovOPlaN module checkouts for development. `requirements-release.txt` installs the packaged modules from tagged git refs for release builds. See [RELEASE_DEPENDENCIES.md](docs/RELEASE_DEPENDENCIES.md). +For the install/runtime configuration contract and operator deployment flow, see [DEPLOYMENT_OPERATOR_GUIDE.md](docs/DEPLOYMENT_OPERATOR_GUIDE.md). + ## WebUI development Install and run from the core WebUI host: diff --git a/Repo-docs-ACCESS-EXTRACTION-PLAN.md b/Repo-docs-ACCESS-EXTRACTION-PLAN.md index 6ecdbe8..6035c2a 100644 --- a/Repo-docs-ACCESS-EXTRACTION-PLAN.md +++ b/Repo-docs-ACCESS-EXTRACTION-PLAN.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/ACCESS_EXTRACTION_PLAN.md`. > Origin: `repository`. @@ -85,9 +85,9 @@ WebUI contributions without changing the core shell route wiring again. Feature modules no longer import core auth dependency wrappers or access-owned ORM models. Backend routers import the access-published FastAPI dependency API -from `govoplan_access.backend.auth.dependencies`; runtime cooperation uses -kernel capabilities such as `access.directory`, `campaigns.access`, -`campaigns.mailPolicyContext`, and `campaigns.deliveryTasks`. +from `govoplan_access.auth`; runtime cooperation uses kernel capabilities such +as `access.directory`, `campaigns.access`, `campaigns.mailPolicyContext`, and +`campaigns.deliveryTasks`. ## Target Ownership @@ -280,9 +280,8 @@ Tasks: Acceptance criteria: - Auth behavior works through the access module. -- Feature modules do not import access internals except the published - `govoplan_access.backend.auth.dependencies` dependency API used by FastAPI - routers. +- Feature modules do not import access internals. FastAPI routers use the + published `govoplan_access.auth` dependency API. - Core can explain startup failure clearly if auth-required routes are enabled without the access capability. @@ -316,12 +315,14 @@ Remove direct ORM/model imports from files, mail, and campaign. Tasks: -- Replace `Group`, `Tenant`, `User`, and `UserGroupMembership` imports with +- [x] Replace `Group`, `Tenant`, `User`, and `UserGroupMembership` imports with directory/capability lookups. -- Replace direct cross-module cleanup/count queries with registered providers, +- [x] Replace direct cross-module cleanup/count queries with registered providers, events, or module-owned API contracts. -- Replace mail-profile ownership resolution with stable owner references. -- Replace campaign/file access checks with resource ACL provider contracts. +- [x] Replace mail-profile ownership resolution with stable owner references. +- [x] Replace campaign/file access checks with resource ACL provider contracts. +- [x] Move feature routers from the old backend auth dependency path to the + public `govoplan_access.auth` dependency API. Acceptance criteria: @@ -335,11 +336,11 @@ Finalize the split. Tasks: -- Remove obsolete core compatibility aliases. -- Keep the dependency-boundary checker allowlist empty. -- Update release dependency docs. -- Update operator migration notes. -- Add full module permutation tests including access-present and access-absent +- [x] Remove obsolete core compatibility aliases. +- [x] Keep the dependency-boundary checker allowlist empty. +- [x] Update release dependency docs. +- [x] Update operator migration notes. +- [x] Add full module permutation tests including access-present and access-absent behavior. Acceptance criteria: @@ -377,7 +378,7 @@ Acceptance criteria: - [x] Create `govoplan-access` repository workspace and issue workflow scaffold. - [x] Create `govoplan-access` repository/package skeleton. - [x] Move `govoplan_core/access` seed package into `govoplan-access`. -- [x] Add compatibility wrappers in core for old import paths. +- [x] Remove compatibility wrappers in core for old import paths. - [x] Route existing auth dependency wrappers through access principal/evaluator capabilities. - [x] Move FastAPI auth dependency wrappers out of core and into `govoplan-access`. @@ -447,7 +448,9 @@ Acceptance criteria: capability? - Audit log storage lives in `govoplan-audit`; policy provenance can build on that module boundary. -- How long should core keep compatibility route paths for existing clients? +- Core no longer keeps compatibility import paths for access/auth/admin service + modules. HTTP route paths remain stable because the access manifest + contributes the same `/api/v1/auth/*` and `/api/v1/admin/*` routes. ## Verification diff --git a/Repo-docs-API-CONDITIONAL-DELTA.md b/Repo-docs-API-CONDITIONAL-DELTA.md new file mode 100644 index 0000000..59935a3 --- /dev/null +++ b/Repo-docs-API-CONDITIONAL-DELTA.md @@ -0,0 +1,71 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/API_CONDITIONAL_DELTA.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Conditional GET And Delta Collections + +GovOPlaN uses two complementary mechanisms to reduce reload cost. + +## Conditional GET + +Core applies conditional GET handling centrally for successful JSON `GET` +responses: + +- Responses receive a weak `ETag` based on the serialized JSON body. +- Responses are marked `Cache-Control: private, no-cache`. +- Responses vary by `Authorization`, `Cookie`, `X-API-Key`, and + `Accept-Language`. +- Matching `If-None-Match` requests return `304 Not Modified` without a body. +- Responses with `Set-Cookie`, `Content-Disposition`, `Content-Encoding`, a + non-JSON content type, a non-200 status, or `Cache-Control: no-store` are not + converted. + +The WebUI `apiFetch` client keeps an in-memory conditional cache for reusable +safe requests. It sends `If-None-Match` after an endpoint has returned an ETag, +returns the cached payload on `304`, and clears the cache generation after +unsafe methods. + +This avoids retransmitting unchanged snapshots. It does not identify which row +changed inside a collection. + +## Delta Collections + +Collection endpoints that can expose row-level changes should use the shared +delta contract instead of inventing module-specific formats. + +Backend shape: + +```json +{ + "items": [], + "deleted": [], + "watermark": "opaque-next-watermark", + "has_more": false, + "full": false +} +``` + +Fields: + +- `items`: changed or current items since the requested watermark. +- `deleted`: deleted item markers with at least `id`, and optionally `revision` + and `deleted_at`. +- `watermark`: opaque value the client sends as `since` on the next request. +- `has_more`: true when the client should request the next page with the + returned watermark. +- `full`: true when the response is a full snapshot rather than an incremental + delta. + +Recommended query parameters: + +- `since`: opaque previous watermark. If omitted or expired, return a full + snapshot with `full: true`. +- `limit`: maximum number of changed items plus deleted markers. +- `include_deleted`: whether deleted markers should be returned. + +Modules should base watermarks on a monotonic revision, audit/event sequence, or +updated/deleted timestamp that is scoped to the same tenant and authorization +rules as the endpoint response. diff --git a/Repo-docs-CATALOG-TRUST-AND-LICENSING.md b/Repo-docs-CATALOG-TRUST-AND-LICENSING.md index 3d189cb..7ebc49e 100644 --- a/Repo-docs-CATALOG-TRUST-AND-LICENSING.md +++ b/Repo-docs-CATALOG-TRUST-AND-LICENSING.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/CATALOG_TRUST_AND_LICENSING.md`. > Origin: `repository`. @@ -131,6 +131,39 @@ are rejected. Catalogs should always expire. Long-lived catalogs make rollback and key compromise harder to reason about. +### Sequence-State Recovery + +The sequence state file is operational state, not a trust root. Keep it on +persistent storage and include it in normal backups: + +```json +{ + "channels": { + "stable": { + "last_sequence": 42, + "accepted_at": "2026-07-07T12:00:00Z", + "key_id": "release-key-1", + "source": "https://govoplan.example/catalogs/v1/channels/stable.json" + } + } +} +``` + +If the file is lost, restore it from backup. If no backup exists, reconstruct +each channel from the highest sequence already accepted in installer run +records, release records, or the currently deployed module package set. Do not +lower `last_sequence` to make an older catalog pass; publish a new higher +sequence catalog when the accepted point is uncertain. + +If the file is corrupted, copy it aside for incident review, validate the +current signed catalog with channel and freshness enforcement, then rewrite the +state with the known accepted sequence. Keep +`GOVOPLAN_MODULE_PACKAGE_CATALOG_REQUIRE_SIGNATURE=true` and approved-channel +checks enabled during recovery. Temporarily disabling +`GOVOPLAN_MODULE_PACKAGE_CATALOG_ENFORCE_SEQUENCE` allows revalidating the same +sequence, but older sequences remain rejected once the reconstructed +`last_sequence` is in place. + ## Release Channels Approved channels are deployment policy: @@ -169,10 +202,59 @@ License files are JSON objects with: - `valid_until` - `signature` +Issue or renew a license from an operator/release shell that has the Ed25519 +private key: + +```bash +govoplan-module-installer \ + --issue-license /srv/govoplan/license.json \ + --license-id customer-2026-07 \ + --license-subject "Example Municipality" \ + --license-feature module.mail \ + --license-feature support.standard \ + --license-valid-until 2027-07-31T23:59:59Z \ + --license-signing-key-id license-issuer-1 \ + --license-signing-private-key /srv/govoplan/secrets/license-issuer-1.pem \ + --format json +``` + +Validate an imported license without exposing secrets: + +```bash +govoplan-module-installer \ + --validate-license /srv/govoplan/license.json \ + --license-trusted-key license-issuer-1="" \ + --require-trusted-license \ + --license-required-feature module.mail \ + --format json +``` + +The CLI and admin module catalog panel report the license id, subject, +validity window, signing key id, signed/trusted state, available features, and +missing entitlements for the configured package catalog. They do not expose +private signing material. + License enforcement can run in observe-only mode by leaving `GOVOPLAN_LICENSE_ENFORCEMENT` unset. In that mode, missing or invalid license data is surfaced as a warning but does not block planning. +Renewal is an ordinary re-issuance with a new `license_id`, extended +`valid_until`, and the full intended feature set. Import the renewed JSON to +`GOVOPLAN_LICENSE_FILE`, keep the previous file for audit, and validate it +before setting enforcement. + +Revocation is handled through the trusted license keyring. Mark a compromised +or invalid issuer key as `revoked` or `disabled`, publish or deploy the updated +keyring, then reissue affected licenses with an active key. Installations that +run with `GOVOPLAN_LICENSE_ENFORCEMENT=true` reject licenses signed only by a +revoked key after the local keyring is updated. + +Emergency fallback is deliberately explicit. Operators can temporarily unset +`GOVOPLAN_LICENSE_ENFORCEMENT` to keep package planning observable while a +license or keyring is recovered. Record the change in the operational incident +log, keep catalog signature and channel enforcement enabled, and restore +license enforcement after a trusted renewal validates successfully. + Licensing is intentionally separate from open-source code licensing. The catalog/license mechanism can govern support channels, official release eligibility, hosted update access, professional support, or commercial @@ -180,12 +262,10 @@ entitlements without changing the source license of the repositories. ## Production Gaps -The current implementation validates signed catalog metadata and offline -license entitlements. Production-grade distribution still needs: +The current implementation validates signed catalog metadata, offline license +entitlements, and local artifact SHA-256 when artifact paths are supplied in an +install plan. Production-grade distribution still needs: -- artifact digest verification for package archives or registry artifacts -- SBOM/provenance fields and verification workflow +- remote registry/git artifact resolution before package-manager apply - hardened catalog publishing pipeline in `govoplan-web` - automated key rotation runbook and emergency revocation procedure -- license issuance and renewal tooling -- tests for remote catalog cache fallback and replay-state upgrades diff --git a/Repo-docs-CONFIGURATION-PACKAGES.md b/Repo-docs-CONFIGURATION-PACKAGES.md index 75c2942..0046f75 100644 --- a/Repo-docs-CONFIGURATION-PACKAGES.md +++ b/Repo-docs-CONFIGURATION-PACKAGES.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/CONFIGURATION_PACKAGES.md`. > Origin: `repository`. @@ -111,6 +111,33 @@ export(selection, context) -> fragment, data_placeholders, warnings health(import_result, context) -> diagnostics ``` +The initial core contract lives in +`govoplan_core.core.configuration_packages`. Modules register providers through +module-specific capability keys and implement the `ConfigurationProvider` +protocol. The generic `configuration.provider` key names the contract and can be +used in package requirements; concrete providers such as `access.configuration` +are resolved by the admin package wizard. The first DTO surface includes package +manifests, module/capability requirements, module-owned fragments, diagnostics, +required operator data, dry-run plan items, apply results, export selections, +and export results. + +The initial implementation includes provider-neutral orchestration helpers: + +- `dry_run_configuration_package(...)` +- `apply_configuration_package(...)` +- `export_configuration_package(...)` + +The first concrete provider is `govoplan_access.backend.configuration_provider`. +It supports access-owned `roles`, `groups`, and `group_role_assignments` +fragments and applies them idempotently. + +The admin wizard backend starts with these routes: + +- `GET /api/v1/admin/configuration-packages/catalog` +- `POST /api/v1/admin/configuration-packages/dry-run` +- `POST /api/v1/admin/configuration-packages/apply` +- `POST /api/v1/admin/configuration-packages/export` + ## Import Flow 1. Select a package from a trusted catalog or upload a package file. @@ -182,6 +209,20 @@ Configuration catalogs should follow the existing module package catalog model: a file-backed or remotely fetched JSON catalog with Ed25519 signatures, channel gating, trusted key ids, and operator-controlled trust policy. +The initial catalog validator mirrors the module catalog environment model with +configuration-specific names: + +```bash +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG=/srv/govoplan/configuration-catalogs/stable.json +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_URL=https://govoplan.example/configuration-catalogs/stable.json +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_CACHE=/srv/govoplan/runtime/configuration-catalog-cache/stable.json +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_REQUIRE_SIGNATURE=true +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_APPROVED_CHANNELS=stable,lts +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_TRUSTED_KEYS_FILE=/srv/govoplan/trust/configuration-catalog-keyring.json +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_SEQUENCE_STATE=/srv/govoplan/runtime/configuration-catalog-sequences.json +GOVOPLAN_CONFIGURATION_PACKAGE_CATALOG_ENFORCE_SEQUENCE=true +``` + Catalog entries should include: - package id, version, name, description, publisher, tags, and channel diff --git a/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md b/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md new file mode 100644 index 0000000..71d063a --- /dev/null +++ b/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md @@ -0,0 +1,310 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/DEPLOYMENT_OPERATOR_GUIDE.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# GovOPlaN Deployment Operator Guide + +This guide defines the current install/runtime configuration contract and the +operator flow for a production-realistic self-hosted deployment. Keep secrets in +the deployment environment or a secret manager; do not commit populated `.env` +files. + +## Runtime Configuration Contract + +### Required Runtime Identity + +| Setting | Required outside dev | Purpose | +| --- | --- | --- | +| `APP_ENV` | yes | Runtime profile. Use `prod`, `staging`, or a deployment-specific value outside local development. | +| `MASTER_KEY_B64` | yes | Fernet key or base64 encoded 32-byte key used for encrypted module secrets. Rotate through an explicit operator plan. | +| `DATABASE_URL` | yes | SQLAlchemy database URL for core and installed modules. SQLite is supported for dev/small installs; PostgreSQL is the preferred production target. | +| `ENABLED_MODULES` | yes | Comma-separated startup module set. Keep `tenancy,access` enabled; keep `admin` enabled for operator UI. | + +Generate a local key for a new non-production environment: + +```bash +python - <<'PY' +from cryptography.fernet import Fernet +print(Fernet.generate_key().decode()) +PY +``` + +### Database And Migrations + +| Setting | Default | Notes | +| --- | --- | --- | +| `DATABASE_URL` | `postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev` | Local development and production-like profiles use PostgreSQL. Use `GOVOPLAN_DEV_DATABASE_BACKEND=sqlite` only for disposable SQLite runs. | +| `DEV_AUTO_MIGRATE_ENABLED` | `true` | Dev convenience only. Production should run migration commands explicitly during deployment. | +| `DEV_BOOTSTRAP_ENABLED` | `false` | Dev bootstrap only. `govoplan_core.devserver` and `scripts/launch-dev.sh` default it to `true`; use controlled first-admin creation outside dev. | + +Operator rule: take a database backup before applying migrations or destructive +module retirement. For non-SQLite databases, configure deployment-specific +backup/restore hooks for the module installer. + +### PostgreSQL Production Target + +PostgreSQL is the primary development and production target. SQLite remains +supported only for tiny disposable profiles and unit-test style smoke runs. +Production/staging deployments should use a managed PostgreSQL database and +explicit migration commands. + +Install the server extra so the `psycopg` driver is available: + +```bash +cd /mnt/DATA/git/govoplan-core +./.venv/bin/python -m pip install -r requirements-release.txt +``` + +Example runtime database URLs: + +```bash +export DATABASE_URL='postgresql+psycopg://govoplan:change-me@db.example.internal:5432/govoplan' +export GOVOPLAN_DATABASE_URL_PGTOOLS='postgresql://govoplan:change-me@db.example.internal:5432/govoplan' +``` + +Use the SQLAlchemy URL for GovOPlaN. Use the pg-tools URL for `pg_dump`, +`pg_restore`, and `psql`; these tools do not understand the +`postgresql+psycopg://` driver marker. + +Bootstrap or upgrade the schema explicitly during deployment: + +```bash +export APP_ENV=prod +export ENABLED_MODULES=tenancy,access,admin,policy,audit,campaigns,files,mail,calendar,docs,ops +./.venv/bin/python -m govoplan_core.commands.init_db \ + --database-url "$DATABASE_URL" +``` + +Backup and restore-check before migration-bearing package changes: + +```bash +pg_dump --format=custom \ + --file "$PWD/runtime/govoplan-$(date +%Y%m%d%H%M%S).dump" \ + "$GOVOPLAN_DATABASE_URL_PGTOOLS" +pg_restore --list "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump" >/dev/null +``` + +Restore a checked backup to the target database: + +```bash +pg_restore --clean --if-exists \ + --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" \ + "$PWD/runtime/govoplan-YYYYMMDDHHMMSS.dump" +``` + +For local development, create the host database described in +`dev/postgres/README.md`, then run: + +```bash +cd /mnt/DATA/git/govoplan-core +./.venv/bin/python -m govoplan_core.commands.init_db \ + --database-url postgresql+psycopg://govoplan_dev@127.0.0.1:5432/govoplan_dev \ + --with-dev-data +./.venv/bin/python -m govoplan_core.devserver --smoke --no-reload +scripts/launch-dev.sh +``` + +For disposable local validation against a throwaway PostgreSQL instance, use +the bundled PostgreSQL testbed: + +```bash +cd /mnt/DATA/git/govoplan-core/dev/postgres +cp .env.example .env +docker compose --env-file .env up -d + +cd /mnt/DATA/git/govoplan-core +set -a +. dev/postgres/.env +set +a +./.venv/bin/python scripts/postgres-integration-check.py \ + --database-url "$GOVOPLAN_POSTGRES_DATABASE_URL" \ + --reset-schema +``` + +The integration check runs migrations and startup smoke checks across the +standard module permutations. `--reset-schema` is destructive and belongs only +on throwaway databases. + +### Broker And Workers + +| Setting | Default | Notes | +| --- | --- | --- | +| `REDIS_URL` | `redis://redis:6379/0` | Celery broker/result backend when async workers are enabled. | +| `CELERY_ENABLED` | `false` | Local/dev can send synchronously. Production campaign delivery should run workers and set this to `true`. | +| `CELERY_QUEUES` | `send_email,append_sent,default` | Queue list expected by worker/process manager definitions. | + +Worker command: + +```bash +python -m celery -A govoplan_core.celery_app:celery worker \ + --queues send_email,append_sent,default \ + --loglevel INFO +``` + +### Storage + +| Setting | Default | Notes | +| --- | --- | --- | +| `FILE_STORAGE_BACKEND` | `local` | Use `local` for dev/small deployments; use object storage when files must scale independently. | +| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Must live on durable storage and be backed up when `FILE_STORAGE_BACKEND=local`. | +| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Read-only fallback roots for migrated local files. | +| `FILE_STORAGE_S3_ENDPOINT_URL` | empty | Object-store endpoint for the files module. | +| `FILE_STORAGE_S3_REGION` | empty | Object-store region. | +| `FILE_STORAGE_S3_ACCESS_KEY_ID` | empty | Secret; inject through deployment environment. | +| `FILE_STORAGE_S3_SECRET_ACCESS_KEY` | empty | Secret; inject through deployment environment. | +| `FILE_STORAGE_S3_BUCKET` | `files` | Managed-file object bucket. | + +Legacy `S3_*` settings remain for older storage paths but new deployments should +prefer `FILE_STORAGE_*`. + +### HTTP, Cookies, And Base URLs + +| Setting | Default | Notes | +| --- | --- | --- | +| `CORS_ORIGINS` | local dev origins | Set to the exact WebUI origins in staging/production. | +| `AUTH_SESSION_COOKIE_NAME` | `msm_session` | Change only through a controlled rollout because it logs users out. | +| `AUTH_CSRF_COOKIE_NAME` | `msm_csrf` | Must match WebUI/API deployment. | +| `AUTH_COOKIE_SECURE` | `false` | Set `true` behind HTTPS. | +| `AUTH_COOKIE_SAMESITE` | `lax` | Use a stricter value only after testing login and CSRF flows. | +| `AUTH_COOKIE_DOMAIN` | empty | Set only when the API and WebUI intentionally share a parent domain. | + +Public URLs are currently supplied by deployment/reverse-proxy configuration and +module settings. Do not hardcode them in core; configuration packages should ask +for portal, WebUI, postbox, and notification URLs when they become relevant. + +### Module Catalogs, Licenses, And Trust Roots + +| Setting | Purpose | +| --- | --- | +| `GOVOPLAN_MODULE_PACKAGE_CATALOG_URL` or `GOVOPLAN_MODULE_PACKAGE_CATALOG` | Module package catalog source. | +| `GOVOPLAN_MODULE_PACKAGE_CATALOG_TRUSTED_KEYS_FILE` | Preferred production keyring path. | +| `GOVOPLAN_MODULE_PACKAGE_CATALOG_APPROVED_CHANNEL` | Approved catalog channel, for example `stable`. | +| `GOVOPLAN_LICENSE_TRUSTED_KEYS_FILE` | Trusted license issuer keyring path. | +| `GOVOPLAN_LICENSE_ENFORCEMENT` | Enables license enforcement when set to `true`. | + +Trust roots are deployment-managed and should not be editable through the +running WebUI. + +### Mail Test Credentials + +Dedicated SMTP/IMAP test credentials belong to the mail/campaign test-bed +configuration, not the core runtime contract. Store them in a local ignored +`.env` file for the test bed or in CI secrets. Required values are: + +- SMTP host, port, TLS mode, username, password, and envelope/from address. +- IMAP host, port, TLS mode, username, password, and append folder. +- At least one recipient mailbox that is safe for automated send tests. + +## First Deployment Flow + +1. Create an environment file or secret set with the runtime contract above. +2. Install the tagged core and module packages from `requirements-release.txt`. +3. Build the WebUI from `webui/package.release.json` or deploy a prebuilt + artifact from the same release tag. +4. Run database migrations with the target `DATABASE_URL`. +5. Create the first tenant and system owner through the controlled bootstrap or + one-time admin command for the deployment. +6. Start the API service with `govoplan_core.server.app:app`. +7. Start workers when `CELERY_ENABLED=true`. +8. Start the WebUI/reverse proxy and verify CORS/cookie settings. +9. Open Admin > System > Modules, verify enabled modules, and save desired + module state if it differs from `ENABLED_MODULES`. +10. Run health checks: + +```bash +curl -fsS http://127.0.0.1:8000/health +curl -fsS http://127.0.0.1:8000/api/v1/platform/modules +``` + +Authenticated health details require `system:settings:read`: + +```bash +curl -fsS -H "X-API-Key: $GOVOPLAN_HEALTH_API_KEY" \ + http://127.0.0.1:8000/health/details +``` + +## Production-Like Dev Profile + +Use this profile to verify deployment behavior without publishing packages or +using real production credentials. The canonical launcher keeps API, worker, and +WebUI code in the editable repositories while Docker provides PostgreSQL and +Redis: + +```bash +cd /mnt/DATA/git/govoplan-core +scripts/launch-production-like-dev.sh +``` + +The launcher uses `dev/production-like/.env` when present, otherwise the checked +in `.env.example`. It runs: + +- PostgreSQL on `127.0.0.1:55433` +- Redis on `127.0.0.1:56379` +- explicit `ENABLED_MODULES` +- explicit migrations and `--with-dev-data` bootstrap +- API via the module-aware devserver +- a Celery worker for `send_email,append_sent,default` +- WebUI through the Vite dev server +- durable local files under `runtime/production-like/files` + +This profile validates explicit migration execution, config loading, module +discovery, route aggregation, local storage paths, Redis broker connectivity, +worker heartbeats, and health/readiness startup without real production +credentials. It does not replace a managed PostgreSQL/Redis/WebUI/worker +deployment test. + +To stop PostgreSQL and Redis when the launcher exits: + +```bash +GOVOPLAN_STOP_PROFILE_DEPENDENCIES_ON_EXIT=1 scripts/launch-production-like-dev.sh +``` + +## Module Install/Uninstall Operations + +Use Admin > System > Modules for planning. Use the operator shell for package +changes: + +```bash +govoplan-module-installer --format shell +govoplan-module-installer --apply --build-webui +``` + +For production-like runs, prefer supervised mode with restart and health checks: + +```bash +govoplan-module-installer \ + --supervise \ + --migrate \ + --restart-command 'systemctl restart govoplan-api' \ + --restart-command 'systemctl restart govoplan-worker' \ + --health-url http://127.0.0.1:8000/health \ + --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ + --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \ + --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' +``` + +Run the rollback drill before relying on installer automation in a new +environment: + +```bash +./.venv/bin/python scripts/module-installer-rollback-drill.py --format json +``` + +See `RELEASE_DEPENDENCIES.md` for release package refs, catalog trust, +installer daemon operation, rollback records, and destructive module retirement. + +## Operator Checklist + +- Runtime secrets are injected outside git. +- `MASTER_KEY_B64` is set and backed up securely. +- Database backup and restore commands are tested. +- File/object storage is durable and backed up. +- `CORS_ORIGINS` and cookie settings match the deployed WebUI origin. +- Redis and workers are running before `CELERY_ENABLED=true`. +- Module catalog and license keyrings are pinned locally. +- Health endpoints are monitored. +- Test SMTP/IMAP credentials are non-production and isolated. +- Module installer rollback drill has passed in the deployment environment. diff --git a/Repo-docs-EVENTS-AND-AUDIT.md b/Repo-docs-EVENTS-AND-AUDIT.md index 975e9a5..704151c 100644 --- a/Repo-docs-EVENTS-AND-AUDIT.md +++ b/Repo-docs-EVENTS-AND-AUDIT.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/EVENTS_AND_AUDIT.md`. > Origin: `repository`. @@ -13,18 +13,49 @@ there is a concrete need for durable asynchronous command orchestration. Events are facts about completed work and are safe for audit, projections, optional module reactions, and operator diagnostics. -## Current Decision +## Production Transport Decision + +The first production target is a **database outbox plus in-process immediate +dispatch**: - Use `govoplan_core.core.events.PlatformEvent` for domain and platform events. -- Use `EventBus` as the in-process dispatch contract for immediate module - reactions. -- Keep durable dispatch out of the first API shape. A database outbox can wrap - the same `PlatformEvent` envelope later without changing event producers. +- Use `EventBus` as the in-process dispatch contract for same-process module + reactions that are safe to run inline. +- Persist durable integration/workflow events through a database outbox before + acknowledging the state change that produced them. +- Drain the outbox through a small dispatcher process. The dispatcher may call + in-process handlers in the same deployment first, but its storage contract is + database-backed. +- Treat Redis/Celery as worker/job infrastructure, not as the authoritative + first event transport. A Celery dispatcher can consume the outbox later. +- Keep the dispatch implementation pluggable behind the `PlatformEvent` + envelope so a future message broker can be added without changing event + producers. - Keep commands out of the kernel until workflows need retryable, durable, operator-visible command records. -This keeps the first contract small while preserving room for a DB outbox, -Redis/Celery dispatch, or workflow-owned command orchestration later. +This keeps the first contract small, PostgreSQL-friendly, auditable, and +recoverable after process crashes. It also avoids making Redis a correctness +dependency for deployments that only need synchronous mail/tests or light +background work. + +## Dispatch Semantics + +Event producers should write their domain state and outbox event in the same +database transaction wherever possible. Handlers must be idempotent because the +outbox dispatcher can retry after a crash or timeout. + +Recommended first outbox columns: + +- `event_id`, `event_type`, `module_id` +- `correlation_id`, `causation_id` +- `payload`, `occurred_at` +- `available_at`, `attempt_count`, `claimed_at`, `claim_token` +- `processed_at`, `last_error` + +Inline `EventBus` handlers are allowed only for non-critical local reactions. +Anything that must survive process failure, restart, package update, or worker +redeployment belongs in the outbox. ## Trace IDs @@ -43,6 +74,12 @@ Audit logging reads the current event context and stores trace IDs in `details._trace`. Callers can also pass explicit `correlation_id` and `causation_id` to `audit_event` or `audit_from_principal`. +Admin and lifecycle code should use the compact operational detail shape +documented in `govoplan-audit/docs/AUDIT_TRACE_CONTEXT.md`. The core +`audit_operation_context` helper preserves `module_id`, `request_id`, `run_id`, +`outcome`, and `_trace` while applying the shared audit redaction pass to +additional detail values. + ## Audit MVP Boundary `govoplan-audit` owns: diff --git a/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md b/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md index eb66c86..2ab1815 100644 --- a/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md +++ b/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/GOVERNMENT_OPERATIONS_VISION.md`. > Origin: `repository`. @@ -97,6 +97,11 @@ The output should be a ranked connector backlog with owners, protocols, authentication models, data shapes, operational risks, and minimum viable integration tests. +The core classification index lives in +`docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md`; the detailed connector catalogue +and executable target inventory live in +`govoplan-connectors/docs/PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md`. + ## Recurring Data Extraction And Transformation A concrete recurring use case is monthly extraction and transformation of data @@ -182,6 +187,10 @@ Sizing should account for tenants, concurrent users, cases, files, storage volume, upload/download rates, campaign volume, workflow jobs, transformation runs, reporting load, and retention/audit growth. +See `docs/SCALABILITY_AND_SIZING.md` for the first deployment topology, +readiness/degraded-mode model, worker scaling assumptions, PostgreSQL production +path, and sizing matrix. + ## Hardware Requirements Hardware guidance should be empirical and deployment-shaped. The first sizing @@ -197,6 +206,9 @@ Each profile should document CPU, memory, disk, database, storage, queue/cache, backup, and monitoring requirements, plus the assumptions behind the numbers. A later calculator can turn operator inputs into recommended profiles. +The initial matrix lives in `docs/SCALABILITY_AND_SIZING.md`; the calculator is +kept as a follow-up once real deployment measurements are available. + ## Collaboration Collaboration should be integration-first. GovOPlaN should connect with strong diff --git a/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md b/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md index 30e65f8..e40298c 100644 --- a/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md +++ b/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MODULE_ROADMAP.md`. > Origin: `repository`. @@ -7,7 +7,11 @@ --- # GovOPlaN Module And Integration Roadmap -This page maps current module and integration ideas to existing GovOPlaN repositories or to missing-module decisions. Issues are the active backlog. This document is durable routing context and should be mirrored to the Gitea wiki. +This page maps current module and integration ideas to existing GovOPlaN +repositories or to explicit missing-module decisions. Issues are the active +backlog. This document is durable routing context and should be mirrored to the +Gitea wiki. Boundary decisions are recorded in +`docs/MODULE_BOUNDARY_DECISIONS.md`. ## Current Routing @@ -18,48 +22,78 @@ This page maps current module and integration ideas to existing GovOPlaN reposit | Fully UI-managed configuration with safety controls | `govoplan-admin`, `govoplan-core`, `govoplan-policy`, `govoplan-access`, `govoplan-audit` | `add-ideas/govoplan-core#218` | | Access as a module | `govoplan-access` | `add-ideas/govoplan-access#7` | | OpenProject API / project management connector | `govoplan-connectors` | `add-ideas/govoplan-connectors#1` | -| Native project-management module decision | `govoplan-core` | `add-ideas/govoplan-core#196` | -| Datasources for databases, CSV, files, APIs | proposed `govoplan-datasources` | `add-ideas/govoplan-core#197` | -| Dataflow for pipelines, BI, publication | proposed `govoplan-dataflow` | `add-ideas/govoplan-core#198` | -| Monthly datasource and transformation workflows | proposed `govoplan-datasources`, proposed `govoplan-dataflow`, `govoplan-workflow`, `govoplan-reporting` | `add-ideas/govoplan-core#216` | -| Templates for letters, emails, forms, reports | `govoplan-templates` | `add-ideas/govoplan-templates#1` | -| Reporting and BI | `govoplan-reporting` | `add-ideas/govoplan-reporting#1` | +| Native project-management module decision | connector-first through `govoplan-connectors`; no native project module yet | `add-ideas/govoplan-core#196`, `add-ideas/govoplan-connectors#1` | +| Datasources for databases, CSV, files, APIs | no repository yet; start with connectors/files/reporting and create `govoplan-datasources` only after the first package proves shared source-catalog ownership | `add-ideas/govoplan-core#197` | +| Dataflow for pipelines, BI, publication | no repository yet; start with workflow/reporting/connectors and create `govoplan-dataflow` only after repeated pipeline/lineage contracts emerge | `add-ideas/govoplan-core#198` | +| Monthly datasource and transformation workflows | first as configuration package across connectors, files, workflow, reporting, and templates | `add-ideas/govoplan-core#216` | +| Templates for letters, emails, forms, reports | `govoplan-templates`, separate from reporting | `add-ideas/govoplan-core#190`, `add-ideas/govoplan-templates#1` | +| Reporting and BI | `govoplan-reporting`, separate from templates | `add-ideas/govoplan-core#190`, `add-ideas/govoplan-reporting#1` | | File connectors: Nextcloud, Seafile, SMB, NFS | `govoplan-files` | `add-ideas/govoplan-files#15` | -| Public-sector software integration catalogue | `govoplan-connectors` | `add-ideas/govoplan-connectors#2` | +| Public-sector software integration catalogue | `govoplan-connectors` with core strategy index | `add-ideas/govoplan-core#191`, `add-ideas/govoplan-connectors#2` | | Public-sector integration landscape catalogue | `govoplan-connectors` with core tracking | `add-ideas/govoplan-core#215` | +| Cases module concept | `govoplan-cases` | `add-ideas/govoplan-core#174` | +| Workflow module concept | `govoplan-workflow` | `add-ideas/govoplan-core#175` | +| Connectors module concept | `govoplan-connectors` | `add-ideas/govoplan-core#176` | | Adrema-style address and distribution-list management | `govoplan-addresses` | `add-ideas/govoplan-addresses#1` | | Consume sources and become a governed source | `govoplan-connectors` plus proposed `govoplan-dataflow` | `add-ideas/govoplan-connectors#3`, `add-ideas/govoplan-core#198` | -| Terminfindung and meeting scheduling polls | `govoplan-scheduling` | local scaffold; remote creation tracked by `add-ideas/govoplan-core#199` | -| Terminplaner and calendar primitives | `govoplan-calendar` | `add-ideas/govoplan-calendar#1` | -| Terminbuchung appointment booking | `govoplan-appointments` | `add-ideas/govoplan-appointments#1` | +| Terminfindung and meeting scheduling polls | `govoplan-scheduling`; calendar primitives remain in calendar | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-scheduling#1` | +| Terminplaner and calendar primitives | `govoplan-calendar` | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-calendar#1` | +| Terminbuchung appointment booking | `govoplan-appointments` | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-appointments#1` | | Collaborative documents | `govoplan-dms` | `add-ideas/govoplan-dms#1` | -| Forms | `govoplan-forms` | `add-ideas/govoplan-forms#1` | +| Forms | `govoplan-forms` for definitions and `govoplan-forms-runtime` for submissions/runtime behavior | `add-ideas/govoplan-core#194`, `add-ideas/govoplan-forms#1` | | RSS consume and emit | `govoplan-connectors` | `add-ideas/govoplan-connectors#4` | | LDAP, Active Directory, OpenDesk identity | `govoplan-idm` | `add-ideas/govoplan-idm#1` | -| OpenDesk stack integration map | `govoplan-connectors` | `add-ideas/govoplan-connectors#5` | +| OpenDesk stack integration map | integration profile across IDM/access, mail/calendar, files/DMS, and connectors; not a monolithic module | `add-ideas/govoplan-core#195`, `add-ideas/govoplan-connectors#5` | | Open-Xchange mail/groupware | `govoplan-mail` | `add-ideas/govoplan-mail#5` | | Open-Xchange calendar | `govoplan-calendar` | `add-ideas/govoplan-calendar#2` | | Scalability profiles and autoscaling readiness | `govoplan-ops`, `govoplan-core` | `add-ideas/govoplan-core#217` | | Hardware sizing matrix and requirements calculator | `govoplan-ops`, `govoplan-core` | `add-ideas/govoplan-core#219` | | Collaboration suite integration strategy | `govoplan-connectors`, `govoplan-dms`, `govoplan-workflow`, `govoplan-tasks`, `govoplan-appointments`, `govoplan-calendar` | `add-ideas/govoplan-core#220` | +| Install/runtime configuration contract | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#19` | +| Installer/deployment operator flow | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#26` | +| Production-like deployment documentation | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#28` | ## Proposed Missing Modules -`govoplan-datasources` should exist only if source catalog ownership becomes broad enough to justify a module separate from connectors and reporting. Candidate responsibilities: +## Boundary Decision Register + +`docs/MODULE_BOUNDARY_DECISIONS.md` is the durable decision register for older +roadmap and boundary issues. Current decisions: + +- templates and reporting are separate modules +- RSS/source consume-publish starts in connectors; datasources/dataflow are not + repositories yet +- calendar, scheduling, and appointments are three separate modules +- forms definitions and forms runtime are separate responsibilities +- OpenDesk is an integration profile across modules, not a monolithic module +- OpenProject is connector-first; no native projects module yet +- public-sector integration strategy stays in core; executable catalogue work + lives in connectors + +## Proposed Missing Modules + +`govoplan-datasources` is not created yet. It should exist only if source +catalog ownership becomes broad enough to justify a module separate from +connectors, files, and reporting. Candidate responsibilities: - source catalogue for SQL databases, CSV/Excel files, uploaded files, APIs, RSS feeds, and governed file locations - credentials and connection profiles - schema discovery and refresh cadence - provenance, freshness, permission boundaries, and audit events -`govoplan-dataflow` should exist only if pipelines and publication become first-class product behavior. Candidate responsibilities: +`govoplan-dataflow` is not created yet. It should exist only if pipelines and +publication become first-class product behavior beyond workflow, connectors, +and reporting. Candidate responsibilities: - ingestion, transformation, validation, scheduling, and lineage - connecting datasources to reports, APIs, RSS, exports, and downstream systems - the "consume sources, become source" lifecycle - publication-state and audit integration -`govoplan-projects` is undecided. The default path should be an OpenProject connector first. A native module is justified only if GovOPlaN must own project semantics beyond cases, tasks, workflow, appointments, documents, and reporting. +`govoplan-projects` is intentionally not created yet. The decision is +connector-first through OpenProject. A native module is justified only if +GovOPlaN must own project semantics beyond cases, tasks, workflow, +appointments, documents, and reporting. ## Boundary Notes @@ -74,6 +108,18 @@ This page maps current module and integration ideas to existing GovOPlaN reposit - `govoplan-reporting` owns report definitions, BI views, scheduled outputs, and export targets. - `govoplan-templates` owns reusable renderable templates, not data selection or persistence. +## Integration Catalogue Routing + +Core keeps the strategy index in +`docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md`: integration postures, default +ownership, and prioritization rules. `govoplan-connectors` owns the detailed +target inventory and connector entry shape in +`govoplan-connectors/docs/PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md`. + +When a target needs executable behavior, create the implementation issue in the +owning module repository and keep only the cross-module routing or architecture +decision in core. + ## Release Tooling Note `scripts/push-release-tag.sh` covers the released full-product package set: @@ -91,4 +137,6 @@ This page maps current module and integration ideas to existing GovOPlaN reposit It also includes existing roadmap/scaffold module repositories such as addresses, appointments, connectors, DMS, forms, IDM, reporting, scheduling, templates, workflow, XOE/V, and XRechnung as tag-only repositories. Tag-only repositories are committed, tagged, and pushed with the same release tag, but they are not added to `requirements-release.txt` or `webui/package.release.json` until they contain installable package metadata. -Proposed modules without repositories, such as `govoplan-datasources`, `govoplan-dataflow`, and possibly `govoplan-projects`, cannot be included until their repositories are created. +Proposed modules without repositories, such as `govoplan-datasources`, +`govoplan-dataflow`, and `govoplan-projects`, cannot be included until a later +decision creates their repositories. diff --git a/Repo-docs-MODULE-ARCHITECTURE.md b/Repo-docs-MODULE-ARCHITECTURE.md index a00d8f0..28ee8e8 100644 --- a/Repo-docs-MODULE-ARCHITECTURE.md +++ b/Repo-docs-MODULE-ARCHITECTURE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/MODULE_ARCHITECTURE.md`. > Origin: `repository`. @@ -13,6 +13,12 @@ The current package name is still `govoplan-core`, but the architecture target i The concrete access/auth/RBAC extraction path is tracked in [`ACCESS_EXTRACTION_PLAN.md`](ACCESS_EXTRACTION_PLAN.md). +The event and audit trace contract is tracked in +[`EVENTS_AND_AUDIT.md`](EVENTS_AND_AUDIT.md). +Policy decision, source provenance, and explain-response contracts are tracked +in [`POLICY_CONTRACTS.md`](POLICY_CONTRACTS.md). +The experimental remote WebUI bundle loading design is tracked in +[`REMOTE_WEBUI_BUNDLES.md`](REMOTE_WEBUI_BUNDLES.md). ## Layer Model @@ -42,14 +48,15 @@ The kernel must not own product semantics such as users, tenants, RBAC decisions ## Current Compatibility Responsibilities During the staged split, `govoplan-core` still contains compatibility surfaces -for access, auth, tenancy, RBAC, governance, audit, CSRF/API helpers, and -secret helpers. The extracted access implementation now lives in +for tenancy settings, governance/policy contracts, audit helpers, CSRF/API +helpers, and secret helpers. The extracted access implementation lives in `govoplan-access`; live legacy ORM table definitions have been split across their platform owners while retaining historical table names. The old core -model, route, admin-service, and access-security import shims have been -removed; callers must use module-owned imports or kernel capabilities. The -remaining compatibility surfaces are temporary until the matching platform -modules are fully self-contained: +route, admin-service, and access-security re-export modules have been removed. +Callers must use module-owned imports, the public `govoplan_access.auth` request +dependency API, or kernel capabilities. +The remaining platform compatibility surfaces are temporary until the matching +platform modules are fully self-contained: - `govoplan-access` - `govoplan-tenancy` @@ -78,6 +85,7 @@ The following contracts are the baseline API that modules can rely on: - WebUI module contribution contract - navigation metadata contract - command/event envelope contract +- policy decision and source provenance contract in `govoplan_core.core.policy` Changes to these contracts must be versioned or accompanied by compatibility shims. @@ -106,10 +114,10 @@ access/tenant ORM models when they need labels, group membership, default access provisioning, counts, audit actor labels, or tenant metadata. FastAPI route dependencies for authenticated endpoints are access-owned and -published from `govoplan_access.backend.auth.dependencies`. Routers may import -that dependency module directly until a more generic request-principal adapter -exists; they must not import access ORM models or other access implementation -internals. +published from `govoplan_access.auth`. Routers may import that public API for +`ApiPrincipal`, `get_api_principal`, `has_scope`, `require_scope`, and +`require_any_scope`; they must not import access ORM models or +`govoplan_access.backend.*` implementation internals. Current live table ownership: @@ -216,6 +224,10 @@ Rules: - Optional module migrations may create multiple Alembic heads. Verification should compare the database heads to the configured script heads instead of assuming one linear revision when multiple modules are enabled. +- Treat migrations as release artifacts. Unreleased migrations may be squashed + or rewritten before a stable release; released revision IDs are immutable + once an installation may have recorded them. Each stable release records its + public migration heads in `docs/migration-release-baselines.json`. ## Install, Uninstall, And Catalogs @@ -361,8 +373,7 @@ The repository includes `scripts/check_dependency_boundaries.py`. It enforces th - access source may not import files/mail/campaign internals - feature modules may not import access implementation internals - feature modules may not add new direct imports of sibling feature modules -- FastAPI routers may import the published - `govoplan_access.backend.auth.dependencies` dependency API +- FastAPI routers may import the published `govoplan_access.auth` dependency API - the transitional allowlist is expected to stay empty Any future exception is extraction debt and must be temporary, documented in the @@ -572,6 +583,8 @@ directory with: - `GOVOPLAN_INSTALLER_RUN_DIR` - `GOVOPLAN_DATABASE_URL` +- `GOVOPLAN_DATABASE_URL_PGTOOLS` for PostgreSQL URLs converted to the + `postgresql://` form expected by `pg_dump`, `pg_restore`, and `psql` - `GOVOPLAN_DATABASE_BACKUP_PATH` - `GOVOPLAN_DATABASE_BACKUP_METADATA` diff --git a/Repo-docs-MODULE-BOUNDARY-DECISIONS.md b/Repo-docs-MODULE-BOUNDARY-DECISIONS.md new file mode 100644 index 0000000..7d6789b --- /dev/null +++ b/Repo-docs-MODULE-BOUNDARY-DECISIONS.md @@ -0,0 +1,215 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/MODULE_BOUNDARY_DECISIONS.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Module Boundary Decisions + +This document records durable boundary decisions that close older exploratory +core issues. Implementation work should move to the owning module repositories +once a boundary is clear. + +## Decision Principles + +- Prefer connector-first when an external specialist system is likely to remain + the system of record. +- Create a native module only when GovOPlaN must own domain semantics, + permissions, audit, retention, configuration-package fragments, or workflow + state. +- Keep optional behavior behind core-mediated capabilities, events, DTOs, route + contributions, and UI contribution points. +- Do not create repositories just because a possible product area exists. + +## Templates And Reporting + +Tracking: `govoplan-core#190`, `govoplan-templates#1`, +`govoplan-reporting#1`. + +Decision: templates and reporting are separate modules. + +`govoplan-templates` owns: + +- reusable renderable templates for letters, permits, emails, forms, reports, + certificates, and notices +- template versioning, merge-field declarations, rendering profiles, output + format choices, and preview contracts +- template package fragments that other modules can reference + +`govoplan-reporting` owns: + +- report definitions, data selection, dashboards, BI views, scheduled outputs, + and export targets +- report permissions, report execution history, generated report evidence, and + report-specific retention inputs +- downstream export handoff to files, dataflow, connectors, or publication + surfaces + +Boundary: + +- Templates do not own data selection, aggregation, scheduling, or BI semantics. +- Reporting may call template rendering through a capability when a formatted + report output is needed. +- Campaign, mail, files, workflow, and cases use templates/reporting through + capabilities and DTOs, never direct imports. + +## Sources, RSS, Datasources, And Dataflow + +Tracking: `govoplan-core#192`, `govoplan-core#197`, +`govoplan-core#198`, `govoplan-connectors#3`, +`govoplan-connectors#4`. + +Decision: do not create `govoplan-datasources` or `govoplan-dataflow` until a +first executable use case proves that connector/reporting/workflow ownership is +too narrow. + +First slice: + +- `govoplan-connectors` owns RSS/Atom consume/emit connector profiles, + connector health, external references, source lifecycle metadata, and + source/publish capability boundaries. +- `govoplan-files` owns file-backed governed locations and uploaded/stored file + evidence. +- `govoplan-reporting` owns report/data views and scheduled outputs. +- `govoplan-workflow` owns process state, approvals, scheduling of process + steps, and human review. + +Future `govoplan-datasources` is justified when GovOPlaN needs a broad source +catalogue for SQL databases, CSV/Excel files, APIs, RSS feeds, uploaded files, +and governed file locations with shared ownership, credentials, schema +discovery, refresh cadence, provenance, and permission boundaries. + +Future `govoplan-dataflow` is justified when GovOPlaN needs first-class +pipelines for ingestion, transformation, validation, scheduling, lineage, +publication, audit events, reruns, and source-to-source workflows. + +Monthly extraction/transformation work should start as a configuration package +and module collaboration across connectors, files, workflow, reporting, and +possibly templates. Create datasources/dataflow repositories only after that +package exposes repeated contracts that do not belong to an existing module. + +## Calendar, Scheduling, And Appointments + +Tracking: `govoplan-core#193`, `govoplan-calendar#1`, +`govoplan-calendar#2`, `govoplan-scheduling#1`, +`govoplan-appointments#1`. + +Decision: use three separate modules. + +`govoplan-calendar` owns: + +- calendar collections, events, recurrence, availability/free-busy, resources, + iCalendar import/export, CalDAV/Open-Xchange-style calendar adapters, and + calendar WebUI surfaces + +`govoplan-scheduling` owns: + +- Terminfindung, meeting-time polls, participant availability collection, + candidate-slot ranking, conflict explanations, reminders, and the handoff + from a selected slot to calendar/appointment/workflow modules + +`govoplan-appointments` owns: + +- Terminbuchung/fixed-slot appointment booking, appointment types, booking + rules, capacity, cancellation/no-show state, public/internal booking flows, + and appointment evidence + +Boundary: + +- Calendar provides time primitives and external calendar integration. +- Scheduling chooses a suitable time. +- Appointments owns booked appointment workflows and public/internal booking + semantics. +- Mail and notifications deliver invitations/reminders through capabilities. + +## Forms And Workflow Handoff + +Tracking: `govoplan-core#194`, `govoplan-forms#1`. + +Decision: forms are a reusable module boundary, with runtime behavior separated +from workflow semantics. + +`govoplan-forms` owns: + +- form definitions, schemas, validation rules, field visibility rules, + localization, versioning, admin editing, and reusable form package fragments + +`govoplan-forms-runtime` owns, when implemented: + +- public/internal submissions, drafts, submitted values, validation evidence, + attachment references, submission receipts, and handoff events + +Boundary: + +- Forms do not own cases, workflow transitions, tasks, or portal identity. +- Workflow/cases consume form submission events and evidence references. +- Files owns uploaded file storage and file permissions. +- Reporting/dataflow may consume submitted data through governed DTOs or + source lifecycle contracts. + +## OpenDesk Integration Profile + +Tracking: `govoplan-core#195`, `govoplan-connectors#5`, +`govoplan-idm#1`, `govoplan-mail#5`, `govoplan-calendar#2`, +`govoplan-connectors#1`. + +Decision: OpenDesk is an integration profile, not a monolithic module. + +Ownership: + +- identity: `govoplan-idm` plus `govoplan-access` +- mail/groupware: `govoplan-mail` +- calendar: `govoplan-calendar` +- files/documents: `govoplan-files` and later `govoplan-dms` +- projects/tasks: `govoplan-connectors` OpenProject connector first +- inventory/health/profile diagnostics: `govoplan-connectors` + +The OpenDesk profile should describe required connector profiles, shared +identity assumptions, health checks, and optional module combinations. It must +not create direct module-to-module imports. + +## Project Management And OpenProject + +Tracking: `govoplan-core#196`, `govoplan-connectors#1`. + +Decision: connector-first. Do not create a native `govoplan-projects` module +yet. + +OpenProject integration belongs in `govoplan-connectors` first: + +- profile test +- project and work-package lookup +- external-reference storage +- selected publish/synchronize capabilities for tasks, workflow, or cases + +A native project module is justified only if GovOPlaN needs to own project +semantics beyond cases, tasks, workflow, appointments, documents, and reporting, +for example portfolios, project budgets, project-level resource planning, or +governed project records that cannot remain in OpenProject. + +## Public-Sector Integration Landscape + +Tracking: `govoplan-core#186`, `govoplan-core#215`, +`govoplan-connectors#2`, `govoplan-connectors#3`. + +Decision: core owns strategy and routing; connectors owns executable +integration catalogue entries and operator inventory. + +Core documents: + +- product-level integration strategy +- native-vs-connector decisions +- owning module routing +- roadmap sequencing + +`govoplan-connectors` owns: + +- connector entry schema +- external system catalogue +- connector profiles and diagnostics +- source consume/publish lifecycle +- external references + +When a target needs executable behavior, create the implementation issue in the +owning module repository and keep only cross-module decisions in core. diff --git a/Repo-docs-POLICY-CONTRACTS.md b/Repo-docs-POLICY-CONTRACTS.md new file mode 100644 index 0000000..308acd8 --- /dev/null +++ b/Repo-docs-POLICY-CONTRACTS.md @@ -0,0 +1,112 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/POLICY_CONTRACTS.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# GovOPlaN Policy Contracts + +GovOPlaN has several policy families that are moving out of core into owning +modules. The shared kernel contract keeps their decision and provenance shape +consistent while each module still owns its domain rules. + +## Current Policy Inventory + +| Policy area | Current owner | Runtime surface | Notes | +| --- | --- | --- | --- | +| Privacy retention | `govoplan-policy` routes with compatibility helpers in core | `/api/v1/admin/privacy-retention/policies/{scope}` and `/explain` | System, tenant, user, group, and campaign sources merge into the effective retention policy. Parent locks block lower-level widening. | +| Mail profile policy | `govoplan-mail` | `/api/v1/mail/policies/{scope}` | Uses the same source-step path format for system, tenant, owner, and campaign provenance. | +| RBAC/access policy | `govoplan-access` | access capabilities in `govoplan_core.core.access` | Permission decisions should use access capability contracts. Explain responses should adopt `PolicyDecision` when an API-level explanation is added. | +| Governance defaults | `govoplan-admin` plus `govoplan-access` materializer | admin settings, governance template routes, access materialization capability | System governance can block tenant-local groups, roles, and API keys. | +| Delegation and ownership policy | access/campaign/mail/files modules | capability checks and owner-scoped APIs | Source provenance should use this contract when policies become externally explainable. | + +## Policy Decision + +The shared DTO lives in `govoplan_core.core.policy.PolicyDecision`. + +```json +{ + "allowed": false, + "reason": "Parent retention policy locks lower-level changes.", + "source_path": [ + { + "scope_type": "system", + "scope_id": null, + "path": "system", + "label": "System", + "applied_fields": ["allow_lower_level_limits"], + "policy": {} + } + ], + "requirements": ["raw_campaign_json_retention_days"], + "details": { + "blocked_fields": ["raw_campaign_json_retention_days"] + } +} +``` + +`allowed` is the effective answer for the checked action. `reason` is a stable, +human-readable summary. `source_path` lists the policy sources that explain the +answer. `requirements` lists machine-readable blockers or prerequisites, and +`details` carries domain-specific structured context. + +Every source step should be concrete enough for an operator to understand the +decision without knowing internal merge rules. Use real scope labels such as +`System`, `Tenant`, `Owner user`, `Group`, or a campaign/profile name. Include +the stable `path`, the fields applied by that step, and the local policy +fragment that caused them. This lets UIs render explanations like +`System: Allow > Tenant: Deny without override` without additional lookups. +If a policy family cannot expose the full local fragment for security reasons, +it must still include a redacted structured value that identifies the applied +field and the effective allow/deny or lock state. + +## Source Path Format + +Policy source paths are stable string identifiers for provenance steps: + +- `system` +- `:` + +Supported scope types are `system`, `tenant`, `user`, `group`, and `campaign`. +Examples: + +- `tenant:4a45b4fe-1d86-43ce-9d10-6022333f4d4b` +- `campaign:campaign%2Fwith%20space` + +Use `policy_source_path()` and `parse_policy_source_path()` instead of building +or splitting these strings manually. + +## Retention Explain Endpoint + +`GET /api/v1/admin/privacy-retention/policies/{scope_type}/explain` returns: + +- `scope_type` and optional `scope_id` +- `decision`, using the shared `PolicyDecision` shape +- `effective_policy` +- optional `parent_policy` +- `effective_policy_sources` +- `parent_policy_sources` +- `blocked_fields` + +The endpoint is read-only. Enforcement remains in the existing policy write +path. For lower-level scopes, `blocked_fields` is derived from the parent +policy's `allow_lower_level_limits`; clients can use it to disable local +controls before attempting a write. + +## Frontend Contract + +Policy UIs must: + +- render effective source provenance when `effective_policy_sources` is present +- display a field-level path when the source data is shown next to a specific + setting, using concrete source labels and stop at the first non-overridable + deny/lock +- disable local field controls when the parent policy sets that field's + lower-level limit to `false` +- avoid sending locked fields or re-enable attempts in save payloads +- show inherited values separately from local overrides + +The core WebUI helper `privacyRetentionParentAllowsField()` centralizes the +field-lock decision used by the retention editor and its lightweight module +tests. diff --git a/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md b/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md new file mode 100644 index 0000000..46465f5 --- /dev/null +++ b/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md @@ -0,0 +1,283 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Public-Sector Integration Strategy + +GovOPlaN should integrate with the existing public-sector software landscape +before deciding to replace specialist workflows. This document is the core +strategy index. The executable connector catalogue lives in +`govoplan-connectors/docs/PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md`. + +## Strategy Labels + +Use one or more of these labels for every external system family: + +- `integrate`: GovOPlaN talks to the system through a stable API/protocol. +- `link`: GovOPlaN stores external references and opens the external system for + source-of-truth work. +- `import`: GovOPlaN consumes data or files into governed module storage. +- `synchronize`: GovOPlaN keeps selected records aligned both ways or through a + source-of-truth rule. +- `replace selected workflow`: GovOPlaN may own a narrow workflow where the + external product is weak, but does not replace the whole product family. +- `no first-class support`: GovOPlaN only stores manual references unless a + deployment project creates a specific connector. + +## Initial Classification + +| System family | Examples | Default strategy | Likely owner | +| --- | --- | --- | --- | +| File providers | SMB/CIFS, WebDAV, Nextcloud, Seafile, SFTP, S3 | integrate, import, link | `govoplan-files`, connector inventory in `govoplan-connectors` | +| Project/task management | OpenProject, Jira, Redmine, Microsoft Planner | link, synchronize selected records, replace selected workflow only after proof | `govoplan-connectors`, later `govoplan-tasks` or workflow modules | +| Identity providers | LDAP, Active Directory, OIDC, SAML, OpenDesk IDM | integrate, synchronize | `govoplan-idm`, `govoplan-access` | +| Mail and groupware | IMAP/SMTP, Open-Xchange, Exchange/M365, CalDAV/CardDAV | integrate, link | `govoplan-mail`, `govoplan-calendar`, `govoplan-connectors` | +| DMS/e-file/archive | d.velop/d.3, enaio, ELO, Fabasoft, CMIS, VIS/eAkte | link, import, synchronize selected metadata | `govoplan-dms`, `govoplan-files`, `govoplan-records` | +| ERP/finance/procurement | SAP, MACH, Infoma, DATEV, procurement feeds | export, import, synchronize; do not replace by default | `govoplan-erp`, `govoplan-procurement`, `govoplan-ledger`, `govoplan-payments` | +| Public-sector transport | FIT-Connect, XTA/OSCI, Peppol access points | integrate, publish, receive | dedicated protocol modules plus `govoplan-connectors` inventory | +| Standards registries | XRepository, XOE/V catalogues | link, import metadata/cache | `govoplan-connectors`, `govoplan-xoev` | +| Publication/data exchange | RSS, open-data APIs, API feeds, CSV/Excel drops | consume, publish, transform | proposed `govoplan-datasources`, proposed `govoplan-dataflow`, `govoplan-reporting` | +| Collaboration suites | Matrix, Jitsi, BigBlueButton, Nextcloud Talk, Collabora/OnlyOffice | integrate, link; native behavior only for governed evidence | `govoplan-connectors`, `govoplan-dms`, `govoplan-workflow` | +| Specialist Fachverfahren | register-specific and domain-specific systems | link first; integrate/import when a real project supplies contracts | domain module or deployment-specific connector | + +## Landscape Catalogue + +This catalogue is intentionally implementation-oriented. Each entry records the +first API/auth/data assumptions needed to turn an inventory entry into a +connector or module issue. + +### Citizen And Service Portals + +- Strategy: integrate/link first; replace selected intake workflow only when a + GovOPlaN portal package owns the complete journey. +- Protocol/API surface: REST/JSON APIs, form submission webhooks, OIDC/SAML + login, eID interfaces where available, file-upload callbacks, case-status + callbacks. +- Auth model: OIDC/SAML service clients, signed webhook secrets, tenant-scoped + API keys, later eID/trust-provider handoff. +- Data shape: applicant identity reference, application form payload, + attachment references, consent declarations, status events, receipt IDs. +- Deployment assumptions: externally reachable HTTPS, reverse proxy, portal DMZ + separation, strict CSRF/origin settings, large upload path. +- Risks: personal data exposure, duplicate identity mapping, partial + submissions, upload malware, inconsistent portal status models. +- MVP test path: submit a test application with one file, create a form + submission/case/task stub, return a receipt and status reference. +- Owner/priority: `govoplan-portal`, `govoplan-forms-runtime`, + `govoplan-files`, Wave 1. + +### DMS, E-File, Records, And Archives + +- Strategy: link/import/synchronize selected metadata; do not replace the DMS by + default. +- Protocol/API surface: CMIS, WebDAV, vendor REST APIs, S3/object archive + staging, file-plan export/import, archive handoff APIs. +- Auth model: service accounts, OAuth/OIDC where supported, mTLS for regulated + archives, secret references for vendor tokens. +- Data shape: document ID, version, file-plan/classification code, retention + metadata, owner/case reference, external URL, checksum, lock/legal-hold state. +- Deployment assumptions: usually internal network or VPN, strict storage + quotas, existing retention policies, archive immutability requirements. +- Risks: record duplication, broken legal hold, permission mismatch, version + drift, destructive retention/export mistakes. +- MVP test path: create a connector inventory entry, test read-only metadata + lookup, link one GovOPlaN file/case evidence item to an external document. +- Owner/priority: `govoplan-dms`, `govoplan-records`, `govoplan-files`, + `govoplan-connectors`, Wave 2/5. + +### ERP, Finance, Procurement, And Accounting + +- Strategy: export/import/synchronize selected records; replacement only by + narrow domain decision. +- Protocol/API surface: vendor REST/SOAP APIs, CSV/XML batch exchange, SFTP, + XRechnung/Peppol, XBestellung/procurement feeds, payment reconciliation files. +- Auth model: service accounts, client certificates, mTLS, SFTP keys, token + references, environment-specific account separation. +- Data shape: debtor/creditor reference, payment request, invoice, order, + budget/cost-center code, booking status, receipt/evidence reference. +- Deployment assumptions: batch windows, finance-system approval workflows, + test tenants often separated from production by vendor process. +- Risks: financial posting errors, double export, tax/legal data retention, + inconsistent master data, irreversible accounting handoff. +- MVP test path: dry-run export of one payment/accounting handoff file with + checksum, validation report, and no remote posting. +- Owner/priority: `govoplan-payments`, `govoplan-ledger`, + `govoplan-xrechnung`, `govoplan-erp`, `govoplan-procurement`, Wave 1/6. + +### Identity, IAM, And Directory Services + +- Strategy: integrate/synchronize; access remains GovOPlaN's local + authorization boundary. +- Protocol/API surface: LDAP, Active Directory, SCIM, OIDC, SAML, OpenDesk IDM + APIs, group membership sync, account deactivation feeds. +- Auth model: bind accounts, service clients, OIDC/SAML metadata, SCIM tokens, + certificate-backed clients where required. +- Data shape: account, user, group, membership, role claim, tenant/org-unit + mapping, status, external directory ID. +- Deployment assumptions: directory is usually internal; identity provider may + be organization-wide and not GovOPlaN-owned. +- Risks: privilege escalation through group mapping, stale memberships, account + collision, deprovisioning latency, tenant-boundary mistakes. +- MVP test path: read-only directory profile test, map one external group to a + tenant group, show a dry-run membership diff. +- Owner/priority: `govoplan-idm`, `govoplan-access`, Wave 1. + +### Groupware, Mail, Calendar, And Collaboration + +- Strategy: integrate/link; native behavior only where GovOPlaN needs governed + evidence or process state. +- Protocol/API surface: IMAP/SMTP, CalDAV/CardDAV, Open-Xchange APIs, Microsoft + Graph/EWS, Matrix APIs, Jitsi/BigBlueButton APIs, Collabora/OnlyOffice + integration points. +- Auth model: service accounts, delegated OAuth/OIDC, app passwords, mailbox + credentials, groupware-specific tokens, secret references. +- Data shape: mailbox folder/message references, event/free-busy data, meeting + URL, chat room ID, participant list, document-editing session reference. +- Deployment assumptions: often internal/existing tenant infrastructure; mail + and calendar may be separate from identity even in OpenDesk-style stacks. +- Risks: mail credential exposure, calendar privacy, double invitations, room + booking conflicts, chat/document data escaping retention rules. +- MVP test path: profile test for mailbox/calendar reachability, read-only + folder/free-busy lookup, create a non-production event/message draft. +- Owner/priority: `govoplan-mail`, `govoplan-calendar`, + `govoplan-connectors`, Wave 1/2. + +### Payment And Public Cashier Systems + +- Strategy: integrate/export/import; keep the payment provider or cashier as + source of settlement truth. +- Protocol/API surface: payment provider APIs, redirect/callback flows, + reconciliation files, SEPA/export formats, cash-register/cashier interfaces. +- Auth model: provider API keys, signed webhooks, client certificates, mTLS, + tenant-specific merchant accounts. +- Data shape: payment intent, amount/currency, payer reference, provider + transaction ID, settlement status, receipt, refund/cancellation reference. +- Deployment assumptions: public callback URLs, strict environment separation, + PCI-sensitive providers, finance reconciliation cadence. +- Risks: duplicate charges, callback replay, amount mismatch, refund workflow + gaps, evidence-retention mistakes. +- MVP test path: sandbox payment intent, signed callback verification, receipt + evidence link, reconciliation dry-run. +- Owner/priority: `govoplan-payments`, `govoplan-ledger`, Wave 1/6. + +### Reporting, BI, Open Data, And Publication + +- Strategy: consume/publish/transform; native reporting owns GovOPlaN views, not + every external BI product. +- Protocol/API surface: SQL read replicas, CSV/Excel export/import, REST APIs, + RSS/Atom, open-data APIs, SFTP/WebDAV publication targets. +- Auth model: read-only DB users, API tokens, SFTP keys, OAuth clients, public + anonymous publication profiles where appropriate. +- Data shape: dataset metadata, schema/version, report parameters, generated + file references, publication URL, freshness/lineage, validation results. +- Deployment assumptions: publication can be public or internal; generated + datasets need retention and provenance. +- Risks: leaking restricted data, stale publications, schema drift, expensive + queries, untraceable manual transformations. +- MVP test path: publish one report/export as a governed file plus RSS/Atom + entry with checksum, timestamp, and permission check. +- Owner/priority: `govoplan-reporting`, `govoplan-connectors`, possible future + `govoplan-datasources`/`govoplan-dataflow`, Wave 2. + +### Public-Sector Protocols And Registries + +- Strategy: integrate/publish/receive; protocol modules own protocol semantics. +- Protocol/API surface: FIT-Connect, XTA/OSCI, XRepository, XOE/V, XRechnung, + XBestellung, Peppol, registry-specific Fachverfahren APIs. +- Auth model: certificates, mTLS, service accounts, destination credentials, + protocol-specific trust anchors and key rotation. +- Data shape: transport envelope, payload schema/version, destination IDs, + receipt/acknowledgement, message status, standard-specific metadata. +- Deployment assumptions: regulated trust chains, test/prod endpoint + separation, formal onboarding, strict logging and retention expectations. +- Risks: invalid schemas, failed delivery receipts, certificate expiry, wrong + destination routing, protocol version drift. +- MVP test path: validate a sample payload against a schema, test endpoint + reachability in sandbox, store receipt/evidence reference. +- Owner/priority: `govoplan-fit-connect`, `govoplan-xoev`, + `govoplan-xrechnung`, `govoplan-xta-osci`, `govoplan-connectors`, Wave 1/2. + +### File Providers And Shared Storage + +- Strategy: integrate/import/link; files module owns GovOPlaN file semantics. +- Protocol/API surface: SMB/CIFS, WebDAV, Nextcloud, Seafile, SFTP, S3, + local/object storage profiles. +- Auth model: service accounts, user credentials, app tokens, OAuth where + supported, secret references, environment variables for deployment-managed + credentials. +- Data shape: file ID/path, provider object ID, checksum, MIME type, size, + version/ETag, owner, permission snapshot, imported file reference. +- Deployment assumptions: internal networks, large files, existing shares, + variable permissions, provider-specific rate limits. +- Risks: permission mismatch, stale imports, overwrites, duplicate files, path + traversal, storage growth. +- MVP test path: profile test, list folder, import one file into governed + storage, keep provider reference and checksum. +- Owner/priority: `govoplan-files`, `govoplan-connectors`, Wave 0/1. + +### Project, Task, And Case-Adjacent Systems + +- Strategy: connector-first for OpenProject/Jira/Redmine; native module only + when GovOPlaN owns project semantics. +- Protocol/API surface: OpenProject API v3, webhooks, Jira/Redmine REST APIs, + Microsoft Graph for Planner/Project where applicable. +- Auth model: API tokens, OAuth/OIDC apps, webhook secrets, service accounts. +- Data shape: project ID, work package/task ID, status, assignee reference, + external URL, version/lock token, publish/sync trace. +- Deployment assumptions: external project tool remains source of truth for + broad project management; GovOPlaN links selected records. +- Risks: task duplication, bidirectional sync conflicts, permission mismatch, + over-mirroring comments/attachments. +- MVP test path: OpenProject profile test, project/work-package lookup, + external-reference round-trip. +- Owner/priority: `govoplan-connectors`, later `govoplan-tasks`/workflow/cases + consumers, Wave 0/2. + +### Specialist Fachverfahren + +- Strategy: link first; integrate/import only when a deployment project supplies + concrete contracts and a domain owner. +- Protocol/API surface: vendor APIs, CSV/XML batch imports, SFTP, database + views, message queues, protocol-specific transports. +- Auth model: usually service accounts, VPN, mTLS, SFTP keys, or vendor tokens. +- Data shape: domain-specific record IDs, status, applicant/person references, + file/evidence references, case/status events. +- Deployment assumptions: strongly local/vendor-specific, often no stable test + API, data model differs by jurisdiction. +- Risks: brittle vendor contracts, legal source-of-truth ambiguity, high + customization cost, migration expectations. +- MVP test path: inventory entry and manual external-reference link; require a + project-specific connector issue before automation. +- Owner/priority: domain module or deployment-specific connector, case by case. + +## Prioritization Rules + +1. Start with connectors that unblock Wave 0 or Wave 1 reference journeys. +2. Prefer open standards and self-hosted/open-source APIs where they are common + in public-sector deployments. +3. Treat inventory-only entries as useful because operators need a map of their + software landscape even before automation exists. +4. Keep connector code in the owning connector/protocol module. Domain modules + consume capabilities, DTOs, external references, and events through core. +5. Every executable connector needs health diagnostics, secret-reference + handling, lifecycle state, audit events, and retirement behavior. + +## Connector Catalogue Handoff + +`govoplan-connectors` owns the detailed catalogue entry shape: + +- connector type key +- category and owner module +- supported directions and trigger modes +- credential and secret handling +- health check and diagnostics payload +- external-reference shape +- required capabilities and optional module combinations +- lifecycle support + +Core should only keep strategy, routing, and cross-module architecture notes. +Connector implementation and public-sector target inventory belong in +`govoplan-connectors`. diff --git a/Repo-docs-RELEASE-CATALOG-WORKFLOW.md b/Repo-docs-RELEASE-CATALOG-WORKFLOW.md index 4e61d8f..6505f79 100644 --- a/Repo-docs-RELEASE-CATALOG-WORKFLOW.md +++ b/Repo-docs-RELEASE-CATALOG-WORKFLOW.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/RELEASE_CATALOG_WORKFLOW.md`. > Origin: `repository`. @@ -54,8 +54,13 @@ The wrapper validates the catalog with core using the generated public keyring. ## Publish After A New Release -For normal module/core releases, first tag and push the module/core repos. Then -publish the website catalog: +For normal module/core releases, first audit and record migration baselines, +then tag and push the module/core repos. Finally publish the website catalog: + +```bash +cd /mnt/DATA/git/govoplan-core +./.venv/bin/python scripts/release-migration-audit.py --strict +``` ```bash cd /mnt/DATA/git/govoplan-core @@ -84,13 +89,16 @@ https://govoplan.add-ideas.de/catalogs/v1/keyring.json ## Integrated Release Script `scripts/push-release-tag.sh` can publish the web catalog after module and core -tags have been pushed: +tags have been pushed. It runs the migration release audit in non-strict mode +by default; add `--strict-migration-audit` for stable releases after +`docs/migration-release-baselines.json` has been updated: ```bash cd /mnt/DATA/git/govoplan-core KEY_DIR="$HOME/.config/govoplan/release-keys" scripts/push-release-tag.sh \ --bump subversion \ + --strict-migration-audit \ --publish-web-catalog \ --catalog-signing-key "release-key-1=$KEY_DIR/release-key-1.pem" \ --build-web-catalog diff --git a/Repo-docs-RELEASE-DEPENDENCIES.md b/Repo-docs-RELEASE-DEPENDENCIES.md index cc24994..d87630e 100644 --- a/Repo-docs-RELEASE-DEPENDENCIES.md +++ b/Repo-docs-RELEASE-DEPENDENCIES.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/RELEASE_DEPENDENCIES.md`. > Origin: `repository`. @@ -35,17 +35,84 @@ cd /mnt/DATA/git/govoplan-core `requirements-release.txt` pins the module repositories to the release tag. Update those refs when cutting a release: ```text -govoplan-access git@git.add-ideas.de:add-ideas/govoplan-access.git v0.1.4 -govoplan-admin git@git.add-ideas.de:add-ideas/govoplan-admin.git v0.1.4 -govoplan-tenancy git@git.add-ideas.de:add-ideas/govoplan-tenancy.git v0.1.4 -govoplan-policy git@git.add-ideas.de:add-ideas/govoplan-policy.git v0.1.4 -govoplan-audit git@git.add-ideas.de:add-ideas/govoplan-audit.git v0.1.4 -govoplan-files git@git.add-ideas.de:add-ideas/govoplan-files.git v0.1.4 -govoplan-mail git@git.add-ideas.de:add-ideas/govoplan-mail.git v0.1.4 -govoplan-campaign git@git.add-ideas.de:add-ideas/govoplan-campaign.git v0.1.4 -govoplan-calendar git@git.add-ideas.de:add-ideas/govoplan-calendar.git v0.1.4 +govoplan-access git@git.add-ideas.de:add-ideas/govoplan-access.git v0.1.6 +govoplan-admin git@git.add-ideas.de:add-ideas/govoplan-admin.git v0.1.6 +govoplan-tenancy git@git.add-ideas.de:add-ideas/govoplan-tenancy.git v0.1.6 +govoplan-policy git@git.add-ideas.de:add-ideas/govoplan-policy.git v0.1.6 +govoplan-audit git@git.add-ideas.de:add-ideas/govoplan-audit.git v0.1.6 +govoplan-files git@git.add-ideas.de:add-ideas/govoplan-files.git v0.1.6 +govoplan-mail git@git.add-ideas.de:add-ideas/govoplan-mail.git v0.1.6 +govoplan-campaign git@git.add-ideas.de:add-ideas/govoplan-campaign.git v0.1.6 +govoplan-calendar git@git.add-ideas.de:add-ideas/govoplan-calendar.git v0.1.6 ``` +### PostgreSQL Release Check + +Release candidates should pass a disposable PostgreSQL migration and startup +smoke check before tagging or publishing catalogs. Start the local testbed, then +run the permutation check from the core checkout: + +```bash +cd /mnt/DATA/git/govoplan-core/dev/postgres +cp .env.example .env +docker compose --env-file .env up -d + +cd /mnt/DATA/git/govoplan-core +set -a +. dev/postgres/.env +set +a +./.venv/bin/python scripts/postgres-integration-check.py \ + --database-url "$GOVOPLAN_POSTGRES_DATABASE_URL" \ + --reset-schema +``` + +The script checks migrations and `/health` startup for core-only, files-only, +mail-only, campaign-only, campaign+files, campaign+mail, and full-product +module sets. `--reset-schema` is destructive and must only be used against a +throwaway database. + +### Migration Baselines + +Development migrations may be small and numerous while a feature is moving. +Before a stable release, unreleased migrations may be rewritten or squashed into +a release-level baseline or release-to-release upgrade migration. After a +release tag has shipped, released migration revision IDs are immutable. + +The release policy is: + +- unreleased migrations may be folded before release; +- released migrations are never rewritten or deleted; +- each stable release records the public migration head revisions in + `docs/migration-release-baselines.json`; +- fresh installations should apply release-level baselines/upgrades, not + unreleased create-then-rename churn; +- release-to-release schema changes should be folded into one reviewed + migration per migration owner where practical. + +Audit the current graph during release preparation: + +```bash +cd /mnt/DATA/git/govoplan-core +./.venv/bin/python scripts/release-migration-audit.py +``` + +Use strict mode after the baseline file has been updated for the release: + +```bash +./.venv/bin/python scripts/release-migration-audit.py --strict +``` + +`scripts/push-release-tag.sh` runs the audit in non-strict mode by default so +release operators see the current migration heads before tagging. Pass +`--strict-migration-audit` when cutting a release whose migration baselines have +already been recorded, or `--skip-migration-audit` only for emergency/manual +release work. + +Before the first stable release, fold the current development chain into the +first public baseline and record that baseline in +`docs/migration-release-baselines.json`. The tracking issue is +`add-ideas/govoplan-core#223`. + ## Runtime Module Package Changes The admin module manager can hot-enable and hot-disable packages that are @@ -91,9 +158,9 @@ govoplan-module-installer \ --daemon \ --migrate \ --build-webui \ - --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ + --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \ - --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ + --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ --health-url http://127.0.0.1:8000/health \ --restart-command '' ``` @@ -123,9 +190,9 @@ hooks: govoplan-module-installer \ --supervise \ --migrate \ - --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ + --database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ --database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' \ - --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ + --database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' \ --health-url http://127.0.0.1:8000/health \ --restart-command '' ``` @@ -178,6 +245,9 @@ Database hook commands run with these environment variables: - `GOVOPLAN_INSTALLER_RUN_DIR`: the run snapshot directory - `GOVOPLAN_DATABASE_URL`: the configured database URL +- `GOVOPLAN_DATABASE_URL_PGTOOLS`: the same PostgreSQL URL converted from + `postgresql+driver://` to `postgresql://` for `pg_dump`, `pg_restore`, and + `psql`; set only for PostgreSQL URLs - `GOVOPLAN_DATABASE_BACKUP_PATH`: a suggested backup artifact path inside the run directory - `GOVOPLAN_DATABASE_BACKUP_METADATA`: optional JSON metadata path that backup @@ -198,6 +268,88 @@ govoplan-module-installer --cancel-request --format json govoplan-module-installer --retry-request --format json ``` +### Rollback Drill + +Before using the installer daemon in production, run the rollback drill in a +controlled shell from the core checkout: + +```bash +./.venv/bin/python scripts/module-installer-rollback-drill.py --format json +``` + +The drill uses temporary SQLite databases and simulated package commands. It +does not install or uninstall real packages. It exercises: + +- package command failure followed by supervised rollback +- migration failure with a SQLite database snapshot +- restart-command failure +- health timeout after restart +- destructive retirement executor failure with database rollback +- PostgreSQL-style backup, restore-check, and restore hooks +- daemon heartbeat, request queue claim/update, retry/cancel, and stale lock + detection/removal + +Keep the drill runtime for inspection with: + +```bash +./.venv/bin/python scripts/module-installer-rollback-drill.py \ + --keep-runtime \ + --runtime-root /srv/govoplan/drills/installer-rollback-$(date +%Y%m%d) +``` + +Each scenario writes an installer run record below +`//installer/runs//record.json`. For a successful +rollback drill, check: + +- `status` is `applied` only for the original package phase when rollback later + happens in the supervisor +- `rollback_status` is `rolled-back` when package/database snapshots were + restored +- `supervisor.status` is `rolled-back` for supervised package, migration, + restart, and health failures +- `supervisor.failure_reason` names the failing restart or health target +- `snapshot.database_backup` exists for migrated SQLite runs and destructive + retirement runs +- PostgreSQL hook drills write `database.external.backup`, + `restore-check.marker`, and `restore.marker` + +The PostgreSQL drill validates the hook interface without connecting to a live +database. In production, replace the documented example hooks with deployment +commands that target a disposable staging database first: + +```bash +--database-backup-command 'pg_dump --format=custom "$GOVOPLAN_DATABASE_URL_PGTOOLS" > "$GOVOPLAN_DATABASE_BACKUP_PATH"' +--database-restore-check-command 'pg_restore --list "$GOVOPLAN_DATABASE_BACKUP_PATH" >/dev/null' +--database-restore-command 'pg_restore --clean --if-exists --dbname "$GOVOPLAN_DATABASE_URL_PGTOOLS" "$GOVOPLAN_DATABASE_BACKUP_PATH"' +``` + +If a daemon request fails, inspect it and retry only after the underlying cause +is corrected: + +```bash +govoplan-module-installer --show-request --format json +govoplan-module-installer --show-run --format json +govoplan-module-installer --retry-request --format json +``` + +Cancel queued duplicate work before retrying: + +```bash +govoplan-module-installer --list-requests --format json +govoplan-module-installer --cancel-request --format json +``` + +For stale locks, first check the lock payload and confirm the recorded process +is no longer running on the host that owns the runtime directory: + +```bash +govoplan-module-installer --lock-status --format json +``` + +Only remove `runtime/module-installer/install.lock` after confirming there is +no active installer process and no package manager command still running. Then +rerun preflight before applying or retrying the request. + Package catalogs can be local files or remote static resources, for example served by `govoplan-web`. Set `GOVOPLAN_MODULE_PACKAGE_CATALOG` for a local file or `GOVOPLAN_MODULE_PACKAGE_CATALOG_URL` for a remote catalog matching @@ -253,6 +405,29 @@ GOVOPLAN_MODULE_PACKAGE_CATALOG_SEQUENCE_STATE=/srv/govoplan/runtime/catalog-seq GOVOPLAN_MODULE_PACKAGE_CATALOG_ENFORCE_SEQUENCE=true ``` +Back up the sequence state with other runtime metadata. If it is lost or +corrupted, reconstruct the highest accepted sequence per channel from installer +run records or release records, never by lowering the sequence to admit an +older catalog. `docs/CATALOG_TRUST_AND_LICENSING.md` includes the recovery JSON +shape and reset procedure. + +Catalog module entries may include `artifact_integrity.python` and +`artifact_integrity.webui` metadata. Each artifact entry can declare: + +- `ref`: the exact Python or WebUI install ref expected in the plan +- `sha256`: the expected SHA-256 digest of the resolved artifact +- `path` or `artifact_path`: a local artifact path that the installer can hash + before applying the plan +- `sbom_url`: the published SBOM for the artifact +- `provenance_url`: the published provenance or attestation for the artifact +- `registry_identity` or `git_ref`: the expected registry or source identity + +When a saved install plan includes this metadata, installer preflight records +verification results in the run record under `preflight.artifact_integrity`. +Set `GOVOPLAN_MODULE_INSTALLER_REQUIRE_ARTIFACT_INTEGRITY=true` in production +to block package changes whose Python/WebUI artifacts are missing integrity +metadata or cannot be locally verified. + Trusted keys can also be loaded from `GOVOPLAN_MODULE_PACKAGE_CATALOG_TRUSTED_KEYS_FILE`. A URL-backed keyring is supported for development and tightly controlled deployments through @@ -270,6 +445,23 @@ GOVOPLAN_LICENSE_TRUSTED_KEYS_FILE=/srv/govoplan/trust/license-keyring.json GOVOPLAN_LICENSE_ENFORCEMENT=true ``` +Operators can validate the imported license and required catalog entitlements: + +```bash +govoplan-module-installer \ + --validate-license /srv/govoplan/license.json \ + --license-trusted-key license-issuer-1="" \ + --require-trusted-license \ + --license-required-feature module.mail \ + --format json +``` + +Release or support operators can issue/renew signed offline licenses with +`govoplan-module-installer --issue-license`. Keep the signing private key off +the application server; the admin UI only displays public diagnostics such as +license id, subject, validity window, trusted key id, features, and missing +entitlements. + See `docs/CATALOG_TRUST_AND_LICENSING.md` for key rotation, replay protection, and licensing details. See `docs/RELEASE_CATALOG_WORKFLOW.md` for the concrete release-machine workflow that generates signing keys and publishes signed @@ -281,7 +473,8 @@ Install rows must use tagged package/git refs or registry packages, not local `file:` or workspace links. The installer daemon can run `npm install` and `npm run build` for WebUI package changes; that is the supported path. Browser remote bundles are still experimental and should be treated as a controlled -deployment option, not the normal install/uninstall mechanism. +deployment option, not the normal install/uninstall mechanism. The target +design and follow-up slices are in `docs/REMOTE_WEBUI_BUNDLES.md`. Module manifests can declare core compatibility bounds and uninstall guard providers. Preflight blocks incompatible manifest contracts/core versions, @@ -313,7 +506,7 @@ The normal release path is automated by `scripts/push-release-tag.sh`: it bumps ```bash cd /mnt/DATA/git/govoplan-core -scripts/push-release-tag.sh --version 0.1.2 +scripts/push-release-tag.sh --version 0.1.6 ``` The script also includes GovOPlaN roadmap/scaffold module repositories that do not yet have package metadata. Those repositories are committed, tagged, and pushed with the same release tag, but they are tag-only until they contain `pyproject.toml`, module manifests, or WebUI packages. Tag-only repositories are not listed in `requirements-release.txt` or `webui/package.release.json`. diff --git a/Repo-docs-REMOTE-WEBUI-BUNDLES.md b/Repo-docs-REMOTE-WEBUI-BUNDLES.md new file mode 100644 index 0000000..0353229 --- /dev/null +++ b/Repo-docs-REMOTE-WEBUI-BUNDLES.md @@ -0,0 +1,168 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/REMOTE_WEBUI_BUNDLES.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Remote WebUI Bundle Loading + +GovOPlaN WebUI modules normally ship through the core WebUI package graph: +install a tagged npm/git dependency, run `npm install`, rebuild the shell, and +restart or reload the served assets. Remote WebUI bundles are an experimental +future path for controlled deployments where a backend-enabled module is not in +the local WebUI package graph and the operator wants the shell to load its +frontend without rebuilding core. + +This design defines the target guardrails. The current rebuild/reload path +remains the default production path. + +## Current Rebuild Path + +The supported release path is: + +1. Install or remove backend and WebUI package dependencies through the trusted + installer CLI/daemon. +2. Snapshot package state before mutation. +3. Run `npm install` and optionally `npm run build`. +4. Restart or reload the served WebUI assets. +5. Use installer rollback if package apply, restart, or health checks fail. + +Strengths: + +- package manager and lockfile semantics stay conventional +- CSP can stay strict because all code is served as built assets +- rollback restores package files and lockfiles +- local development uses sibling workspace dependencies naturally + +Costs: + +- frontend changes require a rebuild/reload +- hot enabling a module with frontend code is not possible in the running shell +- failed rebuilds happen at install time, not at lazy module-load time + +## Remote Bundle Path + +Remote loading is only for modules that are enabled by the backend and absent +from `virtual:govoplan-installed-modules`. The backend manifest exposes: + +- `FrontendModule.asset_manifest` +- `asset_manifest_integrity` +- `asset_manifest_signature` +- `asset_manifest_public_key_id` +- `asset_manifest_contract_version` + +The WebUI shell fetches the asset manifest, verifies manifest integrity and/or +signature, validates the manifest contract, fetches the entry bundle, verifies +the entry integrity, imports the bundle, validates the exported +`PlatformWebModule`, and applies backend metadata before registering routes, +navigation, and UI capabilities. + +Unsigned and unhashed manifests are skipped. Entries without integrity are +skipped. A module id mismatch is skipped. + +## Asset Manifest Contract + +Contract version `1`: + +```json +{ + "contractVersion": "1", + "moduleId": "files", + "entry": "./files-webui.remote.js", + "entryIntegrity": "SHA-256-", + "moduleExport": "default" +} +``` + +The backend manifest carries the manifest-level trust metadata. The remote +asset manifest carries the concrete entry URL and entry digest. + +## Compatibility Checks + +Before a remote bundle is accepted: + +- backend module must be enabled +- local WebUI module with the same id must be absent +- manifest contract must be supported by the shell +- manifest `moduleId`, exported module `id`, and backend module id must match +- backend metadata remains authoritative for label, version, dependencies, + nav, and runtime UI capability exposure +- future contract versions must declare required shell capabilities so old + shells fail closed + +Remote bundles must not broaden backend permissions. They can only contribute +routes/nav/capabilities that the backend metadata and existing permission +checks allow. + +## CSP + +The current implementation imports verified entry bytes through a blob URL. A +production CSP for this mode must explicitly allow: + +- `connect-src` for the approved asset-manifest and bundle origins +- `script-src` for `blob:` only when remote loading is enabled +- no `unsafe-inline` requirement for remote modules + +If an installation cannot allow `blob:` scripts, the follow-up implementation +should use signed same-origin module assets with static URLs instead of blob +imports. Remote loading must be disableable by configuration so strict-CSP +deployments can keep the rebuild path only. + +## Cache Invalidation + +The shell cache key is: + +```text +:: +``` + +Release catalogs should publish immutable manifest and entry URLs or change at +least one cache-key component on every release. Emergency rollback can point the +backend manifest at a previous immutable manifest URL or lower the enabled +module version after the backend package rollback. + +Browsers may still cache remote responses. Approved asset servers should use: + +- long cache lifetimes only for content-addressed immutable assets +- short cache lifetimes or explicit revalidation for channel/latest manifest + aliases +- `Cache-Control: no-store` for emergency override manifests + +## Rollback + +Remote WebUI rollback is metadata rollback, not package-manager rollback: + +- if the backend package install rolls back, the previous backend frontend + metadata returns +- if only remote assets are bad, publish a new manifest URL or revert the + backend/frontend metadata to the last known good manifest +- failed remote loading must degrade by omitting the remote module frontend, + not by breaking the shell + +Installer run records should eventually include remote asset manifest URLs, +integrity, signature key ids, and load-test results when a plan enables a +remote frontend. + +## Local And Development Behavior + +Local development should keep using workspace/file dependencies and Vite. Remote +loading is useful for integration testing release artifacts, not for day-to-day +module UI development. + +Development deployments may use unsigned catalogs and local asset servers only +when signature/integrity enforcement is intentionally disabled. The remote +loader itself still requires an integrity hash or a verifiable signature. + +## Follow-Up Slices + +1. Add a server-side remote-bundle policy flag and surface whether remote + loading is enabled in platform metadata. +2. Add CSP documentation/config generation for strict rebuild-only mode versus + controlled remote-bundle mode. +3. Add an installer/catalog preflight that validates remote WebUI asset + manifests and records verified identity in installer run records. +4. Add Playwright coverage for a signed test remote bundle, failed digest, bad + module id, cache-key refresh, and fallback when the module is unavailable. +5. Add release tooling to emit immutable remote asset manifests with digest, + signature, key id, and rollback metadata. diff --git a/Repo-docs-SCALABILITY-AND-SIZING.md b/Repo-docs-SCALABILITY-AND-SIZING.md new file mode 100644 index 0000000..279a966 --- /dev/null +++ b/Repo-docs-SCALABILITY-AND-SIZING.md @@ -0,0 +1,204 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/SCALABILITY_AND_SIZING.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Scalability And Sizing Profiles + +GovOPlaN scales manually first. Autoscaling can be added only where the runtime +surface is already stateless, health-checked, and backed by shared durable +services. + +## Deployment Topologies + +| Profile | Intended Use | Topology | +| --- | --- | --- | +| Development | Local feature work and module tests | One API process, Vite dev server, SQLite or disposable PostgreSQL, local file storage, synchronous workers. | +| Pilot | Single office, non-critical early usage | One API process, one WebUI build, PostgreSQL, local or object storage, optional Redis, one worker process when queues are enabled. | +| Small Production | One tenant or small agency | Two API processes behind a reverse proxy, PostgreSQL with backups, durable storage, Redis/Celery workers, health monitoring. | +| Medium Deployment | Municipal deployment with multiple departments | Separate WebUI, API, worker, scheduler, PostgreSQL, object storage, Redis, backup host, metrics/log collection. | +| Shared Platform | Multiple tenants or high campaign/workflow volume | Horizontally scaled WebUI/API/workers, managed PostgreSQL, object storage, queue/cache HA, central monitoring, controlled maintenance windows. | + +## Stateless And Stateful Components + +Stateless and horizontally replicable: + +- WebUI static assets +- API workers when `MASTER_KEY_B64`, `DATABASE_URL`, storage, queue, and module + configuration are shared +- Background workers when queues and idempotency keys are used +- Scheduler replicas only when leader election or an external lock exists + +Stateful or singleton-sensitive: + +- PostgreSQL +- local file storage when not replaced by object storage +- Redis/queue state +- module installer daemon and package mutation operations +- migration execution +- scheduler without distributed locking +- outgoing campaign append/send jobs unless claim tokens are enforced + +## Readiness And Degraded Modes + +| Component | Ready When | Degraded Mode | +| --- | --- | --- | +| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood | Read-only/admin-only where routes allow it; otherwise fail closed. | +| WebUI | Static assets match backend module metadata contract | Show unavailable modules/routes with reason; do not invent routes. | +| PostgreSQL | Accepts connections and migration head is current | Block writes and package changes if migration state is unknown. | +| Storage | Configured backend is reachable and writable for write flows | Read-only file views may continue if storage is read-only but reachable. | +| Redis/Celery | Broker reachable and worker queues have heartbeats | Synchronous dev-only workflows may continue; production async send/workflow queues are degraded. | +| Installer daemon | Lock is free or owned by a live daemon; latest status is fresh | Admin UI can plan changes but not execute them. | +| Mail transport | SMTP/IMAP profiles validate for the selected scope | Campaign validation blocks send/append but allows draft editing. | + +## Queue And Worker Scaling + +Worker pools should be split by queue once load appears: + +- `send_email`: SMTP send throughput, rate limits, retries, and outcome + uncertainty. +- `append_sent`: IMAP append latency and mailbox-side throttling. +- `workflow`: process orchestration and case/task state transitions. +- `transform`: datasource extraction, transformation, and export jobs. +- `notifications`: postbox, email notification, calendar, and external + notification fan-out. +- `reporting`: long-running exports, aggregates, and audit/report generation. + +Scaling signal examples: + +- queue depth and oldest queued job age +- retry rate and permanent failure rate +- worker CPU and memory saturation +- database lock time and query latency +- SMTP/IMAP provider throttling responses +- storage upload/download latency + +Autoscaling should have upper bounds per queue. Mail and connector queues often +hit external throttles before CPU is exhausted, so adding workers blindly can +make failures worse. + +## Database Path + +PostgreSQL is the production database. SQLite remains a local-development and +tiny disposable profile only. + +Production migrations should run explicitly before startup or package +activation. Module install/uninstall workflows must use database backup and +restore-check hooks for PostgreSQL before migrations or destructive retirement. + +## First Sizing Matrix + +These numbers are starting assumptions, not guarantees. Measure and adjust once +real workload metrics exist. + +| Profile | CPU | Memory | Database | Storage | Queue/Cache | Backup/Monitoring Assumptions | +| --- | ---: | ---: | --- | --- | --- | --- | +| Development | 2 cores | 4-8 GB | SQLite or local PostgreSQL | local disk | optional | no SLA; manual reset acceptable | +| Pilot | 2-4 cores | 8 GB | PostgreSQL on same host or small managed instance | 100-500 GB durable local/object storage | optional Redis | daily DB backup; basic health checks | +| Small Production | 4-8 cores | 16 GB | dedicated PostgreSQL, 2-4 vCPU, 8-16 GB RAM | 0.5-2 TB object/durable storage | Redis plus 1-2 workers | daily full backup plus WAL/snapshot policy; uptime alerts | +| Medium Deployment | 8-16 API/worker cores total | 32-64 GB total | PostgreSQL 4-8 vCPU, 16-64 GB RAM | 2-10 TB object storage | Redis, separate worker pools | central logs/metrics, tested restore, queue alerts | +| Shared Platform | sized from measured load | 64 GB+ total | managed HA PostgreSQL, read replicas only after profiling | 10 TB+ object storage | HA queue/cache if required | SLOs, restore drills, capacity alerts, maintenance windows | + +## Workload Dimensions For A Calculator + +A later calculator should ask for: + +- tenants and active users +- concurrent sessions and peak request rate +- files per month, average/max file size, and retention window +- cases/tasks/workflows per month +- campaigns per month, recipients per campaign, and send window +- IMAP append and inbound mailbox volume +- datasource/import/export job volume and file sizes +- report/audit query frequency +- retention policy and audit growth +- required recovery point and recovery time objectives + +## Profile Selection Worksheet + +The first calculator can be rule-based. It should recommend the lowest profile +that satisfies all hard constraints, then show the inputs that pushed the +operator upward. + +| Input | Pilot Threshold | Small Production Threshold | Medium Threshold | Shared Platform Threshold | +| --- | ---: | ---: | ---: | ---: | +| Active tenants | 1 | 1-5 | 5-25 | 25+ | +| Concurrent users | up to 10 | up to 50 | up to 250 | measured/contracted | +| Managed files | under 100 GB | 100 GB-2 TB | 2-10 TB | 10 TB+ | +| Campaign recipients/month | under 5,000 | 5,000-100,000 | 100,000-1,000,000 | provider-limited or multi-tenant | +| Workflow/import jobs/day | under 100 | 100-2,000 | 2,000-25,000 | queue-specific scaling | +| Recovery point objective | daily backup | daily plus WAL/snapshots | tested restore, tighter RPO | formal SLO/SLA | +| Process split requirement | optional | API plus worker | API, workers, scheduler split | horizontal replicas | + +Hard constraints override the numeric thresholds: + +- Multiple API replicas require PostgreSQL and shared storage. +- Cross-node file access requires object storage or a shared durable file + service, not node-local disk. +- Async campaign send, append, imports, exports, or workflows require Redis and + workers outside development. +- Package install/uninstall in production requires maintenance mode, backup, + restore-check hooks, and installer daemon visibility. +- Autoscaling requires idempotent job claims, readiness probes, external + throttling limits, and queue-specific maximum replica counts. + +## Calculator Output Contract + +The future UI calculator should emit a structured recommendation: + +```json +{ + "recommended_profile": "small-production", + "minimum_components": [ + "PostgreSQL", + "Redis", + "API process", + "worker process", + "durable file storage" + ], + "reasons": [ + "Campaign recipients/month exceed pilot threshold.", + "Async mail delivery requires worker split." + ], + "warnings": [ + "Object storage is recommended before adding a second API node." + ], + "open_measurements": [ + "Peak concurrent users", + "Database backup restore duration" + ] +} +``` + +Until the calculator is implemented, operators should fill the worksheet +manually and compare it with the Ops page's current profile, readiness checks, +worker queues, and sizing assumptions. + +## Minimum Production Requirements + +A production deployment, even a small one, should have: + +- PostgreSQL, explicit migrations, backups, and a tested restore path. +- A stable `MASTER_KEY_B64` stored outside the repository. +- HTTPS, exact CORS origins, secure cookies, and a reverse proxy. +- Durable file storage with backup or object-store lifecycle policy. +- Redis plus at least one worker when any queued module behavior is enabled. +- Health/readiness checks visible in `govoplan-ops`. +- Maintenance-mode access assigned to at least one operator account. +- Module catalog trust roots and license trust roots pinned by deployment + configuration. + +## Manual Before Automatic + +The first supported production scaling path is manual: + +1. Move from SQLite/local storage to PostgreSQL and durable storage. +2. Add workers and Redis for queue-backed operations. +3. Split WebUI/API/worker processes. +4. Add health checks and deployment profile warnings. +5. Add metrics and queue-depth alerts. +6. Scale API and worker replicas with fixed limits. +7. Add autoscaling only after idempotency, readiness, and external throttles are + understood.