diff --git a/Codex-Project-Index.md b/Codex-Project-Index.md index 36c2f50..4c25355 100644 --- a/Codex-Project-Index.md +++ b/Codex-Project-Index.md @@ -6,9 +6,7 @@ This page is generated from repository and product-directory project files. - [Product govoplan-split-concept-action-plan](Product govoplan-split-concept-action-plan) - `/mnt/DATA/Nextcloud/ADD ideas UG/Products/govoplan/split-concept-action-plan.md` - [Repo-README](Repo-README) - `/mnt/DATA/git/govoplan-core/README.md` -- [Repo-docs-ACCESS-EXTRACTION-PLAN](Repo-docs-ACCESS-EXTRACTION-PLAN) - `/mnt/DATA/git/govoplan-core/docs/ACCESS_EXTRACTION_PLAN.md` - [Repo-docs-ACCESS-RBAC-MODEL](Repo-docs-ACCESS-RBAC-MODEL) - `/mnt/DATA/git/govoplan-core/docs/ACCESS_RBAC_MODEL.md` -- [Repo-docs-API-CONDITIONAL-DELTA](Repo-docs-API-CONDITIONAL-DELTA) - `/mnt/DATA/git/govoplan-core/docs/API_CONDITIONAL_DELTA.md` - [Repo-docs-CATALOG-TRUST-AND-LICENSING](Repo-docs-CATALOG-TRUST-AND-LICENSING) - `/mnt/DATA/git/govoplan-core/docs/CATALOG_TRUST_AND_LICENSING.md` - [Repo-docs-CODEX-WORKFLOW](Repo-docs-CODEX-WORKFLOW) - `/mnt/DATA/git/govoplan-core/docs/CODEX_WORKFLOW.md` - [Repo-docs-CONFIGURATION-PACKAGES](Repo-docs-CONFIGURATION-PACKAGES) - `/mnt/DATA/git/govoplan-core/docs/CONFIGURATION_PACKAGES.md` @@ -19,12 +17,9 @@ This page is generated from repository and product-directory project files. - [Repo-docs-GOVERNANCE-MODEL](Repo-docs-GOVERNANCE-MODEL) - `/mnt/DATA/git/govoplan-core/docs/GOVERNANCE_MODEL.md` - [Repo-docs-GOVERNMENT-OPERATIONS-VISION](Repo-docs-GOVERNMENT-OPERATIONS-VISION) - `/mnt/DATA/git/govoplan-core/docs/GOVERNMENT_OPERATIONS_VISION.md` - [Repo-docs-GOVOPLAN-MASTER-ROADMAP](Repo-docs-GOVOPLAN-MASTER-ROADMAP) - `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MASTER_ROADMAP.md` -- [Repo-docs-GOVOPLAN-MODULE-ROADMAP](Repo-docs-GOVOPLAN-MODULE-ROADMAP) - `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MODULE_ROADMAP.md` - [Repo-docs-MODULE-ARCHITECTURE](Repo-docs-MODULE-ARCHITECTURE) - `/mnt/DATA/git/govoplan-core/docs/MODULE_ARCHITECTURE.md` -- [Repo-docs-MODULE-BOUNDARY-DECISIONS](Repo-docs-MODULE-BOUNDARY-DECISIONS) - `/mnt/DATA/git/govoplan-core/docs/MODULE_BOUNDARY_DECISIONS.md` - [Repo-docs-POLICY-CONTRACTS](Repo-docs-POLICY-CONTRACTS) - `/mnt/DATA/git/govoplan-core/docs/POLICY_CONTRACTS.md` - [Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY](Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY) - `/mnt/DATA/git/govoplan-core/docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md` - [Repo-docs-RELEASE-CATALOG-WORKFLOW](Repo-docs-RELEASE-CATALOG-WORKFLOW) - `/mnt/DATA/git/govoplan-core/docs/RELEASE_CATALOG_WORKFLOW.md` - [Repo-docs-RELEASE-DEPENDENCIES](Repo-docs-RELEASE-DEPENDENCIES) - `/mnt/DATA/git/govoplan-core/docs/RELEASE_DEPENDENCIES.md` - [Repo-docs-REMOTE-WEBUI-BUNDLES](Repo-docs-REMOTE-WEBUI-BUNDLES) - `/mnt/DATA/git/govoplan-core/docs/REMOTE_WEBUI_BUNDLES.md` -- [Repo-docs-SCALABILITY-AND-SIZING](Repo-docs-SCALABILITY-AND-SIZING) - `/mnt/DATA/git/govoplan-core/docs/SCALABILITY_AND_SIZING.md` diff --git a/Repo-docs-ACCESS-EXTRACTION-PLAN.md b/Repo-docs-ACCESS-EXTRACTION-PLAN.md deleted file mode 100644 index 6035c2a..0000000 --- a/Repo-docs-ACCESS-EXTRACTION-PLAN.md +++ /dev/null @@ -1,466 +0,0 @@ - - -> Mirrored from `/mnt/DATA/git/govoplan-core/docs/ACCESS_EXTRACTION_PLAN.md`. -> Origin: `repository`. -> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. - ---- -# GovOPlaN Access Extraction Plan - -> Backlog state migrated to Gitea issues on 2026-07-06. Keep this document as -> durable extraction context and architecture notes. Track active tasks, -> decisions, blockers, and implementation state in `add-ideas/govoplan-core` -> issues with `module/access` and `source/backlog-import`. - -This plan describes how to extract access, authentication, RBAC, and related -administration behavior from the current compatibility core into a dedicated -`govoplan-access` platform module. - -The goal is not to make access optional in every real deployment. The goal is to -make ownership explicit: the kernel composes modules and exposes contracts; -`govoplan-access` owns identity, sessions, API keys, roles, groups, and access -administration. - -## Current Inventory - -### Existing Access Module Seed - -`govoplan-access` contains the extracted module-shaped seed package under -`src/govoplan_access/backend`. `govoplan-core` keeps compatibility import shims -under `src/govoplan_core/access`: - -- `manifest.py` declares `ModuleManifest(id="access")`. -- `db/models.py` defines access-owned model candidates. -- `auth/principals.py`, `auth/roles.py`, and `auth/tokens.py` contain auth - helper concepts. -- `permissions/definitions.py`, `permissions/evaluator.py`, and - `permissions/registry.py` contain permission catalogue/evaluation concepts. -- `tenancy/datastore.py` contains early tenancy access helpers. - -This seed package is now the starting point for the access module, but it is -not yet the full live source of truth for the running product. - -### Live Compatibility Ownership In Core - -The active implementation still uses core-owned models and services: - -- `src/govoplan_core/db/models.py` - - `Account` - - `Tenant` - - `User` - - `Group` - - `Role` - - `SystemSettings` - - `GovernanceTemplate` - - `GovernanceTemplateAssignment` - - `SystemRoleAssignment` - - `UserGroupMembership` - - `UserRoleAssignment` - - `GroupRoleAssignment` - - `ApiKey` - - `AuthSession` - - `AuditLog` -- `src/govoplan_core/auth/dependencies.py` -- `src/govoplan_core/security/api_keys.py` -- `src/govoplan_core/security/permissions.py` -- `src/govoplan_core/security/sessions.py` -- `src/govoplan_core/security/passwords.py` -- `src/govoplan_core/security/secrets.py` -- `src/govoplan_core/security/module_permissions.py` -- `src/govoplan_core/admin/service.py` -- `src/govoplan_core/admin/governance.py` -- `src/govoplan_core/api/v1/auth.py` -- `src/govoplan_core/api/v1/admin.py` -- `src/govoplan_core/api/v1/admin_schemas.py` -- `src/govoplan_core/api/v1/audit.py` -- `src/govoplan_core/db/bootstrap.py` - -Core still hosts the generic login/settings shell and shared WebUI primitives. -The legacy administration page has moved to the `govoplan-access` WebUI package -and is contributed as the `/admin` route by the access module. Individual admin -panels can now be split further into access, admin, tenancy, policy, and audit -WebUI contributions without changing the core shell route wiring again. - -### Current Module Consumers - -Feature modules no longer import core auth dependency wrappers or access-owned -ORM models. Backend routers import the access-published FastAPI dependency API -from `govoplan_access.auth`; runtime cooperation uses kernel capabilities such -as `access.directory`, `campaigns.access`, `campaigns.mailPolicyContext`, and -`campaigns.deliveryTasks`. - -## Target Ownership - -### Kernel - -The kernel keeps only platform composition and stable contracts: - -- module discovery and registry validation -- route aggregation -- migration orchestration -- database engine/session lifecycle -- permission catalogue aggregation -- capability registry -- event/command envelopes -- dependency injection hooks -- health and diagnostics -- compatibility facades while modules migrate - -The kernel should not own account, tenant, role, group, session, or API-key -semantics. - -### `govoplan-access` - -`govoplan-access` owns: - -- accounts -- authentication routes -- session lifecycle -- API keys -- users -- groups and memberships -- roles and assignments -- principal resolution -- permission evaluation -- access administration routes -- access administration WebUI contributions -- access-related migrations -- access permissions and role templates - -### Later Platform Modules - -Some current access-adjacent behavior can remain in access initially, but should -have clean seams for later extraction: - -- `govoplan-tenancy`: tenants, tenant lifecycle, tenant switching, tenant - metadata, tenant settings boundaries. -- `govoplan-policy`: hierarchical policy/effective policy resolution and - provenance display contracts. -- `govoplan-audit`: audit log storage, audit routes, retention hooks, evidence - exports. -- `govoplan-admin`: generic administration shell contributions if they are not - owned by access/tenancy/policy/audit directly. - -## Kernel Contracts To Stabilize First - -Before moving live code, define contracts that feature modules can depend on -without importing access ORM models: - -- `PrincipalResolver` - - resolves the current actor from a request/session/API key. - - returns a stable DTO, not an ORM model. -- `AccessDirectory` - - resolves users, groups, memberships, and display labels by stable IDs. - - supports batch lookups for grids and policy screens. -- `TenantResolver` - - resolves current tenant context and tenant metadata by stable ID. -- `PermissionEvaluator` - - evaluates required scopes for a principal and resource. -- `ResourceAccessProvider` - - lets modules register resource ACL behavior without importing each other. -- `SecretProvider` - - encrypts/decrypts named secrets without tying consumers to access models. -- `AuditSink` - - records audit events without importing audit storage models. - -DTOs should be small and serializable: - -- `PrincipalRef` -- `AccountRef` -- `TenantRef` -- `UserRef` -- `GroupRef` -- `RoleRef` - -## Extraction Stages - -### Stage 0: Baseline Safeguards - -Status: started. - -- Document the kernel/platform module model. -- Add dependency-boundary checks. -- Add backend module permutation startup tests. -- Inventory access/auth/RBAC imports. -- Draft this extraction plan. - -Acceptance criteria: - -- `scripts/check_dependency_boundaries.py` passes. -- backend module permutation tests start every supported module combination. -- current direct imports are tracked as explicit transitional debt. - -### Stage 1: Contract Definitions - -Add kernel-owned protocols and DTOs for access-related interaction. - -Tasks: - -- Add protocol definitions under a kernel contract package. -- Add registry/capability names for access services. -- Keep existing core dependency functions as compatibility wrappers. -- Make wrappers resolve through capabilities when `govoplan-access` is present. -- Add tests for missing-capability behavior and clear error messages. - -Acceptance criteria: - -- Feature modules can receive principal/tenant/group/user references without - importing access ORM models. -- Existing routes continue to work through compatibility wrappers. - -### Stage 2: Create `govoplan-access` - -Create the new repository/package and move the access seed package into it. - -Tasks: - -- Create package metadata and entry points for `govoplan-access`. -- Move `govoplan_core/access` implementation into the new package. -- Publish `ModuleManifest(id="access")`. -- Register access permissions, role templates, routers, and migrations. -- Add compatibility imports in core where needed. -- Add release/dev dependency entries in core. - -Acceptance criteria: - -- `govoplan-core + govoplan-access` starts through normal module discovery. -- `govoplan-core` compatibility imports still work during transition. -- access manifest, migrations, and route contributions are discovered by core. - -### Stage 3: Move Live Models And Migrations - -Move active identity/access models out of `govoplan_core.db.models`. - -Current state: the stable table naming strategy is to keep legacy table names -and move SQLAlchemy class definitions under their platform owners. The old -`govoplan_core.db.models` compatibility re-export has been removed; callers -must import module-owned models or use kernel capabilities. The earlier -`access_*` candidate tables are not active metadata. - -Current table ownership: - -- `govoplan-tenancy`: `tenants` -- `govoplan-access`: `accounts`, tenant memberships in `users`, `groups`, - `roles`, role/group assignment tables, `api_keys`, and `auth_sessions` -- `govoplan-admin`: governance templates and assignments -- `govoplan-audit`: audit log -- `govoplan-core`: system settings - -Tasks: - -- [x] Decide the stable table naming strategy. -- [x] Map legacy model names to access-owned model classes. -- [x] Keep database table names where possible to avoid data migration churn. -- [x] Add Alembic migration metadata owned by the platform module owners. -- [x] Remove core model compatibility aliases after callers moved to - module-owned imports. -- [x] Update bootstrap/create-all compatibility paths. - -Acceptance criteria: - -- Existing development databases migrate without data loss. -- New databases initialize with module-owned metadata. -- Core no longer owns or re-exports live account/user/group/role/session/API-key - model definitions. - -### Stage 4: Move Auth Routes And Dependencies - -Move authentication, sessions, API keys, and principal dependencies. - -Tasks: - -- Move `/api/v1/auth/*` implementation to access. -- Move session/API-key services to access. -- Keep core route compatibility only if required by clients. -- Replace feature-module imports of core auth dependencies with the - access-published FastAPI dependency API. -- Add tests for login, session refresh, tenant selection, API-key auth, and - missing access capability failures. - -Acceptance criteria: - -- Auth behavior works through the access module. -- Feature modules do not import access internals. FastAPI routers use the - published `govoplan_access.auth` dependency API. -- Core can explain startup failure clearly if auth-required routes are enabled - without the access capability. - -### Stage 5: Move Admin And WebUI Contributions - -Move access administration UI/API ownership into modules. - -Tasks: - -- [x] Move users, groups, roles, API keys, and access settings pages into - `govoplan-access`. -- [ ] Move tenant-specific pages to `govoplan-tenancy` when that module exists, or - keep them temporarily in access with clear boundaries. -- [x] Move overview, system settings, and governance-template panels into - `govoplan-admin`. -- [x] Register admin navigation and admin sections through module - contributions. -- [x] Keep core shell, layout, route rendering, and generic components only. - -Acceptance criteria: - -- Core WebUI shell renders admin/access pages from module contributions. -- Core does not import access page components directly. -- Module nav and route metadata remain serializable. -- The access admin shell consumes module-owned `admin.sections` capability - contributions without importing sibling module panels. - -### Stage 6: Decouple Feature Modules - -Remove direct ORM/model imports from files, mail, and campaign. - -Tasks: - -- [x] Replace `Group`, `Tenant`, `User`, and `UserGroupMembership` imports with - directory/capability lookups. -- [x] Replace direct cross-module cleanup/count queries with registered providers, - events, or module-owned API contracts. -- [x] Replace mail-profile ownership resolution with stable owner references. -- [x] Replace campaign/file access checks with resource ACL provider contracts. -- [x] Move feature routers from the old backend auth dependency path to the - public `govoplan_access.auth` dependency API. - -Acceptance criteria: - -- `govoplan-files`, `govoplan-mail`, and `govoplan-campaign` do not import - access implementation modules or sibling feature modules. -- Dependency-boundary allowlist stays empty after each replacement. - -### Stage 7: Remove Compatibility Debt - -Finalize the split. - -Tasks: - -- [x] Remove obsolete core compatibility aliases. -- [x] Keep the dependency-boundary checker allowlist empty. -- [x] Update release dependency docs. -- [x] Update operator migration notes. -- [x] Add full module permutation tests including access-present and access-absent - behavior. - -Acceptance criteria: - -- Core is a composition kernel. -- Access behavior is owned by `govoplan-access`. -- Feature modules communicate through kernel contracts, capabilities, events, - and their own APIs. - -## Immediate Backlog - -- [x] Document kernel/platform module model. -- [x] Add dependency-boundary check. -- [x] Add backend permutation startup smoke checks. -- [x] Inventory access/auth/RBAC import debt. -- [x] Draft access extraction plan. -- [x] Define kernel access DTOs and protocols. -- [x] Add access capability registry names. -- [x] Register live access directory and tenant resolver capability - implementations backed by the current compatibility tables. -- [x] Register an access tenant-provisioner capability so tenancy can seed - access-owned default roles without importing access service internals. -- [x] Replace `govoplan-files` direct user/group/tenant model imports with the - `access.directory` capability for group membership and share-target checks. -- [x] Replace `govoplan-files` direct campaign model imports with the - `campaigns.access` capability for campaign share/existence checks. -- [x] Replace `govoplan-mail` direct user/group/tenant model imports with - mail-owned policy storage plus access-directory validation and legacy-read - fallback. -- [x] Replace `govoplan-mail` direct campaign model imports with the - `campaigns.mailPolicyContext` capability for campaign-scoped mail policy and - owner context. -- [x] Replace `govoplan-campaign` runtime user/group/tenant model imports with - access-directory lookups and string-based ORM relationships. -- [x] Create `govoplan-access` repository workspace and issue workflow scaffold. -- [x] Create `govoplan-access` repository/package skeleton. -- [x] Move `govoplan_core/access` seed package into `govoplan-access`. -- [x] Remove compatibility wrappers in core for old import paths. -- [x] Route existing auth dependency wrappers through access principal/evaluator capabilities. -- [x] Move FastAPI auth dependency wrappers out of core and into - `govoplan-access`. -- [x] Move session, API-key, and password helper services into `govoplan-access`. -- [x] Move interactive auth/session routes behind access module manifest. -- [x] Move legacy admin/API-key routes behind access module manifest. -- [x] Move access-owned legacy admin service helpers into `govoplan-access`. -- [x] Move governance-template CRUD helpers and routes into `govoplan-admin`. -- [x] Move governance-template materialization of access-owned groups and roles - behind the `access.governanceMaterializer` capability. -- [x] Replace legacy access-to-files/campaign admin lookups with module - tenant-summary and group-delete veto providers. -- [x] Split legacy admin route contribution across `govoplan-admin`, - `govoplan-tenancy`, `govoplan-policy`, `govoplan-audit`, and - `govoplan-access` route slices. -- [x] Move legacy admin route-handler ownership out of the access compatibility - router into access, admin, tenancy, policy, and audit module routers. -- [x] Move shared system-settings, tenant-governance, slug/error, and - tenant-count helpers out of access internals into core settings/tenancy - helpers used by the split platform route modules. -- [x] Move campaign schema routes into the campaign route contribution. -- [x] Move development mailbox routes into the mail route contribution. -- [x] Replace core Celery direct campaign/mail imports with the - `campaigns.deliveryTasks` capability. -- [x] Replace core retention direct campaign queries with - `campaigns.policyContext` and `campaigns.retention` capabilities. -- [x] Replace core `create_all` feature-model imports with module registry - metadata discovery. -- [x] Remove transitional boundary checker allowlist entries. -- [x] Move live legacy model definitions out of core and into - `govoplan-access` while preserving existing table names. -- [x] Split the transitional access-owned legacy model graph further into - tenancy, audit, admin, access, and core settings ownership. -- [x] Reverse the tenancy/access dependency direction so `govoplan-access` - depends on `govoplan-tenancy`, and the registry inserts tenancy before access. -- [x] Replace tenancy-to-access default role seeding with an access - tenant-provisioner capability. -- [x] Replace tenancy-to-access owner candidate and owner membership handling - with the access tenant-provisioner capability. -- [x] Replace admin overview counts and audit actor lookup/filtering with the - `access.administration` capability. -- [x] Replace feature-module direct user/group/tenant model imports. - - [x] `govoplan-files` - - [x] `govoplan-mail` - - [x] `govoplan-campaign` -- [x] Replace files/mail/campaign direct cross-module SQL lookups with - provider/capability contracts. -- [x] Move access admin WebUI pages to module route contributions. -- [x] Move generic admin-owned WebUI panels into `govoplan-admin` and register - them through the `admin.sections` UI capability. -- [x] Remove transitional boundary checker allowlist entries as each contract - lands. -- [x] Remove old core import shims for access models, access routes, admin - service helpers, and access-owned security services. -- [ ] Move campaign-scoped mail policy ownership fully behind an API/event - workflow if direct synchronous capability calls become too tight for later - deployment boundaries. - -## Open Decisions - -- Is `govoplan-access` required for any authenticated deployment, while - core-only remains a diagnostics/settings shell? -- Tenant models live in `govoplan-tenancy`; `govoplan-access` depends on - tenancy for authenticated platform composition. -- Historical table names remain stable for painless migrations. -- Should secret encryption remain a kernel primitive or become an access-owned - capability? -- Audit log storage lives in `govoplan-audit`; policy provenance can build on - that module boundary. -- Core no longer keeps compatibility import paths for access/auth/admin service - modules. HTTP route paths remain stable because the access manifest - contributes the same `/api/v1/auth/*` and `/api/v1/admin/*` routes. - -## Verification - -Use the following checks while extracting access: - -```bash -cd /mnt/DATA/git/govoplan-core -./.venv/bin/python scripts/check_dependency_boundaries.py -./.venv/bin/python -m unittest tests.test_module_system -``` - -For each removed dependency-boundary exception, add or update a focused test -that proves the replacement contract starts without the old direct import. diff --git a/Repo-docs-API-CONDITIONAL-DELTA.md b/Repo-docs-API-CONDITIONAL-DELTA.md deleted file mode 100644 index dd0a8ea..0000000 --- a/Repo-docs-API-CONDITIONAL-DELTA.md +++ /dev/null @@ -1,88 +0,0 @@ - - -> Mirrored from `/mnt/DATA/git/govoplan-core/docs/API_CONDITIONAL_DELTA.md`. -> Origin: `repository`. -> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. - ---- -# Conditional GET And Delta Collections - -GovOPlaN uses two complementary mechanisms to reduce reload cost. - -## Conditional GET - -Core applies conditional GET handling centrally for successful JSON `GET` -responses: - -- Responses receive a weak `ETag` based on the serialized JSON body. -- Responses are marked `Cache-Control: private, no-cache`. -- Responses vary by `Authorization`, `Cookie`, `X-API-Key`, and - `Accept-Language`. -- Matching `If-None-Match` requests return `304 Not Modified` without a body. -- Responses with `Set-Cookie`, `Content-Disposition`, `Content-Encoding`, a - non-JSON content type, a non-200 status, or `Cache-Control: no-store` are not - converted. - -The WebUI `apiFetch` client keeps an in-memory conditional cache for reusable -safe requests. It sends `If-None-Match` after an endpoint has returned an ETag, -returns the cached payload on `304`, and clears the cache generation after -unsafe methods. - -This avoids retransmitting unchanged snapshots. It does not identify which row -changed inside a collection. - -## Delta Collections - -Collection endpoints that can expose row-level changes should use the shared -delta contract instead of inventing module-specific formats. - -Core provides `core_change_sequence` as the shared monotonic change sequence. -Modules record append-only entries in the same database transaction as the -resource write. Watermarks are encoded as `seq:` and should be treated -as opaque by clients. - -Backend shape: - -```json -{ - "items": [], - "deleted": [], - "watermark": "opaque-next-watermark", - "has_more": false, - "full": false -} -``` - -Fields: - -- `items`: changed or current items since the requested watermark. -- `deleted`: deleted item markers with at least `id`, and optionally - `resource_type`, `revision`, and `deleted_at`. -- `watermark`: opaque value the client sends as `since` on the next request. -- `has_more`: true when the client should request the next page with the - returned watermark. -- `full`: true when the response is a full snapshot rather than an incremental - delta. - -Recommended query parameters: - -- `since`: opaque previous watermark. If omitted or expired, return a full - snapshot with `full: true`. -- `limit`: maximum number of changed items plus deleted markers. -- `include_deleted`: whether deleted markers should be returned. - -Modules should record changes with: - -- `module_id`: the owning module, for example `files`. -- `collection`: the delta collection, for example `files.assets`. -- `resource_type`: stable row kind, for example `file` or `folder`. -- `resource_id`: stable resource identifier. -- `operation`: `created`, `updated`, or `deleted`. -- `tenant_id`: tenant scope when the change is tenant-owned. -- `payload`: small, non-secret routing metadata that helps determine whether a - tombstone belongs to the requested view. - -The first concrete consumer is `GET /api/v1/files/delta`. Without `since`, it -returns the current files/folders snapshot for the requested owner/campaign -scope. With `since=seq:`, it returns changed files, changed folders, -and tombstones for resources that left the current view. diff --git a/Repo-docs-DOCUMENTATION-MAP.md b/Repo-docs-DOCUMENTATION-MAP.md index aa4e496..69f5b24 100644 --- a/Repo-docs-DOCUMENTATION-MAP.md +++ b/Repo-docs-DOCUMENTATION-MAP.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/DOCUMENTATION_MAP.md`. > Origin: `repository`. @@ -15,8 +15,7 @@ operator, and roadmap pages. | Topic | Canonical document | Notes | | --- | --- | --- | -| Module architecture and kernel contracts | `MODULE_ARCHITECTURE.md` | Stable module contracts, boundaries, lifecycle, and WebUI contribution rules. | -| Access extraction history | `ACCESS_EXTRACTION_PLAN.md` | Historical extraction plan and remaining ownership context only. | +| Module architecture and kernel contracts | `MODULE_ARCHITECTURE.md` | Stable module contracts, API efficiency contracts, durable boundary decisions, lifecycle, and WebUI contribution rules. | | RBAC and resource access | `ACCESS_RBAC_MODEL.md` | Current permission, role, API-key, and resource-access model. | | Governance hierarchy | `GOVERNANCE_MODEL.md` | System, tenant, user/group, campaign policy inheritance and admin UI structure. | | Policy decision DTOs and provenance | `POLICY_CONTRACTS.md` | Shared explain/provenance shape; module-specific policy docs should link here. | @@ -31,15 +30,12 @@ operator, and roadmap pages. | Catalog publishing workflow | `RELEASE_CATALOG_WORKFLOW.md` | Release-machine steps for signed catalog/keyring publication through `govoplan-web`. | | Catalog trust and licensing rules | `CATALOG_TRUST_AND_LICENSING.md` | Catalog shape, signatures, keyrings, replay protection, and offline licenses. | | Remote WebUI bundle design | `REMOTE_WEBUI_BUNDLES.md` | Experimental controlled-deployment design; normal releases still use package builds. | -| Scalability and sizing | `SCALABILITY_AND_SIZING.md` | Core sizing profiles until `govoplan-ops` owns executable operations tooling. | ## Product And Module Planning | Topic | Canonical document | Notes | | --- | --- | --- | -| Product roadmap waves | `GOVOPLAN_MASTER_ROADMAP.md` | Product-level sequencing and implementation gates. | -| Module and issue routing | `GOVOPLAN_MODULE_ROADMAP.md` | Quick map from ideas to owning repositories and issues. | -| Durable boundary decisions | `MODULE_BOUNDARY_DECISIONS.md` | Native-vs-connector decisions and missing-module criteria. | +| Product roadmap and module routing | `GOVOPLAN_MASTER_ROADMAP.md` | Product-level sequencing, implementation gates, issue routing, and missing-module decisions. | | Public-sector integration posture | `PUBLIC_SECTOR_INTEGRATION_STRATEGY.md` | Strategy index; executable target inventory lives in `govoplan-connectors`. | | Configuration packages | `CONFIGURATION_PACKAGES.md` | Package model, provider contract, import/export flow, and tracking slices. | | Government operations vision | `GOVERNMENT_OPERATIONS_VISION.md` | Product vision and reference journeys; active work belongs in issues. | diff --git a/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md b/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md index 2ab1815..ff54877 100644 --- a/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md +++ b/Repo-docs-GOVERNMENT-OPERATIONS-VISION.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/GOVERNMENT_OPERATIONS_VISION.md`. > Origin: `repository`. @@ -187,9 +187,9 @@ Sizing should account for tenants, concurrent users, cases, files, storage volume, upload/download rates, campaign volume, workflow jobs, transformation runs, reporting load, and retention/audit growth. -See `docs/SCALABILITY_AND_SIZING.md` for the first deployment topology, -readiness/degraded-mode model, worker scaling assumptions, PostgreSQL production -path, and sizing matrix. +See `govoplan-ops/docs/SCALABILITY_PROFILES.md` for the first deployment +topology, readiness/degraded-mode model, worker scaling assumptions, PostgreSQL +production path, and sizing matrix. ## Hardware Requirements @@ -206,8 +206,9 @@ Each profile should document CPU, memory, disk, database, storage, queue/cache, backup, and monitoring requirements, plus the assumptions behind the numbers. A later calculator can turn operator inputs into recommended profiles. -The initial matrix lives in `docs/SCALABILITY_AND_SIZING.md`; the calculator is -kept as a follow-up once real deployment measurements are available. +The initial matrix lives in `govoplan-ops/docs/SCALABILITY_PROFILES.md`; the +calculator is kept as a follow-up once real deployment measurements are +available. ## Collaboration diff --git a/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md b/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md index bc81ceb..aedd434 100644 --- a/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md +++ b/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MASTER_ROADMAP.md`. > Origin: `repository`. @@ -11,9 +11,10 @@ This roadmap orders GovOPlaN work so the product moves toward the government operations vision without turning every possible public-sector need into an immediate implementation track. -Use this document for sequencing. Use `GOVERNMENT_OPERATIONS_VISION.md` for the -north star and `GOVOPLAN_MODULE_ROADMAP.md` for routing specific ideas to module -repositories and issues. +Use this document for sequencing and module routing. Use +`GOVERNMENT_OPERATIONS_VISION.md` for the north star. Issues are the active +backlog; this document is durable planning context and should be mirrored to the +Gitea wiki. ## Product Thesis @@ -398,6 +399,79 @@ Defer these until a reference journey proves the need: These may become important, but they should not distract from the first complete administrative journeys. +## Module And Integration Routing + +This table maps current module and integration ideas to existing GovOPlaN +repositories or to explicit missing-module decisions. + +| Idea | Owner | Tracking | +| --- | --- | --- | +| Government operations backbone reference model | `govoplan-core` | `add-ideas/govoplan-core#213` | +| Permit-to-payment configuration package | `govoplan-core` plus participating modules | `add-ideas/govoplan-core#214` | +| Fully UI-managed configuration with safety controls | `govoplan-admin`, `govoplan-core`, `govoplan-policy`, `govoplan-access`, `govoplan-audit` | `add-ideas/govoplan-core#218` | +| Access as a module | `govoplan-access` | `add-ideas/govoplan-access#7` | +| OpenProject API / project management connector | `govoplan-connectors` | `add-ideas/govoplan-connectors#1` | +| Native project-management module decision | connector-first through `govoplan-connectors`; no native project module yet | `add-ideas/govoplan-core#196`, `add-ideas/govoplan-connectors#1` | +| Datasources for databases, CSV, files, APIs | no repository yet; start with connectors/files/reporting and create `govoplan-datasources` only after the first package proves shared source-catalog ownership | `add-ideas/govoplan-core#197` | +| Dataflow for pipelines, BI, publication | no repository yet; start with workflow/reporting/connectors and create `govoplan-dataflow` only after repeated pipeline/lineage contracts emerge | `add-ideas/govoplan-core#198` | +| Monthly datasource and transformation workflows | first as configuration package across connectors, files, workflow, reporting, and templates | `add-ideas/govoplan-core#216` | +| Templates for letters, emails, forms, reports | `govoplan-templates`, separate from reporting | `add-ideas/govoplan-core#190`, `add-ideas/govoplan-templates#1` | +| Reporting and BI | `govoplan-reporting`, separate from templates | `add-ideas/govoplan-core#190`, `add-ideas/govoplan-reporting#1` | +| File connectors: Nextcloud, Seafile, SMB, NFS | `govoplan-files` | `add-ideas/govoplan-files#15` | +| Public-sector software integration catalogue | `govoplan-connectors` with core strategy index | `add-ideas/govoplan-core#191`, `add-ideas/govoplan-connectors#2` | +| Public-sector integration landscape catalogue | `govoplan-connectors` with core tracking | `add-ideas/govoplan-core#215` | +| Cases module concept | `govoplan-cases` | `add-ideas/govoplan-core#174` | +| Workflow module concept | `govoplan-workflow` | `add-ideas/govoplan-core#175` | +| Connectors module concept | `govoplan-connectors` | `add-ideas/govoplan-core#176` | +| Adrema-style address and distribution-list management | `govoplan-addresses` | `add-ideas/govoplan-addresses#1` | +| Consume sources and become a governed source | `govoplan-connectors` plus possible future `govoplan-dataflow` | `add-ideas/govoplan-connectors#3`, `add-ideas/govoplan-core#198` | +| Terminfindung and meeting scheduling polls | `govoplan-scheduling`; calendar primitives remain in calendar | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-scheduling#1` | +| Terminplaner and calendar primitives | `govoplan-calendar` | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-calendar#1` | +| Terminbuchung appointment booking | `govoplan-appointments` | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-appointments#1` | +| Collaborative documents | `govoplan-dms` | `add-ideas/govoplan-dms#1` | +| Forms | `govoplan-forms` for definitions and `govoplan-forms-runtime` for submissions/runtime behavior | `add-ideas/govoplan-core#194`, `add-ideas/govoplan-forms#1` | +| RSS consume and emit | `govoplan-connectors` | `add-ideas/govoplan-connectors#4` | +| LDAP, Active Directory, OpenDesk identity | `govoplan-idm` | `add-ideas/govoplan-idm#1` | +| OpenDesk stack integration map | integration profile across IDM/access, mail/calendar, files/DMS, and connectors; not a monolithic module | `add-ideas/govoplan-core#195`, `add-ideas/govoplan-connectors#5` | +| Open-Xchange mail/groupware | `govoplan-mail` | `add-ideas/govoplan-mail#5` | +| Open-Xchange calendar | `govoplan-calendar` | `add-ideas/govoplan-calendar#2` | +| Scalability profiles and autoscaling readiness | `govoplan-ops`, `govoplan-core` | `add-ideas/govoplan-core#217` | +| Hardware sizing matrix and requirements calculator | `govoplan-ops`, `govoplan-core` | `add-ideas/govoplan-core#219` | +| Collaboration suite integration strategy | `govoplan-connectors`, `govoplan-dms`, `govoplan-workflow`, `govoplan-tasks`, `govoplan-appointments`, `govoplan-calendar` | `add-ideas/govoplan-core#220` | +| Install/runtime configuration contract | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#19` | +| Installer/deployment operator flow | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#26` | +| Production-like deployment documentation | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#28` | + +Boundary rationale lives in `MODULE_ARCHITECTURE.md`. Current decisions: + +- templates and reporting are separate modules +- RSS/source consume-publish starts in connectors; datasources/dataflow are not + repositories yet +- calendar, scheduling, and appointments are three separate modules +- forms definitions and forms runtime are separate responsibilities +- OpenDesk is an integration profile across modules, not a monolithic module +- OpenProject is connector-first; no native projects module yet +- public-sector integration strategy stays in core; executable catalogue work + lives in connectors + +The following modules are intentionally not created yet: + +- `govoplan-datasources` +- `govoplan-dataflow` +- `govoplan-projects` + +Create a repository only after a concrete implementation package proves that +existing connector, files, reporting, workflow, or task ownership is too narrow. + +Core keeps the strategy index in +`PUBLIC_SECTOR_INTEGRATION_STRATEGY.md`: integration postures, default +ownership, and prioritization rules. `govoplan-connectors` owns the detailed +target inventory and connector entry shape in +`govoplan-connectors/docs/PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md`. + +Release composition and tag-only repository handling are documented in +`RELEASE_DEPENDENCIES.md`. + ## Next Practical Work The next planning step should create or update Gitea issues for Wave 0 and Wave diff --git a/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md b/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md deleted file mode 100644 index 03911af..0000000 --- a/Repo-docs-GOVOPLAN-MODULE-ROADMAP.md +++ /dev/null @@ -1,99 +0,0 @@ - - -> Mirrored from `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MODULE_ROADMAP.md`. -> Origin: `repository`. -> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. - ---- -# GovOPlaN Module And Integration Roadmap - -This page maps current module and integration ideas to existing GovOPlaN -repositories or to explicit missing-module decisions. Issues are the active -backlog. This document is durable routing context and should be mirrored to the -Gitea wiki. Boundary decisions are recorded in -`docs/MODULE_BOUNDARY_DECISIONS.md`. - -## Current Routing - -| Idea | Owner | Tracking | -| --- | --- | --- | -| Government operations backbone reference model | `govoplan-core` | `add-ideas/govoplan-core#213` | -| Permit-to-payment configuration package | `govoplan-core` plus participating modules | `add-ideas/govoplan-core#214` | -| Fully UI-managed configuration with safety controls | `govoplan-admin`, `govoplan-core`, `govoplan-policy`, `govoplan-access`, `govoplan-audit` | `add-ideas/govoplan-core#218` | -| Access as a module | `govoplan-access` | `add-ideas/govoplan-access#7` | -| OpenProject API / project management connector | `govoplan-connectors` | `add-ideas/govoplan-connectors#1` | -| Native project-management module decision | connector-first through `govoplan-connectors`; no native project module yet | `add-ideas/govoplan-core#196`, `add-ideas/govoplan-connectors#1` | -| Datasources for databases, CSV, files, APIs | no repository yet; start with connectors/files/reporting and create `govoplan-datasources` only after the first package proves shared source-catalog ownership | `add-ideas/govoplan-core#197` | -| Dataflow for pipelines, BI, publication | no repository yet; start with workflow/reporting/connectors and create `govoplan-dataflow` only after repeated pipeline/lineage contracts emerge | `add-ideas/govoplan-core#198` | -| Monthly datasource and transformation workflows | first as configuration package across connectors, files, workflow, reporting, and templates | `add-ideas/govoplan-core#216` | -| Templates for letters, emails, forms, reports | `govoplan-templates`, separate from reporting | `add-ideas/govoplan-core#190`, `add-ideas/govoplan-templates#1` | -| Reporting and BI | `govoplan-reporting`, separate from templates | `add-ideas/govoplan-core#190`, `add-ideas/govoplan-reporting#1` | -| File connectors: Nextcloud, Seafile, SMB, NFS | `govoplan-files` | `add-ideas/govoplan-files#15` | -| Public-sector software integration catalogue | `govoplan-connectors` with core strategy index | `add-ideas/govoplan-core#191`, `add-ideas/govoplan-connectors#2` | -| Public-sector integration landscape catalogue | `govoplan-connectors` with core tracking | `add-ideas/govoplan-core#215` | -| Cases module concept | `govoplan-cases` | `add-ideas/govoplan-core#174` | -| Workflow module concept | `govoplan-workflow` | `add-ideas/govoplan-core#175` | -| Connectors module concept | `govoplan-connectors` | `add-ideas/govoplan-core#176` | -| Adrema-style address and distribution-list management | `govoplan-addresses` | `add-ideas/govoplan-addresses#1` | -| Consume sources and become a governed source | `govoplan-connectors` plus proposed `govoplan-dataflow` | `add-ideas/govoplan-connectors#3`, `add-ideas/govoplan-core#198` | -| Terminfindung and meeting scheduling polls | `govoplan-scheduling`; calendar primitives remain in calendar | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-scheduling#1` | -| Terminplaner and calendar primitives | `govoplan-calendar` | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-calendar#1` | -| Terminbuchung appointment booking | `govoplan-appointments` | `add-ideas/govoplan-core#193`, `add-ideas/govoplan-appointments#1` | -| Collaborative documents | `govoplan-dms` | `add-ideas/govoplan-dms#1` | -| Forms | `govoplan-forms` for definitions and `govoplan-forms-runtime` for submissions/runtime behavior | `add-ideas/govoplan-core#194`, `add-ideas/govoplan-forms#1` | -| RSS consume and emit | `govoplan-connectors` | `add-ideas/govoplan-connectors#4` | -| LDAP, Active Directory, OpenDesk identity | `govoplan-idm` | `add-ideas/govoplan-idm#1` | -| OpenDesk stack integration map | integration profile across IDM/access, mail/calendar, files/DMS, and connectors; not a monolithic module | `add-ideas/govoplan-core#195`, `add-ideas/govoplan-connectors#5` | -| Open-Xchange mail/groupware | `govoplan-mail` | `add-ideas/govoplan-mail#5` | -| Open-Xchange calendar | `govoplan-calendar` | `add-ideas/govoplan-calendar#2` | -| Scalability profiles and autoscaling readiness | `govoplan-ops`, `govoplan-core` | `add-ideas/govoplan-core#217` | -| Hardware sizing matrix and requirements calculator | `govoplan-ops`, `govoplan-core` | `add-ideas/govoplan-core#219` | -| Collaboration suite integration strategy | `govoplan-connectors`, `govoplan-dms`, `govoplan-workflow`, `govoplan-tasks`, `govoplan-appointments`, `govoplan-calendar` | `add-ideas/govoplan-core#220` | -| Install/runtime configuration contract | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#19` | -| Installer/deployment operator flow | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#26` | -| Production-like deployment documentation | `govoplan-core`, later `govoplan-ops` | `add-ideas/govoplan-core#28` | - -## Boundary Decision Register - -`docs/MODULE_BOUNDARY_DECISIONS.md` is the durable decision register for older -roadmap and boundary issues. Keep decision rationale there; keep this page as a -quick routing map from ideas to repositories and tracking issues. - -Current decisions: - -- templates and reporting are separate modules -- RSS/source consume-publish starts in connectors; datasources/dataflow are not - repositories yet -- calendar, scheduling, and appointments are three separate modules -- forms definitions and forms runtime are separate responsibilities -- OpenDesk is an integration profile across modules, not a monolithic module -- OpenProject is connector-first; no native projects module yet -- public-sector integration strategy stays in core; executable catalogue work - lives in connectors - -## Proposed Missing Modules - -The following modules are intentionally not created yet. Their candidate -responsibilities and creation criteria live in `MODULE_BOUNDARY_DECISIONS.md`: - -- `govoplan-datasources` -- `govoplan-dataflow` -- `govoplan-projects` - -Create a repository only after a concrete implementation package proves that -existing connector, files, reporting, workflow, or task ownership is too narrow. - -## Integration Catalogue Routing - -Core keeps the strategy index in -`docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md`: integration postures, default -ownership, and prioritization rules. `govoplan-connectors` owns the detailed -target inventory and connector entry shape in -`govoplan-connectors/docs/PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md`. - -When a target needs executable behavior, create the implementation issue in the -owning module repository and keep only the cross-module routing or architecture -decision in core. - -Release composition and tag-only repository handling are documented in -`RELEASE_DEPENDENCIES.md`. diff --git a/Repo-docs-MODULE-ARCHITECTURE.md b/Repo-docs-MODULE-ARCHITECTURE.md index 220ff3e..3930cea 100644 --- a/Repo-docs-MODULE-ARCHITECTURE.md +++ b/Repo-docs-MODULE-ARCHITECTURE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/MODULE_ARCHITECTURE.md`. > Origin: `repository`. @@ -9,10 +9,11 @@ GovOPlaN is structured as a platform kernel plus installable modules. The kernel starts and composes the platform. Modules own product behavior and contribute backend routes, database metadata, permissions, WebUI routes, navigation metadata, capabilities, and events. -The current package name is still `govoplan-core`, but the architecture target is a smaller kernel. Access, tenancy, policy, audit, and admin semantics are platform-module responsibilities and should be extracted in stages. +The current package name is still `govoplan-core`, but the architecture target is a smaller kernel. Access, tenancy, policy, audit, and admin semantics are platform-module responsibilities. -The concrete access/auth/RBAC extraction path is tracked in -[`ACCESS_EXTRACTION_PLAN.md`](ACCESS_EXTRACTION_PLAN.md). +Access extraction is complete enough that current ownership is described here, +in [`ACCESS_RBAC_MODEL.md`](ACCESS_RBAC_MODEL.md), and in the +`govoplan-access` repository docs. The event and audit trace contract is tracked in [`EVENTS_AND_AUDIT.md`](EVENTS_AND_AUDIT.md). Policy decision, source provenance, and explain-response contracts are tracked @@ -158,6 +159,89 @@ campaign file-share access, and core retention can call campaign-owned cleanup logic without importing campaign ORM models. Keep these contracts small DTO/protocol surfaces and register concrete behavior from the owning module. +## API Efficiency Contracts + +GovOPlaN uses conditional GET and delta collections to reduce reload cost +without giving every module a custom synchronization format. + +### Conditional GET + +Core applies conditional GET handling centrally for successful JSON `GET` +responses: + +- Responses receive a weak `ETag` based on the serialized JSON body. +- Responses are marked `Cache-Control: private, no-cache`. +- Responses vary by `Authorization`, `Cookie`, `X-API-Key`, and + `Accept-Language`. +- Matching `If-None-Match` requests return `304 Not Modified` without a body. +- Responses with `Set-Cookie`, `Content-Disposition`, `Content-Encoding`, a + non-JSON content type, a non-200 status, or `Cache-Control: no-store` are not + converted. + +The WebUI `apiFetch` client keeps an in-memory conditional cache for reusable +safe requests. It sends `If-None-Match` after an endpoint has returned an ETag, +returns the cached payload on `304`, and clears the cache generation after +unsafe methods. + +This avoids retransmitting unchanged snapshots. It does not identify which row +changed inside a collection. + +### Delta Collections + +Collection endpoints that can expose row-level changes should use the shared +delta contract instead of inventing module-specific formats. + +Core provides `core_change_sequence` as the shared monotonic change sequence. +Modules record append-only entries in the same database transaction as the +resource write. Watermarks are encoded as `seq:` and should be treated +as opaque by clients. + +Backend shape: + +```json +{ + "items": [], + "deleted": [], + "watermark": "opaque-next-watermark", + "has_more": false, + "full": false +} +``` + +Fields: + +- `items`: changed or current items since the requested watermark. +- `deleted`: deleted item markers with at least `id`, and optionally + `resource_type`, `revision`, and `deleted_at`. +- `watermark`: opaque value the client sends as `since` on the next request. +- `has_more`: true when the client should request the next page with the + returned watermark. +- `full`: true when the response is a full snapshot rather than an incremental + delta. + +Recommended query parameters: + +- `since`: opaque previous watermark. If omitted or expired, return a full + snapshot with `full: true`. +- `limit`: maximum number of changed items plus deleted markers. +- `include_deleted`: whether deleted markers should be returned. + +Modules should record changes with: + +- `module_id`: the owning module, for example `files`. +- `collection`: the delta collection, for example `files.assets`. +- `resource_type`: stable row kind, for example `file` or `folder`. +- `resource_id`: stable resource identifier. +- `operation`: `created`, `updated`, or `deleted`. +- `tenant_id`: tenant scope when the change is tenant-owned. +- `payload`: small, non-secret routing metadata that helps determine whether a + tombstone belongs to the requested view. + +The first concrete consumer is `GET /api/v1/files/delta`. Without `since`, it +returns the current files/folders snapshot for the requested owner/campaign +scope. With `since=seq:`, it returns changed files, changed folders, +and tombstones for resources that left the current view. + ## Module Responsibilities A module owns one bounded feature area. A module can include both backend and WebUI code in the same repository so feature behavior and frontend integration evolve together. @@ -380,6 +464,214 @@ Any future exception is extraction debt and must be temporary, documented in the script with a reason, and removed when a capability/API/event contract replaces it. +## Boundary Decision Register + +These durable decisions close older exploratory core issues. Implementation +work should live in the owning module repositories once a boundary is clear. + +Decision principles: + +- Prefer connector-first when an external specialist system is likely to remain + the system of record. +- Create a native module only when GovOPlaN must own domain semantics, + permissions, audit, retention, configuration-package fragments, or workflow + state. +- Keep optional behavior behind core-mediated capabilities, events, DTOs, route + contributions, and UI contribution points. +- Do not create repositories just because a possible product area exists. + +### Templates And Reporting + +Tracking: `govoplan-core#190`, `govoplan-templates#1`, +`govoplan-reporting#1`. + +Decision: templates and reporting are separate modules. + +`govoplan-templates` owns: + +- reusable renderable templates for letters, permits, emails, forms, reports, + certificates, and notices +- template versioning, merge-field declarations, rendering profiles, output + format choices, and preview contracts +- template package fragments that other modules can reference + +`govoplan-reporting` owns: + +- report definitions, data selection, dashboards, BI views, scheduled outputs, + and export targets +- report permissions, report execution history, generated report evidence, and + report-specific retention inputs +- downstream export handoff to files, dataflow, connectors, or publication + surfaces + +Boundary: + +- Templates do not own data selection, aggregation, scheduling, or BI semantics. +- Reporting may call template rendering through a capability when a formatted + report output is needed. +- Campaign, mail, files, workflow, and cases use templates/reporting through + capabilities and DTOs, never direct imports. + +### Sources, RSS, Datasources, And Dataflow + +Tracking: `govoplan-core#192`, `govoplan-core#197`, +`govoplan-core#198`, `govoplan-connectors#3`, +`govoplan-connectors#4`. + +Decision: do not create `govoplan-datasources` or `govoplan-dataflow` until a +first executable use case proves that connector/reporting/workflow ownership is +too narrow. + +First slice: + +- `govoplan-connectors` owns RSS/Atom consume/emit connector profiles, + connector health, external references, source lifecycle metadata, and + source/publish capability boundaries. +- `govoplan-files` owns file-backed governed locations and uploaded/stored file + evidence. +- `govoplan-reporting` owns report/data views and scheduled outputs. +- `govoplan-workflow` owns process state, approvals, scheduling of process + steps, and human review. + +Future `govoplan-datasources` is justified when GovOPlaN needs a broad source +catalogue for SQL databases, CSV/Excel files, APIs, RSS feeds, uploaded files, +and governed file locations with shared ownership, credentials, schema +discovery, refresh cadence, provenance, and permission boundaries. + +Future `govoplan-dataflow` is justified when GovOPlaN needs first-class +pipelines for ingestion, transformation, validation, scheduling, lineage, +publication, audit events, reruns, and source-to-source workflows. + +Monthly extraction/transformation work should start as a configuration package +and module collaboration across connectors, files, workflow, reporting, and +possibly templates. Create datasources/dataflow repositories only after that +package exposes repeated contracts that do not belong to an existing module. + +### Calendar, Scheduling, And Appointments + +Tracking: `govoplan-core#193`, `govoplan-calendar#1`, +`govoplan-calendar#2`, `govoplan-scheduling#1`, +`govoplan-appointments#1`. + +Decision: use three separate modules. + +`govoplan-calendar` owns: + +- calendar collections, events, recurrence, availability/free-busy, resources, + iCalendar import/export, CalDAV/Open-Xchange-style calendar adapters, and + calendar WebUI surfaces + +`govoplan-scheduling` owns: + +- Terminfindung, meeting-time polls, participant availability collection, + candidate-slot ranking, conflict explanations, reminders, and the handoff + from a selected slot to calendar/appointment/workflow modules + +`govoplan-appointments` owns: + +- Terminbuchung/fixed-slot appointment booking, appointment types, booking + rules, capacity, cancellation/no-show state, public/internal booking flows, + and appointment evidence + +Boundary: + +- Calendar provides time primitives and external calendar integration. +- Scheduling chooses a suitable time. +- Appointments owns booked appointment workflows and public/internal booking + semantics. +- Mail and notifications deliver invitations/reminders through capabilities. + +### Forms And Workflow Handoff + +Tracking: `govoplan-core#194`, `govoplan-forms#1`. + +Decision: forms are a reusable module boundary, with runtime behavior separated +from workflow semantics. + +`govoplan-forms` owns: + +- form definitions, schemas, validation rules, field visibility rules, + localization, versioning, admin editing, and reusable form package fragments + +`govoplan-forms-runtime` owns, when implemented: + +- public/internal submissions, drafts, submitted values, validation evidence, + attachment references, submission receipts, and handoff events + +Boundary: + +- Forms do not own cases, workflow transitions, tasks, or portal identity. +- Workflow/cases consume form submission events and evidence references. +- Files owns uploaded file storage and file permissions. +- Reporting/dataflow may consume submitted data through governed DTOs or + source lifecycle contracts. + +### OpenDesk Integration Profile + +Tracking: `govoplan-core#195`, `govoplan-connectors#5`, +`govoplan-idm#1`, `govoplan-mail#5`, `govoplan-calendar#2`, +`govoplan-connectors#1`. + +Decision: OpenDesk is an integration profile, not a monolithic module. + +Ownership: + +- identity: `govoplan-idm` plus `govoplan-access` +- mail/groupware: `govoplan-mail` +- calendar: `govoplan-calendar` +- files/documents: `govoplan-files` and later `govoplan-dms` +- projects/tasks: `govoplan-connectors` OpenProject connector first +- inventory/health/profile diagnostics: `govoplan-connectors` + +The OpenDesk profile should describe required connector profiles, shared +identity assumptions, health checks, and optional module combinations. It must +not create direct module-to-module imports. + +### Project Management And OpenProject + +Tracking: `govoplan-core#196`, `govoplan-connectors#1`. + +Decision: connector-first. Do not create a native `govoplan-projects` module +yet. + +OpenProject integration belongs in `govoplan-connectors` first: + +- profile test +- project and work-package lookup +- external-reference storage +- selected publish/synchronize capabilities for tasks, workflow, or cases + +A native project module is justified only if GovOPlaN needs to own project +semantics beyond cases, tasks, workflow, appointments, documents, and reporting, +for example portfolios, project budgets, project-level resource planning, or +governed project records that cannot remain in OpenProject. + +### Public-Sector Integration Landscape + +Tracking: `govoplan-core#186`, `govoplan-core#215`, +`govoplan-connectors#2`, `govoplan-connectors#3`. + +Decision: core owns strategy and routing; connectors owns executable +integration catalogue entries and operator inventory. + +Core documents: + +- product-level integration strategy +- native-vs-connector decisions +- owning module routing +- roadmap sequencing + +`govoplan-connectors` owns: + +- connector entry schema +- external system catalogue +- connector profiles and diagnostics +- source consume/publish lifecycle +- external references + +When a target needs executable behavior, create the implementation issue in the +owning module repository and keep only cross-module decisions in core. + ## Module Lifecycle Core exposes the installed module catalog through the admin API and WebUI. The diff --git a/Repo-docs-MODULE-BOUNDARY-DECISIONS.md b/Repo-docs-MODULE-BOUNDARY-DECISIONS.md deleted file mode 100644 index 7d6789b..0000000 --- a/Repo-docs-MODULE-BOUNDARY-DECISIONS.md +++ /dev/null @@ -1,215 +0,0 @@ - - -> Mirrored from `/mnt/DATA/git/govoplan-core/docs/MODULE_BOUNDARY_DECISIONS.md`. -> Origin: `repository`. -> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. - ---- -# Module Boundary Decisions - -This document records durable boundary decisions that close older exploratory -core issues. Implementation work should move to the owning module repositories -once a boundary is clear. - -## Decision Principles - -- Prefer connector-first when an external specialist system is likely to remain - the system of record. -- Create a native module only when GovOPlaN must own domain semantics, - permissions, audit, retention, configuration-package fragments, or workflow - state. -- Keep optional behavior behind core-mediated capabilities, events, DTOs, route - contributions, and UI contribution points. -- Do not create repositories just because a possible product area exists. - -## Templates And Reporting - -Tracking: `govoplan-core#190`, `govoplan-templates#1`, -`govoplan-reporting#1`. - -Decision: templates and reporting are separate modules. - -`govoplan-templates` owns: - -- reusable renderable templates for letters, permits, emails, forms, reports, - certificates, and notices -- template versioning, merge-field declarations, rendering profiles, output - format choices, and preview contracts -- template package fragments that other modules can reference - -`govoplan-reporting` owns: - -- report definitions, data selection, dashboards, BI views, scheduled outputs, - and export targets -- report permissions, report execution history, generated report evidence, and - report-specific retention inputs -- downstream export handoff to files, dataflow, connectors, or publication - surfaces - -Boundary: - -- Templates do not own data selection, aggregation, scheduling, or BI semantics. -- Reporting may call template rendering through a capability when a formatted - report output is needed. -- Campaign, mail, files, workflow, and cases use templates/reporting through - capabilities and DTOs, never direct imports. - -## Sources, RSS, Datasources, And Dataflow - -Tracking: `govoplan-core#192`, `govoplan-core#197`, -`govoplan-core#198`, `govoplan-connectors#3`, -`govoplan-connectors#4`. - -Decision: do not create `govoplan-datasources` or `govoplan-dataflow` until a -first executable use case proves that connector/reporting/workflow ownership is -too narrow. - -First slice: - -- `govoplan-connectors` owns RSS/Atom consume/emit connector profiles, - connector health, external references, source lifecycle metadata, and - source/publish capability boundaries. -- `govoplan-files` owns file-backed governed locations and uploaded/stored file - evidence. -- `govoplan-reporting` owns report/data views and scheduled outputs. -- `govoplan-workflow` owns process state, approvals, scheduling of process - steps, and human review. - -Future `govoplan-datasources` is justified when GovOPlaN needs a broad source -catalogue for SQL databases, CSV/Excel files, APIs, RSS feeds, uploaded files, -and governed file locations with shared ownership, credentials, schema -discovery, refresh cadence, provenance, and permission boundaries. - -Future `govoplan-dataflow` is justified when GovOPlaN needs first-class -pipelines for ingestion, transformation, validation, scheduling, lineage, -publication, audit events, reruns, and source-to-source workflows. - -Monthly extraction/transformation work should start as a configuration package -and module collaboration across connectors, files, workflow, reporting, and -possibly templates. Create datasources/dataflow repositories only after that -package exposes repeated contracts that do not belong to an existing module. - -## Calendar, Scheduling, And Appointments - -Tracking: `govoplan-core#193`, `govoplan-calendar#1`, -`govoplan-calendar#2`, `govoplan-scheduling#1`, -`govoplan-appointments#1`. - -Decision: use three separate modules. - -`govoplan-calendar` owns: - -- calendar collections, events, recurrence, availability/free-busy, resources, - iCalendar import/export, CalDAV/Open-Xchange-style calendar adapters, and - calendar WebUI surfaces - -`govoplan-scheduling` owns: - -- Terminfindung, meeting-time polls, participant availability collection, - candidate-slot ranking, conflict explanations, reminders, and the handoff - from a selected slot to calendar/appointment/workflow modules - -`govoplan-appointments` owns: - -- Terminbuchung/fixed-slot appointment booking, appointment types, booking - rules, capacity, cancellation/no-show state, public/internal booking flows, - and appointment evidence - -Boundary: - -- Calendar provides time primitives and external calendar integration. -- Scheduling chooses a suitable time. -- Appointments owns booked appointment workflows and public/internal booking - semantics. -- Mail and notifications deliver invitations/reminders through capabilities. - -## Forms And Workflow Handoff - -Tracking: `govoplan-core#194`, `govoplan-forms#1`. - -Decision: forms are a reusable module boundary, with runtime behavior separated -from workflow semantics. - -`govoplan-forms` owns: - -- form definitions, schemas, validation rules, field visibility rules, - localization, versioning, admin editing, and reusable form package fragments - -`govoplan-forms-runtime` owns, when implemented: - -- public/internal submissions, drafts, submitted values, validation evidence, - attachment references, submission receipts, and handoff events - -Boundary: - -- Forms do not own cases, workflow transitions, tasks, or portal identity. -- Workflow/cases consume form submission events and evidence references. -- Files owns uploaded file storage and file permissions. -- Reporting/dataflow may consume submitted data through governed DTOs or - source lifecycle contracts. - -## OpenDesk Integration Profile - -Tracking: `govoplan-core#195`, `govoplan-connectors#5`, -`govoplan-idm#1`, `govoplan-mail#5`, `govoplan-calendar#2`, -`govoplan-connectors#1`. - -Decision: OpenDesk is an integration profile, not a monolithic module. - -Ownership: - -- identity: `govoplan-idm` plus `govoplan-access` -- mail/groupware: `govoplan-mail` -- calendar: `govoplan-calendar` -- files/documents: `govoplan-files` and later `govoplan-dms` -- projects/tasks: `govoplan-connectors` OpenProject connector first -- inventory/health/profile diagnostics: `govoplan-connectors` - -The OpenDesk profile should describe required connector profiles, shared -identity assumptions, health checks, and optional module combinations. It must -not create direct module-to-module imports. - -## Project Management And OpenProject - -Tracking: `govoplan-core#196`, `govoplan-connectors#1`. - -Decision: connector-first. Do not create a native `govoplan-projects` module -yet. - -OpenProject integration belongs in `govoplan-connectors` first: - -- profile test -- project and work-package lookup -- external-reference storage -- selected publish/synchronize capabilities for tasks, workflow, or cases - -A native project module is justified only if GovOPlaN needs to own project -semantics beyond cases, tasks, workflow, appointments, documents, and reporting, -for example portfolios, project budgets, project-level resource planning, or -governed project records that cannot remain in OpenProject. - -## Public-Sector Integration Landscape - -Tracking: `govoplan-core#186`, `govoplan-core#215`, -`govoplan-connectors#2`, `govoplan-connectors#3`. - -Decision: core owns strategy and routing; connectors owns executable -integration catalogue entries and operator inventory. - -Core documents: - -- product-level integration strategy -- native-vs-connector decisions -- owning module routing -- roadmap sequencing - -`govoplan-connectors` owns: - -- connector entry schema -- external system catalogue -- connector profiles and diagnostics -- source consume/publish lifecycle -- external references - -When a target needs executable behavior, create the implementation issue in the -owning module repository and keep only cross-module decisions in core. diff --git a/Repo-docs-SCALABILITY-AND-SIZING.md b/Repo-docs-SCALABILITY-AND-SIZING.md deleted file mode 100644 index 279a966..0000000 --- a/Repo-docs-SCALABILITY-AND-SIZING.md +++ /dev/null @@ -1,204 +0,0 @@ - - -> Mirrored from `/mnt/DATA/git/govoplan-core/docs/SCALABILITY_AND_SIZING.md`. -> Origin: `repository`. -> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. - ---- -# Scalability And Sizing Profiles - -GovOPlaN scales manually first. Autoscaling can be added only where the runtime -surface is already stateless, health-checked, and backed by shared durable -services. - -## Deployment Topologies - -| Profile | Intended Use | Topology | -| --- | --- | --- | -| Development | Local feature work and module tests | One API process, Vite dev server, SQLite or disposable PostgreSQL, local file storage, synchronous workers. | -| Pilot | Single office, non-critical early usage | One API process, one WebUI build, PostgreSQL, local or object storage, optional Redis, one worker process when queues are enabled. | -| Small Production | One tenant or small agency | Two API processes behind a reverse proxy, PostgreSQL with backups, durable storage, Redis/Celery workers, health monitoring. | -| Medium Deployment | Municipal deployment with multiple departments | Separate WebUI, API, worker, scheduler, PostgreSQL, object storage, Redis, backup host, metrics/log collection. | -| Shared Platform | Multiple tenants or high campaign/workflow volume | Horizontally scaled WebUI/API/workers, managed PostgreSQL, object storage, queue/cache HA, central monitoring, controlled maintenance windows. | - -## Stateless And Stateful Components - -Stateless and horizontally replicable: - -- WebUI static assets -- API workers when `MASTER_KEY_B64`, `DATABASE_URL`, storage, queue, and module - configuration are shared -- Background workers when queues and idempotency keys are used -- Scheduler replicas only when leader election or an external lock exists - -Stateful or singleton-sensitive: - -- PostgreSQL -- local file storage when not replaced by object storage -- Redis/queue state -- module installer daemon and package mutation operations -- migration execution -- scheduler without distributed locking -- outgoing campaign append/send jobs unless claim tokens are enforced - -## Readiness And Degraded Modes - -| Component | Ready When | Degraded Mode | -| --- | --- | --- | -| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood | Read-only/admin-only where routes allow it; otherwise fail closed. | -| WebUI | Static assets match backend module metadata contract | Show unavailable modules/routes with reason; do not invent routes. | -| PostgreSQL | Accepts connections and migration head is current | Block writes and package changes if migration state is unknown. | -| Storage | Configured backend is reachable and writable for write flows | Read-only file views may continue if storage is read-only but reachable. | -| Redis/Celery | Broker reachable and worker queues have heartbeats | Synchronous dev-only workflows may continue; production async send/workflow queues are degraded. | -| Installer daemon | Lock is free or owned by a live daemon; latest status is fresh | Admin UI can plan changes but not execute them. | -| Mail transport | SMTP/IMAP profiles validate for the selected scope | Campaign validation blocks send/append but allows draft editing. | - -## Queue And Worker Scaling - -Worker pools should be split by queue once load appears: - -- `send_email`: SMTP send throughput, rate limits, retries, and outcome - uncertainty. -- `append_sent`: IMAP append latency and mailbox-side throttling. -- `workflow`: process orchestration and case/task state transitions. -- `transform`: datasource extraction, transformation, and export jobs. -- `notifications`: postbox, email notification, calendar, and external - notification fan-out. -- `reporting`: long-running exports, aggregates, and audit/report generation. - -Scaling signal examples: - -- queue depth and oldest queued job age -- retry rate and permanent failure rate -- worker CPU and memory saturation -- database lock time and query latency -- SMTP/IMAP provider throttling responses -- storage upload/download latency - -Autoscaling should have upper bounds per queue. Mail and connector queues often -hit external throttles before CPU is exhausted, so adding workers blindly can -make failures worse. - -## Database Path - -PostgreSQL is the production database. SQLite remains a local-development and -tiny disposable profile only. - -Production migrations should run explicitly before startup or package -activation. Module install/uninstall workflows must use database backup and -restore-check hooks for PostgreSQL before migrations or destructive retirement. - -## First Sizing Matrix - -These numbers are starting assumptions, not guarantees. Measure and adjust once -real workload metrics exist. - -| Profile | CPU | Memory | Database | Storage | Queue/Cache | Backup/Monitoring Assumptions | -| --- | ---: | ---: | --- | --- | --- | --- | -| Development | 2 cores | 4-8 GB | SQLite or local PostgreSQL | local disk | optional | no SLA; manual reset acceptable | -| Pilot | 2-4 cores | 8 GB | PostgreSQL on same host or small managed instance | 100-500 GB durable local/object storage | optional Redis | daily DB backup; basic health checks | -| Small Production | 4-8 cores | 16 GB | dedicated PostgreSQL, 2-4 vCPU, 8-16 GB RAM | 0.5-2 TB object/durable storage | Redis plus 1-2 workers | daily full backup plus WAL/snapshot policy; uptime alerts | -| Medium Deployment | 8-16 API/worker cores total | 32-64 GB total | PostgreSQL 4-8 vCPU, 16-64 GB RAM | 2-10 TB object storage | Redis, separate worker pools | central logs/metrics, tested restore, queue alerts | -| Shared Platform | sized from measured load | 64 GB+ total | managed HA PostgreSQL, read replicas only after profiling | 10 TB+ object storage | HA queue/cache if required | SLOs, restore drills, capacity alerts, maintenance windows | - -## Workload Dimensions For A Calculator - -A later calculator should ask for: - -- tenants and active users -- concurrent sessions and peak request rate -- files per month, average/max file size, and retention window -- cases/tasks/workflows per month -- campaigns per month, recipients per campaign, and send window -- IMAP append and inbound mailbox volume -- datasource/import/export job volume and file sizes -- report/audit query frequency -- retention policy and audit growth -- required recovery point and recovery time objectives - -## Profile Selection Worksheet - -The first calculator can be rule-based. It should recommend the lowest profile -that satisfies all hard constraints, then show the inputs that pushed the -operator upward. - -| Input | Pilot Threshold | Small Production Threshold | Medium Threshold | Shared Platform Threshold | -| --- | ---: | ---: | ---: | ---: | -| Active tenants | 1 | 1-5 | 5-25 | 25+ | -| Concurrent users | up to 10 | up to 50 | up to 250 | measured/contracted | -| Managed files | under 100 GB | 100 GB-2 TB | 2-10 TB | 10 TB+ | -| Campaign recipients/month | under 5,000 | 5,000-100,000 | 100,000-1,000,000 | provider-limited or multi-tenant | -| Workflow/import jobs/day | under 100 | 100-2,000 | 2,000-25,000 | queue-specific scaling | -| Recovery point objective | daily backup | daily plus WAL/snapshots | tested restore, tighter RPO | formal SLO/SLA | -| Process split requirement | optional | API plus worker | API, workers, scheduler split | horizontal replicas | - -Hard constraints override the numeric thresholds: - -- Multiple API replicas require PostgreSQL and shared storage. -- Cross-node file access requires object storage or a shared durable file - service, not node-local disk. -- Async campaign send, append, imports, exports, or workflows require Redis and - workers outside development. -- Package install/uninstall in production requires maintenance mode, backup, - restore-check hooks, and installer daemon visibility. -- Autoscaling requires idempotent job claims, readiness probes, external - throttling limits, and queue-specific maximum replica counts. - -## Calculator Output Contract - -The future UI calculator should emit a structured recommendation: - -```json -{ - "recommended_profile": "small-production", - "minimum_components": [ - "PostgreSQL", - "Redis", - "API process", - "worker process", - "durable file storage" - ], - "reasons": [ - "Campaign recipients/month exceed pilot threshold.", - "Async mail delivery requires worker split." - ], - "warnings": [ - "Object storage is recommended before adding a second API node." - ], - "open_measurements": [ - "Peak concurrent users", - "Database backup restore duration" - ] -} -``` - -Until the calculator is implemented, operators should fill the worksheet -manually and compare it with the Ops page's current profile, readiness checks, -worker queues, and sizing assumptions. - -## Minimum Production Requirements - -A production deployment, even a small one, should have: - -- PostgreSQL, explicit migrations, backups, and a tested restore path. -- A stable `MASTER_KEY_B64` stored outside the repository. -- HTTPS, exact CORS origins, secure cookies, and a reverse proxy. -- Durable file storage with backup or object-store lifecycle policy. -- Redis plus at least one worker when any queued module behavior is enabled. -- Health/readiness checks visible in `govoplan-ops`. -- Maintenance-mode access assigned to at least one operator account. -- Module catalog trust roots and license trust roots pinned by deployment - configuration. - -## Manual Before Automatic - -The first supported production scaling path is manual: - -1. Move from SQLite/local storage to PostgreSQL and durable storage. -2. Add workers and Redis for queue-backed operations. -3. Split WebUI/API/worker processes. -4. Add health checks and deployment profile warnings. -5. Add metrics and queue-depth alerts. -6. Scale API and worker replicas with fixed limits. -7. Add autoscaling only after idempotency, readiness, and external throttles are - understood.