diff --git a/Codex-Project-Index.md b/Codex-Project-Index.md index e476394..e7f3ab3 100644 --- a/Codex-Project-Index.md +++ b/Codex-Project-Index.md @@ -11,15 +11,18 @@ This page is generated from repository and product-directory project files. - [Repo-docs-FRONTEND-LAYOUT-PRINCIPLES](Repo-docs-FRONTEND-LAYOUT-PRINCIPLES) - `/mnt/DATA/git/govoplan/docs/FRONTEND_LAYOUT_PRINCIPLES.md` - [Repo-docs-GITEA-ISSUES](Repo-docs-GITEA-ISSUES) - `/mnt/DATA/git/govoplan/docs/GITEA_ISSUES.md` - [Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE](Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE) - `/mnt/DATA/git/govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md` +- [Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE](Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE) - `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md` - [Repo-docs-INTERFACE-PATTERN-LANGUAGE](Repo-docs-INTERFACE-PATTERN-LANGUAGE) - `/mnt/DATA/git/govoplan/docs/INTERFACE_PATTERN_LANGUAGE.md` - [Repo-docs-INTERFACE-SURFACE-INVENTORY](Repo-docs-INTERFACE-SURFACE-INVENTORY) - `/mnt/DATA/git/govoplan/docs/INTERFACE_SURFACE_INVENTORY.md` - [Repo-docs-META-REPO-SCAN](Repo-docs-META-REPO-SCAN) - `/mnt/DATA/git/govoplan/docs/META_REPO_SCAN.md` - [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md` - [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md` +- [Repo-docs-PLATFORM-CONTROL-PLANE](Repo-docs-PLATFORM-CONTROL-PLANE) - `/mnt/DATA/git/govoplan/docs/PLATFORM_CONTROL_PLANE.md` - [Repo-docs-REFERENCE-JOURNEY-PROGRAM](Repo-docs-REFERENCE-JOURNEY-PROGRAM) - `/mnt/DATA/git/govoplan/docs/REFERENCE_JOURNEY_PROGRAM.md` - [Repo-docs-RELEASE-CONSOLE](Repo-docs-RELEASE-CONSOLE) - `/mnt/DATA/git/govoplan/docs/RELEASE_CONSOLE.md` - [Repo-docs-REPOSITORY-INDEX](Repo-docs-REPOSITORY-INDEX) - `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md` - [Repo-docs-REPOSITORY-STRUCTURE](Repo-docs-REPOSITORY-STRUCTURE) - `/mnt/DATA/git/govoplan/docs/REPOSITORY_STRUCTURE.md` +- [Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT](Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT) - `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md` - [Repo-docs-SECURITY-AUDIT](Repo-docs-SECURITY-AUDIT) - `/mnt/DATA/git/govoplan/docs/SECURITY_AUDIT.md` - [Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY](Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY) - `/mnt/DATA/git/govoplan/docs/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md` - [Repo-docs-VIEWS-ARCHITECTURE](Repo-docs-VIEWS-ARCHITECTURE) - `/mnt/DATA/git/govoplan/docs/VIEWS_ARCHITECTURE.md` diff --git a/Repo-README.md b/Repo-README.md index 5196c7b..f5dfb7d 100644 --- a/Repo-README.md +++ b/Repo-README.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/README.md`. > Origin: `repository`. @@ -160,6 +160,8 @@ Create and validate a private, declarative installation bundle: The current executable slice and remaining production gates are documented in [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). +Same-host replica balancing and the multi-host promotion boundary are documented +in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). ## Configuration @@ -187,6 +189,10 @@ contract are documented in The cross-product destination, stakeholder visions, configuration archetypes, connected outcome stories, and capability horizons are documented in the [Connected Governance Platform Roadmap](docs/CONNECTED_GOVERNANCE_PLATFORM_ROADMAP.md). +The reconciled institutional semantics, source-authority modes, module layers, +candidate Mandates/Services/Parties/Decisions boundaries, and migration +sequence are documented in the +[Institutional Governance Target Architecture](docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md). The selected Campaign-to-Postbox-to-data-to-collaboration implementation path, including stage gates and shared documentation expectations, is in the [Reference Journey Program](docs/REFERENCE_JOURNEY_PROGRAM.md). @@ -196,6 +202,8 @@ installation, scale-out, and reversible environment promotion is defined in The corresponding host deployment compiler, managed/external component choices, reconfiguration semantics, and safe Web update boundary are defined in [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). +The concrete replica, worker-node, load-balancer, and shared-state topology is +defined in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). The first Campaign-centric capability and infrastructure fit assessment is in `docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. Its rerun tooling can collect and verify a bounded installed composition; target, provider and production claims diff --git a/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md b/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md index 1eedf82..83f45a1 100644 --- a/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md +++ b/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. > Origin: `repository`. @@ -21,6 +21,14 @@ | Machine-readable companion | [`capability-fit-current.json`](capability-fit-current.json) | | Input schema | [`capability-fit.schema.json`](capability-fit.schema.json) | +**Snapshot notice:** this assessment remains valid only for the pinned +2026-07-22 composition above. Workflow Engine, the optional Workflow editor, +Datasources, Dataflow, Search, encryption contracts, and other later main-branch +work must not be inferred into this evidence record. The current product +direction and implemented-state reconciliation are documented separately in +the +[Institutional Governance Target Architecture](INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md). + This is a fit assessment, not a production approval or security certification. It deliberately does not infer implementation from a repository, issue, or manifest existing. A conclusion needs code plus a route/contract, test, diff --git a/Repo-docs-CONNECTED-GOVERNANCE-PLATFORM-ROADMAP.md b/Repo-docs-CONNECTED-GOVERNANCE-PLATFORM-ROADMAP.md index 56c1f3b..013a0cb 100644 --- a/Repo-docs-CONNECTED-GOVERNANCE-PLATFORM-ROADMAP.md +++ b/Repo-docs-CONNECTED-GOVERNANCE-PLATFORM-ROADMAP.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/CONNECTED_GOVERNANCE_PLATFORM_ROADMAP.md`. > Origin: `repository`. @@ -23,6 +23,7 @@ cross-product vision that sequence serves. Read it together with: +- the [institutional governance target architecture](INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md) - the [selected reference-journey program](REFERENCE_JOURNEY_PROGRAM.md) - the [current capability and infrastructure fit assessment](CAPABILITY_AND_INFRASTRUCTURE_FIT.md) - the [interface pattern language](INTERFACE_PATTERN_LANGUAGE.md) @@ -158,7 +159,7 @@ responsibilities, not a new dependency hierarchy. | Participation and channels | Let internal and external actors enter, receive, discuss, schedule, and respond through suitable channels. | Portal, postbox, mail, notifications, calendar, scheduling, appointments, campaign, consultation, poll | | Work coordination | Turn an input into owned, reviewable work and make exceptions visible. | Forms/runtime, cases, tasks, approvals, workflow, booking, resources, domain modules | | Evidence and institutional memory | Preserve what was known, decided, produced, sent, received, retained, corrected, and disclosed. | Files, templates, DMS, records, audit, search, reporting, transparency | -| Institutional governance | Establish who may act, in which capacity, for which organization and tenant, under which policy. | Identity, access, IDM, tenancy, organizations, policy, identity trust, risk/compliance | +| Institutional governance | Establish who may act, in which capacity, for which organization and tenant, under which mandate, jurisdiction, and policy. | Identity, access, IDM, tenancy, organizations, policy, identity trust, risk/compliance; candidate mandate and decision contracts | | Integration and operations | Connect sources and destinations, operate them safely, and prove their health and recovery. | Connectors, REST/SOAP and public-sector protocols, mail/calendar/file adapters, ERP handoffs, ops, release and configuration packages | Across these planes, GovOPlaN should maintain a connected context graph rather @@ -1143,11 +1144,12 @@ boundary, and retain operational, security, and evidence guarantees. | Addresses and directories | Implemented adapters exist but configuration/target proof varies | Directory source, privacy, conflict and lifecycle package | Reusable people/contact source capability | | Docs/admin/ops/dashboard | Useful cross-product surfaces with incomplete rollout | Configured-system inventory, guided config, monitoring/recovery evidence | Explainability and operation of the configured product | | Organizations/identity/IDM/access/postbox | Normalized ownership concepts and uneven runtime slices | Function-bound delivery, reassignment/delegation, vacancy and access-evidence proof | Institutional responsibility and durable communication spine | -| Forms/cases/tasks/approvals/search | Concepts and uneven first slices | Extend the proven responsibility path into one manual end-to-end work/evidence journey | Reusable administrative coordination layer | -| Templates/reporting/data sources | Boundary concepts or scaffolds | One reproducible data-backed document/report and safe HIS-style launch | Governed document production, reports, dashboards, and analytical consumption | -| Analytical data products/dataflow | Selected direction; platform contracts not yet implemented | One bounded university source-to-indicator path with staging, quality, lineage and promotion proof | Transparent institutional BI and cross-process reporting | +| Forms/cases/tasks/approvals | Concepts and uneven first slices | Extend the proven responsibility path into one manual end-to-end work/evidence journey, with shared service, party, mandate, and decision semantics | Reusable administrative coordination layer | +| Search | PostgreSQL-backed permission-aware provider and global/contextual UI foundation | Complete provider rollout, indexing operations, and target authorization/performance evidence | Optional cross-module discovery with OpenSearch only as an adapter | +| Templates/reporting/data sources | Templates and governed Datasources foundations exist; Reporting remains an early semantic-model slice | One reproducible data-backed document/report and safe HIS-style launch | Governed document production, reports, dashboards, and analytical consumption | +| Analytical data products/dataflow | Typed graph/runtime foundations, triggers, staging integration, reusable definitions, and golden-flow fixtures exist | One bounded university source-to-indicator path with quality, lineage, promotion, and target-performance proof | Transparent institutional BI and cross-process reporting | | DMS/collaborative editing | Boundary concept and tag-only scaffold | One Files-backed version lifecycle, then one provider-neutral editing session | Collaborative documents, review, approval and records-ready renditions | -| Workflow/automation | Concept only; no discoverable Workflow runtime; program postponed | Stable action/effect providers and an explicitly reprioritized bounded journey | Configurable governed process coordination | +| Workflow/automation | Headless Workflow Engine and optional editor are implemented foundations with versioned module baselines, BPMN interchange, action/effect execution, and reconciliation | Prove one resumable human/system journey with target recovery and conformance evidence | Configurable governed process coordination without becoming a second domain layer | | Domain modules | Mostly boundary concepts or seeds | Only the modules required by a reference package | Reusable semantics above the shared spine | | Connectors and protocols | Catalogue/strategy plus several module-specific adapters | Profile/runtime, source-of-truth, health and one real landscape | Coexistence with institutional IT | | Records/transparency/risk-compliance/export screening | Planned or early concepts | Evidence contracts and one regulated reference story | Institutional memory, oversight, and assurance | @@ -1257,8 +1259,9 @@ user-story implementation remains paused. target acceptance for calendar/scheduling/poll and other shipped foundations without displacing the selected reference path; implement new feature programs only when they are required by a stage or explicitly reprioritized. -7. **Resume Workflow only by explicit priority decision.** When resumed, start - with stable actions from a demonstrated package; do not turn it into a +7. **Use Workflow only through bounded journeys.** The headless engine and + optional editor now exist. Extend them through stable module-owned actions, + versioned baselines, and demonstrated packages; do not turn Workflow into a second domain layer. ## Product decisions to make progressively @@ -1270,8 +1273,8 @@ them; they do not block the product vision today. reference release? - Which real identity, groupware, file/DMS, and deployment stack should define the first integration profile? -- When should the postponed Workflow program resume, and which single package - will constrain its first implementation? +- Which demonstrated package should provide the first target-accepted Workflow + execution and recovery profile? - Which objects and fields remain authoritative in GovOPlaN versus each target system, and which conflict/failure behavior is acceptable? - Which default participant privacy profiles should ship for scheduling, diff --git a/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md b/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md index 0569355..2e5534c 100644 --- a/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md +++ b/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md`. > Origin: `repository`. @@ -30,7 +30,12 @@ It currently supports: - managed, external, or evaluation-only disabled Redis; - disabled mail, an external relay declaration, or an evaluation-only GreenMail service; -- durable local file storage or external S3-compatible storage; +- durable local file storage, managed single-node Garage S3, or external + S3-compatible storage; +- an explicit HAProxy service that load-balances configured WebUI and API + replicas without access to the Docker socket; +- declarative API, WebUI, and worker replica counts while keeping migrations + and the scheduler singleton; - Core, base, or full initial module selections; - deterministic Compose JSON accepted by Compose v2; - generated secrets stored in a private `0600` file; @@ -54,7 +59,11 @@ Create a local evaluation bundle: --profile evaluation \ --postgres managed \ --redis managed \ + --storage garage \ --mail test-mail \ + --api-replicas 2 \ + --web-replicas 2 \ + --worker-replicas 2 \ --module-set base ``` @@ -81,6 +90,8 @@ The private installation directory contains: | `installation.json` | Versioned, non-secret desired state | | `secrets.env` | Deployment-local secrets and external service bindings | | `compose.json` | Deterministic generated Compose definition | +| `garage.toml` | Non-secret managed Garage server configuration | +| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration | | `plan.json` | Latest desired-state diff and readiness findings | | `receipt.json` | Last successfully applied immutable identities | | `.deployment.lock` | Same-host operation exclusion | @@ -133,7 +144,9 @@ a production distribution: 5. **Deployment agent.** Web updates need a separate privileged reconciler with a typed command allowlist. The API and browser must never receive the Docker socket or arbitrary shell access. -6. **Ingress and certificates.** A self-hosted profile needs an explicit choice +6. **Ingress and certificates.** The managed HAProxy service provides HTTP + load balancing inside the deployment boundary; it does not issue or renew + certificates. A self-hosted profile still needs an explicit choice between an existing reverse proxy and a supported managed ingress, including trusted-proxy boundaries, TLS certificate issuance, renewal, and health probing through the public route. @@ -192,11 +205,49 @@ presenting a container as a complete mail service. ### File Storage `local` uses a durable Compose volume and is appropriate for one-host -installations. `s3` requires endpoint, region, access key, secret key, and -bucket values. Self-hosted S3 endpoints must use HTTPS. +installations. `garage` provisions Garage 2.3 in its supported single-node +bootstrap mode, generates a private application key and bucket, and connects +the Files S3 backend to the exact installer-owned internal endpoint. The +deployment-only trust marker cannot authorize another S3 host; arbitrary +external SDK endpoints remain fail-closed until peer pinning is implemented. +Garage metadata and object data use separate persistent volumes. `s3` requires +an external endpoint, region, access key, secret key, and bucket values. +Self-hosted external S3 endpoints must use HTTPS. Local storage must be included in backup and restore drills. Horizontal API or -worker scale-out requires shared/object storage. +worker scale-out requires shared/object storage. The managed Garage profile is +persistent but has no data redundancy; availability-sensitive installations +must use a tested multi-node Garage cluster or another external S3 service. + +### Load Balancing And Replicas + +The generated Compose topology publishes only `load-balancer`. HAProxy uses +Docker DNS service discovery to distribute public traffic across WebUI replicas +and WebUI API proxy traffic across API replicas. The WebUI and API services do +not publish host ports. HAProxy has no Docker socket and discovers only the +bounded replica slots rendered into `load-balancer.cfg`. + +Replica counts are desired state: + +```sh +./.venv/bin/python tools/deployment/govoplan-deploy.py configure \ + --directory /tmp/govoplan-evaluation \ + --api-replicas 3 \ + --web-replicas 2 \ + --worker-replicas 4 +./.venv/bin/python tools/deployment/govoplan-deploy.py apply \ + --directory /tmp/govoplan-evaluation +``` + +Workers are queue consumers, so they are scaled through Redis rather than put +behind an HTTP load balancer. The migration runner and Celery scheduler remain +singletons. Multiple API replicas are rejected when Redis is disabled because +distributed throttling and queued work cannot then be shared correctly. + +This is same-host scaling. Docker Compose uses a bridge network and does not +place containers on another machine. See +[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) for +the supported topology and promotion path. ## Reconfiguration Semantics @@ -214,6 +265,8 @@ applied state. `plan` compares their canonical hashes and service sets. - Managed-to-external transitions require the new endpoint in the same operation. - Migrations run as a one-shot service before API/worker replacement. +- The first upgrade from a direct WebUI host port stops that legacy WebUI + container immediately before HAProxy claims the same endpoint. - Health must recover before a new receipt is committed. This is sufficient for one-host reconciliation. Production updates additionally @@ -268,7 +321,8 @@ Run the focused tests: ``` The tests cover profile restrictions, secret persistence, external endpoint -requirements, S3 policy, Compose service selection, secret non-disclosure, +requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy +discovery configuration, Compose service selection, secret non-disclosure, service-specific environment isolation, private file modes, external endpoint preflight, first-plan generation, apply ordering, and receipt-based idempotency. diff --git a/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md b/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md new file mode 100644 index 0000000..6fa7f1c --- /dev/null +++ b/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md @@ -0,0 +1,453 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Institutional Governance Target Architecture + +## Status and sources + +This document is the accepted architectural reconciliation of two product +concepts prepared outside the repositories: + +- `govoplan_concept_dev.md` +- `software_big_picture.md` + +The source concepts describe GovOPlaN as an operational governance platform for +public institutions. This document merges that direction with the implemented +platform state as of 2026-07-31. It is the canonical repository version of the +direction. Gitea issues remain the source of truth for delivery state. + +Read this together with: + +- [Connected Governance Platform Roadmap](CONNECTED_GOVERNANCE_PLATFORM_ROADMAP.md) +- [Reference Journey Program](REFERENCE_JOURNEY_PROGRAM.md) +- [Module Contracts and Install Boundaries](MODULE_CONTRACTS_AND_INSTALLS.md) +- [Datasource and Definition Graph Architecture](DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md) +- [Capability and Infrastructure Fit](CAPABILITY_AND_INFRASTRUCTURE_FIT.md) +- [Core Module Architecture](../../govoplan-core/docs/MODULE_ARCHITECTURE.md) +- [Core External References and Integration Maturity](../../govoplan-core/docs/EXTERNAL_REFERENCES_AND_INTEGRATION_MATURITY.md) +- [Core Action, Effect, and Automation Layer](../../govoplan-core/docs/ACTION_EFFECT_AUTOMATION_LAYER.md) + +## Decision + +GovOPlaN is a configurable **institutional governance and operations layer** for +public institutions. It should model the institution, coordinate its work, +connect its specialist systems, and preserve why and under whose authority an +action occurred. + +GovOPlaN is not intended to become one universal ERP, DMS, groupware suite, +workflow editor, or specialist procedure. It should own the governance concepts +that must remain understandable across those systems and support native, +external, mirrored, synchronized, overlay, and link-only operation explicitly. + +This changes product emphasis, not the modular architecture: + +1. The current kernel and optional-module model remains. +2. Existing domain owners keep their data and behavior. +3. Cross-module semantics become explicit, versioned contracts. +4. Successful compositions become product and sector packages, not forks or + monolithic replacement applications. +5. Repository creation follows a proof threshold; a noun in the information + model does not automatically require a module. + +## What recent work already supersedes + +The source concepts predate several implemented foundations. These items are +accepted as the current baseline and must not be reopened as greenfield work. + +| Concept requirement | Reconciled current state | +| --- | --- | +| Slim kernel plus installable modules | Implemented through entry-point discovery, `ModuleManifest`, migrations, capabilities, interfaces, WebUI contributions, and permutation checks. | +| Versioned cross-module contracts | Implemented through named interface ranges, capability protocols, static workspace graph checks, activation validation, and release checks. | +| Separate headless workflow runtime and editor | Implemented as `govoplan-workflow-engine` and optional `govoplan-workflow`. Module-owned workflow baselines are versioned and reconciled without replacing local overrides. | +| Provider-neutral external references | Implemented in Core with stable external identity and cumulative integration maturity from discovery through replacement. | +| Governed asynchronous effects | Implemented foundations include the action/effect contract, transactional platform event outbox, module outboxes, idempotency, outcome-unknown states, reconciliation, and worker health. Coverage still varies by provider. | +| Acting identity, function assignment, and ownership recovery | Implemented foundations span Identity, Organizations, IDM, Access, generic ownership transfer/recovery, and audit provenance. Mandate and jurisdiction context remain missing. | +| Governed data foundations | Connectors, Datasources, Dataflow, Reporting, and Search now exist. Datasources already provides live/cached/static modes, staging, immutable materializations, and publication contracts. | +| Task-focused projections and configured documentation | Views, view-surface declarations, configurable dashboards, and manifest-driven user/admin documentation exist. Rollout and content depth remain incremental. | +| Encryption as an optional capability | `govoplan-encryption` now defines key-vault, content-protection, recovery, and disable-preflight boundaries. It is not a reason to move domain ownership into Core. | +| Search without mandatory OpenSearch | PostgreSQL-backed, permission-aware search and module provider contracts exist; OpenSearch remains an optional adapter. | +| Scale-out and recovery architecture | Stateless API/worker, shared database/object storage, event delivery, deployment, and recovery contracts are documented and partly exercised. Production profiles and drills remain active work. | + +The important remaining work is primarily **institutional semantics, consistent +provider declarations, and product packaging**, not another runtime rewrite. + +## Target capability layers + +The layers describe ownership and dependency direction. They are not navigation +groups and do not imply that every installation exposes every module. + +| Layer | Responsibility | Current and candidate owners | +| --- | --- | --- | +| 0. Runtime and meta | Composition, release, migrations, shared contracts, operations, deployment | Core, meta repository, Admin, Ops | +| 1. Institutional foundation | Institution, tenant, identity, organization, function, authority, access, trust | Tenancy, Identity, Organizations, IDM, Access, Identity Trust, Encryption; candidate Mandates | +| 2. Governance and accountability | Policy, audit, risk, control, explainability, configured projection | Policy, Audit, Risk Compliance, Docs, Views, Search; candidate Decisions | +| 3. Human work and procedure | Intake, cases, tasks, approvals, process execution and editing | Forms, Forms Runtime, Cases, Tasks, Approvals, Workflow Engine, Workflow, Tickets; candidates Services and Parties | +| 4. Communication and participation | Delivery, participation, scheduling, channels, consultation | Portal, Postbox, Notifications, Mail, Campaign, Calendar, Scheduling, Poll, Appointments, Booking, Consultation, Committee, Addresses, Distribution Lists | +| 5. Content, records, and evidence | Managed content, templates, records, knowledge, disclosure | Files, Templates, DMS, Records, Wiki, Transparency, Certificates | +| 6. Data, reporting, and integration | Source access, staging, transformation, search, analytics, protocols | Connectors, Datasources, Dataflow, Reporting, Dashboard, REST, SOAP, XOE/V, XTA/OSCI, FIT-Connect, XRechnung, ERP adapters | +| 7. Domain capabilities | Reusable public-sector subject matter | Projects, Procurement, Contracts, Grants, Resources, Assets, Facilities, Learning, Payments, Ledger, Permits, Inspections, Evaluation, Helpdesk | +| 8. Product and sector packages | Versioned compositions, terminology, forms, processes, controls, reports, integration profiles | Signed configuration packages and reference packages; not runtime modules by default | + +## Canonical institutional semantics + +The connected model must keep these concepts distinct even where one UI +combines them. + +| Concept | Canonical answer | Owner or direction | +| --- | --- | --- | +| Institution and tenant | In which governed installation and tenant does work occur? | Tenancy and Organizations | +| Organization and unit | Where is responsibility situated? | Organizations | +| Function | Which named organizational responsibility can an incumbent hold? | Organizations | +| Identity and account | Who is the person or machine, and through which account do they act? | Identity and Access | +| Function assignment | Who holds or represents a function, for which interval and source? | IDM | +| Role and permission | What application behavior may the acting principal perform? | Access, constrained by Policy | +| Mandate and jurisdiction | Why is an institution, unit, or function competent to act on this subject, territory, population, or interval? | Candidate Mandates capability | +| Service | What governed promise can an institution offer, to whom, under which prerequisites, evidence, channel, deadline, and responsibility? | Candidate Services capability; Portal presents it | +| Case | Which concrete administrative matter is being handled? | Cases | +| Party | In what procedural capacity does a person or organization participate, and who may represent or receive for it? | Candidate Parties capability; Identity/Organizations remain the subject owners | +| Work item | What must a responsible actor do next? | Tasks and domain modules | +| Workflow | How is work coordinated, including waits, human hand-offs, and governed actions? | Workflow Engine; Workflow is the optional editor | +| Approval | Has a proposed action passed a configured review or separation-of-duties gate? | Approvals | +| Decision | What formal institutional outcome was reached, by which competent authority, on which facts, rules, evidence, reasoning, and review path? | Candidate Decisions capability | +| Evidence and record | What proves the input, state, action, effect, correction, and retained institutional memory? | Domain owner, Files/DMS/Records, and Audit | + +### Candidate semantic modules + +Four horizontal concepts are not cleanly owned today. They are repository +candidates, not immediate repository mandates. + +#### Mandates + +Mandates should own public or internal tasks, jurisdiction, responsibility, +decision/signature authority, legal or organizational basis, and effective +history. Organizations continues to own structures and functions; IDM owns +incumbency; Access owns permissions; Policy owns constraints. + +The first proof should answer: *Was this function competent to act for this +case at the relevant time, and on what basis?* Start with a Core-neutral DTO and +provider contract used by one Case/Decision journey. Create +`govoplan-mandates` when the model has independent persistence, lifecycle, +security, and more than one consumer. + +#### Services + +Services should own versioned service definitions: audience, prerequisites, +legal basis, evidence, fees, deadlines, channels, responsible unit/function, +jurisdiction, forms, case/workflow/result bindings, remedies, service levels, +and publication status. Portal presents and starts services but should not own +their institutional definition. + +The existing Portal service-directory concept remains the first presentation +surface. Extract `govoplan-services` when service definitions are consumed by +Portal plus at least one of Cases, Forms, Workflow, Reporting, or external +publication independently of Portal. + +#### Parties + +Parties should own procedure-local roles and relationships: applicant, +respondent, beneficiary, representative, joint applicant, delivery recipient, +power or authority to represent, and permitted/preferred channels for the +matter. Identity answers who the subject is; Organizations answers which +institutional unit it is; Addresses owns contact points; Parties answers how +the subject participates here. + +Cases currently sketches local party rows. That is a compatibility seed, not a +reason to copy representation semantics into every domain. Define a shared +reference contract first and extract `govoplan-parties` when a second procedure +needs the same effective-dated representation lifecycle. + +#### Decisions + +Decisions should own formal outcomes: subject, type, competent authority, +facts, evidence, applicable rule versions, reasoning, operative result, +conditions, effect, delivery/publication, remedy/review, correction, revocation, +and links to observed effects. Approvals own review gates; Poll owns response +collection; Committee owns deliberation, meetings, and votes; Workflow owns +coordination. + +Committee and Cases may retain temporary decision references, but the first +regulated journey should use one shared Decision DTO and evidence contract. +Create `govoplan-decisions` when the lifecycle is persisted and reused outside +one domain. + +## Source authority and integration maturity + +Two independent dimensions must be recorded. They must not be collapsed into a +single `sync` flag. + +### Source-authority mode + +| Mode | Meaning | +| --- | --- | +| `native_authoritative` | GovOPlaN owns the authoritative object and lifecycle. | +| `external_authoritative` | The external system owns the object; GovOPlaN reads or acts through it. | +| `external_mirror` | The external system is authoritative and GovOPlaN keeps a governed local projection or immutable snapshots. | +| `governed_sync` | Both sides may change supported fields under explicit conflict and reconciliation rules. | +| `governance_overlay` | GovOPlaN owns policy, responsibility, evidence, or coordination around an externally executed object. | +| `linked_reference` | GovOPlaN keeps only a stable link and minimal display/provenance metadata. | + +Authority may be declared per tenant, organization, service, object type, +object, field group, or process step. A broad default must not hide a narrower +override. + +### Integration maturity + +The implemented maturity ladder remains `discover`, `link`, `search`, `read`, +`publish`, `synchronize`, `migrate`, and `replace`. Maturity says what an +adapter can do. Source-authority mode says who owns truth in a particular +configuration. For example, a connector may support `synchronize`, while a +tenant deliberately configures it as `external_mirror`. + +### Provider declaration + +Every provider that reads or causes external effects should eventually declare: + +- owned object and field groups; +- supported source-authority modes and integration maturity; +- read, write, delete, search, preview, and dry-run operations; +- revision/concurrency tokens, freshness, health, and bounded-read limits; +- idempotency, retry, timeout, conflict, and outcome-unknown behavior; +- evidence, audit, correction, rollback/compensation, and reconciliation paths; +- degraded and outage behavior; +- classification, purpose, retention, and secret-handling requirements. + +Current external-reference, action/effect, connector lifecycle, capability, +operational-check, and documentation contracts provide most of the primitives. +The required change is a common declaration that composes them and can be +validated by Core, release tooling, Ops, Docs, and configuration-package +preflight. + +## Cross-cutting contracts + +The following contracts are mandatory for consequential domain objects. They +should be shared reference DTOs and provider protocols, not shared domain +tables in Core. + +1. **Time and history:** valid-from/to, recorded-at, superseded-at, revision, + change reason, and stable identity. +2. **Actor and representation:** real account/identity, system or service + account, represented account/function/party, delegation or power, and + mandate reference. +3. **Institutional context:** tenant, institution, organization unit, function, + task/mandate, jurisdiction, service, case, and decision references. +4. **Legal and policy basis:** typed, versioned references to rules, + obligations, policies, exceptions, and the effective decision source. +5. **Requested and observed effect:** intent, approval, dispatch, possible + execution, confirmation, reconciliation, correction, and terminal evidence. +6. **Evidence and provenance:** source, version, checksum, derivation, + responsible actor, timestamps, and inspection links. +7. **Information governance:** classification, purpose, legal basis, retention, + hold, minimization, and disclosure state. +8. **External source:** system/profile/object identity, authority mode, + maturity, version, freshness, health, and conflict state. +9. **Presentation:** language, accessibility, channel, explanation, and + configured availability. + +Existing contracts already cover substantial parts of items 1, 2, 5, 6, 8, +and 9. New work should extend those contracts instead of creating parallel DTO +families. + +## Existing module direction changes + +### Datasources becomes the governed data and register catalogue + +Keep the implemented live/cached/static, staging, immutable materialization, +and publication model. Add typed governance metadata for owner/steward, +authoritative source and authority mode, legal basis and purpose, semantic +definition, quality and freshness policy, classification, transfer agreement, +correction process, affected services/processes, and dependent flows, +reports, controls, and decisions. Connector credentials and protocol behavior +remain outside Datasources. + +### Projects grows into portfolio and change governance + +The Projects boundary already includes portfolios and goals. Extend it through +versioned objectives/outcomes, dependencies, capacity, benefits, change impact, +and links to mandates, services, risks, contracts, resources, and indicators. +Do not create a separate Goals module before more than one domain proves an +independent goal lifecycle. + +### Reporting becomes evidence-backed institutional measurement + +Every report, measure, and indicator should explain the institutional question +or obligation it serves, owner, source/materialization and flow revision, +freshness/quality, calculation version, visibility/purpose limits, publication, +and decisions or actions that consumed it. Reporting owns presentation and +execution; source and transformation owners retain their domains. + +### Risk Compliance becomes the horizontal assurance model + +Sanctions screening remains a complete vertical slice. The broader reusable +model is: + +```text +Obligation -> governed object -> risk -> control -> evidence -> finding -> measure -> effectiveness review +``` + +Risk Compliance should own that assurance graph and review lifecycle. Policy +owns enforceable rules and decisions; Audit owns immutable event evidence; +domain modules own the governed objects and corrective actions. + +### Connectors exposes authority and effect behavior + +Connector direction (`consume`, `publish`, `bidirectional`) remains useful but +is not enough. Profiles and bindings need the source-authority mode and +provider declaration above. ERP remains an integration family: finance, +workforce, procurement, asset, or other domain modules own semantics while +connectors own transport and source interaction. + +### Geography starts as a reference contract + +Before adding a `govoplan-geo` module, define a common reference shape for +coordinates, geometry, administrative area, address/location, CRS, source, +accuracy, validity, and external GIS identity. Create a repository only when +GovOPlaN must own spatial datasets, topology, or independent geospatial +lifecycles rather than link to an external GIS. + +## Product and sector packages + +A module says what capability can exist. A product package says how capabilities +work together for a bounded outcome. A sector package specializes vocabulary, +forms, rules, process baselines, controls, reports, and integration profiles +without forking the platform. + +The existing signed configuration-package mechanism is the implementation +base. It should distinguish: + +- **reference package:** tested composition proving a journey and its recovery + behavior; +- **product package:** reusable operating capability such as governed + communication, service-to-decision, procurement/contracts, or governed BI; +- **sector package:** institutional specialization such as municipality, + university/research, ministry/program, regulator, grants authority, or + committee/council; +- **deployment profile:** supported infrastructure and operational topology; +- **integration profile:** supported set of external systems, authority modes, + bindings, and health expectations. + +Packages may require modules and capabilities, but package definitions remain +configuration and evidence. They do not gain access to module-owned tables. + +## Module portfolio metadata + +Repository category is not capability maturity. The runtime manifest, release +catalog, Docs projection, and meta repository inventory should converge on a +machine-readable declaration with at least: + +- architecture layer and module kind; +- lifecycle/maturity claim: `concept`, `scaffold`, `vertical_slice`, + `reference_ready`, `supported`, or `lts`; +- evidence supporting the claim and known limits; +- supported source-authority modes; +- owned and explicitly non-owned concepts; +- provided/required capabilities and interfaces; +- reference packages and target-tested providers; +- migration, upgrade, recovery, security, and operations documentation. + +Maturity is a release claim and must be checked against evidence. A manifest +must not become “supported” merely because a maintainer changes one string. + +Create a repository only when the capability has distinct data ownership, +independent installability, technical assets, a security/lifecycle profile, a +release reason, more than one consumer or a proven reference process, and tests +that justify the boundary. Otherwise use a shared DTO, provider capability, +submodule, configuration fragment, package, or profile. + +## Migration sequence + +### 0. Align the portfolio and contracts + +- Publish this reconciliation and correct stale roadmap/module statements. +- Add machine-readable portfolio/source-authority metadata with validation and + evidence-based maturity claims. +- Compose the existing external-reference, action/effect, operational-health, + ownership, policy, audit, and documentation primitives into one provider + declaration. +- Add common institutional-context, legal-basis, evidence, and geo reference + DTOs only where current contracts do not already cover them. + +### 1. Prove responsibility and formal outcome + +- Define Mandate and Decision contracts. +- Use them in one bounded Case or Committee path with effective-time authority, + approval, reasoning, evidence, observed effect, correction, and review. +- Create separate repositories only when persistence and reuse pass the + repository threshold. + +### 2. Separate service and party semantics + +- Keep Portal as the service presentation surface while extracting a reusable, + versioned service-definition contract. +- Replace duplicated case/procedure participant fields with a common Party + reference and representation contract. +- Prove both through one general administrative service package. + +### 3. Complete governed data and assurance + +- Extend the Datasources catalogue with typed governance and dependency links. +- Complete Reporting semantic models and institutional indicator provenance. +- Generalize the Risk Compliance assurance graph after the sanctions vertical + is accepted. +- Extend Projects from execution tracking into portfolio/outcome governance as + a consuming domain, not as a second policy or reporting engine. + +### 4. Package repeatable public-sector outcomes + +- Finish the active Campaign/Postbox/data/BI/document reference program. +- Package governance foundation, governed communication, function-bound + collaboration, service-to-decision, procurement/contracts, and governed BI. +- Add sector packages only after a reference package has target-environment, + recovery, accessibility, privacy, and operator evidence. + +## Delivery tracking + +The cross-repository epic is +[GovOPlaN #29](https://git.add-ideas.de/GovOPlaN/govoplan/issues/29). +Its new work packages are: + +- [Core #279](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/279): + validated module architecture and provider authority declarations; +- [Core #280](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/280): + shared institutional-context and governed reference primitives; +- [GovOPlaN #30](https://git.add-ideas.de/GovOPlaN/govoplan/issues/30): + Mandates semantics; +- [GovOPlaN #31](https://git.add-ideas.de/GovOPlaN/govoplan/issues/31): + Services semantics; +- [GovOPlaN #32](https://git.add-ideas.de/GovOPlaN/govoplan/issues/32): + Parties and representation semantics; +- [GovOPlaN #33](https://git.add-ideas.de/GovOPlaN/govoplan/issues/33): + formal Decisions semantics; +- [Datasources #6](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/6): + governed data/register catalogue; +- [Risk Compliance #7](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance/issues/7): + horizontal assurance graph; +- [GovOPlaN #34](https://git.add-ideas.de/GovOPlaN/govoplan/issues/34): + product and sector package classes; and +- [Docs #19](https://git.add-ideas.de/GovOPlaN/govoplan-docs/issues/19): + configured architecture, maturity, and source-authority explanations. + +Existing Projects #1, Reporting #4, Portal #1, Cases #1, Datasources #1, +Risk Compliance #2, GovOPlaN #14, and GovOPlaN #19 carry reconciliation notes +instead of duplicate issues. + +## Completion criteria + +The direction is established when: + +- a user can move from service entry through case, party, work, decision, + effect, communication, record, and review without losing institutional + context; +- the system can answer who acted, for whom, in which function, under which + mandate and jurisdiction, using which rule and evidence versions; +- every external binding states authority mode, maturity, health, freshness, + conflict, and recovery behavior; +- every material report or decision can be reconstructed from governed source + and transformation versions; +- product and sector packages are portable without cross-module table access or + code forks; and +- documentation and Ops explain the configured composition and its limits to + users, administrators, operators, and auditors. diff --git a/Repo-docs-INTERFACE-SURFACE-INVENTORY.md b/Repo-docs-INTERFACE-SURFACE-INVENTORY.md index 8614b1f..fb6d1c5 100644 --- a/Repo-docs-INTERFACE-SURFACE-INVENTORY.md +++ b/Repo-docs-INTERFACE-SURFACE-INVENTORY.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/INTERFACE_SURFACE_INVENTORY.md`. > Origin: `repository`. @@ -16,6 +16,12 @@ The applicable design contract is ## Snapshot And Method +The source-derived inventory command is documented in +[`PLATFORM_CONTROL_PLANE.md`](PLATFORM_CONTROL_PLANE.md). It produces +machine-readable field, label, translation, route, API-reference, and module +manifest evidence. This hand-maintained document remains the reviewed product +interpretation and rollout ledger; generated evidence does not replace it. + Snapshot refreshed: 2026-07-22. Evidence was read from tracked Git `HEAD` in the local GovOPlaN checkouts: @@ -222,10 +228,11 @@ backend-only modules may remain intentionally headless. | 13 | Remaining direct routes | Routes are contributed; most are unreviewed | Per-module bounded audit and migration plan | New module issues derived from this inventory | Applicable definition-of-done gates | P2 after Campaign, not a bulk rewrite | | 14 | Manifest/runtime alignment | Several executable routes are absent from manifest metadata | Declared alignment or explicit validated exception | Core contract issue to create | Automated manifest/module route check and configured Docs verification | Discovery follow-up | -Workflow/user-story implementation is postponed. It is not on the critical path -for this rollout matrix. Focused views can be specified, manually selected, and -tested through core composition contracts; a later workflow step may become one -activation source without changing the proven surface patterns. +Workflow remains outside this rollout matrix because it has its own runtime and +editor workstream, not because it is postponed. Focused views can be specified, +manually selected, and tested through core composition contracts today. +Workflow steps may activate those views through the same contract without +changing the proven surface patterns. ## Inventory Maintenance diff --git a/Repo-docs-PLATFORM-CONTROL-PLANE.md b/Repo-docs-PLATFORM-CONTROL-PLANE.md new file mode 100644 index 0000000..ddf75e5 --- /dev/null +++ b/Repo-docs-PLATFORM-CONTROL-PLANE.md @@ -0,0 +1,133 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan/docs/PLATFORM_CONTROL_PLANE.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Platform Control Plane And Self-Description + +## Objective + +GovOPlaN should be able to describe its installed structure without becoming a +self-modifying application. The platform model is a declarative control plane: +module manifests, UI contributions, schemas, policy provenance, runtime +capabilities, and generated source evidence describe what can be configured. +Ordinary administrators edit validated data through those contracts; they do +not edit Python, TypeScript, routes, or database code from the product UI. + +This distinction provides the requested overview while preserving reviewable +releases, module boundaries, migrations, and security controls. + +## Canonical Sources + +| Concern | Canonical source | +| --- | --- | +| Installed modules and dependency graph | Runtime `ModuleManifest` registry | +| Backend routes | Registered FastAPI application; Python AST is build-time evidence | +| Frontend routes and navigation | `PlatformWebModule` contributions | +| View-filterable regions | Versioned `viewSurfaces` declarations | +| Admin sections and module settings | `admin.sections`, including `moduleId`, `kind`, scope group, permission guards, and surface ID | +| User settings | `settings.sections` and core settings schemas | +| Labels and translations | Generated translation catalogs plus source usage | +| Fields and help coverage | Shared form components plus generated TypeScript AST inventory | +| API use by the WebUI | Typed API clients plus generated static reference inventory | +| Effective configuration | Owning module data plus Policy provenance | + +Runtime introspection is authoritative for an installed system. Static source +inventory is authoritative evidence for a checkout or release candidate. The +two should be compared in CI and by Ops, not conflated. + +## Generated Inventory + +Run: + +```bash +cd /mnt/DATA/git/govoplan +./.venv/bin/python tools/inventory/platform-interface-inventory.py +``` + +The command writes: + +- `audit-reports/platform-inventory/platform-interface-inventory.json` +- `audit-reports/platform-inventory/platform-interface-inventory.md` + +It combines: + +1. loaded module manifests +2. TypeScript AST extraction of fields, label attributes, visible text, + translations, frontend routes, navigation, capabilities, and API references +3. Python AST extraction of FastAPI route decorators and router prefixes + +The JSON includes exact repository, file, and line evidence. A missing-help +entry is a review candidate because dynamic parent components may supply help. +A backend route without a static frontend reference is also a review candidate: +public APIs, workers, callbacks, health checks, connectors, and dynamic URL +assembly are valid explanations. + +`--strict` currently enforces only translation-catalog completeness. Endpoint +and help classifications need narrow reviewed baselines before they can become +release gates. + +## Admin Information Architecture + +The Admin host uses a tree because system, tenant, group, user, and module +settings form a hierarchy rather than one flat list. Every contributed section +can identify: + +- its owning `moduleId` +- whether it is `management` or `settings` +- its system/tenant/group/user scope group +- an optional future `parentId` +- permission and View visibility requirements + +Existing panels remain their own render owners. The tree only changes discovery +and grouping. A later embedded-settings contract may add named slots inside an +owning page; it must not allow one module to import another module's private +component. + +## Navigation And Workflow + +The intended maximum visible navigation stack is: + +1. global shell context +2. one task/object navigation surface +3. one workflow stage surface when a workflow is active + +Workflow instance pages should reuse the Campaign stage language: clear stage +state, optional/skipped/blocked semantics, partial progress, and a stable current +step. Workflow definition pages remain graph editors. Views may activate a +focused workflow view that suppresses unrelated shell and module surfaces while +retaining an explicit way out. + +Nested module submenus should not be added merely because a data hierarchy +exists. Prefer a tree inside configuration/directory surfaces, tabs for sibling +views, and the workflow stage rail for ordered work. + +## Safe Meta-Configuration + +The platform can eventually render many configuration editors from versioned +JSON Schema and UI Schema supplied by modules. Generated editors remain bounded +by: + +- explicit typed schemas and migrations +- module-owned validation and preview +- Policy locks and provenance +- permission and View filtering +- preflight, consequence, and rollback information +- auditable apply operations + +Custom code, new routes, arbitrary SQL, and executable workflow nodes remain +release artifacts. Modeling them as ordinary configuration would create an +unreviewed code-execution and migration channel. + +## Next Enforcement Slices + +1. Require every WebUI module route and admin/settings contribution to have + matching manifest metadata or a reviewed exception. +2. Add stable field IDs and optional help-topic IDs to shared field components. +3. Classify each statically unreferenced backend endpoint by consumer type. +4. Compare a running installation's OpenAPI and module registry against the + release inventory. +5. Publish the sanitized installed-system structure through Ops/Docs for + authorized administrators. diff --git a/Repo-docs-REFERENCE-JOURNEY-PROGRAM.md b/Repo-docs-REFERENCE-JOURNEY-PROGRAM.md index 72894c9..73c2bd5 100644 --- a/Repo-docs-REFERENCE-JOURNEY-PROGRAM.md +++ b/Repo-docs-REFERENCE-JOURNEY-PROGRAM.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/REFERENCE_JOURNEY_PROGRAM.md`. > Origin: `repository`. @@ -9,9 +9,16 @@ ## Status -This is the selected product-development sequence as of 2026-07-21. It turns -the long-term connected-platform roadmap into five demonstrable journeys while -the Workflow program remains deliberately deferred. +This is the selected product-development sequence, originally chosen on +2026-07-21 and reconciled with the implemented platform on 2026-07-31. It turns +the long-term connected-platform roadmap into five demonstrable journeys. +Workflow Engine and the optional Workflow editor now exist, but they are used +by a stage only when its package explicitly composes and proves them; Workflow +is not an automatic dependency of every journey. + +The institutional semantics and source-authority model applied to these stages +are defined in the +[Institutional Governance Target Architecture](INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md). The stages are ordered, but they are not monolithic releases. Each stage is delivered as small, reviewable, green increments and is complete only when its diff --git a/Repo-docs-RELEASE-CONSOLE.md b/Repo-docs-RELEASE-CONSOLE.md index 8264b97..68c3bfb 100644 --- a/Repo-docs-RELEASE-CONSOLE.md +++ b/Repo-docs-RELEASE-CONSOLE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/RELEASE_CONSOLE.md`. > Origin: `repository`. @@ -29,10 +29,17 @@ effects can be verified safely: - freeze a selective plan as a durable, resumable local release run - durably preflight a creation-time-bound repository, create its annotated tag, and publish its branch/tag pair atomically +- deterministically update recognized package/manifest version declarations and + commit only the receipt-bound metadata paths +- order selected module providers before consumers, create module tags before + Core, regenerate Core's selected WebUI release lock, and re-run alignment + before any remote push - build selected Python wheels and generate a private, signed, receipt-bound catalog candidate - publish that exact candidate through a verified website commit and immutable tag after explicit confirmation +- install selected candidate wheels into a private no-network/no-dependency + target and verify their installed metadata against the frozen plan Start it from the meta repository: @@ -74,15 +81,35 @@ shows the dry-run commands for the selected rows, and `Generate Candidate` creates a signed catalog candidate that advances only selected repositories that already have a catalog entry. +The full-width **Release Workflow** guide projects the server state into seven +operator phases: Inspect, Targets, Validate, Source, Package, Publish, and +Verify. It does not maintain a second workflow state. Completed, current, +blocked, locked, and unavailable phases are derived from the dashboard, +selective plan, and durable run record. The next-action panel opens the exact +section or durable step that needs attention. Changing a channel, target +version, repository selection, or release gate detaches the browser from the +current run and invalidates the draft plan; the persisted run remains available +from the saved-run selector. Problems in unselected repositories remain visible +as workspace notices but do not lock an unrelated release; the selective plan +is the authority for blockers in the selected repository set. + +Installation verification is an explicit durable step after catalog +publication. It verifies the exact candidate receipt, installs every selected +Python wheel into a temporary target with network and dependency resolution +disabled, and compares installed names and versions with the frozen plan. +Deployment startup, database upgrades, and module-combination smoke tests remain +release-integration CI gates; the console does not represent its local wheel +check as a production deployment. + `Build Plan` also returns structured release-gate findings for each selected repository. The plan names the recommended next action and gives an explicit remediation for source-version, lockfile, Core WebUI composition, Git state, and -worktree findings. A target version that has not yet been applied consistently -to the selected source tree is therefore explained before the source-tag -preflight rather than appearing only as an error after the operator tries to -tag. `source_preflight_ready` means that the plan-visible source gates pass; the -non-mutating `Preview Tag + Publish` remains mandatory for remote, manifest, and -immutable-tag checks. +worktree findings. A target version that differs from internally consistent +source metadata becomes a bounded `UPDATE` step. Unsupported, missing, or +internally inconsistent declarations remain a blocker with exact remediation. +`source_preflight_ready` means that plan-visible source gates pass or have a +bounded deterministic mutation; the non-mutating `Preview Tag + Publish` +remains mandatory for remote, manifest, and immutable-tag checks. ## Durable release runs @@ -92,11 +119,12 @@ requires the plan to resolve exactly the requested repositories and target versions; the browser cannot submit or replace the plan snapshot. The input and plan are then immutable and covered by a canonical SHA-256 integrity digest. Every executable repository step also carries its creation-time full HEAD, -branch, worktree state, target tag, and a SHA-256 over both the fetch and push -URLs of `origin`. Both URLs must exactly equal the remote registered in -`repositories.json`; a changed HEAD, branch, worktree, remote, or push URL -requires a new run rather than silently retargeting the frozen compatibility -decision. +branch, target tag, a SHA-256 over both the fetch and push URLs of `origin`, and +a SHA-256 over bounded dirty-path names and bytes. Both URLs must exactly equal +the remote registered in `repositories.json`; a changed HEAD, branch, worktree, +remote, push URL, or metadata byte requires a new run or explicit +interrupted-step reconciliation rather than silently retargeting the frozen +compatibility decision. The complete record also has a checksum so a valid-looking manual edit to its mutable state fails closed. File permissions remain the authority boundary; these digests detect accidental or manual corruption, not an attacker who can @@ -184,24 +212,32 @@ retry. The UI keeps unavailable controls visible and disabled. Supported executors durably claim the step before invoking an effect. Exact attempt replays return the recorded outcome and never invoke the executor a -second time. Successful repository preflight, tag, and push steps persist a -bounded repository-state receipt. Tag reconciliation independently requires an +second time. Successful repository preflight, version, commit, Core-bundle, +tag, and push steps persist a bounded repository-state receipt. Version +reconciliation requires aligned declarations in the same commit; commit +reconciliation requires the expected single-parent release commit and only +recognized metadata paths. Tag reconciliation independently requires an annotated local tag at the frozen HEAD; push reconciliation additionally requires both the remote annotated tag object and remote branch to match. Catalog generation persists only its server-issued opaque candidate ID and canonical catalog SHA-256, then re-resolves and re-hashes that private candidate before publication. -The safe baseline is intentionally narrower than the complete dry-run plan. -Dirty worktrees and version changes still show commit/version steps, but those -steps have no durable executor because the frozen run does not bind the exact -proposed file content. A run selecting Core together with one or more modules -begins with a disabled dependency-ordering barrier: local module tags, Core -release-lock regeneration/commit, Core tagging, alignment verification, and -pushes need an explicit DAG executor before that composition can mutate. -Operators must prepare and review those changes outside the console and create -a new run from the resulting clean HEAD. The console never skips these steps or -claims an end-to-end release succeeded. +Repository capabilities are frozen into each plan unit (`python-package`, +`webui-package`, `module-manifest`, `database-migrations`, `documentation`, +`core-release-bundle`, and the universal `git-source`) and determine which +steps appear. Internally aligned version changes are rendered deterministically +from recognized TOML, JSON, lockfile, manifest, and package declarations. +Pre-existing dirty worktrees remain visible but have no commit executor; the +console never absorbs unrelated operator changes. + +For mixed releases, module interface providers are ordered before consumers +and Core is tagged last. The durable sequence creates and commits module +metadata, creates local module tags, updates Core's selected WebUI references, +regenerates the release lock against those local tags, commits/tags Core, and +runs a receipt-bound alignment gate before exposing any atomic branch/tag push. +A failed step stops later steps while preserving prior receipts for explicit +retry or reconciliation. The browser likewise retains the request identifier for an uncertain resume/retry/reconciliation response and replays it after reload. A successful @@ -233,8 +269,9 @@ other `/api/` route: - `POST /api/release-runs/{run_id}/steps/{step_id}/preview` provides the non-mutating preview for receipt-bound catalog publication. -Run-storage errors are confined to the Durable Run State card; dashboard and -release-preview collection continue and show the bounded storage remediation. +Run-storage errors are confined to the Durable Release Run section; dashboard +and release-preview collection continue and the workflow guide points to the +bounded storage remediation. The run record is execution evidence only for a supported step whose durable claim and bounded result receipt were persisted. The console never infers diff --git a/Repo-docs-REPOSITORY-INDEX.md b/Repo-docs-REPOSITORY-INDEX.md index a5b0653..e14c570 100644 --- a/Repo-docs-REPOSITORY-INDEX.md +++ b/Repo-docs-REPOSITORY-INDEX.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`. > Origin: `repository`. @@ -41,6 +41,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s | `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) | | `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) | | `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) | +| `govoplan-encryption` | `platform` | `../govoplan-encryption` | [govoplan-encryption](https://git.add-ideas.de/GovOPlaN/govoplan-encryption) | | `govoplan-erp` | `domain` | `../govoplan-erp` | [govoplan-erp](https://git.add-ideas.de/GovOPlaN/govoplan-erp) | | `govoplan-evaluation` | `domain` | `../govoplan-evaluation` | [govoplan-evaluation](https://git.add-ideas.de/GovOPlaN/govoplan-evaluation) | | `govoplan-facilities` | `domain` | `../govoplan-facilities` | [govoplan-facilities](https://git.add-ideas.de/GovOPlaN/govoplan-facilities) | @@ -81,6 +82,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s | `govoplan-views` | `platform` | `../govoplan-views` | [govoplan-views](https://git.add-ideas.de/GovOPlaN/govoplan-views) | | `govoplan-wiki` | `domain` | `../govoplan-wiki` | [govoplan-wiki](https://git.add-ideas.de/GovOPlaN/govoplan-wiki) | | `govoplan-workflow` | `platform` | `../govoplan-workflow` | [govoplan-workflow](https://git.add-ideas.de/GovOPlaN/govoplan-workflow) | +| `govoplan-workflow-engine` | `platform` | `../govoplan-workflow-engine` | [govoplan-workflow-engine](https://git.add-ideas.de/GovOPlaN/govoplan-workflow-engine) | ## Connector diff --git a/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md b/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md new file mode 100644 index 0000000..3abe95a --- /dev/null +++ b/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md @@ -0,0 +1,163 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Scaling And Multi-Host Deployment + +## What Is Implemented + +The default installer now supports explicit same-host horizontal scaling: + +- one HAProxy container accepts the published HTTP endpoint; +- one or more WebUI containers serve assets and proxy API requests; +- one or more API containers serve normal domain traffic; +- one or more Celery workers consume shared Redis queues; +- one migration job runs before replacement; +- exactly one Celery scheduler runs; +- PostgreSQL, Redis, and file storage are shared by every runtime replica. + +HAProxy discovers Compose replicas through Docker's internal DNS and performs +health-checked round-robin WebUI balancing and least-connection API balancing. +It does not mount the Docker socket. Replica counts live in +`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on +the desired topology. + +This is not multi-host scheduling. Docker Compose's bridge network belongs to +one Docker Engine. Adding a second machine requires an orchestrator or +deployment manager that can place equivalent role definitions on multiple +hosts. + +## Recommended Topologies + +### One Host + +Use the generated Compose bundle: + +```text +client + -> external TLS proxy, when required + -> managed HAProxy + -> WebUI replica(s) + -> managed HAProxy API listener + -> API replica(s) + +API/worker/scheduler + -> PostgreSQL + -> Redis + -> local volume, managed Garage, or external S3 +``` + +This improves concurrency and permits rolling process replacement, but the host +and every managed stateful component remain single failure domains. Two API +containers on one failed host do not provide host-level availability. + +### Multiple Hosts + +Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not +extend the Compose installer into a proprietary scheduler. The target topology +is: + +```text +public TLS ingress/load balancer + -> WebUI replicas on at least two nodes + -> internal API service/load balancer + -> API replicas on at least two nodes + +queue-specific worker pools on worker nodes +one fenced scheduler +one fenced migration/deployment job + +shared PostgreSQL +shared Redis +multi-node Garage or external S3 +shared secret/config provider +central logs, metrics, and health alerts +``` + +All API and worker nodes must receive the same immutable software composition, +`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object +storage binding. Node-local file storage is not valid in this topology. + +## Adding Capacity + +### API And WebUI + +Increase API replicas for measured request concurrency, CPU saturation, or +latency after checking database load. Increase WebUI replicas for static asset +and proxy capacity. No sticky session should be required because durable +sessions and throttling use shared services, but this must remain covered by +multi-replica integration tests. + +Every added API process also adds database connections. Set the application +pool size and the total replica ceiling against PostgreSQL's connection budget; +adding replicas can otherwise reduce throughput. + +### Workers + +Workers are not placed behind a load balancer. They compete for jobs on shared +Redis queues. Add workers by queue and cap each pool according to the external +system it calls. SMTP, IMAP, directory, and connector jobs often reach provider +rate limits before CPU limits. + +Use distinct pools when load warrants it: + +- mail send and Sent-folder append; +- notifications; +- calendar and connector synchronization; +- dataflow/workflow execution; +- reporting and export; +- platform events and default work. + +Before stopping a worker, mark it draining, stop new claims, and let or safely +requeue active jobs. Worker registration and drain control remain part of the +larger platform scale-out story. + +### Stateful Services + +- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover + design appropriate to the service level. +- Redis needs persistence and, for high availability, a supported failover + topology. Queue loss is not equivalent to a harmless cache loss. +- Managed Garage from the Compose installer is single-node. For redundant + storage, deploy a multi-node Garage cluster separately and select external + `s3`, or use another compatible object store. +- The scheduler stays at one replica until distributed leader election or a + fenced lease is implemented. +- Migrations and module lifecycle mutations remain one-at-a-time operations. + +## Promotion Path + +1. Measure concurrent users, request latency, database query time, connection + use, queue age, worker saturation, storage latency, and external throttling. +2. Move files to managed Garage or external S3 before introducing independent + runtime hosts. +3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy + their reviewed HA operators. +4. Publish immutable API/WebUI images and one versioned configuration/secret + contract. +5. Translate the generated role commands, environment allowlists, health + checks, and singleton constraints into the chosen orchestrator. +6. Put API and WebUI replicas behind health-aware services and a TLS ingress. +7. Add queue-specific workers with fixed upper bounds. +8. Prove replica loss, rolling replacement, job redelivery, session continuity, + migration exclusion, backup restore, and storage-node loss before claiming + high availability. + +## Remaining Platform Work + +The one-host installer does not yet provide: + +- a Kubernetes, Nomad, or Swarm deployment export/profile; +- distributed deployment locks and fenced scheduler leadership; +- worker registration, composition-skew reporting, and drain controls; +- managed PostgreSQL, Redis, or multi-node Garage high availability; +- autoscaling policy; +- managed TLS certificate issuance and renewal; +- measured multi-replica and failover integration evidence. + +These are separate production capabilities. The local load balancer and +replica model provide the contract they should implement, but they do not by +themselves make a cluster highly available.