Sync wiki from project files

2026-08-01 14:56:45 +02:00
parent 5d87768c10
commit 9070e4e7c8
9 changed files with 467 additions and 158 deletions
+1
@@ -18,6 +18,7 @@ This page is generated from repository and product-directory project files.
- [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md` - [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md`
- [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md` - [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md`
- [Repo-docs-PLATFORM-CONTROL-PLANE](Repo-docs-PLATFORM-CONTROL-PLANE) - `/mnt/DATA/git/govoplan/docs/PLATFORM_CONTROL_PLANE.md` - [Repo-docs-PLATFORM-CONTROL-PLANE](Repo-docs-PLATFORM-CONTROL-PLANE) - `/mnt/DATA/git/govoplan/docs/PLATFORM_CONTROL_PLANE.md`
- [Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES](Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES) - `/mnt/DATA/git/govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md`
- [Repo-docs-REFERENCE-JOURNEY-PROGRAM](Repo-docs-REFERENCE-JOURNEY-PROGRAM) - `/mnt/DATA/git/govoplan/docs/REFERENCE_JOURNEY_PROGRAM.md` - [Repo-docs-REFERENCE-JOURNEY-PROGRAM](Repo-docs-REFERENCE-JOURNEY-PROGRAM) - `/mnt/DATA/git/govoplan/docs/REFERENCE_JOURNEY_PROGRAM.md`
- [Repo-docs-RELEASE-CONSOLE](Repo-docs-RELEASE-CONSOLE) - `/mnt/DATA/git/govoplan/docs/RELEASE_CONSOLE.md` - [Repo-docs-RELEASE-CONSOLE](Repo-docs-RELEASE-CONSOLE) - `/mnt/DATA/git/govoplan/docs/RELEASE_CONSOLE.md`
- [Repo-docs-REPOSITORY-INDEX](Repo-docs-REPOSITORY-INDEX) - `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md` - [Repo-docs-REPOSITORY-INDEX](Repo-docs-REPOSITORY-INDEX) - `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`
+8 -1
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:e8f6d622ddc3176e628d09c3 --> <!-- codex-wiki-sync:28496fdfb2db07160032de9f -->
> Mirrored from `/mnt/DATA/git/govoplan/README.md`. > Mirrored from `/mnt/DATA/git/govoplan/README.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -162,6 +162,9 @@ The current executable slice and remaining production gates are documented in
[Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
Same-host replica balancing and the multi-host promotion boundary are documented Same-host replica balancing and the multi-host promotion boundary are documented
in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
The recovery state machine, migration rollback boundary, and required restore
drills are documented in
[Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md).
## Configuration ## Configuration
@@ -204,6 +207,10 @@ reconfiguration semantics, and safe Web update boundary are defined in
[Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
The concrete replica, worker-node, load-balancer, and shared-state topology is The concrete replica, worker-node, load-balancer, and shared-state topology is
defined in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). defined in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
Durable deployment journals, Core recovery evidence, and the distinction
between pre-migration configuration restore and post-migration forward recovery
are defined in
[Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md).
The first Campaign-centric capability and infrastructure fit assessment is in The first Campaign-centric capability and infrastructure fit assessment is in
`docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. Its rerun tooling can collect and `docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. Its rerun tooling can collect and
verify a bounded installed composition; target, provider and production claims verify a bounded installed composition; target, provider and production claims
+15 -9
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:4765944ec28fc1daab87fa3c --> <!-- codex-wiki-sync:e4b2c1763e740cebf64cba2a -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. > Mirrored from `/mnt/DATA/git/govoplan/docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -483,10 +483,11 @@ which modules a running service activated, migrations, configuration, health,
or reference-journey behavior. Runtime activation therefore remains an explicit or reference-journey behavior. Runtime activation therefore remains an explicit
unchecked boundary. unchecked boundary.
### Target, provider and production proof boundary ### Reference-readiness, provider and production proof boundary
Installed evidence cannot approve a target environment, an external provider, Installed evidence cannot establish target acceptance, accessibility, privacy,
or production use. These scopes use a separate, expiring security, operations, recovery, an external provider, or production use. These
scopes use a separate, expiring
[`capability-fit-boundary-evidence.schema.json`](capability-fit-boundary-evidence.schema.json) [`capability-fit-boundary-evidence.schema.json`](capability-fit-boundary-evidence.schema.json)
bundle. The bundle is bound to the assessment ID, assessment release and exact bundle. The bundle is bound to the assessment ID, assessment release and exact
installed-evidence SHA-256 digest. It contains only opaque subject/control/result installed-evidence SHA-256 digest. It contains only opaque subject/control/result
@@ -495,8 +496,9 @@ IDs and content hashes, not endpoints, credentials, people or raw result files.
Boundary evidence is accepted only when at least one Ed25519 signature validates Boundary evidence is accepted only when at least one Ed25519 signature validates
against a separately provisioned against a separately provisioned
[`capability-fit-proof-authority-keyring.schema.json`](capability-fit-proof-authority-keyring.schema.json). [`capability-fit-proof-authority-keyring.schema.json`](capability-fit-proof-authority-keyring.schema.json).
Each authority key explicitly lists the scopes it may attest. Target and provider Each authority key explicitly lists the scopes it may attest. Target,
claims use `passed` or `failed`; production claims use `approved` or `rejected`. accessibility, privacy, security, operations, recovery, and provider claims use
`passed` or `failed`; production claims use `approved` or `rejected`.
One claim per scope, unique control/artifact IDs, `issued_at < expires_at`, current One claim per scope, unique control/artifact IDs, `issued_at < expires_at`, current
validity and exact digest binding are mandatory. Any schema, binding, time, validity and exact digest binding are mandatory. Any schema, binding, time,
signature or authority blocker leaves every supplied boundary claim unchecked; signature or authority blocker leaves every supplied boundary claim unchecked;
@@ -569,6 +571,10 @@ Target-environment and production-approval claims must use the assessment's
operator to supply a bounded opaque expected subject with operator to supply a bounded opaque expected subject with
`--expected-external-provider-subject`; without it, such a claim remains `--expected-external-provider-subject`; without it, such a claim remains
unchecked and blocks. Expected and observed IDs are retained in proof scope. unchecked and blocks. Expected and observed IDs are retained in proof scope.
Accessibility, privacy, security, operations, and recovery claims use the same
deployment subject. The report emits a cumulative `reference_readiness` verdict
only when all six required scopes are checked and positive. This verdict remains
separate from production approval and from provider-specific acceptance.
Both authority keyrings are governance trust roots. Installer receipt keys use Both authority keyrings are governance trust roots. Installer receipt keys use
the strict the strict
@@ -590,9 +596,9 @@ cannot relabel it as current. Live admission must omit it and use the actual
current time. current time.
No boundary bundle or production authority has been supplied for this current No boundary bundle or production authority has been supplied for this current
assessment. Target environment, provider and production proof therefore remain assessment. Reference-readiness, provider, and production proof therefore
explicitly unchecked rather than inferred from the local GreenMail journey, remain explicitly unchecked rather than inferred from the local GreenMail
source tests or signed release metadata. journey, source tests, or signed release metadata.
## Evidence used in this slice ## Evidence used in this slice
+80 -14
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:04ef6f09ecabe28900177d68 --> <!-- codex-wiki-sync:1e4a0a14da7f197bb96cac2c -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md`. > Mirrored from `/mnt/DATA/git/govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -41,9 +41,15 @@ It currently supports:
- generated secrets stored in a private `0600` file; - generated secrets stored in a private `0600` file;
- service-specific environment allowlists so infrastructure containers do not - service-specific environment allowlists so infrastructure containers do not
receive unrelated application credentials; receive unrelated application credentials;
- plan, render, doctor, status, and apply commands; - plan, render, doctor, status, apply, Kubernetes export, operation history,
and bounded recovery commands;
- an installation lock, migration-before-start ordering, readiness polling, - an installation lock, migration-before-start ordering, readiness polling,
and an applied-state receipt; and an applied-state receipt;
- a durable hash-chained deployment journal captured before runtime mutation;
- PostgreSQL advisory serialization for Core and module migrations;
- runtime initialization that waits for exact configured migration heads
without mutating schema;
- runtime node registration, heartbeats, drain state, and a fenced scheduler;
- idempotent reconfiguration that preserves generated secrets; - idempotent reconfiguration that preserves generated secrets;
- a keyed environment fingerprint that detects private binding changes without - a keyed environment fingerprint that detects private binding changes without
writing secret values to plans or receipts; writing secret values to plans or receipts;
@@ -94,6 +100,9 @@ The private installation directory contains:
| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration | | `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
| `plan.json` | Latest desired-state diff and readiness findings | | `plan.json` | Latest desired-state diff and readiness findings |
| `receipt.json` | Last successfully applied immutable identities | | `receipt.json` | Last successfully applied immutable identities |
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
| `.deployment.lock` | Same-host operation exclusion | | `.deployment.lock` | Same-host operation exclusion |
The specification contract is The specification contract is
@@ -208,11 +217,13 @@ presenting a container as a complete mail service.
installations. `garage` provisions Garage 2.3 in its supported single-node installations. `garage` provisions Garage 2.3 in its supported single-node
bootstrap mode, generates a private application key and bucket, and connects bootstrap mode, generates a private application key and bucket, and connects
the Files S3 backend to the exact installer-owned internal endpoint. The the Files S3 backend to the exact installer-owned internal endpoint. The
deployment-only trust marker cannot authorize another S3 host; arbitrary managed trust marker cannot authorize another S3 host.
external SDK endpoints remain fail-closed until peer pinning is implemented.
Garage metadata and object data use separate persistent volumes. `s3` requires Garage metadata and object data use separate persistent volumes. `s3` requires
an external endpoint, region, access key, secret key, and bucket values. an external endpoint, region, access key, secret key, and bucket values.
Self-hosted external S3 endpoints must use HTTPS. Self-hosted external S3 endpoints must be clean HTTPS origins. The generated
runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that
operator-selected endpoint. The trust flag is not accepted for local storage
and cannot be combined with installer-managed Garage trust.
Local storage must be included in backup and restore drills. Horizontal API or Local storage must be included in backup and restore drills. Horizontal API or
worker scale-out requires shared/object storage. The managed Garage profile is worker scale-out requires shared/object storage. The managed Garage profile is
@@ -240,9 +251,10 @@ Replica counts are desired state:
``` ```
Workers are queue consumers, so they are scaled through Redis rather than put Workers are queue consumers, so they are scaled through Redis rather than put
behind an HTTP load balancer. The migration runner and Celery scheduler remain behind an HTTP load balancer. Migrations are serialized with a deployment-wide
singletons. Multiple API replicas are rejected when Redis is disabled because PostgreSQL advisory lock. The Celery scheduler is run under a renewable,
distributed throttling and queued work cannot then be shared correctly. fencing-token lease. Multiple API replicas are rejected when Redis is disabled
because distributed throttling and queued work cannot then be shared correctly.
This is same-host scaling. Docker Compose uses a bridge network and does not This is same-host scaling. Docker Compose uses a bridge network and does not
place containers on another machine. See place containers on another machine. See
@@ -265,13 +277,66 @@ applied state. `plan` compares their canonical hashes and service sets.
- Managed-to-external transitions require the new endpoint in the same - Managed-to-external transitions require the new endpoint in the same
operation. operation.
- Migrations run as a one-shot service before API/worker replacement. - Migrations run as a one-shot service before API/worker replacement.
- API, worker, and scheduler start commands wait for exact configured migration
heads; only the migration command is permitted to change schema.
- API and worker replicas register their software/module composition and
heartbeat in PostgreSQL. Ops can request and cancel a node drain.
- The first upgrade from a direct WebUI host port stops that legacy WebUI - The first upgrade from a direct WebUI host port stops that legacy WebUI
container immediately before HAProxy claims the same endpoint. container immediately before HAProxy claims the same endpoint.
- Health must recover before a new receipt is committed. - Health must recover before a new receipt and applied-state snapshot are
committed.
This is sufficient for one-host reconciliation. Production updates additionally Every apply operation is journalled before image pulls or runtime mutation. A
need backup/restore gates, maintenance/drain state, database compatibility failure before migration may restore a verified previous bundle. Once migration
windows, image signature verification, and rollback/forward-recovery policy. starts, recovery is forward-only unless an independently verified database
backup is restored. See
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
Production updates still need an operator-provided database backup/restore
gate, database compatibility declaration, image signature verification, and
deployment-specific drain policy. The deployment journal proves its own
actions; it does not manufacture backup evidence.
## Stateless Kubernetes Runtime
`render-kubernetes` exports the application tier for a standard orchestrator.
It requires external PostgreSQL, Redis, and S3 and emits no stateful service or
secret value:
```sh
python tools/deployment/govoplan-deploy.py render-kubernetes \
--directory /srv/govoplan/default \
--namespace govoplan \
--secret-name govoplan-runtime
```
The output includes a release-specific migration Job, database-head wait init
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
the named Secret through the cluster's secret manager and review ingress proxy
CIDRs before deployment. Detailed rollout and scaling rules live in
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
## Recovery Commands
List durable deployment operations:
```sh
python tools/deployment/govoplan-deploy.py operations \
--directory /srv/govoplan/default
```
Recover a selected failed operation after reviewing its stage evidence:
```sh
python tools/deployment/govoplan-deploy.py recover \
--directory /srv/govoplan/default \
--operation-id <operation-id>
```
The command reports whether it restored the pre-migration applied bundle,
requires forward recovery, or needs manual intervention. Add `--apply` only
after that decision has been reviewed.
## Web Update Boundary ## Web Update Boundary
@@ -324,5 +389,6 @@ The tests cover profile restrictions, secret persistence, external endpoint
requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
discovery configuration, Compose service selection, secret non-disclosure, discovery configuration, Compose service selection, secret non-disclosure,
service-specific environment isolation, private file modes, external endpoint service-specific environment isolation, private file modes, external endpoint
preflight, first-plan generation, apply ordering, and receipt-based preflight, first-plan generation, apply ordering, receipt idempotency,
idempotency. hash-chained recovery journals, migration recovery boundaries, and stateless
Kubernetes rendering.
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:46024a5062a8085c6ffc152f --> <!-- codex-wiki-sync:7b1befe1315b504d421507a4 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md`. > Mirrored from `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -90,11 +90,12 @@ were proven now have independent persistent owners:
| External providers | Core validates provider objects/field groups, operations, integration maturity, source authority, bounded reads, freshness/health, idempotency, conflicts, outcome-unknown handling, evidence, correction, reconciliation, outage, classification, purpose, retention, and secret handling. Addresses/CardDAV, Files remote storage, Mail SMTP/IMAP, Calendar CalDAV/ICS/Graph/EWS, and Connectors tabular/sanctions providers declare the contract and tenant-bounded secret-free runtime state. | Registry validation rejects any declared external provider without a sanitized state provider. Future adapters must cross the same gate before activation. | | External providers | Core validates provider objects/field groups, operations, integration maturity, source authority, bounded reads, freshness/health, idempotency, conflicts, outcome-unknown handling, evidence, correction, reconciliation, outage, classification, purpose, retention, and secret handling. Addresses/CardDAV, Files remote storage, Mail SMTP/IMAP, Calendar CalDAV/ICS/Graph/EWS, and Connectors tabular/sanctions providers declare the contract and tenant-bounded secret-free runtime state. | Registry validation rejects any declared external provider without a sanitized state provider. Future adapters must cross the same gate before activation. |
| Institutional context | Core provides versioned temporal, actor/representation, institution/unit/function/task/mandate/jurisdiction/service/case/party/work-item/workflow/approval/decision/record, legal-basis, evidence, information-governance, external-source, presentation, and geographic references. Events, automation actions, audit records, and the transactional Audit outbox preserve the envelope. | Owning modules must progressively require the relevant subset for consequential operations. | | Institutional context | Core provides versioned temporal, actor/representation, institution/unit/function/task/mandate/jurisdiction/service/case/party/work-item/workflow/approval/decision/record, legal-basis, evidence, information-governance, external-source, presentation, and geographic references. Events, automation actions, audit records, and the transactional Audit outbox preserve the envelope. | Owning modules must progressively require the relevant subset for consequential operations. |
| Semantic provider contracts | Provider-neutral DTOs and protocols cover Mandate resolution, versioned Service definitions, procedure Parties/representation, and formal Decisions. `govoplan-mandates`, `govoplan-services`, `govoplan-parties`, and `govoplan-decisions` now persist immutable revisions behind those contracts with tenant isolation, bounded reads, replay safety, OCC, migrations, uninstall guards, permissions, APIs, capability documentation, and recovery documentation. | The owners are deliberately headless. Procedure-specific UI remains with consuming modules. | | Semantic provider contracts | Provider-neutral DTOs and protocols cover Mandate resolution, versioned Service definitions, procedure Parties/representation, and formal Decisions. `govoplan-mandates`, `govoplan-services`, `govoplan-parties`, and `govoplan-decisions` now persist immutable revisions behind those contracts with tenant isolation, bounded reads, replay safety, OCC, migrations, uninstall guards, permissions, APIs, capability documentation, and recovery documentation. | The owners are deliberately headless. Procedure-specific UI remains with consuming modules. |
| Formal-outcome proof | Committee persists bodies, meetings, agenda items, votes, minutes, lifecycle events, and an optional protected local Decision projection. It resolves effective Mandate authority and records reconstructable formal Decisions through `decisions.registry` when installed. OCC, replay safety, tenant isolation, bounded reads, and closure guards cover the aggregate. | Committee remains API-first and does not yet provide its later deliberation WebUI or specialized secret/external ballot adapters. | | Formal-outcome proof | Committee persists bodies, meetings, agenda items, votes, minutes, lifecycle events, and an optional protected local Decision projection. It resolves effective Mandate authority and records reconstructable formal Decisions through `decisions.registry` when installed. OCC, replay safety, tenant isolation, bounded reads, closure guards, and a full-height body/meeting/agenda/vote/minutes WebUI cover the aggregate. Provider-bound external or secret ballots use dynamic adapter capabilities; Committee validates and retains only aggregate counts, receipt/hash, and evidence, and rejects generic manual closure. | Concrete ballot-provider packages remain integration work because their protocol, custody, credentials, and operator evidence depend on the selected provider. This does not leave a Committee-owned ballot contract unspecified. |
| Service-to-case proof | Services owns the persistent exact definitions consumed by Portal discovery and Cases intake. Portal exposes a tenant-scoped API and compact service-directory WebUI whose audiences are derived from trusted principal/function state. Cases persists catalogs, immutable revisions, parties, assignments, evidence/decision/record references, deadlines, lifecycle events, exact Service versions, OCC, replay safety, tenant isolation, API permissions, ACL filtering, uninstall guards, tenant summary state, and list/detail/history/timeline WebUI. | Specialized form/case/workflow launchers and procedure panels remain with their owners. Cases still needs finer case-sharing policy. Neither gap changes Service ownership. | | Service-to-case proof | Services owns the persistent exact definitions consumed by Portal discovery and Cases intake. Portal exposes a tenant-scoped API and service-directory WebUI whose audiences are derived from trusted principal/function state. Launch re-fetches and re-evaluates the exact Service revision. URL launch is validated; Cases and Workflow Engine provide tenant-bound, replay-safe owner launchers. Cases persists catalogs, immutable revisions, parties, assignments, evidence/decision/record references, deadlines, lifecycle events, exact Service versions, OCC, replay safety, tenant isolation, explicit and assignment-derived object ACLs, uninstall guards, tenant summary state, and list/detail/history/timeline/share WebUI. | Form-bound launch remains unavailable until Forms and Forms Runtime move beyond scaffolds to immutable definitions, validation, submission persistence, and a real owner launcher. Portal fails this case closed and explains the missing capability. |
| Procedure-party proof | Parties persists effective procedure roles, frozen contact snapshots, and representation powers. Existing powers cannot disappear or be silently rewritten; explicit OCC-guarded revocation is required. Cases resolves the provider capability and excludes expired/revoked authority from downstream delivery. | Procedure modules still decide which contextual fields and actions to present. | | Procedure-party proof | Parties persists effective procedure roles, frozen contact snapshots, and representation powers. Existing powers cannot disappear or be silently rewritten; explicit OCC-guarded revocation is required. Cases resolves the provider capability and excludes expired/revoked authority from downstream delivery. | Procedure modules still decide which contextual fields and actions to present. |
| Integrated institutional journey | The executable `product.service-to-decision` fixture uses real SQL-backed Services, Cases, Parties, Mandates, Committee, and Decisions providers. It carries one exact Service version through persisted Case intake, representation and frozen delivery authority, effective Mandate resolution, a body/meeting/agendum/vote/minute sequence, a persisted formal Decision, confirmed Postbox effect, Audit/record evidence, remedy/review, and protected reconstruction. | This is architecture and composition evidence. Signed, release-bound target accessibility, privacy, security, operator, delivery-provider, and recovery-drill evidence is still required before the package may claim `reference_ready`. | | Integrated institutional journey | The executable `product.service-to-decision` fixture uses real SQL-backed Services, Cases, Parties, Mandates, Committee, and Decisions providers. It carries one exact Service version through persisted Case intake, representation and frozen delivery authority, effective Mandate resolution, a body/meeting/agendum/vote/minute sequence, a persisted formal Decision, confirmed Postbox effect, Audit/record evidence, remedy/review, and protected reconstruction. | This is architecture and composition evidence. Signed, release-bound target accessibility, privacy, security, operator, delivery-provider, and recovery-drill evidence is still required before the package may claim `reference_ready`. |
| Governed data catalogue | Datasources stores typed governance metadata, exposes bounded tenant-scoped filters and update APIs/UI, carries governance through staging, and snapshots it into immutable materializations. | Rich dependency/impact traversal and policy-specific field visibility can grow on the catalogue contract without moving connector or transformation ownership. | | Governed data catalogue | Datasources stores typed governance metadata, exposes bounded tenant-scoped filters and update APIs/UI, carries governance through staging, and snapshots it into immutable materializations. Reporting now persists immutable dataset, semantic-model, report, quality-plan, saved-view, and schedule revisions; executes typed semantic queries with quality gates, access checks, replay, pivoting, export/import assessment, and provenance; and exposes the governed analytical WebUI. | Rich dependency/impact traversal, additional expression functions, and policy-specific field visibility can grow on the established contracts without moving connector, transformation, or source ownership. |
| Portfolio and change governance | Projects now persists tenant-safe, immutable portfolio/project/milestone revisions with OCC, replay, lifecycle rules, restricted memberships, Search ACL indexing, outcomes, benefits, dependencies, capacity assumptions, change impact, and institutional references. Its WebUI exposes the planning catalogue and core planning fields. | Advanced planning structures already accepted by the API can receive deeper specialized editors without creating a second Policy, Reporting, Resources, or Goals owner. |
| Product/package governance | Signed configuration packages distinguish reference, product, sector, deployment, and integration classes; preserve parent/evidence provenance; prevent derived packages from loosening constraints; and preflight provider authority, maturity, exact binding, health, freshness, and recovery expectations. Executable product manifests now exist for governed communication and governed data/assurance and are checked in the module matrix. | Both artifacts deliberately remain product-class until target, accessibility, privacy, security, operations, and recovery evidence justifies reference readiness. | | Product/package governance | Signed configuration packages distinguish reference, product, sector, deployment, and integration classes; preserve parent/evidence provenance; prevent derived packages from loosening constraints; and preflight provider authority, maturity, exact binding, health, freshness, and recovery expectations. Executable product manifests now exist for governed communication and governed data/assurance and are checked in the module matrix. | Both artifacts deliberately remain product-class until target, accessibility, privacy, security, operations, and recovery evidence justifies reference readiness. |
| Projection and release | Platform metadata, signed module catalogs, release synthesis, Ops, and role-aware Docs retain and display architecture/provider declarations. Module-owned state providers add bounded configured/active, authority, health, freshness, conflict, recovery, and observation state; ordinary-user Docs omits binding detail. Static checks validate evidence paths, and the WebUI build verifies consuming types. | Runtime-state adoption and broader portfolio presentation follow truthful provider declaration rollout. | | Projection and release | Platform metadata, signed module catalogs, release synthesis, Ops, and role-aware Docs retain and display architecture/provider declarations. Module-owned state providers add bounded configured/active, authority, health, freshness, conflict, recovery, and observation state; ordinary-user Docs omits binding detail. Static checks validate evidence paths, and the WebUI build verifies consuming types. | Runtime-state adoption and broader portfolio presentation follow truthful provider declaration rollout. |
@@ -424,17 +425,17 @@ submodule, configuration fragment, package, or profile.
- `product.service-to-decision` proves both through a portable administrative - `product.service-to-decision` proves both through a portable administrative
service composition. service composition.
### 3. Complete governed data and assurance - architecture complete ### 3. Complete governed data, portfolio, and assurance - vertical slices complete
- Datasources carries typed governance through staging and immutable - Datasources carries typed governance through staging and immutable
materializations, with bounded catalogue filters and dependency references. materializations, with bounded catalogue filters and dependency references.
- Reporting owns semantic presentation/provenance contracts without taking - Reporting owns immutable semantic definitions, safe execution, quality gates,
source or transformation ownership; feature depth remains on its module provenance, schedules, saved views, pivoting, and export/import assessment
backlog. without taking source or transformation ownership.
- Risk Compliance persists the horizontal obligation/risk/control/evidence/ - Risk Compliance persists the horizontal obligation/risk/control/evidence/
finding/measure graph and projects sanctions runs idempotently. finding/measure graph and projects sanctions runs idempotently.
- Projects declares portfolio/outcome ownership as a consuming domain without - Projects persists portfolio/outcome/change-governance revisions as a
becoming a second policy or reporting engine. consuming domain without becoming a second policy or reporting engine.
### 4. Package repeatable public-sector outcomes - complete at product maturity ### 4. Package repeatable public-sector outcomes - complete at product maturity
@@ -446,6 +447,36 @@ submodule, configuration fragment, package, or profile.
recovery, accessibility, privacy, security, and operator evidence. This is a recovery, accessibility, privacy, security, and operator evidence. This is a
maturity gate, not missing architecture implementation. maturity gate, not missing architecture implementation.
## What remains after the executable architecture slice
The remaining work is not another Core or cross-module architecture rewrite.
It falls into three explicitly different categories:
1. **Forms and Forms Runtime vertical:** both repositories remain scaffolds.
Full service entry requires immutable form-definition lookup, server-side
validation, draft/submission persistence, status/evidence history,
attachments/signatures/handoff references, replay and OCC, APIs, accessible
WebUI, migrations, recovery, and `forms_runtime.service_launcher`. Portal
already advertises this absence as an availability blocker. Implementing a
placeholder launcher before those controls would violate the accepted
contract.
2. **Concrete provider packages:** the Committee ballot adapter contract is
complete, but a real secret/electronic ballot provider requires a selected
protocol and product decisions for voter eligibility, custody, secrecy,
recount, challenge, retention, and operational assurance. Equivalent future
adapters must satisfy the declared provider and recovery gates.
3. **Target-produced maturity evidence:** `reference_ready`, `supported`, and
`lts` cannot be generated from source code. An exact release and deployment
must produce signed, expiring accessibility, privacy, security, operator,
provider, backup/restore, rollback, and recovery-drill evidence. The verifier
and schemas are implemented; the actual claims require those real runs.
Everything else described as architecture in this document now has a
repository owner, versioned contract, bounded implementation, migration and
recovery boundary where state exists, documentation, and executable evidence.
Further work in those modules is product breadth, UX depth, provider adoption,
and evidence renewal.
## Delivery tracking ## Delivery tracking
The completed cross-repository architecture epic is The completed cross-repository architecture epic is
+154
@@ -0,0 +1,154 @@
<!-- codex-wiki-sync:e432d0c3b51f11d9f6c79468 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md`.
> Origin: `repository`.
> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
---
# Recovery And Rollback Guarantees
## Principle
GovOPlaN must prove recovery claims with durable state recorded before and
after side effects. A failed operation is not automatically rolled back merely
because the previous application image still exists. Database schema and
external effects may make release rollback unsafe.
Core therefore distinguishes five recovery modes:
| Mode | Meaning |
| --- | --- |
| `atomic` | One database transaction either commits or rolls back. No external effect is claimed. |
| `compensation` | Durable evidence identifies explicit inverse actions for completed effects. |
| `snapshot_restore` | A separately verified backup reference and restore procedure exist. |
| `forward_recovery` | Repair or resume the current version; reverting code/configuration is not claimed safe. |
| `irreversible` | No automated recovery is claimed and an approval reference is mandatory. |
An operation plan must include verification steps. Compensation requires named
compensation steps, snapshot restore requires a verified backup reference,
forward recovery requires repair steps, and irreversible work requires explicit
approval.
## Core Recovery Ledger
Core stores recovery operations and append-only, hash-chained checkpoints in
PostgreSQL. The contract provides:
- installation/module/resource identity;
- an idempotency key bound to a canonical request hash;
- recovery mode, preconditions, verification steps, and references;
- optional runtime lease holder and fencing token;
- explicit planned, prepared, running, recovery-required, recovering,
succeeded, recovered, failed, outcome-unknown, and manual-intervention states;
- an evidence-chain head and sequence count;
- rejection of plaintext secrets in metadata or evidence.
Preparation cannot succeed without durable precondition evidence. A non-atomic
operation cannot hide a partial effect by transitioning directly from running
to failed. Success and recovery require explicit verification evidence with at
least one check. The ledger verifies its hash chain before evidence is trusted.
This is a platform contract, not an assertion that every existing module
operation has adopted it. Module operations with external or multi-resource
effects must be migrated to the ledger before claiming these guarantees.
## Deployment Journal
Every `govoplan-deploy apply` begins an operation journal before it pulls images
or mutates runtime state. The private installation directory records:
```text
operations/<operation-id>/operation.json
operations/<operation-id>/before/
applied-state/
```
Each stage is hash-chained. The previous applied bundle is copied with per-file
SHA-256 evidence. Applied state is replaced atomically after health verification;
an interrupted replacement restores its previous directory.
New journals also bind the complete desired deployment plan, snapshot
availability, failure summary, recovery mode, and terminal status to the
evidence chain. Recovery verifies every snapshot entry and checksum before it
changes any live bundle file, then replaces each live file atomically. A crash
between file replacements is recoverable by rerunning the same idempotent
recovery command under the deployment lock.
Inspect operations:
```sh
python tools/deployment/govoplan-deploy.py operations \
--directory /srv/govoplan/default
```
Recover the latest failed operation, or provide its identifier:
```sh
python tools/deployment/govoplan-deploy.py recover \
--directory /srv/govoplan/default \
--operation-id 20260801T120000Z-1234abcd
```
Add `--apply` only after reviewing the reported action.
## Migration Boundary
Before database migration starts, a failed deployment with a verified prior
applied snapshot may restore its prior release/configuration bundle and
reconcile that desired state.
As soon as migration starts, the journal permanently changes to
`forward_recovery`. It will not restore old application configuration because
old code may not understand the new schema. Recovery then means one of:
1. fix and re-run the current release;
2. deploy a newer compatible repair release;
3. restore a separately verified, coordinated database/object/key backup and
then deploy the matching release.
The deployment tool does not create or validate that database backup. A
`backup-required` annotation on the Kubernetes migration Job is an operator
gate, not backup evidence. Production automation must provide a backup hook or
external backup controller whose artifact, timestamp, scope, encryption key,
and restore test can be referenced from the recovery record.
## Scaled Nodes
Recovery actions must be safe across replicas:
- drain affected API and worker nodes before incompatible changes;
- use the deployment-wide PostgreSQL advisory lock for schema migration;
- use distributed leases and fencing tokens for singleton or externally
visible effects;
- use idempotency keys for retried commands and jobs;
- retain shared object keys and database references until deletion succeeds;
- classify uncertain external outcomes instead of retrying blindly;
- verify the exact software/module composition after replacement.
Campaign generated-message objects now follow this model: object writes are
compensated when a build fails before database commit, workers verify stored
size and digest before delivery, and retention keeps the database reference
when storage deletion fails. A hard process loss between object creation and
database commit can still leave an orphan object; an inventory reconciler is a
separate operational slice and must use the build-specific object prefix.
## Required Drills
Record evidence for at least these scenarios before production acceptance:
1. Kill an API replica and verify traffic continues without session loss.
2. Drain and replace a worker while work is queued and while one job is active.
3. Start two migration jobs and verify only one mutates schema.
4. Kill the fenced scheduler and verify one replacement acquires a higher
fencing token.
5. Fail deployment before migration and restore the prior applied bundle.
6. Fail deployment after migration and verify old configuration is not
restored.
7. Restore PostgreSQL, object storage, and encryption keys to one coordinated
recovery point and verify representative object hashes.
8. Interrupt object storage during Campaign build and retention and verify
compensation/reference-preservation behavior.
9. Tamper with a deployment or Core recovery checkpoint and verify chain
validation rejects it.
No runbook, status badge, or green health endpoint substitutes for a dated,
repeatable restore drill against the actual deployment topology.
+5 -1
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:074fbacd975d50c1718b7e70 --> <!-- codex-wiki-sync:3a6877289005519a3e438aaf -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`. > Mirrored from `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -38,6 +38,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard) | | `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard) |
| `govoplan-dataflow` | `platform` | `../govoplan-dataflow` | [govoplan-dataflow](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow) | | `govoplan-dataflow` | `platform` | `../govoplan-dataflow` | [govoplan-dataflow](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow) |
| `govoplan-datasources` | `platform` | `../govoplan-datasources` | [govoplan-datasources](https://git.add-ideas.de/GovOPlaN/govoplan-datasources) | | `govoplan-datasources` | `platform` | `../govoplan-datasources` | [govoplan-datasources](https://git.add-ideas.de/GovOPlaN/govoplan-datasources) |
| `govoplan-decisions` | `domain` | `../govoplan-decisions` | [govoplan-decisions](https://git.add-ideas.de/GovOPlaN/govoplan-decisions) |
| `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) | | `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) |
| `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) | | `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) |
| `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) | | `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) |
@@ -57,10 +58,12 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-learning` | `domain` | `../govoplan-learning` | [govoplan-learning](https://git.add-ideas.de/GovOPlaN/govoplan-learning) | | `govoplan-learning` | `domain` | `../govoplan-learning` | [govoplan-learning](https://git.add-ideas.de/GovOPlaN/govoplan-learning) |
| `govoplan-ledger` | `domain` | `../govoplan-ledger` | [govoplan-ledger](https://git.add-ideas.de/GovOPlaN/govoplan-ledger) | | `govoplan-ledger` | `domain` | `../govoplan-ledger` | [govoplan-ledger](https://git.add-ideas.de/GovOPlaN/govoplan-ledger) |
| `govoplan-mail` | `domain` | `../govoplan-mail` | [govoplan-mail](https://git.add-ideas.de/GovOPlaN/govoplan-mail) | | `govoplan-mail` | `domain` | `../govoplan-mail` | [govoplan-mail](https://git.add-ideas.de/GovOPlaN/govoplan-mail) |
| `govoplan-mandates` | `domain` | `../govoplan-mandates` | [govoplan-mandates](https://git.add-ideas.de/GovOPlaN/govoplan-mandates) |
| `govoplan-notifications` | `platform` | `../govoplan-notifications` | [govoplan-notifications](https://git.add-ideas.de/GovOPlaN/govoplan-notifications) | | `govoplan-notifications` | `platform` | `../govoplan-notifications` | [govoplan-notifications](https://git.add-ideas.de/GovOPlaN/govoplan-notifications) |
| `govoplan-ops` | `platform` | `../govoplan-ops` | [govoplan-ops](https://git.add-ideas.de/GovOPlaN/govoplan-ops) | | `govoplan-ops` | `platform` | `../govoplan-ops` | [govoplan-ops](https://git.add-ideas.de/GovOPlaN/govoplan-ops) |
| `govoplan-organizations` | `platform` | `../govoplan-organizations` | [govoplan-organizations](https://git.add-ideas.de/GovOPlaN/govoplan-organizations) | | `govoplan-organizations` | `platform` | `../govoplan-organizations` | [govoplan-organizations](https://git.add-ideas.de/GovOPlaN/govoplan-organizations) |
| `govoplan-payments` | `domain` | `../govoplan-payments` | [govoplan-payments](https://git.add-ideas.de/GovOPlaN/govoplan-payments) | | `govoplan-payments` | `domain` | `../govoplan-payments` | [govoplan-payments](https://git.add-ideas.de/GovOPlaN/govoplan-payments) |
| `govoplan-parties` | `domain` | `../govoplan-parties` | [govoplan-parties](https://git.add-ideas.de/GovOPlaN/govoplan-parties) |
| `govoplan-permits` | `domain` | `../govoplan-permits` | [govoplan-permits](https://git.add-ideas.de/GovOPlaN/govoplan-permits) | | `govoplan-permits` | `domain` | `../govoplan-permits` | [govoplan-permits](https://git.add-ideas.de/GovOPlaN/govoplan-permits) |
| `govoplan-policy` | `platform` | `../govoplan-policy` | [govoplan-policy](https://git.add-ideas.de/GovOPlaN/govoplan-policy) | | `govoplan-policy` | `platform` | `../govoplan-policy` | [govoplan-policy](https://git.add-ideas.de/GovOPlaN/govoplan-policy) |
| `govoplan-poll` | `domain` | `../govoplan-poll` | [govoplan-poll](https://git.add-ideas.de/GovOPlaN/govoplan-poll) | | `govoplan-poll` | `domain` | `../govoplan-poll` | [govoplan-poll](https://git.add-ideas.de/GovOPlaN/govoplan-poll) |
@@ -74,6 +77,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-risk-compliance` | `domain` | `../govoplan-risk-compliance` | [govoplan-risk-compliance](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance) | | `govoplan-risk-compliance` | `domain` | `../govoplan-risk-compliance` | [govoplan-risk-compliance](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance) |
| `govoplan-scheduling` | `domain` | `../govoplan-scheduling` | [govoplan-scheduling](https://git.add-ideas.de/GovOPlaN/govoplan-scheduling) | | `govoplan-scheduling` | `domain` | `../govoplan-scheduling` | [govoplan-scheduling](https://git.add-ideas.de/GovOPlaN/govoplan-scheduling) |
| `govoplan-search` | `platform` | `../govoplan-search` | [govoplan-search](https://git.add-ideas.de/GovOPlaN/govoplan-search) | | `govoplan-search` | `platform` | `../govoplan-search` | [govoplan-search](https://git.add-ideas.de/GovOPlaN/govoplan-search) |
| `govoplan-services` | `domain` | `../govoplan-services` | [govoplan-services](https://git.add-ideas.de/GovOPlaN/govoplan-services) |
| `govoplan-tasks` | `domain` | `../govoplan-tasks` | [govoplan-tasks](https://git.add-ideas.de/GovOPlaN/govoplan-tasks) | | `govoplan-tasks` | `domain` | `../govoplan-tasks` | [govoplan-tasks](https://git.add-ideas.de/GovOPlaN/govoplan-tasks) |
| `govoplan-templates` | `domain` | `../govoplan-templates` | [govoplan-templates](https://git.add-ideas.de/GovOPlaN/govoplan-templates) | | `govoplan-templates` | `domain` | `../govoplan-templates` | [govoplan-templates](https://git.add-ideas.de/GovOPlaN/govoplan-templates) |
| `govoplan-tenancy` | `platform` | `../govoplan-tenancy` | [govoplan-tenancy](https://git.add-ideas.de/GovOPlaN/govoplan-tenancy) | | `govoplan-tenancy` | `platform` | `../govoplan-tenancy` | [govoplan-tenancy](https://git.add-ideas.de/GovOPlaN/govoplan-tenancy) |
+144 -117
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:75469c08423f21fc61ec7c15 --> <!-- codex-wiki-sync:c2e1a6a893c11e2057885bf5 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`. > Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -7,157 +7,184 @@
--- ---
# Scaling And Multi-Host Deployment # Scaling And Multi-Host Deployment
## What Is Implemented ## Implemented Contract
The default installer now supports explicit same-host horizontal scaling: GovOPlaN now supports a stateless application tier backed by logically shared
state services. The runtime roles are independently replaceable API, WebUI,
worker, and scheduler processes. Every replica in one installation must use the
same immutable release composition and the same:
- one HAProxy container accepts the published HTTP endpoint; - `GOVOPLAN_INSTALLATION_ID`;
- one or more WebUI containers serve assets and proxy API requests; - PostgreSQL database;
- one or more API containers serve normal domain traffic; - Redis broker and coordination service;
- one or more Celery workers consume shared Redis queues; - `MASTER_KEY_B64` and deployment secret references;
- one migration job runs before replacement; - enabled-module graph;
- exactly one Celery scheduler runs; - S3-compatible object-storage namespace.
- PostgreSQL, Redis, and file storage are shared by every runtime replica.
HAProxy discovers Compose replicas through Docker's internal DNS and performs The application tier must not use node-local durable business data in a
health-checked round-robin WebUI balancing and least-connection API balancing. multi-host deployment. Files owns managed file metadata while Core provides the
It does not mount the Docker socket. Replica counts live in storage-backend contract. Campaign build artifacts are stored under opaque
`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on object keys and workers read those objects from the shared backend. Temporary
the desired topology. build and materialization directories may remain node-local because they are
discardable.
This is not multi-host scheduling. Docker Compose's bridge network belongs to Core validates three explicit state profiles:
one Docker Engine. Adding a second machine requires an orchestrator or
deployment manager that can place equivalent role definitions on multiple
hosts.
## Recommended Topologies | Profile | Supported shape | Storage rule |
| --- | --- | --- |
| `local` | One API and one worker process for development | Local filesystem permitted. |
| `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. |
| `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. |
### One Host `shared` also requires a stable installation identifier. Module package
mutation is blocked in this profile: build and verify a new immutable release,
then roll the complete cluster to it.
Use the generated Compose bundle: ## Same-Host Compose
The generated Compose bundle provides:
```text ```text
client client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas
-> external TLS proxy, when required
-> managed HAProxy
-> WebUI replica(s)
-> managed HAProxy API listener
-> API replica(s)
API/worker/scheduler API/worker/scheduler -> PostgreSQL
-> PostgreSQL
-> Redis -> Redis
-> local volume, managed Garage, or external S3 -> local volume, managed Garage, or external S3
``` ```
This improves concurrency and permits rolling process replacement, but the host HAProxy discovers Compose replicas through Docker DNS and performs health-aware
and every managed stateful component remain single failure domains. Two API balancing without mounting the Docker socket. This improves concurrency and
containers on one failed host do not provide host-level availability. permits process replacement, but the Docker host and installer-managed stateful
services remain single failure domains. Generated Compose therefore declares
the `host-shared` state profile even when its shared storage happens to be an
external S3 service. Its API backend checks `/health/ready`, so drain or
coordination loss removes a replica from rotation. Container, load-balancer,
and Kubernetes probes send the configured public host explicitly, keeping
readiness compatible with strict trusted-host validation.
### Multiple Hosts Managed Garage is a convenient single-node S3-compatible service. It is not a
multi-host storage cluster. Use an independently operated Garage cluster or
another S3-compatible service for the `shared` profile.
Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not ## Kubernetes Export
extend the Compose installer into a proprietary scheduler. The target topology
is:
```text The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL,
public TLS ingress/load balancer Redis, and S3 are all external:
-> WebUI replicas on at least two nodes
-> internal API service/load balancer
-> API replicas on at least two nodes
queue-specific worker pools on worker nodes ```sh
one fenced scheduler python tools/deployment/govoplan-deploy.py render-kubernetes \
one fenced migration/deployment job --directory /srv/govoplan/default \
--namespace govoplan \
shared PostgreSQL --secret-name govoplan-runtime \
shared Redis --tls-secret-name govoplan-tls \
multi-node Garage or external S3 --ingress-class-name nginx \
shared secret/config provider --output /srv/govoplan/default/kubernetes.json
central logs, metrics, and health alerts
``` ```
All API and worker nodes must receive the same immutable software composition, The export contains a Namespace, tokenless ServiceAccount, non-secret
`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption
storage binding. Node-local file storage is not valid in this topology. budgets, Ingress, and a release-specific migration Job. It deliberately emits
no Secret values, persistent volume, PostgreSQL, Redis, or object-store
deployment. Export is rejected unless both release images use immutable
`image@sha256:...` references.
## Adding Capacity Create the named Secret through the cluster's secret-management path. The
command prints the exact required key contract. Review the generated
`FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network
before production use.
### API And WebUI The generated containers run as non-root with a read-only root filesystem and
an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database
migration heads before starting. The API exposes `/health/ready`, which fails
while that API node is draining or cannot prove its runtime-coordination
heartbeat.
Increase API replicas for measured request concurrency, CPU saturation, or ## Runtime Coordination
latency after checking database load. Increase WebUI replicas for static asset
and proxy capacity. No sticky session should be required because durable
sessions and throttling use shared services, but this must remain covered by
multi-replica integration tests.
Every added API process also adds database connections. Set the application Each API and worker incarnation registers in PostgreSQL with its role, software
pool size and the total replica ceiling against PostgreSQL's connection budget; version, module-composition hash, queue set, and heartbeat. Ops shows active,
adding replicas can otherwise reduce throughput. draining, stopped, and stale nodes and compares active counts with configured
replica expectations.
### Workers An operator may request or cancel drain from Ops:
Workers are not placed behind a load balancer. They compete for jobs on shared - API readiness becomes unavailable on the next heartbeat so the load balancer
Redis queues. Add workers by queue and cap each pool according to the external stops assigning new requests.
system it calls. SMTP, IMAP, directory, and connector jobs often reach provider - A worker stops consuming its configured queues and may finish work already
rate limits before CPU limits. claimed by that process.
- A stale process incarnation cannot overwrite a replacement incarnation's
heartbeat.
- A coordination outage removes API readiness and cancels worker consumers;
the existing incarnation must heartbeat successfully before either resumes.
Use distinct pools when load warrants it: Singleton work uses PostgreSQL-backed leases with monotonically increasing
fencing tokens. The generated scheduler runs Celery beat through
`govoplan_core.commands.fenced_run`; loss of its lease terminates the child and
returns a distinct failure code. A fenced business operation must validate the
same lease token immediately before committing its effect.
- mail send and Sent-folder append; ## Release Ordering
- notifications;
- calendar and connector synchronization;
- dataflow/workflow execution;
- reporting and export;
- platform events and default work.
Before stopping a worker, mark it draining, stop new claims, and let or safely Use this order for every multi-replica rollout:
requeue active jobs. Worker registration and drain control remain part of the
larger platform scale-out story.
### Stateful Services 1. Verify immutable image identities, module composition, external state
reachability, backup evidence, and the generated plan.
2. Drain application replicas when the migration compatibility declaration
requires it.
3. Run the release-specific migration Job exactly once. PostgreSQL advisory
locking serializes all Core and module migration tasks across competing
deployment jobs.
4. Let runtime init containers run `wait_for_database`. They wait for exact
configured Alembic heads and never mutate schema.
5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
6. Verify runtime composition, expected replica counts, queue consumers,
object-storage round trips, and recovery status in Ops.
- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover Applying the complete generated manifest is fail-closed: runtime pods remain in
design appropriate to the service level. their init phase until the migration Job reaches the expected heads. A second
- Redis needs persistence and, for high availability, a supported failover release may be submitted concurrently, but advisory locking prevents concurrent
topology. Queue loss is not equivalent to a harmless cache loss. schema mutation and each release has a distinct migration Job name.
- Managed Garage from the Compose installer is single-node. For redundant
storage, deploy a multi-node Garage cluster separately and select external
`s3`, or use another compatible object store.
- The scheduler stays at one replica until distributed leader election or a
fenced lease is implemented.
- Migrations and module lifecycle mutations remain one-at-a-time operations.
## Promotion Path ## Storage Trust Boundary
1. Measure concurrent users, request latency, database query time, connection Installer-managed Garage uses its exact generated endpoint. An arbitrary
use, queue age, worker saturation, storage latency, and external throttling. external S3 endpoint is accepted only when the deployment explicitly sets
2. Move files to managed Garage or external S3 before introducing independent `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS
runtime hosts. origin without embedded credentials, query, fragment, or path. This is an
3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy operator trust declaration, not a user-controlled connector bypass. Operators
their reviewed HA operators. remain responsible for DNS, certificate, network-egress, bucket-policy,
4. Publish immutable API/WebUI images and one versioned configuration/secret versioning, and lifecycle controls.
contract.
5. Translate the generated role commands, environment allowlists, health
checks, and singleton constraints into the chosen orchestrator.
6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
7. Add queue-specific workers with fixed upper bounds.
8. Prove replica loss, rolling replacement, job redelivery, session continuity,
migration exclusion, backup restore, and storage-node loss before claiming
high availability.
## Remaining Platform Work ## Capacity
The one-host installer does not yet provide: - Scale API replicas only within the PostgreSQL connection budget.
- Scale workers by queue, with upper bounds based on external provider limits.
- Keep one fenced scheduler rather than load-balancing schedulers.
- Increase WebUI replicas for asset/proxy capacity.
- Measure request latency, database query time and locks, active connections,
queue age, retry rate, storage latency, and provider throttling before adding
replicas.
- a Kubernetes, Nomad, or Swarm deployment export/profile; Workers compete for Redis-backed work and are not placed behind a load balancer.
- distributed deployment locks and fenced scheduler leadership; SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often
- worker registration, composition-skew reporting, and drain controls; hit external-system limits before host CPU is exhausted.
- managed PostgreSQL, Redis, or multi-node Garage high availability;
## What This Does Not Claim
The implemented contract provides stateless runtime placement, shared artifact
access, node visibility, drain controls, migration serialization, and scheduler
fencing. It does not by itself provide:
- a highly available PostgreSQL, Redis, or object-store deployment;
- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
- autoscaling policy; - autoscaling policy;
- managed TLS certificate issuance and renewal; - central logs, metrics, traces, or alert routing;
- measured multi-replica and failover integration evidence. - managed ingress certificates;
- automatic reconciliation of every possible module side effect;
- a service-level availability guarantee.
These are separate production capabilities. The local load balancer and Those are deployment and module-adoption requirements. Before claiming high
replica model provide the contract they should implement, but they do not by availability, drill replica loss, rolling replacement, session continuity, job
themselves make a cluster highly available. redelivery, scheduler failover, migration exclusion, object-store outage, and a
coordinated database/object/key restore. Recovery rules and evidence are
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:76e4d772814a0842238fd114 --> <!-- codex-wiki-sync:978cfaf4b9d5c9243c07ec19 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md`. > Mirrored from `/mnt/DATA/git/govoplan/docs/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md`.
> Origin: `repository`. > Origin: `repository`.
@@ -146,6 +146,17 @@ The canonical backlog item is
## Implementation slices ## Implementation slices
Implementation status as of the current source tree:
- Slice 6 has a working application-tier foundation: state profiles, shared
object storage, runtime node registration/heartbeats/drain, fenced scheduler,
migration serialization, exact-head startup waiting, Ops visibility, and a
Kubernetes export. Production acceptance still requires topology-specific
failover and restore drills.
- The recovery foundation for slices 4 and 5 is implemented as a Core recovery
ledger and deployment operation journal. Automatic database backup and broad
adoption by module-owned external effects remain open work.
1. **Reproducible Core-only distribution.** Publish pinned multi-architecture 1. **Reproducible Core-only distribution.** Publish pinned multi-architecture
images, signed distribution manifest, Core-only Compose profile, bootstrap images, signed distribution manifest, Core-only Compose profile, bootstrap
preflight, generated secrets, readiness, and idempotent rerun/repair. preflight, generated secrets, readiness, and idempotent rerun/repair.
@@ -162,9 +173,11 @@ The canonical backlog item is
5. **Safe module update.** Add drain/maintenance coordination, backup gate, 5. **Safe module update.** Add drain/maintenance coordination, backup gate,
migration compatibility window, reconnectable progress, health verification, migration compatibility window, reconnectable progress, health verification,
retry/recovery, and update notification. retry/recovery, and update notification.
6. **Stateless replica profile.** Remove remaining local-runtime assumptions, 6. **Stateless replica profile.** Continue module adoption and operational
expose role-specific commands/images, implement worker registration/drain, proof for the implemented role commands, shared-state validation, runtime
and prove multiple API and worker replicas against shared dependencies. registration/drain, fenced scheduler, and Kubernetes application-tier
export. Prove multiple API and worker replicas against the target shared
dependencies.
7. **Configuration revision model.** Define provider export/import schemas, 7. **Configuration revision model.** Define provider export/import schemas,
canonical serialization, secret references, validation/diff, immutable canonical serialization, secret references, validation/diff, immutable
revision storage, audit, apply, and undo-as-new-revision. revision storage, audit, apply, and undo-as-new-revision.