GovOPlaN/govoplan
2
0
Fork 0
Code Issues 18 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Sync wiki from project files

zemion
2026-08-01 14:56:45 +02:00
parent 5d87768c10
commit 9070e4e7c8
9 changed files with 467 additions and 158 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Codex-Project-Index.md
+1
@@ -18,6 +18,7 @@ This page is generated from repository and product-directory project files.
- [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md`
- [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md`
- [Repo-docs-PLATFORM-CONTROL-PLANE](Repo-docs-PLATFORM-CONTROL-PLANE) - `/mnt/DATA/git/govoplan/docs/PLATFORM_CONTROL_PLANE.md`
- [Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES](Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES) - `/mnt/DATA/git/govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md`
- [Repo-docs-REFERENCE-JOURNEY-PROGRAM](Repo-docs-REFERENCE-JOURNEY-PROGRAM) - `/mnt/DATA/git/govoplan/docs/REFERENCE_JOURNEY_PROGRAM.md`
- [Repo-docs-RELEASE-CONSOLE](Repo-docs-RELEASE-CONSOLE) - `/mnt/DATA/git/govoplan/docs/RELEASE_CONSOLE.md`
- [Repo-docs-REPOSITORY-INDEX](Repo-docs-REPOSITORY-INDEX) - `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`
Repo-README.md
+8 -1
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:e8f6d622ddc3176e628d09c3 -->
<!-- codex-wiki-sync:28496fdfb2db07160032de9f -->
> Mirrored from `/mnt/DATA/git/govoplan/README.md`.
> Origin: `repository`.
@@ -162,6 +162,9 @@ The current executable slice and remaining production gates are documented in
[Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
Same-host replica balancing and the multi-host promotion boundary are documented
in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
The recovery state machine, migration rollback boundary, and required restore
drills are documented in
[Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md).
## Configuration
@@ -204,6 +207,10 @@ reconfiguration semantics, and safe Web update boundary are defined in
[Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
The concrete replica, worker-node, load-balancer, and shared-state topology is
defined in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
Durable deployment journals, Core recovery evidence, and the distinction
between pre-migration configuration restore and post-migration forward recovery
are defined in
[Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md).
The first Campaign-centric capability and infrastructure fit assessment is in
`docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. Its rerun tooling can collect and
verify a bounded installed composition; target, provider and production claims
Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md
+15 -9
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:4765944ec28fc1daab87fa3c -->
<!-- codex-wiki-sync:e4b2c1763e740cebf64cba2a -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`.
> Origin: `repository`.
@@ -483,10 +483,11 @@ which modules a running service activated, migrations, configuration, health,
or reference-journey behavior. Runtime activation therefore remains an explicit
unchecked boundary.
### Target, provider and production proof boundary
### Reference-readiness, provider and production proof boundary
Installed evidence cannot approve a target environment, an external provider,
or production use. These scopes use a separate, expiring
Installed evidence cannot establish target acceptance, accessibility, privacy,
security, operations, recovery, an external provider, or production use. These
scopes use a separate, expiring
[`capability-fit-boundary-evidence.schema.json`](capability-fit-boundary-evidence.schema.json)
bundle. The bundle is bound to the assessment ID, assessment release and exact
installed-evidence SHA-256 digest. It contains only opaque subject/control/result
@@ -495,8 +496,9 @@ IDs and content hashes, not endpoints, credentials, people or raw result files.
Boundary evidence is accepted only when at least one Ed25519 signature validates
against a separately provisioned
[`capability-fit-proof-authority-keyring.schema.json`](capability-fit-proof-authority-keyring.schema.json).
Each authority key explicitly lists the scopes it may attest. Target and provider
claims use `passed` or `failed`; production claims use `approved` or `rejected`.
Each authority key explicitly lists the scopes it may attest. Target,
accessibility, privacy, security, operations, recovery, and provider claims use
`passed` or `failed`; production claims use `approved` or `rejected`.
One claim per scope, unique control/artifact IDs, `issued_at < expires_at`, current
validity and exact digest binding are mandatory. Any schema, binding, time,
signature or authority blocker leaves every supplied boundary claim unchecked;
@@ -569,6 +571,10 @@ Target-environment and production-approval claims must use the assessment's
operator to supply a bounded opaque expected subject with
`--expected-external-provider-subject`; without it, such a claim remains
unchecked and blocks. Expected and observed IDs are retained in proof scope.
Accessibility, privacy, security, operations, and recovery claims use the same
deployment subject. The report emits a cumulative `reference_readiness` verdict
only when all six required scopes are checked and positive. This verdict remains
separate from production approval and from provider-specific acceptance.
Both authority keyrings are governance trust roots. Installer receipt keys use
the strict
@@ -590,9 +596,9 @@ cannot relabel it as current. Live admission must omit it and use the actual
current time.
No boundary bundle or production authority has been supplied for this current
assessment. Target environment, provider and production proof therefore remain
explicitly unchecked rather than inferred from the local GreenMail journey,
source tests or signed release metadata.
assessment. Reference-readiness, provider, and production proof therefore
remain explicitly unchecked rather than inferred from the local GreenMail
journey, source tests, or signed release metadata.
## Evidence used in this slice
Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md
+80 -14
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:04ef6f09ecabe28900177d68 -->
<!-- codex-wiki-sync:1e4a0a14da7f197bb96cac2c -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md`.
> Origin: `repository`.
@@ -41,9 +41,15 @@ It currently supports:
- generated secrets stored in a private `0600` file;
- service-specific environment allowlists so infrastructure containers do not
receive unrelated application credentials;
- plan, render, doctor, status, and apply commands;
- plan, render, doctor, status, apply, Kubernetes export, operation history,
and bounded recovery commands;
- an installation lock, migration-before-start ordering, readiness polling,
and an applied-state receipt;
- a durable hash-chained deployment journal captured before runtime mutation;
- PostgreSQL advisory serialization for Core and module migrations;
- runtime initialization that waits for exact configured migration heads
without mutating schema;
- runtime node registration, heartbeats, drain state, and a fenced scheduler;
- idempotent reconfiguration that preserves generated secrets;
- a keyed environment fingerprint that detects private binding changes without
writing secret values to plans or receipts;
@@ -94,6 +100,9 @@ The private installation directory contains:
| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
| `plan.json` | Latest desired-state diff and readiness findings |
| `receipt.json` | Last successfully applied immutable identities |
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
| `.deployment.lock` | Same-host operation exclusion |
The specification contract is
@@ -208,11 +217,13 @@ presenting a container as a complete mail service.
installations. `garage` provisions Garage 2.3 in its supported single-node
bootstrap mode, generates a private application key and bucket, and connects
the Files S3 backend to the exact installer-owned internal endpoint. The
deployment-only trust marker cannot authorize another S3 host; arbitrary
external SDK endpoints remain fail-closed until peer pinning is implemented.
managed trust marker cannot authorize another S3 host.
Garage metadata and object data use separate persistent volumes. `s3` requires
an external endpoint, region, access key, secret key, and bucket values.
Self-hosted external S3 endpoints must use HTTPS.
Self-hosted external S3 endpoints must be clean HTTPS origins. The generated
runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that
operator-selected endpoint. The trust flag is not accepted for local storage
and cannot be combined with installer-managed Garage trust.
Local storage must be included in backup and restore drills. Horizontal API or
worker scale-out requires shared/object storage. The managed Garage profile is
@@ -240,9 +251,10 @@ Replica counts are desired state:
```
Workers are queue consumers, so they are scaled through Redis rather than put
behind an HTTP load balancer. The migration runner and Celery scheduler remain
singletons. Multiple API replicas are rejected when Redis is disabled because
distributed throttling and queued work cannot then be shared correctly.
behind an HTTP load balancer. Migrations are serialized with a deployment-wide
PostgreSQL advisory lock. The Celery scheduler is run under a renewable,
fencing-token lease. Multiple API replicas are rejected when Redis is disabled
because distributed throttling and queued work cannot then be shared correctly.
This is same-host scaling. Docker Compose uses a bridge network and does not
place containers on another machine. See
@@ -265,13 +277,66 @@ applied state. `plan` compares their canonical hashes and service sets.
- Managed-to-external transitions require the new endpoint in the same
operation.
- Migrations run as a one-shot service before API/worker replacement.
- API, worker, and scheduler start commands wait for exact configured migration
heads; only the migration command is permitted to change schema.
- API and worker replicas register their software/module composition and
heartbeat in PostgreSQL. Ops can request and cancel a node drain.
- The first upgrade from a direct WebUI host port stops that legacy WebUI
container immediately before HAProxy claims the same endpoint.
- Health must recover before a new receipt is committed.
- Health must recover before a new receipt and applied-state snapshot are
committed.
This is sufficient for one-host reconciliation. Production updates additionally
need backup/restore gates, maintenance/drain state, database compatibility
windows, image signature verification, and rollback/forward-recovery policy.
Every apply operation is journalled before image pulls or runtime mutation. A
failure before migration may restore a verified previous bundle. Once migration
starts, recovery is forward-only unless an independently verified database
backup is restored. See
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
Production updates still need an operator-provided database backup/restore
gate, database compatibility declaration, image signature verification, and
deployment-specific drain policy. The deployment journal proves its own
actions; it does not manufacture backup evidence.
## Stateless Kubernetes Runtime
`render-kubernetes` exports the application tier for a standard orchestrator.
It requires external PostgreSQL, Redis, and S3 and emits no stateful service or
secret value:
```sh
python tools/deployment/govoplan-deploy.py render-kubernetes \
--directory /srv/govoplan/default \
--namespace govoplan \
--secret-name govoplan-runtime
```
The output includes a release-specific migration Job, database-head wait init
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
the named Secret through the cluster's secret manager and review ingress proxy
CIDRs before deployment. Detailed rollout and scaling rules live in
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
## Recovery Commands
List durable deployment operations:
```sh
python tools/deployment/govoplan-deploy.py operations \
--directory /srv/govoplan/default
```
Recover a selected failed operation after reviewing its stage evidence:
```sh
python tools/deployment/govoplan-deploy.py recover \
--directory /srv/govoplan/default \
--operation-id <operation-id>
```
The command reports whether it restored the pre-migration applied bundle,
requires forward recovery, or needs manual intervention. Add `--apply` only
after that decision has been reviewed.
## Web Update Boundary
@@ -324,5 +389,6 @@ The tests cover profile restrictions, secret persistence, external endpoint
requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
discovery configuration, Compose service selection, secret non-disclosure,
service-specific environment isolation, private file modes, external endpoint
preflight, first-plan generation, apply ordering, and receipt-based
idempotency.
preflight, first-plan generation, apply ordering, receipt idempotency,
hash-chained recovery journals, migration recovery boundaries, and stateless
Kubernetes rendering.
Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md
+41 -10
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:46024a5062a8085c6ffc152f -->
<!-- codex-wiki-sync:7b1befe1315b504d421507a4 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md`.
> Origin: `repository`.
@@ -90,11 +90,12 @@ were proven now have independent persistent owners:
| External providers | Core validates provider objects/field groups, operations, integration maturity, source authority, bounded reads, freshness/health, idempotency, conflicts, outcome-unknown handling, evidence, correction, reconciliation, outage, classification, purpose, retention, and secret handling. Addresses/CardDAV, Files remote storage, Mail SMTP/IMAP, Calendar CalDAV/ICS/Graph/EWS, and Connectors tabular/sanctions providers declare the contract and tenant-bounded secret-free runtime state. | Registry validation rejects any declared external provider without a sanitized state provider. Future adapters must cross the same gate before activation. |
| Institutional context | Core provides versioned temporal, actor/representation, institution/unit/function/task/mandate/jurisdiction/service/case/party/work-item/workflow/approval/decision/record, legal-basis, evidence, information-governance, external-source, presentation, and geographic references. Events, automation actions, audit records, and the transactional Audit outbox preserve the envelope. | Owning modules must progressively require the relevant subset for consequential operations. |
| Semantic provider contracts | Provider-neutral DTOs and protocols cover Mandate resolution, versioned Service definitions, procedure Parties/representation, and formal Decisions. `govoplan-mandates`, `govoplan-services`, `govoplan-parties`, and `govoplan-decisions` now persist immutable revisions behind those contracts with tenant isolation, bounded reads, replay safety, OCC, migrations, uninstall guards, permissions, APIs, capability documentation, and recovery documentation. | The owners are deliberately headless. Procedure-specific UI remains with consuming modules. |
| Formal-outcome proof | Committee persists bodies, meetings, agenda items, votes, minutes, lifecycle events, and an optional protected local Decision projection. It resolves effective Mandate authority and records reconstructable formal Decisions through `decisions.registry` when installed. OCC, replay safety, tenant isolation, bounded reads, and closure guards cover the aggregate. | Committee remains API-first and does not yet provide its later deliberation WebUI or specialized secret/external ballot adapters. |
| Service-to-case proof | Services owns the persistent exact definitions consumed by Portal discovery and Cases intake. Portal exposes a tenant-scoped API and compact service-directory WebUI whose audiences are derived from trusted principal/function state. Cases persists catalogs, immutable revisions, parties, assignments, evidence/decision/record references, deadlines, lifecycle events, exact Service versions, OCC, replay safety, tenant isolation, API permissions, ACL filtering, uninstall guards, tenant summary state, and list/detail/history/timeline WebUI. | Specialized form/case/workflow launchers and procedure panels remain with their owners. Cases still needs finer case-sharing policy. Neither gap changes Service ownership. |
| Formal-outcome proof | Committee persists bodies, meetings, agenda items, votes, minutes, lifecycle events, and an optional protected local Decision projection. It resolves effective Mandate authority and records reconstructable formal Decisions through `decisions.registry` when installed. OCC, replay safety, tenant isolation, bounded reads, closure guards, and a full-height body/meeting/agenda/vote/minutes WebUI cover the aggregate. Provider-bound external or secret ballots use dynamic adapter capabilities; Committee validates and retains only aggregate counts, receipt/hash, and evidence, and rejects generic manual closure. | Concrete ballot-provider packages remain integration work because their protocol, custody, credentials, and operator evidence depend on the selected provider. This does not leave a Committee-owned ballot contract unspecified. |
| Service-to-case proof | Services owns the persistent exact definitions consumed by Portal discovery and Cases intake. Portal exposes a tenant-scoped API and service-directory WebUI whose audiences are derived from trusted principal/function state. Launch re-fetches and re-evaluates the exact Service revision. URL launch is validated; Cases and Workflow Engine provide tenant-bound, replay-safe owner launchers. Cases persists catalogs, immutable revisions, parties, assignments, evidence/decision/record references, deadlines, lifecycle events, exact Service versions, OCC, replay safety, tenant isolation, explicit and assignment-derived object ACLs, uninstall guards, tenant summary state, and list/detail/history/timeline/share WebUI. | Form-bound launch remains unavailable until Forms and Forms Runtime move beyond scaffolds to immutable definitions, validation, submission persistence, and a real owner launcher. Portal fails this case closed and explains the missing capability. |
| Procedure-party proof | Parties persists effective procedure roles, frozen contact snapshots, and representation powers. Existing powers cannot disappear or be silently rewritten; explicit OCC-guarded revocation is required. Cases resolves the provider capability and excludes expired/revoked authority from downstream delivery. | Procedure modules still decide which contextual fields and actions to present. |
| Integrated institutional journey | The executable `product.service-to-decision` fixture uses real SQL-backed Services, Cases, Parties, Mandates, Committee, and Decisions providers. It carries one exact Service version through persisted Case intake, representation and frozen delivery authority, effective Mandate resolution, a body/meeting/agendum/vote/minute sequence, a persisted formal Decision, confirmed Postbox effect, Audit/record evidence, remedy/review, and protected reconstruction. | This is architecture and composition evidence. Signed, release-bound target accessibility, privacy, security, operator, delivery-provider, and recovery-drill evidence is still required before the package may claim `reference_ready`. |
| Governed data catalogue | Datasources stores typed governance metadata, exposes bounded tenant-scoped filters and update APIs/UI, carries governance through staging, and snapshots it into immutable materializations. | Rich dependency/impact traversal and policy-specific field visibility can grow on the catalogue contract without moving connector or transformation ownership. |
| Governed data catalogue | Datasources stores typed governance metadata, exposes bounded tenant-scoped filters and update APIs/UI, carries governance through staging, and snapshots it into immutable materializations. Reporting now persists immutable dataset, semantic-model, report, quality-plan, saved-view, and schedule revisions; executes typed semantic queries with quality gates, access checks, replay, pivoting, export/import assessment, and provenance; and exposes the governed analytical WebUI. | Rich dependency/impact traversal, additional expression functions, and policy-specific field visibility can grow on the established contracts without moving connector, transformation, or source ownership. |
| Portfolio and change governance | Projects now persists tenant-safe, immutable portfolio/project/milestone revisions with OCC, replay, lifecycle rules, restricted memberships, Search ACL indexing, outcomes, benefits, dependencies, capacity assumptions, change impact, and institutional references. Its WebUI exposes the planning catalogue and core planning fields. | Advanced planning structures already accepted by the API can receive deeper specialized editors without creating a second Policy, Reporting, Resources, or Goals owner. |
| Product/package governance | Signed configuration packages distinguish reference, product, sector, deployment, and integration classes; preserve parent/evidence provenance; prevent derived packages from loosening constraints; and preflight provider authority, maturity, exact binding, health, freshness, and recovery expectations. Executable product manifests now exist for governed communication and governed data/assurance and are checked in the module matrix. | Both artifacts deliberately remain product-class until target, accessibility, privacy, security, operations, and recovery evidence justifies reference readiness. |
| Projection and release | Platform metadata, signed module catalogs, release synthesis, Ops, and role-aware Docs retain and display architecture/provider declarations. Module-owned state providers add bounded configured/active, authority, health, freshness, conflict, recovery, and observation state; ordinary-user Docs omits binding detail. Static checks validate evidence paths, and the WebUI build verifies consuming types. | Runtime-state adoption and broader portfolio presentation follow truthful provider declaration rollout. |
@@ -424,17 +425,17 @@ submodule, configuration fragment, package, or profile.
- `product.service-to-decision` proves both through a portable administrative
service composition.
### 3. Complete governed data and assurance - architecture complete
### 3. Complete governed data, portfolio, and assurance - vertical slices complete
- Datasources carries typed governance through staging and immutable
materializations, with bounded catalogue filters and dependency references.
- Reporting owns semantic presentation/provenance contracts without taking
source or transformation ownership; feature depth remains on its module
backlog.
- Reporting owns immutable semantic definitions, safe execution, quality gates,
provenance, schedules, saved views, pivoting, and export/import assessment
without taking source or transformation ownership.
- Risk Compliance persists the horizontal obligation/risk/control/evidence/
finding/measure graph and projects sanctions runs idempotently.
- Projects declares portfolio/outcome ownership as a consuming domain without
becoming a second policy or reporting engine.
- Projects persists portfolio/outcome/change-governance revisions as a
consuming domain without becoming a second policy or reporting engine.
### 4. Package repeatable public-sector outcomes - complete at product maturity
@@ -446,6 +447,36 @@ submodule, configuration fragment, package, or profile.
recovery, accessibility, privacy, security, and operator evidence. This is a
maturity gate, not missing architecture implementation.
## What remains after the executable architecture slice
The remaining work is not another Core or cross-module architecture rewrite.
It falls into three explicitly different categories:
1. **Forms and Forms Runtime vertical:** both repositories remain scaffolds.
Full service entry requires immutable form-definition lookup, server-side
validation, draft/submission persistence, status/evidence history,
attachments/signatures/handoff references, replay and OCC, APIs, accessible
WebUI, migrations, recovery, and `forms_runtime.service_launcher`. Portal
already advertises this absence as an availability blocker. Implementing a
placeholder launcher before those controls would violate the accepted
contract.
2. **Concrete provider packages:** the Committee ballot adapter contract is
complete, but a real secret/electronic ballot provider requires a selected
protocol and product decisions for voter eligibility, custody, secrecy,
recount, challenge, retention, and operational assurance. Equivalent future
adapters must satisfy the declared provider and recovery gates.
3. **Target-produced maturity evidence:** `reference_ready`, `supported`, and
`lts` cannot be generated from source code. An exact release and deployment
must produce signed, expiring accessibility, privacy, security, operator,
provider, backup/restore, rollback, and recovery-drill evidence. The verifier
and schemas are implemented; the actual claims require those real runs.
Everything else described as architecture in this document now has a
repository owner, versioned contract, bounded implementation, migration and
recovery boundary where state exists, documentation, and executable evidence.
Further work in those modules is product breadth, UX depth, provider adoption,
and evidence renewal.
## Delivery tracking
The completed cross-repository architecture epic is
Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES.md
+154
@@ -0,0 +1,154 @@
<!-- codex-wiki-sync:e432d0c3b51f11d9f6c79468 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md`.
> Origin: `repository`.
> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
---
# Recovery And Rollback Guarantees
## Principle
GovOPlaN must prove recovery claims with durable state recorded before and
after side effects. A failed operation is not automatically rolled back merely
because the previous application image still exists. Database schema and
external effects may make release rollback unsafe.
Core therefore distinguishes five recovery modes:
| Mode | Meaning |
| --- | --- |
| `atomic` | One database transaction either commits or rolls back. No external effect is claimed. |
| `compensation` | Durable evidence identifies explicit inverse actions for completed effects. |
| `snapshot_restore` | A separately verified backup reference and restore procedure exist. |
| `forward_recovery` | Repair or resume the current version; reverting code/configuration is not claimed safe. |
| `irreversible` | No automated recovery is claimed and an approval reference is mandatory. |
An operation plan must include verification steps. Compensation requires named
compensation steps, snapshot restore requires a verified backup reference,
forward recovery requires repair steps, and irreversible work requires explicit
approval.
## Core Recovery Ledger
Core stores recovery operations and append-only, hash-chained checkpoints in
PostgreSQL. The contract provides:
- installation/module/resource identity;
- an idempotency key bound to a canonical request hash;
- recovery mode, preconditions, verification steps, and references;
- optional runtime lease holder and fencing token;
- explicit planned, prepared, running, recovery-required, recovering,
succeeded, recovered, failed, outcome-unknown, and manual-intervention states;
- an evidence-chain head and sequence count;
- rejection of plaintext secrets in metadata or evidence.
Preparation cannot succeed without durable precondition evidence. A non-atomic
operation cannot hide a partial effect by transitioning directly from running
to failed. Success and recovery require explicit verification evidence with at
least one check. The ledger verifies its hash chain before evidence is trusted.
This is a platform contract, not an assertion that every existing module
operation has adopted it. Module operations with external or multi-resource
effects must be migrated to the ledger before claiming these guarantees.
## Deployment Journal
Every `govoplan-deploy apply` begins an operation journal before it pulls images
or mutates runtime state. The private installation directory records:
```text
operations/<operation-id>/operation.json
operations/<operation-id>/before/
applied-state/
```
Each stage is hash-chained. The previous applied bundle is copied with per-file
SHA-256 evidence. Applied state is replaced atomically after health verification;
an interrupted replacement restores its previous directory.
New journals also bind the complete desired deployment plan, snapshot
availability, failure summary, recovery mode, and terminal status to the
evidence chain. Recovery verifies every snapshot entry and checksum before it
changes any live bundle file, then replaces each live file atomically. A crash
between file replacements is recoverable by rerunning the same idempotent
recovery command under the deployment lock.
Inspect operations:
```sh
python tools/deployment/govoplan-deploy.py operations \
--directory /srv/govoplan/default
```
Recover the latest failed operation, or provide its identifier:
```sh
python tools/deployment/govoplan-deploy.py recover \
--directory /srv/govoplan/default \
--operation-id 20260801T120000Z-1234abcd
```
Add `--apply` only after reviewing the reported action.
## Migration Boundary
Before database migration starts, a failed deployment with a verified prior
applied snapshot may restore its prior release/configuration bundle and
reconcile that desired state.
As soon as migration starts, the journal permanently changes to
`forward_recovery`. It will not restore old application configuration because
old code may not understand the new schema. Recovery then means one of:
1. fix and re-run the current release;
2. deploy a newer compatible repair release;
3. restore a separately verified, coordinated database/object/key backup and
then deploy the matching release.
The deployment tool does not create or validate that database backup. A
`backup-required` annotation on the Kubernetes migration Job is an operator
gate, not backup evidence. Production automation must provide a backup hook or
external backup controller whose artifact, timestamp, scope, encryption key,
and restore test can be referenced from the recovery record.
## Scaled Nodes
Recovery actions must be safe across replicas:
- drain affected API and worker nodes before incompatible changes;
- use the deployment-wide PostgreSQL advisory lock for schema migration;
- use distributed leases and fencing tokens for singleton or externally
visible effects;
- use idempotency keys for retried commands and jobs;
- retain shared object keys and database references until deletion succeeds;
- classify uncertain external outcomes instead of retrying blindly;
- verify the exact software/module composition after replacement.
Campaign generated-message objects now follow this model: object writes are
compensated when a build fails before database commit, workers verify stored
size and digest before delivery, and retention keeps the database reference
when storage deletion fails. A hard process loss between object creation and
database commit can still leave an orphan object; an inventory reconciler is a
separate operational slice and must use the build-specific object prefix.
## Required Drills
Record evidence for at least these scenarios before production acceptance:
1. Kill an API replica and verify traffic continues without session loss.
2. Drain and replace a worker while work is queued and while one job is active.
3. Start two migration jobs and verify only one mutates schema.
4. Kill the fenced scheduler and verify one replacement acquires a higher
fencing token.
5. Fail deployment before migration and restore the prior applied bundle.
6. Fail deployment after migration and verify old configuration is not
restored.
7. Restore PostgreSQL, object storage, and encryption keys to one coordinated
recovery point and verify representative object hashes.
8. Interrupt object storage during Campaign build and retention and verify
compensation/reference-preservation behavior.
9. Tamper with a deployment or Core recovery checkpoint and verify chain
validation rejects it.
No runbook, status badge, or green health endpoint substitutes for a dated,
repeatable restore drill against the actual deployment topology.
Repo-docs-REPOSITORY-INDEX.md
+5 -1
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:074fbacd975d50c1718b7e70 -->
<!-- codex-wiki-sync:3a6877289005519a3e438aaf -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`.
> Origin: `repository`.
@@ -38,6 +38,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard) |
| `govoplan-dataflow` | `platform` | `../govoplan-dataflow` | [govoplan-dataflow](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow) |
| `govoplan-datasources` | `platform` | `../govoplan-datasources` | [govoplan-datasources](https://git.add-ideas.de/GovOPlaN/govoplan-datasources) |
| `govoplan-decisions` | `domain` | `../govoplan-decisions` | [govoplan-decisions](https://git.add-ideas.de/GovOPlaN/govoplan-decisions) |
| `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) |
| `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) |
| `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) |
@@ -57,10 +58,12 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-learning` | `domain` | `../govoplan-learning` | [govoplan-learning](https://git.add-ideas.de/GovOPlaN/govoplan-learning) |
| `govoplan-ledger` | `domain` | `../govoplan-ledger` | [govoplan-ledger](https://git.add-ideas.de/GovOPlaN/govoplan-ledger) |
| `govoplan-mail` | `domain` | `../govoplan-mail` | [govoplan-mail](https://git.add-ideas.de/GovOPlaN/govoplan-mail) |
| `govoplan-mandates` | `domain` | `../govoplan-mandates` | [govoplan-mandates](https://git.add-ideas.de/GovOPlaN/govoplan-mandates) |
| `govoplan-notifications` | `platform` | `../govoplan-notifications` | [govoplan-notifications](https://git.add-ideas.de/GovOPlaN/govoplan-notifications) |
| `govoplan-ops` | `platform` | `../govoplan-ops` | [govoplan-ops](https://git.add-ideas.de/GovOPlaN/govoplan-ops) |
| `govoplan-organizations` | `platform` | `../govoplan-organizations` | [govoplan-organizations](https://git.add-ideas.de/GovOPlaN/govoplan-organizations) |
| `govoplan-payments` | `domain` | `../govoplan-payments` | [govoplan-payments](https://git.add-ideas.de/GovOPlaN/govoplan-payments) |
| `govoplan-parties` | `domain` | `../govoplan-parties` | [govoplan-parties](https://git.add-ideas.de/GovOPlaN/govoplan-parties) |
| `govoplan-permits` | `domain` | `../govoplan-permits` | [govoplan-permits](https://git.add-ideas.de/GovOPlaN/govoplan-permits) |
| `govoplan-policy` | `platform` | `../govoplan-policy` | [govoplan-policy](https://git.add-ideas.de/GovOPlaN/govoplan-policy) |
| `govoplan-poll` | `domain` | `../govoplan-poll` | [govoplan-poll](https://git.add-ideas.de/GovOPlaN/govoplan-poll) |
@@ -74,6 +77,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-risk-compliance` | `domain` | `../govoplan-risk-compliance` | [govoplan-risk-compliance](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance) |
| `govoplan-scheduling` | `domain` | `../govoplan-scheduling` | [govoplan-scheduling](https://git.add-ideas.de/GovOPlaN/govoplan-scheduling) |
| `govoplan-search` | `platform` | `../govoplan-search` | [govoplan-search](https://git.add-ideas.de/GovOPlaN/govoplan-search) |
| `govoplan-services` | `domain` | `../govoplan-services` | [govoplan-services](https://git.add-ideas.de/GovOPlaN/govoplan-services) |
| `govoplan-tasks` | `domain` | `../govoplan-tasks` | [govoplan-tasks](https://git.add-ideas.de/GovOPlaN/govoplan-tasks) |
| `govoplan-templates` | `domain` | `../govoplan-templates` | [govoplan-templates](https://git.add-ideas.de/GovOPlaN/govoplan-templates) |
| `govoplan-tenancy` | `platform` | `../govoplan-tenancy` | [govoplan-tenancy](https://git.add-ideas.de/GovOPlaN/govoplan-tenancy) |
Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md
+146 -119
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:75469c08423f21fc61ec7c15 -->
<!-- codex-wiki-sync:c2e1a6a893c11e2057885bf5 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`.
> Origin: `repository`.
@@ -7,157 +7,184 @@
---
# Scaling And Multi-Host Deployment
## What Is Implemented
## Implemented Contract
The default installer now supports explicit same-host horizontal scaling:
GovOPlaN now supports a stateless application tier backed by logically shared
state services. The runtime roles are independently replaceable API, WebUI,
worker, and scheduler processes. Every replica in one installation must use the
same immutable release composition and the same:
- one HAProxy container accepts the published HTTP endpoint;
- one or more WebUI containers serve assets and proxy API requests;
- one or more API containers serve normal domain traffic;
- one or more Celery workers consume shared Redis queues;
- one migration job runs before replacement;
- exactly one Celery scheduler runs;
- PostgreSQL, Redis, and file storage are shared by every runtime replica.
- `GOVOPLAN_INSTALLATION_ID`;
- PostgreSQL database;
- Redis broker and coordination service;
- `MASTER_KEY_B64` and deployment secret references;
- enabled-module graph;
- S3-compatible object-storage namespace.
HAProxy discovers Compose replicas through Docker's internal DNS and performs
health-checked round-robin WebUI balancing and least-connection API balancing.
It does not mount the Docker socket. Replica counts live in
`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on
the desired topology.
The application tier must not use node-local durable business data in a
multi-host deployment. Files owns managed file metadata while Core provides the
storage-backend contract. Campaign build artifacts are stored under opaque
object keys and workers read those objects from the shared backend. Temporary
build and materialization directories may remain node-local because they are
discardable.
This is not multi-host scheduling. Docker Compose's bridge network belongs to
one Docker Engine. Adding a second machine requires an orchestrator or
deployment manager that can place equivalent role definitions on multiple
hosts.
Core validates three explicit state profiles:
## Recommended Topologies
| Profile | Supported shape | Storage rule |
| --- | --- | --- |
| `local` | One API and one worker process for development | Local filesystem permitted. |
| `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. |
| `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. |
### One Host
`shared` also requires a stable installation identifier. Module package
mutation is blocked in this profile: build and verify a new immutable release,
then roll the complete cluster to it.
Use the generated Compose bundle:
## Same-Host Compose
The generated Compose bundle provides:
```text
client
-> external TLS proxy, when required
-> managed HAProxy
-> WebUI replica(s)
-> managed HAProxy API listener
-> API replica(s)
client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas
API/worker/scheduler
-> PostgreSQL
-> Redis
-> local volume, managed Garage, or external S3
API/worker/scheduler -> PostgreSQL
-> Redis
-> local volume, managed Garage, or external S3
```
This improves concurrency and permits rolling process replacement, but the host
and every managed stateful component remain single failure domains. Two API
containers on one failed host do not provide host-level availability.
HAProxy discovers Compose replicas through Docker DNS and performs health-aware
balancing without mounting the Docker socket. This improves concurrency and
permits process replacement, but the Docker host and installer-managed stateful
services remain single failure domains. Generated Compose therefore declares
the `host-shared` state profile even when its shared storage happens to be an
external S3 service. Its API backend checks `/health/ready`, so drain or
coordination loss removes a replica from rotation. Container, load-balancer,
and Kubernetes probes send the configured public host explicitly, keeping
readiness compatible with strict trusted-host validation.
### Multiple Hosts
Managed Garage is a convenient single-node S3-compatible service. It is not a
multi-host storage cluster. Use an independently operated Garage cluster or
another S3-compatible service for the `shared` profile.
Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not
extend the Compose installer into a proprietary scheduler. The target topology
is:
## Kubernetes Export
```text
public TLS ingress/load balancer
-> WebUI replicas on at least two nodes
-> internal API service/load balancer
-> API replicas on at least two nodes
The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL,
Redis, and S3 are all external:
queue-specific worker pools on worker nodes
one fenced scheduler
one fenced migration/deployment job
shared PostgreSQL
shared Redis
multi-node Garage or external S3
shared secret/config provider
central logs, metrics, and health alerts
```sh
python tools/deployment/govoplan-deploy.py render-kubernetes \
--directory /srv/govoplan/default \
--namespace govoplan \
--secret-name govoplan-runtime \
--tls-secret-name govoplan-tls \
--ingress-class-name nginx \
--output /srv/govoplan/default/kubernetes.json
```
All API and worker nodes must receive the same immutable software composition,
`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object
storage binding. Node-local file storage is not valid in this topology.
The export contains a Namespace, tokenless ServiceAccount, non-secret
ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption
budgets, Ingress, and a release-specific migration Job. It deliberately emits
no Secret values, persistent volume, PostgreSQL, Redis, or object-store
deployment. Export is rejected unless both release images use immutable
`image@sha256:...` references.
## Adding Capacity
Create the named Secret through the cluster's secret-management path. The
command prints the exact required key contract. Review the generated
`FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network
before production use.
### API And WebUI
The generated containers run as non-root with a read-only root filesystem and
an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database
migration heads before starting. The API exposes `/health/ready`, which fails
while that API node is draining or cannot prove its runtime-coordination
heartbeat.
Increase API replicas for measured request concurrency, CPU saturation, or
latency after checking database load. Increase WebUI replicas for static asset
and proxy capacity. No sticky session should be required because durable
sessions and throttling use shared services, but this must remain covered by
multi-replica integration tests.
## Runtime Coordination
Every added API process also adds database connections. Set the application
pool size and the total replica ceiling against PostgreSQL's connection budget;
adding replicas can otherwise reduce throughput.
Each API and worker incarnation registers in PostgreSQL with its role, software
version, module-composition hash, queue set, and heartbeat. Ops shows active,
draining, stopped, and stale nodes and compares active counts with configured
replica expectations.
### Workers
An operator may request or cancel drain from Ops:
Workers are not placed behind a load balancer. They compete for jobs on shared
Redis queues. Add workers by queue and cap each pool according to the external
system it calls. SMTP, IMAP, directory, and connector jobs often reach provider
rate limits before CPU limits.
- API readiness becomes unavailable on the next heartbeat so the load balancer
stops assigning new requests.
- A worker stops consuming its configured queues and may finish work already
claimed by that process.
- A stale process incarnation cannot overwrite a replacement incarnation's
heartbeat.
- A coordination outage removes API readiness and cancels worker consumers;
the existing incarnation must heartbeat successfully before either resumes.
Use distinct pools when load warrants it:
Singleton work uses PostgreSQL-backed leases with monotonically increasing
fencing tokens. The generated scheduler runs Celery beat through
`govoplan_core.commands.fenced_run`; loss of its lease terminates the child and
returns a distinct failure code. A fenced business operation must validate the
same lease token immediately before committing its effect.
- mail send and Sent-folder append;
- notifications;
- calendar and connector synchronization;
- dataflow/workflow execution;
- reporting and export;
- platform events and default work.
## Release Ordering
Before stopping a worker, mark it draining, stop new claims, and let or safely
requeue active jobs. Worker registration and drain control remain part of the
larger platform scale-out story.
Use this order for every multi-replica rollout:
### Stateful Services
1. Verify immutable image identities, module composition, external state
reachability, backup evidence, and the generated plan.
2. Drain application replicas when the migration compatibility declaration
requires it.
3. Run the release-specific migration Job exactly once. PostgreSQL advisory
locking serializes all Core and module migration tasks across competing
deployment jobs.
4. Let runtime init containers run `wait_for_database`. They wait for exact
configured Alembic heads and never mutate schema.
5. Roll API, workers, scheduler, and WebUI using health-aware replacement.
6. Verify runtime composition, expected replica counts, queue consumers,
object-storage round trips, and recovery status in Ops.
- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover
design appropriate to the service level.
- Redis needs persistence and, for high availability, a supported failover
topology. Queue loss is not equivalent to a harmless cache loss.
- Managed Garage from the Compose installer is single-node. For redundant
storage, deploy a multi-node Garage cluster separately and select external
`s3`, or use another compatible object store.
- The scheduler stays at one replica until distributed leader election or a
fenced lease is implemented.
- Migrations and module lifecycle mutations remain one-at-a-time operations.
Applying the complete generated manifest is fail-closed: runtime pods remain in
their init phase until the migration Job reaches the expected heads. A second
release may be submitted concurrently, but advisory locking prevents concurrent
schema mutation and each release has a distinct migration Job name.
## Promotion Path
## Storage Trust Boundary
1. Measure concurrent users, request latency, database query time, connection
use, queue age, worker saturation, storage latency, and external throttling.
2. Move files to managed Garage or external S3 before introducing independent
runtime hosts.
3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy
their reviewed HA operators.
4. Publish immutable API/WebUI images and one versioned configuration/secret
contract.
5. Translate the generated role commands, environment allowlists, health
checks, and singleton constraints into the chosen orchestrator.
6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
7. Add queue-specific workers with fixed upper bounds.
8. Prove replica loss, rolling replacement, job redelivery, session continuity,
migration exclusion, backup restore, and storage-node loss before claiming
high availability.
Installer-managed Garage uses its exact generated endpoint. An arbitrary
external S3 endpoint is accepted only when the deployment explicitly sets
`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS
origin without embedded credentials, query, fragment, or path. This is an
operator trust declaration, not a user-controlled connector bypass. Operators
remain responsible for DNS, certificate, network-egress, bucket-policy,
versioning, and lifecycle controls.
## Remaining Platform Work
## Capacity
The one-host installer does not yet provide:
- Scale API replicas only within the PostgreSQL connection budget.
- Scale workers by queue, with upper bounds based on external provider limits.
- Keep one fenced scheduler rather than load-balancing schedulers.
- Increase WebUI replicas for asset/proxy capacity.
- Measure request latency, database query time and locks, active connections,
queue age, retry rate, storage latency, and provider throttling before adding
replicas.
- a Kubernetes, Nomad, or Swarm deployment export/profile;
- distributed deployment locks and fenced scheduler leadership;
- worker registration, composition-skew reporting, and drain controls;
- managed PostgreSQL, Redis, or multi-node Garage high availability;
Workers compete for Redis-backed work and are not placed behind a load balancer.
SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often
hit external-system limits before host CPU is exhausted.
## What This Does Not Claim
The implemented contract provides stateless runtime placement, shared artifact
access, node visibility, drain controls, migration serialization, and scheduler
fencing. It does not by itself provide:
- a highly available PostgreSQL, Redis, or object-store deployment;
- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
- autoscaling policy;
- managed TLS certificate issuance and renewal;
- measured multi-replica and failover integration evidence.
- central logs, metrics, traces, or alert routing;
- managed ingress certificates;
- automatic reconciliation of every possible module side effect;
- a service-level availability guarantee.
These are separate production capabilities. The local load balancer and
replica model provide the contract they should implement, but they do not by
themselves make a cluster highly available.
Those are deployment and module-adoption requirements. Before claiming high
availability, drill replica loss, rolling replacement, session continuity, job
redelivery, scheduler failover, migration exclusion, object-store outage, and a
coordinated database/object/key restore. Recovery rules and evidence are
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY.md
+17 -4
@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:76e4d772814a0842238fd114 -->
<!-- codex-wiki-sync:978cfaf4b9d5c9243c07ec19 -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md`.
> Origin: `repository`.
@@ -146,6 +146,17 @@ The canonical backlog item is
## Implementation slices
Implementation status as of the current source tree:
- Slice 6 has a working application-tier foundation: state profiles, shared
object storage, runtime node registration/heartbeats/drain, fenced scheduler,
migration serialization, exact-head startup waiting, Ops visibility, and a
Kubernetes export. Production acceptance still requires topology-specific
failover and restore drills.
- The recovery foundation for slices 4 and 5 is implemented as a Core recovery
ledger and deployment operation journal. Automatic database backup and broad
adoption by module-owned external effects remain open work.
1. **Reproducible Core-only distribution.** Publish pinned multi-architecture
images, signed distribution manifest, Core-only Compose profile, bootstrap
preflight, generated secrets, readiness, and idempotent rerun/repair.
@@ -162,9 +173,11 @@ The canonical backlog item is
5. **Safe module update.** Add drain/maintenance coordination, backup gate,
migration compatibility window, reconnectable progress, health verification,
retry/recovery, and update notification.
6. **Stateless replica profile.** Remove remaining local-runtime assumptions,
expose role-specific commands/images, implement worker registration/drain,
and prove multiple API and worker replicas against shared dependencies.
6. **Stateless replica profile.** Continue module adoption and operational
proof for the implemented role commands, shared-state validation, runtime
registration/drain, fenced scheduler, and Kubernetes application-tier
export. Prove multiple API and worker replicas against the target shared
dependencies.
7. **Configuration revision model.** Define provider export/import schemas,
canonical serialization, secret references, validation/diff, immutable
revision storage, audit, apply, and undo-as-new-revision.