diff --git a/Codex-Project-Index.md b/Codex-Project-Index.md index e7f3ab3..d36dce5 100644 --- a/Codex-Project-Index.md +++ b/Codex-Project-Index.md @@ -18,6 +18,7 @@ This page is generated from repository and product-directory project files. - [Repo-docs-META-REPOSITORY-MIGRATION-AUDIT](Repo-docs-META-REPOSITORY-MIGRATION-AUDIT) - `/mnt/DATA/git/govoplan/docs/META_REPOSITORY_MIGRATION_AUDIT.md` - [Repo-docs-MODULE-CONTRACTS-AND-INSTALLS](Repo-docs-MODULE-CONTRACTS-AND-INSTALLS) - `/mnt/DATA/git/govoplan/docs/MODULE_CONTRACTS_AND_INSTALLS.md` - [Repo-docs-PLATFORM-CONTROL-PLANE](Repo-docs-PLATFORM-CONTROL-PLANE) - `/mnt/DATA/git/govoplan/docs/PLATFORM_CONTROL_PLANE.md` +- [Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES](Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES) - `/mnt/DATA/git/govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md` - [Repo-docs-REFERENCE-JOURNEY-PROGRAM](Repo-docs-REFERENCE-JOURNEY-PROGRAM) - `/mnt/DATA/git/govoplan/docs/REFERENCE_JOURNEY_PROGRAM.md` - [Repo-docs-RELEASE-CONSOLE](Repo-docs-RELEASE-CONSOLE) - `/mnt/DATA/git/govoplan/docs/RELEASE_CONSOLE.md` - [Repo-docs-REPOSITORY-INDEX](Repo-docs-REPOSITORY-INDEX) - `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md` diff --git a/Repo-README.md b/Repo-README.md index f5dfb7d..c558548 100644 --- a/Repo-README.md +++ b/Repo-README.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/README.md`. > Origin: `repository`. @@ -162,6 +162,9 @@ The current executable slice and remaining production gates are documented in [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). Same-host replica balancing and the multi-host promotion boundary are documented in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). +The recovery state machine, migration rollback boundary, and required restore +drills are documented in +[Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md). ## Configuration @@ -204,6 +207,10 @@ reconfiguration semantics, and safe Web update boundary are defined in [Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md). The concrete replica, worker-node, load-balancer, and shared-state topology is defined in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md). +Durable deployment journals, Core recovery evidence, and the distinction +between pre-migration configuration restore and post-migration forward recovery +are defined in +[Recovery and Rollback Guarantees](docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md). The first Campaign-centric capability and infrastructure fit assessment is in `docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. Its rerun tooling can collect and verify a bounded installed composition; target, provider and production claims diff --git a/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md b/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md index 83f45a1..98bc091 100644 --- a/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md +++ b/Repo-docs-CAPABILITY-AND-INFRASTRUCTURE-FIT.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. > Origin: `repository`. @@ -483,10 +483,11 @@ which modules a running service activated, migrations, configuration, health, or reference-journey behavior. Runtime activation therefore remains an explicit unchecked boundary. -### Target, provider and production proof boundary +### Reference-readiness, provider and production proof boundary -Installed evidence cannot approve a target environment, an external provider, -or production use. These scopes use a separate, expiring +Installed evidence cannot establish target acceptance, accessibility, privacy, +security, operations, recovery, an external provider, or production use. These +scopes use a separate, expiring [`capability-fit-boundary-evidence.schema.json`](capability-fit-boundary-evidence.schema.json) bundle. The bundle is bound to the assessment ID, assessment release and exact installed-evidence SHA-256 digest. It contains only opaque subject/control/result @@ -495,8 +496,9 @@ IDs and content hashes, not endpoints, credentials, people or raw result files. Boundary evidence is accepted only when at least one Ed25519 signature validates against a separately provisioned [`capability-fit-proof-authority-keyring.schema.json`](capability-fit-proof-authority-keyring.schema.json). -Each authority key explicitly lists the scopes it may attest. Target and provider -claims use `passed` or `failed`; production claims use `approved` or `rejected`. +Each authority key explicitly lists the scopes it may attest. Target, +accessibility, privacy, security, operations, recovery, and provider claims use +`passed` or `failed`; production claims use `approved` or `rejected`. One claim per scope, unique control/artifact IDs, `issued_at < expires_at`, current validity and exact digest binding are mandatory. Any schema, binding, time, signature or authority blocker leaves every supplied boundary claim unchecked; @@ -569,6 +571,10 @@ Target-environment and production-approval claims must use the assessment's operator to supply a bounded opaque expected subject with `--expected-external-provider-subject`; without it, such a claim remains unchecked and blocks. Expected and observed IDs are retained in proof scope. +Accessibility, privacy, security, operations, and recovery claims use the same +deployment subject. The report emits a cumulative `reference_readiness` verdict +only when all six required scopes are checked and positive. This verdict remains +separate from production approval and from provider-specific acceptance. Both authority keyrings are governance trust roots. Installer receipt keys use the strict @@ -590,9 +596,9 @@ cannot relabel it as current. Live admission must omit it and use the actual current time. No boundary bundle or production authority has been supplied for this current -assessment. Target environment, provider and production proof therefore remain -explicitly unchecked rather than inferred from the local GreenMail journey, -source tests or signed release metadata. +assessment. Reference-readiness, provider, and production proof therefore +remain explicitly unchecked rather than inferred from the local GreenMail +journey, source tests, or signed release metadata. ## Evidence used in this slice diff --git a/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md b/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md index 2e5534c..aebde18 100644 --- a/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md +++ b/Repo-docs-INSTALLATION-AND-DEPLOYMENT-ARCHITECTURE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md`. > Origin: `repository`. @@ -41,9 +41,15 @@ It currently supports: - generated secrets stored in a private `0600` file; - service-specific environment allowlists so infrastructure containers do not receive unrelated application credentials; -- plan, render, doctor, status, and apply commands; +- plan, render, doctor, status, apply, Kubernetes export, operation history, + and bounded recovery commands; - an installation lock, migration-before-start ordering, readiness polling, and an applied-state receipt; +- a durable hash-chained deployment journal captured before runtime mutation; +- PostgreSQL advisory serialization for Core and module migrations; +- runtime initialization that waits for exact configured migration heads + without mutating schema; +- runtime node registration, heartbeats, drain state, and a fenced scheduler; - idempotent reconfiguration that preserves generated secrets; - a keyed environment fingerprint that detects private binding changes without writing secret values to plans or receipts; @@ -94,6 +100,9 @@ The private installation directory contains: | `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration | | `plan.json` | Latest desired-state diff and readiness findings | | `receipt.json` | Last successfully applied immutable identities | +| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle | +| `operations//` | Private hash-chained deployment progress and recovery evidence | +| `kubernetes.json` | Optional stateless multi-host Kubernetes export | | `.deployment.lock` | Same-host operation exclusion | The specification contract is @@ -208,11 +217,13 @@ presenting a container as a complete mail service. installations. `garage` provisions Garage 2.3 in its supported single-node bootstrap mode, generates a private application key and bucket, and connects the Files S3 backend to the exact installer-owned internal endpoint. The -deployment-only trust marker cannot authorize another S3 host; arbitrary -external SDK endpoints remain fail-closed until peer pinning is implemented. +managed trust marker cannot authorize another S3 host. Garage metadata and object data use separate persistent volumes. `s3` requires an external endpoint, region, access key, secret key, and bucket values. -Self-hosted external S3 endpoints must use HTTPS. +Self-hosted external S3 endpoints must be clean HTTPS origins. The generated +runtime explicitly sets `FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` for that +operator-selected endpoint. The trust flag is not accepted for local storage +and cannot be combined with installer-managed Garage trust. Local storage must be included in backup and restore drills. Horizontal API or worker scale-out requires shared/object storage. The managed Garage profile is @@ -240,9 +251,10 @@ Replica counts are desired state: ``` Workers are queue consumers, so they are scaled through Redis rather than put -behind an HTTP load balancer. The migration runner and Celery scheduler remain -singletons. Multiple API replicas are rejected when Redis is disabled because -distributed throttling and queued work cannot then be shared correctly. +behind an HTTP load balancer. Migrations are serialized with a deployment-wide +PostgreSQL advisory lock. The Celery scheduler is run under a renewable, +fencing-token lease. Multiple API replicas are rejected when Redis is disabled +because distributed throttling and queued work cannot then be shared correctly. This is same-host scaling. Docker Compose uses a bridge network and does not place containers on another machine. See @@ -265,13 +277,66 @@ applied state. `plan` compares their canonical hashes and service sets. - Managed-to-external transitions require the new endpoint in the same operation. - Migrations run as a one-shot service before API/worker replacement. +- API, worker, and scheduler start commands wait for exact configured migration + heads; only the migration command is permitted to change schema. +- API and worker replicas register their software/module composition and + heartbeat in PostgreSQL. Ops can request and cancel a node drain. - The first upgrade from a direct WebUI host port stops that legacy WebUI container immediately before HAProxy claims the same endpoint. -- Health must recover before a new receipt is committed. +- Health must recover before a new receipt and applied-state snapshot are + committed. -This is sufficient for one-host reconciliation. Production updates additionally -need backup/restore gates, maintenance/drain state, database compatibility -windows, image signature verification, and rollback/forward-recovery policy. +Every apply operation is journalled before image pulls or runtime mutation. A +failure before migration may restore a verified previous bundle. Once migration +starts, recovery is forward-only unless an independently verified database +backup is restored. See +[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md). + +Production updates still need an operator-provided database backup/restore +gate, database compatibility declaration, image signature verification, and +deployment-specific drain policy. The deployment journal proves its own +actions; it does not manufacture backup evidence. + +## Stateless Kubernetes Runtime + +`render-kubernetes` exports the application tier for a standard orchestrator. +It requires external PostgreSQL, Redis, and S3 and emits no stateful service or +secret value: + +```sh +python tools/deployment/govoplan-deploy.py render-kubernetes \ + --directory /srv/govoplan/default \ + --namespace govoplan \ + --secret-name govoplan-runtime +``` + +The output includes a release-specific migration Job, database-head wait init +containers, API readiness/liveness probes, rolling Deployments, Services, Pod +disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply +the named Secret through the cluster's secret manager and review ingress proxy +CIDRs before deployment. Detailed rollout and scaling rules live in +[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md). + +## Recovery Commands + +List durable deployment operations: + +```sh +python tools/deployment/govoplan-deploy.py operations \ + --directory /srv/govoplan/default +``` + +Recover a selected failed operation after reviewing its stage evidence: + +```sh +python tools/deployment/govoplan-deploy.py recover \ + --directory /srv/govoplan/default \ + --operation-id +``` + +The command reports whether it restored the pre-migration applied bundle, +requires forward recovery, or needs manual intervention. Add `--apply` only +after that decision has been reviewed. ## Web Update Boundary @@ -324,5 +389,6 @@ The tests cover profile restrictions, secret persistence, external endpoint requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy discovery configuration, Compose service selection, secret non-disclosure, service-specific environment isolation, private file modes, external endpoint -preflight, first-plan generation, apply ordering, and receipt-based -idempotency. +preflight, first-plan generation, apply ordering, receipt idempotency, +hash-chained recovery journals, migration recovery boundaries, and stateless +Kubernetes rendering. diff --git a/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md b/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md index 09f22a0..400313a 100644 --- a/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md +++ b/Repo-docs-INSTITUTIONAL-GOVERNANCE-TARGET-ARCHITECTURE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/INSTITUTIONAL_GOVERNANCE_TARGET_ARCHITECTURE.md`. > Origin: `repository`. @@ -90,11 +90,12 @@ were proven now have independent persistent owners: | External providers | Core validates provider objects/field groups, operations, integration maturity, source authority, bounded reads, freshness/health, idempotency, conflicts, outcome-unknown handling, evidence, correction, reconciliation, outage, classification, purpose, retention, and secret handling. Addresses/CardDAV, Files remote storage, Mail SMTP/IMAP, Calendar CalDAV/ICS/Graph/EWS, and Connectors tabular/sanctions providers declare the contract and tenant-bounded secret-free runtime state. | Registry validation rejects any declared external provider without a sanitized state provider. Future adapters must cross the same gate before activation. | | Institutional context | Core provides versioned temporal, actor/representation, institution/unit/function/task/mandate/jurisdiction/service/case/party/work-item/workflow/approval/decision/record, legal-basis, evidence, information-governance, external-source, presentation, and geographic references. Events, automation actions, audit records, and the transactional Audit outbox preserve the envelope. | Owning modules must progressively require the relevant subset for consequential operations. | | Semantic provider contracts | Provider-neutral DTOs and protocols cover Mandate resolution, versioned Service definitions, procedure Parties/representation, and formal Decisions. `govoplan-mandates`, `govoplan-services`, `govoplan-parties`, and `govoplan-decisions` now persist immutable revisions behind those contracts with tenant isolation, bounded reads, replay safety, OCC, migrations, uninstall guards, permissions, APIs, capability documentation, and recovery documentation. | The owners are deliberately headless. Procedure-specific UI remains with consuming modules. | -| Formal-outcome proof | Committee persists bodies, meetings, agenda items, votes, minutes, lifecycle events, and an optional protected local Decision projection. It resolves effective Mandate authority and records reconstructable formal Decisions through `decisions.registry` when installed. OCC, replay safety, tenant isolation, bounded reads, and closure guards cover the aggregate. | Committee remains API-first and does not yet provide its later deliberation WebUI or specialized secret/external ballot adapters. | -| Service-to-case proof | Services owns the persistent exact definitions consumed by Portal discovery and Cases intake. Portal exposes a tenant-scoped API and compact service-directory WebUI whose audiences are derived from trusted principal/function state. Cases persists catalogs, immutable revisions, parties, assignments, evidence/decision/record references, deadlines, lifecycle events, exact Service versions, OCC, replay safety, tenant isolation, API permissions, ACL filtering, uninstall guards, tenant summary state, and list/detail/history/timeline WebUI. | Specialized form/case/workflow launchers and procedure panels remain with their owners. Cases still needs finer case-sharing policy. Neither gap changes Service ownership. | +| Formal-outcome proof | Committee persists bodies, meetings, agenda items, votes, minutes, lifecycle events, and an optional protected local Decision projection. It resolves effective Mandate authority and records reconstructable formal Decisions through `decisions.registry` when installed. OCC, replay safety, tenant isolation, bounded reads, closure guards, and a full-height body/meeting/agenda/vote/minutes WebUI cover the aggregate. Provider-bound external or secret ballots use dynamic adapter capabilities; Committee validates and retains only aggregate counts, receipt/hash, and evidence, and rejects generic manual closure. | Concrete ballot-provider packages remain integration work because their protocol, custody, credentials, and operator evidence depend on the selected provider. This does not leave a Committee-owned ballot contract unspecified. | +| Service-to-case proof | Services owns the persistent exact definitions consumed by Portal discovery and Cases intake. Portal exposes a tenant-scoped API and service-directory WebUI whose audiences are derived from trusted principal/function state. Launch re-fetches and re-evaluates the exact Service revision. URL launch is validated; Cases and Workflow Engine provide tenant-bound, replay-safe owner launchers. Cases persists catalogs, immutable revisions, parties, assignments, evidence/decision/record references, deadlines, lifecycle events, exact Service versions, OCC, replay safety, tenant isolation, explicit and assignment-derived object ACLs, uninstall guards, tenant summary state, and list/detail/history/timeline/share WebUI. | Form-bound launch remains unavailable until Forms and Forms Runtime move beyond scaffolds to immutable definitions, validation, submission persistence, and a real owner launcher. Portal fails this case closed and explains the missing capability. | | Procedure-party proof | Parties persists effective procedure roles, frozen contact snapshots, and representation powers. Existing powers cannot disappear or be silently rewritten; explicit OCC-guarded revocation is required. Cases resolves the provider capability and excludes expired/revoked authority from downstream delivery. | Procedure modules still decide which contextual fields and actions to present. | | Integrated institutional journey | The executable `product.service-to-decision` fixture uses real SQL-backed Services, Cases, Parties, Mandates, Committee, and Decisions providers. It carries one exact Service version through persisted Case intake, representation and frozen delivery authority, effective Mandate resolution, a body/meeting/agendum/vote/minute sequence, a persisted formal Decision, confirmed Postbox effect, Audit/record evidence, remedy/review, and protected reconstruction. | This is architecture and composition evidence. Signed, release-bound target accessibility, privacy, security, operator, delivery-provider, and recovery-drill evidence is still required before the package may claim `reference_ready`. | -| Governed data catalogue | Datasources stores typed governance metadata, exposes bounded tenant-scoped filters and update APIs/UI, carries governance through staging, and snapshots it into immutable materializations. | Rich dependency/impact traversal and policy-specific field visibility can grow on the catalogue contract without moving connector or transformation ownership. | +| Governed data catalogue | Datasources stores typed governance metadata, exposes bounded tenant-scoped filters and update APIs/UI, carries governance through staging, and snapshots it into immutable materializations. Reporting now persists immutable dataset, semantic-model, report, quality-plan, saved-view, and schedule revisions; executes typed semantic queries with quality gates, access checks, replay, pivoting, export/import assessment, and provenance; and exposes the governed analytical WebUI. | Rich dependency/impact traversal, additional expression functions, and policy-specific field visibility can grow on the established contracts without moving connector, transformation, or source ownership. | +| Portfolio and change governance | Projects now persists tenant-safe, immutable portfolio/project/milestone revisions with OCC, replay, lifecycle rules, restricted memberships, Search ACL indexing, outcomes, benefits, dependencies, capacity assumptions, change impact, and institutional references. Its WebUI exposes the planning catalogue and core planning fields. | Advanced planning structures already accepted by the API can receive deeper specialized editors without creating a second Policy, Reporting, Resources, or Goals owner. | | Product/package governance | Signed configuration packages distinguish reference, product, sector, deployment, and integration classes; preserve parent/evidence provenance; prevent derived packages from loosening constraints; and preflight provider authority, maturity, exact binding, health, freshness, and recovery expectations. Executable product manifests now exist for governed communication and governed data/assurance and are checked in the module matrix. | Both artifacts deliberately remain product-class until target, accessibility, privacy, security, operations, and recovery evidence justifies reference readiness. | | Projection and release | Platform metadata, signed module catalogs, release synthesis, Ops, and role-aware Docs retain and display architecture/provider declarations. Module-owned state providers add bounded configured/active, authority, health, freshness, conflict, recovery, and observation state; ordinary-user Docs omits binding detail. Static checks validate evidence paths, and the WebUI build verifies consuming types. | Runtime-state adoption and broader portfolio presentation follow truthful provider declaration rollout. | @@ -424,17 +425,17 @@ submodule, configuration fragment, package, or profile. - `product.service-to-decision` proves both through a portable administrative service composition. -### 3. Complete governed data and assurance - architecture complete +### 3. Complete governed data, portfolio, and assurance - vertical slices complete - Datasources carries typed governance through staging and immutable materializations, with bounded catalogue filters and dependency references. -- Reporting owns semantic presentation/provenance contracts without taking - source or transformation ownership; feature depth remains on its module - backlog. +- Reporting owns immutable semantic definitions, safe execution, quality gates, + provenance, schedules, saved views, pivoting, and export/import assessment + without taking source or transformation ownership. - Risk Compliance persists the horizontal obligation/risk/control/evidence/ finding/measure graph and projects sanctions runs idempotently. -- Projects declares portfolio/outcome ownership as a consuming domain without - becoming a second policy or reporting engine. +- Projects persists portfolio/outcome/change-governance revisions as a + consuming domain without becoming a second policy or reporting engine. ### 4. Package repeatable public-sector outcomes - complete at product maturity @@ -446,6 +447,36 @@ submodule, configuration fragment, package, or profile. recovery, accessibility, privacy, security, and operator evidence. This is a maturity gate, not missing architecture implementation. +## What remains after the executable architecture slice + +The remaining work is not another Core or cross-module architecture rewrite. +It falls into three explicitly different categories: + +1. **Forms and Forms Runtime vertical:** both repositories remain scaffolds. + Full service entry requires immutable form-definition lookup, server-side + validation, draft/submission persistence, status/evidence history, + attachments/signatures/handoff references, replay and OCC, APIs, accessible + WebUI, migrations, recovery, and `forms_runtime.service_launcher`. Portal + already advertises this absence as an availability blocker. Implementing a + placeholder launcher before those controls would violate the accepted + contract. +2. **Concrete provider packages:** the Committee ballot adapter contract is + complete, but a real secret/electronic ballot provider requires a selected + protocol and product decisions for voter eligibility, custody, secrecy, + recount, challenge, retention, and operational assurance. Equivalent future + adapters must satisfy the declared provider and recovery gates. +3. **Target-produced maturity evidence:** `reference_ready`, `supported`, and + `lts` cannot be generated from source code. An exact release and deployment + must produce signed, expiring accessibility, privacy, security, operator, + provider, backup/restore, rollback, and recovery-drill evidence. The verifier + and schemas are implemented; the actual claims require those real runs. + +Everything else described as architecture in this document now has a +repository owner, versioned contract, bounded implementation, migration and +recovery boundary where state exists, documentation, and executable evidence. +Further work in those modules is product breadth, UX depth, provider adoption, +and evidence renewal. + ## Delivery tracking The completed cross-repository architecture epic is diff --git a/Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES.md b/Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES.md new file mode 100644 index 0000000..f013045 --- /dev/null +++ b/Repo-docs-RECOVERY-AND-ROLLBACK-GUARANTEES.md @@ -0,0 +1,154 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Recovery And Rollback Guarantees + +## Principle + +GovOPlaN must prove recovery claims with durable state recorded before and +after side effects. A failed operation is not automatically rolled back merely +because the previous application image still exists. Database schema and +external effects may make release rollback unsafe. + +Core therefore distinguishes five recovery modes: + +| Mode | Meaning | +| --- | --- | +| `atomic` | One database transaction either commits or rolls back. No external effect is claimed. | +| `compensation` | Durable evidence identifies explicit inverse actions for completed effects. | +| `snapshot_restore` | A separately verified backup reference and restore procedure exist. | +| `forward_recovery` | Repair or resume the current version; reverting code/configuration is not claimed safe. | +| `irreversible` | No automated recovery is claimed and an approval reference is mandatory. | + +An operation plan must include verification steps. Compensation requires named +compensation steps, snapshot restore requires a verified backup reference, +forward recovery requires repair steps, and irreversible work requires explicit +approval. + +## Core Recovery Ledger + +Core stores recovery operations and append-only, hash-chained checkpoints in +PostgreSQL. The contract provides: + +- installation/module/resource identity; +- an idempotency key bound to a canonical request hash; +- recovery mode, preconditions, verification steps, and references; +- optional runtime lease holder and fencing token; +- explicit planned, prepared, running, recovery-required, recovering, + succeeded, recovered, failed, outcome-unknown, and manual-intervention states; +- an evidence-chain head and sequence count; +- rejection of plaintext secrets in metadata or evidence. + +Preparation cannot succeed without durable precondition evidence. A non-atomic +operation cannot hide a partial effect by transitioning directly from running +to failed. Success and recovery require explicit verification evidence with at +least one check. The ledger verifies its hash chain before evidence is trusted. + +This is a platform contract, not an assertion that every existing module +operation has adopted it. Module operations with external or multi-resource +effects must be migrated to the ledger before claiming these guarantees. + +## Deployment Journal + +Every `govoplan-deploy apply` begins an operation journal before it pulls images +or mutates runtime state. The private installation directory records: + +```text +operations//operation.json +operations//before/ +applied-state/ +``` + +Each stage is hash-chained. The previous applied bundle is copied with per-file +SHA-256 evidence. Applied state is replaced atomically after health verification; +an interrupted replacement restores its previous directory. +New journals also bind the complete desired deployment plan, snapshot +availability, failure summary, recovery mode, and terminal status to the +evidence chain. Recovery verifies every snapshot entry and checksum before it +changes any live bundle file, then replaces each live file atomically. A crash +between file replacements is recoverable by rerunning the same idempotent +recovery command under the deployment lock. + +Inspect operations: + +```sh +python tools/deployment/govoplan-deploy.py operations \ + --directory /srv/govoplan/default +``` + +Recover the latest failed operation, or provide its identifier: + +```sh +python tools/deployment/govoplan-deploy.py recover \ + --directory /srv/govoplan/default \ + --operation-id 20260801T120000Z-1234abcd +``` + +Add `--apply` only after reviewing the reported action. + +## Migration Boundary + +Before database migration starts, a failed deployment with a verified prior +applied snapshot may restore its prior release/configuration bundle and +reconcile that desired state. + +As soon as migration starts, the journal permanently changes to +`forward_recovery`. It will not restore old application configuration because +old code may not understand the new schema. Recovery then means one of: + +1. fix and re-run the current release; +2. deploy a newer compatible repair release; +3. restore a separately verified, coordinated database/object/key backup and + then deploy the matching release. + +The deployment tool does not create or validate that database backup. A +`backup-required` annotation on the Kubernetes migration Job is an operator +gate, not backup evidence. Production automation must provide a backup hook or +external backup controller whose artifact, timestamp, scope, encryption key, +and restore test can be referenced from the recovery record. + +## Scaled Nodes + +Recovery actions must be safe across replicas: + +- drain affected API and worker nodes before incompatible changes; +- use the deployment-wide PostgreSQL advisory lock for schema migration; +- use distributed leases and fencing tokens for singleton or externally + visible effects; +- use idempotency keys for retried commands and jobs; +- retain shared object keys and database references until deletion succeeds; +- classify uncertain external outcomes instead of retrying blindly; +- verify the exact software/module composition after replacement. + +Campaign generated-message objects now follow this model: object writes are +compensated when a build fails before database commit, workers verify stored +size and digest before delivery, and retention keeps the database reference +when storage deletion fails. A hard process loss between object creation and +database commit can still leave an orphan object; an inventory reconciler is a +separate operational slice and must use the build-specific object prefix. + +## Required Drills + +Record evidence for at least these scenarios before production acceptance: + +1. Kill an API replica and verify traffic continues without session loss. +2. Drain and replace a worker while work is queued and while one job is active. +3. Start two migration jobs and verify only one mutates schema. +4. Kill the fenced scheduler and verify one replacement acquires a higher + fencing token. +5. Fail deployment before migration and restore the prior applied bundle. +6. Fail deployment after migration and verify old configuration is not + restored. +7. Restore PostgreSQL, object storage, and encryption keys to one coordinated + recovery point and verify representative object hashes. +8. Interrupt object storage during Campaign build and retention and verify + compensation/reference-preservation behavior. +9. Tamper with a deployment or Core recovery checkpoint and verify chain + validation rejects it. + +No runbook, status badge, or green health endpoint substitutes for a dated, +repeatable restore drill against the actual deployment topology. diff --git a/Repo-docs-REPOSITORY-INDEX.md b/Repo-docs-REPOSITORY-INDEX.md index e14c570..1f12338 100644 --- a/Repo-docs-REPOSITORY-INDEX.md +++ b/Repo-docs-REPOSITORY-INDEX.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/REPOSITORY_INDEX.md`. > Origin: `repository`. @@ -38,6 +38,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s | `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard) | | `govoplan-dataflow` | `platform` | `../govoplan-dataflow` | [govoplan-dataflow](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow) | | `govoplan-datasources` | `platform` | `../govoplan-datasources` | [govoplan-datasources](https://git.add-ideas.de/GovOPlaN/govoplan-datasources) | +| `govoplan-decisions` | `domain` | `../govoplan-decisions` | [govoplan-decisions](https://git.add-ideas.de/GovOPlaN/govoplan-decisions) | | `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) | | `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) | | `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) | @@ -57,10 +58,12 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s | `govoplan-learning` | `domain` | `../govoplan-learning` | [govoplan-learning](https://git.add-ideas.de/GovOPlaN/govoplan-learning) | | `govoplan-ledger` | `domain` | `../govoplan-ledger` | [govoplan-ledger](https://git.add-ideas.de/GovOPlaN/govoplan-ledger) | | `govoplan-mail` | `domain` | `../govoplan-mail` | [govoplan-mail](https://git.add-ideas.de/GovOPlaN/govoplan-mail) | +| `govoplan-mandates` | `domain` | `../govoplan-mandates` | [govoplan-mandates](https://git.add-ideas.de/GovOPlaN/govoplan-mandates) | | `govoplan-notifications` | `platform` | `../govoplan-notifications` | [govoplan-notifications](https://git.add-ideas.de/GovOPlaN/govoplan-notifications) | | `govoplan-ops` | `platform` | `../govoplan-ops` | [govoplan-ops](https://git.add-ideas.de/GovOPlaN/govoplan-ops) | | `govoplan-organizations` | `platform` | `../govoplan-organizations` | [govoplan-organizations](https://git.add-ideas.de/GovOPlaN/govoplan-organizations) | | `govoplan-payments` | `domain` | `../govoplan-payments` | [govoplan-payments](https://git.add-ideas.de/GovOPlaN/govoplan-payments) | +| `govoplan-parties` | `domain` | `../govoplan-parties` | [govoplan-parties](https://git.add-ideas.de/GovOPlaN/govoplan-parties) | | `govoplan-permits` | `domain` | `../govoplan-permits` | [govoplan-permits](https://git.add-ideas.de/GovOPlaN/govoplan-permits) | | `govoplan-policy` | `platform` | `../govoplan-policy` | [govoplan-policy](https://git.add-ideas.de/GovOPlaN/govoplan-policy) | | `govoplan-poll` | `domain` | `../govoplan-poll` | [govoplan-poll](https://git.add-ideas.de/GovOPlaN/govoplan-poll) | @@ -74,6 +77,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s | `govoplan-risk-compliance` | `domain` | `../govoplan-risk-compliance` | [govoplan-risk-compliance](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance) | | `govoplan-scheduling` | `domain` | `../govoplan-scheduling` | [govoplan-scheduling](https://git.add-ideas.de/GovOPlaN/govoplan-scheduling) | | `govoplan-search` | `platform` | `../govoplan-search` | [govoplan-search](https://git.add-ideas.de/GovOPlaN/govoplan-search) | +| `govoplan-services` | `domain` | `../govoplan-services` | [govoplan-services](https://git.add-ideas.de/GovOPlaN/govoplan-services) | | `govoplan-tasks` | `domain` | `../govoplan-tasks` | [govoplan-tasks](https://git.add-ideas.de/GovOPlaN/govoplan-tasks) | | `govoplan-templates` | `domain` | `../govoplan-templates` | [govoplan-templates](https://git.add-ideas.de/GovOPlaN/govoplan-templates) | | `govoplan-tenancy` | `platform` | `../govoplan-tenancy` | [govoplan-tenancy](https://git.add-ideas.de/GovOPlaN/govoplan-tenancy) | diff --git a/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md b/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md index 3abe95a..f2324ca 100644 --- a/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md +++ b/Repo-docs-SCALING-AND-MULTI-HOST-DEPLOYMENT.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md`. > Origin: `repository`. @@ -7,157 +7,184 @@ --- # Scaling And Multi-Host Deployment -## What Is Implemented +## Implemented Contract -The default installer now supports explicit same-host horizontal scaling: +GovOPlaN now supports a stateless application tier backed by logically shared +state services. The runtime roles are independently replaceable API, WebUI, +worker, and scheduler processes. Every replica in one installation must use the +same immutable release composition and the same: -- one HAProxy container accepts the published HTTP endpoint; -- one or more WebUI containers serve assets and proxy API requests; -- one or more API containers serve normal domain traffic; -- one or more Celery workers consume shared Redis queues; -- one migration job runs before replacement; -- exactly one Celery scheduler runs; -- PostgreSQL, Redis, and file storage are shared by every runtime replica. +- `GOVOPLAN_INSTALLATION_ID`; +- PostgreSQL database; +- Redis broker and coordination service; +- `MASTER_KEY_B64` and deployment secret references; +- enabled-module graph; +- S3-compatible object-storage namespace. -HAProxy discovers Compose replicas through Docker's internal DNS and performs -health-checked round-robin WebUI balancing and least-connection API balancing. -It does not mount the Docker socket. Replica counts live in -`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on -the desired topology. +The application tier must not use node-local durable business data in a +multi-host deployment. Files owns managed file metadata while Core provides the +storage-backend contract. Campaign build artifacts are stored under opaque +object keys and workers read those objects from the shared backend. Temporary +build and materialization directories may remain node-local because they are +discardable. -This is not multi-host scheduling. Docker Compose's bridge network belongs to -one Docker Engine. Adding a second machine requires an orchestrator or -deployment manager that can place equivalent role definitions on multiple -hosts. +Core validates three explicit state profiles: -## Recommended Topologies +| Profile | Supported shape | Storage rule | +| --- | --- | --- | +| `local` | One API and one worker process for development | Local filesystem permitted. | +| `host-shared` | Multiple processes on one Docker host | A shared host volume is permitted; PostgreSQL and Redis are required. | +| `shared` | Multiple independent hosts | PostgreSQL, Redis, and S3-compatible object storage are required. | -### One Host +`shared` also requires a stable installation identifier. Module package +mutation is blocked in this profile: build and verify a new immutable release, +then roll the complete cluster to it. -Use the generated Compose bundle: +## Same-Host Compose + +The generated Compose bundle provides: ```text -client - -> external TLS proxy, when required - -> managed HAProxy - -> WebUI replica(s) - -> managed HAProxy API listener - -> API replica(s) +client -> TLS proxy -> HAProxy -> WebUI replicas -> HAProxy -> API replicas -API/worker/scheduler - -> PostgreSQL - -> Redis - -> local volume, managed Garage, or external S3 +API/worker/scheduler -> PostgreSQL + -> Redis + -> local volume, managed Garage, or external S3 ``` -This improves concurrency and permits rolling process replacement, but the host -and every managed stateful component remain single failure domains. Two API -containers on one failed host do not provide host-level availability. +HAProxy discovers Compose replicas through Docker DNS and performs health-aware +balancing without mounting the Docker socket. This improves concurrency and +permits process replacement, but the Docker host and installer-managed stateful +services remain single failure domains. Generated Compose therefore declares +the `host-shared` state profile even when its shared storage happens to be an +external S3 service. Its API backend checks `/health/ready`, so drain or +coordination loss removes a replica from rotation. Container, load-balancer, +and Kubernetes probes send the configured public host explicitly, keeping +readiness compatible with strict trusted-host validation. -### Multiple Hosts +Managed Garage is a convenient single-node S3-compatible service. It is not a +multi-host storage cluster. Use an independently operated Garage cluster or +another S3-compatible service for the `shared` profile. -Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not -extend the Compose installer into a proprietary scheduler. The target topology -is: +## Kubernetes Export -```text -public TLS ingress/load balancer - -> WebUI replicas on at least two nodes - -> internal API service/load balancer - -> API replicas on at least two nodes +The deployment compiler exports a stateless Kubernetes runtime when PostgreSQL, +Redis, and S3 are all external: -queue-specific worker pools on worker nodes -one fenced scheduler -one fenced migration/deployment job - -shared PostgreSQL -shared Redis -multi-node Garage or external S3 -shared secret/config provider -central logs, metrics, and health alerts +```sh +python tools/deployment/govoplan-deploy.py render-kubernetes \ + --directory /srv/govoplan/default \ + --namespace govoplan \ + --secret-name govoplan-runtime \ + --tls-secret-name govoplan-tls \ + --ingress-class-name nginx \ + --output /srv/govoplan/default/kubernetes.json ``` -All API and worker nodes must receive the same immutable software composition, -`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object -storage binding. Node-local file storage is not valid in this topology. +The export contains a Namespace, tokenless ServiceAccount, non-secret +ConfigMap, API/WebUI/worker/scheduler Deployments, Services, Pod disruption +budgets, Ingress, and a release-specific migration Job. It deliberately emits +no Secret values, persistent volume, PostgreSQL, Redis, or object-store +deployment. Export is rejected unless both release images use immutable +`image@sha256:...` references. -## Adding Capacity +Create the named Secret through the cluster's secret-management path. The +command prints the exact required key contract. Review the generated +`FORWARDED_ALLOW_IPS` value and replace it with the exact ingress-proxy network +before production use. -### API And WebUI +The generated containers run as non-root with a read-only root filesystem and +an ephemeral `/tmp`. Runtime Deployments wait for the exact configured database +migration heads before starting. The API exposes `/health/ready`, which fails +while that API node is draining or cannot prove its runtime-coordination +heartbeat. -Increase API replicas for measured request concurrency, CPU saturation, or -latency after checking database load. Increase WebUI replicas for static asset -and proxy capacity. No sticky session should be required because durable -sessions and throttling use shared services, but this must remain covered by -multi-replica integration tests. +## Runtime Coordination -Every added API process also adds database connections. Set the application -pool size and the total replica ceiling against PostgreSQL's connection budget; -adding replicas can otherwise reduce throughput. +Each API and worker incarnation registers in PostgreSQL with its role, software +version, module-composition hash, queue set, and heartbeat. Ops shows active, +draining, stopped, and stale nodes and compares active counts with configured +replica expectations. -### Workers +An operator may request or cancel drain from Ops: -Workers are not placed behind a load balancer. They compete for jobs on shared -Redis queues. Add workers by queue and cap each pool according to the external -system it calls. SMTP, IMAP, directory, and connector jobs often reach provider -rate limits before CPU limits. +- API readiness becomes unavailable on the next heartbeat so the load balancer + stops assigning new requests. +- A worker stops consuming its configured queues and may finish work already + claimed by that process. +- A stale process incarnation cannot overwrite a replacement incarnation's + heartbeat. +- A coordination outage removes API readiness and cancels worker consumers; + the existing incarnation must heartbeat successfully before either resumes. -Use distinct pools when load warrants it: +Singleton work uses PostgreSQL-backed leases with monotonically increasing +fencing tokens. The generated scheduler runs Celery beat through +`govoplan_core.commands.fenced_run`; loss of its lease terminates the child and +returns a distinct failure code. A fenced business operation must validate the +same lease token immediately before committing its effect. -- mail send and Sent-folder append; -- notifications; -- calendar and connector synchronization; -- dataflow/workflow execution; -- reporting and export; -- platform events and default work. +## Release Ordering -Before stopping a worker, mark it draining, stop new claims, and let or safely -requeue active jobs. Worker registration and drain control remain part of the -larger platform scale-out story. +Use this order for every multi-replica rollout: -### Stateful Services +1. Verify immutable image identities, module composition, external state + reachability, backup evidence, and the generated plan. +2. Drain application replicas when the migration compatibility declaration + requires it. +3. Run the release-specific migration Job exactly once. PostgreSQL advisory + locking serializes all Core and module migration tasks across competing + deployment jobs. +4. Let runtime init containers run `wait_for_database`. They wait for exact + configured Alembic heads and never mutate schema. +5. Roll API, workers, scheduler, and WebUI using health-aware replacement. +6. Verify runtime composition, expected replica counts, queue consumers, + object-storage round trips, and recovery status in Ops. -- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover - design appropriate to the service level. -- Redis needs persistence and, for high availability, a supported failover - topology. Queue loss is not equivalent to a harmless cache loss. -- Managed Garage from the Compose installer is single-node. For redundant - storage, deploy a multi-node Garage cluster separately and select external - `s3`, or use another compatible object store. -- The scheduler stays at one replica until distributed leader election or a - fenced lease is implemented. -- Migrations and module lifecycle mutations remain one-at-a-time operations. +Applying the complete generated manifest is fail-closed: runtime pods remain in +their init phase until the migration Job reaches the expected heads. A second +release may be submitted concurrently, but advisory locking prevents concurrent +schema mutation and each release has a distinct migration Job name. -## Promotion Path +## Storage Trust Boundary -1. Measure concurrent users, request latency, database query time, connection - use, queue age, worker saturation, storage latency, and external throttling. -2. Move files to managed Garage or external S3 before introducing independent - runtime hosts. -3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy - their reviewed HA operators. -4. Publish immutable API/WebUI images and one versioned configuration/secret - contract. -5. Translate the generated role commands, environment allowlists, health - checks, and singleton constraints into the chosen orchestrator. -6. Put API and WebUI replicas behind health-aware services and a TLS ingress. -7. Add queue-specific workers with fixed upper bounds. -8. Prove replica loss, rolling replacement, job redelivery, session continuity, - migration exclusion, backup restore, and storage-node loss before claiming - high availability. +Installer-managed Garage uses its exact generated endpoint. An arbitrary +external S3 endpoint is accepted only when the deployment explicitly sets +`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`; that endpoint must be a clean HTTPS +origin without embedded credentials, query, fragment, or path. This is an +operator trust declaration, not a user-controlled connector bypass. Operators +remain responsible for DNS, certificate, network-egress, bucket-policy, +versioning, and lifecycle controls. -## Remaining Platform Work +## Capacity -The one-host installer does not yet provide: +- Scale API replicas only within the PostgreSQL connection budget. +- Scale workers by queue, with upper bounds based on external provider limits. +- Keep one fenced scheduler rather than load-balancing schedulers. +- Increase WebUI replicas for asset/proxy capacity. +- Measure request latency, database query time and locks, active connections, + queue age, retry rate, storage latency, and provider throttling before adding + replicas. -- a Kubernetes, Nomad, or Swarm deployment export/profile; -- distributed deployment locks and fenced scheduler leadership; -- worker registration, composition-skew reporting, and drain controls; -- managed PostgreSQL, Redis, or multi-node Garage high availability; +Workers compete for Redis-backed work and are not placed behind a load balancer. +SMTP, IMAP, directory, connector, workflow, dataflow, and reporting queues often +hit external-system limits before host CPU is exhausted. + +## What This Does Not Claim + +The implemented contract provides stateless runtime placement, shared artifact +access, node visibility, drain controls, migration serialization, and scheduler +fencing. It does not by itself provide: + +- a highly available PostgreSQL, Redis, or object-store deployment; +- automatic PostgreSQL backup, point-in-time recovery, or restore verification; - autoscaling policy; -- managed TLS certificate issuance and renewal; -- measured multi-replica and failover integration evidence. +- central logs, metrics, traces, or alert routing; +- managed ingress certificates; +- automatic reconciliation of every possible module side effect; +- a service-level availability guarantee. -These are separate production capabilities. The local load balancer and -replica model provide the contract they should implement, but they do not by -themselves make a cluster highly available. +Those are deployment and module-adoption requirements. Before claiming high +availability, drill replica loss, rolling replacement, session continuity, job +redelivery, scheduler failover, migration exclusion, object-store outage, and a +coordinated database/object/key restore. Recovery rules and evidence are +defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md). diff --git a/Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY.md b/Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY.md index a2deb43..93d0549 100644 --- a/Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY.md +++ b/Repo-docs-SYSTEM-ADMINISTRATOR-LIFECYCLE-USER-STORY.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md`. > Origin: `repository`. @@ -146,6 +146,17 @@ The canonical backlog item is ## Implementation slices +Implementation status as of the current source tree: + +- Slice 6 has a working application-tier foundation: state profiles, shared + object storage, runtime node registration/heartbeats/drain, fenced scheduler, + migration serialization, exact-head startup waiting, Ops visibility, and a + Kubernetes export. Production acceptance still requires topology-specific + failover and restore drills. +- The recovery foundation for slices 4 and 5 is implemented as a Core recovery + ledger and deployment operation journal. Automatic database backup and broad + adoption by module-owned external effects remain open work. + 1. **Reproducible Core-only distribution.** Publish pinned multi-architecture images, signed distribution manifest, Core-only Compose profile, bootstrap preflight, generated secrets, readiness, and idempotent rerun/repair. @@ -162,9 +173,11 @@ The canonical backlog item is 5. **Safe module update.** Add drain/maintenance coordination, backup gate, migration compatibility window, reconnectable progress, health verification, retry/recovery, and update notification. -6. **Stateless replica profile.** Remove remaining local-runtime assumptions, - expose role-specific commands/images, implement worker registration/drain, - and prove multiple API and worker replicas against shared dependencies. +6. **Stateless replica profile.** Continue module adoption and operational + proof for the implemented role commands, shared-state validation, runtime + registration/drain, fenced scheduler, and Kubernetes application-tier + export. Prove multiple API and worker replicas against the target shared + dependencies. 7. **Configuration revision model.** Define provider export/import schemas, canonical serialization, secret references, validation/diff, immutable revision storage, audit, apply, and undo-as-new-revision.