[Task] Add multi-host orchestrator profile, fencing, and worker lifecycle #27

New Issue

2026-07-30T18:56:08+02:00

zemion commented

2026-07-30 18:56:08 +02:00

Objective

Extend the one-host replica contract delivered under #13 into a supported multi-host deployment profile without turning the Compose installer into a proprietary scheduler.

Parent story: GovOPlaN/govoplan#13.

Current baseline

The generated one-host Compose topology has managed HAProxy discovery and balancing for WebUI/API replicas.
API, WebUI, and Celery worker counts are declarative installation state.
Migrations and Celery beat remain singleton roles.
Managed Garage is available as persistent single-node S3-compatible storage.
docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md defines the role and shared-state boundary.

Scope

Select and implement one supported orchestrator export/profile (Kubernetes, Nomad, or Swarm) using the existing immutable role commands, environment allowlist, and health contracts.
Place WebUI/API replicas across nodes behind health-aware services and a TLS ingress.
Keep PostgreSQL, Redis, object storage, secrets, and configuration shared and external to stateless runtime nodes.
Add distributed fencing for migrations, module lifecycle mutations, and the singleton scheduler.
Add worker identity, software-composition heartbeat, queue declaration, drain, and safe retirement/requeue behavior.
Detect composition/configuration skew across runtime nodes.
Define bounded queue-specific worker pools and explicit database connection budgets.
Prove rolling replacement and loss of one stateless node without session, job, or request corruption.

Acceptance criteria

One documented orchestrator profile deploys equivalent API, WebUI, worker, migration, and scheduler roles across at least two nodes.
Public and internal traffic use health-aware service discovery; no application container receives the orchestrator or Docker control socket.
Only one migration/module mutation or scheduler lease can commit, including after lease expiry and network delay.
Operators can inspect worker identity, queues, version/composition, heartbeat, and drain state.
Node loss and rolling replacement preserve sessions, throttling, idempotency, and accepted jobs.
Stateful-service HA expectations and unsupported single-node modes are explicit in preflight and documentation.
Integration evidence covers failover, composition skew, worker draining, stale fencing tokens, and database connection ceilings.

Non-goals

Building a custom cluster scheduler.
Claiming multi-node Garage, PostgreSQL, or Redis HA from the one-host Compose profile.
Autoscaling before workload metrics and hard capacity ceilings exist.

## Objective Extend the one-host replica contract delivered under #13 into a supported multi-host deployment profile without turning the Compose installer into a proprietary scheduler. Parent story: GovOPlaN/govoplan#13. ## Current baseline - The generated one-host Compose topology has managed HAProxy discovery and balancing for WebUI/API replicas. - API, WebUI, and Celery worker counts are declarative installation state. - Migrations and Celery beat remain singleton roles. - Managed Garage is available as persistent single-node S3-compatible storage. - `docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md` defines the role and shared-state boundary. ## Scope - Select and implement one supported orchestrator export/profile (Kubernetes, Nomad, or Swarm) using the existing immutable role commands, environment allowlist, and health contracts. - Place WebUI/API replicas across nodes behind health-aware services and a TLS ingress. - Keep PostgreSQL, Redis, object storage, secrets, and configuration shared and external to stateless runtime nodes. - Add distributed fencing for migrations, module lifecycle mutations, and the singleton scheduler. - Add worker identity, software-composition heartbeat, queue declaration, drain, and safe retirement/requeue behavior. - Detect composition/configuration skew across runtime nodes. - Define bounded queue-specific worker pools and explicit database connection budgets. - Prove rolling replacement and loss of one stateless node without session, job, or request corruption. ## Acceptance criteria - [ ] One documented orchestrator profile deploys equivalent API, WebUI, worker, migration, and scheduler roles across at least two nodes. - [ ] Public and internal traffic use health-aware service discovery; no application container receives the orchestrator or Docker control socket. - [ ] Only one migration/module mutation or scheduler lease can commit, including after lease expiry and network delay. - [ ] Operators can inspect worker identity, queues, version/composition, heartbeat, and drain state. - [ ] Node loss and rolling replacement preserve sessions, throttling, idempotency, and accepted jobs. - [ ] Stateful-service HA expectations and unsupported single-node modes are explicit in preflight and documentation. - [ ] Integration evidence covers failover, composition skew, worker draining, stale fencing tokens, and database connection ceilings. ## Non-goals - Building a custom cluster scheduler. - Claiming multi-node Garage, PostgreSQL, or Redis HA from the one-host Compose profile. - Autoscaling before workload metrics and hard capacity ceilings exist.

zemion added the module/ops

type

task

area/security codex/ready area/release area/devex

module/core labels 2026-07-30 18:56:08 +02:00

zemion referenced this issue

2026-07-30 18:56:33 +02:00

[User Story] One-command Core bootstrap, module lifecycle, scale-out, and environment promotion #13

zemion commented

2026-08-01 01:48:17 +02:00

Codex State: progress

Summary

Implemented the stateless multi-host application-tier foundation locally (pending push): external PostgreSQL/Redis/S3 validation, deterministic Kubernetes export, health-aware API/WebUI services, non-root runtime settings, exact-head init waits, and a release-specific migration Job.
Added PostgreSQL runtime-node identity/heartbeat/composition/drain state, Ops projection and audited drain controls, migration advisory serialization, and a renewable fencing-token scheduler runner.
Moved durable Campaign generated EML to Core shared object storage and blocked in-place package mutation for shared immutable clusters.

Changed Files

tools/deployment/govoplan_deploy/kubernetes.py
docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

Verification

tools/checks/check-focused.sh: passed, including 54 manifest contracts and 35 WebUI module permutations

Next / Blocked

Run the generated profile on at least two real cluster nodes and record node-loss, rolling-replacement, active-job drain/requeue, session continuity, composition-skew, and database-connection-ceiling evidence before closing.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented the stateless multi-host application-tier foundation locally (pending push): external PostgreSQL/Redis/S3 validation, deterministic Kubernetes export, health-aware API/WebUI services, non-root runtime settings, exact-head init waits, and a release-specific migration Job. - Added PostgreSQL runtime-node identity/heartbeat/composition/drain state, Ops projection and audited drain controls, migration advisory serialization, and a renewable fencing-token scheduler runner. - Moved durable Campaign generated EML to Core shared object storage and blocked in-place package mutation for shared immutable clusters. ### Changed Files - `tools/deployment/govoplan_deploy/kubernetes.py` - `docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md` ### Verification - `tools/checks/check-focused.sh: passed, including 54 manifest contracts and 35 WebUI module permutations` ### Next / Blocked - Run the generated profile on at least two real cluster nodes and record node-loss, rolling-replacement, active-job drain/requeue, session continuity, composition-skew, and database-connection-ceiling evidence before closing. Suggested status label: `status/in-progress`

zemion referenced this issue from GovOPlaN/govoplan-campaign

2026-08-01 01:48:58 +02:00

[Debt] Reconcile orphaned Campaign build artifacts in shared object storage #91

zemion commented

2026-08-01 02:04:14 +02:00

Codex State: progress

Summary

Implemented the stateless shared-state runtime profile, immutable digest-pinned Kubernetes export, runtime node registration/drain, and fenced singleton scheduling.
API readiness and worker consumers now fail closed when runtime coordination cannot be proven.

Changed Files

docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md
tools/deployment/govoplan_deploy/kubernetes.py

Verification

33 deployment-installer tests passed

Next / Blocked

Production acceptance still requires real multi-node failure drills and externally operated HA PostgreSQL, Redis, and object storage.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented the stateless shared-state runtime profile, immutable digest-pinned Kubernetes export, runtime node registration/drain, and fenced singleton scheduling. - API readiness and worker consumers now fail closed when runtime coordination cannot be proven. ### Changed Files - `docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md` - `tools/deployment/govoplan_deploy/kubernetes.py` ### Verification - `33 deployment-installer tests passed` ### Next / Blocked - Production acceptance still requires real multi-node failure drills and externally operated HA PostgreSQL, Redis, and object storage. Suggested status label: `status/in-progress`

zemion commented

2026-08-02 05:32:57 +02:00

All source-controlled implementation that can be proven locally is pushed:

Kubernetes export now has topology-spread API/WebUI replicas, health-aware Services/Ingress, singleton migration/scheduler roles, queue-specific worker Deployments/PDBs, explicit concurrency, and enforceable database connection budgets (GovOPlaN/govoplan@ed31409).
Core resets inherited SQLAlchemy pools in Celery children, publishes pool/concurrency identity, computes order-independent composition hashes, and exposes capacity settings (govoplan-core@d6255f9).
Ops checks expected replica composition/version, queue coverage, worker pools, and database headroom (govoplan-ops@b464d01).
govoplan-deploy verify-kubernetes records sanitized evidence and can perform a bounded API-pod-loss drill while observing readiness and replacement. It requires at least two ready nodes and never stores the API key.

Verification: deployment/inventory/publication suite 54 passed, 2 subtests; Core contract/runtime suite 128 passed, 211 subtests; Ops 5 passed; production WebUI build passed.

This issue remains open only for target-environment evidence: deploy a pinned release on a real two-node Kubernetes cluster, then run:

export GOVOPLAN_OPS_API_KEY='...'
python tools/deployment/govoplan-deploy.py verify-kubernetes \
  --directory /srv/govoplan/installation \
  --namespace govoplan \
  --exercise-api-pod-loss

The broader session/job/state-service recovery evidence remains tracked by #37. Source and local tests cannot manufacture that operational proof, so the issue is moved to needs-info/needs-human rather than closed.

All source-controlled implementation that can be proven locally is pushed: - Kubernetes export now has topology-spread API/WebUI replicas, health-aware Services/Ingress, singleton migration/scheduler roles, queue-specific worker Deployments/PDBs, explicit concurrency, and enforceable database connection budgets (`GovOPlaN/govoplan@ed31409`). - Core resets inherited SQLAlchemy pools in Celery children, publishes pool/concurrency identity, computes order-independent composition hashes, and exposes capacity settings (`govoplan-core@d6255f9`). - Ops checks expected replica composition/version, queue coverage, worker pools, and database headroom (`govoplan-ops@b464d01`). - `govoplan-deploy verify-kubernetes` records sanitized evidence and can perform a bounded API-pod-loss drill while observing readiness and replacement. It requires at least two ready nodes and never stores the API key. Verification: deployment/inventory/publication suite `54 passed, 2 subtests`; Core contract/runtime suite `128 passed, 211 subtests`; Ops `5 passed`; production WebUI build passed. This issue remains open only for target-environment evidence: deploy a pinned release on a real two-node Kubernetes cluster, then run: ```bash export GOVOPLAN_OPS_API_KEY='...' python tools/deployment/govoplan-deploy.py verify-kubernetes \ --directory /srv/govoplan/installation \ --namespace govoplan \ --exercise-api-pod-loss ``` The broader session/job/state-service recovery evidence remains tracked by #37. Source and local tests cannot manufacture that operational proof, so the issue is moved to `needs-info`/`needs-human` rather than closed.

zemion added codex/needs-human

and removed

codex/ready labels 2026-08-02 05:33:09 +02:00

zemion commented

2026-08-03 20:20:13 +02:00

The previously missing immutable subject now exists and is independently verified:

Release: GovOPlaN v0.1.14
Source/tag/release target: 1f039dd39c1ce2672f4978c8abc6dff862ef1445
Runtime Distribution: run #459
Signed manifest SHA-256: d703267e01855dee63200cb20921c91c3f95fbff550c8ca76e9a35cba3f69109
Signing key: runtime-distribution-2026-01
API index: git.add-ideas.de/govoplan/runtime-api@sha256:197ed01790986f2bc927eaa5d8348fa118702e5d2dc05feb851fc2643c23764a
WebUI index: git.add-ideas.de/govoplan/runtime-web@sha256:e936cca124f1fad29a067834cf17627d4c236410fdc3fa129e0ccb26b8193812

The release proves both architecture images and the one-host managed-ingress boundary. This pins the exact subject for the live Kubernetes acceptance run but does not replace it.

Remaining target inputs are now concrete: a reachable Kubernetes cluster with at least two schedulable nodes; a namespace and public target; external shared PostgreSQL, Redis, and S3 bindings; the adopted v0.1.14 installation state; and a temporary Ops read API key. During an approved node-loss window, run:

export GOVOPLAN_OPS_API_KEY='...'
python tools/deployment/govoplan-deploy.py verify-kubernetes \
  --directory /srv/govoplan/installation \
  --namespace govoplan \
  --exercise-api-pod-loss

The current workspace has no Kubernetes client, kubeconfig, cluster credentials, Ops API key, or target endpoint, so it cannot truthfully produce this operational receipt. Once supplied on the target, the command fails closed unless API/WebUI pods span at least two nodes, all deployments are available, Ops reports the pinned release/composition, every worker queue is covered, database capacity is within budget, and the API pod-loss drill preserves readiness. Attach the sanitized receipt here; broader state-service/session/job recovery remains #37.

The previously missing immutable subject now exists and is independently verified: - Release: [GovOPlaN v0.1.14](https://git.add-ideas.de/GovOPlaN/govoplan/releases/tag/v0.1.14) - Source/tag/release target: `1f039dd39c1ce2672f4978c8abc6dff862ef1445` - Runtime Distribution: [run #459](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/459) - Signed manifest SHA-256: `d703267e01855dee63200cb20921c91c3f95fbff550c8ca76e9a35cba3f69109` - Signing key: `runtime-distribution-2026-01` - API index: `git.add-ideas.de/govoplan/runtime-api@sha256:197ed01790986f2bc927eaa5d8348fa118702e5d2dc05feb851fc2643c23764a` - WebUI index: `git.add-ideas.de/govoplan/runtime-web@sha256:e936cca124f1fad29a067834cf17627d4c236410fdc3fa129e0ccb26b8193812` The release proves both architecture images and the one-host managed-ingress boundary. This pins the exact subject for the live Kubernetes acceptance run but does not replace it. Remaining target inputs are now concrete: a reachable Kubernetes cluster with at least two schedulable nodes; a namespace and public target; external shared PostgreSQL, Redis, and S3 bindings; the adopted v0.1.14 installation state; and a temporary Ops read API key. During an approved node-loss window, run: ```bash export GOVOPLAN_OPS_API_KEY='...' python tools/deployment/govoplan-deploy.py verify-kubernetes \ --directory /srv/govoplan/installation \ --namespace govoplan \ --exercise-api-pod-loss ``` The current workspace has no Kubernetes client, kubeconfig, cluster credentials, Ops API key, or target endpoint, so it cannot truthfully produce this operational receipt. Once supplied on the target, the command fails closed unless API/WebUI pods span at least two nodes, all deployments are available, Ops reports the pinned release/composition, every worker queue is covered, database capacity is within budget, and the API pod-loss drill preserves readiness. Attach the sanitized receipt here; broader state-service/session/job recovery remains #37.

zemion referenced this issue

2026-08-03 20:20:14 +02:00

[Task] Produce signed target maturity and recovery evidence for a pinned release #37

zemion commented

2026-08-04 14:37:56 +02:00

Codex State: needs-info

Summary

The exact two-node target handoff and least-privilege evidence collector contract are documented in PRODUCTION_TARGET_HANDOFF.md (be51a9c).
The verifier requires two ready schedulable nodes, API and WebUI spread across both, runtime/queue/database consistency, and an API pod-loss drill without readiness failure.

Changed Files

docs/PRODUCTION_TARGET_HANDOFF.md
docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

Verification

tests.test_assessment_authority_keypair: passed

Next / Blocked

Target owner must provide a secure kubeconfig, two independent worker failure domains, external PostgreSQL/Redis/S3, ingress/TLS inputs, a scoped ops:operations:read key, and an approved pod-loss window.

Suggested status label: status/needs-info

## Codex State: needs-info ### Summary - The exact two-node target handoff and least-privilege evidence collector contract are documented in PRODUCTION_TARGET_HANDOFF.md (be51a9c). - The verifier requires two ready schedulable nodes, API and WebUI spread across both, runtime/queue/database consistency, and an API pod-loss drill without readiness failure. ### Changed Files - `docs/PRODUCTION_TARGET_HANDOFF.md` - `docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md` ### Verification - `tests.test_assessment_authority_keypair: passed` ### Next / Blocked - Target owner must provide a secure kubeconfig, two independent worker failure domains, external PostgreSQL/Redis/S3, ingress/TLS inputs, a scoped ops:operations:read key, and an approved pod-loss window. Suggested status label: `status/needs-info`

zemion commented

2026-08-04 16:55:07 +02:00

Codex State: needs-info

Summary

GovOPlaN v0.1.15 is now the preferred immutable drill subject: protected commit a24c94435e, signed manifest SHA-256 09ac1ade6ede4958bab0dfb7fd8f99246f4d991846308db1f410b25b46267840, and passing amd64/arm64 runtime receipts.
All repository-controlled Kubernetes rendering, topology verification, Ops readiness, queue/composition checks, connection budgeting, and API pod-loss exercise are implemented.

Next / Blocked

Target owner must provide a secure kubeconfig path for two workers in independent failure domains, target identity/TLS/ingress details, shared PostgreSQL/Redis/S3 services and CA paths, runtime secret-file paths, a short-lived ops:operations:read API key file, and an approved pod-deletion drill window as documented in docs/PRODUCTION_TARGET_HANDOFF.md.

Suggested status label: status/needs-info

## Codex State: needs-info ### Summary - GovOPlaN v0.1.15 is now the preferred immutable drill subject: protected commit a24c94435eecd6c96f3d4dae09675aa0c6a954b7, signed manifest SHA-256 09ac1ade6ede4958bab0dfb7fd8f99246f4d991846308db1f410b25b46267840, and passing amd64/arm64 runtime receipts. - All repository-controlled Kubernetes rendering, topology verification, Ops readiness, queue/composition checks, connection budgeting, and API pod-loss exercise are implemented. ### Next / Blocked - Target owner must provide a secure kubeconfig path for two workers in independent failure domains, target identity/TLS/ingress details, shared PostgreSQL/Redis/S3 services and CA paths, runtime secret-file paths, a short-lived ops:operations:read API key file, and an approved pod-deletion drill window as documented in docs/PRODUCTION_TARGET_HANDOFF.md. Suggested status label: `status/needs-info`

zemion commented

2026-08-05 17:42:07 +02:00

Codex State: progress

Summary

Implemented a guarded libvirt/K3s VM lifecycle for same-host rehearsal and independently hosted acceptance targets.
Added strict topology validation, pinned Ubuntu/K3s/release inputs, signed release adoption, external PostgreSQL/Redis/Garage state, private-CA trust, serial updates, pause/resume, teardown and live evidence collection.

Changed Files

tools/lab/govoplan-lab.py
tools/lab/govoplan_lab
tools/lab/govoplan-lab.example.toml
tools/lab/govoplan-lab.acceptance.example.toml
docs/KUBERNETES_TEST_LAB.md
tools/deployment/govoplan_deploy/kubernetes.py

Verification

python -m unittest tests.test_kubernetes_lab tests.test_deployment_installer: 47 tests passed
tools/checks/check-focused.sh: passed, including 50 WebUI module permutations
Pinned v0.1.15 manifest, keyring and K3s installer digests reverified against their HTTPS sources

Next / Blocked

Run the acceptance inventory on workers backed by two genuinely independent physical hypervisors, enroll an operator, and execute verify --exercise-api-pod-loss.
Keep #27 open until that live target evidence passes; a one-host run is explicitly classified as rehearsal evidence.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented a guarded libvirt/K3s VM lifecycle for same-host rehearsal and independently hosted acceptance targets. - Added strict topology validation, pinned Ubuntu/K3s/release inputs, signed release adoption, external PostgreSQL/Redis/Garage state, private-CA trust, serial updates, pause/resume, teardown and live evidence collection. ### Changed Files - `tools/lab/govoplan-lab.py` - `tools/lab/govoplan_lab` - `tools/lab/govoplan-lab.example.toml` - `tools/lab/govoplan-lab.acceptance.example.toml` - `docs/KUBERNETES_TEST_LAB.md` - `tools/deployment/govoplan_deploy/kubernetes.py` ### Verification - `python -m unittest tests.test_kubernetes_lab tests.test_deployment_installer: 47 tests passed` - `tools/checks/check-focused.sh: passed, including 50 WebUI module permutations` - `Pinned v0.1.15 manifest, keyring and K3s installer digests reverified against their HTTPS sources` ### Next / Blocked - Run the acceptance inventory on workers backed by two genuinely independent physical hypervisors, enroll an operator, and execute verify --exercise-api-pod-loss. - Keep #27 open until that live target evidence passes; a one-host run is explicitly classified as rehearsal evidence. Suggested status label: `status/in-progress`

zemion commented

2026-08-05 19:06:11 +02:00

Codex State: progress

Summary

Provisioned a reproducible four-VM libvirt/K3s rehearsal on the dedicated 192.168.123.0/24 lab network, with PostgreSQL, Redis, Garage, GreenMail, TLS ingress and signed GovOPlaN v0.1.15 runtime images.
Completed first-administrator enrollment and proved that the one-time bootstrap credential retired and its local artifact was removed.
Collected a passing bounded API-pod-loss rehearsal: API and WebUI each span two worker nodes, runtime composition/version and queues are coherent, database peak is 43 of 90 usable connections, and replacement caused zero observed readiness failures.
The rehearsal exposed and source-fixed migration-head dependency resolution, WebUI upstream configuration, Celery Beat read-only-root behavior, rollout topology spread, kubectl mutation handling and API runtime-version identity.

Changed Files

tools/lab/
tools/deployment/govoplan_deploy/cluster_evidence.py
tools/deployment/govoplan_deploy/kubernetes.py
tools/release/runtime/
docs/KUBERNETES_TEST_LAB.md
../govoplan-core/src/govoplan_core/db/migrations.py
../govoplan-core/src/govoplan_core/server/default_config.py

Verification

66 deployment/runtime/lab unit tests passed
8 Core runtime-agent/database-wait tests passed
tools/checks/check-focused.sh passed, including 50 WebUI module permutations
Sanitized evidence: ~/.local/share/govoplan/labs/govoplan-k8s-lab/evidence/kubernetes-multi-host.json (passed)

Next / Blocked

This is same-host rehearsal evidence and does not satisfy independent failure-domain acceptance for #27.
Publish a new signed release containing these source fixes, redeploy it without live compatibility overrides, and repeat on a real two-host Kubernetes target.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Provisioned a reproducible four-VM libvirt/K3s rehearsal on the dedicated 192.168.123.0/24 lab network, with PostgreSQL, Redis, Garage, GreenMail, TLS ingress and signed GovOPlaN v0.1.15 runtime images. - Completed first-administrator enrollment and proved that the one-time bootstrap credential retired and its local artifact was removed. - Collected a passing bounded API-pod-loss rehearsal: API and WebUI each span two worker nodes, runtime composition/version and queues are coherent, database peak is 43 of 90 usable connections, and replacement caused zero observed readiness failures. - The rehearsal exposed and source-fixed migration-head dependency resolution, WebUI upstream configuration, Celery Beat read-only-root behavior, rollout topology spread, kubectl mutation handling and API runtime-version identity. ### Changed Files - `tools/lab/` - `tools/deployment/govoplan_deploy/cluster_evidence.py` - `tools/deployment/govoplan_deploy/kubernetes.py` - `tools/release/runtime/` - `docs/KUBERNETES_TEST_LAB.md` - `../govoplan-core/src/govoplan_core/db/migrations.py` - `../govoplan-core/src/govoplan_core/server/default_config.py` ### Verification - `66 deployment/runtime/lab unit tests passed` - `8 Core runtime-agent/database-wait tests passed` - `tools/checks/check-focused.sh passed, including 50 WebUI module permutations` - `Sanitized evidence: ~/.local/share/govoplan/labs/govoplan-k8s-lab/evidence/kubernetes-multi-host.json (passed)` ### Next / Blocked - This is same-host rehearsal evidence and does not satisfy independent failure-domain acceptance for #27. - Publish a new signed release containing these source fixes, redeploy it without live compatibility overrides, and repeat on a real two-host Kubernetes target. Suggested status label: `status/in-progress`

zemion commented

2026-08-05 22:00:55 +02:00

Codex State: target rehearsal passed

Immutable subject

GovOPlaN release: v0.1.18
Signed distribution manifest SHA-256: 7003232331780add84a167169398918c707b19a9f8e8329c0d105184ee4d6009
API OCI index: git.add-ideas.de/govoplan/runtime-api@sha256:be9fb2b14b03232b820ee9bdb57d4ad34b753f28765813eac728bf1869ccb8dc
Web OCI index: git.add-ideas.de/govoplan/runtime-web@sha256:54a1a6ab0a304f22a852d6fa131c598de4ec8287a5a5a880e36b2a3f0217f2d6
Runtime Distribution run #842 passed clean amd64 and arm64 builds, migrations, non-root API/WebUI and worker smoke.
The formerly ignored WebUI entrypoint is tracked in the release tag and was exercised by those clean builds.

Live rehearsal

A four-VM K3s rehearsal deployed the signed release with one control, two workers and one external shared-state VM. API, WebUI and workers span both Kubernetes workers. All readiness, deployment, composition, version, queue and database-budget checks passed.

The first pod-loss run exposed a real endpoint-termination race. The generated API workload now has a 10-second pre-stop drain and 30-second termination grace period (GovOPlaN/govoplan@389df7c). A repeated public-path drill then deleted and replaced one API pod with zero observed readiness failures.

Private sanitized receipt:

file: kubernetes-multi-host-v0.1.18-20260805T195355Z.json
SHA-256: 259f8e33b7b7a7161e1278f3ae94d52d99d88fcc2659fd1f1f8a2808cfb08abc
result: passed
ready schedulable nodes: 3
API/WebUI spread: 2 ready pods on 2 worker nodes each
database budget: 43 peak of 90 available connections
replacement: passed, readiness failures: 0

The strict private-CA profile was also corrected and verified with openssl verify -x509_strict (GovOPlaN/govoplan@3b9ae90).

Remaining acceptance boundary

This inventory is deliberately mode = rehearsal: both worker VMs share the same physical hypervisor and failure domain. It proves Kubernetes behavior and removes the previous release/tooling blockers, but it cannot prove independent-host failure.

Keep this issue open until the same pinned process is run with at least two independently controlled physical hypervisors or availability zones and the operator supplies that out-of-band topology evidence. The broader session/job/state-service and maturity evidence remains #37.

## Codex State: target rehearsal passed ### Immutable subject - GovOPlaN release: `v0.1.18` - Signed distribution manifest SHA-256: `7003232331780add84a167169398918c707b19a9f8e8329c0d105184ee4d6009` - API OCI index: `git.add-ideas.de/govoplan/runtime-api@sha256:be9fb2b14b03232b820ee9bdb57d4ad34b753f28765813eac728bf1869ccb8dc` - Web OCI index: `git.add-ideas.de/govoplan/runtime-web@sha256:54a1a6ab0a304f22a852d6fa131c598de4ec8287a5a5a880e36b2a3f0217f2d6` - Runtime Distribution run #842 passed clean amd64 and arm64 builds, migrations, non-root API/WebUI and worker smoke. - The formerly ignored WebUI entrypoint is tracked in the release tag and was exercised by those clean builds. ### Live rehearsal A four-VM K3s rehearsal deployed the signed release with one control, two workers and one external shared-state VM. API, WebUI and workers span both Kubernetes workers. All readiness, deployment, composition, version, queue and database-budget checks passed. The first pod-loss run exposed a real endpoint-termination race. The generated API workload now has a 10-second pre-stop drain and 30-second termination grace period (`GovOPlaN/govoplan@389df7c`). A repeated public-path drill then deleted and replaced one API pod with **zero observed readiness failures**. Private sanitized receipt: - file: `kubernetes-multi-host-v0.1.18-20260805T195355Z.json` - SHA-256: `259f8e33b7b7a7161e1278f3ae94d52d99d88fcc2659fd1f1f8a2808cfb08abc` - result: `passed` - ready schedulable nodes: 3 - API/WebUI spread: 2 ready pods on 2 worker nodes each - database budget: 43 peak of 90 available connections - replacement: passed, readiness failures: 0 The strict private-CA profile was also corrected and verified with `openssl verify -x509_strict` (`GovOPlaN/govoplan@3b9ae90`). ### Remaining acceptance boundary This inventory is deliberately `mode = rehearsal`: both worker VMs share the same physical hypervisor and failure domain. It proves Kubernetes behavior and removes the previous release/tooling blockers, but it cannot prove independent-host failure. Keep this issue open until the same pinned process is run with at least two independently controlled physical hypervisors or availability zones and the operator supplies that out-of-band topology evidence. The broader session/job/state-service and maturity evidence remains #37.

zemion referenced this issue

2026-08-05 22:01:15 +02:00

[Task] Produce signed target maturity and recovery evidence for a pinned release #37

zemion referenced this issue

2026-08-05 22:03:04 +02:00

[User Story] One-command Core bootstrap, module lifecycle, scale-out, and environment promotion #13

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan#27