feat: add bounded Kubernetes runtime verification
Dependency Audit / dependency-audit (push) Failing after 9s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Failing after 8s

This commit is contained in:
2026-08-02 05:30:26 +02:00
parent 50e9607e72
commit ed31409034
6 changed files with 1094 additions and 11 deletions
+47 -1
View File
@@ -150,8 +150,29 @@ versioning, and lifecycle controls.
## Capacity
- Scale API replicas only within the PostgreSQL connection budget.
- Set `GOVOPLAN_DB_CONNECTION_LIMIT` to the PostgreSQL role's effective
connection limit. The Kubernetes export reserves
`GOVOPLAN_DB_CONNECTION_RESERVE` connections and rejects a topology whose
calculated rolling-update peak would exceed the remainder. The calculation
includes API pools, every Celery parent and prefork child, the scheduler,
migration, and one surge replica per deployment. Role-specific pool and
overflow values are emitted into each workload rather than inherited from one
unconstrained global default.
- Scale workers by queue, with upper bounds based on external provider limits.
`GOVOPLAN_WORKER_POOLS` may contain a JSON list of exact queue owners, for
example:
```json
[
{"name":"delivery","queues":["send_email","append_sent"],"replicas":2,"concurrency":2},
{"name":"platform","queues":["events","workflow","default"],"replicas":2,"concurrency":2}
]
```
Pool replica totals must equal `replicas.worker`, and the pools must cover
`CELERY_QUEUES` exactly without duplicate ownership. Each pool receives its
own Deployment, disruption budget, topology-spread selector, runtime identity,
and declared concurrency.
- Keep one fenced scheduler rather than load-balancing schedulers.
- Increase WebUI replicas for asset/proxy capacity.
- Measure request latency, database query time and locks, active connections,
@@ -181,3 +202,28 @@ availability, drill replica loss, rolling replacement, session continuity, job
redelivery, scheduler failover, migration exclusion, object-store outage, and a
coordinated database/object/key restore. Recovery rules and evidence are
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
## Live Multi-Host Evidence
After deploying a pinned release on at least two Kubernetes nodes, create an API
key with Ops read scope and run:
```bash
export GOVOPLAN_OPS_API_KEY='...'
python tools/deployment/govoplan-deploy.py verify-kubernetes \
--directory /srv/govoplan/installation \
--namespace govoplan
```
The command fails unless API and WebUI pods are ready on at least two nodes,
all rendered deployments are available, Ops reports a consistent release and
module composition, every declared worker queue is served, and the calculated
database peak remains below its budget. It writes a private, sanitized JSON
record under the installation evidence directory and never retains the API key.
Use `--exercise-api-pod-loss` in an approved drill window to delete one API pod,
observe the public readiness path continuously, and record its replacement.
This proves the bounded stateless-node-loss slice only. Session continuity,
accepted-job redelivery, state-service failover, and coordinated restore remain
separate target exercises whose signed evidence is governed by
`docs/TARGET_MATURITY_EVIDENCE_RUNBOOK.md` and GovOPlaN #37.