feat: add bounded Kubernetes runtime verification
This commit is contained in:
@@ -150,8 +150,29 @@ versioning, and lifecycle controls.
|
||||
|
||||
## Capacity
|
||||
|
||||
- Scale API replicas only within the PostgreSQL connection budget.
|
||||
- Set `GOVOPLAN_DB_CONNECTION_LIMIT` to the PostgreSQL role's effective
|
||||
connection limit. The Kubernetes export reserves
|
||||
`GOVOPLAN_DB_CONNECTION_RESERVE` connections and rejects a topology whose
|
||||
calculated rolling-update peak would exceed the remainder. The calculation
|
||||
includes API pools, every Celery parent and prefork child, the scheduler,
|
||||
migration, and one surge replica per deployment. Role-specific pool and
|
||||
overflow values are emitted into each workload rather than inherited from one
|
||||
unconstrained global default.
|
||||
- Scale workers by queue, with upper bounds based on external provider limits.
|
||||
`GOVOPLAN_WORKER_POOLS` may contain a JSON list of exact queue owners, for
|
||||
example:
|
||||
|
||||
```json
|
||||
[
|
||||
{"name":"delivery","queues":["send_email","append_sent"],"replicas":2,"concurrency":2},
|
||||
{"name":"platform","queues":["events","workflow","default"],"replicas":2,"concurrency":2}
|
||||
]
|
||||
```
|
||||
|
||||
Pool replica totals must equal `replicas.worker`, and the pools must cover
|
||||
`CELERY_QUEUES` exactly without duplicate ownership. Each pool receives its
|
||||
own Deployment, disruption budget, topology-spread selector, runtime identity,
|
||||
and declared concurrency.
|
||||
- Keep one fenced scheduler rather than load-balancing schedulers.
|
||||
- Increase WebUI replicas for asset/proxy capacity.
|
||||
- Measure request latency, database query time and locks, active connections,
|
||||
@@ -181,3 +202,28 @@ availability, drill replica loss, rolling replacement, session continuity, job
|
||||
redelivery, scheduler failover, migration exclusion, object-store outage, and a
|
||||
coordinated database/object/key restore. Recovery rules and evidence are
|
||||
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
|
||||
|
||||
## Live Multi-Host Evidence
|
||||
|
||||
After deploying a pinned release on at least two Kubernetes nodes, create an API
|
||||
key with Ops read scope and run:
|
||||
|
||||
```bash
|
||||
export GOVOPLAN_OPS_API_KEY='...'
|
||||
python tools/deployment/govoplan-deploy.py verify-kubernetes \
|
||||
--directory /srv/govoplan/installation \
|
||||
--namespace govoplan
|
||||
```
|
||||
|
||||
The command fails unless API and WebUI pods are ready on at least two nodes,
|
||||
all rendered deployments are available, Ops reports a consistent release and
|
||||
module composition, every declared worker queue is served, and the calculated
|
||||
database peak remains below its budget. It writes a private, sanitized JSON
|
||||
record under the installation evidence directory and never retains the API key.
|
||||
|
||||
Use `--exercise-api-pod-loss` in an approved drill window to delete one API pod,
|
||||
observe the public readiness path continuously, and record its replacement.
|
||||
This proves the bounded stateless-node-loss slice only. Session continuity,
|
||||
accepted-job redelivery, state-service failover, and coordinated restore remain
|
||||
separate target exercises whose signed evidence is governed by
|
||||
`docs/TARGET_MATURITY_EVIDENCE_RUNBOOK.md` and GovOPlaN #37.
|
||||
|
||||
Reference in New Issue
Block a user