Automate worker runtime delivery drill
Dependency Audit / dependency-audit (push) Failing after 9s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Failing after 8s

This commit is contained in:
2026-08-03 00:20:57 +02:00
parent be4410ef1a
commit d3713bf2ee
4 changed files with 477 additions and 0 deletions
+24
View File
@@ -203,6 +203,30 @@ redelivery, scheduler failover, migration exclusion, object-store outage, and a
coordinated database/object/key restore. Recovery rules and evidence are
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
## Worker Delivery Evidence
The module-matrix workflow runs `tools/checks/worker-runtime-drill.py` against a
real isolated Redis database. The drill starts supervised Celery worker
processes and records four guarantees without accessing tenant data:
1. a task published through the broker is consumed exactly once;
2. an application retry is delivered again and completes;
3. warm `SIGTERM` lets an in-flight late-ack task complete before shutdown; and
4. loss of a worker after task start causes the unacknowledged task to be
redelivered after the configured visibility timeout.
Run the same drill with the release Python environment and target Redis before
promoting a worker composition. Use a dedicated Redis database, retain the JSON
evidence, and set `CELERY_VISIBILITY_TIMEOUT_SECONDS` above the longest supported
business-task duration. The short visibility timeout used by CI is an isolated
test setting, not a production recommendation.
```bash
GOVOPLAN_WORKER_DRILL_REDIS_URL=redis://redis.example.test:6379/15 \
.venv/bin/python tools/checks/worker-runtime-drill.py \
--output evidence/worker-runtime.json
```
## Live Multi-Host Evidence
After deploying a pinned release on at least two Kubernetes nodes, create an API