Automate worker runtime delivery drill
This commit is contained in:
@@ -203,6 +203,30 @@ redelivery, scheduler failover, migration exclusion, object-store outage, and a
|
||||
coordinated database/object/key restore. Recovery rules and evidence are
|
||||
defined in [Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
|
||||
|
||||
## Worker Delivery Evidence
|
||||
|
||||
The module-matrix workflow runs `tools/checks/worker-runtime-drill.py` against a
|
||||
real isolated Redis database. The drill starts supervised Celery worker
|
||||
processes and records four guarantees without accessing tenant data:
|
||||
|
||||
1. a task published through the broker is consumed exactly once;
|
||||
2. an application retry is delivered again and completes;
|
||||
3. warm `SIGTERM` lets an in-flight late-ack task complete before shutdown; and
|
||||
4. loss of a worker after task start causes the unacknowledged task to be
|
||||
redelivered after the configured visibility timeout.
|
||||
|
||||
Run the same drill with the release Python environment and target Redis before
|
||||
promoting a worker composition. Use a dedicated Redis database, retain the JSON
|
||||
evidence, and set `CELERY_VISIBILITY_TIMEOUT_SECONDS` above the longest supported
|
||||
business-task duration. The short visibility timeout used by CI is an isolated
|
||||
test setting, not a production recommendation.
|
||||
|
||||
```bash
|
||||
GOVOPLAN_WORKER_DRILL_REDIS_URL=redis://redis.example.test:6379/15 \
|
||||
.venv/bin/python tools/checks/worker-runtime-drill.py \
|
||||
--output evidence/worker-runtime.json
|
||||
```
|
||||
|
||||
## Live Multi-Host Evidence
|
||||
|
||||
After deploying a pinned release on at least two Kubernetes nodes, create an API
|
||||
|
||||
Reference in New Issue
Block a user