docs(campaign): define broker redelivery evidence

This commit is contained in:
2026-07-22 20:19:51 +02:00
parent 06786e86ef
commit 90677348ff
4 changed files with 78 additions and 3 deletions

View File

@@ -103,6 +103,47 @@ proof, not proof of a target provider's behavior or Redis/Celery broker
redelivery. Use `--success-only` only when testing the success journey without redelivery. Use `--success-only` only when testing the success journey without
the local failure endpoints. the local failure endpoints.
## Redis/Celery Redelivery Acceptance
Run the maintained broker/worker-loss acceptance separately from the GreenMail
journey:
```bash
cd /mnt/DATA/git/govoplan-campaign/dev/mail-testbed
set -a
. ./.env
set +a
/mnt/DATA/git/govoplan/.venv/bin/python run_celery_redelivery_acceptance.py \
--evidence /tmp/govoplan-campaign-celery-redelivery-evidence.json
```
The runner creates a unique Compose project, starts only its loopback-bound,
AOF-enabled Redis service, creates an isolated temporary GovOPlaN database,
and starts a real Celery worker subscribed to `send_email`. A controlled SMTP
server holds the transaction after complete DATA and before the final response.
The runner kills that solo worker with the task still unacknowledged and starts
a replacement worker. After the configured Redis visibility timeout, the same Celery task identity must be redelivered.
The replacement must turn the durable
unfinished attempt into `outcome_unknown`, acknowledge the task, drain the
broker queue/unacked records, and leave the SMTP endpoint at exactly one
connection and one DATA transaction.
Only bounded counts, classifications, and booleans are retained. Worker logs,
task IDs, database identifiers, endpoints, credentials, and raw diagnostics are
kept in the temporary runtime and deleted. The evidence proves the local Redis
transport, real Celery process boundary, runner-supervised replacement, and
Campaign's duplicate-effect guard. It deliberately keeps production daemon supervision,
target-provider behavior, and source-artifact provenance false.
It does not claim that systemd, Kubernetes, another container orchestrator, or
an institution's Redis/SMTP deployment behaves identically.
The default run requires Docker CLI/Compose/daemon access and permission to
pull `redis:7-alpine`; the Celery workers execute from the current Python
environment. The isolated Compose project and volume are removed on exit.
`GOVOPLAN_CAMPAIGN_TEST_REDIS_VISIBILITY_TIMEOUT_SECONDS` defaults to three
seconds only to make this destructive local drill finish promptly; it is not a
production recommendation.
Starting the maintained test bed requires a working Docker CLI, Compose plugin, Starting the maintained test bed requires a working Docker CLI, Compose plugin,
daemon/socket access, and permission to pull `greenmail/standalone:2.1.9`. The daemon/socket access, and permission to pull `greenmail/standalone:2.1.9`. The
Campaign runner needs only the already-running loopback endpoints; it neither Campaign runner needs only the already-running loopback endpoints; it neither

View File

@@ -118,9 +118,21 @@ The runner also terminates a dedicated OS process executing the registered
Campaign send task after complete DATA, then invokes the task in a fresh Campaign send task after complete DATA, then invokes the task in a fresh
process. The unfinished durable attempt must become `outcome_unknown` and the process. The unfinished durable attempt must become `outcome_unknown` and the
endpoint must observe no second connection or DATA transaction. This covers endpoint must observe no second connection or DATA transaction. This covers
the worker task/process boundary. It does not cover Redis/Celery delivery, the worker task/process boundary but not a broker or daemon.
acknowledgement, broker redelivery, or daemon supervision; repeat the drill in
that infrastructure before approving a production worker deployment. Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` for the maintained
Redis/Celery delivery and broker redelivery boundary. It starts an isolated
Redis Compose service and real Celery workers, kills the first solo worker after complete DATA while the
late-ack task is unacknowledged, and requires the same task identity to reach a
replacement worker after Redis visibility recovery. Passing evidence also
requires durable `outcome_unknown`, an empty broker queue/unacked set, and
exactly one SMTP connection and DATA transaction. Raw worker logs and task,
database, endpoint, and credential identifiers are never retained.
That second runner proves local runner-supervised process replacement, not the
production process manager. Repeat the worker-loss drill under the selected
systemd, container, Kubernetes, or other production supervisor and the target
Redis/SMTP infrastructure before deployment approval.
## Reporting Checks ## Reporting Checks

View File

@@ -65,6 +65,13 @@ Before tagging a campaign release:
- Treat the task-process restart proof separately from the still-open - Treat the task-process restart proof separately from the still-open
Redis/Celery broker redelivery and daemon-supervision check; the coverage Redis/Celery broker redelivery and daemon-supervision check; the coverage
projection must keep `celery_broker_redelivery` false. projection must keep `celery_broker_redelivery` false.
- Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` as a separate
destructive worker-loss check. Retain its bounded evidence only when the same
broker task is observed at both workers, the durable state is
`outcome_unknown`, broker queue/unacked counts are zero, and the controlled
SMTP endpoint observed one connection and one DATA transaction. This closes
local Redis/Celery redelivery coverage, while production supervisor and
target-provider coverage remain false until separately tested.
- Confirm reusable mail profile selection is revalidated after campaign owner - Confirm reusable mail profile selection is revalidated after campaign owner
transfer. transfer.
- Confirm every inline SMTP/IMAP field is rejected on import/write, omitted - Confirm every inline SMTP/IMAP field is rejected on import/write, omitted

View File

@@ -128,6 +128,21 @@ def test_compose_redis_is_isolated_durable_and_health_checked() -> None:
assert "campaign-redis-data:/data" in compose assert "campaign-redis-data:/data" in compose
def test_runbook_keeps_local_redelivery_distinct_from_production_supervision() -> None:
testbed = (REPOSITORY_ROOT / "dev" / "mail-testbed" / "README.md").read_text(
encoding="utf-8"
)
runbook = (REPOSITORY_ROOT / "docs" / "CAMPAIGN_DELIVERY_RUNBOOK.md").read_text(
encoding="utf-8"
)
assert "run_celery_redelivery_acceptance.py" in testbed
assert "same Celery task identity must be redelivered" in testbed
assert "production daemon supervision" in testbed
assert "empty broker queue/unacked set" in runbook
assert "production process manager" in runbook
def test_compose_lifecycle_targets_only_isolated_redis_service() -> None: def test_compose_lifecycle_targets_only_isolated_redis_service() -> None:
up = runner._compose_command( up = runner._compose_command(
compose_file=COMPOSE_PATH, compose_file=COMPOSE_PATH,