diff --git a/dev/mail-testbed/README.md b/dev/mail-testbed/README.md index 22ae159..7ac8981 100644 --- a/dev/mail-testbed/README.md +++ b/dev/mail-testbed/README.md @@ -103,6 +103,47 @@ proof, not proof of a target provider's behavior or Redis/Celery broker redelivery. Use `--success-only` only when testing the success journey without the local failure endpoints. +## Redis/Celery Redelivery Acceptance + +Run the maintained broker/worker-loss acceptance separately from the GreenMail +journey: + +```bash +cd /mnt/DATA/git/govoplan-campaign/dev/mail-testbed +set -a +. ./.env +set +a +/mnt/DATA/git/govoplan/.venv/bin/python run_celery_redelivery_acceptance.py \ + --evidence /tmp/govoplan-campaign-celery-redelivery-evidence.json +``` + +The runner creates a unique Compose project, starts only its loopback-bound, +AOF-enabled Redis service, creates an isolated temporary GovOPlaN database, +and starts a real Celery worker subscribed to `send_email`. A controlled SMTP +server holds the transaction after complete DATA and before the final response. +The runner kills that solo worker with the task still unacknowledged and starts +a replacement worker. After the configured Redis visibility timeout, the same Celery task identity must be redelivered. +The replacement must turn the durable +unfinished attempt into `outcome_unknown`, acknowledge the task, drain the +broker queue/unacked records, and leave the SMTP endpoint at exactly one +connection and one DATA transaction. + +Only bounded counts, classifications, and booleans are retained. Worker logs, +task IDs, database identifiers, endpoints, credentials, and raw diagnostics are +kept in the temporary runtime and deleted. The evidence proves the local Redis +transport, real Celery process boundary, runner-supervised replacement, and +Campaign's duplicate-effect guard. It deliberately keeps production daemon supervision, +target-provider behavior, and source-artifact provenance false. +It does not claim that systemd, Kubernetes, another container orchestrator, or +an institution's Redis/SMTP deployment behaves identically. + +The default run requires Docker CLI/Compose/daemon access and permission to +pull `redis:7-alpine`; the Celery workers execute from the current Python +environment. The isolated Compose project and volume are removed on exit. +`GOVOPLAN_CAMPAIGN_TEST_REDIS_VISIBILITY_TIMEOUT_SECONDS` defaults to three +seconds only to make this destructive local drill finish promptly; it is not a +production recommendation. + Starting the maintained test bed requires a working Docker CLI, Compose plugin, daemon/socket access, and permission to pull `greenmail/standalone:2.1.9`. The Campaign runner needs only the already-running loopback endpoints; it neither diff --git a/docs/CAMPAIGN_DELIVERY_RUNBOOK.md b/docs/CAMPAIGN_DELIVERY_RUNBOOK.md index f4eeb7b..7ad8d8a 100644 --- a/docs/CAMPAIGN_DELIVERY_RUNBOOK.md +++ b/docs/CAMPAIGN_DELIVERY_RUNBOOK.md @@ -118,9 +118,21 @@ The runner also terminates a dedicated OS process executing the registered Campaign send task after complete DATA, then invokes the task in a fresh process. The unfinished durable attempt must become `outcome_unknown` and the endpoint must observe no second connection or DATA transaction. This covers -the worker task/process boundary. It does not cover Redis/Celery delivery, -acknowledgement, broker redelivery, or daemon supervision; repeat the drill in -that infrastructure before approving a production worker deployment. +the worker task/process boundary but not a broker or daemon. + +Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` for the maintained +Redis/Celery delivery and broker redelivery boundary. It starts an isolated +Redis Compose service and real Celery workers, kills the first solo worker after complete DATA while the +late-ack task is unacknowledged, and requires the same task identity to reach a +replacement worker after Redis visibility recovery. Passing evidence also +requires durable `outcome_unknown`, an empty broker queue/unacked set, and +exactly one SMTP connection and DATA transaction. Raw worker logs and task, +database, endpoint, and credential identifiers are never retained. + +That second runner proves local runner-supervised process replacement, not the +production process manager. Repeat the worker-loss drill under the selected +systemd, container, Kubernetes, or other production supervisor and the target +Redis/SMTP infrastructure before deployment approval. ## Reporting Checks diff --git a/docs/EXAMPLE_CAMPAIGNS_AND_RELEASE_CHECKLIST.md b/docs/EXAMPLE_CAMPAIGNS_AND_RELEASE_CHECKLIST.md index 8437a01..eaf1a81 100644 --- a/docs/EXAMPLE_CAMPAIGNS_AND_RELEASE_CHECKLIST.md +++ b/docs/EXAMPLE_CAMPAIGNS_AND_RELEASE_CHECKLIST.md @@ -65,6 +65,13 @@ Before tagging a campaign release: - Treat the task-process restart proof separately from the still-open Redis/Celery broker redelivery and daemon-supervision check; the coverage projection must keep `celery_broker_redelivery` false. +- Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` as a separate + destructive worker-loss check. Retain its bounded evidence only when the same + broker task is observed at both workers, the durable state is + `outcome_unknown`, broker queue/unacked counts are zero, and the controlled + SMTP endpoint observed one connection and one DATA transaction. This closes + local Redis/Celery redelivery coverage, while production supervisor and + target-provider coverage remain false until separately tested. - Confirm reusable mail profile selection is revalidated after campaign owner transfer. - Confirm every inline SMTP/IMAP field is rejected on import/write, omitted diff --git a/tests/test_celery_redelivery_acceptance.py b/tests/test_celery_redelivery_acceptance.py index 0dc161e..3dc5317 100644 --- a/tests/test_celery_redelivery_acceptance.py +++ b/tests/test_celery_redelivery_acceptance.py @@ -128,6 +128,21 @@ def test_compose_redis_is_isolated_durable_and_health_checked() -> None: assert "campaign-redis-data:/data" in compose +def test_runbook_keeps_local_redelivery_distinct_from_production_supervision() -> None: + testbed = (REPOSITORY_ROOT / "dev" / "mail-testbed" / "README.md").read_text( + encoding="utf-8" + ) + runbook = (REPOSITORY_ROOT / "docs" / "CAMPAIGN_DELIVERY_RUNBOOK.md").read_text( + encoding="utf-8" + ) + + assert "run_celery_redelivery_acceptance.py" in testbed + assert "same Celery task identity must be redelivered" in testbed + assert "production daemon supervision" in testbed + assert "empty broker queue/unacked set" in runbook + assert "production process manager" in runbook + + def test_compose_lifecycle_targets_only_isolated_redis_service() -> None: up = runner._compose_command( compose_file=COMPOSE_PATH,