docs(campaign): define broker redelivery evidence
This commit is contained in:
@@ -103,6 +103,47 @@ proof, not proof of a target provider's behavior or Redis/Celery broker
|
||||
redelivery. Use `--success-only` only when testing the success journey without
|
||||
the local failure endpoints.
|
||||
|
||||
## Redis/Celery Redelivery Acceptance
|
||||
|
||||
Run the maintained broker/worker-loss acceptance separately from the GreenMail
|
||||
journey:
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-campaign/dev/mail-testbed
|
||||
set -a
|
||||
. ./.env
|
||||
set +a
|
||||
/mnt/DATA/git/govoplan/.venv/bin/python run_celery_redelivery_acceptance.py \
|
||||
--evidence /tmp/govoplan-campaign-celery-redelivery-evidence.json
|
||||
```
|
||||
|
||||
The runner creates a unique Compose project, starts only its loopback-bound,
|
||||
AOF-enabled Redis service, creates an isolated temporary GovOPlaN database,
|
||||
and starts a real Celery worker subscribed to `send_email`. A controlled SMTP
|
||||
server holds the transaction after complete DATA and before the final response.
|
||||
The runner kills that solo worker with the task still unacknowledged and starts
|
||||
a replacement worker. After the configured Redis visibility timeout, the same Celery task identity must be redelivered.
|
||||
The replacement must turn the durable
|
||||
unfinished attempt into `outcome_unknown`, acknowledge the task, drain the
|
||||
broker queue/unacked records, and leave the SMTP endpoint at exactly one
|
||||
connection and one DATA transaction.
|
||||
|
||||
Only bounded counts, classifications, and booleans are retained. Worker logs,
|
||||
task IDs, database identifiers, endpoints, credentials, and raw diagnostics are
|
||||
kept in the temporary runtime and deleted. The evidence proves the local Redis
|
||||
transport, real Celery process boundary, runner-supervised replacement, and
|
||||
Campaign's duplicate-effect guard. It deliberately keeps production daemon supervision,
|
||||
target-provider behavior, and source-artifact provenance false.
|
||||
It does not claim that systemd, Kubernetes, another container orchestrator, or
|
||||
an institution's Redis/SMTP deployment behaves identically.
|
||||
|
||||
The default run requires Docker CLI/Compose/daemon access and permission to
|
||||
pull `redis:7-alpine`; the Celery workers execute from the current Python
|
||||
environment. The isolated Compose project and volume are removed on exit.
|
||||
`GOVOPLAN_CAMPAIGN_TEST_REDIS_VISIBILITY_TIMEOUT_SECONDS` defaults to three
|
||||
seconds only to make this destructive local drill finish promptly; it is not a
|
||||
production recommendation.
|
||||
|
||||
Starting the maintained test bed requires a working Docker CLI, Compose plugin,
|
||||
daemon/socket access, and permission to pull `greenmail/standalone:2.1.9`. The
|
||||
Campaign runner needs only the already-running loopback endpoints; it neither
|
||||
|
||||
@@ -118,9 +118,21 @@ The runner also terminates a dedicated OS process executing the registered
|
||||
Campaign send task after complete DATA, then invokes the task in a fresh
|
||||
process. The unfinished durable attempt must become `outcome_unknown` and the
|
||||
endpoint must observe no second connection or DATA transaction. This covers
|
||||
the worker task/process boundary. It does not cover Redis/Celery delivery,
|
||||
acknowledgement, broker redelivery, or daemon supervision; repeat the drill in
|
||||
that infrastructure before approving a production worker deployment.
|
||||
the worker task/process boundary but not a broker or daemon.
|
||||
|
||||
Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` for the maintained
|
||||
Redis/Celery delivery and broker redelivery boundary. It starts an isolated
|
||||
Redis Compose service and real Celery workers, kills the first solo worker after complete DATA while the
|
||||
late-ack task is unacknowledged, and requires the same task identity to reach a
|
||||
replacement worker after Redis visibility recovery. Passing evidence also
|
||||
requires durable `outcome_unknown`, an empty broker queue/unacked set, and
|
||||
exactly one SMTP connection and DATA transaction. Raw worker logs and task,
|
||||
database, endpoint, and credential identifiers are never retained.
|
||||
|
||||
That second runner proves local runner-supervised process replacement, not the
|
||||
production process manager. Repeat the worker-loss drill under the selected
|
||||
systemd, container, Kubernetes, or other production supervisor and the target
|
||||
Redis/SMTP infrastructure before deployment approval.
|
||||
|
||||
## Reporting Checks
|
||||
|
||||
|
||||
@@ -65,6 +65,13 @@ Before tagging a campaign release:
|
||||
- Treat the task-process restart proof separately from the still-open
|
||||
Redis/Celery broker redelivery and daemon-supervision check; the coverage
|
||||
projection must keep `celery_broker_redelivery` false.
|
||||
- Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` as a separate
|
||||
destructive worker-loss check. Retain its bounded evidence only when the same
|
||||
broker task is observed at both workers, the durable state is
|
||||
`outcome_unknown`, broker queue/unacked counts are zero, and the controlled
|
||||
SMTP endpoint observed one connection and one DATA transaction. This closes
|
||||
local Redis/Celery redelivery coverage, while production supervisor and
|
||||
target-provider coverage remain false until separately tested.
|
||||
- Confirm reusable mail profile selection is revalidated after campaign owner
|
||||
transfer.
|
||||
- Confirm every inline SMTP/IMAP field is rejected on import/write, omitted
|
||||
|
||||
@@ -128,6 +128,21 @@ def test_compose_redis_is_isolated_durable_and_health_checked() -> None:
|
||||
assert "campaign-redis-data:/data" in compose
|
||||
|
||||
|
||||
def test_runbook_keeps_local_redelivery_distinct_from_production_supervision() -> None:
|
||||
testbed = (REPOSITORY_ROOT / "dev" / "mail-testbed" / "README.md").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
runbook = (REPOSITORY_ROOT / "docs" / "CAMPAIGN_DELIVERY_RUNBOOK.md").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
|
||||
assert "run_celery_redelivery_acceptance.py" in testbed
|
||||
assert "same Celery task identity must be redelivered" in testbed
|
||||
assert "production daemon supervision" in testbed
|
||||
assert "empty broker queue/unacked set" in runbook
|
||||
assert "production process manager" in runbook
|
||||
|
||||
|
||||
def test_compose_lifecycle_targets_only_isolated_redis_service() -> None:
|
||||
up = runner._compose_command(
|
||||
compose_file=COMPOSE_PATH,
|
||||
|
||||
Reference in New Issue
Block a user