docs(campaign): define broker redelivery evidence
This commit is contained in:
@@ -103,6 +103,47 @@ proof, not proof of a target provider's behavior or Redis/Celery broker
|
|||||||
redelivery. Use `--success-only` only when testing the success journey without
|
redelivery. Use `--success-only` only when testing the success journey without
|
||||||
the local failure endpoints.
|
the local failure endpoints.
|
||||||
|
|
||||||
|
## Redis/Celery Redelivery Acceptance
|
||||||
|
|
||||||
|
Run the maintained broker/worker-loss acceptance separately from the GreenMail
|
||||||
|
journey:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /mnt/DATA/git/govoplan-campaign/dev/mail-testbed
|
||||||
|
set -a
|
||||||
|
. ./.env
|
||||||
|
set +a
|
||||||
|
/mnt/DATA/git/govoplan/.venv/bin/python run_celery_redelivery_acceptance.py \
|
||||||
|
--evidence /tmp/govoplan-campaign-celery-redelivery-evidence.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The runner creates a unique Compose project, starts only its loopback-bound,
|
||||||
|
AOF-enabled Redis service, creates an isolated temporary GovOPlaN database,
|
||||||
|
and starts a real Celery worker subscribed to `send_email`. A controlled SMTP
|
||||||
|
server holds the transaction after complete DATA and before the final response.
|
||||||
|
The runner kills that solo worker with the task still unacknowledged and starts
|
||||||
|
a replacement worker. After the configured Redis visibility timeout, the same Celery task identity must be redelivered.
|
||||||
|
The replacement must turn the durable
|
||||||
|
unfinished attempt into `outcome_unknown`, acknowledge the task, drain the
|
||||||
|
broker queue/unacked records, and leave the SMTP endpoint at exactly one
|
||||||
|
connection and one DATA transaction.
|
||||||
|
|
||||||
|
Only bounded counts, classifications, and booleans are retained. Worker logs,
|
||||||
|
task IDs, database identifiers, endpoints, credentials, and raw diagnostics are
|
||||||
|
kept in the temporary runtime and deleted. The evidence proves the local Redis
|
||||||
|
transport, real Celery process boundary, runner-supervised replacement, and
|
||||||
|
Campaign's duplicate-effect guard. It deliberately keeps production daemon supervision,
|
||||||
|
target-provider behavior, and source-artifact provenance false.
|
||||||
|
It does not claim that systemd, Kubernetes, another container orchestrator, or
|
||||||
|
an institution's Redis/SMTP deployment behaves identically.
|
||||||
|
|
||||||
|
The default run requires Docker CLI/Compose/daemon access and permission to
|
||||||
|
pull `redis:7-alpine`; the Celery workers execute from the current Python
|
||||||
|
environment. The isolated Compose project and volume are removed on exit.
|
||||||
|
`GOVOPLAN_CAMPAIGN_TEST_REDIS_VISIBILITY_TIMEOUT_SECONDS` defaults to three
|
||||||
|
seconds only to make this destructive local drill finish promptly; it is not a
|
||||||
|
production recommendation.
|
||||||
|
|
||||||
Starting the maintained test bed requires a working Docker CLI, Compose plugin,
|
Starting the maintained test bed requires a working Docker CLI, Compose plugin,
|
||||||
daemon/socket access, and permission to pull `greenmail/standalone:2.1.9`. The
|
daemon/socket access, and permission to pull `greenmail/standalone:2.1.9`. The
|
||||||
Campaign runner needs only the already-running loopback endpoints; it neither
|
Campaign runner needs only the already-running loopback endpoints; it neither
|
||||||
|
|||||||
@@ -118,9 +118,21 @@ The runner also terminates a dedicated OS process executing the registered
|
|||||||
Campaign send task after complete DATA, then invokes the task in a fresh
|
Campaign send task after complete DATA, then invokes the task in a fresh
|
||||||
process. The unfinished durable attempt must become `outcome_unknown` and the
|
process. The unfinished durable attempt must become `outcome_unknown` and the
|
||||||
endpoint must observe no second connection or DATA transaction. This covers
|
endpoint must observe no second connection or DATA transaction. This covers
|
||||||
the worker task/process boundary. It does not cover Redis/Celery delivery,
|
the worker task/process boundary but not a broker or daemon.
|
||||||
acknowledgement, broker redelivery, or daemon supervision; repeat the drill in
|
|
||||||
that infrastructure before approving a production worker deployment.
|
Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` for the maintained
|
||||||
|
Redis/Celery delivery and broker redelivery boundary. It starts an isolated
|
||||||
|
Redis Compose service and real Celery workers, kills the first solo worker after complete DATA while the
|
||||||
|
late-ack task is unacknowledged, and requires the same task identity to reach a
|
||||||
|
replacement worker after Redis visibility recovery. Passing evidence also
|
||||||
|
requires durable `outcome_unknown`, an empty broker queue/unacked set, and
|
||||||
|
exactly one SMTP connection and DATA transaction. Raw worker logs and task,
|
||||||
|
database, endpoint, and credential identifiers are never retained.
|
||||||
|
|
||||||
|
That second runner proves local runner-supervised process replacement, not the
|
||||||
|
production process manager. Repeat the worker-loss drill under the selected
|
||||||
|
systemd, container, Kubernetes, or other production supervisor and the target
|
||||||
|
Redis/SMTP infrastructure before deployment approval.
|
||||||
|
|
||||||
## Reporting Checks
|
## Reporting Checks
|
||||||
|
|
||||||
|
|||||||
@@ -65,6 +65,13 @@ Before tagging a campaign release:
|
|||||||
- Treat the task-process restart proof separately from the still-open
|
- Treat the task-process restart proof separately from the still-open
|
||||||
Redis/Celery broker redelivery and daemon-supervision check; the coverage
|
Redis/Celery broker redelivery and daemon-supervision check; the coverage
|
||||||
projection must keep `celery_broker_redelivery` false.
|
projection must keep `celery_broker_redelivery` false.
|
||||||
|
- Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` as a separate
|
||||||
|
destructive worker-loss check. Retain its bounded evidence only when the same
|
||||||
|
broker task is observed at both workers, the durable state is
|
||||||
|
`outcome_unknown`, broker queue/unacked counts are zero, and the controlled
|
||||||
|
SMTP endpoint observed one connection and one DATA transaction. This closes
|
||||||
|
local Redis/Celery redelivery coverage, while production supervisor and
|
||||||
|
target-provider coverage remain false until separately tested.
|
||||||
- Confirm reusable mail profile selection is revalidated after campaign owner
|
- Confirm reusable mail profile selection is revalidated after campaign owner
|
||||||
transfer.
|
transfer.
|
||||||
- Confirm every inline SMTP/IMAP field is rejected on import/write, omitted
|
- Confirm every inline SMTP/IMAP field is rejected on import/write, omitted
|
||||||
|
|||||||
@@ -128,6 +128,21 @@ def test_compose_redis_is_isolated_durable_and_health_checked() -> None:
|
|||||||
assert "campaign-redis-data:/data" in compose
|
assert "campaign-redis-data:/data" in compose
|
||||||
|
|
||||||
|
|
||||||
|
def test_runbook_keeps_local_redelivery_distinct_from_production_supervision() -> None:
|
||||||
|
testbed = (REPOSITORY_ROOT / "dev" / "mail-testbed" / "README.md").read_text(
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
runbook = (REPOSITORY_ROOT / "docs" / "CAMPAIGN_DELIVERY_RUNBOOK.md").read_text(
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "run_celery_redelivery_acceptance.py" in testbed
|
||||||
|
assert "same Celery task identity must be redelivered" in testbed
|
||||||
|
assert "production daemon supervision" in testbed
|
||||||
|
assert "empty broker queue/unacked set" in runbook
|
||||||
|
assert "production process manager" in runbook
|
||||||
|
|
||||||
|
|
||||||
def test_compose_lifecycle_targets_only_isolated_redis_service() -> None:
|
def test_compose_lifecycle_targets_only_isolated_redis_service() -> None:
|
||||||
up = runner._compose_command(
|
up = runner._compose_command(
|
||||||
compose_file=COMPOSE_PATH,
|
compose_file=COMPOSE_PATH,
|
||||||
|
|||||||
Reference in New Issue
Block a user