govoplan-campaign/docs/CAMPAIGN_DELIVERY_RUNBOOK.md

Campaign Delivery Runbook

This runbook covers controlled campaign delivery after a campaign version has been validated, built, reviewed, and locked.

Operating Modes

• Local direct send: CELERY_ENABLED=false. Send now is available only for exact built runs within the effective synchronous limit. It preflights the complete batch before the first SMTP effect.
• Worker send: CELERY_ENABLED=true with Redis/Celery workers running. Queueing publishes durable delivery tasks, and Review and send polls their persisted summary counters even after the initiating request has returned.
• Mock send: use only for development review. It does not prove real SMTP/IMAP credentials or server policy.

Before First Live Use

• Use dedicated non-production SMTP/IMAP credentials.
• Store and test those credentials in a Mail-module profile. Campaign must contain only the selected server.mail_profile_id reference.
• Start the repository test bed in dev/mail-testbed/ when a local production-like SMTP/IMAP server is sufficient.
• Use a dedicated mailbox/folder for append-to-Sent tests.
• Confirm policy allows the SMTP host, envelope sender, recipients, and optional IMAP append target.
• Run one campaign each for no attachment, one attachment, and password-protected ZIP before using production recipients.
• Keep the report page open during tests; it is the operational source of truth for attempts, outcomes, and reconciliation.
• Confirm the effective Send now recipient-job limit. The safe default is 25; use Queue for workers for ordinary batches or any run above that limit.

Deliverability Preflight

Before the first live send for a sender domain or mail-server profile:

• Confirm the selected SMTP identity matches the visible From/envelope sender policy.
• Confirm SPF, DKIM, and DMARC are handled by the sending infrastructure or documented as out of scope for the selected test environment.
• Confirm rate limits are explicitly set for the expected provider and recipient volume.
• Confirm bounce/reply/notification addresses are monitored by an operational mailbox or intentionally disabled.
• Confirm large attachments and password-protected ZIPs are acceptable for the recipient systems.
• Confirm owner transfer or policy changes force profile reselection and revalidation before live delivery.

Queue And Send

1. Validate the version with file checks enabled.
2. Build the version and inspect all blocking review items.
3. Queue only after the selected version is the intended immutable execution version. Select Queue for workers, then verify the committed and published counts.
4. Use Send now only if the exact eligible count is non-zero and at or below the effective deployment/tenant limit shown on the page.
5. In worker mode, verify queue counters move from queued/claimed/sending to a terminal SMTP state.
6. If a synchronous request is used, keep Review and send open: it polls the durable counters while the request runs. A rejection occurs before SMTP and directs oversized runs to workers.

Outcome Handling

• smtp_accepted: Do not retry. If IMAP append is enabled and pending, run or enqueue the append action.
• failed_temporary: Retry explicitly after checking the error and retry count.
• failed_permanent: Retry only if the operator has corrected the root cause and intentionally includes permanent failures.
• outcome_unknown: Do not retry directly. Check SMTP logs, mailbox evidence, or provider control panels, then reconcile as accepted or not sent.
• claimed or sending that does not progress: treat as a worker interruption. Re-run worker handling or reconcile if SMTP may already have accepted the message.
• IMAP appending: A worker owns the durable append claim. Do not start a second append; if the worker cannot finish, reconcile only after checking the mailbox.
• IMAP outcome_unknown: Never append automatically. An operator with campaigns:campaign:reconcile must record an evidence note and resolve it as imap_appended or imap_not_appended. Only the latter becomes explicitly retryable.

Reconciliation

• Choose "Accepted" only with external evidence that SMTP accepted the message. The job becomes protected from retry and may proceed to IMAP append.
• Choose "Not sent" only when SMTP did not accept the message. The job becomes a temporary failure and can be selected by explicit retry.
• Add a note that identifies the evidence used, for example SMTP log line, provider message ID, or operator ticket.

Fault Injection Checklist

Use mock infrastructure first, then repeat against the non-production real test bed where possible:

• SMTP temporary failure.
• SMTP permanent failure.
• Recipient refusal after partial SMTP acceptance.
• Connection drop or worker interruption during SMTP.
• IMAP append failure after SMTP acceptance.
• Worker restart with queued, claimed, and sending jobs.

For the maintained loopback baseline, run dev/mail-testbed/run_campaign_acceptance.py. It proves the public Campaign path for SMTP acceptance, IMAP append, repeat-send blocking, an SMTP connection failure before transmission, an explicit SMTP authentication rejection, and an IMAP authentication rejection after SMTP acceptance. Its evidence is an allowlisted classification/count projection; raw provider diagnostics and transport/account identifiers are deliberately excluded.

The runner does not simulate a connection loss after SMTP DATA and does not restart a real worker. Keep those two checklist items open until a controlled fault proxy/target provider and supervised worker environment can produce the corresponding outcome_unknown, restart, retry, and reconciliation evidence.

Reporting Checks

• Partial delivery must show accepted, failed, and unknown counts separately.
• Excluded messages must show SMTP and IMAP as skipped, with skipped counts and filters separate from unattempted or failed delivery.
• Accepted and unknown jobs must not appear in retry selections.
• Reconciled accepted jobs must remain protected from resend.
• Reconciled not-sent jobs must appear only as explicit retry candidates.
• The final CSV export should include message id, resolved envelope headers, attachment evidence, EML reference/checksum, latest SMTP response/error, and latest IMAP folder/error before the campaign is considered operationally closed.