govoplan-campaign/docs/CAMPAIGN_DELIVERY_RUNBOOK.md

Campaign Delivery Runbook

This runbook covers controlled campaign delivery after a campaign version has been validated, built, reviewed, and locked.

Operating Modes

• Local direct send: CELERY_ENABLED=false. Send now is available only for exact built runs within the effective synchronous limit. It preflights the complete batch before the first SMTP effect.
• Worker send: CELERY_ENABLED=true with Redis/Celery workers running. Queueing publishes durable delivery tasks, and Review and send polls their persisted summary counters even after the initiating request has returned.
• Mock send: use only for development review. It does not prove real SMTP/IMAP credentials or server policy.

Before First Live Use

• Use dedicated non-production SMTP/IMAP credentials.
• Store and test those credentials in a Mail-module profile. Campaign must contain only the selected server.mail_profile_id reference.
• Start the repository test bed in dev/mail-testbed/ when a local production-like SMTP/IMAP server is sufficient.
• Use a dedicated mailbox/folder for append-to-Sent tests.
• Confirm policy allows the SMTP host, envelope sender, recipients, and optional IMAP append target.
• Run one campaign each for no attachment, one attachment, and password-protected ZIP before using production recipients.
• Keep the report page open during tests; it is the operational source of truth for attempts, outcomes, and reconciliation.
• Confirm the effective Send now recipient-job limit. The safe default is 25; use Queue for workers for ordinary batches or any run above that limit. System administrators can explicitly configure 0–500 under Administration → SYSTEM → Campaign delivery; an explicit deployment ceiling remains binding, and TENANT policy may only narrow the inherited limit. This setting affects one synchronous request, not campaign size. It is audited and never sends messages; large interactive requests may encounter proxy timeouts.

Deliverability Preflight

The Mail server connection test checks that selected server and credential. It does not authorize the campaign's sender, recipients, or resource selection. SMTP runtime checks require the selected SMTP credential when policy forbids inheritance, independently of IMAP. Sent-folder append checks IMAP credentials independently; full campaign validation still checks both required selections. Preflight errors distinguish Mail profile/credential policy, SMTP configuration, authentication, and connectivity. Do not change TLS or credential policies merely because a campaign preflight failed. A preflight rejection leaves staged jobs uncommitted and starts no message delivery.

Before the first live send for a sender domain or mail-server profile:

• Confirm the selected SMTP identity matches the visible From/envelope sender policy.
• Confirm SPF, DKIM, and DMARC are handled by the sending infrastructure or documented as out of scope for the selected test environment.
• Confirm rate limits are explicitly set for the expected provider and recipient volume.
• Confirm bounce/reply/notification addresses are monitored by an operational mailbox or intentionally disabled.
• Confirm large attachments and password-protected ZIPs are acceptable for the recipient systems.
• Confirm owner transfer or policy changes force profile reselection and revalidation before live delivery.

Queue And Send

1. Link all required managed files, then validate the version with file checks enabled. Locking waits for a fresh attachment preview and asks for explicit Link and lock confirmation when matches are not linked. A locked version cannot change attachment links: use an editable version and repeat validation, build and review rather than assuming unlinked files were included.
2. Build the version and inspect all blocking review items.
3. Queue only after the selected version is the intended immutable execution version. Select Queue for workers, then verify the committed and published counts.
4. Use Send now only if the exact eligible count is non-zero and at or below the effective deployment/system/tenant limit shown on the page.
5. In worker mode, verify queue counters move from queued/claimed/sending to a terminal SMTP state.
6. If a synchronous request is used, keep its blocking progress dialog open. Only its small version-scoped persisted counters refresh; the workspace, recipient list, attachment preview and full summary stay unchanged. Read-only refresh failure retains the last counters and does not prove delivery failed. A disconnected request may still be executing; never repeat it blindly. Oversized initial runs are rejected before SMTP and directed to workers.
7. Review the SMTP batch line. ready means DNS/connectivity/TLS/auth preflight succeeded. Connection and reconnect counts explain reuse. paused means a systemic transport failure stopped the remaining jobs before their SMTP effect; test/correct the Mail profile and explicitly resume the queue.

Outcome Handling

Each real Campaign job delivery is represented by a Core recovery-ledger operation before the worker claims the job or invokes Mail, Postbox, or print. Synchronous batches use the same boundary after their batch-wide preflight. Explicit test/resend actions and post-acceptance IMAP appends use separate action-fenced operations. Operations store only opaque IDs, digests, channel policy, and bounded status evidence. A verified acceptance becomes succeeded, a definitive pre-effect or provider rejection becomes rejected, and uncertain or stranded effects remain outcome_unknown or recovery_required in Ops. Campaign jobs, message actions, and channel-attempt records remain the business source of truth.

For SMTP and Sent-folder APPEND, Campaign also passes the stable job/action and attempt identifier into Mail. Mail establishes its own provider-bound recovery fence after profile authorization and policy checks but before network I/O. This nested ownership is intentional: Campaign proves its business transition, while Mail proves the transport effect. Neither layer replays a completed or unknown provider attempt merely to repair the other layer's state.

• smtp_accepted: Do not retry. If IMAP append is enabled and pending, run or enqueue the append action.
• failed_temporary: Retry explicitly after checking the error and retry count.
• failed_permanent: Retry only if the operator has corrected the root cause and intentionally includes permanent failures.
• paused after a systemic SMTP failure: do not resume until the shared Mail profile passes its connection test. Authentication, sender rejection, and unavailable connectivity affect the batch rather than one recipient.
• outcome_unknown: Do not retry directly. Check SMTP logs, mailbox evidence, or provider control panels, then reconcile as accepted or not sent.
• claimed or sending that does not progress: investigate the owning runtime. Duplicate worker handling leaves active state unchanged. Never infer from elapsed time alone that SMTP did not accept the message. Use the fenced Recover interrupted claim action described below when it is available.
• IMAP appending: A worker owns the durable append claim. Do not start a second append; if the worker cannot finish, reconcile only after checking the mailbox.
• IMAP outcome_unknown: Never append automatically. An operator with campaigns:campaign:reconcile must record an evidence note and resolve it as imap_appended or imap_not_appended. Only the latter becomes explicitly retryable.

Reconciliation

• Choose "Accepted" only with external evidence that SMTP accepted the message. The job becomes protected from retry and may proceed to IMAP append.
• Choose "Not sent" only when SMTP did not accept the message. The job becomes a temporary failure and can be selected by explicit retry.
• Add a note that identifies the evidence used, for example SMTP log line, provider message ID, or operator ticket.

For pure-Mail SMTP and channel-specific IMAP operations, reconciliation updates the original Campaign attempt, matching Campaign recovery operation and audit record atomically under a fresh lease. A SMTP-only decision cannot resolve an entire compound Mail/Postbox/Print operation; that ledger remains separately unresolved until all of its effects are established. Campaign does not rewrite Mail-owned nested provider-effect operations: those remain Mail's separate evidence and operational responsibility. A failed audit or conflicting claim must leave the prior unknown state intact.

Recovery without workers

The Report offers explicit inline retry and continuation when workers are not configured. Retry uses campaigns:campaign:retry plus campaigns:campaign:send; continuation uses campaigns:campaign:queue plus campaigns:campaign:send. Both use the canonical immutable jobs and ordinary attempt ledger, not a separate one-message resend. Current Mail authorization, review/approval, execution integrity, retry limits and rate limits still apply. Each call is bounded by the effective synchronous recipient-job limit and reports remaining eligible work. Continue explicitly until none remains; it never selects accepted, excluded, active, uncertain or known failed jobs. Known failures have their own explicit retry action. These actions do not turn a long HTTP request into a background worker or guarantee exactly-once SMTP when a provider acknowledgement is lost.

For an abandoned active SMTP or IMAP claim, the report exposes recovery only after the original durable lease expires and the runtime registry proves that its owner stopped or was replaced. A stale heartbeat is insufficient. The opaque claim revision and original recovery evidence are rechecked under a fresh lease. Recovery records the effect as outcome unknown, never not sent. Then separately inspect external evidence and reconcile with a factual note before any retry. This requires campaigns:campaign:reconcile. If the original lease, evidence, or stopped-owner proof is missing, preserve the records and investigate through Ops; do not edit delivery rows or force a lease expiry in a real installation.

Progress totals and Sent-folder batching

For each channel, processed includes successful, failed, uncertain and cancelled outcomes. In progress is separate from pending, so the currently sending/appending message remains visible. Paused SMTP work is also separate. Excluded/non-requested channel work is outside the denominator. The endpoint requires campaign read and object access and returns no recipient addresses, message bodies, attachments or credentials. It is not the privacy-thresholded aggregate Reports view and does not grant that view recipient access.

Append-to-Sent targets the selected campaign version, not all historical versions. Each message remains a separately fenced, sequential IMAP APPEND. Mail reuses the authenticated session and resolved folder for at most 100 messages or 300 seconds by default, then rotates the connection. Current authorization, frozen transport revisions, credentials and recovery checks still run for every message. A stale idle connection is checked before another APPEND; an uncertain APPEND is never replayed. This removes repeated connect/login/folder-list round trips, not the time needed to upload each EML. It is not an atomic MULTIAPPEND transaction or parallel delivery.

SMTP and IMAP use the same progress dialog. An acknowledged operation remains successful even if loading its follow-up diagnostics fails: use Reload to refresh display, not to repeat the external effect.

Fault Injection Checklist

Use mock infrastructure first, then repeat against the non-production real test bed where possible:

• SMTP temporary failure.
• SMTP permanent failure.
• Recipient refusal after partial SMTP acceptance.
• Connection drop or worker interruption during SMTP.
• IMAP append failure after SMTP acceptance.
• Worker restart with queued, claimed, and sending jobs.

For the maintained loopback baseline, run dev/mail-testbed/run_campaign_acceptance.py. It proves the public Campaign path for SMTP acceptance, IMAP append, repeat-send blocking, an SMTP connection failure before transmission, an explicit SMTP authentication rejection, an explicit temporary 451 response after DATA, partial RCPT refusal, a connection loss after complete DATA, and an IMAP authentication rejection after SMTP acceptance. Its evidence is an allowlisted classification/count projection; raw provider diagnostics and transport/account identifiers are deliberately excluded.

The runner also terminates a dedicated OS process executing the registered Campaign send task after complete DATA, then invokes the task in a fresh process. Redelivery must leave the active claim unchanged and the endpoint must observe no second connection or DATA transaction. Only a subsequent explicit, fenced recovery with test-fixture stopped-owner and expired-lease proof may change the unfinished attempt to outcome_unknown. This covers the worker task/process boundary but not a broker or daemon.

Run dev/mail-testbed/run_celery_redelivery_acceptance.py for the maintained Redis/Celery delivery and broker redelivery boundary. It starts an isolated Redis Compose service and real Celery workers, kills the first solo worker after complete DATA while the late-ack task is unacknowledged, and requires the same task identity to reach a replacement worker after Redis visibility recovery. Passing evidence also requires unchanged active state on duplicate delivery, followed by explicit fenced recovery to outcome_unknown, an empty broker queue/unacked set, and exactly one SMTP connection and DATA transaction. Lease expiration is simulated only in the isolated fixture after its owner was stopped. Raw worker logs and task, database, endpoint, and credential identifiers are never retained.

That second runner proves local runner-supervised process replacement, not the production process manager. Repeat the worker-loss drill under the selected systemd, container, Kubernetes, or other production supervisor and the target Redis/SMTP infrastructure before deployment approval.

Shared Build Artifacts

Generated EML is stored through Core's configured object-storage backend under opaque Campaign-owned keys. The job records expected byte size, SHA-256 digest, and Message-ID. A worker on another node must retrieve and verify those values before attempting delivery.

• Do not expose object keys to ordinary campaign users or copy them into business fields.
• A build failure deletes objects written before the database transaction can commit.
• Retention starts a job-fenced forward-recovery operation before deletion, commits metadata changes in the Campaign-owned boundary, and independently probes the original locator before reporting success. An unavailable backend leaves an outcome-unknown operation; a deletion/metadata mismatch becomes recovery-required in Ops.
• A hard process loss between object creation and metadata commit can leave an orphan object. Use the operator-only, dry-run-first Campaign artifact reconciler documented in CAMPAIGN_BUILD_RECOVERY.md; it scans one bounded tenant-prefix page, enforces a minimum 24-hour grace period, protects active build fences, and rechecks committed EML and print-output references before deletion.
• Restore Campaign rows, object storage, and the encryption key to one coordinated recovery point before resuming workers.

For a scaled-runtime drill, build on one API replica, consume from another worker, compare the stored evidence, and inject storage failures during build and retention.

Reporting Checks

• The recipient-aware report shows every frozen To/Cc/Bcc address in authored order, with a primary-address fallback only for old rows without that snapshot. SMTP envelope evidence, not the old primary-only UI, establishes how many recipients were actually offered to the provider. SMTP and IMAP diagnostics use list filters and consistent translated labels.
• Partial delivery must show accepted, failed, and unknown counts separately.
• Excluded messages must show SMTP and IMAP as skipped, with skipped counts and filters separate from unattempted or failed delivery.
• Accepted and unknown jobs must not appear in retry selections.
• Reconciled accepted jobs must remain protected from resend.
• Reconciled not-sent jobs must appear only as explicit retry candidates.
• The final CSV export should include message id, resolved envelope headers, attachment evidence, EML reference/checksum, latest SMTP response/error, and latest IMAP folder/error before the campaign is considered operationally closed.