govoplan-campaign/docs/CAMPAIGN_DELIVERY_RUNBOOK.md

# Campaign Delivery Runbook

This runbook covers controlled campaign delivery after a campaign version has
been validated, built, reviewed, and locked.

## Operating Modes

- Local direct send: `CELERY_ENABLED=false`. **Send now** is available only for
  exact built runs within the effective synchronous limit. It preflights the
  complete batch before the first SMTP effect.
- Worker send: `CELERY_ENABLED=true` with Redis/Celery workers running. Queueing
  publishes durable delivery tasks, and Review and send polls their persisted
  summary counters even after the initiating request has returned.
- Mock send: use only for development review. It does not prove real SMTP/IMAP
  credentials or server policy.

## Before First Live Use

- Use dedicated non-production SMTP/IMAP credentials.
- Store and test those credentials in a Mail-module profile. Campaign must
  contain only the selected `server.mail_profile_id` reference.
- Start the repository test bed in `dev/mail-testbed/` when a local
  production-like SMTP/IMAP server is sufficient.
- Use a dedicated mailbox/folder for append-to-Sent tests.
- Confirm policy allows the SMTP host, envelope sender, recipients, and optional
  IMAP append target.
- Run one campaign each for no attachment, one attachment, and password-protected
  ZIP before using production recipients.
- Keep the report page open during tests; it is the operational source of truth
  for attempts, outcomes, and reconciliation.
- Confirm the effective Send now recipient-job limit. The safe default is 25;
  use Queue for workers for ordinary batches or any run above that limit.
  System administrators can explicitly configure 0–500 under Administration →
  SYSTEM → Campaign delivery; an explicit deployment ceiling remains binding,
  and TENANT policy may only narrow the inherited limit. This setting affects
  one synchronous request, not campaign size. It is audited and never sends
  messages; large interactive requests may encounter proxy timeouts.

## Deliverability Preflight

The Mail server connection test checks that selected server and credential. It
does not authorize the campaign's sender, recipients, or resource selection.
SMTP runtime checks require the selected SMTP credential when policy forbids
inheritance, independently of IMAP. Sent-folder append checks IMAP credentials
independently; full campaign validation still checks both required selections.
Preflight errors distinguish Mail profile/credential policy, SMTP configuration,
authentication, and connectivity. Do not change TLS or credential policies merely
because a campaign preflight failed. A preflight rejection leaves staged jobs
uncommitted and starts no message delivery.

Before the first live send for a sender domain or mail-server profile:

- Confirm the selected SMTP identity matches the visible From/envelope sender
  policy.
- Confirm SPF, DKIM, and DMARC are handled by the sending infrastructure or
  documented as out of scope for the selected test environment.
- Confirm rate limits are explicitly set for the expected provider and recipient
  volume.
- Confirm bounce/reply/notification addresses are monitored by an operational
  mailbox or intentionally disabled.
- Confirm large attachments and password-protected ZIPs are acceptable for the
  recipient systems.
- Confirm owner transfer or policy changes force profile reselection and
  revalidation before live delivery.

## Queue And Send

1. Link all required managed files, then validate the version with file checks
   enabled. Locking waits for a fresh attachment preview and asks for explicit
   **Link and lock** confirmation when matches are not linked. A locked version
   cannot change attachment links: use an editable version and repeat validation,
   build and review rather than assuming unlinked files were included.
2. Build the version and inspect all blocking review items.
3. Queue only after the selected version is the intended immutable execution
   version. Select **Queue for workers**, then verify the committed and
   published counts.
4. Use **Send now** only if the exact eligible count is non-zero and at or below
   the effective deployment/system/tenant limit shown on the page.
5. In worker mode, verify queue counters move from queued/claimed/sending to a
   terminal SMTP state.
6. If a synchronous request is used, keep its blocking progress dialog open.
   Only its small version-scoped persisted counters refresh; the workspace,
   recipient list, attachment preview and full summary stay unchanged. Read-only
   refresh failure retains the last counters and does not prove delivery failed.
   A disconnected request may still be executing; never repeat it blindly.
   Oversized initial runs are rejected before SMTP and directed to workers.
7. Review the SMTP batch line. `ready` means DNS/connectivity/TLS/auth preflight
   succeeded. Connection and reconnect counts explain reuse. `paused` means a
   systemic transport failure stopped the remaining jobs before their SMTP
   effect; test/correct the Mail profile and explicitly resume the queue.

## Outcome Handling

Each real Campaign job delivery is represented by a Core recovery-ledger
operation before the worker claims the job or invokes Mail, Postbox, or print.
Synchronous batches use the same boundary after their batch-wide preflight.
Explicit test/resend actions and post-acceptance IMAP appends use separate
action-fenced operations. Operations store only opaque IDs, digests, channel
policy, and bounded status evidence. A verified acceptance becomes
`succeeded`, a definitive pre-effect or provider rejection becomes `rejected`,
and uncertain or stranded effects remain `outcome_unknown` or
`recovery_required` in Ops. Campaign jobs, message actions, and channel-attempt
records remain the business source of truth.

For SMTP and Sent-folder APPEND, Campaign also passes the stable job/action and
attempt identifier into Mail. Mail establishes its own provider-bound recovery
fence after profile authorization and policy checks but before network I/O.
This nested ownership is intentional: Campaign proves its business transition,
while Mail proves the transport effect. Neither layer replays a completed or
unknown provider attempt merely to repair the other layer's state.

- `smtp_accepted`: Do not retry. If IMAP append is enabled and pending, run or
  enqueue the append action.
- `failed_temporary`: Retry explicitly after checking the error and retry count.
- `failed_permanent`: Retry only if the operator has corrected the root cause and
  intentionally includes permanent failures.
- `paused` after a systemic SMTP failure: do not resume until the shared Mail
  profile passes its connection test. Authentication, sender rejection, and
  unavailable connectivity affect the batch rather than one recipient.
- `outcome_unknown`: Do not retry directly. Check SMTP logs, mailbox evidence, or
  provider control panels, then reconcile as accepted or not sent.
- `claimed` or `sending` that does not progress: investigate the owning runtime.
  Duplicate worker handling leaves active state unchanged. Never infer from
  elapsed time alone that SMTP did not accept the message. Use the fenced
  **Recover interrupted claim** action described below when it is available.
- IMAP `appending`: A worker owns the durable append claim. Do not start a
  second append; if the worker cannot finish, reconcile only after checking the
  mailbox.
- IMAP `outcome_unknown`: Never append automatically. An operator with
  `campaigns:campaign:reconcile` must record an evidence note and resolve it as
  `imap_appended` or `imap_not_appended`. Only the latter becomes explicitly
  retryable.

## Reconciliation

- Choose "Accepted" only with external evidence that SMTP accepted the message.
  The job becomes protected from retry and may proceed to IMAP append.
- Choose "Not sent" only when SMTP did not accept the message. The job becomes a
  temporary failure and can be selected by explicit retry.
- Add a note that identifies the evidence used, for example SMTP log line,
  provider message ID, or operator ticket.

For pure-Mail SMTP and channel-specific IMAP operations, reconciliation updates
the original Campaign attempt, matching Campaign recovery operation and audit
record atomically under a fresh lease. A SMTP-only decision cannot resolve an
entire compound Mail/Postbox/Print operation; that ledger remains separately
unresolved until all of its effects are established. Campaign does
not rewrite Mail-owned nested provider-effect operations: those remain Mail's
separate evidence and operational responsibility. A failed audit or conflicting
claim must leave the prior unknown state intact.

### Recovery without workers

The Report offers explicit inline retry and continuation when workers are not
configured. Retry uses `campaigns:campaign:retry` plus
`campaigns:campaign:send`; continuation uses `campaigns:campaign:queue` plus
`campaigns:campaign:send`. Both use the canonical immutable jobs and ordinary
attempt ledger, not a separate one-message resend. Current Mail authorization,
review/approval, execution integrity, retry limits and rate limits still apply.
Each call is bounded by the effective synchronous recipient-job limit and
reports remaining eligible work. Continue explicitly until none remains; it
never selects accepted, excluded, active, uncertain or known failed jobs.
Known failures have their own explicit retry action. These actions do not
turn a long HTTP request into a background worker or guarantee exactly-once
SMTP when a provider acknowledgement is lost.

For an abandoned active SMTP or IMAP claim, the report exposes recovery only
after the original durable lease expires **and** the runtime registry proves
that its owner stopped or was replaced. A stale heartbeat is insufficient.
The opaque claim revision and original recovery evidence are rechecked under a
fresh lease. Recovery records the effect as **outcome unknown**, never not sent.
Then separately inspect external evidence and reconcile with a factual note
before any retry. This requires `campaigns:campaign:reconcile`. If the original
lease, evidence, or stopped-owner proof is missing, preserve the records and
investigate through Ops; do not edit delivery rows or force a lease expiry in
a real installation.

### Progress totals and Sent-folder batching

For each channel, **processed** includes successful, failed, uncertain and
cancelled outcomes. **In progress** is separate from pending, so the currently
sending/appending message remains visible. Paused SMTP work is also separate.
Excluded/non-requested channel work is outside the denominator. The endpoint
requires campaign read and object access and returns no recipient addresses,
message bodies, attachments or credentials. It is not the privacy-thresholded
aggregate Reports view and does not grant that view recipient access.

Append-to-Sent targets the selected campaign version, not all historical
versions. Each message remains a separately fenced, sequential IMAP APPEND.
Mail reuses the authenticated session and resolved folder for at most 100
messages or 300 seconds by default, then rotates the connection. Current
authorization, frozen transport revisions, credentials and recovery checks
still run for every message. A stale idle connection is checked before another
APPEND; an uncertain APPEND is never replayed. This removes repeated
connect/login/folder-list round trips, not the time needed to upload each EML.
It is not an atomic MULTIAPPEND transaction or parallel delivery.

SMTP and IMAP use the same progress dialog. An acknowledged operation remains
successful even if loading its follow-up diagnostics fails: use Reload to
refresh display, not to repeat the external effect.

## Fault Injection Checklist

Use mock infrastructure first, then repeat against the non-production real test
bed where possible:

- SMTP temporary failure.
- SMTP permanent failure.
- Recipient refusal after partial SMTP acceptance.
- Connection drop or worker interruption during SMTP.
- IMAP append failure after SMTP acceptance.
- Worker restart with queued, claimed, and sending jobs.

For the maintained loopback baseline, run
`dev/mail-testbed/run_campaign_acceptance.py`. It proves the public Campaign
path for SMTP acceptance, IMAP append, repeat-send blocking, an SMTP connection
failure before transmission, an explicit SMTP authentication rejection, an
explicit temporary `451` response after DATA, partial RCPT refusal, a
connection loss after complete DATA, and an IMAP authentication rejection
after SMTP acceptance. Its evidence is an allowlisted classification/count
projection; raw provider diagnostics and transport/account identifiers are
deliberately excluded.

The runner also terminates a dedicated OS process executing the registered
Campaign send task after complete DATA, then invokes the task in a fresh
process. Redelivery must leave the active claim unchanged and the endpoint
must observe no second connection or DATA transaction. Only a subsequent
explicit, fenced recovery with test-fixture stopped-owner and expired-lease
proof may change the unfinished attempt to `outcome_unknown`. This covers the
worker task/process boundary but not a broker or daemon.

Run `dev/mail-testbed/run_celery_redelivery_acceptance.py` for the maintained
Redis/Celery delivery and broker redelivery boundary. It starts an isolated
Redis Compose service and real Celery workers, kills the first solo worker after complete DATA while the
late-ack task is unacknowledged, and requires the same task identity to reach a
replacement worker after Redis visibility recovery. Passing evidence also
requires unchanged active state on duplicate delivery, followed by explicit
fenced recovery to `outcome_unknown`, an empty broker queue/unacked set, and
exactly one SMTP connection and DATA transaction. Lease expiration is simulated
only in the isolated fixture after its owner was stopped. Raw worker logs and task,
database, endpoint, and credential identifiers are never retained.

That second runner proves local runner-supervised process replacement, not the
production process manager. Repeat the worker-loss drill under the selected
systemd, container, Kubernetes, or other production supervisor and the target
Redis/SMTP infrastructure before deployment approval.

## Shared Build Artifacts

Generated EML is stored through Core's configured object-storage backend under
opaque Campaign-owned keys. The job records expected byte size, SHA-256 digest,
and Message-ID. A worker on another node must retrieve and verify those values
before attempting delivery.

- Do not expose object keys to ordinary campaign users or copy them into
  business fields.
- A build failure deletes objects written before the database transaction can
  commit.
- Retention starts a job-fenced forward-recovery operation before deletion,
  commits metadata changes in the Campaign-owned boundary, and independently
  probes the original locator before reporting success. An unavailable backend
  leaves an outcome-unknown operation; a deletion/metadata mismatch becomes
  recovery-required in Ops.
- A hard process loss between object creation and metadata commit can leave an
  orphan object. Use the operator-only, dry-run-first Campaign artifact
  reconciler documented in `CAMPAIGN_BUILD_RECOVERY.md`; it scans one bounded
  tenant-prefix page, enforces a minimum 24-hour grace period, protects active
  build fences, and rechecks committed EML and print-output references before
  deletion.
- Restore Campaign rows, object storage, and the encryption key to one
  coordinated recovery point before resuming workers.

For a scaled-runtime drill, build on one API replica, consume from another
worker, compare the stored evidence, and inject storage failures during build
and retention.

## Reporting Checks

- The recipient-aware report shows every frozen To/Cc/Bcc address in authored
  order, with a primary-address fallback only for old rows without that snapshot.
  SMTP envelope evidence, not the old primary-only UI, establishes how many
  recipients were actually offered to the provider. SMTP and IMAP diagnostics
  use list filters and consistent translated labels.
- Partial delivery must show accepted, failed, and unknown counts separately.
- Excluded messages must show SMTP and IMAP as `skipped`, with skipped counts
  and filters separate from unattempted or failed delivery.
- Accepted and unknown jobs must not appear in retry selections.
- Reconciled accepted jobs must remain protected from resend.
- Reconciled not-sent jobs must appear only as explicit retry candidates.
- The final CSV export should include message id, resolved envelope headers,
  attachment evidence, EML reference/checksum, latest SMTP response/error, and
  latest IMAP folder/error before the campaign is considered operationally
  closed.