govoplan-campaign/docs/CAMPAIGN_HANDBOOK.md

# Campaign Handbook

## Purpose and status

This is the canonical, multi-perspective handbook for the Campaign module. It
describes the current implementation, the operational contract it relies on,
and the remaining work required before Campaign can be presented as GovOPlaN's
maintained reference composition.

Use the section that matches the task at hand:

| Perspective | Start here |
| --- | --- |
| Campaign author | [Prepare a campaign](#prepare-a-campaign) |
| Reviewer | [Review and complete review](#review-and-complete-review) |
| Sender or delivery operator | [Deliver and resolve outcomes](#deliver-and-resolve-outcomes) |
| Campaign or tenant administrator | [Administration and policy](#administration-and-policy) |
| Platform operator | [Operations and recovery](#operations-and-recovery) |
| Integrator or developer | [Composition and integration contracts](#composition-and-integration-contracts) |
| Security, privacy, or audit reviewer | [Assurance model](#assurance-model) |
| Release reviewer | [Reference-composition acceptance](#reference-composition-acceptance) |

The shorter task documents remain useful companions:

- [Campaign delivery runbook](CAMPAIGN_DELIVERY_RUNBOOK.md)
- [Mail profile boundary](MAIL_PROFILE_BOUNDARY.md)
- [Recipient import guide](RECIPIENT_IMPORT_GUIDE.md)
- [Recipient and Addresses boundary](RECIPIENT_ADDRESS_BOUNDARY.md)
- [Examples and release checklist](EXAMPLE_CAMPAIGNS_AND_RELEASE_CHECKLIST.md)

## What Campaign is for

Campaign turns governed source data into individually built messages and then
controls their review, delivery, and evidence. It is intentionally a
composition module: it demonstrates how one user journey can use optional Mail,
Files, Addresses, and Notifications capabilities alongside Core access/audit
infrastructure without copying ownership from those modules. Policy may consume
Campaign context through a narrow capability; Campaign does not import Policy.

Campaign owns:

- the communication purpose, content, campaign-local fields, and templates;
- campaign-local recipient snapshots, exclusions, and personalization;
- message and attachment rules for a version;
- validation, review, build, queue, and delivery-control state;
- the durable jobs and attempts needed to explain delivery outcomes; and
- campaign-specific reports, shares, and frozen execution evidence.

Campaign does not own:

- SMTP/IMAP profiles, credentials, protocol adapters, or mailbox policy;
- long-lived address-book master data, consent lifecycle, or deduplication;
- managed file bytes, connector credentials, or external-file provenance;
- general-purpose workflow definitions; or
- global identity, organization, permission, or retention policy.

Those boundaries matter in both persistence and UI. A campaign references an
authorized Mail profile and managed file versions; it must never become a
second secret store, file store, or address directory.

## Process view

The supported process is a controlled progression, not a single "send" call:

```text
create/edit
    -> validate and resolve policy/integrations
    -> review warnings and blockers
    -> build exact recipient messages
    -> complete review and queue
    -> SMTP attempt per job
    -> optional IMAP append per accepted job
    -> report, retry, reconcile, or correct
    -> archive when no active/uncertain delivery remains
```

Campaign status summarizes the whole campaign. Version workflow state describes
the selected immutable/editable version. Each recipient job separately records
build, validation, queue, SMTP, and IMAP state. Operators must use the job-level
states when deciding whether another external effect is safe.

Important distinctions:

- **Validation lock** is the reversible lock created by a successful
  validation/build path. Editing requires unlocking and invalidates derived
  evidence as appropriate.
- **User lock** is an explicit audit-safe lock. A permanently locked or
  delivery-final version is not edited in place; create an editable successor.
- **SMTP accepted** means the provider accepted the message. It does not prove
  inbox delivery, reading, or business acknowledgement.
- **Outcome unknown** means an attempt may have had an external effect. It must
  be investigated and reconciled before retry.
- **IMAP append** is a separate effect after SMTP acceptance. An append failure
  is not evidence that sending failed.
- **Archive** preserves evidence. Draft-only campaigns without built, locked,
  or delivery evidence may be deleted where policy allows; evidence-bearing
  campaigns are archived instead.

## User tasks

### Prepare a campaign

1. Create a campaign and confirm its owner or owning group.
2. Define global settings, fields, templates, and recipient data.
3. Import one-off recipient data or select a reusable Addresses source when the
   optional capability is installed. Review source provenance and stale-source
   warnings; Campaign freezes the selected rows rather than following later
   directory changes silently.
4. Select managed attachments through Files. Server/API campaigns do not accept
   arbitrary local paths. Preview rules and unmatched files before building.
5. Open **Mail settings** and select an available Mail profile. The campaign
   stores only `server.mail_profile_id`; it never accepts SMTP/IMAP settings,
   usernames, passwords, or credential references.
6. Save the editable version, validate the relevant sections, and resolve every
   blocking issue. Warnings remain explicit review decisions.
7. Build the exact messages and inspect recipient, addressing, template,
   attachment, and generated-message evidence.

If a selected optional module is absent, Campaign remains loadable and explains
which function is unavailable. It must not fail startup because Mail, Files, or
Addresses is not installed.

### Review and complete review

The reviewer should verify the immutable candidate that will be delivered, not
just the authoring form:

1. Confirm purpose, owner, selected version, and recipient count.
2. Inspect blocking errors, warnings, exclusions, and recipients requiring
   review.
3. Inspect representative and exceptional rendered messages, including From,
   To/CC/BCC, Reply-To, subject, body, and attachment evidence.
4. Confirm the selected Mail profile is authorized for the campaign's current
   tenant and owner context.
5. Confirm attachment behavior when a rule matches no files, ZIP/password
   behavior, and any recipient-specific files.
6. Record review completion and the inspected message keys through the review
   surface. The current baseline does not persist a distinct approve/reject
   decision or review reason. If content, recipients, attachment inputs, owner
   context, or non-secret transport identity changes, revalidate and rebuild.

Normal readers and reviewers see business state and safe evidence. Process-local
paths, storage keys, worker claim tokens, and raw provider diagnostics require
the dedicated diagnostic permission and must not leak through ordinary campaign,
version, job, or report responses.

Campaign now provides a separate aggregate **Reports** surface for readers with
`campaigns:report:read` and access to the campaign. It loads only the safe
aggregate projections, applies small-cell suppression, and offers no recipient
rows, drill-down, filtering, export, or delivery actions. The recipient-aware
**Campaign Report** still requires recipient-read access and does not yet hide
every action control that the actor lacks. The server authorizes each action,
but permission-aware action visibility on that detailed surface remains open
work; do not confuse it with the aggregate reader experience.

### Deliver and resolve outcomes

Use the [delivery runbook](CAMPAIGN_DELIVERY_RUNBOOK.md) for the detailed
operator sequence.

At a minimum:

1. Queue only a validated, locked, built version. Use **Queue for workers** for
   ordinary batches; the durable progress remains visible after leaving and
   returning to Review and send.
2. Use **Send now** only when the exact persisted eligible build is within the
   effective synchronous limit shown by the UI. The default deployment limit
   is 25 recipient jobs. The backend repeats the count and preflights every
   message and the Mail profile revision before contacting SMTP.
3. Treat `smtp_accepted` as protected from ordinary retry.
4. Retry `failed_temporary` explicitly after inspecting the cause.
5. Include `failed_permanent` only after correcting the cause and making a
   conscious override.
6. Never retry `outcome_unknown` blindly. Inspect SMTP/provider evidence and
   reconcile it as **accepted** or **not sent**, with a note identifying the
   evidence.
7. Process append-to-Sent only for SMTP-accepted jobs and investigate append
   failure independently. Never retry an `outcome_unknown` IMAP append blindly:
   reconcile mailbox evidence as **appended** or **not appended** with a note.
   Only the latter becomes explicitly retryable, and neither decision resends
   the already SMTP-accepted message.
8. Archive only after active and uncertain effects are resolved.

Pause stops new eligible work but cannot undo a provider effect already in
progress. Cancel marks work that has not yet produced a protected SMTP outcome;
it cannot recall accepted mail.

The deployment ceiling is configured with
`GOVOPLAN_CAMPAIGN_SYNCHRONOUS_SEND_MAX_RECIPIENTS` (0 disables Send now; the
accepted range is 0–500). A tenant may only narrow that ceiling with
`tenant.settings.campaign_delivery_policy.synchronous_send_max_recipients`.
The effective value and source are returned by the protected delivery-options
API, recorded for successful/rejected synchronous commands, and stated in the
configured handbook topic.

### Test and one-message actions

The current baseline includes mock send, queue dry-run, synchronous immediate
delivery, sending one selected job, retry selection, and unattempted-job
selection. Their audit and state behavior is not yet the final vocabulary
accepted for issue `govoplan-campaign#69`.

The intended distinction is:

- **Test:** deliver once to the configured test path, audit it, and leave the
  production job unsent.
- **Single send:** send one unsent job, audit it, and mark its production effect
  complete.
- **Single resend:** intentionally send one job again regardless of an earlier
  failure or acceptance, with distinct authority, warning, reason, and audit.

Until that slice is implemented and target-tested, do not present the existing
buttons as a complete resend policy. Ordinary retry protection remains the safe
default.

## Data and evidence model

### Versions and snapshots

Editable campaign JSON is versioned. Build creates recipient jobs and an
execution snapshot. A new profile-only snapshot contains the stable Mail
profile id, delivery policy/evidence, and opaque Mail-issued transport
revisions. It contains no resolved SMTP/IMAP configuration or credentials.

At delivery time Mail re-authorizes the profile, compares the expected opaque
revisions, resolves credentials inside the same Mail-owned operation, and
performs the transport effect. Campaign receives only the sanitized result it
needs for job state and evidence. This same-call check prevents a
validate-then-use race across the module boundary.

Credential rotation that does not change the non-secret transport identity may
continue to satisfy the snapshot. A host, account identity, protocol, sender,
or other revisioned transport change stops delivery until the campaign is
revalidated and rebuilt.

### Existing legacy database records

The current Campaign database may contain versions with inline SMTP/IMAP
material. No separate historical Campaign JSON corpus exists. Inline transport
material in those rows is treated as inert legacy data and is never interpreted
as an executable Mail configuration:

- public responses remove legacy transport fields and secrets;
- validation, build, queue, retry, and delivery fail closed;
- an editable version changes to profile-only form only through an explicit
  Mail-settings save; and
- a locked version is preserved and must be forked to an editable successor.

Normal database backup/restore and access controls cover those rows together
with the rest of the current database. There is no separate historical-JSON,
backup-scanning, or inline-secret migration program. Restoring an existing
legacy row preserves it as inert evidence and does not make it deliverable.

### Recipient and attachment evidence

The delivery record should be able to identify:

- source/import context and the frozen recipient row;
- effective addressing and message id;
- template inputs and unresolved-placeholder decisions;
- managed file/version ids, source provenance, checksums, and ZIP evidence;
- generated EML checksum and size;
- Mail profile reference and opaque transport revisions;
- each SMTP and IMAP attempt, its classification, safe provider response, and
  reconciliation note; and
- actor/system trigger, timestamps, policy context, and corrections.

An excluded recipient/message is a completed validation decision, not a
pending delivery. Its SMTP and IMAP states are both `skipped`; it is counted and
filterable separately from unattempted, failed, accepted, and append outcomes.
No SMTP or IMAP attempt exists for such a row. If historical data contains
actual transport evidence despite an exclusion marker, that evidence is
preserved for audit and reconciliation rather than relabelled.

## Administration and policy

### Roles and permissions

The supplied role templates deliberately separate preparation, review, and
delivery:

- **Campaign manager** prepares, validates, and builds campaigns and recipients.
- **Campaign reviewer** inspects prepared material and records review
  completion for the exact messages checked.
- **Campaign sender** queues, sends, pauses/resumes/cancels, retries, reconciles,
  reads reports, and has diagnostic access.

Administrators may compose narrower roles from the declared permissions. Keep
these separations where institutional policy requires four-eyes approval. Mail
profile use additionally requires `mail:profile:use`; profile and credential
administration remains a Mail permission.

Campaign access is also constrained by tenant, owner/group context, and explicit
shares. A share grants only its declared campaign permission; it does not grant
Mail credentials, Files administration, or tenant-wide recipient access.

### Configuration checklist

- Install compatible Core and Campaign versions. Access and base audit are Core
  infrastructure; install optional Mail, Files, Addresses, Notifications, and
  Policy modules only where the configured journey requires them.
- Configure Mail profiles and policy in Mail, not in campaign fixtures or JSON.
- Configure Files storage/connectors and attachment permissions in Files.
- Define role assignments and separation-of-duty policy.
- Configure Redis/Celery for durable batch workers where production volume
  requires it. Local execution is a development/small-run mode, not horizontal
  worker coordination.
- Set rate limits for the target provider. Mail may use Redis for shared
  throttling when present and a process-local fallback in development.
- Establish retention, deletion, archive, report-export, and diagnostic-access
  policy before production recipient data is loaded.
- Use non-production SMTP/IMAP identities and the maintained examples before
  allowing a production profile.

### Owner transfer

Mail profile visibility can depend on campaign owner or group. Transferring an
editable campaign with a selected profile clears/requires reselection and
revalidation. A locked or delivery-final version is never silently rewritten;
create an editable successor.

## Operations and recovery

### Health to observe

- database and migration health;
- compatible module interface versions;
- queue publication, worker heartbeat, claim age, and backlog;
- SMTP/IMAP profile test result and deployment egress policy;
- Mail profile authorization/transport-revision mismatch;
- Files resolution and frozen attachment availability;
- counts by queue, SMTP, IMAP, and reconciliation state; and
- audit and report generation failures.

### Worker interruption

A crashed worker can leave a job claimed or sending. Do not infer non-delivery
from worker loss. If the SMTP boundary may have been crossed, classify the
outcome as unknown and reconcile from external evidence. Only work that is
provably unattempted may be returned to an ordinary queue.

### Retry and reconciliation

Retries create new attempt evidence; they do not overwrite the previous
attempt. Reconciliation is a privileged factual correction supported by an
operator note. Repeated automated requests should be idempotent for the same
eligible job state; a deliberate resend is a different, not-yet-finalized
business action and must never be disguised as a retry.

### Backups and restoration

Back up Campaign, Mail, Files, Core/Audit, and shared storage consistently for
the composition. A database restore without generated EML/file storage, or file
storage without matching metadata, does not reconstruct the evidence chain.
After restore, keep outbound delivery paused until queue/attempt state and
provider evidence have been reconciled; never let restored accepted jobs send
again merely because a queue message was lost.

### Incident handling

1. Pause new delivery when duplicate or unknown effects are possible.
2. Preserve database, queue, provider, worker, and audit evidence.
3. Scope affected campaigns/jobs without exposing recipient content broadly.
4. Reconcile uncertain jobs individually or through an approved bounded tool.
5. Correct configuration in its owning module, then revalidate/rebuild where
   revisioned transport inputs changed.
6. Record the incident reference and recovery rationale in audit evidence.

## Composition and integration contracts

Campaign has one required platform dependency: Core. Optional module behavior
is discovered through versioned, Core-mediated capabilities; Campaign must not
import optional sibling ORM, services, or WebUI implementation.

Current principal contracts include:

| Contract | Direction | Purpose |
| --- | --- | --- |
| `mail.campaign_delivery` 0.2.x | Mail -> Campaign | Summarize a known reference; authorize and revision-gate it; send/append using Mail-owned configuration and credentials; return sanitized results |
| `files.campaign_attachments` 0.1.x | Files -> Campaign | Select/materialize governed file versions and preserve campaign usage/evidence |
| `addresses.lookup` 0.1.x | Addresses -> Campaign | Optional address suggestions |
| `addresses.recipient_source` 0.1.x | Addresses -> Campaign | Optional versioned recipient-source snapshots |
| `campaigns.access` 0.1.x | Campaign -> platform | Explain campaign access/existence without exporting ORM objects |
| `campaigns.mail_policy_context` 0.1.x | Campaign -> Mail | Resolve campaign tenant/owner context for Mail policy |
| `campaigns.delivery_tasks` 0.1.x | Campaign -> workers | Execute narrow queued send/append tasks |
| `campaigns.retention` 0.1.x | Campaign -> retention | Apply Campaign-owned retention behavior |

Breaking payload or ownership changes require an interface-version bump and a
release-composition alignment gate. Optional absence must be tested physically,
not only hidden in navigation.

### External API expectations

- Tenant and campaign access are evaluated for every operation.
- Writes require CSRF/auth behavior supplied by Core and the specific declared
  scope.
- Queue/retry/reconcile endpoints operate on persisted state and return safe
  summaries; initiating HTTP success is not proof of external delivery.
- Delta endpoints are optimization surfaces, not a separate source of truth.
- Report exports contain permitted business/evidence fields but no credentials,
  local paths, storage keys, or worker claims.

## Assurance model

### Security invariants

- No new campaign payload, fixture, response, or execution snapshot contains
  SMTP/IMAP settings or credentials.
- Mail resolves credentials and performs transports inside Mail-owned calls.
- Every real connector peer is validated and pinned at connection time under
  deployment-wide private-network policy.
- API/server attachment paths use managed Files references; arbitrary and
  traversal-capable local paths are rejected.
- Public responses recursively remove infrastructure locators and secret-like
  legacy fields.
- Accepted and outcome-unknown jobs are protected from ordinary retry.
- Diagnostic permission is separate from campaign read/report access.

### Privacy

Recipient fields and rendered messages may contain personal or sensitive data.
Grant recipient read/export, report export, and diagnostic access separately.
Prefer aggregate status for readers who do not need recipient detail. Define
purpose, lawful basis, minimization, export control, and retention before the
campaign starts; do not use Campaign as a substitute consent or address-master
system.

### Audit and destructive actions

Material authoring, validation, locking, review, queueing, send, retry,
reconciliation, sharing, owner transfer, archive, and permitted deletion actions
emit attributable evidence. Audit details must be non-secret and should refer to
stable ids rather than repeat message bodies or credentials.

Draft deletion is allowed only while no audit-relevant build, delivery job, or
lock exists. Evidence-bearing campaigns are archived. Destructive module
retirement remains a separately confirmed installer operation with backup and
retirement evidence.

## Reference-composition acceptance

Campaign is ready to serve as the demonstration module only when all of the
following are repeatable in a pinned clean installation:

1. The maintained examples validate and build with Mail/Files present, and
   Campaign still starts with each optional module absent.
2. A user can import recipients, select managed files, choose an authorized Mail
   profile, validate, review, build, and queue without entering transport ids or
   secrets manually.
3. Campaign JSON and all ordinary APIs reject/omit inline transport material;
   legacy records are visible as migration-required and cannot execute.
4. SMTP success, temporary/permanent failure, connection loss, worker loss,
   outcome unknown, retry, reconciliation, IMAP success, and IMAP failure have
   tested, non-duplicating outcomes against the target environment.
5. Author, reviewer, sender/operator, reader, and administrator views use the
   central component system and expose only task-relevant actions.
6. Reports and audit can reconstruct recipient/message/file/profile/attempt
   evidence without exposing secrets or ordinary-reader infrastructure details.
7. Clean install, upgrade, backup/restore, module permutations, version
   alignment, security audit, and target SMTP/IMAP tests pass.
8. This handbook and its adaptive Docs topics match the shipped UI wording and
   distinguish implemented behavior from planned work.

## Explicitly planned, not yet claimed

The following are part of the selected reference journey but are not implied by
the current baseline:

- the final audited **test / single send / single resend** semantics;
- reusable SMTP batch sessions and their measured throughput benefit;
- durable, idempotent Campaign report delivery through a Mail-owned outbox
  ([`govoplan-mail#17`](https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/17));
- a fully packaged one-command Campaign reference composition with production
  policy presets and target-provider certification;
- function-bound Postbox delivery (stage 2 of the reference program);
- generic workflow-driven campaign transitions.

Each item needs an owning issue, implementation, failure tests, documentation,
and release evidence before the wording above can move from planned to current.