From 25a69b3fa9810ba271deff12d4e050e288f1fad6 Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Tue, 21 Jul 2026 17:29:01 +0200 Subject: [PATCH] docs: add multi-perspective Campaign handbook --- docs/CAMPAIGN_HANDBOOK.md | 452 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 452 insertions(+) create mode 100644 docs/CAMPAIGN_HANDBOOK.md diff --git a/docs/CAMPAIGN_HANDBOOK.md b/docs/CAMPAIGN_HANDBOOK.md new file mode 100644 index 0000000..1539b8b --- /dev/null +++ b/docs/CAMPAIGN_HANDBOOK.md @@ -0,0 +1,452 @@ +# Campaign Handbook + +## Purpose and status + +This is the canonical, multi-perspective handbook for the Campaign module. It +describes the current implementation, the operational contract it relies on, +and the remaining work required before Campaign can be presented as GovOPlaN's +maintained reference composition. + +Use the section that matches the task at hand: + +| Perspective | Start here | +| --- | --- | +| Campaign author | [Prepare a campaign](#prepare-a-campaign) | +| Reviewer | [Review and complete review](#review-and-complete-review) | +| Sender or delivery operator | [Deliver and resolve outcomes](#deliver-and-resolve-outcomes) | +| Campaign or tenant administrator | [Administration and policy](#administration-and-policy) | +| Platform operator | [Operations and recovery](#operations-and-recovery) | +| Integrator or developer | [Composition and integration contracts](#composition-and-integration-contracts) | +| Security, privacy, or audit reviewer | [Assurance model](#assurance-model) | +| Release reviewer | [Reference-composition acceptance](#reference-composition-acceptance) | + +The shorter task documents remain useful companions: + +- [Campaign delivery runbook](CAMPAIGN_DELIVERY_RUNBOOK.md) +- [Mail profile boundary](MAIL_PROFILE_BOUNDARY.md) +- [Recipient import guide](RECIPIENT_IMPORT_GUIDE.md) +- [Recipient and Addresses boundary](RECIPIENT_ADDRESS_BOUNDARY.md) +- [Examples and release checklist](EXAMPLE_CAMPAIGNS_AND_RELEASE_CHECKLIST.md) + +## What Campaign is for + +Campaign turns governed source data into individually built messages and then +controls their review, delivery, and evidence. It is intentionally a +composition module: it demonstrates how one user journey can use optional Mail, +Files, Addresses, and Notifications capabilities alongside Core access/audit +infrastructure without copying ownership from those modules. Policy may consume +Campaign context through a narrow capability; Campaign does not import Policy. + +Campaign owns: + +- the communication purpose, content, campaign-local fields, and templates; +- campaign-local recipient snapshots, exclusions, and personalization; +- message and attachment rules for a version; +- validation, review, build, queue, and delivery-control state; +- the durable jobs and attempts needed to explain delivery outcomes; and +- campaign-specific reports, shares, and frozen execution evidence. + +Campaign does not own: + +- SMTP/IMAP profiles, credentials, protocol adapters, or mailbox policy; +- long-lived address-book master data, consent lifecycle, or deduplication; +- managed file bytes, connector credentials, or external-file provenance; +- general-purpose workflow definitions; or +- global identity, organization, permission, or retention policy. + +Those boundaries matter in both persistence and UI. A campaign references an +authorized Mail profile and managed file versions; it must never become a +second secret store, file store, or address directory. + +## Process view + +The supported process is a controlled progression, not a single "send" call: + +```text +create/edit + -> validate and resolve policy/integrations + -> review warnings and blockers + -> build exact recipient messages + -> complete review and queue + -> SMTP attempt per job + -> optional IMAP append per accepted job + -> report, retry, reconcile, or correct + -> archive when no active/uncertain delivery remains +``` + +Campaign status summarizes the whole campaign. Version workflow state describes +the selected immutable/editable version. Each recipient job separately records +build, validation, queue, SMTP, and IMAP state. Operators must use the job-level +states when deciding whether another external effect is safe. + +Important distinctions: + +- **Validation lock** is the reversible lock created by a successful + validation/build path. Editing requires unlocking and invalidates derived + evidence as appropriate. +- **User lock** is an explicit audit-safe lock. A permanently locked or + delivery-final version is not edited in place; create an editable successor. +- **SMTP accepted** means the provider accepted the message. It does not prove + inbox delivery, reading, or business acknowledgement. +- **Outcome unknown** means an attempt may have had an external effect. It must + be investigated and reconciled before retry. +- **IMAP append** is a separate effect after SMTP acceptance. An append failure + is not evidence that sending failed. +- **Archive** preserves evidence. Draft-only campaigns without built, locked, + or delivery evidence may be deleted where policy allows; evidence-bearing + campaigns are archived instead. + +## User tasks + +### Prepare a campaign + +1. Create a campaign and confirm its owner or owning group. +2. Define global settings, fields, templates, and recipient data. +3. Import one-off recipient data or select a reusable Addresses source when the + optional capability is installed. Review source provenance and stale-source + warnings; Campaign freezes the selected rows rather than following later + directory changes silently. +4. Select managed attachments through Files. Server/API campaigns do not accept + arbitrary local paths. Preview rules and unmatched files before building. +5. Open **Mail settings** and select an available Mail profile. The campaign + stores only `server.mail_profile_id`; it never accepts SMTP/IMAP settings, + usernames, passwords, or credential references. +6. Save the editable version, validate the relevant sections, and resolve every + blocking issue. Warnings remain explicit review decisions. +7. Build the exact messages and inspect recipient, addressing, template, + attachment, and generated-message evidence. + +If a selected optional module is absent, Campaign remains loadable and explains +which function is unavailable. It must not fail startup because Mail, Files, or +Addresses is not installed. + +### Review and complete review + +The reviewer should verify the immutable candidate that will be delivered, not +just the authoring form: + +1. Confirm purpose, owner, selected version, and recipient count. +2. Inspect blocking errors, warnings, exclusions, and recipients requiring + review. +3. Inspect representative and exceptional rendered messages, including From, + To/CC/BCC, Reply-To, subject, body, and attachment evidence. +4. Confirm the selected Mail profile is authorized for the campaign's current + tenant and owner context. +5. Confirm attachment behavior when a rule matches no files, ZIP/password + behavior, and any recipient-specific files. +6. Record review completion and the inspected message keys through the review + surface. The current baseline does not persist a distinct approve/reject + decision or review reason. If content, recipients, attachment inputs, owner + context, or non-secret transport identity changes, revalidate and rebuild. + +Normal readers and reviewers see business state and safe evidence. Process-local +paths, storage keys, worker claim tokens, and raw provider diagnostics require +the dedicated diagnostic permission and must not leak through ordinary campaign, +version, job, or report responses. + +### Deliver and resolve outcomes + +Use the [delivery runbook](CAMPAIGN_DELIVERY_RUNBOOK.md) for the detailed +operator sequence. + +At a minimum: + +1. Queue only a validated, locked, built version. +2. Use worker delivery for ordinary batches; synchronous send is intentionally + limited to small controlled runs. +3. Treat `smtp_accepted` as protected from ordinary retry. +4. Retry `failed_temporary` explicitly after inspecting the cause. +5. Include `failed_permanent` only after correcting the cause and making a + conscious override. +6. Never retry `outcome_unknown` blindly. Inspect SMTP/provider evidence and + reconcile it as **accepted** or **not sent**, with a note identifying the + evidence. +7. Process append-to-Sent only for SMTP-accepted jobs and investigate append + failure independently. Never retry an `outcome_unknown` IMAP append blindly: + reconcile mailbox evidence as **appended** or **not appended** with a note. + Only the latter becomes explicitly retryable, and neither decision resends + the already SMTP-accepted message. +8. Archive only after active and uncertain effects are resolved. + +Pause stops new eligible work but cannot undo a provider effect already in +progress. Cancel marks work that has not yet produced a protected SMTP outcome; +it cannot recall accepted mail. + +### Test and one-message actions + +The current baseline includes mock send, queue dry-run, synchronous small-run +delivery, sending one selected job, retry selection, and unattempted-job +selection. Their audit and state behavior is not yet the final vocabulary +accepted for issue `govoplan-campaign#69`. + +The intended distinction is: + +- **Test:** deliver once to the configured test path, audit it, and leave the + production job unsent. +- **Single send:** send one unsent job, audit it, and mark its production effect + complete. +- **Single resend:** intentionally send one job again regardless of an earlier + failure or acceptance, with distinct authority, warning, reason, and audit. + +Until that slice is implemented and target-tested, do not present the existing +buttons as a complete resend policy. Ordinary retry protection remains the safe +default. + +## Data and evidence model + +### Versions and snapshots + +Editable campaign JSON is versioned. Build creates recipient jobs and an +execution snapshot. A new profile-only snapshot contains the stable Mail +profile id, delivery policy/evidence, and opaque Mail-issued transport +revisions. It contains no resolved SMTP/IMAP configuration or credentials. + +At delivery time Mail re-authorizes the profile, compares the expected opaque +revisions, resolves credentials inside the same Mail-owned operation, and +performs the transport effect. Campaign receives only the sanitized result it +needs for job state and evidence. This same-call check prevents a +validate-then-use race across the module boundary. + +Credential rotation that does not change the non-secret transport identity may +continue to satisfy the snapshot. A host, account identity, protocol, sender, +or other revisioned transport change stops delivery until the campaign is +revalidated and rebuilt. + +### Legacy campaign records + +Historical versions may contain inline SMTP/IMAP material. They are retained as +audit records, but: + +- public responses remove legacy transport fields and secrets; +- validation, build, queue, retry, and delivery fail closed; +- an editable version changes to profile-only form only through an explicit + Mail-settings save; and +- a locked version is preserved and must be forked to an editable successor. + +This is execution containment, not complete historical secret remediation. +Operators still need a separately approved migration/retention procedure for +old database snapshots and backups that may contain legacy campaign secrets; +that decision and its acceptance conditions are tracked in +[`govoplan-campaign#77`](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/77). + +### Recipient and attachment evidence + +The delivery record should be able to identify: + +- source/import context and the frozen recipient row; +- effective addressing and message id; +- template inputs and unresolved-placeholder decisions; +- managed file/version ids, source provenance, checksums, and ZIP evidence; +- generated EML checksum and size; +- Mail profile reference and opaque transport revisions; +- each SMTP and IMAP attempt, its classification, safe provider response, and + reconciliation note; and +- actor/system trigger, timestamps, policy context, and corrections. + +## Administration and policy + +### Roles and permissions + +The supplied role templates deliberately separate preparation, review, and +delivery: + +- **Campaign manager** prepares, validates, and builds campaigns and recipients. +- **Campaign reviewer** inspects prepared material and records review + completion for the exact messages checked. +- **Campaign sender** queues, sends, pauses/resumes/cancels, retries, reconciles, + reads reports, and has diagnostic access. + +Administrators may compose narrower roles from the declared permissions. Keep +these separations where institutional policy requires four-eyes approval. Mail +profile use additionally requires `mail:profile:use`; profile and credential +administration remains a Mail permission. + +Campaign access is also constrained by tenant, owner/group context, and explicit +shares. A share grants only its declared campaign permission; it does not grant +Mail credentials, Files administration, or tenant-wide recipient access. + +### Configuration checklist + +- Install compatible Core and Campaign versions. Access and base audit are Core + infrastructure; install optional Mail, Files, Addresses, Notifications, and + Policy modules only where the configured journey requires them. +- Configure Mail profiles and policy in Mail, not in campaign fixtures or JSON. +- Configure Files storage/connectors and attachment permissions in Files. +- Define role assignments and separation-of-duty policy. +- Configure Redis/Celery for durable batch workers where production volume + requires it. Local execution is a development/small-run mode, not horizontal + worker coordination. +- Set rate limits for the target provider. Mail may use Redis for shared + throttling when present and a process-local fallback in development. +- Establish retention, deletion, archive, report-export, and diagnostic-access + policy before production recipient data is loaded. +- Use non-production SMTP/IMAP identities and the maintained examples before + allowing a production profile. + +### Owner transfer + +Mail profile visibility can depend on campaign owner or group. Transferring an +editable campaign with a selected profile clears/requires reselection and +revalidation. A locked or delivery-final version is never silently rewritten; +create an editable successor. + +## Operations and recovery + +### Health to observe + +- database and migration health; +- compatible module interface versions; +- queue publication, worker heartbeat, claim age, and backlog; +- SMTP/IMAP profile test result and deployment egress policy; +- Mail profile authorization/transport-revision mismatch; +- Files resolution and frozen attachment availability; +- counts by queue, SMTP, IMAP, and reconciliation state; and +- audit and report generation failures. + +### Worker interruption + +A crashed worker can leave a job claimed or sending. Do not infer non-delivery +from worker loss. If the SMTP boundary may have been crossed, classify the +outcome as unknown and reconcile from external evidence. Only work that is +provably unattempted may be returned to an ordinary queue. + +### Retry and reconciliation + +Retries create new attempt evidence; they do not overwrite the previous +attempt. Reconciliation is a privileged factual correction supported by an +operator note. Repeated automated requests should be idempotent for the same +eligible job state; a deliberate resend is a different, not-yet-finalized +business action and must never be disguised as a retry. + +### Backups and restoration + +Back up Campaign, Mail, Files, Core/Audit, and shared storage consistently for +the composition. A database restore without generated EML/file storage, or file +storage without matching metadata, does not reconstruct the evidence chain. +After restore, keep outbound delivery paused until queue/attempt state and +provider evidence have been reconciled; never let restored accepted jobs send +again merely because a queue message was lost. + +### Incident handling + +1. Pause new delivery when duplicate or unknown effects are possible. +2. Preserve database, queue, provider, worker, and audit evidence. +3. Scope affected campaigns/jobs without exposing recipient content broadly. +4. Reconcile uncertain jobs individually or through an approved bounded tool. +5. Correct configuration in its owning module, then revalidate/rebuild where + revisioned transport inputs changed. +6. Record the incident reference and recovery rationale in audit evidence. + +## Composition and integration contracts + +Campaign has one required platform dependency: Core. Optional module behavior +is discovered through versioned, Core-mediated capabilities; Campaign must not +import optional sibling ORM, services, or WebUI implementation. + +Current principal contracts include: + +| Contract | Direction | Purpose | +| --- | --- | --- | +| `mail.campaign_delivery` 0.2.x | Mail -> Campaign | Summarize a known reference; authorize and revision-gate it; send/append using Mail-owned configuration and credentials; return sanitized results | +| `files.campaign_attachments` 0.1.x | Files -> Campaign | Select/materialize governed file versions and preserve campaign usage/evidence | +| `addresses.lookup` 0.1.x | Addresses -> Campaign | Optional address suggestions | +| `addresses.recipient_source` 0.1.x | Addresses -> Campaign | Optional versioned recipient-source snapshots | +| `campaigns.access` 0.1.x | Campaign -> platform | Explain campaign access/existence without exporting ORM objects | +| `campaigns.mail_policy_context` 0.1.x | Campaign -> Mail | Resolve campaign tenant/owner context for Mail policy | +| `campaigns.delivery_tasks` 0.1.x | Campaign -> workers | Execute narrow queued send/append tasks | +| `campaigns.retention` 0.1.x | Campaign -> retention | Apply Campaign-owned retention behavior | + +Breaking payload or ownership changes require an interface-version bump and a +release-composition alignment gate. Optional absence must be tested physically, +not only hidden in navigation. + +### External API expectations + +- Tenant and campaign access are evaluated for every operation. +- Writes require CSRF/auth behavior supplied by Core and the specific declared + scope. +- Queue/retry/reconcile endpoints operate on persisted state and return safe + summaries; initiating HTTP success is not proof of external delivery. +- Delta endpoints are optimization surfaces, not a separate source of truth. +- Report exports contain permitted business/evidence fields but no credentials, + local paths, storage keys, or worker claims. + +## Assurance model + +### Security invariants + +- No new campaign payload, fixture, response, or execution snapshot contains + SMTP/IMAP settings or credentials. +- Mail resolves credentials and performs transports inside Mail-owned calls. +- Every real connector peer is validated and pinned at connection time under + deployment-wide private-network policy. +- API/server attachment paths use managed Files references; arbitrary and + traversal-capable local paths are rejected. +- Public responses recursively remove infrastructure locators and secret-like + legacy fields. +- Accepted and outcome-unknown jobs are protected from ordinary retry. +- Diagnostic permission is separate from campaign read/report access. + +### Privacy + +Recipient fields and rendered messages may contain personal or sensitive data. +Grant recipient read/export, report export, and diagnostic access separately. +Prefer aggregate status for readers who do not need recipient detail. Define +purpose, lawful basis, minimization, export control, and retention before the +campaign starts; do not use Campaign as a substitute consent or address-master +system. + +### Audit and destructive actions + +Material authoring, validation, locking, review, queueing, send, retry, +reconciliation, sharing, owner transfer, archive, and permitted deletion actions +emit attributable evidence. Audit details must be non-secret and should refer to +stable ids rather than repeat message bodies or credentials. + +Draft deletion is allowed only while no audit-relevant build, delivery job, or +lock exists. Evidence-bearing campaigns are archived. Destructive module +retirement remains a separately confirmed installer operation with backup and +retirement evidence. + +## Reference-composition acceptance + +Campaign is ready to serve as the demonstration module only when all of the +following are repeatable in a pinned clean installation: + +1. The maintained examples validate and build with Mail/Files present, and + Campaign still starts with each optional module absent. +2. A user can import recipients, select managed files, choose an authorized Mail + profile, validate, review, build, and queue without entering transport ids or + secrets manually. +3. Campaign JSON and all ordinary APIs reject/omit inline transport material; + legacy records are visible as migration-required and cannot execute. +4. SMTP success, temporary/permanent failure, connection loss, worker loss, + outcome unknown, retry, reconciliation, IMAP success, and IMAP failure have + tested, non-duplicating outcomes against the target environment. +5. Author, reviewer, sender/operator, reader, and administrator views use the + central component system and expose only task-relevant actions. +6. Reports and audit can reconstruct recipient/message/file/profile/attempt + evidence without exposing secrets or ordinary-reader infrastructure details. +7. Clean install, upgrade, backup/restore, module permutations, version + alignment, security audit, and target SMTP/IMAP tests pass. +8. This handbook and its adaptive Docs topics match the shipped UI wording and + distinguish implemented behavior from planned work. + +## Explicitly planned, not yet claimed + +The following are part of the selected reference journey but are not implied by +the current baseline: + +- the final audited **test / single send / single resend** semantics; +- reusable SMTP batch sessions and their measured throughput benefit; +- durable, idempotent Campaign report delivery through a Mail-owned outbox + ([`govoplan-mail#17`](https://git.add-ideas.de/add-ideas/govoplan-mail/issues/17)); +- a fully packaged one-command Campaign reference composition with production + policy presets and target-provider certification; +- function-bound Postbox delivery (stage 2 of the reference program); +- generic workflow-driven campaign transitions; and +- complete remediation of legacy secrets in historical databases and backups + ([`govoplan-campaign#77`](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/77)). + +Each item needs an owning issue, implementation, failure tests, documentation, +and release evidence before the wording above can move from planned to current.