From 0f1327638ed2345f0fd25bcef64e6f1ad5bba790 Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Tue, 21 Jul 2026 17:29:01 +0200 Subject: [PATCH] docs: add multi-perspective Mail handbook --- docs/MAIL_HANDBOOK.md | 379 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 379 insertions(+) create mode 100644 docs/MAIL_HANDBOOK.md diff --git a/docs/MAIL_HANDBOOK.md b/docs/MAIL_HANDBOOK.md new file mode 100644 index 0000000..1818734 --- /dev/null +++ b/docs/MAIL_HANDBOOK.md @@ -0,0 +1,379 @@ +# Mail Handbook + +## Purpose and status + +This handbook is the canonical multi-perspective description of the GovOPlaN +Mail module. Mail is a governed transport and mailbox capability. It is not a +campaign definition, contact database, or general records store. + +| Perspective | Start here | +| --- | --- | +| User selecting a profile or reading a mailbox | [User tasks](#user-tasks) | +| Mail profile administrator | [Profile administration](#profile-administration) | +| Policy or tenant administrator | [Policy hierarchy](#policy-hierarchy) | +| Delivery operator | [Operations and recovery](#operations-and-recovery) | +| Integrator | [Capability contract](#capability-contract) | +| Security or audit reviewer | [Security, deletion, and audit](#security-deletion-and-audit) | +| Release reviewer | [Acceptance checklist](#acceptance-checklist) | + +See also [Mail protocol roadmap](MAIL_PROTOCOL_ROADMAP.md) and the Campaign +[Mail profile boundary](https://git.add-ideas.de/add-ideas/govoplan-campaign/src/branch/main/docs/MAIL_PROFILE_BOUNDARY.md). + +## Domain ownership + +Mail owns: + +- reusable SMTP/IMAP profile definitions and scope; +- encrypted SMTP/IMAP credentials and safe credential replacement; +- effective profile policy and visibility/authorization decisions; +- connection tests and protocol adapters; +- SMTP send and IMAP append operations exposed to consumers; +- read-only mailbox folder/message access and its bounded indexes; and +- the transport sanitization boundary and throttling behavior. A general + Mail-owned provider-attempt/diagnostic ledger remains planned. + +Consumers own their business intent. For example, Campaign owns message +content, recipients, approval, jobs, and delivery evidence, while Mail owns the +profile and performs the provider operation. A consumer stores only a stable +Mail profile reference and non-secret evidence returned by Mail. It never +receives decrypted credentials or a raw resolved transport configuration. + +Core owns authentication, tenant context, permission evaluation, database +session primitives, cryptographic secret helpers, audit infrastructure, and the +module registry. Optional consumers provide narrow context through capabilities; +Mail does not import their ORM or service implementations. + +## Concepts + +### Profile + +A profile is a reusable, named delivery identity with optional SMTP and IMAP +configuration. It has a stable id, lifecycle state, scope, owner context, and +non-secret connection metadata. Passwords are write-only encrypted values and +are never returned through list/read/capability responses. + +Profiles may be scoped to system, tenant, user, group, or campaign context. +Scope controls where a profile can be discovered; effective policy can narrow +that further. A visible profile is not automatically authorized for every +operation: using, testing, managing, and managing secrets are separate rights. + +### Policy + +Mail policy controls approved profile ids, which lower scopes may define +profiles, allowed/denied hosts and addressing patterns, credential inheritance, +and lower-level limits. The effective result is assembled from applicable +system, tenant, user/group, and campaign context. Denials and locked parent +limits cannot be relaxed by a lower scope. + +### Transport identity and revision + +Mail owns opaque random revisions for relevant SMTP/IMAP identity and +configuration. They are concurrency tokens, not deterministic hashes that a +consumer could use to guess a host or account. Credentials are excluded. A +consumer can freeze these revisions at build time and ask Mail to require the +same identity when the effect is later performed. Credential rotation alone +therefore need not invalidate prepared work, while a host/account/protocol +identity change does. + +The revision is evidence and a concurrency guard, not a substitute for +authorization. Mail re-evaluates profile activity, visibility, policy, and +revision immediately before it resolves credentials and performs the +effect. + +### Provider outcomes + +SMTP acceptance, recipient refusal, temporary/permanent failure, connection +loss, and unknown outcome are distinct. IMAP append is a separate operation and +outcome. Mail returns a sanitized transport result; the consuming domain owns +the durable business job and its retry/reconciliation semantics. + +## User tasks + +### Choose a profile + +1. Open the task that requires mail, such as Campaign **Mail settings**. +2. Choose from the profiles visible and authorized for the current tenant, + owner, group, and task context. Never enter or copy a profile id manually + when the UI can present a picker. +3. Review the safe summary: name, scope, active state, SMTP/IMAP availability, + and policy-relevant sender identity. Credentials and raw provider internals + are not visible. +4. Save the reference in the consuming module. If ownership or policy changes, + select and validate again. + +The user needs `mail:profile:use`; reading profile summaries and testing may +require their own scopes. The consuming module still requires its own action +permission. + +### Test a profile + +An authorized profile test verifies connection and authentication for the +selected active/visible SMTP or IMAP profile using Mail-owned credentials. The +consumer-use path evaluates effective Mail policy separately. Use a +non-production provider and mailbox first. A successful connection test does +not prove policy authorization for a later Campaign context, deliverability, +recipient acceptance, SPF/DKIM/DMARC alignment, or future availability. + +Raw settings test endpoints accept new settings only for actors who may both +test profiles and manage secrets. They are an administration aid, not a way for +ordinary consumers to bypass reusable profiles. + +### Read a mailbox + +The current mailbox UI and API are read-only. An authorized user can list IMAP +folders, page through messages, and inspect a bounded full message. Folder +names are parsed and quoted defensively; Sent-folder discovery uses provider +flags and common names. + +Mailbox access requires both `mail:mailbox:read` and `mail:profile:use`. It must +not mutate read/unread, delete, move, or reply state. Message responses are +bounded by the deployment response policy; ordinary UI should avoid loading a +whole large mailbox or attachment merely to show a list. + +## Profile administration + +### Roles + +The supplied templates are: + +- **Mail profile user:** read/use/test approved profiles and read permitted + mailboxes without reading secrets. +- **Mail profile administrator:** additionally create/update profiles and + create/replace encrypted credentials. + +The specific permissions are `mail:profile:read`, `mail:profile:use`, +`mail:profile:test`, `mail:mailbox:read`, `mail:profile:write`, and +`mail:secret:manage`. System-scoped definitions use the corresponding system +settings authority. Keep secret management separate when an institution wants +profile metadata administrators not to know or replace credentials. + +### Create or change a profile + +1. Choose the narrowest suitable scope and a stable, descriptive name/slug. +2. Configure SMTP, optional IMAP, TLS mode, account identity, Sent-folder + behavior, and timeouts. Sender/envelope/recipient constraints belong to + effective Mail policy; Campaign rate limits remain delivery configuration. +3. Enter credentials only in the dedicated credential fields. Returned profile + data indicates whether credentials are configured without returning them. +4. Save and run SMTP/IMAP tests against a non-production target. +5. Verify the effective policy for every intended consumer context. +6. Communicate changes that alter the non-secret transport identity; prepared + consumer snapshots will deliberately stop until rebuilt. + +An update that omits a password preserves the current encrypted password. A +credential replacement never depends on reading the old cleartext value back. + +### Delete a profile + +Profile deletion is immediate for Mail-owned secrets and audit evidence: + +1. Mail clears both encrypted SMTP and IMAP passwords in the same transaction. +2. It deactivates the remaining non-secret tombstone state so historical stable + references can fail safely rather than resolve to another profile. +3. When owned encrypted secrets existed, it emits + `mail.profile_credentials_deleted` with profile id, scope, reason, and the + kinds of secret removed—never values, usernames/passwords, or raw + configuration. +4. If scrubbing or audit insertion fails, the transaction rolls back and the + deletion fails. +5. Repeating deletion after the secrets are gone is an idempotent no-op for the + secret-deletion effect. + +Consumers that still reference the profile fail closed at validation or +execution. Deletion does not silently rebind them to a default profile. + +## Policy hierarchy + +Effective policy is contextual. Administrators should document: + +- which profile ids are approved globally or for a tenant; +- whether tenant, user, group, or campaign-scoped profiles may be created; +- allowed and denied SMTP/IMAP hosts; +- permitted From, envelope sender (including bounce address), and envelope + recipient-domain patterns; +- whether SMTP/IMAP credentials inherit from the reusable profile; and +- which lower-level settings are locked by a parent policy. + +Campaign delivery requires reusable profile credentials. A legacy policy that +requires campaign-local credentials fails closed with guidance to store them on +the Mail profile and enable effective inheritance. This preserves compatibility +of the policy model without reopening a consumer-owned secret store. + +Policy reads are available through system/tenant/context routes to suitably +authorized actors. Adaptive Docs exposes a safe explanation of the effective +tenant posture; it does not expose credential material. + +## Operations and recovery + +### Deployment egress policy + +Private-network connector access is controlled deployment-wide by +`GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS`. Whether private or public targets +are allowed, SMTP and IMAP resolve, validate, and connect to the exact approved +address records at connection time while retaining the original hostname for +TLS SNI and certificate verification. A DNS change cannot redirect the socket +after validation. + +Transports that cannot pin every connection peer or revalidate protocol-managed +redirects/referrals must fail before client construction. Do not weaken this +rule to make a target test pass; change the connector implementation or the +target topology. + +### Response and time bounds + +The shared defaults are 16 MiB for structured connector responses and 512 MiB +for file transfers. Mailbox full-message fetch additionally uses bounded IMAP +ranges and the applicable deployment limit. Operators may lower limits for +their environment. A declared or streamed over-limit response fails visibly +rather than being retained partially as if complete. + +Configure connection/read timeouts and provider-specific message/attachment +limits. A timeout after the SMTP effect may be an unknown outcome; it must not +be flattened into a safe-to-retry connection failure. + +### Throttling + +Mail rate limiting uses a Redis lock and next-allowed timestamp across worker +processes when worker mode is enabled. Direct development uses a process-local +limiter. If Redis fails in worker mode, the current implementation silently +falls back to a limiter that protects only that process; it does not provide +cluster-wide coordination or emit its own health signal. Production operators +must therefore monitor Redis independently and should stop or restrict +multi-worker delivery during an outage until explicit degradation telemetry is +implemented. + +### SMTP/IMAP incidents + +1. Stop new consumer work if duplicate effects or credential compromise are + possible. +2. Preserve safe Mail, consumer-job, worker, and provider evidence. +3. Rotate credentials in Mail. Rebuild consumers only if the revisioned + non-secret identity also changed. +4. Treat post-command connection loss as potentially unknown until provider + evidence establishes whether SMTP accepted the message. +5. Handle IMAP append failure independently; do not resend an accepted message + merely to recreate a Sent copy. +6. Record the incident/reconciliation reference in the consuming domain's audit + trail without copying raw provider secrets or message content unnecessarily. + +### Backup, restore, and retirement + +Backups contain encrypted credentials and therefore need the same protection as +the live database and key material. Restoring a Mail database without the +matching encryption key makes credentials unusable; restoring it with keys can +reactivate sensitive historical state and must be controlled. + +Destructive module retirement first applies the same immediate credential +scrub/audit rule to every remaining profile, then drops Mail-owned tables after +the installer snapshot/confirmation gate. Any scrub or audit failure blocks +retirement. Historical backups are separate retained copies and require the +deployment's approved backup-destruction policy. + +## Capability contract + +Mail provides `mail.campaign_delivery` version `0.2.x`. This version is a +breaking ownership boundary: it exposes reference-oriented operations, not raw +resolved configuration or credentials. + +The Mail REST API supplies the profile list used by pickers. Given a stable +profile reference, the capability supports these kinds of operation: + +- resolve a profile reference and evaluate its Mail context/policy to produce a + non-secret delivery summary and + opaque Mail-owned revisions; +- send Campaign-owned RFC message bytes using the selected profile; +- optionally append an accepted message to Sent; and +- return sanitized outcome/revision evidence. + +For a real effect, one Mail call must: + +1. resolve current campaign/tenant/owner context through the optional narrow + context capability; +2. load the selected active profile; +3. evaluate profile-scope eligibility and effective policy; +4. compare the consumer's expected opaque transport revision; +5. decrypt credentials in Mail memory; +6. perform the SMTP or IMAP operation; and +7. return only the safe result. + +Keeping comparison, credential resolution, and transport inside one operation +avoids a time-of-check/time-of-use gap and prevents the consumer from becoming +a credential-processing boundary. The capability does not receive the acting +principal: the Campaign route must first enforce `mail:profile:use`, while Mail +still re-evaluates active profile scope and contextual policy inside the call. + +Mail optionally consumes `campaigns.mail_policy_context` and +`addresses.lookup`. Their physical absence must leave Mail installable and its +unrelated profile/mailbox behavior usable. Interface-version compatibility is a +mandatory release gate. + +## Security, deletion, and audit + +Security invariants: + +- Decrypted SMTP/IMAP passwords never cross the Mail capability/API boundary. +- Password fields are write-only and encrypted at rest; safe responses expose + configuration state, not values. +- Consumers persist stable profile references, not transport copies. +- Host policy is deployment-wide and every supported connection is DNS/IP + pinned at socket creation. +- Redirect/referral-capable transports fail closed unless every peer can be + revalidated and pinned. +- Profile deletion and destructive retirement scrub every Mail-owned encrypted + password and emit non-secret audit in the same lifecycle action. +- Mailbox reads are separately authorized, read-only, paginated, and bounded. +- Logs/audit/results minimize message content, addresses, provider responses, + and secret-like values. + +Credential replacement/deletion emits canonical non-secret Mail audit events. +Profile/policy create/update/deactivate currently emits change-feed evidence, +but connection tests and the complete administrative lifecycle do not yet have +equivalent canonical audit events. Consumer send/retry/reconciliation evidence +belongs primarily to the consuming module today. A Mail-owned restricted +provider-attempt ledger with correlation, retention, and unknown-outcome +reconciliation remains part of the durable outbox work in +[`govoplan-mail#17`](https://git.add-ideas.de/add-ideas/govoplan-mail/issues/17). + +## Acceptance checklist + +Before claiming a Mail composition is production-ready: + +1. Install Mail alone with Core and with each optional consumer absent. +2. Prove profile scope, visibility, use, test, write, and secret permissions for + representative tenant/user/group contexts. +3. Prove policy deny precedence, locked lower limits, approved-profile mode, and + the campaign-local-credential fail-closed path. +4. Verify all read/list/capability/audit/log responses contain no plaintext or + encrypted password, raw credential token, or consumer-owned inline config. +5. Test DNS rebinding resistance and private/public deployment policy for SMTP + and IMAP. +6. Test SMTP acceptance, refusal, temporary/permanent failure, timeout/unknown + outcome, and IMAP append success/failure against the target provider. +7. Test Redis worker throttling and prove that independent infrastructure + monitoring detects Redis loss; restrict multi-worker delivery during the + current silent local-only fallback. +8. Delete tenant and system profiles, inject audit failure, repeat deletion, and + prove transactional scrub/rollback/idempotency. +9. Run destructive-retirement preflight on a snapshot and prove credentials are + scrubbed/audited before table drop. +10. Pass module permutations, interface/version alignment, WebUI/i18n, full + security audit, backup/restore, and target-environment release checks. + +## Explicitly planned, not yet claimed + +- Durable, idempotent Campaign report delivery with Mail-owned attempts, + unknown-outcome reconciliation, and partial-refusal evidence + ([`govoplan-mail#17`](https://git.add-ideas.de/add-ideas/govoplan-mail/issues/17)). +- Canonical audit events for profile tests and the remaining profile/policy + administration lifecycle, plus an operator-visible Redis-throttling + degradation signal. +- Reusable SMTP batch sessions and their final connection/error/isolation + semantics (`govoplan-mail#16`). +- Final Campaign **test / single send / single resend** semantics; those are a + Campaign business-action contract built on Mail transport operations. +- JMAP mailbox synchronization/search; it is preferred only after the IMAP MVP + is stable. +- POP3 except for a future explicit legacy download/import requirement. +- A full mail client with compose/reply/move/delete/read-state mutation. +- Proof that process-local throttling coordinates multiple workers when Redis + is unavailable; it deliberately does not.