diff --git a/Repo-docs-MAIL-HANDBOOK.md b/Repo-docs-MAIL-HANDBOOK.md new file mode 100644 index 0000000..ca21d8b --- /dev/null +++ b/Repo-docs-MAIL-HANDBOOK.md @@ -0,0 +1,491 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-mail/docs/MAIL_HANDBOOK.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# Mail Handbook + +## Purpose and status + +This handbook is the canonical multi-perspective description of the GovOPlaN +Mail module. Mail is a governed transport and mailbox capability. It is not a +campaign definition, contact database, or general records store. + +| Perspective | Start here | +| --- | --- | +| User selecting a profile or reading a mailbox | [User tasks](#user-tasks) | +| Mail profile administrator | [Profile administration](#profile-administration) | +| Policy or tenant administrator | [Policy hierarchy](#policy-hierarchy) | +| Delivery operator | [Operations and recovery](#operations-and-recovery) | +| Integrator | [Capability contract](#capability-contract) | +| Security or audit reviewer | [Security, deletion, and audit](#security-deletion-and-audit) | +| Release reviewer | [Acceptance checklist](#acceptance-checklist) | + +See also [Mail protocol roadmap](MAIL_PROTOCOL_ROADMAP.md) and the Campaign +[Mail profile boundary](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/src/branch/main/docs/MAIL_PROFILE_BOUNDARY.md). + +## Domain ownership + +Mail owns: + +- reusable SMTP/IMAP profile definitions and scope; +- encrypted SMTP/IMAP credentials and safe credential replacement; +- effective profile policy and visibility/authorization decisions; +- connection tests and protocol adapters; +- SMTP send and IMAP append operations exposed to consumers; +- read-only mailbox folder/message access and its bounded indexes; and +- the transport sanitization boundary and throttling behavior. A general + Mail-owned provider-attempt/diagnostic ledger remains planned. + +Consumers own their business intent. For example, Campaign owns message +content, recipients, approval, jobs, and delivery evidence, while Mail owns the +profile and performs the provider operation. A consumer stores only a stable +Mail profile reference and non-secret evidence returned by Mail. It never +receives decrypted credentials or a raw resolved transport configuration. + +Core owns authentication, tenant context, permission evaluation, database +session primitives, cryptographic secret helpers, audit infrastructure, and the +module registry. Optional consumers provide narrow context through capabilities; +Mail does not import their ORM or service implementations. + +## Deployment configuration packages + +Mail registers the `mail.configuration` capability for `smtp_profile` +fragments. The provider reads the validated `mail.smtp` entry from the +installer-generated infrastructure capability receipt. Receipt host and port +are authoritative; the generic package workflow asks for missing non-secret +transport fields such as security mode. An external relay may operate without +authentication, or the operator may select an existing credential-envelope id. +Inline usernames, passwords, tokens, and secret values are rejected. + +Tenant scope is the default. A system-scoped profile requires system settings +or governance write authority. The fragment's stable slug is its idempotency +identity: an absent profile is created, an exact profile is skipped, and a +conflicting profile is preserved unless the reviewed fragment explicitly sets +`on_conflict` to `update`. Credential bindings are added idempotently and are +never removed merely because a package omits a credential reference. + +Preflight does not prove SMTP reachability. After apply, use the normal Mail +profile test and Ops health surfaces. If the receipt says SMTP is unavailable, +is invalid, or is not mounted, import is blocked with an operator-facing +resolution instead of creating a partial profile. + +## Interface patterns and unavailable actions + +Mail uses the platform's shared explorer, connection tree, adaptive form, +effective-policy, dialog, status, alert, and confirmation components. The +module-owned surface inventory and consequence classification are recorded in +[Mail interface pattern migration](INTERFACE_PATTERN_MIGRATION.md). + +An unavailable action remains visible when it belongs to the current task. Its +hover/focus explanation identifies the missing field, active operation, or +permission. Larger blockers state what must change, who can change it, and +where to go. Contextual Help Center links resolve to the configured Docs module +when installed and otherwise use the public documentation fallback. + +Mailbox reads, profile reloads, connection tests, policy saves, bounce scans, +and destructive actions each expose their current loading or blocked state. +Removing a bounce watcher and deactivating profile hierarchy items use shared +confirmation dialogs. Removing a watcher retains existing observations; +deactivating a profile may scrub Mail-owned credentials as described below. + +## Concepts + +### Profile + +A profile is a reusable, named delivery identity with optional SMTP and IMAP +configuration. It has a stable id, lifecycle state, scope, owner context, and +non-secret connection metadata. Passwords are write-only encrypted values and +are never returned through list/read/capability responses. + +Profiles may be scoped to system, tenant, user, group, or campaign context. +Scope controls where a profile can be discovered; effective policy can narrow +that further. A visible profile is not automatically authorized for every +operation: using, testing, managing, and managing secrets are separate rights. + +### Policy + +Mail policy controls approved profile ids, which lower scopes may define +profiles, allowed/denied hosts and addressing patterns, credential inheritance, +and lower-level limits. The effective result is assembled from applicable +system, tenant, user/group, and campaign context. Denials and locked parent +limits cannot be relaxed by a lower scope. + +### Transport identity and revision + +Mail owns opaque random revisions for relevant SMTP/IMAP identity and +configuration. They are concurrency tokens, not deterministic hashes that a +consumer could use to guess a host or account. Credentials are excluded. A +consumer can freeze these revisions at build time and ask Mail to require the +same identity when the effect is later performed. Credential rotation alone +therefore need not invalidate prepared work, while a host/account/protocol +identity change does. + +The revision is evidence and a concurrency guard, not a substitute for +authorization. Mail re-evaluates profile activity, visibility, policy, and +revision immediately before it resolves credentials and performs the +effect. + +### Provider outcomes + +SMTP acceptance, partial or complete recipient refusal, temporary/permanent +failure, connection loss, and unknown outcome are distinct. IMAP append is a +separate operation and outcome. Ordinary Campaign recipient delivery retains +its Campaign-owned job ledger. Mail owns an encrypted durable command and +attempt ledger for report delivery and other effects that do not have such a +consumer ledger. + +The Mail worker commits an attempt before it starts SMTP and then commits an +effect-start marker before opening the provider effect. Worker redelivery may +recover a stale pre-effect claim, but an accepted, partially accepted, +in-progress-after-effect, or unknown command is never sent automatically. +Unknown outcomes require a separately authorized reconciliation with an +external evidence reference. A deliberate resend creates a new command with a +new idempotency key and links it to the prior command. + +## User tasks + +### Choose a profile + +1. Open the task that requires mail, such as Campaign **Mail settings**. +2. Choose from the profiles visible and authorized for the current tenant, + owner, group, and task context. Never enter or copy a profile id manually + when the UI can present a picker. +3. Review the safe summary: name, scope, active state, SMTP/IMAP availability, + and policy-relevant sender identity. Credentials and raw provider internals + are not visible. +4. Save the reference in the consuming module. If ownership or policy changes, + select and validate again. + +The user needs `mail:profile:use`; reading profile summaries and testing may +require their own scopes. The consuming module still requires its own action +permission. + +### Test a profile + +An authorized profile test verifies connection and authentication for the +selected active/visible SMTP or IMAP profile using Mail-owned credentials. The +consumer-use path evaluates effective Mail policy separately. Use a +non-production provider and mailbox first. A successful connection test does +not prove policy authorization for a later Campaign context, deliverability, +recipient acceptance, SPF/DKIM/DMARC alignment, or future availability. + +Testing a saved profile requires both `mail:profile:test` and +`mail:profile:use`, and the profile must be active. Profile creation or test +authority alone is not enough. + +Raw settings test endpoints accept new settings only for actors who may both +test profiles and manage secrets. They are an administration aid, not a way for +ordinary consumers to bypass reusable profiles. + +### Read a mailbox + +The current mailbox UI and API are read-only. An authorized user can list IMAP +folders, page through messages, and inspect a bounded full message. Folder +names are parsed and quoted defensively; Sent-folder discovery uses provider +flags and common names. + +Mailbox access requires both `mail:mailbox:read` and `mail:profile:use`. It must +not mutate read/unread, delete, move, or reply state. Message responses are +bounded by the deployment response policy; ordinary UI should avoid loading a +whole large mailbox or attachment merely to show a list. + +## Profile administration + +### Roles + +The supplied templates are: + +- **Mail profile user:** read/use/test approved profiles and read permitted + mailboxes without reading secrets. +- **Mail profile self-service user:** additionally create, edit, deactivate, + and manage credentials only for the current account's own user-scoped + profiles, subject to the effective Mail policy. +- **Mail profile administrator:** additionally create/update profiles and + create/replace encrypted credentials across tenant-owned scopes. + +The specific permissions are `mail:profile:read`, `mail:profile:use`, +`mail:profile:test`, `mail:mailbox:read`, `mail:profile:write_own`, +`mail:secret:manage_own`, `mail:profile:write`, and `mail:secret:manage`. +The `_own` permissions are enforced against the authenticated membership id and +never authorize a tenant, group, campaign, system, or another user's profile. +They also do not authorize profile-policy changes. System-scoped definitions +use the corresponding system settings authority. Keep secret management +separate when an institution wants profile metadata administrators not to know +or replace credentials. + +That separation is fail-closed for transport rebinding: changing an SMTP or +IMAP host, port, or security mode while the profile retains a stored password +requires the matching secret-management permission. A credential-free profile +can be deactivated with profile-write authority alone; deactivation that +scrubs a stored password also requires secret-management authority and records +the deletion in the audit log. + +### Create or change a profile + +The configured Help Center exposes **Create a custom Mail profile** only when +the current actor has broad or self-service profile-write authority and the +effective user-scope policy permits user profiles. It states the active SMTP/IMAP hostname +allow-list groups and deny rules, plus the actor's separate credential, test, +use, and approval requirements. The Settings task creates in the current +account's user scope. Grant `mail:profile:write_own` for self-service; +`mail:profile:write` remains broad profile administration authority. + +1. Choose the narrowest suitable scope and a stable, descriptive name/slug. +2. Configure SMTP, optional IMAP, TLS mode, account identity, Sent-folder + behavior, and timeouts. Sender/envelope/recipient constraints belong to + effective Mail policy; Campaign rate limits remain delivery configuration. +3. Enter credentials only in the dedicated credential fields. Returned profile + data indicates whether credentials are configured without returning them. +4. Save and run SMTP/IMAP tests against a non-production target. +5. Verify the effective policy for every intended consumer context. +6. Communicate changes that alter the non-secret transport identity; prepared + consumer snapshots will deliberately stop until rebuilt. + +An update that omits a password preserves the current encrypted password. A +credential replacement never depends on reading the old cleartext value back. + +### Delete a profile + +Profile deletion is immediate for Mail-owned secrets and audit evidence: + +1. Mail clears both encrypted SMTP and IMAP passwords in the same transaction. +2. It deactivates the remaining non-secret tombstone state so historical stable + references can fail safely rather than resolve to another profile. +3. When owned encrypted secrets existed, it emits + `mail.profile_credentials_deleted` with profile id, scope, reason, and the + kinds of secret removed—never values, usernames/passwords, or raw + configuration. +4. If scrubbing or audit insertion fails, the transaction rolls back and the + deletion fails. +5. Repeating deletion after the secrets are gone is an idempotent no-op for the + secret-deletion effect. + +Consumers that still reference the profile fail closed at validation or +execution. Deletion does not silently rebind them to a default profile. + +## Policy hierarchy + +Effective policy is contextual. Administrators should document: + +- which profile ids are approved globally or for a tenant; +- whether tenant, user, group, or campaign-scoped profiles may be created; +- allowed and denied SMTP/IMAP hosts; +- permitted From, envelope sender (including bounce address), and envelope + recipient-domain patterns; +- whether SMTP/IMAP credentials inherit from the reusable profile; and +- which lower-level settings are locked by a parent policy. + +Campaign delivery requires reusable profile credentials. A legacy policy that +requires campaign-local credentials fails closed with guidance to store them on +the Mail profile and enable effective inheritance. This preserves compatibility +of the policy model without reopening a consumer-owned secret store. + +Policy reads are available through system/tenant/context routes to suitably +authorized actors. Adaptive Docs exposes a safe explanation of the effective +tenant posture; it does not expose credential material. + +## Operations and recovery + +### Deployment egress policy + +Private-network connector access is controlled deployment-wide by +`GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS`. Whether private or public targets +are allowed, SMTP and IMAP resolve, validate, and connect to the exact approved +address records at connection time while retaining the original hostname for +TLS SNI and certificate verification. A DNS change cannot redirect the socket +after validation. + +Transports that cannot pin every connection peer or revalidate protocol-managed +redirects/referrals must fail before client construction. Do not weaken this +rule to make a target test pass; change the connector implementation or the +target topology. + +### Response and time bounds + +The shared defaults are 16 MiB for structured connector responses and 512 MiB +for file transfers. Mailbox full-message fetch additionally uses bounded IMAP +ranges and the applicable deployment limit. Operators may lower limits for +their environment. A declared or streamed over-limit response fails visibly +rather than being retained partially as if complete. + +Configure connection/read timeouts and provider-specific message/attachment +limits. A timeout after the SMTP effect may be an unknown outcome; it must not +be flattened into a safe-to-retry connection failure. + +### Throttling + +Mail rate limiting uses a Redis lock and next-allowed timestamp across worker +processes when worker mode is enabled. Direct development uses a process-local +limiter. If Redis fails in worker mode, the current implementation silently +falls back to a limiter that protects only that process; it does not provide +cluster-wide coordination or emit its own health signal. Production operators +must therefore monitor Redis independently and should stop or restrict +multi-worker delivery during an outage until explicit degradation telemetry is +implemented. + +### SMTP/IMAP incidents + +Current SMTP delivery and Sent-folder APPEND paths start a Mail-owned Core +recovery operation before the network effect. Outbox attempts use the durable +command and attempt number; Campaign jobs and single-message actions pass their +own stable attempt identifiers through the versioned capability. A matching +completed identifier is never sent or appended again merely to reconstruct +caller state. Provider acceptance, definitive rejection, and outcome-unknown +states are recorded independently of the consuming transaction. + +Mailbox folder/message indexing and configured bounce/calendar-reply scans are +read-only provider operations. They acquire distributed per-profile/folder or +per-source fences, commit bounded projection state, and verify that state in an +independent session. A failed read is rolled back and can be repeated; it is not +treated as an unknown provider mutation. + +1. Stop new consumer work if duplicate effects or credential compromise are + possible. +2. Preserve safe Mail, consumer-job, worker, and provider evidence. +3. Rotate credentials in Mail. Rebuild consumers only if the revisioned + non-secret identity also changed. +4. Treat post-command connection loss as potentially unknown until provider + evidence establishes whether SMTP accepted the message. +5. Handle IMAP append failure independently; do not resend an accepted message + merely to recreate a Sent copy. +6. Record the incident/reconciliation reference in the consuming domain's audit + trail without copying raw provider secrets or message content unnecessarily. +7. For an outbox `outcome_unknown`, reconcile the Mail command with provider + evidence. Confirmed acceptance closes the provider operation as succeeded; + confirmed absence records verified recovery and permits only a new, + deliberate attempt identifier. + +### Delivery-status and calendar-reply sources + +An authorized Mail bounce source scans a bounded IMAP UID range without +changing mailbox flags. It correlates DSN reports with durable Mail commands +and, when the optional Calendar invitation capability is active, forwards +`text/calendar` or `.ics` `METHOD:REPLY` parts to Calendar. Calendar remains +owner of attendee state; Mail records only a raw digest, mailbox coordinates, +Message-ID, and audit linkage. Ordinary or malformed calendar messages do not +block DSN progress. A repeated UID/message digest produces no second Calendar +state transition or outbound synchronization effect. + +### Backup, restore, and retirement + +Backups contain encrypted credentials and therefore need the same protection as +the live database and key material. Restoring a Mail database without the +matching encryption key makes credentials unusable; restoring it with keys can +reactivate sensitive historical state and must be controlled. + +Destructive module retirement first applies the same immediate credential +scrub/audit rule to every remaining profile, then drops Mail-owned tables after +the installer snapshot/confirmation gate. Any scrub or audit failure blocks +retirement. Historical backups are separate retained copies and require the +deployment's approved backup-destruction policy. + +## Capability contract + +Mail provides `mail.campaign_delivery` version `0.2.x`. This version is a +breaking ownership boundary: it exposes reference-oriented operations, not raw +resolved configuration or credentials. + +The Mail REST API supplies the profile list used by pickers. Given a stable +profile reference, the capability supports these kinds of operation: + +- resolve a profile reference and evaluate its Mail context/policy to produce a + non-secret delivery summary and + opaque Mail-owned revisions; +- send Campaign-owned RFC message bytes using the selected profile; +- optionally append an accepted message to Sent; and +- return sanitized outcome/revision evidence. + +For a real effect, one Mail call must: + +1. resolve current campaign/tenant/owner context through the optional narrow + context capability; +2. load the selected active profile; +3. evaluate profile-scope eligibility and effective policy; +4. compare the consumer's expected opaque transport revision; +5. decrypt credentials in Mail memory; +6. perform the SMTP or IMAP operation; and +7. return only the safe result. + +Keeping comparison, credential resolution, and transport inside one operation +avoids a time-of-check/time-of-use gap and prevents the consumer from becoming +a credential-processing boundary. The capability does not receive the acting +principal: the Campaign route must first enforce `mail:profile:use`, while Mail +still re-evaluates active profile scope and contextual policy inside the call. + +Mail optionally consumes `campaigns.mail_policy_context` and +`addresses.lookup`. Their physical absence must leave Mail installable and its +unrelated profile/mailbox behavior usable. Interface-version compatibility is a +mandatory release gate. + +## Security, deletion, and audit + +Security invariants: + +- Decrypted SMTP/IMAP passwords never cross the Mail capability/API boundary. +- Password fields are write-only and encrypted at rest; safe responses expose + configuration state, not values. +- Consumers persist stable profile references, not transport copies. +- Host policy is deployment-wide and every supported connection is DNS/IP + pinned at socket creation. +- Redirect/referral-capable transports fail closed unless every peer can be + revalidated and pinned. +- Profile deletion and destructive retirement scrub every Mail-owned encrypted + password and emit non-secret audit in the same lifecycle action. +- Mailbox reads are separately authorized, read-only, paginated, and bounded. +- Logs/audit/results minimize message content, addresses, provider responses, + and secret-like values. + +Credential replacement/deletion emits canonical non-secret Mail audit events. +Profile/policy create/update/deactivate currently emits change-feed evidence, +but connection tests and the complete administrative lifecycle do not yet have +equivalent canonical audit events. Mail delivery commands emit request, +terminal-outcome, reconciliation, and deliberate-resend events without +addresses or raw provider responses. MIME, envelope addresses, detailed +refusals, and reconciliation notes are encrypted and minimized after their +retention deadline while hashes, counts, attempts, and decisions remain. + +## Acceptance checklist + +Before claiming a Mail composition is production-ready: + +1. Install Mail alone with Core and with each optional consumer absent. +2. Prove profile scope, visibility, use, test, write, and secret permissions for + representative tenant/user/group contexts. +3. Prove policy deny precedence, locked lower limits, approved-profile mode, and + the campaign-local-credential fail-closed path. +4. Verify all read/list/capability/audit/log responses contain no plaintext or + encrypted password, raw credential token, or consumer-owned inline config. +5. Test DNS rebinding resistance and private/public deployment policy for SMTP + and IMAP. +6. Test SMTP acceptance, refusal, temporary/permanent failure, timeout/unknown + outcome, and IMAP append success/failure against the target provider. +7. Test Redis worker throttling and prove that independent infrastructure + monitoring detects Redis loss; restrict multi-worker delivery during the + current silent local-only fallback. +8. Delete tenant and system profiles, inject audit failure, repeat deletion, and + prove transactional scrub/rollback/idempotency. +9. Run destructive-retirement preflight on a snapshot and prove credentials are + scrubbed/audited before table drop. +10. Pass module permutations, interface/version alignment, WebUI/i18n, full + security audit, backup/restore, and target-environment release checks. + +## Explicitly planned, not yet claimed + +- Canonical audit events for profile tests and the remaining profile/policy + administration lifecycle, plus an operator-visible Redis-throttling + degradation signal. +- Reusable SMTP batch sessions and their final connection/error/isolation + semantics (`govoplan-mail#16`). +- Final Campaign **test / single send / single resend** semantics; those are a + Campaign business-action contract built on Mail transport operations. +- JMAP mailbox synchronization/search; it is preferred only after the IMAP MVP + is stable. +- POP3 except for a future explicit legacy download/import requirement. +- A full mail client with compose/reply/move/delete/read-state mutation. +- Recovery-ledger adoption for future provider-side move, delete, and flag + mutations; no such production path exists in the current read-only mailbox. +- Proof that process-local throttling coordinates multiple workers when Redis + is unavailable; it deliberately does not.