govoplan-mail/docs/MAIL_HANDBOOK.md

Mail Handbook

Purpose and status

This handbook is the canonical multi-perspective description of the GovOPlaN Mail module. Mail is a governed transport and mailbox capability. It is not a campaign definition, contact database, or general records store.

Perspective	Start here
User selecting a profile or reading a mailbox	User tasks
Mail profile administrator	Profile administration
Policy or tenant administrator	Policy hierarchy
Delivery operator	Operations and recovery
Integrator	Capability contract
Security or audit reviewer	Security, deletion, and audit
Release reviewer	Acceptance checklist

See also Mail protocol roadmap and the Campaign Mail profile boundary.

Domain ownership

Mail owns:

• reusable SMTP/IMAP profile definitions, dedicated legacy POP3 sources, and scope;
• encrypted SMTP/IMAP/POP3 credentials and safe credential replacement;
• effective profile policy and visibility/authorization decisions;
• connection tests and protocol adapters;
• SMTP send and IMAP append operations exposed to consumers;
• read-only mailbox folder/message access, bounded indexes, and encrypted pending-review records imported from legacy POP3 sources; and
• the transport sanitization boundary and throttling behavior. A general Mail-owned provider-attempt/diagnostic ledger remains planned.

Consumers own their business intent. For example, Campaign owns message content, recipients, approval, jobs, and delivery evidence, while Mail owns the profile and performs the provider operation. A consumer stores only a stable Mail profile reference and non-secret evidence returned by Mail. It never receives decrypted credentials or a raw resolved transport configuration.

Core owns authentication, tenant context, permission evaluation, database session primitives, cryptographic secret helpers, audit infrastructure, and the module registry. Optional consumers provide narrow context through capabilities; Mail does not import their ORM or service implementations.

Deployment configuration packages

Mail registers the mail.configuration capability for smtp_profile fragments. The provider reads the validated mail.smtp entry from the installer-generated infrastructure capability receipt. Receipt host and port are authoritative; the generic package workflow asks for missing non-secret transport fields such as security mode. An external relay may operate without authentication, or the operator may select an existing credential-envelope id. Inline usernames, passwords, tokens, and secret values are rejected.

Tenant scope is the default. A system-scoped profile requires system settings or governance write authority. The fragment's stable slug is its idempotency identity: an absent profile is created, an exact profile is skipped, and a conflicting profile is preserved unless the reviewed fragment explicitly sets on_conflict to update. Credential bindings are added idempotently and are never removed merely because a package omits a credential reference.

Preflight does not prove SMTP reachability. After apply, use the normal Mail profile test and Ops health surfaces. If the receipt says SMTP is unavailable, is invalid, or is not mounted, import is blocked with an operator-facing resolution instead of creating a partial profile.

Configuration packages remain SMTP-focused. A POP3 legacy source is a deliberate operational migration action and is not silently exported, cloned, or enabled by an SMTP profile package.

Interface patterns and unavailable actions

Mail uses the platform's shared explorer, connection tree, adaptive form, effective-policy, dialog, status, alert, and confirmation components. The module-owned surface inventory and consequence classification are recorded in Mail interface pattern migration.

An unavailable action remains visible when it belongs to the current task. Its hover/focus explanation identifies the missing field, active operation, or permission. Larger blockers state what must change, who can change it, and where to go. Contextual Help Center links resolve to the configured Docs module when installed and otherwise use the public documentation fallback.

Mailbox reads, profile reloads, connection tests, policy saves, bounce scans, and destructive actions each expose their current loading or blocked state. Removing a bounce watcher and deactivating profile hierarchy items use shared confirmation dialogs. Removing a watcher retains existing observations; deactivating a profile may scrub Mail-owned credentials as described below.

Concepts

Profile

A profile is a reusable, named delivery identity with optional SMTP and IMAP configuration. It can additionally own a dedicated POP3 endpoint for an explicit legacy-import workflow. It has a stable id, lifecycle state, scope, owner context, and non-secret connection metadata. Passwords are write-only encrypted values and are never returned through list/read/capability responses.

An IMAP server may map the standard Inbox, Sent, Drafts, Trash, Archive, and Junk roles to exact provider folder names. These mappings belong to the reusable profile/server. Empty roles retain automatic behavior. The historical imap.sent_folder value is read as the Sent mapping and remains synchronized for compatibility; a Campaign-specific Sent override still wins for that Campaign.

Profiles may be scoped to system, tenant, user, group, or campaign context. Scope controls where a profile can be discovered; effective policy can narrow that further. A visible profile is not automatically authorized for every operation: using, testing, managing, and managing secrets are separate rights.

Policy

Mail policy controls approved profile ids, which lower scopes may define profiles, allowed/denied hosts and addressing patterns, credential inheritance, and lower-level limits. The effective result is assembled from applicable system, tenant, user/group, and campaign context. Denials and locked parent limits cannot be relaxed by a lower scope.

Transport identity and revision

Mail owns opaque random revisions for relevant SMTP/IMAP identity and configuration. They are concurrency tokens, not deterministic hashes that a consumer could use to guess a host or account. Credentials are excluded. A consumer can freeze these revisions at build time and ask Mail to require the same identity when the effect is later performed. Credential rotation alone therefore need not invalidate prepared work, while a host/account/protocol identity change does.

The revision is evidence and a concurrency guard, not a substitute for authorization. Mail re-evaluates profile activity, visibility, policy, and revision immediately before it resolves credentials and performs the effect.

POP3 imports pin the endpoint/credential transport revision at preview time. A changed revision or a missing provider UIDL stops import and requires a fresh preview. POP3 does not participate in the ordinary mailbox folder/message projection.

Provider outcomes

SMTP acceptance, partial or complete recipient refusal, temporary/permanent failure, connection loss, and unknown outcome are distinct. IMAP append is a separate operation and outcome. Ordinary Campaign recipient delivery retains its Campaign-owned job ledger. Mail owns an encrypted durable command and attempt ledger for report delivery and other effects that do not have such a consumer ledger.

The Mail worker commits an attempt before it starts SMTP and then commits an effect-start marker before opening the provider effect. Worker redelivery may recover a stale pre-effect claim, but an accepted, partially accepted, in-progress-after-effect, or unknown command is never sent automatically. Unknown outcomes require a separately authorized reconciliation with an external evidence reference. A deliberate resend creates a new command with a new idempotency key and links it to the prior command.

User tasks

Choose a profile

1. Open the task that requires mail, such as Campaign Mail settings.
2. Choose from the profiles visible and authorized for the current tenant, owner, group, and task context. Never enter or copy a profile id manually when the UI can present a picker.
3. Review the safe summary: name, scope, active state, SMTP/IMAP availability, and policy-relevant sender identity. Credentials and raw provider internals are not visible.
4. Save the reference in the consuming module. If ownership or policy changes, select and validate again.

The user needs mail:profile:use; reading profile summaries and testing may require their own scopes. The consuming module still requires its own action permission.

Test a profile

An authorized profile test verifies connection and authentication for the selected active/visible SMTP or IMAP profile using Mail-owned credentials. The consumer-use path evaluates effective Mail policy separately. Use a non-production provider and mailbox first. A successful connection test does not prove policy authorization for a later Campaign context, deliverability, recipient acceptance, SPF/DKIM/DMARC alignment, or future availability.

Testing a saved profile requires both mail:profile:test and mail:profile:use, and the profile must be active. Profile creation or test authority alone is not enough.

Raw settings test endpoints accept new settings only for actors who may both test profiles and manage secrets. They are an administration aid, not a way for ordinary consumers to bypass reusable profiles.

Read a mailbox

The current mailbox UI and API are read-only. An authorized user can list IMAP folders, page through messages, and inspect a bounded full message. Folder names are parsed and quoted defensively; Sent-folder discovery uses provider flags and common names. The list exposes the provider's Seen flag as a read/unread indicator without changing it. It also labels whether the current page came directly from the provider, from the bounded mailbox index, or from an index while a refresh is in progress, including the index timestamp when available.

Message HTML is displayed only in the shared sandboxed message component. Remote URLs and active markup are removed, embedded data:/cid: image references remain isolated, and plain text is always available when supplied. Attachments and provider/content failures remain explicit rather than being silently interpreted as an empty message.

Mailbox access requires both mail:mailbox:read and mail:profile:use. It must not mutate read/unread, delete, move, or reply state. Message responses are bounded by the deployment response policy; ordinary UI should avoid loading a whole large mailbox or attachment merely to show a list.

Import a legacy POP3 mailbox

POP3 is available only for bounded migration from a legacy server that cannot provide IMAP or JMAP. It is disabled by default and is not a replacement for the read-only mailbox UI.

1. An actor with profile-write, secret-management, and mail:pop3:manage authority opens Legacy POP3 import, selects a Mail profile, and creates a dedicated source. The UI stages the endpoint disabled, stores its encrypted username/password credential, and enables it only after both operations succeed.
2. The administrator explicitly enables legacy import, sets TLS mode, timeout, maximum message and batch sizes, and preview body lines, and tests connection, authentication, TLS, and provider message count. Plain transport remains subject to deployment egress/security policy and should not be used across an untrusted network.
3. An operator with mail:profile:use and mail:pop3:import refreshes a live preview of at most 100 messages. Preview sends no DELE, changes no flags, and exposes only bounded headers/body text. A server without stable UIDL identifiers is rejected. If TOP is unavailable, Mail uses RETR only inside the configured size bound and suppresses an oversized preview.
4. The operator selects messages. Mail downloads within the size gate and creates encrypted local pending_review records. Tenant, profile, endpoint, and UIDL form the duplicate boundary. Raw content never appears in list, provider-state, audit, or DSAR output.
5. Source messages remain untouched by default. Delete-after-import requires the endpoint's separate allow_delete_after_import policy, the operator's mail:pop3:delete permission, an explicit per-batch choice, and destructive confirmation. Mail commits the local import plus mail.pop3.imported audit evidence before sending DELE. It separately records mail.pop3.source_deletion; disconnect during QUIT is outcome-unknown and must be reconciled before another destructive attempt.

The supplied Mail legacy import operator role can test, preview, and import without deleting. The Mail profile administrator role also contains source management and destructive-delete permissions; deployments should remove or split mail:pop3:delete when operators must never delete provider messages. Imported records follow configured Mail/records retention and require manual review for a data-subject request or deletion decision.

Contextual help is available from the page and from each policy-sensitive source setting, credential field, size limit, import action, and destructive confirmation. Press F1 while a control has focus to open the Mail-owned German reference for the exact POP3 context; the same topic remains available through the page help action.

Profile administration

Roles

The supplied templates are:

• Mail profile user: read/use/test approved profiles and read permitted mailboxes without reading secrets.
• Mail profile self-service user: additionally create, edit, deactivate, and manage credentials only for the current account's own user-scoped profiles, subject to the effective Mail policy.
• Mail profile administrator: additionally create/update profiles and create/replace encrypted credentials across tenant-owned scopes, configure legacy POP3 imports, and—unless the template is narrowed—request source deletion after import.
• Mail legacy import operator: test approved POP3 sources and preview/import messages without permission to delete them at the provider.

The specific permissions are mail:profile:read, mail:profile:use, mail:profile:test, mail:mailbox:read, mail:profile:write_own, mail:secret:manage_own, mail:profile:write, mail:secret:manage, mail:pop3:manage, mail:pop3:import, and mail:pop3:delete. The _own permissions are enforced against the authenticated membership id and never authorize a tenant, group, campaign, system, or another user's profile. They also do not authorize profile-policy changes. System-scoped definitions use the corresponding system settings authority. Keep secret management separate when an institution wants profile metadata administrators not to know or replace credentials.

That separation is fail-closed for transport rebinding: changing an SMTP or IMAP host, port, or security mode while the profile retains a stored password requires the matching secret-management permission. A credential-free profile can be deactivated with profile-write authority alone; deactivation that scrubs a stored password also requires secret-management authority and records the deletion in the audit log.

Create or change a profile

The configured Help Center exposes Create a custom Mail profile only when the current actor has broad or self-service profile-write authority and the effective user-scope policy permits user profiles. It states the active SMTP/IMAP hostname allow-list groups and deny rules, plus the actor's separate credential, test, use, and approval requirements. The Settings task creates in the current account's user scope. Grant mail:profile:write_own for self-service; mail:profile:write remains broad profile administration authority.

1. Choose the narrowest suitable scope and a stable, descriptive name/slug.
2. Configure SMTP, optional IMAP, TLS mode, account identity, standard folder mappings, and timeouts. Folder discovery proposes provider-visible Inbox, Sent, Drafts, Trash, Archive, and Junk names without mutating the mailbox. Sender/envelope/recipient constraints belong to effective Mail policy; Campaign rate limits remain delivery configuration.
3. Enter credentials only in the dedicated credential fields. Returned profile data indicates whether credentials are configured without returning them.
4. Save and run SMTP/IMAP tests against a non-production target.
5. Verify the effective policy for every intended consumer context.
6. Communicate changes that alter the non-secret transport identity; prepared consumer snapshots will deliberately stop until rebuilt.

An update that omits a password preserves the current encrypted password. A credential replacement never depends on reading the old cleartext value back.

Delete a profile

Profile deletion is immediate for Mail-owned secrets and audit evidence:

1. Mail clears both encrypted SMTP and IMAP passwords in the same transaction.
2. It deactivates the remaining non-secret tombstone state so historical stable references can fail safely rather than resolve to another profile.
3. When owned encrypted secrets existed, it emits mail.profile_credentials_deleted with profile id, scope, reason, and the kinds of secret removed—never values, usernames/passwords, or raw configuration.
4. If scrubbing or audit insertion fails, the transaction rolls back and the deletion fails.
5. Repeating deletion after the secrets are gone is an idempotent no-op for the secret-deletion effect.

Consumers that still reference the profile fail closed at validation or execution. Deletion does not silently rebind them to a default profile.

Policy hierarchy

Effective policy is contextual. Administrators should document:

• which profile ids are approved globally or for a tenant;
• whether tenant, user, group, or campaign-scoped profiles may be created;
• allowed and denied SMTP/IMAP hosts;
• permitted From, envelope sender (including bounce address), and envelope recipient-domain patterns;
• whether SMTP/IMAP credentials inherit from the reusable profile; and
• which lower-level settings are locked by a parent policy.

Campaign delivery requires reusable profile credentials. A legacy policy that requires campaign-local credentials fails closed with guidance to store them on the Mail profile and enable effective inheritance. This preserves compatibility of the policy model without reopening a consumer-owned secret store.

Policy reads are available through system/tenant/context routes to suitably authorized actors. Adaptive Docs exposes a safe explanation of the effective tenant posture; it does not expose credential material.

Operations and recovery

Deployment egress policy

Private-network connector access is controlled deployment-wide by GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS. Whether private or public targets are allowed, SMTP, IMAP, and legacy POP3 resolve, validate, and connect to the exact approved address records at connection time while retaining the original hostname for TLS SNI and certificate verification. A DNS change cannot redirect the socket after validation.

Transports that cannot pin every connection peer or revalidate protocol-managed redirects/referrals must fail before client construction. Do not weaken this rule to make a target test pass; change the connector implementation or the target topology.

Response and time bounds

The shared defaults are 16 MiB for structured connector responses and 512 MiB for file transfers. Mailbox full-message fetch additionally uses bounded IMAP ranges and the applicable deployment limit. Operators may lower limits for their environment. A declared or streamed over-limit response fails visibly rather than being retained partially as if complete.

Configure connection/read timeouts and provider-specific message/attachment limits. A timeout after the SMTP effect may be an unknown outcome; it must not be flattened into a safe-to-retry connection failure.

Throttling

Mail rate limiting uses a Redis lock and next-allowed timestamp across worker processes when worker mode is enabled. Direct development uses a process-local limiter. If Redis fails in worker mode, the current implementation silently falls back to a limiter that protects only that process; it does not provide cluster-wide coordination or emit its own health signal. Production operators must therefore monitor Redis independently and should stop or restrict multi-worker delivery during an outage until explicit degradation telemetry is implemented.

SMTP/IMAP incidents

Current SMTP delivery and Sent-folder APPEND paths start a Mail-owned Core recovery operation before the network effect. Outbox attempts use the durable command and attempt number; Campaign jobs and single-message actions pass their own stable attempt identifiers through the versioned capability. A matching completed identifier is never sent or appended again merely to reconstruct caller state. Provider acceptance, definitive rejection, and outcome-unknown states are recorded independently of the consuming transaction.

Mailbox folder/message indexing and configured bounce/calendar-reply scans are read-only provider operations. They acquire distributed per-profile/folder or per-source fences, commit bounded projection state, and verify that state in an independent session. A failed read is rolled back and can be repeated; it is not treated as an unknown provider mutation.

1. Stop new consumer work if duplicate effects or credential compromise are possible.
2. Preserve safe Mail, consumer-job, worker, and provider evidence.
3. Rotate credentials in Mail. Rebuild consumers only if the revisioned non-secret identity also changed.
4. Treat post-command connection loss as potentially unknown until provider evidence establishes whether SMTP accepted the message.
5. Handle IMAP append failure independently; do not resend an accepted message merely to recreate a Sent copy.
6. Record the incident/reconciliation reference in the consuming domain's audit trail without copying raw provider secrets or message content unnecessarily.
7. For an outbox outcome_unknown, reconcile the Mail command with provider evidence. Confirmed acceptance closes the provider operation as succeeded; confirmed absence records verified recovery and permits only a new, deliberate attempt identifier.

Delivery-status and calendar-reply sources

An authorized Mail bounce source scans a bounded IMAP UID range without changing mailbox flags. It correlates DSN reports with durable Mail commands and, when the optional Calendar invitation capability is active, forwards text/calendar or .ics METHOD:REPLY parts to Calendar. Calendar remains owner of attendee state; Mail records only a raw digest, mailbox coordinates, Message-ID, and audit linkage. Ordinary or malformed calendar messages do not block DSN progress. A repeated UID/message digest produces no second Calendar state transition or outbound synchronization effect.

Backup, restore, and retirement

Backups contain encrypted credentials and encrypted raw POP3 import records and therefore need the same protection as the live database and key material. Restoring a Mail database without the matching encryption key makes those records unusable; restoring it with keys can reactivate sensitive historical state and must be controlled.

Destructive module retirement first applies the same immediate credential scrub/audit rule to every remaining profile, then drops Mail-owned tables after the installer snapshot/confirmation gate. Any scrub or audit failure blocks retirement. Historical backups are separate retained copies and require the deployment's approved backup-destruction policy.

Capability contract

Mail provides mail.campaign_delivery version 0.2.x. This version is a breaking ownership boundary: it exposes reference-oriented operations, not raw resolved configuration or credentials.

The Mail REST API supplies the profile list used by pickers. Given a stable profile reference, the capability supports these kinds of operation:

• resolve a profile reference and evaluate its Mail context/policy to produce a non-secret delivery summary and opaque Mail-owned revisions;
• send Campaign-owned RFC message bytes using the selected profile;
• optionally append an accepted message to Sent; and
• return sanitized outcome/revision evidence.

For a real effect, one Mail call must:

1. resolve current campaign/tenant/owner context through the optional narrow context capability;
2. load the selected active profile;
3. evaluate profile-scope eligibility and effective policy;
4. compare the consumer's expected opaque transport revision;
5. decrypt credentials in Mail memory;
6. perform the SMTP or IMAP operation; and
7. return only the safe result.

Keeping comparison, credential resolution, and transport inside one operation avoids a time-of-check/time-of-use gap and prevents the consumer from becoming a credential-processing boundary. The capability does not receive the acting principal: the Campaign route must first enforce mail:profile:use, while Mail still re-evaluates active profile scope and contextual policy inside the call.

Mail optionally consumes campaigns.mail_policy_context and addresses.lookup. Their physical absence must leave Mail installable and its unrelated profile/mailbox behavior usable. Interface-version compatibility is a mandatory release gate.

Security, deletion, and audit

Security invariants:

• Decrypted SMTP/IMAP passwords never cross the Mail capability/API boundary.
• Password fields are write-only and encrypted at rest; safe responses expose configuration state, not values.
• Consumers persist stable profile references, not transport copies.
• Host policy is deployment-wide and every supported connection is DNS/IP pinned at socket creation.
• Redirect/referral-capable transports fail closed unless every peer can be revalidated and pinned.
• Profile deletion and destructive retirement scrub every Mail-owned encrypted password and emit non-secret audit in the same lifecycle action.
• Mailbox reads are separately authorized, read-only, paginated, and bounded.
• Logs/audit/results minimize message content, addresses, provider responses, and secret-like values.

Credential replacement/deletion emits canonical non-secret Mail audit events. Profile/policy create/update/deactivate currently emits change-feed evidence, but connection tests and the complete administrative lifecycle do not yet have equivalent canonical audit events. Mail delivery commands emit request, terminal-outcome, reconciliation, and deliberate-resend events without addresses or raw provider responses. MIME, envelope addresses, detailed refusals, and reconciliation notes are encrypted and minimized after their retention deadline while hashes, counts, attempts, and decisions remain.

Acceptance checklist

Before claiming a Mail composition is production-ready:

1. Install Mail alone with Core and with each optional consumer absent.
2. Prove profile scope, visibility, use, test, write, and secret permissions for representative tenant/user/group contexts.
3. Prove policy deny precedence, locked lower limits, approved-profile mode, and the campaign-local-credential fail-closed path.
4. Verify all read/list/capability/audit/log responses contain no plaintext or encrypted password, raw credential token, or consumer-owned inline config.
5. Test DNS rebinding resistance and private/public deployment policy for SMTP and IMAP.
6. Test SMTP acceptance, refusal, temporary/permanent failure, timeout/unknown outcome, and IMAP append success/failure against the target provider.
7. Test Redis worker throttling and prove that independent infrastructure monitoring detects Redis loss; restrict multi-worker delivery during the current silent local-only fallback.
8. Delete tenant and system profiles, inject audit failure, repeat deletion, and prove transactional scrub/rollback/idempotency.
9. Run destructive-retirement preflight on a snapshot and prove credentials are scrubbed/audited before table drop.
10. Pass module permutations, interface/version alignment, WebUI/i18n, full security audit, backup/restore, and target-environment release checks.

Explicitly planned, not yet claimed

• Canonical audit events for profile tests and the remaining profile/policy administration lifecycle, plus an operator-visible Redis-throttling degradation signal.
• Reusable SMTP batch sessions and their final connection/error/isolation semantics (govoplan-mail#16).
• Final Campaign test / single send / single resend semantics; those are a Campaign business-action contract built on Mail transport operations.
• JMAP mailbox synchronization/search; it is preferred only after the IMAP MVP is stable.
• Expanding POP3 beyond the implemented explicit legacy download/import workflow; it has no folder, flag, search, or synchronization contract.
• A full mail client with compose/reply/move/delete/read-state mutation.
• Quick Access may launch the operating environment's configured composer via mailto:. That explicit handoff is not a GovOPlaN Mail delivery: it selects no Mail profile or credential, bypasses no Mail policy, and reports no GovOPlaN delivery result. Recent-message and Drafts links remain read-only deep links into the authorized Mail profile and preserve their Quick Access return context.
• Recovery-ledger adoption for future provider-side move, delete, and flag mutations; no such production path exists in the current read-only mailbox.
• Proof that process-local throttling coordinates multiple workers when Redis is unavailable; it deliberately does not.