govoplan-mail/docs/MAIL_HANDBOOK.md

# Mail Handbook

## Purpose and status

This handbook is the canonical multi-perspective description of the GovOPlaN
Mail module. Mail is a governed transport and mailbox capability. It is not a
campaign definition, contact database, or general records store.

| Perspective | Start here |
| --- | --- |
| User selecting a profile or reading a mailbox | [User tasks](#user-tasks) |
| Mail profile administrator | [Profile administration](#profile-administration) |
| Policy or tenant administrator | [Policy hierarchy](#policy-hierarchy) |
| Delivery operator | [Operations and recovery](#operations-and-recovery) |
| Integrator | [Capability contract](#capability-contract) |
| Security or audit reviewer | [Security, deletion, and audit](#security-deletion-and-audit) |
| Release reviewer | [Acceptance checklist](#acceptance-checklist) |

See also [Mail protocol roadmap](MAIL_PROTOCOL_ROADMAP.md) and the Campaign
[Mail profile boundary](https://git.add-ideas.de/add-ideas/govoplan-campaign/src/branch/main/docs/MAIL_PROFILE_BOUNDARY.md).

## Domain ownership

Mail owns:

- reusable SMTP/IMAP profile definitions and scope;
- encrypted SMTP/IMAP credentials and safe credential replacement;
- effective profile policy and visibility/authorization decisions;
- connection tests and protocol adapters;
- SMTP send and IMAP append operations exposed to consumers;
- read-only mailbox folder/message access and its bounded indexes; and
- the transport sanitization boundary and throttling behavior. A general
  Mail-owned provider-attempt/diagnostic ledger remains planned.

Consumers own their business intent. For example, Campaign owns message
content, recipients, approval, jobs, and delivery evidence, while Mail owns the
profile and performs the provider operation. A consumer stores only a stable
Mail profile reference and non-secret evidence returned by Mail. It never
receives decrypted credentials or a raw resolved transport configuration.

Core owns authentication, tenant context, permission evaluation, database
session primitives, cryptographic secret helpers, audit infrastructure, and the
module registry. Optional consumers provide narrow context through capabilities;
Mail does not import their ORM or service implementations.

## Concepts

### Profile

A profile is a reusable, named delivery identity with optional SMTP and IMAP
configuration. It has a stable id, lifecycle state, scope, owner context, and
non-secret connection metadata. Passwords are write-only encrypted values and
are never returned through list/read/capability responses.

Profiles may be scoped to system, tenant, user, group, or campaign context.
Scope controls where a profile can be discovered; effective policy can narrow
that further. A visible profile is not automatically authorized for every
operation: using, testing, managing, and managing secrets are separate rights.

### Policy

Mail policy controls approved profile ids, which lower scopes may define
profiles, allowed/denied hosts and addressing patterns, credential inheritance,
and lower-level limits. The effective result is assembled from applicable
system, tenant, user/group, and campaign context. Denials and locked parent
limits cannot be relaxed by a lower scope.

### Transport identity and revision

Mail owns opaque random revisions for relevant SMTP/IMAP identity and
configuration. They are concurrency tokens, not deterministic hashes that a
consumer could use to guess a host or account. Credentials are excluded. A
consumer can freeze these revisions at build time and ask Mail to require the
same identity when the effect is later performed. Credential rotation alone
therefore need not invalidate prepared work, while a host/account/protocol
identity change does.

The revision is evidence and a concurrency guard, not a substitute for
authorization. Mail re-evaluates profile activity, visibility, policy, and
revision immediately before it resolves credentials and performs the
effect.

### Provider outcomes

SMTP acceptance, recipient refusal, temporary/permanent failure, connection
loss, and unknown outcome are distinct. IMAP append is a separate operation and
outcome. Mail returns a sanitized transport result; the consuming domain owns
the durable business job and its retry/reconciliation semantics.

## User tasks

### Choose a profile

1. Open the task that requires mail, such as Campaign **Mail settings**.
2. Choose from the profiles visible and authorized for the current tenant,
   owner, group, and task context. Never enter or copy a profile id manually
   when the UI can present a picker.
3. Review the safe summary: name, scope, active state, SMTP/IMAP availability,
   and policy-relevant sender identity. Credentials and raw provider internals
   are not visible.
4. Save the reference in the consuming module. If ownership or policy changes,
   select and validate again.

The user needs `mail:profile:use`; reading profile summaries and testing may
require their own scopes. The consuming module still requires its own action
permission.

### Test a profile

An authorized profile test verifies connection and authentication for the
selected active/visible SMTP or IMAP profile using Mail-owned credentials. The
consumer-use path evaluates effective Mail policy separately. Use a
non-production provider and mailbox first. A successful connection test does
not prove policy authorization for a later Campaign context, deliverability,
recipient acceptance, SPF/DKIM/DMARC alignment, or future availability.

Testing a saved profile requires both `mail:profile:test` and
`mail:profile:use`, and the profile must be active. Profile creation or test
authority alone is not enough.

Raw settings test endpoints accept new settings only for actors who may both
test profiles and manage secrets. They are an administration aid, not a way for
ordinary consumers to bypass reusable profiles.

### Read a mailbox

The current mailbox UI and API are read-only. An authorized user can list IMAP
folders, page through messages, and inspect a bounded full message. Folder
names are parsed and quoted defensively; Sent-folder discovery uses provider
flags and common names.

Mailbox access requires both `mail:mailbox:read` and `mail:profile:use`. It must
not mutate read/unread, delete, move, or reply state. Message responses are
bounded by the deployment response policy; ordinary UI should avoid loading a
whole large mailbox or attachment merely to show a list.

## Profile administration

### Roles

The supplied templates are:

- **Mail profile user:** read/use/test approved profiles and read permitted
  mailboxes without reading secrets.
- **Mail profile self-service user:** additionally create, edit, deactivate,
  and manage credentials only for the current account's own user-scoped
  profiles, subject to the effective Mail policy.
- **Mail profile administrator:** additionally create/update profiles and
  create/replace encrypted credentials across tenant-owned scopes.

The specific permissions are `mail:profile:read`, `mail:profile:use`,
`mail:profile:test`, `mail:mailbox:read`, `mail:profile:write_own`,
`mail:secret:manage_own`, `mail:profile:write`, and `mail:secret:manage`.
The `_own` permissions are enforced against the authenticated membership id and
never authorize a tenant, group, campaign, system, or another user's profile.
They also do not authorize profile-policy changes. System-scoped definitions
use the corresponding system settings authority. Keep secret management
separate when an institution wants profile metadata administrators not to know
or replace credentials.

That separation is fail-closed for transport rebinding: changing an SMTP or
IMAP host, port, or security mode while the profile retains a stored password
requires the matching secret-management permission. A credential-free profile
can be deactivated with profile-write authority alone; deactivation that
scrubs a stored password also requires secret-management authority and records
the deletion in the audit log.

### Create or change a profile

The configured Help Center exposes **Create a custom Mail profile** only when
the current actor has broad or self-service profile-write authority and the
effective user-scope policy permits user profiles. It states the active SMTP/IMAP hostname
allow-list groups and deny rules, plus the actor's separate credential, test,
use, and approval requirements. The Settings task creates in the current
account's user scope. Grant `mail:profile:write_own` for self-service;
`mail:profile:write` remains broad profile administration authority.

1. Choose the narrowest suitable scope and a stable, descriptive name/slug.
2. Configure SMTP, optional IMAP, TLS mode, account identity, Sent-folder
   behavior, and timeouts. Sender/envelope/recipient constraints belong to
   effective Mail policy; Campaign rate limits remain delivery configuration.
3. Enter credentials only in the dedicated credential fields. Returned profile
   data indicates whether credentials are configured without returning them.
4. Save and run SMTP/IMAP tests against a non-production target.
5. Verify the effective policy for every intended consumer context.
6. Communicate changes that alter the non-secret transport identity; prepared
   consumer snapshots will deliberately stop until rebuilt.

An update that omits a password preserves the current encrypted password. A
credential replacement never depends on reading the old cleartext value back.

### Delete a profile

Profile deletion is immediate for Mail-owned secrets and audit evidence:

1. Mail clears both encrypted SMTP and IMAP passwords in the same transaction.
2. It deactivates the remaining non-secret tombstone state so historical stable
   references can fail safely rather than resolve to another profile.
3. When owned encrypted secrets existed, it emits
   `mail.profile_credentials_deleted` with profile id, scope, reason, and the
   kinds of secret removed—never values, usernames/passwords, or raw
   configuration.
4. If scrubbing or audit insertion fails, the transaction rolls back and the
   deletion fails.
5. Repeating deletion after the secrets are gone is an idempotent no-op for the
   secret-deletion effect.

Consumers that still reference the profile fail closed at validation or
execution. Deletion does not silently rebind them to a default profile.

## Policy hierarchy

Effective policy is contextual. Administrators should document:

- which profile ids are approved globally or for a tenant;
- whether tenant, user, group, or campaign-scoped profiles may be created;
- allowed and denied SMTP/IMAP hosts;
- permitted From, envelope sender (including bounce address), and envelope
  recipient-domain patterns;
- whether SMTP/IMAP credentials inherit from the reusable profile; and
- which lower-level settings are locked by a parent policy.

Campaign delivery requires reusable profile credentials. A legacy policy that
requires campaign-local credentials fails closed with guidance to store them on
the Mail profile and enable effective inheritance. This preserves compatibility
of the policy model without reopening a consumer-owned secret store.

Policy reads are available through system/tenant/context routes to suitably
authorized actors. Adaptive Docs exposes a safe explanation of the effective
tenant posture; it does not expose credential material.

## Operations and recovery

### Deployment egress policy

Private-network connector access is controlled deployment-wide by
`GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS`. Whether private or public targets
are allowed, SMTP and IMAP resolve, validate, and connect to the exact approved
address records at connection time while retaining the original hostname for
TLS SNI and certificate verification. A DNS change cannot redirect the socket
after validation.

Transports that cannot pin every connection peer or revalidate protocol-managed
redirects/referrals must fail before client construction. Do not weaken this
rule to make a target test pass; change the connector implementation or the
target topology.

### Response and time bounds

The shared defaults are 16 MiB for structured connector responses and 512 MiB
for file transfers. Mailbox full-message fetch additionally uses bounded IMAP
ranges and the applicable deployment limit. Operators may lower limits for
their environment. A declared or streamed over-limit response fails visibly
rather than being retained partially as if complete.

Configure connection/read timeouts and provider-specific message/attachment
limits. A timeout after the SMTP effect may be an unknown outcome; it must not
be flattened into a safe-to-retry connection failure.

### Throttling

Mail rate limiting uses a Redis lock and next-allowed timestamp across worker
processes when worker mode is enabled. Direct development uses a process-local
limiter. If Redis fails in worker mode, the current implementation silently
falls back to a limiter that protects only that process; it does not provide
cluster-wide coordination or emit its own health signal. Production operators
must therefore monitor Redis independently and should stop or restrict
multi-worker delivery during an outage until explicit degradation telemetry is
implemented.

### SMTP/IMAP incidents

1. Stop new consumer work if duplicate effects or credential compromise are
   possible.
2. Preserve safe Mail, consumer-job, worker, and provider evidence.
3. Rotate credentials in Mail. Rebuild consumers only if the revisioned
   non-secret identity also changed.
4. Treat post-command connection loss as potentially unknown until provider
   evidence establishes whether SMTP accepted the message.
5. Handle IMAP append failure independently; do not resend an accepted message
   merely to recreate a Sent copy.
6. Record the incident/reconciliation reference in the consuming domain's audit
   trail without copying raw provider secrets or message content unnecessarily.

### Backup, restore, and retirement

Backups contain encrypted credentials and therefore need the same protection as
the live database and key material. Restoring a Mail database without the
matching encryption key makes credentials unusable; restoring it with keys can
reactivate sensitive historical state and must be controlled.

Destructive module retirement first applies the same immediate credential
scrub/audit rule to every remaining profile, then drops Mail-owned tables after
the installer snapshot/confirmation gate. Any scrub or audit failure blocks
retirement. Historical backups are separate retained copies and require the
deployment's approved backup-destruction policy.

## Capability contract

Mail provides `mail.campaign_delivery` version `0.2.x`. This version is a
breaking ownership boundary: it exposes reference-oriented operations, not raw
resolved configuration or credentials.

The Mail REST API supplies the profile list used by pickers. Given a stable
profile reference, the capability supports these kinds of operation:

- resolve a profile reference and evaluate its Mail context/policy to produce a
  non-secret delivery summary and
  opaque Mail-owned revisions;
- send Campaign-owned RFC message bytes using the selected profile;
- optionally append an accepted message to Sent; and
- return sanitized outcome/revision evidence.

For a real effect, one Mail call must:

1. resolve current campaign/tenant/owner context through the optional narrow
   context capability;
2. load the selected active profile;
3. evaluate profile-scope eligibility and effective policy;
4. compare the consumer's expected opaque transport revision;
5. decrypt credentials in Mail memory;
6. perform the SMTP or IMAP operation; and
7. return only the safe result.

Keeping comparison, credential resolution, and transport inside one operation
avoids a time-of-check/time-of-use gap and prevents the consumer from becoming
a credential-processing boundary. The capability does not receive the acting
principal: the Campaign route must first enforce `mail:profile:use`, while Mail
still re-evaluates active profile scope and contextual policy inside the call.

Mail optionally consumes `campaigns.mail_policy_context` and
`addresses.lookup`. Their physical absence must leave Mail installable and its
unrelated profile/mailbox behavior usable. Interface-version compatibility is a
mandatory release gate.

## Security, deletion, and audit

Security invariants:

- Decrypted SMTP/IMAP passwords never cross the Mail capability/API boundary.
- Password fields are write-only and encrypted at rest; safe responses expose
  configuration state, not values.
- Consumers persist stable profile references, not transport copies.
- Host policy is deployment-wide and every supported connection is DNS/IP
  pinned at socket creation.
- Redirect/referral-capable transports fail closed unless every peer can be
  revalidated and pinned.
- Profile deletion and destructive retirement scrub every Mail-owned encrypted
  password and emit non-secret audit in the same lifecycle action.
- Mailbox reads are separately authorized, read-only, paginated, and bounded.
- Logs/audit/results minimize message content, addresses, provider responses,
  and secret-like values.

Credential replacement/deletion emits canonical non-secret Mail audit events.
Profile/policy create/update/deactivate currently emits change-feed evidence,
but connection tests and the complete administrative lifecycle do not yet have
equivalent canonical audit events. Consumer send/retry/reconciliation evidence
belongs primarily to the consuming module today. A Mail-owned restricted
provider-attempt ledger with correlation, retention, and unknown-outcome
reconciliation remains part of the durable outbox work in
[`govoplan-mail#17`](https://git.add-ideas.de/add-ideas/govoplan-mail/issues/17).

## Acceptance checklist

Before claiming a Mail composition is production-ready:

1. Install Mail alone with Core and with each optional consumer absent.
2. Prove profile scope, visibility, use, test, write, and secret permissions for
   representative tenant/user/group contexts.
3. Prove policy deny precedence, locked lower limits, approved-profile mode, and
   the campaign-local-credential fail-closed path.
4. Verify all read/list/capability/audit/log responses contain no plaintext or
   encrypted password, raw credential token, or consumer-owned inline config.
5. Test DNS rebinding resistance and private/public deployment policy for SMTP
   and IMAP.
6. Test SMTP acceptance, refusal, temporary/permanent failure, timeout/unknown
   outcome, and IMAP append success/failure against the target provider.
7. Test Redis worker throttling and prove that independent infrastructure
   monitoring detects Redis loss; restrict multi-worker delivery during the
   current silent local-only fallback.
8. Delete tenant and system profiles, inject audit failure, repeat deletion, and
   prove transactional scrub/rollback/idempotency.
9. Run destructive-retirement preflight on a snapshot and prove credentials are
   scrubbed/audited before table drop.
10. Pass module permutations, interface/version alignment, WebUI/i18n, full
    security audit, backup/restore, and target-environment release checks.

## Explicitly planned, not yet claimed

- Durable, idempotent Campaign report delivery with Mail-owned attempts,
  unknown-outcome reconciliation, and partial-refusal evidence
  ([`govoplan-mail#17`](https://git.add-ideas.de/add-ideas/govoplan-mail/issues/17)).
- Canonical audit events for profile tests and the remaining profile/policy
  administration lifecycle, plus an operator-visible Redis-throttling
  degradation signal.
- Reusable SMTP batch sessions and their final connection/error/isolation
  semantics (`govoplan-mail#16`).
- Final Campaign **test / single send / single resend** semantics; those are a
  Campaign business-action contract built on Mail transport operations.
- JMAP mailbox synchronization/search; it is preferred only after the IMAP MVP
  is stable.
- POP3 except for a future explicit legacy download/import requirement.
- A full mail client with compose/reply/move/delete/read-state mutation.
- Proof that process-local throttling coordinates multiple workers when Redis
  is unavailable; it deliberately does not.