multi-seal-mail/README.md

# MultiMailer skeleton

Local-first bulk mailer backend skeleton:

- FastAPI API
- Celery worker(s)
- Redis queue
- Postgres database
- Garage S3-compatible object storage
- Optional local Traefik
- Modular Compose layout so you can reuse existing infrastructure

## Quick start: full local development stack

```bash
cp .env.example .env
./scripts/up-local.sh
```

Equivalent explicit command:

```bash
docker compose -f compose.yml -f compose.local.yml up --build
```

Health check:

```bash
curl http://localhost:8080/health
# or direct API port:
curl http://localhost:8000/health
```

Garage S3 local endpoint:

```text
http://localhost:3900
```

## App-only mode with existing Postgres, Redis, Garage, Traefik

1. Copy `.env.example` to `.env`
2. Point these values to your existing services:

```dotenv
DATABASE_URL=postgresql+psycopg://user:password@existing-postgres:5432/multimailer
REDIS_URL=redis://existing-redis:6379/0
S3_ENDPOINT_URL=https://garage.example.org
S3_REGION=garage
S3_BUCKET=attachments
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...
```

3. Start app services only:

```bash
docker compose -f compose.yml up --build
```

Or use the example external Traefik override:

```bash
docker compose -f compose.yml -f compose.external.example.yml up --build
```

## Scale within one worker container

For small PDFs zipped per recipient, start with long-lived Celery worker processes:

```dotenv
CELERY_CONCURRENCY=4
CELERY_PREFETCH_MULTIPLIER=1
CELERY_MAX_TASKS_PER_CHILD=200
```

## Scale with more containers

```bash
docker compose -f compose.yml -f compose.local.yml up --build --scale worker=3 --scale api=2
```

## Notes

- Set `MASTER_KEY_B64` before storing real SMTP/IMAP credentials.
- Garage 2.3+ can create the local default bucket and key from environment variables when started with `--single-node --default-bucket`.
- The local Garage config uses `replication_factor = 1`; this is fine for development, not a production storage strategy.

## Campaign JSON

The project now includes a first schema and example campaign file:

```text
server/app/mailer/schemas/campaign.schema.json
server/app/mailer/examples/campaign.json
```

The schema preserves the Java object model concepts: global recipients, individual recipients, combine flags, campaign-level `attachments.base_path`, global and entry-specific attachment configs, validation behavior for missing/ambiguous attachments, rate limits, and optional IMAP append-to-Sent handling.

## Attachment resolver dry run

The attachment resolver loads the campaign JSON, normalizes inline or external entries, renders `${global::...}` and `${local::...}` placeholders in `base_dir` and `file_filter`, and reports missing or ambiguous attachment matches without sending mail.

```bash
cd server
python -m app.mailer.commands.resolve_attachments \
  --campaign app/mailer/examples/campaign.json \
  --verbose
```

Machine-readable output for the future web UI:

```bash
python -m app.mailer.commands.resolve_attachments \
  --campaign app/mailer/examples/campaign.json \
  --json
```

## Build message drafts without sending

After validating the campaign and resolving attachments, build reviewable message drafts:

```bash
cd server
python -m app.mailer.commands.build_messages \
  --campaign app/mailer/examples/campaign.json \
  --output-dir ./build/messages \
  --verbose
```

This writes `.eml` files for review and prints per-entry status:

- `ready`: queueable
- `warning`: queueable, but with non-blocking issues
- `needs_review`: user decision required before queueing
- `blocked`: must not be queued
- `excluded`: excluded by policy, but still reported
- `inactive`: inactive entry

Machine-readable report:

```bash
python -m app.mailer.commands.build_messages \
  --campaign app/mailer/examples/campaign.json \
  --json
```

## SMTP test send

After building messages, you can send one generated message to a test recipient without changing campaign/job status:

```bash
cd server
python -m app.mailer.commands.send_test_message \
  --campaign app/mailer/examples/campaign.json \
  --to your.test.address@example.org \
  --dry-run \
  --write-eml ./build/test-message.eml
```

When ready to actually connect to SMTP, remove `--dry-run`. SMTP settings are read from `campaign.json`, but can be overridden without editing the file:

```bash
MULTIMAILER_SMTP_PASSWORD='...' \
python -m app.mailer.commands.send_test_message \
  --campaign app/mailer/examples/campaign.json \
  --to your.test.address@example.org \
  --smtp-host smtp.example.org \
  --smtp-port 587 \
  --smtp-security starttls \
  --smtp-username user@example.org
```

Supported override environment variables:

```text
MULTIMAILER_SMTP_HOST
MULTIMAILER_SMTP_PORT
MULTIMAILER_SMTP_SECURITY  # plain | tls | starttls
MULTIMAILER_SMTP_USERNAME
MULTIMAILER_SMTP_PASSWORD
```

The test-send command always replaces the real To/Cc/Bcc recipients with the test recipient before sending.

## Persistence/API-key milestone

This version adds the first persistent backend layer:

- SQLAlchemy models for tenants, users, roles, API keys, campaigns, campaign versions, jobs, issues, attachments, send attempts, IMAP append attempts and audit log entries.
- Alembic setup under `server/alembic`.
- Development database bootstrap command.
- API-key authentication for campaign API endpoints.
- Campaign JSON import into versioned DB snapshots.
- Campaign validation/build results persisted as campaign jobs.

### Local DB quick start without Docker

From `server/`:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# SQLite is the default when DATABASE_URL is not set.
python -m app.mailer.commands.init_db --with-dev-data

python -m app.mailer.commands.import_campaign \
  --campaign app/mailer/examples/campaign.json \
  --api-key dev-multimailer-api-key \
  --validate \
  --build

python -m app.mailer.commands.list_db_campaigns \
  --api-key dev-multimailer-api-key
```

### API quick smoke test

Start the API:

```bash
uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
```

The dev bootstrap runs automatically in `APP_ENV=dev` and creates this API key unless disabled:

```text
dev-multimailer-api-key
```

List campaigns:

```bash
curl -H 'X-API-Key: dev-multimailer-api-key' \
  http://127.0.0.1:8000/api/v1/campaigns
```

Create a campaign from JSON:

```bash
python - <<'PY'
import json, requests
config = json.load(open('app/mailer/examples/campaign.json'))
response = requests.post(
    'http://127.0.0.1:8000/api/v1/campaigns',
    headers={'X-API-Key': 'dev-multimailer-api-key'},
    json={'config': config, 'source_filename': 'app/mailer/examples/campaign.json'},
)
print(response.status_code)
print(response.text)
PY
```

### API endpoints added

```text
GET  /api/v1/campaigns
POST /api/v1/campaigns
GET  /api/v1/campaigns/{campaign_id}
GET  /api/v1/campaigns/{campaign_id}/versions
POST /api/v1/campaigns/versions/{version_id}/validate
POST /api/v1/campaigns/versions/{version_id}/build
GET  /api/v1/campaigns/{campaign_id}/jobs
POST /api/v1/admin/api-keys
```

All campaign/admin endpoints require an API key via either:

```text
X-API-Key: ...
Authorization: Bearer ...
```

### Note about local file paths

When importing a file-based campaign JSON, pass `source_filename` or use the CLI importer. Relative paths inside the campaign are normalized to absolute paths at import time so the DB snapshot can still find CSV files, templates and attachment folders later.

## Queueing and sending step

This version adds the first real bulk-delivery transition:

```text
built campaign jobs -> queued jobs -> SMTP worker/direct sender -> send attempts
```

### Queue jobs without starting Celery

```bash
cd server
python -m app.mailer.commands.queue_campaign \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --no-celery
```

Use `--dry-run` to see what would be queued without modifying the database.

### Process queued jobs directly from CLI

This is useful before the web UI and before running a Celery worker:

```bash
python -m app.mailer.commands.send_queued_jobs \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run
```

Remove `--dry-run` only after SMTP credentials in the campaign JSON point to a real server.

### Queue jobs and send through Celery

```bash
python -m app.mailer.commands.queue_campaign \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key

celery -A app.celery_app.celery worker \
  --loglevel=INFO \
  --queues=send_email,append_sent,default \
  --concurrency=1 \
  --prefetch-multiplier=1
```

### API endpoints added

```text
POST /api/v1/campaigns/{campaign_id}/queue
POST /api/v1/campaigns/{campaign_id}/pause
POST /api/v1/campaigns/{campaign_id}/resume
POST /api/v1/campaigns/{campaign_id}/cancel
```

Queueing still respects the review gate: only `ready` and, by default, `warning` jobs are queued. `needs_review`, `blocked`, `excluded`, and `inactive` jobs are not queued.

### Rate limiting

SMTP sending now applies campaign-level rate limiting from the campaign JSON:

```json
"delivery": {
  "rate_limit": {
    "messages_per_minute": 5,
    "concurrency": 1
  }
}
```

The worker uses Redis for distributed throttling when available. If Redis is unavailable in local development, the sender falls back gracefully, but production should use Redis.

### IMAP status

SMTP success now keeps IMAP state separate:

```text
send_status = sent
imap_status = pending | not_requested
```

Actual IMAP APPEND is still the next implementation step.

## IMAP append-to-Sent

SMTP sending and IMAP Sent-folder append are intentionally separate states. A job can be `send_status=sent` while `imap_status=failed`; the email was still sent, but saving the copy to Sent needs attention.

When Celery is used, successful SMTP delivery automatically enqueues an `append_sent` task if the campaign has:

```json
"delivery": {
  "imap_append_sent": { "enabled": true, "folder": "auto" }
}
```

and `server.imap.enabled=true`.

Direct CLI sending can append immediately in the same process:

```bash
python -m app.mailer.commands.send_queued_jobs \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --append-sent
```

Or process pending/failed appends separately:

```bash
python -m app.mailer.commands.append_pending_sent \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key
```

Dry-run IMAP append check:

```bash
python -m app.mailer.commands.append_pending_sent \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run
```

API endpoint:

```text
POST /api/v1/campaigns/{campaign_id}/append-sent
```

with body:

```json
{
  "enqueue_celery": true,
  "dry_run": false
}
```

Folder resolution order:

1. `delivery.imap_append_sent.folder` if not `auto`
2. `server.imap.sent_folder` if not `auto`
3. IMAP `LIST` discovery using `\\Sent` special-use flags
4. common fallback names such as `Sent`, `Sent Items`, `Gesendet`

## Campaign reporting / dashboard data

This version adds dashboard/report payloads that are intended to feed the first web interface.
They summarize persisted jobs, validation issues, attachment resolution, sending attempts, IMAP append state, and rate-limit estimates.

CLI report:

```bash
cd server
python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key
```

Machine-readable report:

```bash
python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --json \
  --include-jobs
```

Per-job CSV export:

```bash
python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --jobs-csv ./build/campaign-jobs.csv
```

API endpoints:

```text
GET /api/v1/campaigns/{campaign_id}/summary
GET /api/v1/campaigns/{campaign_id}/report
GET /api/v1/campaigns/{campaign_id}/report/jobs.csv
```

These endpoints are API-key protected with the `campaign:read` scope and return data shaped for a web dashboard: top-level cards, status counters, attachment summary, issue summary, send/IMAP attempts, recent failures, and optional per-job rows.

## Audit log and report emailing

This version adds a tenant-scoped audit log and a first report-emailing flow.

Audit log entries are written for API key creation, campaign import/create, validation, message building, queue actions, IMAP append enqueueing and report email sending/dry runs.

List audit entries from the CLI:

```bash
cd server
python -m app.mailer.commands.audit_log \
  --api-key dev-multimailer-api-key \
  --limit 50
```

Machine-readable audit output:

```bash
python -m app.mailer.commands.audit_log \
  --api-key dev-multimailer-api-key \
  --json
```

API endpoint:

```text
GET /api/v1/audit
```

Optional filters:

```text
?action=campaign.queued&object_type=campaign&object_id=<campaign-db-uuid>&limit=100&offset=0
```

Email a campaign report:

```bash
python -m app.mailer.commands.email_campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --to recipient@example.org \
  --dry-run
```

Actual sending uses the campaign's `server.smtp` configuration and the configured campaign sender. Remove `--dry-run` only once SMTP is configured.

Attach the full JSON report as well as the default per-job CSV:

```bash
python -m app.mailer.commands.email_campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --to recipient@example.org \
  --attach-report-json
```

API endpoint:

```text
POST /api/v1/campaigns/{campaign_id}/report/email
```

Body:

```json
{
  "to": ["recipient@example.org"],
  "include_jobs": false,
  "attach_jobs_csv": true,
  "attach_report_json": false,
  "dry_run": true
}
```

Required API scopes:

```text
audit:read
reports:send
```

## Supported CLI commands

All commands are run from `server/`.

### Database and API-key setup

```bash
python -m app.mailer.commands.init_db --with-dev-data
```

Creates local tables and development data, including the default development API key if enabled.

### Campaign JSON validation

```bash
python -m app.mailer.commands.validate_campaign \
  --campaign app/mailer/examples/campaign.json
```

Use `--check-files` to verify referenced CSV/template/attachment paths.

### Attachment resolution dry run

```bash
python -m app.mailer.commands.resolve_attachments \
  --campaign app/mailer/examples/campaign.json \
  --verbose
```

Use `--json` for machine-readable output.

### Build message drafts from a campaign JSON file

```bash
python -m app.mailer.commands.build_messages \
  --campaign app/mailer/examples/campaign.json \
  --output-dir ./build/messages \
  --verbose
```

Builds reviewable `.eml` files without sending. Use `--json` for machine-readable output.

### SMTP test-send one generated message

```bash
python -m app.mailer.commands.send_test_message \
  --campaign app/mailer/examples/campaign.json \
  --to your.test.address@example.org \
  --dry-run \
  --write-eml ./build/test-message.eml
```

Supports SMTP overrides via CLI flags or these environment variables:

```text
MULTIMAILER_SMTP_HOST
MULTIMAILER_SMTP_PORT
MULTIMAILER_SMTP_SECURITY
MULTIMAILER_SMTP_USERNAME
MULTIMAILER_SMTP_PASSWORD
```

### Import campaign JSON into the database

```bash
python -m app.mailer.commands.import_campaign \
  --campaign app/mailer/examples/campaign.json \
  --api-key dev-multimailer-api-key \
  --validate \
  --build
```

Creates a versioned DB campaign snapshot, optionally validates and builds DB-backed jobs.

### List database campaigns

```bash
python -m app.mailer.commands.list_db_campaigns \
  --api-key dev-multimailer-api-key
```

### Queue built campaign jobs

```bash
python -m app.mailer.commands.queue_campaign \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --no-celery
```

Useful flags:

```text
--dry-run
--exclude-warnings
--version-id <campaign-version-uuid>
--json
```

### Process queued jobs directly without Celery

```bash
python -m app.mailer.commands.send_queued_jobs \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run
```

Useful flags:

```text
--limit <n>
--no-rate-limit
--append-sent
--json
```

### Append already-sent messages to IMAP Sent

```bash
python -m app.mailer.commands.append_pending_sent \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run
```

Useful flags:

```text
--limit <n>
--include-failed
--json
```

### Generate campaign report

```bash
python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key
```

Useful flags:

```text
--json
--include-jobs
--jobs-csv ./build/campaign-jobs.csv
```

### Email campaign report

```bash
python -m app.mailer.commands.email_campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --to recipient@example.org \
  --dry-run
```

Useful flags:

```text
--to <email>              # repeat for multiple recipients
--include-jobs
--no-jobs-csv
--attach-report-json
--json
```

### List audit log

```bash
python -m app.mailer.commands.audit_log \
  --api-key dev-multimailer-api-key
```

Useful filters:

```text
--action campaign.queued
--object-type campaign
--object-id <campaign-db-uuid>
--limit 100
--offset 0
--json
```

## Web UI MVP

The project now includes a first React/Vite WebUI in `webui/`. It is intentionally API-first: the browser talks to the same FastAPI endpoints that the CLI uses.

### Run API locally and WebUI in VSCodium/dev mode

Terminal 1, from `server/`:

```bash
python -m uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
```

Terminal 2, from `webui/`:

```bash
npm install
npm run dev
```

Open:

```text
http://127.0.0.1:5173
```

Use the development API key:

```text
dev-multimailer-api-key
```

The Vite dev server proxies `/api` to `http://127.0.0.1:8000` by default. Override this with:

```bash
VITE_DEV_API_PROXY_TARGET=http://127.0.0.1:8000 npm run dev
```

### Run full local stack with Docker Compose

```bash
cp .env.example .env
./scripts/up-local.sh
```

Open:

```text
http://localhost:8080
```

Traefik routes:

```text
/api/* and /health -> FastAPI
/*                 -> WebUI
```

Direct ports are also published by default:

```text
API:   http://localhost:8000
WebUI: http://localhost:5173
```

### WebUI features included in this MVP

- API-key based connection screen
- Campaign list and selection
- Campaign JSON import from file or textarea
- Version listing
- Campaign validation
- Message building
- Queue dry-run and queue action
- Pause/resume/cancel actions
- Report dashboard cards
- Job table with build/validation/queue/send/IMAP states
- Jobs CSV download
- IMAP append dry-run/action
- Campaign report email dry-run/action
- Audit log view
- Local WebUI settings

### Important security note

For this MVP, the WebUI stores the API base URL and API key in browser `localStorage`. This is acceptable for local development, but production should move to proper login/session handling and short-lived browser tokens. External tools and desktop clients can continue using explicit scoped API keys.

## WebUI source layout

```text
webui/
├─ Dockerfile
├─ nginx.conf
├─ package.json
├─ vite.config.ts
├─ index.html
└─ src/
   ├─ api.ts
   ├─ main.tsx
   ├─ styles.css
   └─ types.ts
```

## WebUI next steps

Recommended next UI iterations:

1. Add a dedicated campaign configuration editor instead of JSON-only import.
2. Add a review screen for `needs_review`, `blocked`, missing attachments and ambiguous attachments.
3. Add mail-account administration with encrypted credential storage.
4. Add users/groups/RBAC/API-key administration.
5. Add attachment upload and Garage browser.
6. Add housekeeping/retention/quota screens.
7. Add backup/export screens.
8. Add scheduled campaign runs.

## Editable campaign versions / WebUI editor API

Campaigns now use normal `Campaign` + `CampaignVersion` records for draft/edit/autosave workflows. There is no separate `CampaignDraft` table: an editable campaign is simply a campaign version with `workflow_state=editing` and editor metadata.

New version/editor fields on `campaign_versions`:

```text
source_base_path
workflow_state      # editing, under_review, approved, built, queued, sending, completed, cancelled, archived
current_flow        # create, review, send, manual, json
current_step        # basics, sender, fields, recipients, template, attachments, review, send, json, ...
is_complete
editor_state
autosaved_at
published_at
locked_at
locked_by_user_id
```

New API endpoints:

```text
GET  /api/v1/schemas/campaign
POST /api/v1/campaigns/new
GET  /api/v1/campaigns/{campaign_id}/versions/{version_id}
PUT  /api/v1/campaigns/{campaign_id}/versions/{version_id}
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/autosave
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/set-step
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/validate-partial
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/publish
```

Minimal campaign creation example:

```bash
curl -X POST 'http://127.0.0.1:8000/api/v1/campaigns/new' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: dev-multimailer-api-key' \
  -d '{
    "external_id": "my-new-campaign",
    "name": "My New Campaign",
    "description": "Created from the WebUI wizard",
    "current_flow": "create",
    "current_step": "basics"
  }'
```

Autosave example:

```bash
curl -X POST 'http://127.0.0.1:8000/api/v1/campaigns/<campaign-id>/versions/<version-id>/autosave' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: dev-multimailer-api-key' \
  -d '{
    "campaign_json": {"version": "1.0", "campaign": {"id": "my-new-campaign", "name": "My New Campaign", "mode": "draft"}},
    "current_flow": "create",
    "current_step": "recipients",
    "is_complete": false
  }'
```

Partial validation example:

```bash
curl -X POST 'http://127.0.0.1:8000/api/v1/campaigns/<campaign-id>/versions/<version-id>/validate-partial' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: dev-multimailer-api-key' \
  -d '{"section": "recipients"}'
```

Strict validation/build/send endpoints are unchanged. The WebUI should use partial validation while editing, and only call strict validation/build when the user reaches Review/Send.