add-ideas/multi-seal-mail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f3db5fc5cf01b1ac2247fd0541f2544ff1ef410d
multi-seal-mail/README.md

23 KiB
Raw Blame History

MultiMailer skeleton

Local-first bulk mailer backend skeleton:

  • FastAPI API
  • Celery worker(s)
  • Redis queue
  • Postgres database
  • Garage S3-compatible object storage
  • Optional local Traefik
  • Modular Compose layout so you can reuse existing infrastructure

Quick start: full local development stack

cp .env.example .env
./scripts/up-local.sh

Equivalent explicit command:

docker compose -f compose.yml -f compose.local.yml up --build

Health check:

curl http://localhost:8080/health
# or direct API port:
curl http://localhost:8000/health

Garage S3 local endpoint:

http://localhost:3900

App-only mode with existing Postgres, Redis, Garage, Traefik

  1. Copy .env.example to .env
  2. Point these values to your existing services:
DATABASE_URL=postgresql+psycopg://user:password@existing-postgres:5432/multimailer
REDIS_URL=redis://existing-redis:6379/0
S3_ENDPOINT_URL=https://garage.example.org
S3_REGION=garage
S3_BUCKET=attachments
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...
  1. Start app services only:
docker compose -f compose.yml up --build

Or use the example external Traefik override:

docker compose -f compose.yml -f compose.external.example.yml up --build

Scale within one worker container

For small PDFs zipped per recipient, start with long-lived Celery worker processes:

CELERY_CONCURRENCY=4
CELERY_PREFETCH_MULTIPLIER=1
CELERY_MAX_TASKS_PER_CHILD=200

Scale with more containers

docker compose -f compose.yml -f compose.local.yml up --build --scale worker=3 --scale api=2

Notes

  • Set MASTER_KEY_B64 before storing real SMTP/IMAP credentials.
  • Garage 2.3+ can create the local default bucket and key from environment variables when started with --single-node --default-bucket.
  • The local Garage config uses replication_factor = 1; this is fine for development, not a production storage strategy.

Campaign JSON

The project now includes a first schema and example campaign file:

server/app/mailer/schemas/campaign.schema.json
server/app/mailer/examples/campaign.json

The schema preserves the Java object model concepts: global recipients, individual recipients, combine flags, campaign-level attachments.base_path, global and entry-specific attachment configs, validation behavior for missing/ambiguous attachments, rate limits, and optional IMAP append-to-Sent handling.

Attachment resolver dry run

The attachment resolver loads the campaign JSON, normalizes inline or external entries, renders ${global::...} and ${local::...} placeholders in base_dir and file_filter, and reports missing or ambiguous attachment matches without sending mail.

cd server
python -m app.mailer.commands.resolve_attachments \
  --campaign app/mailer/examples/campaign.json \
  --verbose

Machine-readable output for the future web UI:

python -m app.mailer.commands.resolve_attachments \
  --campaign app/mailer/examples/campaign.json \
  --json

Build message drafts without sending

After validating the campaign and resolving attachments, build reviewable message drafts:

cd server
python -m app.mailer.commands.build_messages \
  --campaign app/mailer/examples/campaign.json \
  --output-dir ./build/messages \
  --verbose

This writes .eml files for review and prints per-entry status:

  • ready: queueable
  • warning: queueable, but with non-blocking issues
  • needs_review: user decision required before queueing
  • blocked: must not be queued
  • excluded: excluded by policy, but still reported
  • inactive: inactive entry

Machine-readable report:

python -m app.mailer.commands.build_messages \
  --campaign app/mailer/examples/campaign.json \
  --json

SMTP test send

After building messages, you can send one generated message to a test recipient without changing campaign/job status:

cd server
python -m app.mailer.commands.send_test_message \
  --campaign app/mailer/examples/campaign.json \
  --to your.test.address@example.org \
  --dry-run \
  --write-eml ./build/test-message.eml

When ready to actually connect to SMTP, remove --dry-run. SMTP settings are read from campaign.json, but can be overridden without editing the file:

MULTIMAILER_SMTP_PASSWORD='...' \
python -m app.mailer.commands.send_test_message \
  --campaign app/mailer/examples/campaign.json \
  --to your.test.address@example.org \
  --smtp-host smtp.example.org \
  --smtp-port 587 \
  --smtp-security starttls \
  --smtp-username user@example.org

Supported override environment variables:

MULTIMAILER_SMTP_HOST
MULTIMAILER_SMTP_PORT
MULTIMAILER_SMTP_SECURITY  # plain | tls | starttls
MULTIMAILER_SMTP_USERNAME
MULTIMAILER_SMTP_PASSWORD

The test-send command always replaces the real To/Cc/Bcc recipients with the test recipient before sending.

Persistence/API-key milestone

This version adds the first persistent backend layer:

  • SQLAlchemy models for tenants, users, roles, API keys, campaigns, campaign versions, jobs, issues, attachments, send attempts, IMAP append attempts and audit log entries.
  • Alembic setup under server/alembic.
  • Development database bootstrap command.
  • API-key authentication for campaign API endpoints.
  • Campaign JSON import into versioned DB snapshots.
  • Campaign validation/build results persisted as campaign jobs.

Local DB quick start without Docker

From server/:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# SQLite is the default when DATABASE_URL is not set.
python -m app.mailer.commands.init_db --with-dev-data

python -m app.mailer.commands.import_campaign \
  --campaign app/mailer/examples/campaign.json \
  --api-key dev-multimailer-api-key \
  --validate \
  --build

python -m app.mailer.commands.list_db_campaigns \
  --api-key dev-multimailer-api-key

API quick smoke test

Start the API:

uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

The dev bootstrap runs automatically in APP_ENV=dev and creates this API key unless disabled:

dev-multimailer-api-key

List campaigns:

curl -H 'X-API-Key: dev-multimailer-api-key' \
  http://127.0.0.1:8000/api/v1/campaigns

Create a campaign from JSON:

python - <<'PY'
import json, requests
config = json.load(open('app/mailer/examples/campaign.json'))
response = requests.post(
    'http://127.0.0.1:8000/api/v1/campaigns',
    headers={'X-API-Key': 'dev-multimailer-api-key'},
    json={'config': config, 'source_filename': 'app/mailer/examples/campaign.json'},
)
print(response.status_code)
print(response.text)
PY

API endpoints added

GET  /api/v1/campaigns
POST /api/v1/campaigns
GET  /api/v1/campaigns/{campaign_id}
GET  /api/v1/campaigns/{campaign_id}/versions
POST /api/v1/campaigns/versions/{version_id}/validate
POST /api/v1/campaigns/versions/{version_id}/build
GET  /api/v1/campaigns/{campaign_id}/jobs
POST /api/v1/admin/api-keys

All campaign/admin endpoints require an API key via either:

X-API-Key: ...
Authorization: Bearer ...

Note about local file paths

When importing a file-based campaign JSON, pass source_filename or use the CLI importer. Relative paths inside the campaign are normalized to absolute paths at import time so the DB snapshot can still find CSV files, templates and attachment folders later.

Queueing and sending step

This version adds the first real bulk-delivery transition:

built campaign jobs -> queued jobs -> SMTP worker/direct sender -> send attempts

Queue jobs without starting Celery

cd server
python -m app.mailer.commands.queue_campaign \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --no-celery

Use --dry-run to see what would be queued without modifying the database.

Process queued jobs directly from CLI

This is useful before the web UI and before running a Celery worker:

python -m app.mailer.commands.send_queued_jobs \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run

Remove --dry-run only after SMTP credentials in the campaign JSON point to a real server.

Queue jobs and send through Celery

python -m app.mailer.commands.queue_campaign \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key

celery -A app.celery_app.celery worker \
  --loglevel=INFO \
  --queues=send_email,append_sent,default \
  --concurrency=1 \
  --prefetch-multiplier=1

API endpoints added

POST /api/v1/campaigns/{campaign_id}/queue
POST /api/v1/campaigns/{campaign_id}/pause
POST /api/v1/campaigns/{campaign_id}/resume
POST /api/v1/campaigns/{campaign_id}/cancel

Queueing still respects the review gate: only ready and, by default, warning jobs are queued. needs_review, blocked, excluded, and inactive jobs are not queued.

Rate limiting

SMTP sending now applies campaign-level rate limiting from the campaign JSON:

"delivery": {
  "rate_limit": {
    "messages_per_minute": 5,
    "concurrency": 1
  }
}

The worker uses Redis for distributed throttling when available. If Redis is unavailable in local development, the sender falls back gracefully, but production should use Redis.

IMAP status

SMTP success now keeps IMAP state separate:

send_status = sent
imap_status = pending | not_requested

Actual IMAP APPEND is still the next implementation step.

IMAP append-to-Sent

SMTP sending and IMAP Sent-folder append are intentionally separate states. A job can be send_status=sent while imap_status=failed; the email was still sent, but saving the copy to Sent needs attention.

When Celery is used, successful SMTP delivery automatically enqueues an append_sent task if the campaign has:

"delivery": {
  "imap_append_sent": { "enabled": true, "folder": "auto" }
}

and server.imap.enabled=true.

Direct CLI sending can append immediately in the same process:

python -m app.mailer.commands.send_queued_jobs \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --append-sent

Or process pending/failed appends separately:

python -m app.mailer.commands.append_pending_sent \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key

Dry-run IMAP append check:

python -m app.mailer.commands.append_pending_sent \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run

API endpoint:

POST /api/v1/campaigns/{campaign_id}/append-sent

with body:

{
  "enqueue_celery": true,
  "dry_run": false
}

Folder resolution order:

  1. delivery.imap_append_sent.folder if not auto
  2. server.imap.sent_folder if not auto
  3. IMAP LIST discovery using \\Sent special-use flags
  4. common fallback names such as Sent, Sent Items, Gesendet

Campaign reporting / dashboard data

This version adds dashboard/report payloads that are intended to feed the first web interface. They summarize persisted jobs, validation issues, attachment resolution, sending attempts, IMAP append state, and rate-limit estimates.

CLI report:

cd server
python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key

Machine-readable report:

python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --json \
  --include-jobs

Per-job CSV export:

python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --jobs-csv ./build/campaign-jobs.csv

API endpoints:

GET /api/v1/campaigns/{campaign_id}/summary
GET /api/v1/campaigns/{campaign_id}/report
GET /api/v1/campaigns/{campaign_id}/report/jobs.csv

These endpoints are API-key protected with the campaign:read scope and return data shaped for a web dashboard: top-level cards, status counters, attachment summary, issue summary, send/IMAP attempts, recent failures, and optional per-job rows.

Audit log and report emailing

This version adds a tenant-scoped audit log and a first report-emailing flow.

Audit log entries are written for API key creation, campaign import/create, validation, message building, queue actions, IMAP append enqueueing and report email sending/dry runs.

List audit entries from the CLI:

cd server
python -m app.mailer.commands.audit_log \
  --api-key dev-multimailer-api-key \
  --limit 50

Machine-readable audit output:

python -m app.mailer.commands.audit_log \
  --api-key dev-multimailer-api-key \
  --json

API endpoint:

GET /api/v1/audit

Optional filters:

?action=campaign.queued&object_type=campaign&object_id=<campaign-db-uuid>&limit=100&offset=0

Email a campaign report:

python -m app.mailer.commands.email_campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --to recipient@example.org \
  --dry-run

Actual sending uses the campaign's server.smtp configuration and the configured campaign sender. Remove --dry-run only once SMTP is configured.

Attach the full JSON report as well as the default per-job CSV:

python -m app.mailer.commands.email_campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --to recipient@example.org \
  --attach-report-json

API endpoint:

POST /api/v1/campaigns/{campaign_id}/report/email

Body:

{
  "to": ["recipient@example.org"],
  "include_jobs": false,
  "attach_jobs_csv": true,
  "attach_report_json": false,
  "dry_run": true
}

Required API scopes:

audit:read
reports:send

Supported CLI commands

All commands are run from server/.

Database and API-key setup

python -m app.mailer.commands.init_db --with-dev-data

Creates local tables and development data, including the default development API key if enabled.

Campaign JSON validation

python -m app.mailer.commands.validate_campaign \
  --campaign app/mailer/examples/campaign.json

Use --check-files to verify referenced CSV/template/attachment paths.

Attachment resolution dry run

python -m app.mailer.commands.resolve_attachments \
  --campaign app/mailer/examples/campaign.json \
  --verbose

Use --json for machine-readable output.

Build message drafts from a campaign JSON file

python -m app.mailer.commands.build_messages \
  --campaign app/mailer/examples/campaign.json \
  --output-dir ./build/messages \
  --verbose

Builds reviewable .eml files without sending. Use --json for machine-readable output.

SMTP test-send one generated message

python -m app.mailer.commands.send_test_message \
  --campaign app/mailer/examples/campaign.json \
  --to your.test.address@example.org \
  --dry-run \
  --write-eml ./build/test-message.eml

Supports SMTP overrides via CLI flags or these environment variables:

MULTIMAILER_SMTP_HOST
MULTIMAILER_SMTP_PORT
MULTIMAILER_SMTP_SECURITY
MULTIMAILER_SMTP_USERNAME
MULTIMAILER_SMTP_PASSWORD

Import campaign JSON into the database

python -m app.mailer.commands.import_campaign \
  --campaign app/mailer/examples/campaign.json \
  --api-key dev-multimailer-api-key \
  --validate \
  --build

Creates a versioned DB campaign snapshot, optionally validates and builds DB-backed jobs.

List database campaigns

python -m app.mailer.commands.list_db_campaigns \
  --api-key dev-multimailer-api-key

Queue built campaign jobs

python -m app.mailer.commands.queue_campaign \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --no-celery

Useful flags:

--dry-run
--exclude-warnings
--version-id <campaign-version-uuid>
--json

Process queued jobs directly without Celery

python -m app.mailer.commands.send_queued_jobs \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run

Useful flags:

--limit <n>
--no-rate-limit
--append-sent
--json

Append already-sent messages to IMAP Sent

python -m app.mailer.commands.append_pending_sent \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --dry-run

Useful flags:

--limit <n>
--include-failed
--json

Generate campaign report

python -m app.mailer.commands.campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key

Useful flags:

--json
--include-jobs
--jobs-csv ./build/campaign-jobs.csv

Email campaign report

python -m app.mailer.commands.email_campaign_report \
  --campaign-id <campaign-db-uuid> \
  --api-key dev-multimailer-api-key \
  --to recipient@example.org \
  --dry-run

Useful flags:

--to <email>              # repeat for multiple recipients
--include-jobs
--no-jobs-csv
--attach-report-json
--json

List audit log

python -m app.mailer.commands.audit_log \
  --api-key dev-multimailer-api-key

Useful filters:

--action campaign.queued
--object-type campaign
--object-id <campaign-db-uuid>
--limit 100
--offset 0
--json

Web UI MVP

The project now includes a first React/Vite WebUI in webui/. It is intentionally API-first: the browser talks to the same FastAPI endpoints that the CLI uses.

Run API locally and WebUI in VSCodium/dev mode

Terminal 1, from server/:

python -m uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

Terminal 2, from webui/:

npm install
npm run dev

Open:

http://127.0.0.1:5173

Use the development API key:

dev-multimailer-api-key

The Vite dev server proxies /api to http://127.0.0.1:8000 by default. Override this with:

VITE_DEV_API_PROXY_TARGET=http://127.0.0.1:8000 npm run dev

Run full local stack with Docker Compose

cp .env.example .env
./scripts/up-local.sh

Open:

http://localhost:8080

Traefik routes:

/api/* and /health -> FastAPI
/*                 -> WebUI

Direct ports are also published by default:

API:   http://localhost:8000
WebUI: http://localhost:5173

WebUI features included in this MVP

  • API-key based connection screen
  • Campaign list and selection
  • Campaign JSON import from file or textarea
  • Version listing
  • Campaign validation
  • Message building
  • Queue dry-run and queue action
  • Pause/resume/cancel actions
  • Report dashboard cards
  • Job table with build/validation/queue/send/IMAP states
  • Jobs CSV download
  • IMAP append dry-run/action
  • Campaign report email dry-run/action
  • Audit log view
  • Local WebUI settings

Important security note

For this MVP, the WebUI stores the API base URL and API key in browser localStorage. This is acceptable for local development, but production should move to proper login/session handling and short-lived browser tokens. External tools and desktop clients can continue using explicit scoped API keys.

WebUI source layout

webui/
├─ Dockerfile
├─ nginx.conf
├─ package.json
├─ vite.config.ts
├─ index.html
└─ src/
   ├─ api.ts
   ├─ main.tsx
   ├─ styles.css
   └─ types.ts

WebUI next steps

Recommended next UI iterations:

  1. Add a dedicated campaign configuration editor instead of JSON-only import.
  2. Add a review screen for needs_review, blocked, missing attachments and ambiguous attachments.
  3. Add mail-account administration with encrypted credential storage.
  4. Add users/groups/RBAC/API-key administration.
  5. Add attachment upload and Garage browser.
  6. Add housekeeping/retention/quota screens.
  7. Add backup/export screens.
  8. Add scheduled campaign runs.

Editable campaign versions / WebUI editor API

Campaigns now use normal Campaign + CampaignVersion records for draft/edit/autosave workflows. There is no separate CampaignDraft table: an editable campaign is simply a campaign version with workflow_state=editing and editor metadata.

New version/editor fields on campaign_versions:

source_base_path
workflow_state      # editing, under_review, approved, built, queued, sending, completed, cancelled, archived
current_flow        # create, review, send, manual, json
current_step        # basics, sender, fields, recipients, template, attachments, review, send, json, ...
is_complete
editor_state
autosaved_at
published_at
locked_at
locked_by_user_id

New API endpoints:

GET  /api/v1/schemas/campaign
POST /api/v1/campaigns/new
GET  /api/v1/campaigns/{campaign_id}/versions/{version_id}
PUT  /api/v1/campaigns/{campaign_id}/versions/{version_id}
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/autosave
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/set-step
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/validate-partial
POST /api/v1/campaigns/{campaign_id}/versions/{version_id}/publish

Minimal campaign creation example:

curl -X POST 'http://127.0.0.1:8000/api/v1/campaigns/new' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: dev-multimailer-api-key' \
  -d '{
    "external_id": "my-new-campaign",
    "name": "My New Campaign",
    "description": "Created from the WebUI wizard",
    "current_flow": "create",
    "current_step": "basics"
  }'

Autosave example:

curl -X POST 'http://127.0.0.1:8000/api/v1/campaigns/<campaign-id>/versions/<version-id>/autosave' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: dev-multimailer-api-key' \
  -d '{
    "campaign_json": {"version": "1.0", "campaign": {"id": "my-new-campaign", "name": "My New Campaign", "mode": "draft"}},
    "current_flow": "create",
    "current_step": "recipients",
    "is_complete": false
  }'

Partial validation example:

curl -X POST 'http://127.0.0.1:8000/api/v1/campaigns/<campaign-id>/versions/<version-id>/validate-partial' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: dev-multimailer-api-key' \
  -d '{"section": "recipients"}'

Strict validation/build/send endpoints are unchanged. The WebUI should use partial validation while editing, and only call strict validation/build when the user reaches Review/Send.

Browser login / session auth

The backend now supports both automation API keys and browser session tokens.

Development login after init_db --with-dev-data or dev bootstrap:

Tenant: default
Email: admin@example.local
Password: dev-admin

Login endpoint:

curl -X POST http://127.0.0.1:8000/api/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"tenant_slug":"default","email":"admin@example.local","password":"dev-admin"}'

Use the returned access_token as Authorization: Bearer <token>. Existing API keys still work via X-API-Key.

RBAC scaffolding now includes users, groups, roles, direct user-role assignments, group-role assignments and login sessions. The development user receives the owner role.