govoplan-ops/docs/SCALABILITY_PROFILES.md

GovOPlaN Ops Scalability Profiles

govoplan-ops presents the operator-facing view of deployment health, readiness, worker split, and sizing assumptions. This runbook is the canonical home for sizing profiles and explains how an operator should use them with the live Ops page.

Operator Flow

1. Open Admin > Ops.
2. Confirm the active runtime profile and readiness state.
3. Check whether Redis and worker queues match the intended deployment shape.
4. Compare the installation with the sizing worksheet in the core guide.
5. Resolve readiness blockers before enabling package operations, queue-heavy campaigns, imports, exports, or workflow automation.
6. Record the current profile and open measurements before moving to a larger topology.

Live Profile Signals

The Ops API reports:

• current profile (local-dev, production-like-dev, single-process, or split-worker)
• module count
• redacted database and Redis URLs
• Celery enabled state and queue list
• maintenance mode state
• file-storage backend
• HTTP/certificate deployment posture through the deployment_security check
• readiness blockers
• reference deployment profiles and sizing assumptions

These values are intentionally diagnostic. They do not replace deployment configuration management, backups, monitoring, or restore drills.

deployment_security is inactive for local/test profiles. In staging or pilot profiles it warns when secure cookies or CORS origins still look local. In production it becomes readiness-critical because TLS certificates, proxy headers, secure cookies, and explicit WebUI origins must be handled by the deployment before normal traffic.

Promotion Rules

Promote from local development to a production-like profile when a feature depends on PostgreSQL, Redis, Celery, module package lifecycle, or durable file storage.

Promote from a single-process profile to a split-worker profile when queued work becomes part of normal operation:

• campaign sending or IMAP append
• connector import/export
• report generation
• workflow/task automation
• scheduled maintenance jobs
• package install/uninstall daemon work

Promote from local storage to object/shared storage before running more than one API process.

Measurements To Capture

Before increasing capacity, capture:

• peak concurrent users and API latency
• database size, slow queries, and backup/restore duration
• file count, total bytes, and monthly growth
• queue depth, oldest job age, retry rate, and permanent failure rate
• SMTP/IMAP provider throttling responses
• connector transfer duration and failure rate
• worker CPU/memory saturation
• audit/retention growth

The future requirements calculator should accept these measurements and return the structured recommendation described below.

Deployment Topologies

Profile	Intended Use	Topology
Development	Local feature work and module tests	One API process, Vite dev server, SQLite or disposable PostgreSQL, local file storage, synchronous workers.
Pilot	Single office, non-critical early usage	One API process, one WebUI build, PostgreSQL, local or object storage, optional Redis, one worker process when queues are enabled.
Small Production	One tenant or small agency	Two API processes behind a reverse proxy, PostgreSQL with backups, durable storage, Redis/Celery workers, health monitoring.
Medium Deployment	Municipal deployment with multiple departments	Separate WebUI, API, worker, scheduler, PostgreSQL, object storage, Redis, backup host, metrics/log collection.
Shared Platform	Multiple tenants or high campaign/workflow volume	Horizontally scaled WebUI/API/workers, managed PostgreSQL, object storage, queue/cache HA, central monitoring, controlled maintenance windows.

Stateless And Stateful Components

Stateless and horizontally replicable:

• WebUI static assets
• API workers when MASTER_KEY_B64, DATABASE_URL, storage, queue, and module configuration are shared
• Background workers when queues and idempotency keys are used
• Scheduler replicas only when leader election or an external lock exists

Stateful or singleton-sensitive:

• PostgreSQL
• local file storage when not replaced by object storage
• Redis/queue state
• module installer daemon and package mutation operations
• migration execution
• scheduler without distributed locking
• outgoing campaign append/send jobs unless claim tokens are enforced

Readiness And Degraded Modes

Component	Ready When	Degraded Mode
API	Database reachable, migrations current, enabled module registry builds, maintenance mode understood	Read-only/admin-only where routes allow it; otherwise fail closed.
WebUI	Static assets match backend module metadata contract	Show unavailable modules/routes with reason; do not invent routes.
PostgreSQL	Accepts connections and migration head is current	Block writes and package changes if migration state is unknown.
Storage	Configured backend is reachable and writable for write flows	Read-only file views may continue if storage is read-only but reachable.
Redis/Celery	Broker reachable and worker queues have heartbeats	Synchronous dev-only workflows may continue; production async send/workflow queues are degraded.
Installer daemon	Lock is free or owned by a live daemon; latest status is fresh	Admin UI can plan changes but not execute them.
Mail transport	SMTP/IMAP profiles validate for the selected scope	Campaign validation blocks send/append but allows draft editing.

Queue And Worker Scaling

Worker pools should be split by queue once load appears:

• send_email: SMTP send throughput, rate limits, retries, and outcome uncertainty.
• append_sent: IMAP append latency and mailbox-side throttling.
• workflow: process orchestration and case/task state transitions.
• transform: datasource extraction, transformation, and export jobs.
• notifications: postbox, email notification, calendar, and external notification fan-out.
• reporting: long-running exports, aggregates, and audit/report generation.

Scaling signal examples:

• queue depth and oldest queued job age
• retry rate and permanent failure rate
• worker CPU and memory saturation
• database lock time and query latency
• SMTP/IMAP provider throttling responses
• storage upload/download latency

Autoscaling should have upper bounds per queue. Mail and connector queues often hit external throttles before CPU is exhausted, so adding workers blindly can make failures worse.

Database Path

PostgreSQL is the production database. SQLite remains a local-development and tiny disposable profile only.

Production migrations should run explicitly before startup or package activation. Module install/uninstall workflows must use database backup and restore-check hooks for PostgreSQL before migrations or destructive retirement.

First Sizing Matrix

These numbers are starting assumptions, not guarantees. Measure and adjust once real workload metrics exist.

Profile	CPU	Memory	Database	Storage	Queue/Cache	Backup/Monitoring Assumptions
Development	2 cores	4-8 GB	SQLite or local PostgreSQL	local disk	optional	no SLA; manual reset acceptable
Pilot	2-4 cores	8 GB	PostgreSQL on same host or small managed instance	100-500 GB durable local/object storage	optional Redis	daily DB backup; basic health checks
Small Production	4-8 cores	16 GB	dedicated PostgreSQL, 2-4 vCPU, 8-16 GB RAM	0.5-2 TB object/durable storage	Redis plus 1-2 workers	daily full backup plus WAL/snapshot policy; uptime alerts
Medium Deployment	8-16 API/worker cores total	32-64 GB total	PostgreSQL 4-8 vCPU, 16-64 GB RAM	2-10 TB object storage	Redis, separate worker pools	central logs/metrics, tested restore, queue alerts
Shared Platform	sized from measured load	64 GB+ total	managed HA PostgreSQL, read replicas only after profiling	10 TB+ object storage	HA queue/cache if required	SLOs, restore drills, capacity alerts, maintenance windows

Workload Dimensions For A Calculator

A later calculator should ask for:

• tenants and active users
• concurrent sessions and peak request rate
• files per month, average/max file size, and retention window
• cases/tasks/workflows per month
• campaigns per month, recipients per campaign, and send window
• IMAP append and inbound mailbox volume
• datasource/import/export job volume and file sizes
• report/audit query frequency
• retention policy and audit growth
• required recovery point and recovery time objectives

Profile Selection Worksheet

The first calculator can be rule-based. It should recommend the lowest profile that satisfies all hard constraints, then show the inputs that pushed the operator upward.

Input	Pilot Threshold	Small Production Threshold	Medium Threshold	Shared Platform Threshold
Active tenants	1	1-5	5-25	25+
Concurrent users	up to 10	up to 50	up to 250	measured/contracted
Managed files	under 100 GB	100 GB-2 TB	2-10 TB	10 TB+
Campaign recipients/month	under 5,000	5,000-100,000	100,000-1,000,000	provider-limited or multi-tenant
Workflow/import jobs/day	under 100	100-2,000	2,000-25,000	queue-specific scaling
Recovery point objective	daily backup	daily plus WAL/snapshots	tested restore, tighter RPO	formal SLO/SLA
Process split requirement	optional	API plus worker	API, workers, scheduler split	horizontal replicas

Hard constraints override the numeric thresholds:

• Multiple API replicas require PostgreSQL and shared storage.
• Cross-node file access requires object storage or a shared durable file service, not node-local disk.
• Async campaign send, append, imports, exports, or workflows require Redis and workers outside development.
• Package install/uninstall in production requires maintenance mode, backup, restore-check hooks, and installer daemon visibility.
• Autoscaling requires idempotent job claims, readiness probes, external throttling limits, and queue-specific maximum replica counts.

Calculator Output Contract

The future UI calculator should emit a structured recommendation:

{
  "recommended_profile": "small-production",
  "minimum_components": [
    "PostgreSQL",
    "Redis",
    "API process",
    "worker process",
    "durable file storage"
  ],
  "reasons": [
    "Campaign recipients/month exceed pilot threshold.",
    "Async mail delivery requires worker split."
  ],
  "warnings": [
    "Object storage is recommended before adding a second API node."
  ],
  "open_measurements": [
    "Peak concurrent users",
    "Database backup restore duration"
  ]
}

Until the calculator is implemented, operators should fill the worksheet manually and compare it with the Ops page's current profile, readiness checks, worker queues, and sizing assumptions.

Minimum Production Requirements

A production deployment, even a small one, should have:

• PostgreSQL, explicit migrations, backups, and a tested restore path.
• A stable MASTER_KEY_B64 stored outside the repository.
• HTTPS, exact CORS origins, secure cookies, and a reverse proxy.
• Durable file storage with backup or object-store lifecycle policy.
• Redis plus at least one worker when any queued module behavior is enabled.
• Health/readiness checks visible in govoplan-ops.
• Maintenance-mode access assigned to at least one operator account.
• Module catalog trust roots and license trust roots pinned by deployment configuration.

Manual Before Automatic

The first supported production scaling path is manual:

1. Move from SQLite/local storage to PostgreSQL and durable storage.
2. Add workers and Redis for queue-backed operations.
3. Split WebUI/API/worker processes.
4. Add health checks and deployment profile warnings.
5. Add metrics and queue-depth alerts.
6. Scale API and worker replicas with fixed limits.
7. Add autoscaling only after idempotency, readiness, and external throttles are understood.