govoplan-ops/docs/SCALABILITY_PROFILES.md

# GovOPlaN Ops Scalability Profiles

`govoplan-ops` presents the operator-facing view of deployment health,
readiness, worker split, and sizing assumptions. This runbook is the canonical
home for sizing profiles and explains how an operator should use them with the
live Ops page.

## Operator Flow

1. Open Admin > Ops.
2. Confirm the active runtime profile and readiness state.
3. Check whether Redis and worker queues match the intended deployment shape.
4. Compare the installation with the sizing worksheet in the core guide.
5. Resolve readiness blockers before enabling package operations, queue-heavy
   campaigns, imports, exports, or workflow automation.
6. Record the current profile and open measurements before moving to a larger
   topology.

## Live Profile Signals

The Ops API reports:

- current profile (`local-dev`, `production-like-dev`, `single-process`, or
  `split-worker`)
- module count
- redacted database and Redis URLs
- Celery enabled state and queue list
- maintenance mode state
- file-storage backend
- HTTP/certificate deployment posture through the `deployment_security` check
- readiness blockers
- reference deployment profiles and sizing assumptions

These values are intentionally diagnostic. They do not replace deployment
configuration management, backups, monitoring, or restore drills.

`deployment_security` is inactive for local/test profiles. In staging or pilot
profiles it warns when secure cookies or CORS origins still look local. In
production it becomes readiness-critical because TLS certificates, proxy
headers, secure cookies, and explicit WebUI origins must be handled by the
deployment before normal traffic.

## Promotion Rules

Promote from local development to a production-like profile when a feature
depends on PostgreSQL, Redis, Celery, module package lifecycle, or durable file
storage.

Promote from a single-process profile to a split-worker profile when queued
work becomes part of normal operation:

- campaign sending or IMAP append
- connector import/export
- report generation
- workflow/task automation
- scheduled maintenance jobs
- package install/uninstall daemon work

Promote from local storage to object/shared storage before running more than
one API process.

## Measurements To Capture

Before increasing capacity, capture:

- peak concurrent users and API latency
- database size, slow queries, and backup/restore duration
- file count, total bytes, and monthly growth
- queue depth, oldest job age, retry rate, and permanent failure rate
- SMTP/IMAP provider throttling responses
- connector transfer duration and failure rate
- worker CPU/memory saturation
- audit/retention growth

The future requirements calculator should accept these measurements and return
the structured recommendation described below.

## Deployment Topologies

| Profile | Intended Use | Topology |
| --- | --- | --- |
| Development | Local feature work and module tests | One API process, Vite dev server, SQLite or disposable PostgreSQL, local file storage, synchronous workers. |
| Pilot | Single office, non-critical early usage | One API process, one WebUI build, PostgreSQL, local or object storage, optional Redis, one worker process when queues are enabled. |
| Small Production | One tenant or small agency | Two API processes behind a reverse proxy, PostgreSQL with backups, durable storage, Redis/Celery workers, health monitoring. |
| Medium Deployment | Municipal deployment with multiple departments | Separate WebUI, API, worker, scheduler, PostgreSQL, object storage, Redis, backup host, metrics/log collection. |
| Shared Platform | Multiple tenants or high campaign/workflow volume | Horizontally scaled WebUI/API/workers, managed PostgreSQL, object storage, queue/cache HA, central monitoring, controlled maintenance windows. |

## Stateless And Stateful Components

Stateless and horizontally replicable:

- WebUI static assets
- API workers when `MASTER_KEY_B64`, `DATABASE_URL`, storage, queue, and module
  configuration are shared
- Background workers when queues and idempotency keys are used
- Scheduler replicas only when leader election or an external lock exists

Stateful or singleton-sensitive:

- PostgreSQL
- local file storage when not replaced by object storage
- Redis/queue state
- module installer daemon and package mutation operations
- migration execution
- scheduler without distributed locking
- outgoing campaign append/send jobs unless claim tokens are enforced

## Readiness And Degraded Modes

| Component | Ready When | Degraded Mode |
| --- | --- | --- |
| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood | Read-only/admin-only where routes allow it; otherwise fail closed. |
| WebUI | Static assets match backend module metadata contract | Show unavailable modules/routes with reason; do not invent routes. |
| PostgreSQL | Accepts connections and migration head is current | Block writes and package changes if migration state is unknown. |
| Storage | Configured backend is reachable and writable for write flows | Read-only file views may continue if storage is read-only but reachable. |
| Redis/Celery | Broker reachable and worker queues have heartbeats | Synchronous dev-only workflows may continue; production async send/workflow queues are degraded. |
| Installer daemon | Lock is free or owned by a live daemon; latest status is fresh | Admin UI can plan changes but not execute them. |
| Mail transport | SMTP/IMAP profiles validate for the selected scope | Campaign validation blocks send/append but allows draft editing. |

## Queue And Worker Scaling

Worker pools should be split by queue once load appears:

- `send_email`: SMTP send throughput, rate limits, retries, and outcome
  uncertainty.
- `append_sent`: IMAP append latency and mailbox-side throttling.
- `workflow`: process orchestration and case/task state transitions.
- `transform`: datasource extraction, transformation, and export jobs.
- `notifications`: postbox, email notification, calendar, and external
  notification fan-out.
- `reporting`: long-running exports, aggregates, and audit/report generation.

Scaling signal examples:

- queue depth and oldest queued job age
- retry rate and permanent failure rate
- worker CPU and memory saturation
- database lock time and query latency
- SMTP/IMAP provider throttling responses
- storage upload/download latency

Autoscaling should have upper bounds per queue. Mail and connector queues often
hit external throttles before CPU is exhausted, so adding workers blindly can
make failures worse.

## Database Path

PostgreSQL is the production database. SQLite remains a local-development and
tiny disposable profile only.

Production migrations should run explicitly before startup or package
activation. Module install/uninstall workflows must use database backup and
restore-check hooks for PostgreSQL before migrations or destructive retirement.

## First Sizing Matrix

These numbers are starting assumptions, not guarantees. Measure and adjust once
real workload metrics exist.

| Profile | CPU | Memory | Database | Storage | Queue/Cache | Backup/Monitoring Assumptions |
| --- | ---: | ---: | --- | --- | --- | --- |
| Development | 2 cores | 4-8 GB | SQLite or local PostgreSQL | local disk | optional | no SLA; manual reset acceptable |
| Pilot | 2-4 cores | 8 GB | PostgreSQL on same host or small managed instance | 100-500 GB durable local/object storage | optional Redis | daily DB backup; basic health checks |
| Small Production | 4-8 cores | 16 GB | dedicated PostgreSQL, 2-4 vCPU, 8-16 GB RAM | 0.5-2 TB object/durable storage | Redis plus 1-2 workers | daily full backup plus WAL/snapshot policy; uptime alerts |
| Medium Deployment | 8-16 API/worker cores total | 32-64 GB total | PostgreSQL 4-8 vCPU, 16-64 GB RAM | 2-10 TB object storage | Redis, separate worker pools | central logs/metrics, tested restore, queue alerts |
| Shared Platform | sized from measured load | 64 GB+ total | managed HA PostgreSQL, read replicas only after profiling | 10 TB+ object storage | HA queue/cache if required | SLOs, restore drills, capacity alerts, maintenance windows |

## Workload Dimensions For A Calculator

A later calculator should ask for:

- tenants and active users
- concurrent sessions and peak request rate
- files per month, average/max file size, and retention window
- cases/tasks/workflows per month
- campaigns per month, recipients per campaign, and send window
- IMAP append and inbound mailbox volume
- datasource/import/export job volume and file sizes
- report/audit query frequency
- retention policy and audit growth
- required recovery point and recovery time objectives

## Profile Selection Worksheet

The first calculator can be rule-based. It should recommend the lowest profile
that satisfies all hard constraints, then show the inputs that pushed the
operator upward.

| Input | Pilot Threshold | Small Production Threshold | Medium Threshold | Shared Platform Threshold |
| --- | ---: | ---: | ---: | ---: |
| Active tenants | 1 | 1-5 | 5-25 | 25+ |
| Concurrent users | up to 10 | up to 50 | up to 250 | measured/contracted |
| Managed files | under 100 GB | 100 GB-2 TB | 2-10 TB | 10 TB+ |
| Campaign recipients/month | under 5,000 | 5,000-100,000 | 100,000-1,000,000 | provider-limited or multi-tenant |
| Workflow/import jobs/day | under 100 | 100-2,000 | 2,000-25,000 | queue-specific scaling |
| Recovery point objective | daily backup | daily plus WAL/snapshots | tested restore, tighter RPO | formal SLO/SLA |
| Process split requirement | optional | API plus worker | API, workers, scheduler split | horizontal replicas |

Hard constraints override the numeric thresholds:

- Multiple API replicas require PostgreSQL and shared storage.
- Cross-node file access requires object storage or a shared durable file
  service, not node-local disk.
- Async campaign send, append, imports, exports, or workflows require Redis and
  workers outside development.
- Package install/uninstall in production requires maintenance mode, backup,
  restore-check hooks, and installer daemon visibility.
- Autoscaling requires idempotent job claims, readiness probes, external
  throttling limits, and queue-specific maximum replica counts.

## Calculator Output Contract

The future UI calculator should emit a structured recommendation:

```json
{
  "recommended_profile": "small-production",
  "minimum_components": [
    "PostgreSQL",
    "Redis",
    "API process",
    "worker process",
    "durable file storage"
  ],
  "reasons": [
    "Campaign recipients/month exceed pilot threshold.",
    "Async mail delivery requires worker split."
  ],
  "warnings": [
    "Object storage is recommended before adding a second API node."
  ],
  "open_measurements": [
    "Peak concurrent users",
    "Database backup restore duration"
  ]
}
```

Until the calculator is implemented, operators should fill the worksheet
manually and compare it with the Ops page's current profile, readiness checks,
worker queues, and sizing assumptions.

## Minimum Production Requirements

A production deployment, even a small one, should have:

- PostgreSQL, explicit migrations, backups, and a tested restore path.
- A stable `MASTER_KEY_B64` stored outside the repository.
- HTTPS, exact CORS origins, secure cookies, and a reverse proxy.
- Durable file storage with backup or object-store lifecycle policy.
- Redis plus at least one worker when any queued module behavior is enabled.
- Health/readiness checks visible in `govoplan-ops`.
- Maintenance-mode access assigned to at least one operator account.
- Module catalog trust roots and license trust roots pinned by deployment
  configuration.

## Manual Before Automatic

The first supported production scaling path is manual:

1. Move from SQLite/local storage to PostgreSQL and durable storage.
2. Add workers and Redis for queue-backed operations.
3. Split WebUI/API/worker processes.
4. Add health checks and deployment profile warnings.
5. Add metrics and queue-depth alerts.
6. Scale API and worker replicas with fixed limits.
7. Add autoscaling only after idempotency, readiness, and external throttles are
   understood.