chore: sync GovOPlaN module split state
This commit is contained in:
268
docs/SCALABILITY_PROFILES.md
Normal file
268
docs/SCALABILITY_PROFILES.md
Normal file
@@ -0,0 +1,268 @@
|
||||
# GovOPlaN Ops Scalability Profiles
|
||||
|
||||
`govoplan-ops` presents the operator-facing view of deployment health,
|
||||
readiness, worker split, and sizing assumptions. This runbook is the canonical
|
||||
home for sizing profiles and explains how an operator should use them with the
|
||||
live Ops page.
|
||||
|
||||
## Operator Flow
|
||||
|
||||
1. Open Admin > Ops.
|
||||
2. Confirm the active runtime profile and readiness state.
|
||||
3. Check whether Redis and worker queues match the intended deployment shape.
|
||||
4. Compare the installation with the sizing worksheet in the core guide.
|
||||
5. Resolve readiness blockers before enabling package operations, queue-heavy
|
||||
campaigns, imports, exports, or workflow automation.
|
||||
6. Record the current profile and open measurements before moving to a larger
|
||||
topology.
|
||||
|
||||
## Live Profile Signals
|
||||
|
||||
The Ops API reports:
|
||||
|
||||
- current profile (`local-dev`, `production-like-dev`, `single-process`, or
|
||||
`split-worker`)
|
||||
- module count
|
||||
- redacted database and Redis URLs
|
||||
- Celery enabled state and queue list
|
||||
- maintenance mode state
|
||||
- file-storage backend
|
||||
- HTTP/certificate deployment posture through the `deployment_security` check
|
||||
- readiness blockers
|
||||
- reference deployment profiles and sizing assumptions
|
||||
|
||||
These values are intentionally diagnostic. They do not replace deployment
|
||||
configuration management, backups, monitoring, or restore drills.
|
||||
|
||||
`deployment_security` is inactive for local/test profiles. In staging or pilot
|
||||
profiles it warns when secure cookies or CORS origins still look local. In
|
||||
production it becomes readiness-critical because TLS certificates, proxy
|
||||
headers, secure cookies, and explicit WebUI origins must be handled by the
|
||||
deployment before normal traffic.
|
||||
|
||||
## Promotion Rules
|
||||
|
||||
Promote from local development to a production-like profile when a feature
|
||||
depends on PostgreSQL, Redis, Celery, module package lifecycle, or durable file
|
||||
storage.
|
||||
|
||||
Promote from a single-process profile to a split-worker profile when queued
|
||||
work becomes part of normal operation:
|
||||
|
||||
- campaign sending or IMAP append
|
||||
- connector import/export
|
||||
- report generation
|
||||
- workflow/task automation
|
||||
- scheduled maintenance jobs
|
||||
- package install/uninstall daemon work
|
||||
|
||||
Promote from local storage to object/shared storage before running more than
|
||||
one API process.
|
||||
|
||||
## Measurements To Capture
|
||||
|
||||
Before increasing capacity, capture:
|
||||
|
||||
- peak concurrent users and API latency
|
||||
- database size, slow queries, and backup/restore duration
|
||||
- file count, total bytes, and monthly growth
|
||||
- queue depth, oldest job age, retry rate, and permanent failure rate
|
||||
- SMTP/IMAP provider throttling responses
|
||||
- connector transfer duration and failure rate
|
||||
- worker CPU/memory saturation
|
||||
- audit/retention growth
|
||||
|
||||
The future requirements calculator should accept these measurements and return
|
||||
the structured recommendation described below.
|
||||
|
||||
## Deployment Topologies
|
||||
|
||||
| Profile | Intended Use | Topology |
|
||||
| --- | --- | --- |
|
||||
| Development | Local feature work and module tests | One API process, Vite dev server, SQLite or disposable PostgreSQL, local file storage, synchronous workers. |
|
||||
| Pilot | Single office, non-critical early usage | One API process, one WebUI build, PostgreSQL, local or object storage, optional Redis, one worker process when queues are enabled. |
|
||||
| Small Production | One tenant or small agency | Two API processes behind a reverse proxy, PostgreSQL with backups, durable storage, Redis/Celery workers, health monitoring. |
|
||||
| Medium Deployment | Municipal deployment with multiple departments | Separate WebUI, API, worker, scheduler, PostgreSQL, object storage, Redis, backup host, metrics/log collection. |
|
||||
| Shared Platform | Multiple tenants or high campaign/workflow volume | Horizontally scaled WebUI/API/workers, managed PostgreSQL, object storage, queue/cache HA, central monitoring, controlled maintenance windows. |
|
||||
|
||||
## Stateless And Stateful Components
|
||||
|
||||
Stateless and horizontally replicable:
|
||||
|
||||
- WebUI static assets
|
||||
- API workers when `MASTER_KEY_B64`, `DATABASE_URL`, storage, queue, and module
|
||||
configuration are shared
|
||||
- Background workers when queues and idempotency keys are used
|
||||
- Scheduler replicas only when leader election or an external lock exists
|
||||
|
||||
Stateful or singleton-sensitive:
|
||||
|
||||
- PostgreSQL
|
||||
- local file storage when not replaced by object storage
|
||||
- Redis/queue state
|
||||
- module installer daemon and package mutation operations
|
||||
- migration execution
|
||||
- scheduler without distributed locking
|
||||
- outgoing campaign append/send jobs unless claim tokens are enforced
|
||||
|
||||
## Readiness And Degraded Modes
|
||||
|
||||
| Component | Ready When | Degraded Mode |
|
||||
| --- | --- | --- |
|
||||
| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood | Read-only/admin-only where routes allow it; otherwise fail closed. |
|
||||
| WebUI | Static assets match backend module metadata contract | Show unavailable modules/routes with reason; do not invent routes. |
|
||||
| PostgreSQL | Accepts connections and migration head is current | Block writes and package changes if migration state is unknown. |
|
||||
| Storage | Configured backend is reachable and writable for write flows | Read-only file views may continue if storage is read-only but reachable. |
|
||||
| Redis/Celery | Broker reachable and worker queues have heartbeats | Synchronous dev-only workflows may continue; production async send/workflow queues are degraded. |
|
||||
| Installer daemon | Lock is free or owned by a live daemon; latest status is fresh | Admin UI can plan changes but not execute them. |
|
||||
| Mail transport | SMTP/IMAP profiles validate for the selected scope | Campaign validation blocks send/append but allows draft editing. |
|
||||
|
||||
## Queue And Worker Scaling
|
||||
|
||||
Worker pools should be split by queue once load appears:
|
||||
|
||||
- `send_email`: SMTP send throughput, rate limits, retries, and outcome
|
||||
uncertainty.
|
||||
- `append_sent`: IMAP append latency and mailbox-side throttling.
|
||||
- `workflow`: process orchestration and case/task state transitions.
|
||||
- `transform`: datasource extraction, transformation, and export jobs.
|
||||
- `notifications`: postbox, email notification, calendar, and external
|
||||
notification fan-out.
|
||||
- `reporting`: long-running exports, aggregates, and audit/report generation.
|
||||
|
||||
Scaling signal examples:
|
||||
|
||||
- queue depth and oldest queued job age
|
||||
- retry rate and permanent failure rate
|
||||
- worker CPU and memory saturation
|
||||
- database lock time and query latency
|
||||
- SMTP/IMAP provider throttling responses
|
||||
- storage upload/download latency
|
||||
|
||||
Autoscaling should have upper bounds per queue. Mail and connector queues often
|
||||
hit external throttles before CPU is exhausted, so adding workers blindly can
|
||||
make failures worse.
|
||||
|
||||
## Database Path
|
||||
|
||||
PostgreSQL is the production database. SQLite remains a local-development and
|
||||
tiny disposable profile only.
|
||||
|
||||
Production migrations should run explicitly before startup or package
|
||||
activation. Module install/uninstall workflows must use database backup and
|
||||
restore-check hooks for PostgreSQL before migrations or destructive retirement.
|
||||
|
||||
## First Sizing Matrix
|
||||
|
||||
These numbers are starting assumptions, not guarantees. Measure and adjust once
|
||||
real workload metrics exist.
|
||||
|
||||
| Profile | CPU | Memory | Database | Storage | Queue/Cache | Backup/Monitoring Assumptions |
|
||||
| --- | ---: | ---: | --- | --- | --- | --- |
|
||||
| Development | 2 cores | 4-8 GB | SQLite or local PostgreSQL | local disk | optional | no SLA; manual reset acceptable |
|
||||
| Pilot | 2-4 cores | 8 GB | PostgreSQL on same host or small managed instance | 100-500 GB durable local/object storage | optional Redis | daily DB backup; basic health checks |
|
||||
| Small Production | 4-8 cores | 16 GB | dedicated PostgreSQL, 2-4 vCPU, 8-16 GB RAM | 0.5-2 TB object/durable storage | Redis plus 1-2 workers | daily full backup plus WAL/snapshot policy; uptime alerts |
|
||||
| Medium Deployment | 8-16 API/worker cores total | 32-64 GB total | PostgreSQL 4-8 vCPU, 16-64 GB RAM | 2-10 TB object storage | Redis, separate worker pools | central logs/metrics, tested restore, queue alerts |
|
||||
| Shared Platform | sized from measured load | 64 GB+ total | managed HA PostgreSQL, read replicas only after profiling | 10 TB+ object storage | HA queue/cache if required | SLOs, restore drills, capacity alerts, maintenance windows |
|
||||
|
||||
## Workload Dimensions For A Calculator
|
||||
|
||||
A later calculator should ask for:
|
||||
|
||||
- tenants and active users
|
||||
- concurrent sessions and peak request rate
|
||||
- files per month, average/max file size, and retention window
|
||||
- cases/tasks/workflows per month
|
||||
- campaigns per month, recipients per campaign, and send window
|
||||
- IMAP append and inbound mailbox volume
|
||||
- datasource/import/export job volume and file sizes
|
||||
- report/audit query frequency
|
||||
- retention policy and audit growth
|
||||
- required recovery point and recovery time objectives
|
||||
|
||||
## Profile Selection Worksheet
|
||||
|
||||
The first calculator can be rule-based. It should recommend the lowest profile
|
||||
that satisfies all hard constraints, then show the inputs that pushed the
|
||||
operator upward.
|
||||
|
||||
| Input | Pilot Threshold | Small Production Threshold | Medium Threshold | Shared Platform Threshold |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Active tenants | 1 | 1-5 | 5-25 | 25+ |
|
||||
| Concurrent users | up to 10 | up to 50 | up to 250 | measured/contracted |
|
||||
| Managed files | under 100 GB | 100 GB-2 TB | 2-10 TB | 10 TB+ |
|
||||
| Campaign recipients/month | under 5,000 | 5,000-100,000 | 100,000-1,000,000 | provider-limited or multi-tenant |
|
||||
| Workflow/import jobs/day | under 100 | 100-2,000 | 2,000-25,000 | queue-specific scaling |
|
||||
| Recovery point objective | daily backup | daily plus WAL/snapshots | tested restore, tighter RPO | formal SLO/SLA |
|
||||
| Process split requirement | optional | API plus worker | API, workers, scheduler split | horizontal replicas |
|
||||
|
||||
Hard constraints override the numeric thresholds:
|
||||
|
||||
- Multiple API replicas require PostgreSQL and shared storage.
|
||||
- Cross-node file access requires object storage or a shared durable file
|
||||
service, not node-local disk.
|
||||
- Async campaign send, append, imports, exports, or workflows require Redis and
|
||||
workers outside development.
|
||||
- Package install/uninstall in production requires maintenance mode, backup,
|
||||
restore-check hooks, and installer daemon visibility.
|
||||
- Autoscaling requires idempotent job claims, readiness probes, external
|
||||
throttling limits, and queue-specific maximum replica counts.
|
||||
|
||||
## Calculator Output Contract
|
||||
|
||||
The future UI calculator should emit a structured recommendation:
|
||||
|
||||
```json
|
||||
{
|
||||
"recommended_profile": "small-production",
|
||||
"minimum_components": [
|
||||
"PostgreSQL",
|
||||
"Redis",
|
||||
"API process",
|
||||
"worker process",
|
||||
"durable file storage"
|
||||
],
|
||||
"reasons": [
|
||||
"Campaign recipients/month exceed pilot threshold.",
|
||||
"Async mail delivery requires worker split."
|
||||
],
|
||||
"warnings": [
|
||||
"Object storage is recommended before adding a second API node."
|
||||
],
|
||||
"open_measurements": [
|
||||
"Peak concurrent users",
|
||||
"Database backup restore duration"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Until the calculator is implemented, operators should fill the worksheet
|
||||
manually and compare it with the Ops page's current profile, readiness checks,
|
||||
worker queues, and sizing assumptions.
|
||||
|
||||
## Minimum Production Requirements
|
||||
|
||||
A production deployment, even a small one, should have:
|
||||
|
||||
- PostgreSQL, explicit migrations, backups, and a tested restore path.
|
||||
- A stable `MASTER_KEY_B64` stored outside the repository.
|
||||
- HTTPS, exact CORS origins, secure cookies, and a reverse proxy.
|
||||
- Durable file storage with backup or object-store lifecycle policy.
|
||||
- Redis plus at least one worker when any queued module behavior is enabled.
|
||||
- Health/readiness checks visible in `govoplan-ops`.
|
||||
- Maintenance-mode access assigned to at least one operator account.
|
||||
- Module catalog trust roots and license trust roots pinned by deployment
|
||||
configuration.
|
||||
|
||||
## Manual Before Automatic
|
||||
|
||||
The first supported production scaling path is manual:
|
||||
|
||||
1. Move from SQLite/local storage to PostgreSQL and durable storage.
|
||||
2. Add workers and Redis for queue-backed operations.
|
||||
3. Split WebUI/API/worker processes.
|
||||
4. Add health checks and deployment profile warnings.
|
||||
5. Add metrics and queue-depth alerts.
|
||||
6. Scale API and worker replicas with fixed limits.
|
||||
7. Add autoscaling only after idempotency, readiness, and external throttles are
|
||||
understood.
|
||||
Reference in New Issue
Block a user