feat: expose institutional architecture and runtime state

docs/SCALABILITY_PROFILES.md

@@ -15,6 +15,12 @@ live Ops page.
    campaigns, imports, exports, or workflow automation.
 6. Record the current profile and open measurements before moving to a larger
    topology.
+7. In a replicated profile, compare active non-stale API and worker counts with
+   configured expectations, and resolve composition skew before rollout.
+8. Drain a node before replacement, then verify it is no longer ready/consuming
+   before terminating it.
+9. Inspect recovery-required and outcome-unknown operations; follow the recorded
+   recovery mode rather than retrying or rolling back blindly.
 
 ## Live Profile Signals
 
@@ -30,10 +36,24 @@ The Ops API reports:
 - HTTP/certificate deployment posture through the `deployment_security` check
 - readiness blockers
 - reference deployment profiles and sizing assumptions
+- runtime node identity, role, software/module composition, queues, heartbeat,
+  stale state, and drain state
+- configured versus active API and worker replica counts
+- recovery operation status, mode, checkpoint count, and last update
 
 These values are intentionally diagnostic. They do not replace deployment
 configuration management, backups, monitoring, or restore drills.
 
+Drain controls are cooperative. API and worker processes observe the request on
+their next heartbeat. API readiness then fails; workers cancel their configured
+queue consumers but may still be finishing already claimed work. Confirm the
+process state and queue evidence before forcefully terminating a node.
+
+Recovery evidence is similarly conservative. A forward-recovery or
+manual-intervention record means an operator must repair the current release or
+restore a separately verified coordinated backup. Ops does not convert that
+state into a safe rollback.
+
 `deployment_security` is inactive for local/test profiles. In staging or pilot
 profiles it warns when secure cookies or CORS origins still look local. In
 production it becomes readiness-critical because TLS certificates, proxy
@@ -93,23 +113,26 @@ Stateless and horizontally replicable:
 - API workers when `MASTER_KEY_B64`, `DATABASE_URL`, storage, queue, and module
   configuration are shared
 - Background workers when queues and idempotency keys are used
-- Scheduler replicas only when leader election or an external lock exists
+- Scheduler processes only through Core's renewable PostgreSQL lease and
+  fencing-token runner; deploy one desired scheduler replica
 
 Stateful or singleton-sensitive:
 
 - PostgreSQL
-- local file storage when not replaced by object storage
+- local file storage when not replaced by object storage (or a one-host shared
+  volume under the `host-shared` profile)
 - Redis/queue state
 - module installer daemon and package mutation operations
 - migration execution
-- scheduler without distributed locking
+- migration execution, although competing jobs are serialized by a PostgreSQL
+  advisory lock
 - outgoing campaign append/send jobs unless claim tokens are enforced
 
 ## Readiness And Degraded Modes
 
 | Component | Ready When | Degraded Mode |
 | --- | --- | --- |
-| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood | Read-only/admin-only where routes allow it; otherwise fail closed. |
+| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood, and the node is not draining | Read-only/admin-only where routes allow it; otherwise fail closed. |
 | WebUI | Static assets match backend module metadata contract | Show unavailable modules/routes with reason; do not invent routes. |
 | PostgreSQL | Accepts connections and migration head is current | Block writes and package changes if migration state is unknown. |
 | Storage | Configured backend is reachable and writable for write flows | Read-only file views may continue if storage is read-only but reachable. |