State And Recovery Contract

State Profiles

Core accepts three runtime state profiles:

Profile	Runtime placement	Durable storage
`local`	One development process set	Local filesystem is permitted.
`host-shared`	Replicas on one host	One host-shared volume plus PostgreSQL and Redis.
`shared`	Independent hosts	PostgreSQL, Redis, and S3-compatible object storage.

All replicas in one installation use one stable GOVOPLAN_INSTALLATION_ID, one immutable module composition, and identical database, broker, encryption-key, and object-storage bindings. Core rejects replicas with the local profile and rejects shared without PostgreSQL, Redis, S3, and a non-default installation identifier.

FILE_STORAGE_S3_ENDPOINT_TRUSTED=true is an explicit deployment trust declaration for a clean HTTPS S3 origin. It does not authorize a user-controlled connector endpoint and it is separate from installer-managed Garage's exact endpoint trust.

Object Storage

govoplan_core.core.object_storage is the shared backend contract for durable module artifacts. It provides bounded read/write/list/stat/delete operations for local and S3-compatible storage. Modules own their object-key namespace and business metadata; Core does not interpret module files.

Rules for modules:

Store only opaque object keys in business records, never local absolute paths.
Use node-local directories only for temporary materialization.
Verify expected size and digest before consuming consequential artifacts.
If object creation precedes database commit, compensate successfully created objects on failure.
If object deletion fails, retain the database reference and report a retryable failure rather than claiming deletion.
Define an orphan-inventory strategy for hard process loss between object creation and metadata commit.

The Files module delegates its backend implementation to this Core contract. Campaign generated EML artifacts use a Campaign-owned object prefix and are read by workers through the same shared backend.

Runtime Nodes And Leases

API and worker incarnations register in core_runtime_nodes with role, software version, module-composition hash, queues, start time, and heartbeat. The registration identity includes a process incarnation so a stale process cannot update a replacement's row.

Drain is durable operator intent:

an API enters not-ready state after observing drain;
a worker cancels queue consumers after observing drain;
cancellation returns an eligible draining node to active state;
clean shutdown marks the matching incarnation stopped.

Coordination loss also fails closed. An API reports coordination_unavailable from /health/ready until a heartbeat succeeds again. A worker cancels its local queue consumers on any heartbeat or database failure and only resumes them after its existing incarnation heartbeats successfully. It never re-registers from the heartbeat path, so a stale worker cannot reclaim a node identity from its replacement.

core_distributed_leases provides installation/resource uniqueness, expiry, holder incarnation, and monotonically increasing fencing tokens. Lease expiry does not make an old process harmless by itself: code performing an effect must assert its exact fence before commit. govoplan_core.commands.fenced_run renews a lease around a subprocess and terminates the child when the lease is lost. The deployment profiles use it for the singleton scheduler.

Migration Ordering

Only govoplan_core.commands.init_db mutates schema. On PostgreSQL it acquires a deterministic installation/track advisory lock before pre-migration tasks, Alembic, and post-migration tasks. The lock is session-scoped and therefore released if the migration process dies.

Runtime roles run govoplan_core.commands.wait_for_database. It waits until the database has exactly the configured Core/module Alembic heads and never upgrades schema. This permits a migration Job and runtime Deployments to be submitted together while keeping startup fail-closed.

Recovery Ledger

govoplan_core.core.recovery provides a durable operation and evidence contract. Recovery modes are:

atomic: one database transaction, no external effect;
compensation: explicit inverse actions;
snapshot_restore: separately verified backup reference;
forward_recovery: repair/resume the current version;
irreversible: explicit approval, no automated recovery claim.

Every plan requires verification steps. Mode-specific evidence is mandatory. Operations bind an idempotency key to a canonical request hash, may bind a runtime fencing token, and emit append-only hash-chained checkpoints. Backup and approval references are part of the hashed plan evidence. Every low-level state transition and checkpoint append revalidates the operation's recorded fence while holding the operation row lock. Plaintext secrets are rejected from metadata and evidence.

The state machine makes partial and uncertain outcomes visible. A non-atomic running operation cannot transition directly to ordinary failure, and success or recovery requires explicit verified checks. Ops projects states requiring attention, but module behavior gains this guarantee only after it adopts the ledger around its own side effects.

Recovery Boundary

Application/configuration rollback and database rollback are not equivalent. Once an incompatible migration starts, old code may be unsafe even if its image is available. Deployment automation must switch to forward recovery unless a coordinated and verified database/object/key backup is restored.

Core does not create production database backups. The deployment owner must provide backup, retention, encryption, restore verification, and recovery-point coordination for PostgreSQL, object storage, and encryption keys. The canonical operator procedure is documented in govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md.

Verification

Focused contracts are covered by:

/mnt/DATA/git/govoplan/.venv/bin/python -m pytest -q \
  tests/test_object_storage.py \
  tests/test_runtime_coordination.py \
  tests/test_runtime_agents.py \
  tests/test_fenced_run.py \
  tests/test_migration_lock.py \
  tests/test_wait_for_database.py \
  tests/test_recovery_guarantees.py

Production acceptance additionally requires multi-node failure and coordinated restore drills against the actual PostgreSQL, Redis, object-store, ingress, and secret-provider topology.

6.4 KiB Raw Blame History