# State And Recovery Contract

## State Profiles

Core accepts three runtime state profiles:

| Profile | Runtime placement | Durable storage |
| --- | --- | --- |
| `local` | One development process set | Local filesystem is permitted. |
| `host-shared` | Replicas on one host | One host-shared volume plus PostgreSQL and Redis. |
| `shared` | Independent hosts | PostgreSQL, Redis, and S3-compatible object storage. |

All replicas in one installation use one stable
`GOVOPLAN_INSTALLATION_ID`, one immutable module composition, and identical
database, broker, encryption-key, and object-storage bindings. Core rejects
replicas with the `local` profile and rejects `shared` without PostgreSQL,
Redis, S3, and a non-default installation identifier.

`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true` is an explicit deployment trust
declaration for a clean HTTPS S3 origin. It does not authorize a
user-controlled connector endpoint and it is separate from installer-managed
Garage's exact endpoint trust.

## Object Storage

`govoplan_core.core.object_storage` is the shared backend contract for durable
module artifacts. It provides bounded read/write/list/stat/delete operations
for local and S3-compatible storage. Modules own their object-key namespace and
business metadata; Core does not interpret module files.

Rules for modules:

- Store only opaque object keys in business records, never local absolute
  paths.
- Use node-local directories only for temporary materialization.
- Verify expected size and digest before consuming consequential artifacts.
- If object creation precedes database commit, compensate successfully created
  objects on failure.
- If object deletion fails, retain the database reference and report a retryable
  failure rather than claiming deletion.
- Define an orphan-inventory strategy for hard process loss between object
  creation and metadata commit.

The Files module delegates its backend implementation to this Core contract.
Campaign generated EML artifacts use a Campaign-owned object prefix and are
read by workers through the same shared backend.

## Runtime Nodes And Leases

API and worker incarnations register in `core_runtime_nodes` with role,
software version, module-composition hash, queues, start time, and heartbeat.
The registration identity includes a process incarnation so a stale process
cannot update a replacement's row.

Drain is durable operator intent:

- an API enters not-ready state after observing drain;
- a worker cancels queue consumers after observing drain;
- cancellation returns an eligible draining node to active state;
- clean shutdown marks the matching incarnation stopped.

Coordination loss also fails closed. An API reports
`coordination_unavailable` from `/health/ready` until a heartbeat succeeds
again. A worker cancels its local queue consumers on any heartbeat or database
failure and only resumes them after its existing incarnation heartbeats
successfully. It never re-registers from the heartbeat path, so a stale worker
cannot reclaim a node identity from its replacement.

`core_distributed_leases` provides installation/resource uniqueness, expiry,
holder incarnation, and monotonically increasing fencing tokens. Lease expiry
does not make an old process harmless by itself: code performing an effect must
assert its exact fence before commit. `govoplan_core.commands.fenced_run`
renews a lease around a subprocess and terminates the child when the lease is
lost. The deployment profiles use it for the singleton scheduler.

## Migration Ordering

Only `govoplan_core.commands.init_db` mutates schema. On PostgreSQL it acquires
a deterministic installation/track advisory lock before pre-migration tasks,
Alembic, and post-migration tasks. The lock is session-scoped and therefore
released if the migration process dies.

Runtime roles run `govoplan_core.commands.wait_for_database`. It waits until
the database has exactly the configured Core/module Alembic heads and never
upgrades schema. This permits a migration Job and runtime Deployments to be
submitted together while keeping startup fail-closed.

## Recovery Ledger

`govoplan_core.core.recovery` provides a durable operation and evidence
contract. Recovery modes are:

- `atomic`: one database transaction, no external effect;
- `compensation`: explicit inverse actions;
- `snapshot_restore`: separately verified backup reference;
- `forward_recovery`: repair/resume the current version;
- `irreversible`: explicit approval, no automated recovery claim.

Every plan requires verification steps. Mode-specific evidence is mandatory.
Operations bind an idempotency key to a canonical request hash, may bind a
runtime fencing token, and emit append-only hash-chained checkpoints. Backup
and approval references are part of the hashed plan evidence. Every low-level
state transition and checkpoint append revalidates the operation's recorded
fence while holding the operation row lock. Plaintext secrets are rejected
from metadata and evidence.

The state machine makes partial and uncertain outcomes visible. A non-atomic
running operation cannot transition directly to ordinary failure, and success
or recovery requires explicit verified checks. Ops projects states requiring
attention, but module behavior gains this guarantee only after it adopts the
ledger around its own side effects.

## Recovery Boundary

Application/configuration rollback and database rollback are not equivalent.
Once an incompatible migration starts, old code may be unsafe even if its image
is available. Deployment automation must switch to forward recovery unless a
coordinated and verified database/object/key backup is restored.

Core does not create production database backups. The deployment owner must
provide backup, retention, encryption, restore verification, and recovery-point
coordination for PostgreSQL, object storage, and encryption keys. The canonical
operator procedure is documented in
`govoplan/docs/RECOVERY_AND_ROLLBACK_GUARANTEES.md`.

## Verification

Focused contracts are covered by:

```sh
/mnt/DATA/git/govoplan/.venv/bin/python -m pytest -q \
  tests/test_object_storage.py \
  tests/test_runtime_coordination.py \
  tests/test_runtime_agents.py \
  tests/test_fenced_run.py \
  tests/test_migration_lock.py \
  tests/test_wait_for_database.py \
  tests/test_recovery_guarantees.py
```

Production acceptance additionally requires multi-node failure and coordinated
restore drills against the actual PostgreSQL, Redis, object-store, ingress, and
secret-provider topology.