1.4 KiB
Durable Recovery Operations
Modules must use begin_durable_recovery_operation for work whose effects can
outlive the caller's SQLAlchemy transaction. The helper commits the canonical
request hash, recovery plan, precondition evidence, running state, and lease
fence before the caller mutates object storage, a queue, a filesystem, or an
external provider.
Each later checkpoint is written through an independent database session. A business-transaction rollback therefore cannot erase evidence of an earlier effect. Successful completion requires concrete verification checks and a valid hash chain. Compensation likewise records recovery-required, recovering, and verified-recovered checkpoints rather than reporting an ordinary failure.
If a runtime disappears, another runtime may claim the operation only after the lease expires. The takeover records both fences. A stale compensatable operation becomes recovery-required; a stale forward-only or irreversible external effect becomes outcome-unknown; a database-only atomic operation is recorded failed because its transaction rolled back. Takeover never re-executes the original request automatically.
Evidence and metadata may contain opaque references, digests, counts, and provider result codes. They must never contain credentials or resolved secrets. Ops is the platform surface for unresolved operation status; owning modules must provide the reconciliation action and business-level explanation.