2.1 KiB
Durable Recovery Operations
Modules must use begin_durable_recovery_operation for work whose effects can
outlive the caller's SQLAlchemy transaction. The helper commits the canonical
request hash, recovery plan, precondition evidence, running state, and lease
fence before the caller mutates object storage, a queue, a filesystem, or an
external provider.
Each later checkpoint is written through an independent database session. A
business-transaction rollback therefore cannot erase evidence of an earlier
effect. Successful completion requires concrete verification checks and a valid
hash chain. Compensation likewise records recovery-required, recovering, and
verified-recovered checkpoints rather than reporting an ordinary failure.
A definitive pre-effect or provider rejection records terminal rejected
evidence instead of being mislabeled as success, atomic rollback, or recovery
work.
If a runtime disappears, another runtime may claim the operation only after the lease expires. The takeover records both fences. A stale compensatable operation becomes recovery-required; a stale forward-only or irreversible external effect becomes outcome-unknown; a database-only atomic operation is recorded failed because its transaction rolled back. Takeover never re-executes the original request automatically.
Evidence and metadata may contain opaque references, digests, counts, and provider result codes. They must never contain credentials or resolved secrets. Ops is the platform surface for unresolved operation status; owning modules must provide the reconciliation action and business-level explanation.
An owning module may reconcile an outcome_unknown provider effect through the
claimed durable handle's resolve_unknown method. External evidence that the
effect occurred records verified success. Evidence that it did not occur moves
the operation through recovery-required and recovering to verified recovered,
so any later attempt must use a new deliberate idempotency key. The method does
not infer provider state and requires the same terminal verification structure
and hash-chain checks as ordinary completion.