29 lines
1.6 KiB
Markdown
29 lines
1.6 KiB
Markdown
# Durable Recovery Operations
|
|
|
|
Modules must use `begin_durable_recovery_operation` for work whose effects can
|
|
outlive the caller's SQLAlchemy transaction. The helper commits the canonical
|
|
request hash, recovery plan, precondition evidence, running state, and lease
|
|
fence before the caller mutates object storage, a queue, a filesystem, or an
|
|
external provider.
|
|
|
|
Each later checkpoint is written through an independent database session. A
|
|
business-transaction rollback therefore cannot erase evidence of an earlier
|
|
effect. Successful completion requires concrete verification checks and a valid
|
|
hash chain. Compensation likewise records recovery-required, recovering, and
|
|
verified-recovered checkpoints rather than reporting an ordinary failure.
|
|
A definitive pre-effect or provider rejection records terminal `rejected`
|
|
evidence instead of being mislabeled as success, atomic rollback, or recovery
|
|
work.
|
|
|
|
If a runtime disappears, another runtime may claim the operation only after the
|
|
lease expires. The takeover records both fences. A stale compensatable operation
|
|
becomes recovery-required; a stale forward-only or irreversible external effect
|
|
becomes outcome-unknown; a database-only atomic operation is recorded failed
|
|
because its transaction rolled back. Takeover never re-executes the original
|
|
request automatically.
|
|
|
|
Evidence and metadata may contain opaque references, digests, counts, and
|
|
provider result codes. They must never contain credentials or resolved secrets.
|
|
Ops is the platform surface for unresolved operation status; owning modules must
|
|
provide the reconciliation action and business-level explanation.
|