2.6 KiB
Durable Recovery Operations
Modules must use begin_durable_recovery_operation for work whose effects can
outlive the caller's SQLAlchemy transaction. The helper commits the canonical
request hash, recovery plan, precondition evidence, running state, and lease
fence before the caller mutates object storage, a queue, a filesystem, or an
external provider.
Each later checkpoint is written through an independent database session. A
business-transaction rollback therefore cannot erase evidence of an earlier
effect. Successful completion requires concrete verification checks and a valid
hash chain. Compensation likewise records recovery-required, recovering, and
verified-recovered checkpoints rather than reporting an ordinary failure.
A definitive pre-effect or provider rejection records terminal rejected
evidence instead of being mislabeled as success, atomic rollback, or recovery
work.
If a runtime disappears, another runtime may claim the operation only after the lease expires. The takeover records both fences. A stale compensatable operation becomes recovery-required; a stale forward-only or irreversible external effect becomes outcome-unknown; a database-only atomic operation is recorded failed because its transaction rolled back. Takeover never re-executes the original request automatically.
Evidence and metadata may contain opaque references, digests, counts, and provider result codes. They must never contain credentials or resolved secrets. Ops is the platform surface for unresolved operation status; owning modules must provide the reconciliation action and business-level explanation.
Database-only operations must use the durable handle's atomic terminal methods
when their module rows and final recovery checkpoint belong to one invariant.
Those methods stage the terminal checkpoint and lease release in the caller's
SQLAlchemy transaction, then commit the domain rows and recovery evidence
together. A failed commit rolls both back and leaves the previously durable
running record available for stale-fence handling; modules must not commit
their domain state first and close an atomic recovery record afterwards.
An owning module may reconcile an outcome_unknown provider effect through the
claimed durable handle's resolve_unknown method. External evidence that the
effect occurred records verified success. Evidence that it did not occur moves
the operation through recovery-required and recovering to verified recovered,
so any later attempt must use a new deliberate idempotency key. The method does
not infer provider state and requires the same terminal verification structure
and hash-chain checks as ordinary completion.