[Debt] Adopt the Core recovery ledger for Files object and connector effects #41

New Issue

2026-08-03T02:11:44+02:00

zemion commented

2026-08-03 02:11:44 +02:00

Objective

Record Files operations that span database state, managed object storage, archive staging, or writable remote connectors through the Core recovery ledger.

Acceptance criteria

Upload/finalization, deletion/purge, archive extraction, integrity repair, and two-way connector writes have explicit recovery modes.
Idempotency and request hashes are durable before object or remote effects.
Shared-resource operations use distributed lease fencing.
Partial and outcome-unknown effects remain visible and are reconciled instead of being reported as ordinary failures.
Crash, retry, stale-fence, tamper, and optional-module tests cover the adopted paths.
Files and Ops documentation describe operator recovery.

Parent: GovOPlaN/govoplan#36. ## Objective Record Files operations that span database state, managed object storage, archive staging, or writable remote connectors through the Core recovery ledger. ## Acceptance criteria - [ ] Upload/finalization, deletion/purge, archive extraction, integrity repair, and two-way connector writes have explicit recovery modes. - [ ] Idempotency and request hashes are durable before object or remote effects. - [ ] Shared-resource operations use distributed lease fencing. - [ ] Partial and outcome-unknown effects remain visible and are reconciled instead of being reported as ordinary failures. - [ ] Crash, retry, stale-fence, tamper, and optional-module tests cover the adopted paths. - [ ] Files and Ops documentation describe operator recovery.

zemion added the

codex/ready labels 2026-08-03 02:11:44 +02:00

zemion commented

2026-08-03 04:22:53 +02:00

Codex State: progress

Files now adopts the Core recovery ledger for every implemented managed-object mutation.

govoplan-files@6baf2a4 starts durable request/digest evidence and a distributed lease before blob creation or repair, settles operations from the owning SQLAlchemy commit/rollback, independently streams and hashes stored bytes, compensates only unreferenced reserved keys, quarantines mismatches, and forward-completes verified existing-blob repairs.
Applied orphan cleanup now uses a separate forward-recovery operation and can complete its durable finding after a caller rollback only after renewed reference and object checks.
New object locators are opaque; protected-content/module absence fails before object effects; unresolved and outcome-unknown states remain visible in Ops.
govoplan@5bef966 records files.upload.finalize and files.integrity.reconcile as adopted.

Verification: Files 104 tests passed; Ruff and diff checks passed. The cross-module matrix run 308 also passed.

This issue remains open for two future effects that are deliberately not claimed as implemented: governed hard purge/blob GC (#38) and writable/two-way remote connector mutation. Current connector sync is a read/import snapshot and never mutates the provider.

## Codex State: progress Files now adopts the Core recovery ledger for every implemented managed-object mutation. - `govoplan-files@6baf2a4` starts durable request/digest evidence and a distributed lease before blob creation or repair, settles operations from the owning SQLAlchemy commit/rollback, independently streams and hashes stored bytes, compensates only unreferenced reserved keys, quarantines mismatches, and forward-completes verified existing-blob repairs. - Applied orphan cleanup now uses a separate forward-recovery operation and can complete its durable finding after a caller rollback only after renewed reference and object checks. - New object locators are opaque; protected-content/module absence fails before object effects; unresolved and outcome-unknown states remain visible in Ops. - `govoplan@5bef966` records `files.upload.finalize` and `files.integrity.reconcile` as adopted. Verification: Files `104 tests` passed; Ruff and diff checks passed. The cross-module matrix run 308 also passed. This issue remains open for two future effects that are deliberately not claimed as implemented: governed hard purge/blob GC (`#38`) and writable/two-way remote connector mutation. Current connector sync is a read/import snapshot and never mutates the provider.

zemion commented

2026-08-07 15:18:23 +02:00

Reproduced during the 2026-08-07 whole-Core discovery run after the new module work was otherwise green.

On SQLite, Files upload/archive code holds the request transaction while begin_blob_write_recovery() opens get_database().SessionLocal and attempts to acquire the distributed lease in a second write transaction. SQLite serializes writers, so the inner INSERT ... ON CONFLICT DO NOTHING waits and ends with sqlite3.OperationalError: database is locked.

Reproducing tests:

test_api_smoke.ApiSmokeTests.test_files_and_folders_support_cursor_windows
test_api_smoke.ApiSmokeTests.test_zip_upload_spools_archive_instead_of_full_buffering

This does not reproduce in the PostgreSQL-focused matrix, but it matters for supported local/dev SQLite operation. The fix should define an explicit SQLite recovery mode rather than merely increasing the timeout: either use the caller transaction with documented reduced crash durability, or move the durable claim before any caller write while preserving the PostgreSQL independent-transaction guarantee. Add direct single-file and archive-upload regression coverage.

Reproduced during the 2026-08-07 whole-Core discovery run after the new module work was otherwise green. On SQLite, Files upload/archive code holds the request transaction while `begin_blob_write_recovery()` opens `get_database().SessionLocal` and attempts to acquire the distributed lease in a second write transaction. SQLite serializes writers, so the inner `INSERT ... ON CONFLICT DO NOTHING` waits and ends with `sqlite3.OperationalError: database is locked`. Reproducing tests: - `test_api_smoke.ApiSmokeTests.test_files_and_folders_support_cursor_windows` - `test_api_smoke.ApiSmokeTests.test_zip_upload_spools_archive_instead_of_full_buffering` This does not reproduce in the PostgreSQL-focused matrix, but it matters for supported local/dev SQLite operation. The fix should define an explicit SQLite recovery mode rather than merely increasing the timeout: either use the caller transaction with documented reduced crash durability, or move the durable claim before any caller write while preserving the PostgreSQL independent-transaction guarantee. Add direct single-file and archive-upload regression coverage.

zemion commented

2026-08-19 19:23:47 +02:00

Progress in ad55d47: Files now has an explicit SQLite caller-transaction recovery mode instead of opening a deadlocking second writer during multi-file and archive uploads. It retains ledger request hashes/checkpoints, DB lease evidence, a process-local competing-effect fence, independent post-commit verification, and reconstructs durable compensation/forward-recovery evidence after handled rollback. PostgreSQL keeps the independent pre-effect commit guarantee. Files and Ops docs state the reduced hard-crash guarantee and required integrity scan. Verification: Files 136 tests, direct multi-blob recovery coverage, both former Core API lock regressions, full Core discovery (578 tests), Ruff, and manifest validation are green. This issue remains open for its already-recorded hard-purge and writable remote-connector acceptance scope.

Progress in `ad55d47`: Files now has an explicit SQLite caller-transaction recovery mode instead of opening a deadlocking second writer during multi-file and archive uploads. It retains ledger request hashes/checkpoints, DB lease evidence, a process-local competing-effect fence, independent post-commit verification, and reconstructs durable compensation/forward-recovery evidence after handled rollback. PostgreSQL keeps the independent pre-effect commit guarantee. Files and Ops docs state the reduced hard-crash guarantee and required integrity scan. Verification: Files 136 tests, direct multi-blob recovery coverage, both former Core API lock regressions, full Core discovery (578 tests), Ruff, and manifest validation are green. This issue remains open for its already-recorded hard-purge and writable remote-connector acceptance scope.

zemion commented

2026-08-20 22:10:25 +02:00

Completed in govoplan-files@6c3cf1c, building on the previously recorded upload/archive/integrity adoption; Ops guidance is updated in govoplan-ops@399e1f8.

The remaining purge/blob-GC and S3 write-back effects now create durable Core recovery operations before external effects, bind distributed resource fences, use canonical request hashes and idempotency, preserve partial/outcome-unknown/recovery-required states, and require verified evidence before success. Blob GC also shares its row/lease serialization boundary with upload and repair paths.

Verification: Files Ruff and 154 tests passed, including retry, stale fence/conflict, tamper, failure, optional-module, and outcome-unknown paths. The full workspace focused gate passed: backend suites, manifest/inventory checks, 59 WebUI permutations, bundle budgets, and 7 Playwright conformance tests.

Completed in `govoplan-files@6c3cf1c`, building on the previously recorded upload/archive/integrity adoption; Ops guidance is updated in `govoplan-ops@399e1f8`. The remaining purge/blob-GC and S3 write-back effects now create durable Core recovery operations before external effects, bind distributed resource fences, use canonical request hashes and idempotency, preserve partial/outcome-unknown/recovery-required states, and require verified evidence before success. Blob GC also shares its row/lease serialization boundary with upload and repair paths. Verification: Files Ruff and 154 tests passed, including retry, stale fence/conflict, tamper, failure, optional-module, and outcome-unknown paths. The full workspace focused gate passed: backend suites, manifest/inventory checks, 59 WebUI permutations, bundle budgets, and 7 Playwright conformance tests.

zemion closed this issue

2026-08-20 22:10:26 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-files#41