diff --git a/docs/SCALABILITY_PROFILES.md b/docs/SCALABILITY_PROFILES.md index 1424219..4cef3be 100644 --- a/docs/SCALABILITY_PROFILES.md +++ b/docs/SCALABILITY_PROFILES.md @@ -216,6 +216,15 @@ Ops ledger row. Run a complete Files integrity scan after such a loss and reconcile reported orphans before resuming writes. PostgreSQL retains the independent pre-effect recovery-intent guarantee required for production. +Files hard-purge and S3 connector-write operations use independently durable +Core recovery records and distributed fences. A purge releases database +references before a separate, reference-checked blob-GC operation deletes +bytes. An S3 write records only an opaque target digest plus request/content +evidence and verifies provider metadata after its conditional effect. When Ops +shows `outcome_unknown` or `recovery_required`, do not replay the action from +Ops: reconcile the exact blob reference/object or S3 request/content markers +through the owning Files workflow first. + Production migrations should run explicitly before startup or package activation. Module install/uninstall workflows must use database backup and restore-check hooks for PostgreSQL before migrations or destructive retirement. diff --git a/src/govoplan_ops/backend/manifest.py b/src/govoplan_ops/backend/manifest.py index c66069d..89cd055 100644 --- a/src/govoplan_ops/backend/manifest.py +++ b/src/govoplan_ops/backend/manifest.py @@ -165,7 +165,7 @@ manifest = ModuleManifest( id="ops.runtime-coordination-and-recovery", title="Drain runtime nodes and inspect recovery evidence", summary="Ops projects shared runtime heartbeats, replica gaps, drain controls, and recovery states that require operator attention.", - body="Use the runtime table to identify stale or composition-skewed API and worker replicas. Drain before replacement so API readiness closes and workers stop taking new queue work; cancellation is available while the node is still draining. The recovery table reports durable Core recovery operations. A rejected operation is a verified provider rejection and needs no recovery; outcome-unknown and recovery-required operations still require reconciliation through the owning module. Core module-lifecycle entries block every later install or live graph change: use the installer run id to verify package, backup, migration, and health evidence before rollback or forward repair. Mail SMTP and IMAP APPEND entries use stable attempt identifiers and digest-only evidence: reconcile the Mail command from provider evidence, never by replaying the original effect from Ops. Files object effects use independently committed recovery intent on PostgreSQL. Development SQLite can show only handled-rollback reconstruction because its Files intent shares the caller transaction; after a hard SQLite process loss, run the owning Files integrity scan because an orphan may have no Ops ledger row. Dataflow database-only runs are atomic, while published-output runs use forward recovery: reconcile the recorded output digest and sink idempotency key before allowing another publication. Backup status separately projects only the sanitized deployment verification receipt: a verified status identifies a coordinated recovery point and isolated restore drill, while absent, expired, or invalid evidence blocks a release-changing migration.", + body="Use the runtime table to identify stale or composition-skewed API and worker replicas. Drain before replacement so API readiness closes and workers stop taking new queue work; cancellation is available while the node is still draining. The recovery table reports durable Core recovery operations. A rejected operation is a verified provider rejection and needs no recovery; outcome-unknown and recovery-required operations still require reconciliation through the owning module. Core module-lifecycle entries block every later install or live graph change: use the installer run id to verify package, backup, migration, and health evidence before rollback or forward repair. Mail SMTP and IMAP APPEND entries use stable attempt identifiers and digest-only evidence: reconcile the Mail command from provider evidence, never by replaying the original effect from Ops. Files blob writes, hard purge, reference-checked garbage collection, and conditional S3 connector writes record Core recovery evidence. For a Files connector outcome, inspect the provider request/content markers and revision before allowing another write to the fenced path; for blob GC, recheck FileVersion references and exact object absence. Development SQLite can show only handled-rollback reconstruction for caller-transaction blob uploads; after a hard SQLite process loss, run the owning Files integrity scan because an orphan may have no Ops ledger row. Dataflow database-only runs are atomic, while published-output runs use forward recovery: reconcile the recorded output digest and sink idempotency key before allowing another publication. Backup status separately projects only the sanitized deployment verification receipt: a verified status identifies a coordinated recovery point and isolated restore drill, while absent, expired, or invalid evidence blocks a release-changing migration.", documentation_types=("admin", "user"), audience=("operator", "system_admin"), conditions=( @@ -183,6 +183,7 @@ manifest = ModuleManifest( "Compare active non-stale nodes with the configured API and worker replica expectations.", "Request drain and wait for the node to report draining before replacing it.", "Inspect every recovery-required, outcome-unknown, or manual-intervention operation and follow its recorded recovery mode.", + "For Files, distinguish blob upload/repair, approved hard purge, blob garbage collection, and S3 connector write-back before checking the owning database/object or provider evidence.", "Confirm that backup evidence is verified and current before authorizing a release-changing migration.", "Verify replacement composition, readiness, queue consumers, and recovery evidence before closing the operation.", ],