Fence managed file object effects

This commit is contained in:
2026-08-03 04:21:52 +02:00
parent b993d8e31a
commit 6baf2a421b
10 changed files with 1162 additions and 25 deletions
+7
View File
@@ -156,6 +156,13 @@ then returns a 30-minute tenant/user-bound preview token. Confirmation reuploads
the original archive and stores only the selected members. Password-protected the original archive and stores only the selected members. Password-protected
ZIP passwords remain request-only and are never included in the preview token. ZIP passwords remain request-only and are never included in the preview token.
Managed blob writes and applied orphan cleanup use Core's durable recovery
ledger. Intent, request digests, recovery mode, and a distributed lease are
committed before physical storage effects; the Files session commit verifies
database and streamed object evidence, while rollback compensates only a newly
reserved unreferenced key. New object keys are opaque and do not retain the
uploaded filename. Uncertain or mismatched effects remain visible through Ops.
Bulk rename and transfer APIs are owner-scoped: callers must provide the active Bulk rename and transfer APIs are owner-scoped: callers must provide the active
user or group file space with `owner_type` and `owner_id`. The storage layer user or group file space with `owner_type` and `owner_id`. The storage layer
keeps a named legacy file-only helper for historical callers that lack owner keeps a named legacy file-only helper for historical callers that lack owner
+52 -3
View File
@@ -517,6 +517,50 @@ dry-run by default, rechecks that no database reference exists, remains scoped
to the scanned tenant prefix, and is idempotent. Both applied and dry-run to the scanned tenant prefix, and is idempotent. Both applied and dry-run
actions emit audit evidence. actions emit audit evidence.
### Recovery ledger for object effects
Every managed blob creation or integrity repair starts a Core recovery
operation in an independent committed transaction before Files protects or
writes bytes. The operation records tenant/blob identifiers, an opaque object
locator or locator digest, semantic SHA-256/size evidence, the recovery mode,
and a distributed lease fence. It never records file contents, ZIP passwords,
connector credentials, or a newly uploaded filename. New object keys are opaque;
legacy filename-bearing keys remain readable but repair operations record only
their digest and recover through the blob ID.
The Files business transaction then creates or updates the blob, version, and
asset rows. Its actual SQLAlchemy commit or rollback settles every pending
operation:
- commit reloads the blob through an independent session and streams the object
to verify its stored-byte SHA-256 and size before recording success;
- rollback deletes only a newly reserved object after independently proving
that no `FileBlob` references it, then records verified compensation;
- a repaired existing object is forward-completed only when its identity,
envelope, semantic evidence, and stored bytes all match;
- missing or mismatched bytes quarantine a committed blob and leave the
operation `recovery_required`; an unavailable probe remains
`outcome_unknown` rather than becoming an ordinary upload failure.
Applied orphan cleanup has its own forward-recovery operation. The database
reference check and tenant-prefix check happen before deletion; object absence
and the durable finding state are verified afterward. If the caller transaction
rolls back after deletion, Files may forward-complete only that existing
finding after rechecking that the key is still unreferenced.
Archive preview and confirmation use bounded process-local temporary staging.
Staging is not authoritative and is removed on every handled exit; extracted
members enter the same per-blob recovery boundary as direct uploads. A hard
process loss may leave a temporary OS file for normal host temporary-file
cleanup, but cannot make that staging path a managed Files object.
Use the Ops recovery-operation view to inspect `files` operations. Do not retry
a busy or unresolved blob blindly: first verify the FileBlob row, object hash,
integrity state, and any Encryption envelope named by the blob. Hard purge,
legal hold, and two-way remote connector mutation are not implemented yet, so
they cannot claim recovery-ledger adoption; their owning work remains tracked
separately.
After restore: After restore:
1. Verify the active tenant and module migration state. 1. Verify the active tenant and module migration state.
@@ -644,6 +688,9 @@ Files baseline indiscriminately.
every protected blob; they are identical for unprotected blobs. every protected blob; they are identical for unprotected blobs.
- Upload and archive-confirm APIs can select an Encryption vault. Protected - Upload and archive-confirm APIs can select an Encryption vault. Protected
writes and reads fail closed if the optional Encryption capability is absent. writes and reads fail closed if the optional Encryption capability is absent.
- Managed-object writes and applied orphan cleanup start lease-fenced Core
recovery operations before their physical effects; terminal success and
compensation require independent database and object checks.
The module does **not** currently provide malware scanning, content disarm and The module does **not** currently provide malware scanning, content disarm and
reconstruction, a file-type allowlist, per-user quota, automatic encryption reconstruction, a file-type allowlist, per-user quota, automatic encryption
@@ -794,8 +841,8 @@ returning different content or credentials.
| Area | Implemented now | Planned or explicitly outside the current boundary | | Area | Implemented now | Planned or explicitly outside the current boundary |
| --- | --- | --- | | --- | --- | --- |
| Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, and dry-run-first orphan cleanup | Scheduled scan execution and deployment-specific S3 HA/backup automation | | Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, dry-run-first orphan cleanup, and Core-ledger verification/forward recovery | Scheduled scan execution and deployment-specific S3 HA/backup automation |
| Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload | | Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts, opaque new object keys, and rollback compensation | Malware scanning, quotas, type policy, resumable/chunked upload |
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore | | Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles | | Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) | | Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
@@ -829,7 +876,9 @@ Before releasing Files:
8. Verify credential deletion scrubs dependents and produces audit evidence. 8. Verify credential deletion scrubs dependents and produces audit evidence.
9. Verify a campaign attachment snapshot still identifies its exact version and 9. Verify a campaign attachment snapshot still identifies its exact version and
checksum after the current file changes. checksum after the current file changes.
10. Update the implemented/planned table whenever a boundary changes. 10. Exercise a committed upload, a rolled-back upload, object tamper detection,
and applied orphan cleanup; inspect their `files` operations in Ops.
11. Update the implemented/planned table whenever a boundary changes.
## Related documents ## Related documents
+7
View File
@@ -10,6 +10,13 @@ versioning, configured blob backend, and conflict behavior. An idempotency key
is represented as source provenance so an unchanged retry does not create an is represented as source provenance so an unchanged retry does not create an
unrelated file version. unrelated file version.
The shared Files session owns finalization. Before a new managed object is
written, Files commits a lease-fenced Core recovery operation containing only
identifiers and digests. The caller's eventual session commit independently
verifies both `FileBlob` metadata and stored bytes; rollback compensates only an
unreferenced key. Producers must therefore complete the supplied transaction
normally and must not bypass or replace Files session lifecycle handling.
The response contains only file/version identifiers, display path, media type, The response contains only file/version identifiers, display path, media type,
size, digest, and storage provenance. Producers must not put credentials, size, digest, and storage provenance. Producers must not put credentials,
tokens, or rendered plaintext into metadata. Storing an artifact proves Files tokens, or rendered plaintext into metadata. Storing an artifact proves Files
+3 -3
View File
@@ -583,7 +583,7 @@ manifest = ModuleManifest(
title="Operate Files integrity, recovery, and connector transport safety", title="Operate Files integrity, recovery, and connector transport safety",
summary="Back up database evidence, blob ciphertext, and Encryption custody as one recovery unit, and keep unsupported SDK transports fail-closed.", summary="Back up database evidence, blob ciphertext, and Encryption custody as one recovery unit, and keep unsupported SDK transports fail-closed.",
body=( body=(
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the matching Encryption tables and original deployment master key, then run the bounded resumable integrity scan and verify representative protected and unprotected access paths. Protected scans verify stored ciphertext before decryption and then verify plaintext semantic evidence. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. " "Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the matching Encryption tables and original deployment master key, then run the bounded resumable integrity scan and verify representative protected and unprotected access paths. Protected scans verify stored ciphertext before decryption and then verify plaintext semantic evidence. Managed blob creation/repair and applied orphan cleanup commit lease-fenced Core recovery intent before object effects; success, compensation, and forward completion require independent database and object checks, while mismatch is quarantined and unresolved work remains visible in Ops. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. "
"Arbitrary external S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Installer-owned Garage storage is supported only at the exact deployment service endpoint with its explicit trust marker. Destructive module retirement drops database tables but does not remove backend blob objects." "Arbitrary external S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Installer-owned Garage storage is supported only at the exact deployment service endpoint with its explicit trust marker. Destructive module retirement drops database tables but does not remove backend blob objects."
), ),
layer="configured", layer="configured",
@@ -631,7 +631,7 @@ manifest = ModuleManifest(
"screen": "System file connections and deployment operations", "screen": "System file connections and deployment operations",
"section": "Storage integrity, backup/recovery, and fail-closed transports", "section": "Storage integrity, backup/recovery, and fail-closed transports",
"recovery_unit": ["Files database rows", "Encryption envelope and wrapped-key rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"], "recovery_unit": ["Files database rows", "Encryption envelope and wrapped-key rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"],
"verification": "After restore, complete a checksum-enabled integrity scan, resolve every missing/corrupt finding, approve or retain every reported orphan, verify authorized and denied access, and test one permitted pinned HTTP connector.", "verification": "After restore, complete a checksum-enabled integrity scan, resolve every missing/corrupt finding, approve or retain every reported orphan, inspect Files recovery operations in Ops, verify authorized and denied access, and test one permitted pinned HTTP connector.",
"related_topic_ids": [ "related_topic_ids": [
"files.governed-connectors-and-provenance", "files.governed-connectors-and-provenance",
"files.reference.snapshot-provenance-and-capabilities", "files.reference.snapshot-provenance-and-capabilities",
@@ -874,7 +874,7 @@ manifest = ModuleManifest(
maturity="vertical_slice", maturity="vertical_slice",
documentation_ref="docs/FILES_HANDBOOK.md", documentation_ref="docs/FILES_HANDBOOK.md",
test_ref="tests/test_storage_backends.py", test_ref="tests/test_storage_backends.py",
known_limits=("Multi-node object-storage recovery evidence and every remote connector profile are not reference-ready.",), known_limits=("Target-environment multi-node recovery drills and writable remote connector effects are not reference-ready; hard purge and legal hold remain unimplemented.",),
supported_authority_modes=( supported_authority_modes=(
"native_authoritative", "native_authoritative",
"external_authoritative", "external_authoritative",
+47 -5
View File
@@ -20,8 +20,9 @@ from govoplan_files.backend.storage.backends import (
get_storage_backend, get_storage_backend,
) )
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
from govoplan_files.backend.storage.integrity import ( from govoplan_files.backend.storage.integrity import (
QUARANTINED_BLOB_STATUSES, QUARANTINED_BLOB_STATUSES,
read_verified_blob_bytes, read_verified_blob_bytes,
@@ -61,8 +62,9 @@ def _storage_backend_name() -> str:
return settings.file_storage_backend.lower().strip() return settings.file_storage_backend.lower().strip()
def _storage_key(*, tenant_id: str, checksum: str, filename: str) -> str: def _storage_key(*, tenant_id: str, checksum: str) -> str:
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}-{safe_storage_component(filename)}" # Object locators remain opaque so recovery evidence never persists names.
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}.blob"
def _get_or_create_blob( def _get_or_create_blob(
@@ -97,7 +99,30 @@ def _get_or_create_blob(
except StorageBackendError as exc: except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc raise FileStorageError(str(exc)) from exc
if repair_required: if repair_required:
repair_token = hashlib.sha256(
repr(
(
blob.integrity_status,
blob.integrity_checked_at,
blob.quarantined_at,
blob.storage_checksum_sha256,
)
).encode("utf-8")
).hexdigest()
recovery = begin_blob_write_recovery(
session,
backend=backend,
tenant_id=tenant_id,
blob_id=blob.id,
storage_key=blob.storage_key,
semantic_checksum_sha256=checksum,
semantic_size_bytes=size,
protection_discriminator=protection_discriminator,
created_new=False,
repair_token=repair_token,
)
stored_data = data stored_data = data
expected_envelope_id = blob.encryption_envelope_id
if vault_id: if vault_id:
from govoplan_files.backend.storage.content_protection import protect_blob_content from govoplan_files.backend.storage.content_protection import protect_blob_content
@@ -115,8 +140,13 @@ def _get_or_create_blob(
raise FileStorageError("The existing encrypted blob has another protection envelope.") raise FileStorageError("The existing encrypted blob has another protection envelope.")
stored_data = protected.ciphertext stored_data = protected.ciphertext
blob.encryption_envelope_id = protected.envelope.envelope_id blob.encryption_envelope_id = protected.envelope.envelope_id
expected_envelope_id = protected.envelope.envelope_id
blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest() blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest()
blob.storage_size_bytes = len(stored_data) blob.storage_size_bytes = len(stored_data)
recovery.prepare_stored_bytes(
stored_data,
envelope_id=expected_envelope_id,
)
try: try:
backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type) backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
except StorageBackendError as exc: except StorageBackendError as exc:
@@ -130,7 +160,19 @@ def _get_or_create_blob(
return blob return blob
blob_id = str(uuid4()) blob_id = str(uuid4())
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum, filename=filename) storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
backend = get_storage_backend()
recovery = begin_blob_write_recovery(
session,
backend=backend,
tenant_id=tenant_id,
blob_id=blob_id,
storage_key=storage_key,
semantic_checksum_sha256=checksum,
semantic_size_bytes=size,
protection_discriminator=protection_discriminator,
created_new=True,
)
stored_data = data stored_data = data
envelope_id = None envelope_id = None
if vault_id: if vault_id:
@@ -148,7 +190,7 @@ def _get_or_create_blob(
) )
stored_data = protected.ciphertext stored_data = protected.ciphertext
envelope_id = protected.envelope.envelope_id envelope_id = protected.envelope.envelope_id
backend = get_storage_backend() recovery.prepare_stored_bytes(stored_data, envelope_id=envelope_id)
try: try:
backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type) backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
except StorageBackendError as exc: except StorageBackendError as exc:
@@ -16,6 +16,9 @@ from govoplan_files.backend.storage.backends import (
StorageObjectMissing, StorageObjectMissing,
get_storage_backend, get_storage_backend,
) )
from govoplan_files.backend.storage.recovery import (
begin_orphan_cleanup_recovery,
)
from govoplan_files.backend.storage.common import FileStorageError, utcnow from govoplan_files.backend.storage.common import FileStorageError, utcnow
@@ -341,6 +344,12 @@ def cleanup_orphan_finding(
raise FileStorageError( raise FileStorageError(
"The configured storage backend does not match the integrity finding" "The configured storage backend does not match the integrity finding"
) )
begin_orphan_cleanup_recovery(
session,
finding,
backend=active_backend,
user_id=user_id,
)
try: try:
active_backend.stat(finding.storage_key) active_backend.stat(finding.storage_key)
except StorageObjectMissing: except StorageObjectMissing:
@@ -0,0 +1,691 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC, datetime
import hashlib
from typing import Protocol
from sqlalchemy import event
from sqlalchemy.orm import Session
from govoplan_core.core.recovery import (
RecoveryGuaranteeError,
RecoveryMode,
RecoveryPlan,
RecoveryStatus,
)
from govoplan_core.core.recovery_runtime import (
DurableRecoveryOperation,
RecoveryOperationBusy,
RecoveryOperationStateConflict,
begin_durable_recovery_operation,
)
from govoplan_core.core.runtime_coordination import process_runtime_identity
from govoplan_core.db.session import get_database
from govoplan_files.backend.db.models import (
FileBlob,
FileIntegrityFinding,
FileIntegrityScan,
)
from govoplan_files.backend.storage.backends import (
StorageBackend,
StorageBackendError,
StorageObjectMissing,
)
from govoplan_files.backend.storage.common import FileStorageError
_PENDING_EFFECTS_KEY = "govoplan_files_pending_recovery_effects"
_HOOKS_INSTALLED_KEY = "govoplan_files_recovery_hooks_installed"
_ROLLBACK_ERRORS_KEY = "govoplan_files_recovery_rollback_errors"
class _PendingEffect(Protocol):
def settle(self, *, committed: bool) -> None: ...
@dataclass(slots=True)
class PendingBlobWrite:
operation: DurableRecoveryOperation
backend: StorageBackend
tenant_id: str
blob_id: str
storage_key: str
semantic_checksum_sha256: str
semantic_size_bytes: int
protection_discriminator: str
created_new: bool
expected_storage_checksum_sha256: str | None = None
expected_storage_size_bytes: int | None = None
expected_envelope_id: str | None = None
def prepare_stored_bytes(
self,
data: bytes,
*,
envelope_id: str | None,
) -> None:
"""Retain process-local verification evidence without recording content."""
self.expected_storage_checksum_sha256 = hashlib.sha256(data).hexdigest()
self.expected_storage_size_bytes = len(data)
self.expected_envelope_id = envelope_id
def settle(self, *, committed: bool) -> None:
evidence = _blob_write_evidence(self)
if _blob_write_complete(evidence):
self.operation.succeed(evidence=evidence)
return
if self.created_new and evidence.get("database_blob_present") is False:
self._settle_unreferenced_new_object(evidence)
return
if not self.created_new and _object_matches(evidence):
if _forward_complete_blob_repair(self):
completed = _blob_write_evidence(self)
if _blob_write_complete(completed):
self.operation.succeed(evidence=completed)
return
if evidence.get("database_blob_present") is True and not _object_matches(
evidence
):
_quarantine_blob_after_failed_verification(self, evidence)
evidence = _blob_write_evidence(self)
status = (
RecoveryStatus.OUTCOME_UNKNOWN
if not evidence.get("verified")
else RecoveryStatus.RECOVERY_REQUIRED
)
transaction = "committed" if committed else "rolled back"
self.operation.unresolved(
status=status,
summary=f"Managed Files blob write remained unresolved after the database transaction {transaction}",
evidence=evidence,
failure_summary=(
"The managed object and Files blob metadata require reconciliation"
),
)
def _settle_unreferenced_new_object(self, evidence: dict[str, object]) -> None:
object_present = evidence.get("object_present")
if object_present is False:
self.operation.reject(
summary="The Files blob write left no durable object or database row",
evidence=evidence,
)
return
if object_present is not True:
self.operation.unresolved(
status=RecoveryStatus.OUTCOME_UNKNOWN,
summary="The unreferenced Files object could not be probed",
evidence=evidence,
failure_summary="Object storage availability prevented upload compensation",
)
return
try:
self.backend.delete(self.storage_key)
except (StorageBackendError, OSError):
self.operation.unresolved(
status=RecoveryStatus.RECOVERY_REQUIRED,
summary="An unreferenced Files object could not be compensated",
evidence=evidence,
failure_summary="Delete the unreferenced managed object after verifying that no FileBlob references it",
)
return
recovered = _blob_write_evidence(self)
if (
recovered.get("verified") is True
and recovered.get("database_blob_present") is False
and recovered.get("object_present") is False
):
self.operation.compensate(
failure_summary="The Files database transaction did not retain the new blob",
failure_evidence=evidence,
recovery_evidence=recovered,
)
return
self.operation.unresolved(
status=RecoveryStatus.RECOVERY_REQUIRED,
summary="Files upload compensation could not be verified",
evidence=recovered,
failure_summary="The unreferenced managed object requires operator reconciliation",
)
@dataclass(slots=True)
class PendingOrphanCleanup:
operation: DurableRecoveryOperation
backend: StorageBackend
finding_id: str
tenant_id: str
storage_key: str
resolved_by_user_id: str
def settle(self, *, committed: bool) -> None:
del committed
evidence = _orphan_cleanup_evidence(self)
if _orphan_cleanup_complete(evidence):
self.operation.succeed(evidence=evidence)
return
if (
evidence.get("verified") is True
and evidence.get("object_present") is True
and evidence.get("finding_deleted") is False
):
self.operation.reject(
summary="The orphan object was retained and the cleanup finding stayed open",
evidence=evidence,
)
return
if evidence.get("object_present") is False:
if _forward_complete_orphan_finding(self):
completed = _orphan_cleanup_evidence(self)
if _orphan_cleanup_complete(completed):
self.operation.succeed(evidence=completed)
return
status = (
RecoveryStatus.OUTCOME_UNKNOWN
if not evidence.get("verified")
else RecoveryStatus.RECOVERY_REQUIRED
)
self.operation.unresolved(
status=status,
summary="Files orphan cleanup requires reconciliation",
evidence=evidence,
failure_summary="Recheck the object and integrity-finding state before another cleanup attempt",
)
def begin_blob_write_recovery(
session: Session,
*,
backend: StorageBackend,
tenant_id: str,
blob_id: str,
storage_key: str,
semantic_checksum_sha256: str,
semantic_size_bytes: int,
protection_discriminator: str,
created_new: bool,
repair_token: str | None = None,
) -> PendingBlobWrite:
disposition = "create" if created_new else "repair"
state_token = repair_token or blob_id
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
idempotency_key = (
f"files-blob-{disposition}:{blob_id}:{state_token[:48]}"
)
try:
started = begin_durable_recovery_operation(
get_database().SessionLocal,
identity=process_runtime_identity(),
module_id="files",
operation_type=f"blob-{disposition}",
idempotency_key=idempotency_key,
request={
"tenant_id": tenant_id,
"blob_id": blob_id,
"storage_key": storage_key if created_new else None,
"storage_key_sha256": key_digest,
"semantic_checksum_sha256": semantic_checksum_sha256,
"semantic_size_bytes": semantic_size_bytes,
"protection_discriminator": protection_discriminator,
"disposition": disposition,
},
recovery_plan=RecoveryPlan(
mode=(
RecoveryMode.COMPENSATION
if created_new
else RecoveryMode.FORWARD_RECOVERY
),
preconditions=(
"the caller has Files write authority for the target owner",
"the storage key belongs to the tenant Files namespace",
"the request records digests rather than file contents",
),
compensation_steps=(
"verify that no FileBlob references the newly reserved key",
"delete only that unreferenced key and verify absence",
)
if created_new
else (),
forward_recovery_steps=(
"verify the expected bytes at the existing blob key",
"forward-complete matching integrity metadata or quarantine the blob",
)
if not created_new
else (),
verification_steps=(
"reload FileBlob metadata through an independent session",
"stream and hash the managed object independently",
),
),
precondition_evidence={
"blob_id": blob_id,
"storage_key_sha256": key_digest,
"semantic_checksum_sha256": semantic_checksum_sha256,
"semantic_size_bytes": semantic_size_bytes,
"created_new": created_new,
},
lease_resource_key=(
f"files:blob:{tenant_id}:{blob_id}"
),
lease_ttl_seconds=15 * 60,
resource_type="file_blob",
resource_id=blob_id,
metadata={
"resources": ["postgresql", "object-storage"],
"storage_backend": backend.name,
},
)
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
raise FileStorageError(
"This managed blob is already owned by another recovery operation"
) from exc
except (RecoveryGuaranteeError, RuntimeError) as exc:
raise FileStorageError(
"The Files recovery ledger is unavailable; no object was written"
) from exc
if started.replayed or started.operation is None:
raise FileStorageError(
"The matching Files blob operation was already completed; reload before retrying"
)
pending = PendingBlobWrite(
operation=started.operation,
backend=backend,
tenant_id=tenant_id,
blob_id=blob_id,
storage_key=storage_key,
semantic_checksum_sha256=semantic_checksum_sha256,
semantic_size_bytes=semantic_size_bytes,
protection_discriminator=protection_discriminator,
created_new=created_new,
)
_register_pending_effect(session, pending)
return pending
def begin_orphan_cleanup_recovery(
session: Session,
finding: FileIntegrityFinding,
*,
backend: StorageBackend,
user_id: str,
) -> PendingOrphanCleanup:
key_digest = hashlib.sha256(finding.storage_key.encode("utf-8")).hexdigest()
try:
started = begin_durable_recovery_operation(
get_database().SessionLocal,
identity=process_runtime_identity(),
module_id="files",
operation_type="integrity-orphan-cleanup",
idempotency_key=f"files-orphan-cleanup:{finding.id}",
request={
"tenant_id": finding.tenant_id,
"finding_id": finding.id,
"storage_key_sha256": key_digest,
},
recovery_plan=RecoveryPlan(
mode=RecoveryMode.FORWARD_RECOVERY,
preconditions=(
"the integrity finding identifies an unreferenced object",
"the object key remains inside the completed scan scope",
"a fresh database reference check found no FileBlob",
),
forward_recovery_steps=(
"verify object absence independently",
"mark the durable finding deleted only after absence is proven",
),
verification_steps=(
"reload the finding and its scan through an independent session",
"probe the original storage key through the configured backend",
),
),
precondition_evidence={
"finding_id": finding.id,
"scan_id": finding.scan_id,
"storage_key_sha256": key_digest,
"finding_state": finding.state,
},
lease_resource_key=(
f"files:orphan-cleanup:{finding.tenant_id}:{key_digest[:40]}"
),
lease_ttl_seconds=15 * 60,
resource_type="file_integrity_finding",
resource_id=finding.id,
metadata={
"resources": ["postgresql", "object-storage"],
"storage_backend": backend.name,
},
)
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
raise FileStorageError(
"This orphan object is already owned by another recovery operation"
) from exc
except (RecoveryGuaranteeError, RuntimeError) as exc:
raise FileStorageError(
"The Files recovery ledger is unavailable; no object was deleted"
) from exc
if started.replayed or started.operation is None:
raise FileStorageError(
"This orphan cleanup was already completed; reload the finding"
)
pending = PendingOrphanCleanup(
operation=started.operation,
backend=backend,
finding_id=finding.id,
tenant_id=finding.tenant_id,
storage_key=finding.storage_key,
resolved_by_user_id=user_id,
)
_register_pending_effect(session, pending)
return pending
def _register_pending_effect(session: Session, effect: _PendingEffect) -> None:
if not session.in_transaction():
session.begin()
pending = session.info.setdefault(_PENDING_EFFECTS_KEY, [])
pending.append(effect)
if session.info.get(_HOOKS_INSTALLED_KEY):
return
event.listen(session, "after_commit", _after_session_commit)
event.listen(session, "after_rollback", _after_session_rollback)
session.info[_HOOKS_INSTALLED_KEY] = True
def _after_session_commit(session: Session) -> None:
_settle_pending_effects(session, committed=True)
def _after_session_rollback(session: Session) -> None:
try:
_settle_pending_effects(session, committed=False)
except RecoveryGuaranteeError as exc:
session.info.setdefault(_ROLLBACK_ERRORS_KEY, []).append(str(exc))
def _settle_pending_effects(session: Session, *, committed: bool) -> None:
pending = list(session.info.pop(_PENDING_EFFECTS_KEY, []))
failures: list[Exception] = []
for effect in pending:
try:
effect.settle(committed=committed)
except Exception as exc: # preserve every effect's chance to settle
failures.append(exc)
operation = getattr(effect, "operation", None)
if operation is not None and not operation.closed:
try:
operation.release_unresolved()
except Exception:
pass
if failures:
raise RecoveryGuaranteeError(
f"{len(failures)} Files recovery operation(s) could not be finalized"
) from failures[0]
def _blob_write_evidence(effect: PendingBlobWrite) -> dict[str, object]:
database_blob_present: bool | None
database_matches: bool | None
database_integrity_verified: bool | None
try:
with effect.operation.session_factory() as session:
blob = session.get(FileBlob, effect.blob_id)
database_blob_present = blob is not None
database_matches = bool(
blob is not None
and blob.tenant_id == effect.tenant_id
and blob.storage_key == effect.storage_key
and blob.checksum_sha256 == effect.semantic_checksum_sha256
and blob.size_bytes == effect.semantic_size_bytes
and blob.protection_discriminator
== effect.protection_discriminator
and blob.encryption_envelope_id == effect.expected_envelope_id
and (
effect.expected_storage_checksum_sha256
== (
blob.storage_checksum_sha256
or blob.checksum_sha256
)
)
) if blob is not None else False
database_integrity_verified = bool(
blob is not None
and blob.integrity_status == "verified"
and blob.quarantined_at is None
) if blob is not None else False
except Exception:
database_blob_present = None
database_matches = None
database_integrity_verified = None
object_present: bool | None
observed_size: int | None = None
observed_checksum: str | None = None
try:
digest = hashlib.sha256()
observed_size = 0
for chunk in effect.backend.iter_bytes(effect.storage_key):
observed_size += len(chunk)
digest.update(chunk)
observed_checksum = digest.hexdigest()
object_present = True
except StorageObjectMissing:
object_present = False
except (StorageBackendError, OSError):
object_present = None
object_size_matches = (
observed_size == effect.expected_storage_size_bytes
if object_present is True and effect.expected_storage_size_bytes is not None
else False if object_present is False else None
)
object_checksum_matches = (
observed_checksum == effect.expected_storage_checksum_sha256
if object_present is True
and effect.expected_storage_checksum_sha256 is not None
else False if object_present is False else None
)
verified = database_blob_present is not None and object_present is not None
return {
"verified": verified,
"checks": {
"database_reloaded": database_blob_present is not None,
"object_probed": object_present is not None,
"database_matches_request": database_matches,
"database_integrity_verified": database_integrity_verified,
"object_size_matches": object_size_matches,
"object_checksum_matches": object_checksum_matches,
},
"blob_id": effect.blob_id,
"database_blob_present": database_blob_present,
"database_matches_request": database_matches,
"database_integrity_verified": database_integrity_verified,
"object_present": object_present,
"observed_size_bytes": observed_size,
"observed_checksum_sha256": observed_checksum,
"expected_storage_size_bytes": effect.expected_storage_size_bytes,
"expected_storage_checksum_sha256": effect.expected_storage_checksum_sha256,
}
def _blob_write_complete(evidence: dict[str, object]) -> bool:
return bool(
evidence.get("verified") is True
and evidence.get("database_matches_request") is True
and evidence.get("database_integrity_verified") is True
and _object_matches(evidence)
)
def _object_matches(evidence: dict[str, object]) -> bool:
checks = evidence.get("checks")
return bool(
isinstance(checks, dict)
and evidence.get("object_present") is True
and checks.get("object_size_matches") is True
and checks.get("object_checksum_matches") is True
)
def _forward_complete_blob_repair(effect: PendingBlobWrite) -> bool:
try:
with effect.operation.session_factory() as session:
blob = session.get(FileBlob, effect.blob_id)
if (
blob is None
or blob.tenant_id != effect.tenant_id
or blob.storage_key != effect.storage_key
or blob.checksum_sha256 != effect.semantic_checksum_sha256
or blob.size_bytes != effect.semantic_size_bytes
or blob.protection_discriminator
!= effect.protection_discriminator
or blob.encryption_envelope_id != effect.expected_envelope_id
):
return False
blob.storage_checksum_sha256 = (
effect.expected_storage_checksum_sha256
if effect.expected_envelope_id
else None
)
blob.storage_size_bytes = (
effect.expected_storage_size_bytes
if effect.expected_envelope_id
else None
)
blob.integrity_status = "verified"
blob.integrity_checked_at = datetime.now(UTC)
blob.integrity_failure = None
blob.quarantined_at = None
session.add(blob)
session.commit()
return True
except Exception:
return False
def _quarantine_blob_after_failed_verification(
effect: PendingBlobWrite,
evidence: dict[str, object],
) -> None:
try:
with effect.operation.session_factory() as session:
blob = session.get(FileBlob, effect.blob_id)
if blob is None or blob.storage_key != effect.storage_key:
return
blob.integrity_status = (
"missing"
if evidence.get("object_present") is False
else "checksum_mismatch"
)
blob.integrity_checked_at = datetime.now(UTC)
blob.integrity_failure = "recovery_verification_failed"
blob.quarantined_at = datetime.now(UTC)
session.add(blob)
session.commit()
except Exception:
return
def _orphan_cleanup_evidence(effect: PendingOrphanCleanup) -> dict[str, object]:
try:
object_present: bool | None = effect.backend.exists(effect.storage_key)
except (StorageBackendError, OSError):
object_present = None
finding_present: bool | None
finding_deleted: bool | None
still_unreferenced: bool | None
try:
with effect.operation.session_factory() as session:
finding = session.get(FileIntegrityFinding, effect.finding_id)
finding_present = finding is not None
finding_deleted = bool(
finding is not None and finding.state == "deleted"
) if finding is not None else False
referenced = (
session.query(FileBlob.id)
.filter(
FileBlob.tenant_id == effect.tenant_id,
FileBlob.storage_key == effect.storage_key,
)
.first()
)
still_unreferenced = referenced is None
except Exception:
finding_present = None
finding_deleted = None
still_unreferenced = None
verified = object_present is not None and finding_present is not None
return {
"verified": verified,
"checks": {
"database_reloaded": finding_present is not None,
"object_probed": object_present is not None,
"finding_marked_deleted": finding_deleted,
"object_absent": (
not object_present if object_present is not None else None
),
"still_unreferenced": still_unreferenced,
},
"finding_id": effect.finding_id,
"finding_present": finding_present,
"finding_deleted": finding_deleted,
"object_present": object_present,
"still_unreferenced": still_unreferenced,
}
def _orphan_cleanup_complete(evidence: dict[str, object]) -> bool:
return bool(
evidence.get("verified") is True
and evidence.get("finding_deleted") is True
and evidence.get("object_present") is False
and evidence.get("still_unreferenced") is True
)
def _forward_complete_orphan_finding(effect: PendingOrphanCleanup) -> bool:
try:
with effect.operation.session_factory() as session:
finding = session.get(FileIntegrityFinding, effect.finding_id)
if (
finding is None
or finding.tenant_id != effect.tenant_id
or finding.storage_key != effect.storage_key
):
return False
scan = session.get(FileIntegrityScan, finding.scan_id)
if scan is None or not finding.storage_key.startswith(
scan.storage_prefix
):
return False
referenced = (
session.query(FileBlob.id)
.filter(
FileBlob.tenant_id == effect.tenant_id,
FileBlob.storage_key == effect.storage_key,
)
.first()
)
if referenced:
return False
finding.state = "deleted"
finding.resolved_at = datetime.now(UTC)
finding.resolved_by_user_id = effect.resolved_by_user_id
session.add(finding)
session.commit()
return True
except Exception:
return False
__all__ = [
"PendingBlobWrite",
"PendingOrphanCleanup",
"begin_blob_write_recovery",
"begin_orphan_cleanup_recovery",
]
+4
View File
@@ -4,6 +4,7 @@ import hashlib
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest.mock import patch
from sqlalchemy import create_engine from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import sessionmaker
@@ -160,6 +161,9 @@ class IntegrityReconciliationTests(unittest.TestCase):
) )
self.assertEqual("would_delete", preview_cleanup.action) self.assertEqual("would_delete", preview_cleanup.action)
self.assertTrue(self.backend.exists(orphan_key)) self.assertTrue(self.backend.exists(orphan_key))
with patch(
"govoplan_files.backend.storage.integrity.begin_orphan_cleanup_recovery"
):
cleanup = cleanup_orphan_finding( cleanup = cleanup_orphan_finding(
self.session, self.session,
orphan_finding, orphan_finding,
+2
View File
@@ -160,6 +160,8 @@ class FilesManifestDocumentationTests(unittest.TestCase):
self.assertIn("quarantined", topic.body) self.assertIn("quarantined", topic.body)
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"]) self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
self.assertIn("Encryption envelope and wrapped-key rows", topic.metadata["recovery_unit"]) self.assertIn("Encryption envelope and wrapped-key rows", topic.metadata["recovery_unit"])
self.assertIn("lease-fenced Core recovery", topic.body)
self.assertIn("Ops", topic.body)
self.assertTrue(topic.metadata["verification"]) self.assertTrue(topic.metadata["verification"])
self.assertIn( self.assertIn(
"/api/v1/files/integrity/scans", "/api/v1/files/integrity/scans",
+326
View File
@@ -0,0 +1,326 @@
from __future__ import annotations
import hashlib
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db.models import Account, Group, User
from govoplan_core.core.recovery import (
RecoveryCheckpoint,
RecoveryOperation,
RecoveryStatus,
)
from govoplan_core.core.runtime_coordination import (
DistributedLease,
RuntimeIdentity,
bind_process_runtime_identity,
)
from govoplan_core.db.base import Base
from govoplan_core.db.session import configure_database, reset_database
from govoplan_files.backend.db.models import (
FileBlob,
FileIntegrityFinding,
FileIntegrityScan,
)
from govoplan_files.backend.storage.backends import (
LocalFilesystemStorageBackend,
)
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import _get_or_create_blob
from govoplan_files.backend.storage.integrity import cleanup_orphan_finding
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
TENANT_ID = "tenant-1"
USER_ID = "user-1"
class StorageRecoveryTests(unittest.TestCase):
def setUp(self) -> None:
self.temporary_directory = tempfile.TemporaryDirectory()
self.addCleanup(self.temporary_directory.cleanup)
root = Path(self.temporary_directory.name)
self.backend = LocalFilesystemStorageBackend(root / "objects")
database_path = root / "recovery.sqlite3"
self.engine = create_engine(f"sqlite:///{database_path}", future=True)
Base.metadata.create_all(
bind=self.engine,
tables=[
Account.__table__,
User.__table__,
Group.__table__,
DistributedLease.__table__,
RecoveryOperation.__table__,
RecoveryCheckpoint.__table__,
FileBlob.__table__,
FileIntegrityScan.__table__,
FileIntegrityFinding.__table__,
],
)
configure_database(
f"sqlite:///{database_path}",
engine=self.engine,
dispose_previous=True,
)
bind_process_runtime_identity(
RuntimeIdentity(
installation_id="files-recovery-test",
node_id="node-1",
incarnation="incarnation-1",
role="api",
software_version="test",
composition_hash="a" * 64,
)
)
self.Session = sessionmaker(
bind=self.engine,
expire_on_commit=False,
future=True,
)
self.enterContext(
patch(
"govoplan_files.backend.storage.files._storage_backend_name",
return_value=self.backend.name,
)
)
self.enterContext(
patch(
"govoplan_files.backend.storage.files._storage_bucket_name",
return_value="",
)
)
self.session = self.Session()
self.addCleanup(self._cleanup_runtime)
def _cleanup_runtime(self) -> None:
self.session.close()
bind_process_runtime_identity(None)
reset_database()
self.engine.dispose()
def test_committed_blob_write_is_independently_verified(self) -> None:
observed_running_operation: list[bool] = []
put_bytes = self.backend.put_bytes
class ObservingBackend:
name = self.backend.name
def __getattr__(backend_self, name):
return getattr(self.backend, name)
def put_bytes(backend_self, key, data, *, content_type=None):
with self.Session() as evidence_session:
operation = evidence_session.query(RecoveryOperation).one()
observed_running_operation.append(
operation.status == RecoveryStatus.RUNNING.value
and len(operation.request_sha256) == 64
)
put_bytes(key, data, content_type=content_type)
observing_backend = ObservingBackend()
with patch(
"govoplan_files.backend.storage.files.get_storage_backend",
return_value=observing_backend,
):
blob = _get_or_create_blob(
self.session,
tenant_id=TENANT_ID,
data=b"durable",
filename="private-name.txt",
content_type="text/plain",
actor_id=USER_ID,
)
self.session.commit()
operation = self._only_operation()
self.assertEqual(RecoveryStatus.SUCCEEDED.value, operation.status)
self.assertEqual([True], observed_running_operation)
self.assertTrue(self.backend.exists(blob.storage_key))
self.assertNotIn("private-name", blob.storage_key)
self.assertEqual(".blob", Path(blob.storage_key).suffix)
def test_rolled_back_blob_write_is_compensated(self) -> None:
with patch(
"govoplan_files.backend.storage.files.get_storage_backend",
return_value=self.backend,
):
blob = _get_or_create_blob(
self.session,
tenant_id=TENANT_ID,
data=b"rollback",
filename="rollback.txt",
content_type="text/plain",
actor_id=USER_ID,
)
storage_key = blob.storage_key
self.assertTrue(self.backend.exists(storage_key))
self.session.rollback()
operation = self._only_operation()
self.assertEqual(RecoveryStatus.RECOVERED.value, operation.status)
self.assertFalse(self.backend.exists(storage_key))
with self.Session() as evidence_session:
self.assertIsNone(evidence_session.get(FileBlob, blob.id))
def test_post_write_tamper_is_quarantined_and_recovery_required(self) -> None:
with patch(
"govoplan_files.backend.storage.files.get_storage_backend",
return_value=self.backend,
):
blob = _get_or_create_blob(
self.session,
tenant_id=TENANT_ID,
data=b"expected",
filename="evidence.bin",
content_type="application/octet-stream",
actor_id=USER_ID,
)
self.backend.put_bytes(blob.storage_key, b"tampered")
self.session.commit()
operation = self._only_operation()
self.assertEqual(
RecoveryStatus.RECOVERY_REQUIRED.value,
operation.status,
)
with self.Session() as evidence_session:
persisted = evidence_session.get(FileBlob, blob.id)
self.assertIsNotNone(persisted)
self.assertEqual("checksum_mismatch", persisted.integrity_status)
self.assertIsNotNone(persisted.quarantined_at)
def test_missing_optional_encryption_fails_before_object_effect(self) -> None:
with patch(
"govoplan_files.backend.storage.files.get_storage_backend",
return_value=self.backend,
), patch(
"govoplan_files.backend.storage.content_protection.encryption_content_cipher",
return_value=None,
), self.assertRaisesRegex(FileStorageError, "Encryption module"):
_get_or_create_blob(
self.session,
tenant_id=TENANT_ID,
data=b"protected",
filename="protected.bin",
content_type="application/octet-stream",
actor_id=USER_ID,
encryption_vault_id="vault-1",
)
self.session.rollback()
operation = self._only_operation()
self.assertEqual(RecoveryStatus.REJECTED.value, operation.status)
objects = self.backend.list_objects(
prefix=f"tenants/{TENANT_ID}/files/",
limit=10,
)
self.assertEqual((), objects.objects)
def test_orphan_cleanup_forward_completes_after_business_rollback(self) -> None:
key = f"tenants/{TENANT_ID}/files/orphan.bin"
self.backend.put_bytes(key, b"orphan")
scan = FileIntegrityScan(
id="scan-1",
tenant_id=TENANT_ID,
storage_backend=self.backend.name,
storage_prefix=f"tenants/{TENANT_ID}/files/",
status="completed",
)
finding = FileIntegrityFinding(
id="finding-1",
scan_id=scan.id,
tenant_id=TENANT_ID,
kind="orphan_object",
state="open",
storage_key=key,
observed_size_bytes=6,
observed_checksum_sha256=hashlib.sha256(b"orphan").hexdigest(),
)
self.session.add_all([scan, finding])
self.session.commit()
cleanup_orphan_finding(
self.session,
finding,
user_id=USER_ID,
dry_run=False,
backend=self.backend,
)
self.assertFalse(self.backend.exists(key))
self.session.rollback()
operation = self._only_operation()
self.assertEqual(RecoveryStatus.SUCCEEDED.value, operation.status)
with self.Session() as evidence_session:
persisted = evidence_session.get(FileIntegrityFinding, finding.id)
self.assertIsNotNone(persisted)
self.assertEqual("deleted", persisted.state)
def test_missing_runtime_identity_blocks_before_object_write(self) -> None:
bind_process_runtime_identity(None)
with patch(
"govoplan_files.backend.storage.files.get_storage_backend",
return_value=self.backend,
), self.assertRaisesRegex(FileStorageError, "recovery ledger"):
_get_or_create_blob(
self.session,
tenant_id=TENANT_ID,
data=b"blocked",
filename="blocked.bin",
content_type="application/octet-stream",
actor_id=USER_ID,
)
self.session.rollback()
objects = self.backend.list_objects(
prefix=f"tenants/{TENANT_ID}/files/",
limit=10,
)
self.assertEqual((), objects.objects)
def test_blob_fence_blocks_a_competing_runtime_before_effect(self) -> None:
checksum = hashlib.sha256(b"fenced").hexdigest()
begin_blob_write_recovery(
self.session,
backend=self.backend,
tenant_id=TENANT_ID,
blob_id="blob-fenced",
storage_key=f"tenants/{TENANT_ID}/files/fenced.blob",
semantic_checksum_sha256=checksum,
semantic_size_bytes=6,
protection_discriminator="plaintext",
created_new=True,
)
with self.Session() as competing_session, self.assertRaisesRegex(
FileStorageError,
"already owned",
):
begin_blob_write_recovery(
competing_session,
backend=self.backend,
tenant_id=TENANT_ID,
blob_id="blob-fenced",
storage_key=f"tenants/{TENANT_ID}/files/fenced.blob",
semantic_checksum_sha256=checksum,
semantic_size_bytes=6,
protection_discriminator="plaintext",
created_new=True,
)
self.session.rollback()
self.assertEqual(RecoveryStatus.REJECTED.value, self._only_operation().status)
def _only_operation(self) -> RecoveryOperation:
with self.Session() as session:
operations = session.query(RecoveryOperation).all()
self.assertEqual(1, len(operations))
session.expunge(operations[0])
return operations[0]
if __name__ == "__main__":
unittest.main()