Fence managed file object effects
This commit is contained in:
@@ -156,6 +156,13 @@ then returns a 30-minute tenant/user-bound preview token. Confirmation reuploads
|
|||||||
the original archive and stores only the selected members. Password-protected
|
the original archive and stores only the selected members. Password-protected
|
||||||
ZIP passwords remain request-only and are never included in the preview token.
|
ZIP passwords remain request-only and are never included in the preview token.
|
||||||
|
|
||||||
|
Managed blob writes and applied orphan cleanup use Core's durable recovery
|
||||||
|
ledger. Intent, request digests, recovery mode, and a distributed lease are
|
||||||
|
committed before physical storage effects; the Files session commit verifies
|
||||||
|
database and streamed object evidence, while rollback compensates only a newly
|
||||||
|
reserved unreferenced key. New object keys are opaque and do not retain the
|
||||||
|
uploaded filename. Uncertain or mismatched effects remain visible through Ops.
|
||||||
|
|
||||||
Bulk rename and transfer APIs are owner-scoped: callers must provide the active
|
Bulk rename and transfer APIs are owner-scoped: callers must provide the active
|
||||||
user or group file space with `owner_type` and `owner_id`. The storage layer
|
user or group file space with `owner_type` and `owner_id`. The storage layer
|
||||||
keeps a named legacy file-only helper for historical callers that lack owner
|
keeps a named legacy file-only helper for historical callers that lack owner
|
||||||
|
|||||||
+52
-3
@@ -517,6 +517,50 @@ dry-run by default, rechecks that no database reference exists, remains scoped
|
|||||||
to the scanned tenant prefix, and is idempotent. Both applied and dry-run
|
to the scanned tenant prefix, and is idempotent. Both applied and dry-run
|
||||||
actions emit audit evidence.
|
actions emit audit evidence.
|
||||||
|
|
||||||
|
### Recovery ledger for object effects
|
||||||
|
|
||||||
|
Every managed blob creation or integrity repair starts a Core recovery
|
||||||
|
operation in an independent committed transaction before Files protects or
|
||||||
|
writes bytes. The operation records tenant/blob identifiers, an opaque object
|
||||||
|
locator or locator digest, semantic SHA-256/size evidence, the recovery mode,
|
||||||
|
and a distributed lease fence. It never records file contents, ZIP passwords,
|
||||||
|
connector credentials, or a newly uploaded filename. New object keys are opaque;
|
||||||
|
legacy filename-bearing keys remain readable but repair operations record only
|
||||||
|
their digest and recover through the blob ID.
|
||||||
|
|
||||||
|
The Files business transaction then creates or updates the blob, version, and
|
||||||
|
asset rows. Its actual SQLAlchemy commit or rollback settles every pending
|
||||||
|
operation:
|
||||||
|
|
||||||
|
- commit reloads the blob through an independent session and streams the object
|
||||||
|
to verify its stored-byte SHA-256 and size before recording success;
|
||||||
|
- rollback deletes only a newly reserved object after independently proving
|
||||||
|
that no `FileBlob` references it, then records verified compensation;
|
||||||
|
- a repaired existing object is forward-completed only when its identity,
|
||||||
|
envelope, semantic evidence, and stored bytes all match;
|
||||||
|
- missing or mismatched bytes quarantine a committed blob and leave the
|
||||||
|
operation `recovery_required`; an unavailable probe remains
|
||||||
|
`outcome_unknown` rather than becoming an ordinary upload failure.
|
||||||
|
|
||||||
|
Applied orphan cleanup has its own forward-recovery operation. The database
|
||||||
|
reference check and tenant-prefix check happen before deletion; object absence
|
||||||
|
and the durable finding state are verified afterward. If the caller transaction
|
||||||
|
rolls back after deletion, Files may forward-complete only that existing
|
||||||
|
finding after rechecking that the key is still unreferenced.
|
||||||
|
|
||||||
|
Archive preview and confirmation use bounded process-local temporary staging.
|
||||||
|
Staging is not authoritative and is removed on every handled exit; extracted
|
||||||
|
members enter the same per-blob recovery boundary as direct uploads. A hard
|
||||||
|
process loss may leave a temporary OS file for normal host temporary-file
|
||||||
|
cleanup, but cannot make that staging path a managed Files object.
|
||||||
|
|
||||||
|
Use the Ops recovery-operation view to inspect `files` operations. Do not retry
|
||||||
|
a busy or unresolved blob blindly: first verify the FileBlob row, object hash,
|
||||||
|
integrity state, and any Encryption envelope named by the blob. Hard purge,
|
||||||
|
legal hold, and two-way remote connector mutation are not implemented yet, so
|
||||||
|
they cannot claim recovery-ledger adoption; their owning work remains tracked
|
||||||
|
separately.
|
||||||
|
|
||||||
After restore:
|
After restore:
|
||||||
|
|
||||||
1. Verify the active tenant and module migration state.
|
1. Verify the active tenant and module migration state.
|
||||||
@@ -644,6 +688,9 @@ Files baseline indiscriminately.
|
|||||||
every protected blob; they are identical for unprotected blobs.
|
every protected blob; they are identical for unprotected blobs.
|
||||||
- Upload and archive-confirm APIs can select an Encryption vault. Protected
|
- Upload and archive-confirm APIs can select an Encryption vault. Protected
|
||||||
writes and reads fail closed if the optional Encryption capability is absent.
|
writes and reads fail closed if the optional Encryption capability is absent.
|
||||||
|
- Managed-object writes and applied orphan cleanup start lease-fenced Core
|
||||||
|
recovery operations before their physical effects; terminal success and
|
||||||
|
compensation require independent database and object checks.
|
||||||
|
|
||||||
The module does **not** currently provide malware scanning, content disarm and
|
The module does **not** currently provide malware scanning, content disarm and
|
||||||
reconstruction, a file-type allowlist, per-user quota, automatic encryption
|
reconstruction, a file-type allowlist, per-user quota, automatic encryption
|
||||||
@@ -794,8 +841,8 @@ returning different content or credentials.
|
|||||||
|
|
||||||
| Area | Implemented now | Planned or explicitly outside the current boundary |
|
| Area | Implemented now | Planned or explicitly outside the current boundary |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, and dry-run-first orphan cleanup | Scheduled scan execution and deployment-specific S3 HA/backup automation |
|
| Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, dry-run-first orphan cleanup, and Core-ledger verification/forward recovery | Scheduled scan execution and deployment-specific S3 HA/backup automation |
|
||||||
| Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload |
|
| Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts, opaque new object keys, and rollback compensation | Malware scanning, quotas, type policy, resumable/chunked upload |
|
||||||
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
|
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
|
||||||
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
|
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
|
||||||
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
|
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
|
||||||
@@ -829,7 +876,9 @@ Before releasing Files:
|
|||||||
8. Verify credential deletion scrubs dependents and produces audit evidence.
|
8. Verify credential deletion scrubs dependents and produces audit evidence.
|
||||||
9. Verify a campaign attachment snapshot still identifies its exact version and
|
9. Verify a campaign attachment snapshot still identifies its exact version and
|
||||||
checksum after the current file changes.
|
checksum after the current file changes.
|
||||||
10. Update the implemented/planned table whenever a boundary changes.
|
10. Exercise a committed upload, a rolled-back upload, object tamper detection,
|
||||||
|
and applied orphan cleanup; inspect their `files` operations in Ops.
|
||||||
|
11. Update the implemented/planned table whenever a boundary changes.
|
||||||
|
|
||||||
## Related documents
|
## Related documents
|
||||||
|
|
||||||
|
|||||||
@@ -10,6 +10,13 @@ versioning, configured blob backend, and conflict behavior. An idempotency key
|
|||||||
is represented as source provenance so an unchanged retry does not create an
|
is represented as source provenance so an unchanged retry does not create an
|
||||||
unrelated file version.
|
unrelated file version.
|
||||||
|
|
||||||
|
The shared Files session owns finalization. Before a new managed object is
|
||||||
|
written, Files commits a lease-fenced Core recovery operation containing only
|
||||||
|
identifiers and digests. The caller's eventual session commit independently
|
||||||
|
verifies both `FileBlob` metadata and stored bytes; rollback compensates only an
|
||||||
|
unreferenced key. Producers must therefore complete the supplied transaction
|
||||||
|
normally and must not bypass or replace Files session lifecycle handling.
|
||||||
|
|
||||||
The response contains only file/version identifiers, display path, media type,
|
The response contains only file/version identifiers, display path, media type,
|
||||||
size, digest, and storage provenance. Producers must not put credentials,
|
size, digest, and storage provenance. Producers must not put credentials,
|
||||||
tokens, or rendered plaintext into metadata. Storing an artifact proves Files
|
tokens, or rendered plaintext into metadata. Storing an artifact proves Files
|
||||||
|
|||||||
@@ -583,7 +583,7 @@ manifest = ModuleManifest(
|
|||||||
title="Operate Files integrity, recovery, and connector transport safety",
|
title="Operate Files integrity, recovery, and connector transport safety",
|
||||||
summary="Back up database evidence, blob ciphertext, and Encryption custody as one recovery unit, and keep unsupported SDK transports fail-closed.",
|
summary="Back up database evidence, blob ciphertext, and Encryption custody as one recovery unit, and keep unsupported SDK transports fail-closed.",
|
||||||
body=(
|
body=(
|
||||||
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the matching Encryption tables and original deployment master key, then run the bounded resumable integrity scan and verify representative protected and unprotected access paths. Protected scans verify stored ciphertext before decryption and then verify plaintext semantic evidence. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. "
|
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the matching Encryption tables and original deployment master key, then run the bounded resumable integrity scan and verify representative protected and unprotected access paths. Protected scans verify stored ciphertext before decryption and then verify plaintext semantic evidence. Managed blob creation/repair and applied orphan cleanup commit lease-fenced Core recovery intent before object effects; success, compensation, and forward completion require independent database and object checks, while mismatch is quarantined and unresolved work remains visible in Ops. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. "
|
||||||
"Arbitrary external S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Installer-owned Garage storage is supported only at the exact deployment service endpoint with its explicit trust marker. Destructive module retirement drops database tables but does not remove backend blob objects."
|
"Arbitrary external S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Installer-owned Garage storage is supported only at the exact deployment service endpoint with its explicit trust marker. Destructive module retirement drops database tables but does not remove backend blob objects."
|
||||||
),
|
),
|
||||||
layer="configured",
|
layer="configured",
|
||||||
@@ -631,7 +631,7 @@ manifest = ModuleManifest(
|
|||||||
"screen": "System file connections and deployment operations",
|
"screen": "System file connections and deployment operations",
|
||||||
"section": "Storage integrity, backup/recovery, and fail-closed transports",
|
"section": "Storage integrity, backup/recovery, and fail-closed transports",
|
||||||
"recovery_unit": ["Files database rows", "Encryption envelope and wrapped-key rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"],
|
"recovery_unit": ["Files database rows", "Encryption envelope and wrapped-key rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"],
|
||||||
"verification": "After restore, complete a checksum-enabled integrity scan, resolve every missing/corrupt finding, approve or retain every reported orphan, verify authorized and denied access, and test one permitted pinned HTTP connector.",
|
"verification": "After restore, complete a checksum-enabled integrity scan, resolve every missing/corrupt finding, approve or retain every reported orphan, inspect Files recovery operations in Ops, verify authorized and denied access, and test one permitted pinned HTTP connector.",
|
||||||
"related_topic_ids": [
|
"related_topic_ids": [
|
||||||
"files.governed-connectors-and-provenance",
|
"files.governed-connectors-and-provenance",
|
||||||
"files.reference.snapshot-provenance-and-capabilities",
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
@@ -874,7 +874,7 @@ manifest = ModuleManifest(
|
|||||||
maturity="vertical_slice",
|
maturity="vertical_slice",
|
||||||
documentation_ref="docs/FILES_HANDBOOK.md",
|
documentation_ref="docs/FILES_HANDBOOK.md",
|
||||||
test_ref="tests/test_storage_backends.py",
|
test_ref="tests/test_storage_backends.py",
|
||||||
known_limits=("Multi-node object-storage recovery evidence and every remote connector profile are not reference-ready.",),
|
known_limits=("Target-environment multi-node recovery drills and writable remote connector effects are not reference-ready; hard purge and legal hold remain unimplemented.",),
|
||||||
supported_authority_modes=(
|
supported_authority_modes=(
|
||||||
"native_authoritative",
|
"native_authoritative",
|
||||||
"external_authoritative",
|
"external_authoritative",
|
||||||
|
|||||||
@@ -20,8 +20,9 @@ from govoplan_files.backend.storage.backends import (
|
|||||||
get_storage_backend,
|
get_storage_backend,
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
||||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component
|
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
|
||||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||||
|
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||||
from govoplan_files.backend.storage.integrity import (
|
from govoplan_files.backend.storage.integrity import (
|
||||||
QUARANTINED_BLOB_STATUSES,
|
QUARANTINED_BLOB_STATUSES,
|
||||||
read_verified_blob_bytes,
|
read_verified_blob_bytes,
|
||||||
@@ -61,8 +62,9 @@ def _storage_backend_name() -> str:
|
|||||||
return settings.file_storage_backend.lower().strip()
|
return settings.file_storage_backend.lower().strip()
|
||||||
|
|
||||||
|
|
||||||
def _storage_key(*, tenant_id: str, checksum: str, filename: str) -> str:
|
def _storage_key(*, tenant_id: str, checksum: str) -> str:
|
||||||
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}-{safe_storage_component(filename)}"
|
# Object locators remain opaque so recovery evidence never persists names.
|
||||||
|
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}.blob"
|
||||||
|
|
||||||
|
|
||||||
def _get_or_create_blob(
|
def _get_or_create_blob(
|
||||||
@@ -97,7 +99,30 @@ def _get_or_create_blob(
|
|||||||
except StorageBackendError as exc:
|
except StorageBackendError as exc:
|
||||||
raise FileStorageError(str(exc)) from exc
|
raise FileStorageError(str(exc)) from exc
|
||||||
if repair_required:
|
if repair_required:
|
||||||
|
repair_token = hashlib.sha256(
|
||||||
|
repr(
|
||||||
|
(
|
||||||
|
blob.integrity_status,
|
||||||
|
blob.integrity_checked_at,
|
||||||
|
blob.quarantined_at,
|
||||||
|
blob.storage_checksum_sha256,
|
||||||
|
)
|
||||||
|
).encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
recovery = begin_blob_write_recovery(
|
||||||
|
session,
|
||||||
|
backend=backend,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob.id,
|
||||||
|
storage_key=blob.storage_key,
|
||||||
|
semantic_checksum_sha256=checksum,
|
||||||
|
semantic_size_bytes=size,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
created_new=False,
|
||||||
|
repair_token=repair_token,
|
||||||
|
)
|
||||||
stored_data = data
|
stored_data = data
|
||||||
|
expected_envelope_id = blob.encryption_envelope_id
|
||||||
if vault_id:
|
if vault_id:
|
||||||
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||||
|
|
||||||
@@ -115,8 +140,13 @@ def _get_or_create_blob(
|
|||||||
raise FileStorageError("The existing encrypted blob has another protection envelope.")
|
raise FileStorageError("The existing encrypted blob has another protection envelope.")
|
||||||
stored_data = protected.ciphertext
|
stored_data = protected.ciphertext
|
||||||
blob.encryption_envelope_id = protected.envelope.envelope_id
|
blob.encryption_envelope_id = protected.envelope.envelope_id
|
||||||
|
expected_envelope_id = protected.envelope.envelope_id
|
||||||
blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest()
|
blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest()
|
||||||
blob.storage_size_bytes = len(stored_data)
|
blob.storage_size_bytes = len(stored_data)
|
||||||
|
recovery.prepare_stored_bytes(
|
||||||
|
stored_data,
|
||||||
|
envelope_id=expected_envelope_id,
|
||||||
|
)
|
||||||
try:
|
try:
|
||||||
backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||||
except StorageBackendError as exc:
|
except StorageBackendError as exc:
|
||||||
@@ -130,7 +160,19 @@ def _get_or_create_blob(
|
|||||||
return blob
|
return blob
|
||||||
|
|
||||||
blob_id = str(uuid4())
|
blob_id = str(uuid4())
|
||||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum, filename=filename)
|
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
|
||||||
|
backend = get_storage_backend()
|
||||||
|
recovery = begin_blob_write_recovery(
|
||||||
|
session,
|
||||||
|
backend=backend,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob_id,
|
||||||
|
storage_key=storage_key,
|
||||||
|
semantic_checksum_sha256=checksum,
|
||||||
|
semantic_size_bytes=size,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
created_new=True,
|
||||||
|
)
|
||||||
stored_data = data
|
stored_data = data
|
||||||
envelope_id = None
|
envelope_id = None
|
||||||
if vault_id:
|
if vault_id:
|
||||||
@@ -148,7 +190,7 @@ def _get_or_create_blob(
|
|||||||
)
|
)
|
||||||
stored_data = protected.ciphertext
|
stored_data = protected.ciphertext
|
||||||
envelope_id = protected.envelope.envelope_id
|
envelope_id = protected.envelope.envelope_id
|
||||||
backend = get_storage_backend()
|
recovery.prepare_stored_bytes(stored_data, envelope_id=envelope_id)
|
||||||
try:
|
try:
|
||||||
backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||||
except StorageBackendError as exc:
|
except StorageBackendError as exc:
|
||||||
|
|||||||
@@ -16,6 +16,9 @@ from govoplan_files.backend.storage.backends import (
|
|||||||
StorageObjectMissing,
|
StorageObjectMissing,
|
||||||
get_storage_backend,
|
get_storage_backend,
|
||||||
)
|
)
|
||||||
|
from govoplan_files.backend.storage.recovery import (
|
||||||
|
begin_orphan_cleanup_recovery,
|
||||||
|
)
|
||||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||||
|
|
||||||
|
|
||||||
@@ -341,6 +344,12 @@ def cleanup_orphan_finding(
|
|||||||
raise FileStorageError(
|
raise FileStorageError(
|
||||||
"The configured storage backend does not match the integrity finding"
|
"The configured storage backend does not match the integrity finding"
|
||||||
)
|
)
|
||||||
|
begin_orphan_cleanup_recovery(
|
||||||
|
session,
|
||||||
|
finding,
|
||||||
|
backend=active_backend,
|
||||||
|
user_id=user_id,
|
||||||
|
)
|
||||||
try:
|
try:
|
||||||
active_backend.stat(finding.storage_key)
|
active_backend.stat(finding.storage_key)
|
||||||
except StorageObjectMissing:
|
except StorageObjectMissing:
|
||||||
|
|||||||
@@ -0,0 +1,691 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
import hashlib
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from sqlalchemy import event
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.recovery import (
|
||||||
|
RecoveryGuaranteeError,
|
||||||
|
RecoveryMode,
|
||||||
|
RecoveryPlan,
|
||||||
|
RecoveryStatus,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.recovery_runtime import (
|
||||||
|
DurableRecoveryOperation,
|
||||||
|
RecoveryOperationBusy,
|
||||||
|
RecoveryOperationStateConflict,
|
||||||
|
begin_durable_recovery_operation,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||||
|
from govoplan_core.db.session import get_database
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
FileBlob,
|
||||||
|
FileIntegrityFinding,
|
||||||
|
FileIntegrityScan,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
StorageBackend,
|
||||||
|
StorageBackendError,
|
||||||
|
StorageObjectMissing,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
|
||||||
|
|
||||||
|
_PENDING_EFFECTS_KEY = "govoplan_files_pending_recovery_effects"
|
||||||
|
_HOOKS_INSTALLED_KEY = "govoplan_files_recovery_hooks_installed"
|
||||||
|
_ROLLBACK_ERRORS_KEY = "govoplan_files_recovery_rollback_errors"
|
||||||
|
|
||||||
|
|
||||||
|
class _PendingEffect(Protocol):
|
||||||
|
def settle(self, *, committed: bool) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class PendingBlobWrite:
|
||||||
|
operation: DurableRecoveryOperation
|
||||||
|
backend: StorageBackend
|
||||||
|
tenant_id: str
|
||||||
|
blob_id: str
|
||||||
|
storage_key: str
|
||||||
|
semantic_checksum_sha256: str
|
||||||
|
semantic_size_bytes: int
|
||||||
|
protection_discriminator: str
|
||||||
|
created_new: bool
|
||||||
|
expected_storage_checksum_sha256: str | None = None
|
||||||
|
expected_storage_size_bytes: int | None = None
|
||||||
|
expected_envelope_id: str | None = None
|
||||||
|
|
||||||
|
def prepare_stored_bytes(
|
||||||
|
self,
|
||||||
|
data: bytes,
|
||||||
|
*,
|
||||||
|
envelope_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
"""Retain process-local verification evidence without recording content."""
|
||||||
|
|
||||||
|
self.expected_storage_checksum_sha256 = hashlib.sha256(data).hexdigest()
|
||||||
|
self.expected_storage_size_bytes = len(data)
|
||||||
|
self.expected_envelope_id = envelope_id
|
||||||
|
|
||||||
|
def settle(self, *, committed: bool) -> None:
|
||||||
|
evidence = _blob_write_evidence(self)
|
||||||
|
if _blob_write_complete(evidence):
|
||||||
|
self.operation.succeed(evidence=evidence)
|
||||||
|
return
|
||||||
|
|
||||||
|
if self.created_new and evidence.get("database_blob_present") is False:
|
||||||
|
self._settle_unreferenced_new_object(evidence)
|
||||||
|
return
|
||||||
|
|
||||||
|
if not self.created_new and _object_matches(evidence):
|
||||||
|
if _forward_complete_blob_repair(self):
|
||||||
|
completed = _blob_write_evidence(self)
|
||||||
|
if _blob_write_complete(completed):
|
||||||
|
self.operation.succeed(evidence=completed)
|
||||||
|
return
|
||||||
|
|
||||||
|
if evidence.get("database_blob_present") is True and not _object_matches(
|
||||||
|
evidence
|
||||||
|
):
|
||||||
|
_quarantine_blob_after_failed_verification(self, evidence)
|
||||||
|
evidence = _blob_write_evidence(self)
|
||||||
|
|
||||||
|
status = (
|
||||||
|
RecoveryStatus.OUTCOME_UNKNOWN
|
||||||
|
if not evidence.get("verified")
|
||||||
|
else RecoveryStatus.RECOVERY_REQUIRED
|
||||||
|
)
|
||||||
|
transaction = "committed" if committed else "rolled back"
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=status,
|
||||||
|
summary=f"Managed Files blob write remained unresolved after the database transaction {transaction}",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary=(
|
||||||
|
"The managed object and Files blob metadata require reconciliation"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _settle_unreferenced_new_object(self, evidence: dict[str, object]) -> None:
|
||||||
|
object_present = evidence.get("object_present")
|
||||||
|
if object_present is False:
|
||||||
|
self.operation.reject(
|
||||||
|
summary="The Files blob write left no durable object or database row",
|
||||||
|
evidence=evidence,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if object_present is not True:
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||||
|
summary="The unreferenced Files object could not be probed",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary="Object storage availability prevented upload compensation",
|
||||||
|
)
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self.backend.delete(self.storage_key)
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||||
|
summary="An unreferenced Files object could not be compensated",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary="Delete the unreferenced managed object after verifying that no FileBlob references it",
|
||||||
|
)
|
||||||
|
return
|
||||||
|
recovered = _blob_write_evidence(self)
|
||||||
|
if (
|
||||||
|
recovered.get("verified") is True
|
||||||
|
and recovered.get("database_blob_present") is False
|
||||||
|
and recovered.get("object_present") is False
|
||||||
|
):
|
||||||
|
self.operation.compensate(
|
||||||
|
failure_summary="The Files database transaction did not retain the new blob",
|
||||||
|
failure_evidence=evidence,
|
||||||
|
recovery_evidence=recovered,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||||
|
summary="Files upload compensation could not be verified",
|
||||||
|
evidence=recovered,
|
||||||
|
failure_summary="The unreferenced managed object requires operator reconciliation",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class PendingOrphanCleanup:
|
||||||
|
operation: DurableRecoveryOperation
|
||||||
|
backend: StorageBackend
|
||||||
|
finding_id: str
|
||||||
|
tenant_id: str
|
||||||
|
storage_key: str
|
||||||
|
resolved_by_user_id: str
|
||||||
|
|
||||||
|
def settle(self, *, committed: bool) -> None:
|
||||||
|
del committed
|
||||||
|
evidence = _orphan_cleanup_evidence(self)
|
||||||
|
if _orphan_cleanup_complete(evidence):
|
||||||
|
self.operation.succeed(evidence=evidence)
|
||||||
|
return
|
||||||
|
if (
|
||||||
|
evidence.get("verified") is True
|
||||||
|
and evidence.get("object_present") is True
|
||||||
|
and evidence.get("finding_deleted") is False
|
||||||
|
):
|
||||||
|
self.operation.reject(
|
||||||
|
summary="The orphan object was retained and the cleanup finding stayed open",
|
||||||
|
evidence=evidence,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if evidence.get("object_present") is False:
|
||||||
|
if _forward_complete_orphan_finding(self):
|
||||||
|
completed = _orphan_cleanup_evidence(self)
|
||||||
|
if _orphan_cleanup_complete(completed):
|
||||||
|
self.operation.succeed(evidence=completed)
|
||||||
|
return
|
||||||
|
status = (
|
||||||
|
RecoveryStatus.OUTCOME_UNKNOWN
|
||||||
|
if not evidence.get("verified")
|
||||||
|
else RecoveryStatus.RECOVERY_REQUIRED
|
||||||
|
)
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=status,
|
||||||
|
summary="Files orphan cleanup requires reconciliation",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary="Recheck the object and integrity-finding state before another cleanup attempt",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def begin_blob_write_recovery(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend,
|
||||||
|
tenant_id: str,
|
||||||
|
blob_id: str,
|
||||||
|
storage_key: str,
|
||||||
|
semantic_checksum_sha256: str,
|
||||||
|
semantic_size_bytes: int,
|
||||||
|
protection_discriminator: str,
|
||||||
|
created_new: bool,
|
||||||
|
repair_token: str | None = None,
|
||||||
|
) -> PendingBlobWrite:
|
||||||
|
disposition = "create" if created_new else "repair"
|
||||||
|
state_token = repair_token or blob_id
|
||||||
|
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||||
|
idempotency_key = (
|
||||||
|
f"files-blob-{disposition}:{blob_id}:{state_token[:48]}"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
started = begin_durable_recovery_operation(
|
||||||
|
get_database().SessionLocal,
|
||||||
|
identity=process_runtime_identity(),
|
||||||
|
module_id="files",
|
||||||
|
operation_type=f"blob-{disposition}",
|
||||||
|
idempotency_key=idempotency_key,
|
||||||
|
request={
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"blob_id": blob_id,
|
||||||
|
"storage_key": storage_key if created_new else None,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||||
|
"semantic_size_bytes": semantic_size_bytes,
|
||||||
|
"protection_discriminator": protection_discriminator,
|
||||||
|
"disposition": disposition,
|
||||||
|
},
|
||||||
|
recovery_plan=RecoveryPlan(
|
||||||
|
mode=(
|
||||||
|
RecoveryMode.COMPENSATION
|
||||||
|
if created_new
|
||||||
|
else RecoveryMode.FORWARD_RECOVERY
|
||||||
|
),
|
||||||
|
preconditions=(
|
||||||
|
"the caller has Files write authority for the target owner",
|
||||||
|
"the storage key belongs to the tenant Files namespace",
|
||||||
|
"the request records digests rather than file contents",
|
||||||
|
),
|
||||||
|
compensation_steps=(
|
||||||
|
"verify that no FileBlob references the newly reserved key",
|
||||||
|
"delete only that unreferenced key and verify absence",
|
||||||
|
)
|
||||||
|
if created_new
|
||||||
|
else (),
|
||||||
|
forward_recovery_steps=(
|
||||||
|
"verify the expected bytes at the existing blob key",
|
||||||
|
"forward-complete matching integrity metadata or quarantine the blob",
|
||||||
|
)
|
||||||
|
if not created_new
|
||||||
|
else (),
|
||||||
|
verification_steps=(
|
||||||
|
"reload FileBlob metadata through an independent session",
|
||||||
|
"stream and hash the managed object independently",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
precondition_evidence={
|
||||||
|
"blob_id": blob_id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||||
|
"semantic_size_bytes": semantic_size_bytes,
|
||||||
|
"created_new": created_new,
|
||||||
|
},
|
||||||
|
lease_resource_key=(
|
||||||
|
f"files:blob:{tenant_id}:{blob_id}"
|
||||||
|
),
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_blob",
|
||||||
|
resource_id=blob_id,
|
||||||
|
metadata={
|
||||||
|
"resources": ["postgresql", "object-storage"],
|
||||||
|
"storage_backend": backend.name,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This managed blob is already owned by another recovery operation"
|
||||||
|
) from exc
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; no object was written"
|
||||||
|
) from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The matching Files blob operation was already completed; reload before retrying"
|
||||||
|
)
|
||||||
|
pending = PendingBlobWrite(
|
||||||
|
operation=started.operation,
|
||||||
|
backend=backend,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob_id,
|
||||||
|
storage_key=storage_key,
|
||||||
|
semantic_checksum_sha256=semantic_checksum_sha256,
|
||||||
|
semantic_size_bytes=semantic_size_bytes,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
created_new=created_new,
|
||||||
|
)
|
||||||
|
_register_pending_effect(session, pending)
|
||||||
|
return pending
|
||||||
|
|
||||||
|
|
||||||
|
def begin_orphan_cleanup_recovery(
|
||||||
|
session: Session,
|
||||||
|
finding: FileIntegrityFinding,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend,
|
||||||
|
user_id: str,
|
||||||
|
) -> PendingOrphanCleanup:
|
||||||
|
key_digest = hashlib.sha256(finding.storage_key.encode("utf-8")).hexdigest()
|
||||||
|
try:
|
||||||
|
started = begin_durable_recovery_operation(
|
||||||
|
get_database().SessionLocal,
|
||||||
|
identity=process_runtime_identity(),
|
||||||
|
module_id="files",
|
||||||
|
operation_type="integrity-orphan-cleanup",
|
||||||
|
idempotency_key=f"files-orphan-cleanup:{finding.id}",
|
||||||
|
request={
|
||||||
|
"tenant_id": finding.tenant_id,
|
||||||
|
"finding_id": finding.id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
},
|
||||||
|
recovery_plan=RecoveryPlan(
|
||||||
|
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||||
|
preconditions=(
|
||||||
|
"the integrity finding identifies an unreferenced object",
|
||||||
|
"the object key remains inside the completed scan scope",
|
||||||
|
"a fresh database reference check found no FileBlob",
|
||||||
|
),
|
||||||
|
forward_recovery_steps=(
|
||||||
|
"verify object absence independently",
|
||||||
|
"mark the durable finding deleted only after absence is proven",
|
||||||
|
),
|
||||||
|
verification_steps=(
|
||||||
|
"reload the finding and its scan through an independent session",
|
||||||
|
"probe the original storage key through the configured backend",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
precondition_evidence={
|
||||||
|
"finding_id": finding.id,
|
||||||
|
"scan_id": finding.scan_id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"finding_state": finding.state,
|
||||||
|
},
|
||||||
|
lease_resource_key=(
|
||||||
|
f"files:orphan-cleanup:{finding.tenant_id}:{key_digest[:40]}"
|
||||||
|
),
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_integrity_finding",
|
||||||
|
resource_id=finding.id,
|
||||||
|
metadata={
|
||||||
|
"resources": ["postgresql", "object-storage"],
|
||||||
|
"storage_backend": backend.name,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This orphan object is already owned by another recovery operation"
|
||||||
|
) from exc
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; no object was deleted"
|
||||||
|
) from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This orphan cleanup was already completed; reload the finding"
|
||||||
|
)
|
||||||
|
pending = PendingOrphanCleanup(
|
||||||
|
operation=started.operation,
|
||||||
|
backend=backend,
|
||||||
|
finding_id=finding.id,
|
||||||
|
tenant_id=finding.tenant_id,
|
||||||
|
storage_key=finding.storage_key,
|
||||||
|
resolved_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
_register_pending_effect(session, pending)
|
||||||
|
return pending
|
||||||
|
|
||||||
|
|
||||||
|
def _register_pending_effect(session: Session, effect: _PendingEffect) -> None:
|
||||||
|
if not session.in_transaction():
|
||||||
|
session.begin()
|
||||||
|
pending = session.info.setdefault(_PENDING_EFFECTS_KEY, [])
|
||||||
|
pending.append(effect)
|
||||||
|
if session.info.get(_HOOKS_INSTALLED_KEY):
|
||||||
|
return
|
||||||
|
event.listen(session, "after_commit", _after_session_commit)
|
||||||
|
event.listen(session, "after_rollback", _after_session_rollback)
|
||||||
|
session.info[_HOOKS_INSTALLED_KEY] = True
|
||||||
|
|
||||||
|
|
||||||
|
def _after_session_commit(session: Session) -> None:
|
||||||
|
_settle_pending_effects(session, committed=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _after_session_rollback(session: Session) -> None:
|
||||||
|
try:
|
||||||
|
_settle_pending_effects(session, committed=False)
|
||||||
|
except RecoveryGuaranteeError as exc:
|
||||||
|
session.info.setdefault(_ROLLBACK_ERRORS_KEY, []).append(str(exc))
|
||||||
|
|
||||||
|
|
||||||
|
def _settle_pending_effects(session: Session, *, committed: bool) -> None:
|
||||||
|
pending = list(session.info.pop(_PENDING_EFFECTS_KEY, []))
|
||||||
|
failures: list[Exception] = []
|
||||||
|
for effect in pending:
|
||||||
|
try:
|
||||||
|
effect.settle(committed=committed)
|
||||||
|
except Exception as exc: # preserve every effect's chance to settle
|
||||||
|
failures.append(exc)
|
||||||
|
operation = getattr(effect, "operation", None)
|
||||||
|
if operation is not None and not operation.closed:
|
||||||
|
try:
|
||||||
|
operation.release_unresolved()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if failures:
|
||||||
|
raise RecoveryGuaranteeError(
|
||||||
|
f"{len(failures)} Files recovery operation(s) could not be finalized"
|
||||||
|
) from failures[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _blob_write_evidence(effect: PendingBlobWrite) -> dict[str, object]:
|
||||||
|
database_blob_present: bool | None
|
||||||
|
database_matches: bool | None
|
||||||
|
database_integrity_verified: bool | None
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
blob = session.get(FileBlob, effect.blob_id)
|
||||||
|
database_blob_present = blob is not None
|
||||||
|
database_matches = bool(
|
||||||
|
blob is not None
|
||||||
|
and blob.tenant_id == effect.tenant_id
|
||||||
|
and blob.storage_key == effect.storage_key
|
||||||
|
and blob.checksum_sha256 == effect.semantic_checksum_sha256
|
||||||
|
and blob.size_bytes == effect.semantic_size_bytes
|
||||||
|
and blob.protection_discriminator
|
||||||
|
== effect.protection_discriminator
|
||||||
|
and blob.encryption_envelope_id == effect.expected_envelope_id
|
||||||
|
and (
|
||||||
|
effect.expected_storage_checksum_sha256
|
||||||
|
== (
|
||||||
|
blob.storage_checksum_sha256
|
||||||
|
or blob.checksum_sha256
|
||||||
|
)
|
||||||
|
)
|
||||||
|
) if blob is not None else False
|
||||||
|
database_integrity_verified = bool(
|
||||||
|
blob is not None
|
||||||
|
and blob.integrity_status == "verified"
|
||||||
|
and blob.quarantined_at is None
|
||||||
|
) if blob is not None else False
|
||||||
|
except Exception:
|
||||||
|
database_blob_present = None
|
||||||
|
database_matches = None
|
||||||
|
database_integrity_verified = None
|
||||||
|
|
||||||
|
object_present: bool | None
|
||||||
|
observed_size: int | None = None
|
||||||
|
observed_checksum: str | None = None
|
||||||
|
try:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
observed_size = 0
|
||||||
|
for chunk in effect.backend.iter_bytes(effect.storage_key):
|
||||||
|
observed_size += len(chunk)
|
||||||
|
digest.update(chunk)
|
||||||
|
observed_checksum = digest.hexdigest()
|
||||||
|
object_present = True
|
||||||
|
except StorageObjectMissing:
|
||||||
|
object_present = False
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
object_present = None
|
||||||
|
|
||||||
|
object_size_matches = (
|
||||||
|
observed_size == effect.expected_storage_size_bytes
|
||||||
|
if object_present is True and effect.expected_storage_size_bytes is not None
|
||||||
|
else False if object_present is False else None
|
||||||
|
)
|
||||||
|
object_checksum_matches = (
|
||||||
|
observed_checksum == effect.expected_storage_checksum_sha256
|
||||||
|
if object_present is True
|
||||||
|
and effect.expected_storage_checksum_sha256 is not None
|
||||||
|
else False if object_present is False else None
|
||||||
|
)
|
||||||
|
verified = database_blob_present is not None and object_present is not None
|
||||||
|
return {
|
||||||
|
"verified": verified,
|
||||||
|
"checks": {
|
||||||
|
"database_reloaded": database_blob_present is not None,
|
||||||
|
"object_probed": object_present is not None,
|
||||||
|
"database_matches_request": database_matches,
|
||||||
|
"database_integrity_verified": database_integrity_verified,
|
||||||
|
"object_size_matches": object_size_matches,
|
||||||
|
"object_checksum_matches": object_checksum_matches,
|
||||||
|
},
|
||||||
|
"blob_id": effect.blob_id,
|
||||||
|
"database_blob_present": database_blob_present,
|
||||||
|
"database_matches_request": database_matches,
|
||||||
|
"database_integrity_verified": database_integrity_verified,
|
||||||
|
"object_present": object_present,
|
||||||
|
"observed_size_bytes": observed_size,
|
||||||
|
"observed_checksum_sha256": observed_checksum,
|
||||||
|
"expected_storage_size_bytes": effect.expected_storage_size_bytes,
|
||||||
|
"expected_storage_checksum_sha256": effect.expected_storage_checksum_sha256,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _blob_write_complete(evidence: dict[str, object]) -> bool:
|
||||||
|
return bool(
|
||||||
|
evidence.get("verified") is True
|
||||||
|
and evidence.get("database_matches_request") is True
|
||||||
|
and evidence.get("database_integrity_verified") is True
|
||||||
|
and _object_matches(evidence)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _object_matches(evidence: dict[str, object]) -> bool:
|
||||||
|
checks = evidence.get("checks")
|
||||||
|
return bool(
|
||||||
|
isinstance(checks, dict)
|
||||||
|
and evidence.get("object_present") is True
|
||||||
|
and checks.get("object_size_matches") is True
|
||||||
|
and checks.get("object_checksum_matches") is True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _forward_complete_blob_repair(effect: PendingBlobWrite) -> bool:
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
blob = session.get(FileBlob, effect.blob_id)
|
||||||
|
if (
|
||||||
|
blob is None
|
||||||
|
or blob.tenant_id != effect.tenant_id
|
||||||
|
or blob.storage_key != effect.storage_key
|
||||||
|
or blob.checksum_sha256 != effect.semantic_checksum_sha256
|
||||||
|
or blob.size_bytes != effect.semantic_size_bytes
|
||||||
|
or blob.protection_discriminator
|
||||||
|
!= effect.protection_discriminator
|
||||||
|
or blob.encryption_envelope_id != effect.expected_envelope_id
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
blob.storage_checksum_sha256 = (
|
||||||
|
effect.expected_storage_checksum_sha256
|
||||||
|
if effect.expected_envelope_id
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
blob.storage_size_bytes = (
|
||||||
|
effect.expected_storage_size_bytes
|
||||||
|
if effect.expected_envelope_id
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
blob.integrity_status = "verified"
|
||||||
|
blob.integrity_checked_at = datetime.now(UTC)
|
||||||
|
blob.integrity_failure = None
|
||||||
|
blob.quarantined_at = None
|
||||||
|
session.add(blob)
|
||||||
|
session.commit()
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _quarantine_blob_after_failed_verification(
|
||||||
|
effect: PendingBlobWrite,
|
||||||
|
evidence: dict[str, object],
|
||||||
|
) -> None:
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
blob = session.get(FileBlob, effect.blob_id)
|
||||||
|
if blob is None or blob.storage_key != effect.storage_key:
|
||||||
|
return
|
||||||
|
blob.integrity_status = (
|
||||||
|
"missing"
|
||||||
|
if evidence.get("object_present") is False
|
||||||
|
else "checksum_mismatch"
|
||||||
|
)
|
||||||
|
blob.integrity_checked_at = datetime.now(UTC)
|
||||||
|
blob.integrity_failure = "recovery_verification_failed"
|
||||||
|
blob.quarantined_at = datetime.now(UTC)
|
||||||
|
session.add(blob)
|
||||||
|
session.commit()
|
||||||
|
except Exception:
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def _orphan_cleanup_evidence(effect: PendingOrphanCleanup) -> dict[str, object]:
|
||||||
|
try:
|
||||||
|
object_present: bool | None = effect.backend.exists(effect.storage_key)
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
object_present = None
|
||||||
|
finding_present: bool | None
|
||||||
|
finding_deleted: bool | None
|
||||||
|
still_unreferenced: bool | None
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
finding = session.get(FileIntegrityFinding, effect.finding_id)
|
||||||
|
finding_present = finding is not None
|
||||||
|
finding_deleted = bool(
|
||||||
|
finding is not None and finding.state == "deleted"
|
||||||
|
) if finding is not None else False
|
||||||
|
referenced = (
|
||||||
|
session.query(FileBlob.id)
|
||||||
|
.filter(
|
||||||
|
FileBlob.tenant_id == effect.tenant_id,
|
||||||
|
FileBlob.storage_key == effect.storage_key,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
still_unreferenced = referenced is None
|
||||||
|
except Exception:
|
||||||
|
finding_present = None
|
||||||
|
finding_deleted = None
|
||||||
|
still_unreferenced = None
|
||||||
|
verified = object_present is not None and finding_present is not None
|
||||||
|
return {
|
||||||
|
"verified": verified,
|
||||||
|
"checks": {
|
||||||
|
"database_reloaded": finding_present is not None,
|
||||||
|
"object_probed": object_present is not None,
|
||||||
|
"finding_marked_deleted": finding_deleted,
|
||||||
|
"object_absent": (
|
||||||
|
not object_present if object_present is not None else None
|
||||||
|
),
|
||||||
|
"still_unreferenced": still_unreferenced,
|
||||||
|
},
|
||||||
|
"finding_id": effect.finding_id,
|
||||||
|
"finding_present": finding_present,
|
||||||
|
"finding_deleted": finding_deleted,
|
||||||
|
"object_present": object_present,
|
||||||
|
"still_unreferenced": still_unreferenced,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _orphan_cleanup_complete(evidence: dict[str, object]) -> bool:
|
||||||
|
return bool(
|
||||||
|
evidence.get("verified") is True
|
||||||
|
and evidence.get("finding_deleted") is True
|
||||||
|
and evidence.get("object_present") is False
|
||||||
|
and evidence.get("still_unreferenced") is True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _forward_complete_orphan_finding(effect: PendingOrphanCleanup) -> bool:
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
finding = session.get(FileIntegrityFinding, effect.finding_id)
|
||||||
|
if (
|
||||||
|
finding is None
|
||||||
|
or finding.tenant_id != effect.tenant_id
|
||||||
|
or finding.storage_key != effect.storage_key
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||||
|
if scan is None or not finding.storage_key.startswith(
|
||||||
|
scan.storage_prefix
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
referenced = (
|
||||||
|
session.query(FileBlob.id)
|
||||||
|
.filter(
|
||||||
|
FileBlob.tenant_id == effect.tenant_id,
|
||||||
|
FileBlob.storage_key == effect.storage_key,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
if referenced:
|
||||||
|
return False
|
||||||
|
finding.state = "deleted"
|
||||||
|
finding.resolved_at = datetime.now(UTC)
|
||||||
|
finding.resolved_by_user_id = effect.resolved_by_user_id
|
||||||
|
session.add(finding)
|
||||||
|
session.commit()
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"PendingBlobWrite",
|
||||||
|
"PendingOrphanCleanup",
|
||||||
|
"begin_blob_write_recovery",
|
||||||
|
"begin_orphan_cleanup_recovery",
|
||||||
|
]
|
||||||
@@ -4,6 +4,7 @@ import hashlib
|
|||||||
import tempfile
|
import tempfile
|
||||||
import unittest
|
import unittest
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
from sqlalchemy import create_engine
|
from sqlalchemy import create_engine
|
||||||
from sqlalchemy.orm import sessionmaker
|
from sqlalchemy.orm import sessionmaker
|
||||||
@@ -160,6 +161,9 @@ class IntegrityReconciliationTests(unittest.TestCase):
|
|||||||
)
|
)
|
||||||
self.assertEqual("would_delete", preview_cleanup.action)
|
self.assertEqual("would_delete", preview_cleanup.action)
|
||||||
self.assertTrue(self.backend.exists(orphan_key))
|
self.assertTrue(self.backend.exists(orphan_key))
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.integrity.begin_orphan_cleanup_recovery"
|
||||||
|
):
|
||||||
cleanup = cleanup_orphan_finding(
|
cleanup = cleanup_orphan_finding(
|
||||||
self.session,
|
self.session,
|
||||||
orphan_finding,
|
orphan_finding,
|
||||||
|
|||||||
@@ -160,6 +160,8 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
|||||||
self.assertIn("quarantined", topic.body)
|
self.assertIn("quarantined", topic.body)
|
||||||
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
|
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
|
||||||
self.assertIn("Encryption envelope and wrapped-key rows", topic.metadata["recovery_unit"])
|
self.assertIn("Encryption envelope and wrapped-key rows", topic.metadata["recovery_unit"])
|
||||||
|
self.assertIn("lease-fenced Core recovery", topic.body)
|
||||||
|
self.assertIn("Ops", topic.body)
|
||||||
self.assertTrue(topic.metadata["verification"])
|
self.assertTrue(topic.metadata["verification"])
|
||||||
self.assertIn(
|
self.assertIn(
|
||||||
"/api/v1/files/integrity/scans",
|
"/api/v1/files/integrity/scans",
|
||||||
|
|||||||
@@ -0,0 +1,326 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from pathlib import Path
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_access.backend.db.models import Account, Group, User
|
||||||
|
from govoplan_core.core.recovery import (
|
||||||
|
RecoveryCheckpoint,
|
||||||
|
RecoveryOperation,
|
||||||
|
RecoveryStatus,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.runtime_coordination import (
|
||||||
|
DistributedLease,
|
||||||
|
RuntimeIdentity,
|
||||||
|
bind_process_runtime_identity,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_core.db.session import configure_database, reset_database
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
FileBlob,
|
||||||
|
FileIntegrityFinding,
|
||||||
|
FileIntegrityScan,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
LocalFilesystemStorageBackend,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.files import _get_or_create_blob
|
||||||
|
from govoplan_files.backend.storage.integrity import cleanup_orphan_finding
|
||||||
|
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||||
|
|
||||||
|
|
||||||
|
TENANT_ID = "tenant-1"
|
||||||
|
USER_ID = "user-1"
|
||||||
|
|
||||||
|
|
||||||
|
class StorageRecoveryTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.temporary_directory = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temporary_directory.cleanup)
|
||||||
|
root = Path(self.temporary_directory.name)
|
||||||
|
self.backend = LocalFilesystemStorageBackend(root / "objects")
|
||||||
|
database_path = root / "recovery.sqlite3"
|
||||||
|
self.engine = create_engine(f"sqlite:///{database_path}", future=True)
|
||||||
|
Base.metadata.create_all(
|
||||||
|
bind=self.engine,
|
||||||
|
tables=[
|
||||||
|
Account.__table__,
|
||||||
|
User.__table__,
|
||||||
|
Group.__table__,
|
||||||
|
DistributedLease.__table__,
|
||||||
|
RecoveryOperation.__table__,
|
||||||
|
RecoveryCheckpoint.__table__,
|
||||||
|
FileBlob.__table__,
|
||||||
|
FileIntegrityScan.__table__,
|
||||||
|
FileIntegrityFinding.__table__,
|
||||||
|
],
|
||||||
|
)
|
||||||
|
configure_database(
|
||||||
|
f"sqlite:///{database_path}",
|
||||||
|
engine=self.engine,
|
||||||
|
dispose_previous=True,
|
||||||
|
)
|
||||||
|
bind_process_runtime_identity(
|
||||||
|
RuntimeIdentity(
|
||||||
|
installation_id="files-recovery-test",
|
||||||
|
node_id="node-1",
|
||||||
|
incarnation="incarnation-1",
|
||||||
|
role="api",
|
||||||
|
software_version="test",
|
||||||
|
composition_hash="a" * 64,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.Session = sessionmaker(
|
||||||
|
bind=self.engine,
|
||||||
|
expire_on_commit=False,
|
||||||
|
future=True,
|
||||||
|
)
|
||||||
|
self.enterContext(
|
||||||
|
patch(
|
||||||
|
"govoplan_files.backend.storage.files._storage_backend_name",
|
||||||
|
return_value=self.backend.name,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.enterContext(
|
||||||
|
patch(
|
||||||
|
"govoplan_files.backend.storage.files._storage_bucket_name",
|
||||||
|
return_value="",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.session = self.Session()
|
||||||
|
self.addCleanup(self._cleanup_runtime)
|
||||||
|
|
||||||
|
def _cleanup_runtime(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
bind_process_runtime_identity(None)
|
||||||
|
reset_database()
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_committed_blob_write_is_independently_verified(self) -> None:
|
||||||
|
observed_running_operation: list[bool] = []
|
||||||
|
put_bytes = self.backend.put_bytes
|
||||||
|
|
||||||
|
class ObservingBackend:
|
||||||
|
name = self.backend.name
|
||||||
|
|
||||||
|
def __getattr__(backend_self, name):
|
||||||
|
return getattr(self.backend, name)
|
||||||
|
|
||||||
|
def put_bytes(backend_self, key, data, *, content_type=None):
|
||||||
|
with self.Session() as evidence_session:
|
||||||
|
operation = evidence_session.query(RecoveryOperation).one()
|
||||||
|
observed_running_operation.append(
|
||||||
|
operation.status == RecoveryStatus.RUNNING.value
|
||||||
|
and len(operation.request_sha256) == 64
|
||||||
|
)
|
||||||
|
put_bytes(key, data, content_type=content_type)
|
||||||
|
|
||||||
|
observing_backend = ObservingBackend()
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||||
|
return_value=observing_backend,
|
||||||
|
):
|
||||||
|
blob = _get_or_create_blob(
|
||||||
|
self.session,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
data=b"durable",
|
||||||
|
filename="private-name.txt",
|
||||||
|
content_type="text/plain",
|
||||||
|
actor_id=USER_ID,
|
||||||
|
)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
operation = self._only_operation()
|
||||||
|
self.assertEqual(RecoveryStatus.SUCCEEDED.value, operation.status)
|
||||||
|
self.assertEqual([True], observed_running_operation)
|
||||||
|
self.assertTrue(self.backend.exists(blob.storage_key))
|
||||||
|
self.assertNotIn("private-name", blob.storage_key)
|
||||||
|
self.assertEqual(".blob", Path(blob.storage_key).suffix)
|
||||||
|
|
||||||
|
def test_rolled_back_blob_write_is_compensated(self) -> None:
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||||
|
return_value=self.backend,
|
||||||
|
):
|
||||||
|
blob = _get_or_create_blob(
|
||||||
|
self.session,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
data=b"rollback",
|
||||||
|
filename="rollback.txt",
|
||||||
|
content_type="text/plain",
|
||||||
|
actor_id=USER_ID,
|
||||||
|
)
|
||||||
|
storage_key = blob.storage_key
|
||||||
|
self.assertTrue(self.backend.exists(storage_key))
|
||||||
|
self.session.rollback()
|
||||||
|
|
||||||
|
operation = self._only_operation()
|
||||||
|
self.assertEqual(RecoveryStatus.RECOVERED.value, operation.status)
|
||||||
|
self.assertFalse(self.backend.exists(storage_key))
|
||||||
|
with self.Session() as evidence_session:
|
||||||
|
self.assertIsNone(evidence_session.get(FileBlob, blob.id))
|
||||||
|
|
||||||
|
def test_post_write_tamper_is_quarantined_and_recovery_required(self) -> None:
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||||
|
return_value=self.backend,
|
||||||
|
):
|
||||||
|
blob = _get_or_create_blob(
|
||||||
|
self.session,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
data=b"expected",
|
||||||
|
filename="evidence.bin",
|
||||||
|
content_type="application/octet-stream",
|
||||||
|
actor_id=USER_ID,
|
||||||
|
)
|
||||||
|
self.backend.put_bytes(blob.storage_key, b"tampered")
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
operation = self._only_operation()
|
||||||
|
self.assertEqual(
|
||||||
|
RecoveryStatus.RECOVERY_REQUIRED.value,
|
||||||
|
operation.status,
|
||||||
|
)
|
||||||
|
with self.Session() as evidence_session:
|
||||||
|
persisted = evidence_session.get(FileBlob, blob.id)
|
||||||
|
self.assertIsNotNone(persisted)
|
||||||
|
self.assertEqual("checksum_mismatch", persisted.integrity_status)
|
||||||
|
self.assertIsNotNone(persisted.quarantined_at)
|
||||||
|
|
||||||
|
def test_missing_optional_encryption_fails_before_object_effect(self) -> None:
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||||
|
return_value=self.backend,
|
||||||
|
), patch(
|
||||||
|
"govoplan_files.backend.storage.content_protection.encryption_content_cipher",
|
||||||
|
return_value=None,
|
||||||
|
), self.assertRaisesRegex(FileStorageError, "Encryption module"):
|
||||||
|
_get_or_create_blob(
|
||||||
|
self.session,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
data=b"protected",
|
||||||
|
filename="protected.bin",
|
||||||
|
content_type="application/octet-stream",
|
||||||
|
actor_id=USER_ID,
|
||||||
|
encryption_vault_id="vault-1",
|
||||||
|
)
|
||||||
|
self.session.rollback()
|
||||||
|
|
||||||
|
operation = self._only_operation()
|
||||||
|
self.assertEqual(RecoveryStatus.REJECTED.value, operation.status)
|
||||||
|
objects = self.backend.list_objects(
|
||||||
|
prefix=f"tenants/{TENANT_ID}/files/",
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
self.assertEqual((), objects.objects)
|
||||||
|
|
||||||
|
def test_orphan_cleanup_forward_completes_after_business_rollback(self) -> None:
|
||||||
|
key = f"tenants/{TENANT_ID}/files/orphan.bin"
|
||||||
|
self.backend.put_bytes(key, b"orphan")
|
||||||
|
scan = FileIntegrityScan(
|
||||||
|
id="scan-1",
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
storage_backend=self.backend.name,
|
||||||
|
storage_prefix=f"tenants/{TENANT_ID}/files/",
|
||||||
|
status="completed",
|
||||||
|
)
|
||||||
|
finding = FileIntegrityFinding(
|
||||||
|
id="finding-1",
|
||||||
|
scan_id=scan.id,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
kind="orphan_object",
|
||||||
|
state="open",
|
||||||
|
storage_key=key,
|
||||||
|
observed_size_bytes=6,
|
||||||
|
observed_checksum_sha256=hashlib.sha256(b"orphan").hexdigest(),
|
||||||
|
)
|
||||||
|
self.session.add_all([scan, finding])
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
cleanup_orphan_finding(
|
||||||
|
self.session,
|
||||||
|
finding,
|
||||||
|
user_id=USER_ID,
|
||||||
|
dry_run=False,
|
||||||
|
backend=self.backend,
|
||||||
|
)
|
||||||
|
self.assertFalse(self.backend.exists(key))
|
||||||
|
self.session.rollback()
|
||||||
|
|
||||||
|
operation = self._only_operation()
|
||||||
|
self.assertEqual(RecoveryStatus.SUCCEEDED.value, operation.status)
|
||||||
|
with self.Session() as evidence_session:
|
||||||
|
persisted = evidence_session.get(FileIntegrityFinding, finding.id)
|
||||||
|
self.assertIsNotNone(persisted)
|
||||||
|
self.assertEqual("deleted", persisted.state)
|
||||||
|
|
||||||
|
def test_missing_runtime_identity_blocks_before_object_write(self) -> None:
|
||||||
|
bind_process_runtime_identity(None)
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||||
|
return_value=self.backend,
|
||||||
|
), self.assertRaisesRegex(FileStorageError, "recovery ledger"):
|
||||||
|
_get_or_create_blob(
|
||||||
|
self.session,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
data=b"blocked",
|
||||||
|
filename="blocked.bin",
|
||||||
|
content_type="application/octet-stream",
|
||||||
|
actor_id=USER_ID,
|
||||||
|
)
|
||||||
|
self.session.rollback()
|
||||||
|
objects = self.backend.list_objects(
|
||||||
|
prefix=f"tenants/{TENANT_ID}/files/",
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
self.assertEqual((), objects.objects)
|
||||||
|
|
||||||
|
def test_blob_fence_blocks_a_competing_runtime_before_effect(self) -> None:
|
||||||
|
checksum = hashlib.sha256(b"fenced").hexdigest()
|
||||||
|
begin_blob_write_recovery(
|
||||||
|
self.session,
|
||||||
|
backend=self.backend,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
blob_id="blob-fenced",
|
||||||
|
storage_key=f"tenants/{TENANT_ID}/files/fenced.blob",
|
||||||
|
semantic_checksum_sha256=checksum,
|
||||||
|
semantic_size_bytes=6,
|
||||||
|
protection_discriminator="plaintext",
|
||||||
|
created_new=True,
|
||||||
|
)
|
||||||
|
with self.Session() as competing_session, self.assertRaisesRegex(
|
||||||
|
FileStorageError,
|
||||||
|
"already owned",
|
||||||
|
):
|
||||||
|
begin_blob_write_recovery(
|
||||||
|
competing_session,
|
||||||
|
backend=self.backend,
|
||||||
|
tenant_id=TENANT_ID,
|
||||||
|
blob_id="blob-fenced",
|
||||||
|
storage_key=f"tenants/{TENANT_ID}/files/fenced.blob",
|
||||||
|
semantic_checksum_sha256=checksum,
|
||||||
|
semantic_size_bytes=6,
|
||||||
|
protection_discriminator="plaintext",
|
||||||
|
created_new=True,
|
||||||
|
)
|
||||||
|
self.session.rollback()
|
||||||
|
self.assertEqual(RecoveryStatus.REJECTED.value, self._only_operation().status)
|
||||||
|
|
||||||
|
def _only_operation(self) -> RecoveryOperation:
|
||||||
|
with self.Session() as session:
|
||||||
|
operations = session.query(RecoveryOperation).all()
|
||||||
|
self.assertEqual(1, len(operations))
|
||||||
|
session.expunge(operations[0])
|
||||||
|
return operations[0]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user