Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ca99d0d584 | ||
|
|
61625fb00f | ||
|
|
13433514b9 | ||
|
|
ff84812f7f | ||
|
|
2baa8f2657 | ||
|
|
6176e9f40e | ||
|
|
11b9b7c4c6 | ||
|
|
ab08c73777 | ||
|
|
74c26090d7 | ||
|
|
62047e7a20 | ||
|
|
378f4d6ac5 | ||
|
|
a8c3192c3a | ||
|
|
ce4eaefbfd | ||
|
|
6c3cf1c55e | ||
|
|
95aef18955 | ||
|
|
a9c7a7a40c | ||
|
|
e1361427b8 | ||
|
|
ad55d47645 | ||
|
|
21122e058e | ||
|
|
25140fb0b6 | ||
|
|
e9561a87e1 | ||
|
|
9d7d3f96ec | ||
|
|
44ba0ec9c1 | ||
|
|
1e511e4c15 | ||
|
|
cb37079465 | ||
|
|
04e2bae190 | ||
|
|
aa9f90c71e | ||
|
|
60c3d14f08 | ||
|
|
3a3360650f | ||
|
|
c9f8e9262e | ||
|
|
edee29af51 |
@@ -55,6 +55,15 @@ for example for managed attachment selection and campaign file sharing. Files
|
||||
does not import campaign internals; campaign share/existence checks use the core
|
||||
`campaigns.access` capability registered by the campaign module.
|
||||
|
||||
Files also publishes the optional `privacy.dsar.files` capability. The Core
|
||||
data-subject workflow can use it to collect bounded, tenant-scoped file,
|
||||
version, share, folder, evidence, and non-secret configuration metadata for a
|
||||
direct membership subject. The provider may revoke a subject-targeted share or
|
||||
detach mutable actor references idempotently, but it never exports credential
|
||||
material or raw bytes and never bypasses Files retention, legal hold, evidence,
|
||||
purge approval, audit, or recovery controls. See the handbook's data-subject
|
||||
request coverage section for the review and erasure boundary.
|
||||
|
||||
Platform RBAC and governance rules are documented in `govoplan-core/docs/`.
|
||||
|
||||
Managed files can carry source provenance for connector and import workflows.
|
||||
@@ -123,8 +132,16 @@ socket policy. Retries, redirects, endpoint discovery, bucket aliases, and new
|
||||
connections are therefore revalidated while TLS keeps the configured hostname
|
||||
for SNI and certificate checks. Connector clients do not use outbound proxies or
|
||||
ambient AWS credential discovery; configure credentials on the governed profile,
|
||||
or explicitly use an anonymous profile for public objects. An incompatible SDK
|
||||
upgrade fails closed before a usable client/session is returned.
|
||||
or explicitly use an anonymous profile for public objects. Keep connector spaces
|
||||
read-only unless a remote write is explicitly needed.
|
||||
An S3 profile with the `write` capability can back a two-way connector space.
|
||||
`POST /api/v1/files/connector-spaces/{space_id}/write-back` then conditionally
|
||||
creates or replaces one remote object from a managed file, records durable
|
||||
recovery intent before the provider effect, and verifies request/content markers
|
||||
afterward. Automatic remote delete, rename, move, and ACL propagation remain
|
||||
disabled; other connector providers remain read-only.
|
||||
An incompatible SDK upgrade fails closed before a usable client/session is
|
||||
returned.
|
||||
|
||||
Durable platform storage has a separate deployment boundary: installer-owned
|
||||
Garage is accepted only at its exact generated endpoint, while an
|
||||
@@ -165,11 +182,16 @@ the original archive and stores only the selected members. Password-protected
|
||||
ZIP passwords remain request-only and are never included in the preview token.
|
||||
|
||||
Managed blob writes and applied orphan cleanup use Core's durable recovery
|
||||
ledger. Intent, request digests, recovery mode, and a distributed lease are
|
||||
committed before physical storage effects; the Files session commit verifies
|
||||
database and streamed object evidence, while rollback compensates only a newly
|
||||
reserved unreferenced key. New object keys are opaque and do not retain the
|
||||
uploaded filename. Uncertain or mismatched effects remain visible through Ops.
|
||||
ledger. On PostgreSQL, intent, request digests, recovery mode, and a distributed
|
||||
lease are committed before physical storage effects; the Files session commit
|
||||
verifies database and streamed object evidence, while rollback compensates only
|
||||
a newly reserved unreferenced key. Development SQLite records blob intent in
|
||||
the caller transaction to avoid its second-writer deadlock, then verifies on
|
||||
commit or reconstructs compensation evidence after handled rollback. Because a
|
||||
hard loss before that commit can leave an unrecorded object, SQLite requires a
|
||||
complete integrity scan after a crash and is not a production recovery profile.
|
||||
New object keys are opaque and do not retain the uploaded filename. Uncertain
|
||||
or mismatched effects remain visible through Ops.
|
||||
|
||||
Operators with `files:file:admin` can run bounded, resumable integrity scans in
|
||||
Administration. Scan batches and finding actions carry monotonic revisions;
|
||||
|
||||
@@ -139,12 +139,20 @@ cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
```
|
||||
|
||||
The script reads `.env` from this directory when present and seeds tiny WebDAV,
|
||||
Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud are browsed and imported
|
||||
through the pinned connector transport. SMB and S3 product access deliberately
|
||||
fails closed until their SDK transports support peer pinning, so their fixtures
|
||||
are retained for transport development and reported as expected optional skips.
|
||||
The `--require-smb` and `--require-s3` switches are useful only while developing
|
||||
that transport support and currently make the smoke check fail by design.
|
||||
Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud use the pinned HTTP
|
||||
transport. SMB uses the Files-owned `smbprotocol` connection cache so initial
|
||||
connections, reconnects, aliases, and DFS referral targets are pinned. S3 uses
|
||||
pinned botocore pools for retries, redirects, discovery, and aliases. Every
|
||||
socket destination is revalidated when it opens, including when private-network
|
||||
access is enabled for these local fixtures.
|
||||
|
||||
SMB and S3 are optional by default: a missing provider dependency or unavailable
|
||||
target is reported as `SKIP`. Use `--require-smb` and `--require-s3` to turn such
|
||||
provider failures into a non-zero smoke result during deployment validation. The
|
||||
bundled Samba service is a deterministic single-share target, not a DFS
|
||||
topology. Referral and reconnect behavior is covered by the transport contract
|
||||
tests; deployments using DFS must additionally run the required SMB smoke check
|
||||
against their actual referral topology and record the resolved peers.
|
||||
|
||||
SMB smoke checks need the optional Python dependency in the environment running
|
||||
the script:
|
||||
|
||||
@@ -61,11 +61,10 @@ Linked connector space:
|
||||
- read-only flag from provider/policy
|
||||
- active/deleted state
|
||||
|
||||
The linked space should be addressable as a normal file space in the files UI.
|
||||
For the first implementation slice, actions may use the existing connector
|
||||
browse/sync APIs and managed file storage. A linked space can therefore browse
|
||||
the remote folder and sync selected files into managed storage. Later slices can
|
||||
add a native remote listing view or background sync jobs.
|
||||
The linked space is addressable as a normal file space in the Files UI. It can
|
||||
browse the remote folder, synchronize one selected file, or run a bounded manual
|
||||
folder synchronization into managed storage. Background sync jobs remain a
|
||||
separate scheduling concern.
|
||||
|
||||
## Policy Semantics
|
||||
|
||||
@@ -105,6 +104,9 @@ Implemented:
|
||||
- Files UI sync dialog for choosing a profile, browsing a remote folder, and
|
||||
syncing a selected file into a managed destination folder
|
||||
- Files UI connector-space view with provider/read-only/manual-sync state
|
||||
- bounded folder-level manual sync with recursive traversal, pagination,
|
||||
per-item review outcomes, source-provenance matching, and one summary audit
|
||||
event
|
||||
- Docker dev stack smoke checks for WebDAV, Nextcloud, and SMB
|
||||
|
||||
Missing:
|
||||
@@ -137,9 +139,14 @@ Missing:
|
||||
- Remaining: edit/manage actions for existing linked spaces.
|
||||
|
||||
4. Add sync orchestration.
|
||||
- Manual sync selected file is already available.
|
||||
- Add folder-level manual sync.
|
||||
- Add optional scheduled/background sync with conflict reporting.
|
||||
- Manual selected-file and folder-level synchronization are available.
|
||||
- Folder sync preserves remote relative paths, bounds file count and depth,
|
||||
and reports created, updated, unchanged, skipped, conflict, policy-denied,
|
||||
and failed items without rolling back successful siblings.
|
||||
- The default conflict strategy is `skip`, so unrelated managed files are
|
||||
left untouched. `rename`, `reject`, and `overwrite` remain explicit choices.
|
||||
- Add optional scheduled/background sync separately; it must reuse the same
|
||||
policy, provenance, result, and audit contracts.
|
||||
|
||||
5. Add provider-specific expansion.
|
||||
- OAuth/secret-store credentials.
|
||||
|
||||
+435
-51
@@ -1,6 +1,6 @@
|
||||
# GovOPlaN Files Handbook
|
||||
|
||||
This handbook describes the Files module as implemented in version `0.1.9`.
|
||||
This handbook describes the Files module as implemented in version `0.1.18`.
|
||||
It is the operational source of truth for users, process owners, administrators,
|
||||
operators, auditors, and module integrators. Statements about future behavior
|
||||
are marked **planned**; an unmarked statement describes the current code.
|
||||
@@ -8,11 +8,25 @@ are marked **planned**; an unmarked statement describes the current code.
|
||||
Files is a governed snapshot store. It owns managed file content, versions,
|
||||
logical folders, shares, source provenance, and the evidence that another
|
||||
GovOPlaN module used a particular file version. It can browse selected external
|
||||
stores read-only and import a frozen copy. It is not a general remote filesystem,
|
||||
stores and import a frozen copy. Explicit, conditional S3 write-back is available
|
||||
only through an administrator-enabled two-way connector space. Files is not a general remote filesystem,
|
||||
a document collaboration engine, or a records-management system.
|
||||
|
||||
## Choose a reading path
|
||||
|
||||
The Files workspace keeps **Reload → Create folder → Upload** at the top right;
|
||||
Upload is the primary everyday action. Reload re-reads the current folder with
|
||||
its active pattern/property filters and retains usable loaded content on failure.
|
||||
For connector spaces it only browses the current remote folder: it never imports
|
||||
or synchronizes content. **Connections and imports** groups explicit import,
|
||||
synchronization, and linked-space management. Opening this chooser makes no
|
||||
remote changes. Select content to use the local **Download**, applicable
|
||||
**Unpack archive**, or **Manage selection** actions. The latter groups move,
|
||||
copy, rename, sharing, and access explanations, with deletion in a separate
|
||||
section and its existing confirmation. All permissions, connector restrictions,
|
||||
conflict checks, retention rules, keyboard shortcuts, and context menus remain
|
||||
in force; unavailable actions explain their blockers.
|
||||
|
||||
| If you need to... | Start with... |
|
||||
| --- | --- |
|
||||
| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) |
|
||||
@@ -50,16 +64,47 @@ The main domain objects are:
|
||||
|
||||
| Object | Meaning | Lifecycle today |
|
||||
| --- | --- | --- |
|
||||
| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted |
|
||||
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained |
|
||||
| File blob | Stored plaintext semantics plus stored-byte integrity, backend key, optional Encryption envelope, reference count, and retention timestamp | Reused only within a tenant and matching protection boundary; no automated garbage collection |
|
||||
| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted |
|
||||
| File asset | The user-facing file identity, owner, logical path, description, metadata, current version, retention deadline, legal hold, and lifecycle revision | Created, organized, shared, soft-deleted, restored, or governed-purged |
|
||||
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained until its asset is eligible for purge |
|
||||
| File blob | Stored plaintext semantics plus stored-byte integrity, backend key, optional Encryption envelope, reference count, and retention timestamp | Reused only within a tenant and matching protection boundary; collected only after a fenced reference recheck |
|
||||
| Folder | An explicit logical path in a user or group space | Created, moved/renamed, soft-deleted, and restored when its original path is free |
|
||||
| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet |
|
||||
| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion |
|
||||
| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion |
|
||||
| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use |
|
||||
| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted |
|
||||
| Connector space | A manually synchronized remote folder/library linked to a user or group space; read-only by default, with explicit S3 two-way mode available | Created, updated, disabled, soft-deleted, and restored |
|
||||
| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence |
|
||||
| Form evidence upload grant | A one-use, hash-only bearer grant tied to an exact Form instance/revision, purpose, custodian, size, and media-type policy | Issued for at most 15 minutes, consumed by one managed upload, then retained as evidence provenance |
|
||||
|
||||
## Deployment configuration packages
|
||||
|
||||
Files registers the `files.configuration` capability for `managed_storage`
|
||||
fragments. Managed content storage is deployment-owned: the installer selects
|
||||
local persistent storage, managed Garage, or an external S3-compatible service
|
||||
and mounts a validated non-secret infrastructure capability receipt. Files
|
||||
does not copy that endpoint or its credentials into module-owned tables.
|
||||
|
||||
Preflight compares the receipt's `files.storage` capability with the effective
|
||||
runtime backend. It validates backend kind, sanitized S3 endpoint, bucket,
|
||||
Garage management marker or external trust marker, an absolute persistent path
|
||||
for local storage, and the presence of Files-owned `env:` secret references.
|
||||
Secret values are never read into a plan, diagnostic, export, or fragment.
|
||||
|
||||
When runtime and receipt agree, the plan reports `skip`: the desired binding is
|
||||
already effective, and repeated apply is a no-op. A mismatch blocks import and
|
||||
explains which deployment setting must be reconciled. The provider deliberately
|
||||
does not mutate process environment, migrate stored objects, probe remote
|
||||
storage, or reinterpret an infrastructure replacement as safe. Use the Files
|
||||
integrity and Ops checks after deployment and complete migration/recovery review
|
||||
before changing an active backend.
|
||||
|
||||
The Files infrastructure dependency provider makes that review concrete. Its
|
||||
authorized, non-secret Ops inventory reports the active runtime binding and
|
||||
aggregates persisted `FileBlob` rows by storage backend with blob counts and
|
||||
byte totals. A host apply that changes `files.storage` requires a fresh,
|
||||
complete inventory from the same installation and shows the migration and
|
||||
checksum-verification action before any service is replaced. Object keys,
|
||||
tenant identifiers and storage credentials are not exported.
|
||||
|
||||
## User tasks
|
||||
|
||||
@@ -82,9 +127,15 @@ Files administrator; governed API operations can administer other tenant-owned
|
||||
Files resources when their owner is specified. Linked connector spaces appear
|
||||
beside managed spaces when they are active and visible to the user.
|
||||
|
||||
A connector space is intentionally read-only. It is a view of an approved
|
||||
remote location and a starting point for importing or synchronizing selected
|
||||
files into managed storage; it is not a mounted write-through filesystem.
|
||||
A connector space is read-only by default. It is a view of an approved remote
|
||||
location and a starting point for importing a selected file or synchronizing a
|
||||
selected file or bounded folder tree into managed storage. Folder sync preserves
|
||||
relative remote paths in the linked owner's managed space and returns a
|
||||
reviewable result for every discovered or deliberately skipped item. An
|
||||
administrator may opt an S3 space into two-way mode
|
||||
only after enabling the profile's `write` capability. Even then, Files exposes
|
||||
explicit, conditional file write-back—not a mounted filesystem. Automatic
|
||||
remote delete, rename, move, and ACL propagation stay disabled.
|
||||
|
||||
### Upload files
|
||||
|
||||
@@ -106,6 +157,15 @@ updates currently belong to connector sync.
|
||||
|
||||
### Preview and unpack archives
|
||||
|
||||
The dialog uses the shared blurred loading overlay, without an envelope
|
||||
animation. Its status distinguishes transfer, inspection, extraction/storage,
|
||||
and transaction finalization. Selected file counts and byte totals are known
|
||||
from the preview; running extraction displays actual server counters. Unknown
|
||||
progress is indeterminate, and all bytes processed does not mean the transaction
|
||||
has committed. Closing, changing the destination, and editing the selection are
|
||||
disabled while work runs. Errors release the overlay and keep the selection
|
||||
available for review; no automatic import retry is performed.
|
||||
|
||||
The UI previews ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ before writing managed
|
||||
files. Users may select individual files or complete folders. The request is
|
||||
spooled to a bounded temporary file rather than buffered wholly in memory. The
|
||||
@@ -123,10 +183,43 @@ defaults are:
|
||||
rejected;
|
||||
- actual bytes read are counted, not only archive header declarations.
|
||||
|
||||
The browser retains the selected archive and password until confirmation.
|
||||
Confirmation reuploads the archive, verifies the token and digest, repeats all
|
||||
safety checks, and commits only the selected files. No preview archive or
|
||||
password is retained server-side.
|
||||
The UI requests private temporary staging and uploads the archive once. Password
|
||||
verification, renewed preview requests and confirmation reuse that same copy;
|
||||
the password remains request-only and is never stored with it. Confirmation
|
||||
rechecks the actor/destination-bound preview token, digest and safety limits and
|
||||
commits only selected files. Repreviewing does not extend the initial staged
|
||||
copy's lifetime. Cancelling or replacing the archive requests its release;
|
||||
successful confirmation removes it. An expired or missing copy requires a fresh
|
||||
upload; when requesting another preview, the UI can safely repeat that preview
|
||||
with the original selected file. It never retries confirmation automatically.
|
||||
API clients that do not opt into `retain_upload`
|
||||
retain the original request-only preview/confirmation upload contract.
|
||||
|
||||
For an archive that is **already uploaded**, select exactly one managed archive
|
||||
and choose **Unpack archive** in the action bar or context menu. Select an
|
||||
accessible destination space and folder, then choose **Preview archive**. The
|
||||
same entry selection, folder selection, password verification and confirmation
|
||||
dialog applies. Bytes stay on the server; no browser download/re-upload is
|
||||
needed. This path resolves verified source bytes within the configured archive
|
||||
size limit before passing them into the ordinary bounded extraction pipeline.
|
||||
|
||||
Managed extraction requires `files:file:read`, `files:file:download` and
|
||||
`files:file:upload`, current source ownership/share access, and destination-owner
|
||||
access. Both stages enforce source tenant, deletion/version checks, integrity,
|
||||
quarantine, envelope decryption availability and effective source/destination
|
||||
connector restrictions. The token additionally binds the exact managed file and
|
||||
version. A changed version, revoked share, expired token or changed destination
|
||||
requires a fresh preview. No existing destination file is overwritten, including
|
||||
the source archive; choose another destination if a member collides. Failed
|
||||
extraction rolls back newly created managed assets through the existing upload
|
||||
transaction/recovery path. The source file, bytes and version are not modified.
|
||||
|
||||
New members use ordinary archive-upload destination defaults: neither ZIP
|
||||
password protection nor a source storage encryption envelope is automatically
|
||||
copied to extracted files. The resulting provenance records source file/version
|
||||
lineage without passwords. Keep the busy dialog open until the synchronous
|
||||
operation completes; this is not a resumable background job. Remote entries in
|
||||
connector spaces must first be synchronized into managed storage.
|
||||
|
||||
### Organize files and folders
|
||||
|
||||
@@ -147,6 +240,73 @@ Recursive folder deletion is the default. It soft-deletes the selected folder,
|
||||
its child folders, and files below it. A non-recursive delete fails when the
|
||||
folder is not empty.
|
||||
|
||||
### Restore, retain, hold, and purge
|
||||
|
||||
Soft deletion is reversible for callers with `files:file:restore`. Restoring a
|
||||
file preserves its asset identity, complete version history, blob references,
|
||||
and source provenance. Recursive folder restore reactivates the folder tree and
|
||||
its deleted files. Connector-space restore reactivates only the local link.
|
||||
Each restore fails if an active resource already occupies the original path or
|
||||
label.
|
||||
|
||||
Retention and legal hold are independent lifecycle controls. A caller with
|
||||
`files:file:retention` supplies the current lifecycle revision, a reason, an
|
||||
optional retained-until time, and legal-hold state. Stale revisions fail rather
|
||||
than overwriting a concurrent decision.
|
||||
|
||||
Hard purge is deliberately separate from ordinary delete:
|
||||
|
||||
1. A caller with `files:file:purge` previews 1–100 soft-deleted assets.
|
||||
2. The preview reports retention, legal-hold, active-share, Campaign-evidence,
|
||||
and Form-evidence blockers and returns a SHA-256 over current lifecycle and
|
||||
blob-reference state.
|
||||
3. Execution requires that exact hash, a stable idempotency key, an approval
|
||||
reference, and the literal `PURGE` confirmation.
|
||||
4. The recovery-ledger operation and tenant purge fence are durable before the
|
||||
irreversible database transaction. A stale preview or blocker rejects the
|
||||
operation without erasure.
|
||||
5. Purge removes eligible asset, version, and inactive-share rows and
|
||||
recalculates blob reference counts. It does not delete bytes inline.
|
||||
6. A separately authorized bounded blob-GC call takes the same per-blob lease
|
||||
used by uploads, rechecks all `FileVersion` references, deletes the exact
|
||||
object, verifies absence, and only then deletes `FileBlob` metadata.
|
||||
|
||||
Automatic time-based purge scheduling is not implemented. Operators initiate
|
||||
preview, execute, and garbage collection under their local retention process.
|
||||
|
||||
### Data-subject request coverage
|
||||
|
||||
Files registers `privacy.dsar.files` when the module is active. The provider
|
||||
requires a corroborated tenant membership identifier (or a namespaced Files
|
||||
user reference), searches only that tenant, and fails explicitly if its bounded
|
||||
result limit would be exceeded. It reports managed assets, exact versions,
|
||||
folders, user-targeted shares, Form and Campaign evidence, connector
|
||||
configuration actor references, and integrity-operation evidence.
|
||||
|
||||
The export includes only governed metadata. It never embeds file bytes, blob
|
||||
storage keys, passwords, tokens, environment-variable names, secret-provider
|
||||
references, or encrypted connector values. A reviewer follows the authorized
|
||||
version download route when the file itself must be inspected.
|
||||
|
||||
The erasure plan deliberately separates four outcomes:
|
||||
|
||||
- an active share aimed at the subject can be revoked idempotently;
|
||||
- mutable creator/updater references can be detached after tenant, subject, and
|
||||
current-value revalidation;
|
||||
- legal hold, active retention, submitted Form evidence, Campaign delivery
|
||||
evidence, connector configuration history, and integrity evidence are
|
||||
retained with a reason; and
|
||||
- unstructured file content, ownership, filenames, and paths require manual
|
||||
review.
|
||||
|
||||
DSAR execution never invokes physical byte deletion. If the privacy decision
|
||||
authorizes erasure, the operator must use the separate Files soft-delete, purge
|
||||
preview, approval, execution, and blob-GC sequence. This preserves its distinct
|
||||
authority, evidence blockers, audit trail, distributed fencing, and recovery
|
||||
ledger semantics. Email, account, or identity selectors alone are insufficient
|
||||
because Files does not import the Access directory; the Access search supplies
|
||||
the corroborated membership reference for Files coverage.
|
||||
|
||||
### Find and download files
|
||||
|
||||
Files can list by owner and path, use cursor pagination, and consume incremental
|
||||
@@ -187,7 +347,27 @@ inside the chosen owner space:
|
||||
- identical checksum and size updates provenance and returns `unchanged`;
|
||||
- changed bytes append a version and return `updated`.
|
||||
|
||||
Browse, import, and sync never write, rename, or delete the remote source.
|
||||
Folder-level manual sync uses
|
||||
`POST /api/v1/files/connector-spaces/{space_id}/sync`. The interactive request
|
||||
is bounded by file count and traversal depth, follows provider pagination, and
|
||||
can include or exclude subfolders. Every file is isolated in a database
|
||||
savepoint, so one conflict, policy denial, invalid provider result, or transport
|
||||
failure remains reviewable without discarding successful siblings. Results are
|
||||
classified as `created`, `updated`, `unchanged`, `skipped`, `conflict`,
|
||||
`policy_denied`, or `failed`. The UI defaults unrelated target collisions to
|
||||
`skip`; `rename`, `reject`, and `overwrite` require an explicit operator choice.
|
||||
The response marks a run as truncated when its bound is reached so the operator
|
||||
can continue with a narrower remote folder. This is interactive orchestration,
|
||||
not background scheduling.
|
||||
|
||||
Browse, import, and inbound sync never mutate the remote source. An S3 space in
|
||||
explicit two-way mode can write a selected managed file to one remote object
|
||||
path through `POST /api/v1/files/connector-spaces/{space_id}/write-back`.
|
||||
Creating a path uses `If-None-Match`; overwriting requires the currently
|
||||
observed ETag or version and uses a conditional request. The provider object is
|
||||
then re-read and must contain both the expected content digest and the recovery
|
||||
operation marker. Uncertain or mismatching outcomes remain fenced and visible
|
||||
in Ops. Remote deletion, rename, move, and permission propagation are disabled.
|
||||
|
||||
Connector administration separates endpoint profiles, reusable credentials,
|
||||
and inherited policy. Ordinary setup uses typed fields and provider discovery;
|
||||
@@ -293,10 +473,14 @@ those relationships first.
|
||||
| --- | --- |
|
||||
| `files:file:read` | List and inspect accessible files, folders, spaces, and visible connectors |
|
||||
| `files:file:download` | Download an accessible current version or ZIP archive |
|
||||
| `files:file:upload` | Upload managed assets and import/sync selected connector files |
|
||||
| `files:file:upload` | Upload managed assets and import/sync selected connector files or bounded connector folders |
|
||||
| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces |
|
||||
| `files:file:share` | Create or update file shares |
|
||||
| `files:file:delete` | Soft-delete accessible writable files and folders |
|
||||
| `files:file:restore` | Restore owned or administered soft-deleted files, folders, and connector-space links |
|
||||
| `files:file:retention` | Set retention deadlines and legal holds with optimistic revision checks |
|
||||
| `files:file:purge` | Preview and execute irreversible metadata purge and collect unreferenced blobs |
|
||||
| `files:connector:write` | Write an accessible managed file to an explicitly writable connector space |
|
||||
| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant |
|
||||
|
||||
The `file_manager` role template grants all normal file operations except
|
||||
@@ -316,12 +500,14 @@ Keep these definitions separate:
|
||||
local policy, and descriptive operation capabilities;
|
||||
- a **policy** restricts what a scope may configure or use;
|
||||
- a **connector space** links one approved profile/library/path to one user or
|
||||
group and always uses manual, read-only synchronization today.
|
||||
group, uses manual synchronization, and is read-only unless explicitly
|
||||
configured for the supported two-way S3 write boundary.
|
||||
|
||||
Profile capability values such as `browse`, `import`, and `sync` are stored and
|
||||
returned, but they are descriptive today. Provider implementation and policy
|
||||
checks enforce actual availability; do not use the capability list as the sole
|
||||
security control.
|
||||
Profile capability values such as `browse`, `import`, `sync`, and `write` are
|
||||
stored and returned. `write` is additionally enforced for S3 write-back, but
|
||||
provider implementation, connector-space mode, operation permission, resource
|
||||
access, and inherited policy all remain mandatory; never use the capability
|
||||
list as the sole security control.
|
||||
|
||||
Profiles, credentials, and policies support `system`, `tenant`, `user`,
|
||||
`group`, and `campaign` scopes. A normal user sees system and active-tenant
|
||||
@@ -406,7 +592,7 @@ in a development/test runtime.
|
||||
| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport |
|
||||
| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport |
|
||||
| SMB | Read-only browse/import/manual sync implemented through a pinned smbprotocol transport for initial peers, reconnects, aliases, and DFS referral targets |
|
||||
| S3 connector | Read-only bucket/prefix browse, import, and manual sync implemented through pinned botocore pools covering retries, redirects, endpoint discovery, and provider aliases |
|
||||
| S3 connector | Bucket/prefix browse, import, manual inbound sync, and explicit conditional write-back implemented through pinned botocore pools covering retries, redirects, endpoint discovery, and provider aliases; automatic remote delete/rename/move/ACL propagation disabled |
|
||||
| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned |
|
||||
| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature |
|
||||
|
||||
@@ -434,6 +620,9 @@ before it can return a usable client or session.
|
||||
| `FILE_ARCHIVE_MAX_EXPANDED_BYTES` | 2 GiB | Maximum expanded archive bytes |
|
||||
| `FILE_ARCHIVE_MAX_EXPANSION_RATIO` | 100 | Maximum expanded-to-compressed ratio |
|
||||
| `FILE_ARCHIVE_PREVIEW_TTL_SECONDS` | 1,800 | Lifetime of the sealed archive preview token |
|
||||
| `FILE_ARCHIVE_WORK_ROOT` | private OS-temporary directory | Shared transient archive/progress workspace; distinct from durable blob storage |
|
||||
| `FILE_ARCHIVE_STAGED_MAX_BYTES` | 2 GiB | Total staged archive bytes admitted per work root |
|
||||
| `FILE_ARCHIVE_STAGED_PER_ACTOR` | 4 | Maximum staged archives per tenant/user; oldest inactive stages may be evicted |
|
||||
| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets |
|
||||
|
||||
The local backend is the operational baseline. It resolves every storage key
|
||||
@@ -455,6 +644,107 @@ durable shared mount only for same-host replicas. Independent hosts require the
|
||||
`shared` profile with external S3, PostgreSQL, Redis, a stable installation id,
|
||||
and one immutable module composition.
|
||||
|
||||
### Archive staging, progress and performance
|
||||
|
||||
Archive preview staging is optional in the API and requested by the UI. It is
|
||||
not durable managed storage or a background job queue. The default work root
|
||||
is below the OS temporary directory, isolated by the service uid and a hash of
|
||||
the configured local storage root. `FILE_ARCHIVE_WORK_ROOT` selects an explicit
|
||||
root. The internal directory must have mode `0700`; archive, lease and progress
|
||||
files use `0600`. Passwords are never retained. Progress records contain only
|
||||
status and file/byte counters, not member names or content.
|
||||
|
||||
All same-host API workers must see the same work directory, including across
|
||||
container mounts. For multiple hosts, provide shared POSIX storage with working
|
||||
`flock` locks, or route an archive's preview, confirmation and progress requests
|
||||
to the same host. Shared S3 blob storage alone does not share this transient
|
||||
workspace. Include upload limits, temporary disk capacity and sufficiently long
|
||||
proxy/request timeouts in deployment checks; confirmation remains synchronous.
|
||||
|
||||
The defaults permit 2 GiB of staged data per work root and four stages per
|
||||
tenant/user. A new stage can evict the actor's oldest inactive copy; active
|
||||
imports are never evicted to admit another one. Staging expires after the
|
||||
configured preview TTL (1,800 seconds by default), measured from initial
|
||||
staging and not renewed by password repreview. Cancel/replacement requests and
|
||||
successful import remove copies; expiry cleanup is opportunistic during later
|
||||
archive work, so an abandoned file is not guaranteed to disappear at its exact
|
||||
idle deadline. Active POSIX leases protect copies in use beyond TTL until the
|
||||
operation releases them; a crashed worker releases its OS lock. Do not delete
|
||||
active work directories as a cleanup shortcut.
|
||||
|
||||
For classic ZIPCrypto archives, an optional maintained system `libarchive`
|
||||
library accelerates decoding through its public read API. It applies only to
|
||||
stored/Deflate ZIP entries with ASCII or UTF-8-flagged names; AES, unsupported
|
||||
formats/encodings and a missing library use the existing Python reader chosen
|
||||
before decoding. Native corruption/password errors never trigger a fallback.
|
||||
Install/update the library through the deployment's supported package process
|
||||
and restart workers after installation. Both paths retain complete header and
|
||||
path validation, selection and actual output limits; the native path also
|
||||
independently checks size and CRC and never extracts to filesystem paths.
|
||||
|
||||
All ZIP/TAR metadata parsing (including ZIP central directories and TAR PAX
|
||||
headers) and decoding now run in fresh, credential-stripped child processes.
|
||||
Preview/password checks have 120 seconds wall time, 90 CPU seconds and 512 MiB
|
||||
address space. Confirmation uses one child for the entire archive, with 600
|
||||
seconds wall time and 300 CPU seconds. Its address-space limit is the larger of
|
||||
512 MiB or `3 × configured member limit + 128 MiB`. A member limit must be positive
|
||||
and at most 2 GiB; the default 50 MiB member limit and the existing 250 MiB request,
|
||||
10,000-entry, 2 GiB expanded and 100:1 limits are unchanged. The kernel file-size
|
||||
ceiling is `max(member limit, 16 KiB)`; actual member/cumulative bytes are checked
|
||||
independently. Metadata transport has a 64 MiB ceiling, final extraction receipts
|
||||
64 KiB, and status records 16 KiB. Raw archive/member bytes are not pipe DTOs.
|
||||
|
||||
`GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY` is shared with other isolated module work:
|
||||
one admitted operation per API/worker process by default, configurable from one
|
||||
to 16. Admission covers source snapshot preparation before starting the child,
|
||||
so a busy request does not first copy its source. Capacity is not global across
|
||||
replicas. Include each replica's address-space and temporary-disk budgets when
|
||||
sizing the deployment. Resource controls are required; there is no inline parser
|
||||
fallback. These are process resource limits, not a filesystem/network sandbox or
|
||||
an aggregate cgroup memory guarantee.
|
||||
|
||||
Each admitted operation creates a separate local OS-temporary `0700` directory
|
||||
and `0600` source snapshot, beyond the upload-once work-root quota described
|
||||
above. Plan additional capacity for one compressed source copy and at most one
|
||||
extracted member per admitted archive. Snapshot copying is bounded to the initial
|
||||
regular-file size and rejects changed sources. The staging wrapper writes only
|
||||
numbered internal filenames, never archive names. The parent rechecks selected
|
||||
logical paths, regular/no-symlink file identity, size, digest and monotonic
|
||||
progress, persists the member with the existing Files authority/transaction, and
|
||||
deletes it before acknowledging the child to decode the next member. A private
|
||||
`0600` FIFO carries one eight-byte sequence acknowledgement; completed members
|
||||
wake the parent immediately, with no fixed per-member sleep. Wake notifications
|
||||
are capped at 32 KiB; configurations allowing more than 32,768 members use Core's
|
||||
ordinary progress polling after that budget, still within the wall-time limit.
|
||||
Passwords remain request-only and are never written to these files. Handled success/error
|
||||
paths reap the child and remove the private directory. Abrupt host/parent loss
|
||||
can leave private temporary files; use the deployment's OS-temporary cleanup
|
||||
policy without deleting active work.
|
||||
|
||||
The confirmation wall budget includes time waiting for parent storage; a slow
|
||||
destination may exhaust it after earlier members were staged for persistence.
|
||||
Existing rollback and blob recovery remain authoritative. Busy capacity, missing
|
||||
controls, CPU/memory/time/output limits, or invalid staging records produce a
|
||||
controlled failure; split the archive or explicitly retry after resolving the
|
||||
cause. There is no resumable worker or automatic confirmation retry. An optional
|
||||
UI progress-write failure remains non-fatal, but a failed private worker status
|
||||
or acknowledgement channel must abort safely.
|
||||
|
||||
Verified members are read and stored one at a time, avoiding a whole expanded
|
||||
archive in memory. Storage-client reuse is confined to one archive operation;
|
||||
authorization, tenant isolation and policy are not cached. Batched response
|
||||
metadata avoids repeated lookups, while every member still follows the durable
|
||||
recovery ledger, fencing, integrity and post-commit verification path. The
|
||||
optional decoder's speedup does not imply an equal end-to-end or S3 speedup.
|
||||
Use `tests/benchmark_archive_storage.py` for isolated synthetic SQLite/local
|
||||
storage and decoder measurements; it is not a production PostgreSQL/S3 benchmark.
|
||||
|
||||
Transfer, inspection, extraction/storage and finalization remain separate
|
||||
phases. File and byte counters report actual work; an unknown amount remains
|
||||
indeterminate. Reading all bytes is not transaction success: the UI waits for
|
||||
confirmation and commit settlement. Failed or unavailable progress reporting
|
||||
does not automatically retry an import or prove it failed.
|
||||
|
||||
### Connector egress
|
||||
|
||||
Connector access to private networks is a deployment-wide decision:
|
||||
@@ -544,21 +834,33 @@ with a dry-run preview and requires separate destructive confirmation. The
|
||||
confirmation reuses the finding revision from that preview, rechecks that no
|
||||
database reference exists, remains scoped to the scanned tenant prefix, and is
|
||||
idempotent. Both applied and dry-run actions emit audit evidence. A shared
|
||||
reference blocks deletion. Files currently has no legal-hold or hard-purge
|
||||
model, so retention-controlled objects must not be treated as cleanup
|
||||
candidates until those controls are implemented.
|
||||
reference blocks orphan cleanup. Managed-asset purge separately enforces asset
|
||||
retention, legal hold, active shares, and retained Campaign/Form evidence before
|
||||
releasing a blob for fenced garbage collection.
|
||||
|
||||
### Recovery ledger for object effects
|
||||
|
||||
Every managed blob creation or integrity repair starts a Core recovery
|
||||
operation in an independent committed transaction before Files protects or
|
||||
writes bytes. The operation records tenant/blob identifiers, an opaque object
|
||||
On PostgreSQL, every managed blob creation or integrity repair starts a Core
|
||||
recovery operation in an independent committed transaction before Files
|
||||
protects or writes bytes. The operation records tenant/blob identifiers, an opaque object
|
||||
locator or locator digest, semantic SHA-256/size evidence, the recovery mode,
|
||||
and a distributed lease fence. It never records file contents, ZIP passwords,
|
||||
connector credentials, or a newly uploaded filename. New object keys are opaque;
|
||||
legacy filename-bearing keys remain readable but repair operations record only
|
||||
their digest and recover through the blob ID.
|
||||
|
||||
SQLite is a supported local-development database but permits only one writer.
|
||||
Files therefore uses an explicit reduced-durability mode there: recovery intent
|
||||
and its lease are written in the caller transaction, while a process-local
|
||||
fence prevents competing effects in the same runtime. Commit makes the intent
|
||||
durable before independent verification. A handled rollback reconstructs a
|
||||
durable recovery operation and verifies compensation or forward completion.
|
||||
A hard process loss before commit can leave an object without a surviving
|
||||
ledger row, so SQLite is not a production recovery profile; after such a loss,
|
||||
run a complete Files integrity scan and reconcile every reported orphan before
|
||||
resuming writes. PostgreSQL retains the independent pre-effect durability
|
||||
guarantee.
|
||||
|
||||
The Files business transaction then creates or updates the blob, version, and
|
||||
asset rows. Its actual SQLAlchemy commit or rollback settles every pending
|
||||
operation:
|
||||
@@ -579,6 +881,22 @@ and the durable finding state are verified afterward. If the caller transaction
|
||||
rolls back after deletion, Files may forward-complete only that existing
|
||||
finding after rechecking that the key is still unreferenced.
|
||||
|
||||
Hard purge uses an irreversible Core recovery plan with an approval reference
|
||||
and a tenant-wide purge lease. The preview hash binds target IDs, deletion
|
||||
state, lifecycle revisions, holds, deadlines, blockers, and blob IDs. Asset and
|
||||
version removal plus audit evidence commit together. Released objects are not
|
||||
deleted by that transaction. Blob GC later takes `files:blob:<tenant>:<blob>`,
|
||||
the same distributed resource fence as upload/repair, locks and rechecks the
|
||||
blob, and verifies object absence before removing metadata. Provider or database
|
||||
ambiguity remains recovery-required or outcome-unknown in Ops.
|
||||
|
||||
Explicit S3 connector writes use forward recovery because the provider cannot
|
||||
join the database transaction. The durable request contains only tenant,
|
||||
profile, opaque target digest, content SHA-256/size, and expected revision. A
|
||||
conditional create or overwrite is followed by a metadata probe for the exact
|
||||
content and operation markers. Never retry an unresolved target with another
|
||||
request key; reconcile the owning Files operation from provider evidence first.
|
||||
|
||||
Archive preview and confirmation use bounded process-local temporary staging.
|
||||
Staging is not authoritative and is removed on every handled exit; extracted
|
||||
members enter the same per-blob recovery boundary as direct uploads. A hard
|
||||
@@ -586,11 +904,9 @@ process loss may leave a temporary OS file for normal host temporary-file
|
||||
cleanup, but cannot make that staging path a managed Files object.
|
||||
|
||||
Use the Ops recovery-operation view to inspect `files` operations. Do not retry
|
||||
a busy or unresolved blob blindly: first verify the FileBlob row, object hash,
|
||||
integrity state, and any Encryption envelope named by the blob. Hard purge,
|
||||
legal hold, and two-way remote connector mutation are not implemented yet, so
|
||||
they cannot claim recovery-ledger adoption; their owning work remains tracked
|
||||
separately.
|
||||
a busy or unresolved blob or connector path blindly: first verify the FileBlob
|
||||
row and object hash, or the remote request/content markers and revision, plus
|
||||
any Encryption envelope named by a managed blob.
|
||||
|
||||
After restore:
|
||||
|
||||
@@ -648,11 +964,26 @@ public HTTP API. They must not import Files ORM models or storage helpers.
|
||||
| --- | --- |
|
||||
| `files.access` (`0.1.6`) | Explain resource access provenance for managed files, explicit folders, and virtual folders |
|
||||
| `files.campaign_attachments` (`0.1.6`) | Resolve managed attachment matches, prepare frozen campaign snapshots, annotate built messages, share assets with a campaign, and record/mark exact attachment use |
|
||||
| `records.source.files` (`1.0.0`) | Recheck current Files access and resolve one exact, integrity-approved managed file version for Records filing |
|
||||
| `forms_runtime.evidence.files` (`1.0.0`) | Issue a one-time managed attachment grant and re-verify the exact file/version/checksum at Form submission |
|
||||
|
||||
Files requires Core principal resolution and permission evaluation. Campaign is
|
||||
an optional dependency; when installed, Files consumes the optional
|
||||
`campaigns.access` interface to verify campaign existence and access. Missing
|
||||
optional Campaign support fails explicitly rather than bypassing the check.
|
||||
Records is also optional. When enabled, the source capability returns the
|
||||
requested `FileVersion` identity, path snapshot, content metadata, SHA-256,
|
||||
integrity/protection state, and launch link. It rejects mutable aliases,
|
||||
cross-tenant requests, missing access, and quarantined or failed blobs. Records
|
||||
stores the filing decision; Files continues to own the version and bytes.
|
||||
Forms Runtime is optional as well. Its public or authenticated participant never
|
||||
receives general Files access through this integration. Forms Runtime asks for a
|
||||
purpose-bound grant, Files verifies an active same-tenant user custodian, stores
|
||||
only the token digest, accepts one bounded upload, and returns an immutable
|
||||
`EvidenceReference`. Draft save and final submit independently recheck the exact
|
||||
Form instance/revision, grant, asset, version, checksum, deletion state, and
|
||||
integrity state. An idempotent replay returns the existing grant without
|
||||
reproducing its bearer secret.
|
||||
|
||||
### API families
|
||||
|
||||
@@ -662,16 +993,32 @@ All routes below are under `/api/v1/files`.
|
||||
| --- | --- |
|
||||
| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` |
|
||||
| Upload and folders | `POST /upload`, `POST /upload-zip` (compatibility), `POST /archive-preview`, `POST /archive-confirm`, `POST /folders`, `POST /folders/delete` |
|
||||
| Existing managed archive extraction | `POST /{file_id}/archive-preview`, `POST /{file_id}/archive-confirm` (JSON source version, destination, request-only password, preview token and selected paths; read/download/upload permissions) |
|
||||
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
|
||||
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
|
||||
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
|
||||
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` |
|
||||
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}`, `POST /connector-spaces/{space_id}/restore`, `POST /connector-spaces/{space_id}/sync`, `POST /connector-spaces/{space_id}/write-back` |
|
||||
| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` |
|
||||
| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` |
|
||||
| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` |
|
||||
| Restore and lifecycle | `POST /assets/{file_id}/restore`, `POST /folders/restore`, `PATCH /{file_id}/lifecycle` |
|
||||
| Governed erasure | `POST /purge/preview`, `POST /purge/execute`, `POST /purge/blobs` |
|
||||
| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` |
|
||||
| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` |
|
||||
| Incremental connector settings | `GET /connectors/settings/delta` |
|
||||
| Form evidence | `POST /form-evidence/upload` with a short-lived `X-Form-Evidence-Token` issued by Forms Runtime |
|
||||
|
||||
The Files workspace exposes **Remove space** only for connector spaces and only
|
||||
to actors with file-organization authority over the owning user
|
||||
or group space. Confirmation explains the exact boundary: removal soft-deletes
|
||||
the local connector-space definition and makes that virtual view disappear. It
|
||||
does not mutate or delete remote provider content, previously imported managed
|
||||
files, their metadata or shares, the connector profile, credentials, or remote
|
||||
object references. Those retained objects therefore do not block removal and
|
||||
the connector location can be linked again later. User and group managed spaces
|
||||
are intrinsic ownership scopes rather than removable records, so they never
|
||||
offer this action. A missing, already removed, cross-tenant, or inaccessible
|
||||
connector space fails through the backend lookup and owner-access checks.
|
||||
|
||||
Consumers should use cursor/watermark contracts instead of assuming an
|
||||
unbounded complete list. The default full-list page size is 500 and public page
|
||||
@@ -709,6 +1056,9 @@ Files baseline indiscriminately.
|
||||
- Upload, archive extraction, connector response, and S3 stream code use bounded
|
||||
reads. Archive previews are sealed and short-lived; ZIP passwords are
|
||||
request-only.
|
||||
- Public Form evidence uploads require a custom-header bearer grant that is
|
||||
stored only as SHA-256, expires after at most 15 minutes, is bound to one
|
||||
exact submission and user custodian, and can create only one managed file.
|
||||
- Connector HTTP sockets use connection-time DNS/IP validation and pinning,
|
||||
redirects are refused, and unsafe SDK transports fail before client creation.
|
||||
- Database-managed connector passwords/tokens are encrypted; responses redact
|
||||
@@ -750,7 +1100,7 @@ events. External metadata is provider/user input and is not a digital signature.
|
||||
Files records canonical audit events for:
|
||||
|
||||
- connector discovery attempts, before the attempted external I/O;
|
||||
- connector imports and manual syncs;
|
||||
- connector imports, selected-file syncs, and bounded folder sync summaries;
|
||||
- download/archive access to connector-originated managed files;
|
||||
- immediate connector profile and credential deletion/scrubbing;
|
||||
- credential scrubbing during destructive module retirement.
|
||||
@@ -758,6 +1108,10 @@ Files records canonical audit events for:
|
||||
Connector audit details include the managed asset/version/blob, checksum, size,
|
||||
operation, source revision, and provenance where applicable. Deletion audit
|
||||
details name secret/reference kinds but never the secret values.
|
||||
`files.connector.folder_synced` additionally records the connector profile,
|
||||
provider, library/root, managed target, traversal mode, truncation state, typed
|
||||
counts, and bounded per-item source/target/action references. It does not embed
|
||||
downloaded bytes, credentials, or transport content.
|
||||
|
||||
Assets, folders, shares, profiles, credentials, policies, and connector spaces
|
||||
also feed Core's incremental change sequence for UI synchronization. A change
|
||||
@@ -781,6 +1135,9 @@ The word "delete" has different meanings by object type:
|
||||
| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata |
|
||||
| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed |
|
||||
| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility |
|
||||
| Governed hard purge | Removes only preview-matched, soft-deleted asset/version rows without active retention, legal hold, active shares, Campaign evidence, or Form evidence |
|
||||
| Blob garbage collection | Deletes an exact managed object only after a fresh zero-reference check under the shared blob fence; metadata follows verified object absence |
|
||||
| Connector write-back | Explicit conditional S3 create/overwrite only; automatic provider delete, rename, move, and ACL propagation remain disabled |
|
||||
|
||||
Credential/profile scrubbing and its audit event use the same database
|
||||
transaction. If audit creation fails, the deletion rolls back. Repeating a
|
||||
@@ -788,16 +1145,19 @@ delete against an already scrubbed tombstone does not recreate secret evidence.
|
||||
|
||||
### Retention boundary
|
||||
|
||||
File versions and blobs are effectively retained indefinitely today. Although a
|
||||
blob has `ref_count` and `retained_until` fields, no complete retention-policy,
|
||||
legal-hold, hard-purge, or garbage-collection service enforces them. There is
|
||||
also no supported user restore endpoint for soft-deleted assets/folders.
|
||||
Integrity reconciliation is operator-triggered and is not a retention or
|
||||
automatic garbage-collection policy.
|
||||
Each asset has an enforceable retained-until value, legal-hold flag, reason, and
|
||||
optimistic lifecycle revision. These controls block hard purge; they do not
|
||||
automatically schedule it. Restore, purge preview/execute, and blob GC are
|
||||
explicit authorized operations. Blob `retained_until` remains an additional
|
||||
storage-level safeguard and must also have expired before automated collection
|
||||
is introduced. Campaign and Form evidence are hard blockers rather than
|
||||
silently cascaded references.
|
||||
|
||||
Do not promise erasure, timed retention, legal hold, or self-service recovery
|
||||
from the current soft-delete behavior. Those require an explicit, auditable
|
||||
retention/purge design that preserves campaign and other evidence references.
|
||||
Do not equate soft deletion with erasure. Erasure is complete only after the
|
||||
approved purge removes asset/version metadata, bounded GC verifies that no
|
||||
version references the blob and removes its bytes/metadata, and any owning
|
||||
Encryption retention/key-custody consequence has been handled under that
|
||||
module's policy.
|
||||
|
||||
## Acceptance scenarios
|
||||
|
||||
@@ -842,6 +1202,16 @@ attempt. Botocore retries, redirects, endpoint discovery, and aliases, plus SMB
|
||||
reconnects and DFS referrals, must pass through the pinned factories. A changed
|
||||
or unsupported SDK seam must fail before a usable client/session is returned.
|
||||
|
||||
SMB deployments use `smbprotocol>=1.13`; signing is required by default and SMB
|
||||
encryption can be required per connector profile. Private targets additionally
|
||||
require the governed private-network setting, but that setting never bypasses
|
||||
per-connection address validation. The bundled development Samba target checks
|
||||
live initial browse/import behavior with `smoke.py --require-smb`. It is a
|
||||
single-share fixture rather than a DFS topology: reconnect and referral handling
|
||||
is enforced by the transport contract suite, while operators using DFS must run
|
||||
the same required smoke check against their institutional referral topology and
|
||||
record the initial and referred peers as deployment evidence.
|
||||
|
||||
### Imported evidence and sync
|
||||
|
||||
Given a permitted WebDAV file is imported, its managed response and audit event
|
||||
@@ -849,6 +1219,13 @@ must carry source identity, revision when available, current version, checksum,
|
||||
and size. Re-syncing identical bytes must return `unchanged`; changed bytes must
|
||||
create a higher version while preserving the previous version.
|
||||
|
||||
Given a permitted linked connector folder contains changed, unchanged,
|
||||
colliding, and policy-denied files, a manual folder sync must preserve remote
|
||||
relative paths, commit the allowed siblings, return one typed outcome for every
|
||||
discovered/skipped item, and emit a matching summary audit event. Reaching the
|
||||
configured file bound must mark the run truncated rather than imply complete
|
||||
coverage.
|
||||
|
||||
### Campaign freeze
|
||||
|
||||
Given a campaign snapshot selected version V1, when the managed asset later
|
||||
@@ -876,18 +1253,20 @@ returning different content or credentials.
|
||||
| --- | --- | --- |
|
||||
| Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, dry-run-first orphan cleanup, and Core-ledger verification/forward recovery | Scheduled scan execution and deployment-specific S3 HA/backup automation |
|
||||
| Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts, opaque new object keys, and rollback compensation | Malware scanning, quotas, type policy, resumable/chunked upload |
|
||||
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
|
||||
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution, and API restoration preserving versions/provenance | General file-history UI and user-driven append-version UI |
|
||||
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
|
||||
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
|
||||
| Deletion/retention | Soft-delete and restore assets/folders/spaces; optimistic retention and legal-hold controls; preview-bound, approval-referenced hard purge; reference-checked blob GC; immediate audited connector-secret scrubbing | Automatic time-based purge scheduling and richer lifecycle administration UI |
|
||||
| Privacy requests | Tenant-scoped bounded DSAR metadata search; retained/manual/revoke/detach planning; idempotent share revocation and mutable actor-reference detachment; explicit separation from byte purge | Content-specific automated redaction and policy-specific approval remain manual or belong to the owning process |
|
||||
| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected |
|
||||
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers |
|
||||
| SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, and redirect/retry/referral transport-contract tests | Live topology smoke evidence, provider-specific OAuth, remote writes, and background indexing remain separate deployment or connector-module concerns |
|
||||
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import, selected-file sync, and folder sync | Background sync, remote mutation, long-running transfer workers |
|
||||
| SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, redirect/retry/referral transport-contract tests, and explicit conditional S3 write-back with Core-ledger recovery | Live topology smoke evidence, provider-specific OAuth, additional provider writes, and background indexing remain separate deployment or connector-module concerns |
|
||||
| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors |
|
||||
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs |
|
||||
| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate |
|
||||
| Connector spaces | User/group link, browse, bounded manual file/folder sync, edit/disable/delete/restore, read-only default, and opt-in S3 two-way mode | Background sync and automatic remote delete/rename/move/ACL propagation |
|
||||
| Profile capabilities | Stored, displayed, and enforced for explicit connector write-back | Broader provider-specific capability negotiation |
|
||||
| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation |
|
||||
| Preview | File metadata and attachment download | Dedicated safe content-preview service |
|
||||
| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign |
|
||||
| Forms Runtime | One-time managed attachment grants plus exact-version, checksum, deletion, tenant, submission, and integrity verification | Malware scanning and advanced/qualified signature providers remain separate assurance depth |
|
||||
| Collaboration | Governed input/output snapshots | Co-editing, comments, review, presence, locks, and semantic document versions belong to Documents/workflow/provider modules |
|
||||
|
||||
## Release and change checklist
|
||||
@@ -911,7 +1290,12 @@ Before releasing Files:
|
||||
checksum after the current file changes.
|
||||
10. Exercise a committed upload, a rolled-back upload, object tamper detection,
|
||||
and applied orphan cleanup; inspect their `files` operations in Ops.
|
||||
11. Update the implemented/planned table whenever a boundary changes.
|
||||
11. Exercise a Form evidence grant, token replay, wrong-submission reference,
|
||||
expired token, unsupported media type, and quarantined-file rejection.
|
||||
12. Exercise a Files DSAR search/plan, repeat an approved reversible action, and
|
||||
confirm retained file content can only be erased through the separate purge
|
||||
authority and recovery path.
|
||||
13. Update the implemented/planned table whenever a boundary changes.
|
||||
|
||||
## Related documents
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ and consequences described here.
|
||||
| Surface | Primary task | Archetype | Consequence | Pattern evidence |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `/files` space and folder panes | Browse managed and connected content without losing location | Directory/explorer | Low for navigation; medium for exposing filenames and provenance | Full-height two-pane workspace, bounded panes, stable selection and contextual Help Center link |
|
||||
| `/files` toolbar and property filters | Find and act on the current selection | Explorer actions and local filtering | Medium for upload, move, copy, share and synchronization; high for delete | Actions remain beside the affected list, disabled controls explain permission/state/selection blockers, destructive work uses `ConfirmDialog` |
|
||||
| `/files` workspace header and selection toolbar | Find content and act at the appropriate scope | Persistent workspace actions plus local selection tools | Medium for upload, move, copy, share and synchronization; high for delete | Reload → Create folder → primary Upload stay above both panes; Connections and imports groups secondary workspace tools; Download, eligible Unpack archive and Manage selection stay list-local; deletion is separated and uses `ConfirmDialog` |
|
||||
| Upload/archive, transfer, rename and connector-import dialogs | Supply and review one bounded change | Adaptive create/edit or guided import | Medium to high because files, paths and external bytes change | Shared `Dialog`, `FileDropZone`, validation, conflict review, unsaved inputs and explicit confirmation |
|
||||
| File share dialog | Inspect and change access | Review/decision | High because another actor gains access | Shared dialog, access explanation, stable row actions and destructive confirmation |
|
||||
| System/tenant/group/user connector surfaces | Compare connections, credentials and effective policy | Administration/configuration | High because endpoints, secrets and inherited policy control external access | Shared `ConnectionTree`, adaptive forms, `ActionBlockerHint`, policy provenance and contextual admin help |
|
||||
@@ -22,6 +22,11 @@ and consequences described here.
|
||||
|
||||
- Loading, errors, success, empty results, access explanations and confirmation
|
||||
use Core components. Files does not reproduce the application shell.
|
||||
- Explicit Reload bypasses the client read cache for the current folder and all
|
||||
active filter pages, never imports/synchronizes content, and keeps usable data
|
||||
on failure. Late responses cannot replace a changed tenant, account or folder.
|
||||
- Selection and connection tools use domain-labelled shared dialogs and form
|
||||
sections; opening a tool chooser does not itself perform a remote operation.
|
||||
- A connector that comes from deployment settings remains visible but read-only;
|
||||
its action explains that bootstrap configuration and a restart are required.
|
||||
- Missing permission, target, selection, endpoint, or compatible provider is an
|
||||
@@ -52,4 +57,3 @@ The focused structural test guards these contracts, optional-module boundaries,
|
||||
confirmation, contextual help, advanced-only JSON and responsive rules. Core's
|
||||
TypeScript build, structural localization audit, module-permutation suite and
|
||||
full-product bundle check provide the integration gates.
|
||||
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.27",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "webui/src/index.ts",
|
||||
@@ -26,7 +26,7 @@
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.15"
|
||||
"@govoplan/core-webui": "^0.1.45"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
+2
-2
@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-files"
|
||||
version = "0.1.15"
|
||||
version = "0.1.27"
|
||||
description = "GovOPlaN files module with backend and WebUI integration."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = { file = "LICENSE" }
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.15",
|
||||
"govoplan-core>=0.1.46",
|
||||
"defusedxml>=0.7,<1",
|
||||
"pyzipper>=0.3.6,<1",
|
||||
"python-multipart>=0.0.31,<1",
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""Bounded, private, transient archive work shared by workers on one host.
|
||||
|
||||
This is not a detached job queue: confirmation still owns its transaction.
|
||||
Progress never contains filenames, passwords, or extracted document content.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
class ArchiveWorkExpired(FileStorageError):
|
||||
pass
|
||||
|
||||
|
||||
def _uuid(value: str) -> str:
|
||||
try:
|
||||
result = str(UUID(value))
|
||||
except (ValueError, AttributeError, TypeError) as exc:
|
||||
raise FileStorageError("Invalid archive operation identifier") from exc
|
||||
if result != value:
|
||||
raise FileStorageError("Invalid archive operation identifier")
|
||||
return result
|
||||
|
||||
|
||||
def _actor(tenant_id: str, user_id: str) -> str:
|
||||
return hashlib.sha256(json.dumps([tenant_id, user_id]).encode()).hexdigest()
|
||||
|
||||
|
||||
def _root(settings) -> Path:
|
||||
configured = getattr(settings, "file_archive_work_root", None)
|
||||
identity = hashlib.sha256(str(Path(getattr(settings, "file_storage_local_root", "runtime/files")).absolute()).encode()).hexdigest()[:16]
|
||||
base = Path(configured) if configured else Path(tempfile.gettempdir()) / f"govoplan-archive-work-{os.getuid()}-{identity}"
|
||||
root = base.absolute() / "v1"
|
||||
# All internal paths are fixed subdirectories or server-generated UUIDs.
|
||||
root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
if root.is_symlink() or root.stat().st_mode & 0o077:
|
||||
raise FileStorageError("Archive work directory must be private (mode 0700)")
|
||||
return root
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _locked(root: Path):
|
||||
# POSIX advisory locks release automatically if a worker exits. GovOPlaN's
|
||||
# server deployment targets POSIX; no credentials are passed to a process.
|
||||
import fcntl
|
||||
descriptor = os.open(root / ".lock", os.O_CREAT | os.O_RDWR, 0o600)
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX)
|
||||
yield
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def _ttl(settings) -> int:
|
||||
return settings.file_archive_preview_ttl_seconds
|
||||
|
||||
|
||||
def _lease_active(path: Path) -> bool:
|
||||
import fcntl
|
||||
try:
|
||||
descriptor = os.open(path, os.O_RDWR | os.O_NOFOLLOW)
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
try:
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
return True
|
||||
return False
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def _cleanup(root: Path, ttl: int) -> None:
|
||||
cutoff = time.time() - ttl
|
||||
for entry in root.iterdir():
|
||||
if not re.fullmatch(r"[a-f0-9]{64}-[a-f0-9-]{36}\.(archive|progress|lease)", entry.name) or entry.is_symlink():
|
||||
continue
|
||||
try:
|
||||
if entry.stat().st_mtime >= cutoff:
|
||||
continue
|
||||
lease = entry.with_suffix(".lease")
|
||||
if entry.suffix in {".archive", ".lease"} and _lease_active(lease):
|
||||
continue
|
||||
entry.unlink(missing_ok=True)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def stage_upload(settings, source: str, *, tenant_id: str, user_id: str) -> str:
|
||||
root = _root(settings)
|
||||
actor = _actor(tenant_id, user_id)
|
||||
stage_id = str(uuid4())
|
||||
target = root / f"{actor}-{stage_id}.archive"
|
||||
size = Path(source).stat().st_size
|
||||
with _locked(root):
|
||||
_cleanup(root, _ttl(settings))
|
||||
stages = [entry for entry in root.glob("*.archive") if not entry.is_symlink()]
|
||||
own = sorted((entry for entry in stages if entry.name.startswith(actor + "-")), key=lambda item: item.stat().st_mtime)
|
||||
limit = getattr(settings, "file_archive_staged_per_actor", 4)
|
||||
while len(own) >= limit:
|
||||
victim = next((entry for entry in own if not _lease_active(entry.with_suffix(".lease"))), None)
|
||||
if victim is None:
|
||||
raise FileStorageError("Too many archive imports are active; wait for one to finish")
|
||||
victim.unlink(missing_ok=True)
|
||||
own.remove(victim)
|
||||
stages.remove(victim)
|
||||
maximum = getattr(settings, "file_archive_staged_max_bytes", 2 * 1024 ** 3)
|
||||
if size + sum(entry.stat().st_size for entry in stages) > maximum:
|
||||
raise FileStorageError("Temporary archive storage is full; retry after existing previews expire")
|
||||
descriptor = os.open(target, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb") as output, open(source, "rb") as stream:
|
||||
shutil.copyfileobj(stream, output, length=1024 * 1024)
|
||||
except BaseException:
|
||||
target.unlink(missing_ok=True)
|
||||
raise
|
||||
return stage_id
|
||||
|
||||
|
||||
@contextmanager
|
||||
def use_staged_upload(settings, stage_id: str, *, tenant_id: str, user_id: str):
|
||||
import fcntl
|
||||
root = _root(settings)
|
||||
path = root / f"{_actor(tenant_id, user_id)}-{_uuid(stage_id)}.archive"
|
||||
lease = path.with_suffix(".lease")
|
||||
with _locked(root):
|
||||
_cleanup(root, _ttl(settings))
|
||||
if not path.is_file() or path.is_symlink() or path.stat().st_mtime < time.time() - _ttl(settings):
|
||||
raise ArchiveWorkExpired("Archive preview expired; select the archive and preview it again")
|
||||
descriptor = os.open(lease, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
os.close(descriptor)
|
||||
raise FileStorageError("This archive is already being processed") from exc
|
||||
try:
|
||||
yield str(path)
|
||||
finally:
|
||||
try:
|
||||
with _locked(root):
|
||||
lease.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def discard_staged_upload(settings, stage_id: str, *, tenant_id: str, user_id: str) -> None:
|
||||
root = _root(settings)
|
||||
path = root / f"{_actor(tenant_id, user_id)}-{_uuid(stage_id)}.archive"
|
||||
with _locked(root):
|
||||
if not _lease_active(path.with_suffix(".lease")):
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
|
||||
class ArchiveProgress:
|
||||
def __init__(self, settings, operation_id: str | None, *, tenant_id: str, user_id: str):
|
||||
self.path: Path | None = None
|
||||
self.value = {"phase": "inspecting", "completed_files": 0, "total_files": 0,
|
||||
"completed_bytes": 0, "total_bytes": 0, "status": "running"}
|
||||
self.last_write = 0.0
|
||||
if not isinstance(operation_id, str):
|
||||
return
|
||||
root = _root(settings)
|
||||
self.path = root / f"{_actor(tenant_id, user_id)}-{_uuid(operation_id)}.progress"
|
||||
with _locked(root):
|
||||
_cleanup(root, _ttl(settings))
|
||||
# Bounded receipts as well as bounded file bytes; keep newest256 per
|
||||
# actor, never remove a live progress record to admit another one.
|
||||
own = sorted(root.glob(f"{_actor(tenant_id, user_id)}-*.progress"), key=lambda entry: entry.stat().st_mtime)
|
||||
for old in own[:-255]:
|
||||
try:
|
||||
if json.loads(old.read_text()).get("status") != "running":
|
||||
old.unlink(missing_ok=True)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
if len(list(root.glob(f"{_actor(tenant_id, user_id)}-*.progress"))) >= 256:
|
||||
raise FileStorageError("Too many active archive operations")
|
||||
try:
|
||||
descriptor = os.open(self.path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
except FileExistsError as exc:
|
||||
raise FileStorageError("Archive operation identifier has already been used; do not resubmit it") from exc
|
||||
with os.fdopen(descriptor, "w") as target:
|
||||
json.dump(self.value, target)
|
||||
|
||||
def __call__(self, phase: str, completed_files: int, total_files: int, completed_bytes: int, total_bytes: int):
|
||||
changed_phase = self.value["phase"] != phase
|
||||
self.value.update(phase=phase, completed_files=completed_files, total_files=total_files,
|
||||
completed_bytes=completed_bytes, total_bytes=total_bytes)
|
||||
if changed_phase or time.monotonic() - self.last_write >= .2:
|
||||
self._write()
|
||||
|
||||
def finish(self, success: bool):
|
||||
self.value.update(status="complete" if success else "failed", phase="complete" if success else "failed")
|
||||
self._write()
|
||||
|
||||
def _write(self):
|
||||
if self.path is None:
|
||||
return
|
||||
temporary = None
|
||||
try:
|
||||
descriptor, temporary = tempfile.mkstemp(prefix=".progress-", dir=self.path.parent)
|
||||
with os.fdopen(descriptor, "w") as target:
|
||||
json.dump(self.value, target)
|
||||
os.replace(temporary, self.path)
|
||||
self.last_write = time.monotonic()
|
||||
except OSError:
|
||||
# Observability cannot turn a committed import into a failed
|
||||
# response or cause the browser to retry a write.
|
||||
pass
|
||||
finally:
|
||||
if temporary:
|
||||
try:
|
||||
Path(temporary).unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def read_progress(settings, operation_id: str, *, tenant_id: str, user_id: str) -> dict:
|
||||
path = _root(settings) / f"{_actor(tenant_id, user_id)}-{_uuid(operation_id)}.progress"
|
||||
try:
|
||||
if path.is_symlink() or path.stat().st_mtime < time.time() - _ttl(settings):
|
||||
raise ArchiveWorkExpired("Archive progress is not available")
|
||||
return json.loads(path.read_text())
|
||||
except (FileNotFoundError, json.JSONDecodeError) as exc:
|
||||
raise ArchiveWorkExpired("Archive progress is not available") from exc
|
||||
@@ -5,18 +5,32 @@ import binascii
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.auth import ApiPrincipal, has_scope
|
||||
from govoplan_core.core.access import AccessDecisionProvenance, PrincipalRef
|
||||
from govoplan_core.core.files import FileAccessProvider
|
||||
from govoplan_core.core.files import (
|
||||
ManagedTabularFile,
|
||||
ManagedTabularFileAccessError,
|
||||
ManagedTabularFileContent,
|
||||
ManagedTabularFileNotFoundError,
|
||||
ManagedTabularFileProvider,
|
||||
ManagedTabularFileUnavailableError,
|
||||
ManagedTabularFileValidationError,
|
||||
PostboxFileReferenceRef,
|
||||
PostboxFileReferenceRequest,
|
||||
PostboxFileReferenceProvider,
|
||||
ManagedArtifactRef,
|
||||
ManagedArtifactStore,
|
||||
ManagedArtifactWriteRequest,
|
||||
)
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_core.security.module_permissions import scopes_grant_compatible
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileFolder, FileShare, FileVersion
|
||||
from govoplan_files.backend.runtime import configure_runtime
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.campaign_attachments import (
|
||||
annotate_built_messages_with_managed_files,
|
||||
managed_match_payloads,
|
||||
@@ -28,6 +42,9 @@ from govoplan_files.backend.storage.campaign_usage import record_campaign_attach
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
list_recent_assets_for_user,
|
||||
read_asset_version_bytes,
|
||||
sync_file_asset_from_source,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
@@ -146,6 +163,137 @@ def artifact_store_capability(context: ModuleContext) -> FilesArtifactStore:
|
||||
return FilesArtifactStore()
|
||||
|
||||
|
||||
class FilesPostboxReferenceService(PostboxFileReferenceProvider):
|
||||
"""Resolve Postbox evidence without turning message access into file access."""
|
||||
|
||||
def resolve_postbox_references(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
requests: tuple[PostboxFileReferenceRequest, ...],
|
||||
) -> tuple[PostboxFileReferenceRef, ...]:
|
||||
if not hasattr(session, "get"):
|
||||
raise TypeError("Postbox file resolution requires a SQLAlchemy session.")
|
||||
principal_tenant = str(getattr(principal, "tenant_id", "") or "")
|
||||
can_download = bool(
|
||||
hasattr(principal, "has") and principal.has("files:file:download")
|
||||
)
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(getattr(user, "id", "") or "")
|
||||
is_admin = bool(
|
||||
hasattr(principal, "has") and principal.has("files:file:admin")
|
||||
)
|
||||
results: list[PostboxFileReferenceRef] = []
|
||||
for request in requests:
|
||||
if principal_tenant != tenant_id:
|
||||
results.append(_unavailable_postbox_reference(request, "tenant_mismatch"))
|
||||
continue
|
||||
if not can_download or not user_id:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(
|
||||
request,
|
||||
"download_permission_missing",
|
||||
)
|
||||
)
|
||||
continue
|
||||
reference_type = request.reference_type.strip().casefold()
|
||||
version: FileVersion | None = None
|
||||
if reference_type in {"file", "file_asset", "files:file"}:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "exact_version_required")
|
||||
)
|
||||
continue
|
||||
elif reference_type in {"file_version", "files:file_version"}:
|
||||
version = session.get(FileVersion, request.reference_id) # type: ignore[attr-defined]
|
||||
asset = (
|
||||
session.get(FileAsset, version.file_asset_id) # type: ignore[attr-defined]
|
||||
if version is not None
|
||||
else None
|
||||
)
|
||||
else:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(
|
||||
request,
|
||||
"unsupported_reference_type",
|
||||
)
|
||||
)
|
||||
continue
|
||||
if (
|
||||
asset is None
|
||||
or version is None
|
||||
or asset.tenant_id != tenant_id
|
||||
or version.tenant_id != tenant_id
|
||||
or asset.deleted_at is not None
|
||||
):
|
||||
results.append(_unavailable_postbox_reference(request, "file_not_found"))
|
||||
continue
|
||||
try:
|
||||
get_asset_for_user(
|
||||
session, # type: ignore[arg-type]
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=asset.id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
except FileStorageError:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "file_access_denied")
|
||||
)
|
||||
continue
|
||||
blob = session.get(FileBlob, version.blob_id) # type: ignore[attr-defined]
|
||||
if blob is None or blob.tenant_id != tenant_id:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "file_payload_missing")
|
||||
)
|
||||
continue
|
||||
results.append(
|
||||
PostboxFileReferenceRef(
|
||||
reference_type=request.reference_type,
|
||||
reference_id=request.reference_id,
|
||||
available=True,
|
||||
reason_code="available",
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=version.id,
|
||||
filename=version.filename_at_upload or asset.filename,
|
||||
content_type=version.content_type or blob.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
sha256=version.checksum_sha256,
|
||||
download_path=(
|
||||
f"/api/v1/files/{asset.id}/versions/{version.id}/download"
|
||||
),
|
||||
provenance={
|
||||
"module": "files",
|
||||
"postbox_id": request.postbox_id,
|
||||
"message_id": request.message_id,
|
||||
"display_path": asset.display_path,
|
||||
"exact_version": True,
|
||||
},
|
||||
)
|
||||
)
|
||||
return tuple(results)
|
||||
|
||||
|
||||
def _unavailable_postbox_reference(
|
||||
request: PostboxFileReferenceRequest,
|
||||
reason_code: str,
|
||||
) -> PostboxFileReferenceRef:
|
||||
return PostboxFileReferenceRef(
|
||||
reference_type=request.reference_type,
|
||||
reference_id=request.reference_id,
|
||||
available=False,
|
||||
reason_code=reason_code,
|
||||
)
|
||||
|
||||
|
||||
def postbox_reference_capability(
|
||||
context: ModuleContext,
|
||||
) -> FilesPostboxReferenceService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesPostboxReferenceService()
|
||||
|
||||
|
||||
class FilesAccessService(FileAccessProvider):
|
||||
def explain_resource_provenance(
|
||||
self,
|
||||
@@ -284,6 +432,228 @@ def access_capability(context: ModuleContext) -> FilesAccessService:
|
||||
return FilesAccessService()
|
||||
|
||||
|
||||
class FilesManagedTabularFileService(ManagedTabularFileProvider):
|
||||
"""Expose authorized immutable CSV/XLSX versions to optional consumers."""
|
||||
|
||||
def list_tabular_files(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
query: str = "",
|
||||
limit: int = 100,
|
||||
) -> tuple[ManagedTabularFile, ...]:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:read",
|
||||
)
|
||||
normalized_query = str(query or "").strip().casefold()
|
||||
requested_limit = max(1, min(int(limit), 100))
|
||||
assets = list_recent_assets_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
limit=min(500, requested_limit * 5),
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
results: list[ManagedTabularFile] = []
|
||||
for asset in assets:
|
||||
if normalized_query and normalized_query not in (
|
||||
f"{asset.filename} {asset.display_path} {asset.description or ''}"
|
||||
).casefold():
|
||||
continue
|
||||
try:
|
||||
version, blob = current_version_and_blob(db, asset)
|
||||
except FileStorageError:
|
||||
continue
|
||||
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||
continue
|
||||
results.append(_managed_tabular_file(asset, version, blob))
|
||||
if len(results) >= requested_limit:
|
||||
break
|
||||
return tuple(results)
|
||||
|
||||
def get_tabular_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str | None = None,
|
||||
) -> ManagedTabularFile | None:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:read",
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_asset_id,
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
except FileStorageError:
|
||||
return None
|
||||
if file_version_id:
|
||||
version = db.get(FileVersion, file_version_id)
|
||||
if (
|
||||
version is None
|
||||
or version.file_asset_id != asset.id
|
||||
or version.tenant_id != api_principal.tenant_id
|
||||
):
|
||||
return None
|
||||
blob = db.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != api_principal.tenant_id:
|
||||
return None
|
||||
else:
|
||||
try:
|
||||
version, blob = current_version_and_blob(db, asset)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileUnavailableError(
|
||||
"Managed file version metadata is unavailable."
|
||||
) from exc
|
||||
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||
return None
|
||||
return _managed_tabular_file(asset, version, blob)
|
||||
|
||||
def read_tabular_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str,
|
||||
max_bytes: int,
|
||||
) -> ManagedTabularFileContent:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:download",
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_asset_id,
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileNotFoundError(
|
||||
"Managed tabular file not found."
|
||||
) from exc
|
||||
metadata = self.get_tabular_file(
|
||||
db,
|
||||
api_principal,
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=file_version_id,
|
||||
)
|
||||
if metadata is None:
|
||||
raise ManagedTabularFileNotFoundError(
|
||||
"Managed tabular file version not found."
|
||||
)
|
||||
effective_max = max(1, int(max_bytes))
|
||||
if metadata.size_bytes > effective_max:
|
||||
raise ManagedTabularFileValidationError(
|
||||
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||
)
|
||||
try:
|
||||
payload, version, blob = read_asset_version_bytes(
|
||||
db,
|
||||
asset,
|
||||
file_version_id,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileUnavailableError(
|
||||
"Managed tabular file content is unavailable or failed integrity verification."
|
||||
) from exc
|
||||
if len(payload) > effective_max:
|
||||
raise ManagedTabularFileValidationError(
|
||||
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||
)
|
||||
result = _managed_tabular_file(asset, version, blob)
|
||||
audit_from_principal(
|
||||
db,
|
||||
api_principal,
|
||||
action="files.tabular_content.read",
|
||||
object_type="file_version",
|
||||
object_id=version.id,
|
||||
details={
|
||||
"file_asset_id": asset.id,
|
||||
"size_bytes": version.size_bytes,
|
||||
"checksum_sha256": version.checksum_sha256,
|
||||
"consumer": "tabular_content",
|
||||
},
|
||||
)
|
||||
return ManagedTabularFileContent(file=result, payload=payload)
|
||||
|
||||
|
||||
def managed_tabular_file_capability(
|
||||
context: ModuleContext,
|
||||
) -> FilesManagedTabularFileService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesManagedTabularFileService()
|
||||
|
||||
|
||||
def _tabular_context(
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
required_scope: str,
|
||||
) -> tuple[Session, ApiPrincipal, str]:
|
||||
if not isinstance(session, Session):
|
||||
raise TypeError("Managed tabular file access requires a SQLAlchemy session.")
|
||||
if not isinstance(principal, ApiPrincipal):
|
||||
raise ManagedTabularFileAccessError(
|
||||
"Managed tabular file access requires a tenant API principal."
|
||||
)
|
||||
if not (
|
||||
has_scope(principal, required_scope)
|
||||
or has_scope(principal, "files:file:admin")
|
||||
):
|
||||
raise ManagedTabularFileAccessError(
|
||||
f"Managed tabular file access requires {required_scope}."
|
||||
)
|
||||
user_id = str(getattr(principal.user, "id", "") or "").strip()
|
||||
if not principal.tenant_id or not user_id:
|
||||
raise ManagedTabularFileAccessError(
|
||||
"Managed tabular file access requires a tenant user principal."
|
||||
)
|
||||
return session, principal, user_id
|
||||
|
||||
|
||||
def _managed_tabular_file(
|
||||
asset: FileAsset,
|
||||
version: FileVersion,
|
||||
blob: FileBlob,
|
||||
) -> ManagedTabularFile:
|
||||
return ManagedTabularFile(
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=version.id,
|
||||
filename=version.filename_at_upload or asset.filename,
|
||||
display_path=version.display_path_at_upload or asset.display_path,
|
||||
content_type=version.content_type or blob.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
sha256=version.checksum_sha256,
|
||||
updated_at=version.created_at,
|
||||
current_version=asset.current_version_id == version.id,
|
||||
)
|
||||
|
||||
|
||||
def _is_tabular_file(filename: str, content_type: str | None) -> bool:
|
||||
normalized_name = str(filename or "").strip().casefold()
|
||||
normalized_type = str(content_type or "").split(";", 1)[0].strip().casefold()
|
||||
return normalized_name.endswith((".csv", ".xlsx")) or normalized_type in {
|
||||
"text/csv",
|
||||
"application/csv",
|
||||
"application/vnd.ms-excel",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
}
|
||||
|
||||
|
||||
def virtual_folder_resource_id(*, tenant_id: str, owner_type: str, owner_id: str, path: str) -> str:
|
||||
normalized_path = normalize_folder(path)
|
||||
encoded_path = base64.urlsafe_b64encode(normalized_path.encode("utf-8")).decode("ascii").rstrip("=")
|
||||
|
||||
@@ -56,6 +56,10 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
"filename",
|
||||
"description",
|
||||
"deleted_at",
|
||||
"retained_until",
|
||||
"legal_hold",
|
||||
"lifecycle_revision",
|
||||
"lifecycle_reason",
|
||||
"metadata_",
|
||||
),
|
||||
)
|
||||
@@ -79,6 +83,9 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
"previous_path": previous_value(asset, "display_path"),
|
||||
"filename": asset.filename,
|
||||
"deleted_at": _isoformat(asset.deleted_at),
|
||||
"retained_until": _isoformat(asset.retained_until),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,668 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from govoplan_core.core.configuration_packages import (
|
||||
ConfigurationApplyResult,
|
||||
ConfigurationDiagnostic,
|
||||
ConfigurationExportResult,
|
||||
ConfigurationExportSelection,
|
||||
ConfigurationPackageFragment,
|
||||
ConfigurationPlanItem,
|
||||
ConfigurationPreflightContext,
|
||||
ConfigurationPreflightResult,
|
||||
ConfigurationProvider,
|
||||
ConfigurationProviderDescription,
|
||||
)
|
||||
from govoplan_core.core.infrastructure_capabilities import (
|
||||
InfrastructureCapability,
|
||||
InfrastructureCapabilityReceipt,
|
||||
InfrastructureDependency,
|
||||
InfrastructureDependencyProvider,
|
||||
)
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import FileBlob
|
||||
from govoplan_files.backend.runtime import settings as runtime_settings
|
||||
|
||||
|
||||
FILES_CONFIGURATION_CAPABILITY = "files.configuration"
|
||||
FILES_INFRASTRUCTURE_DEPENDENCY_CAPABILITY = (
|
||||
"infrastructure.dependency_inventory.files"
|
||||
)
|
||||
MANAGED_STORAGE_FRAGMENT = "managed_storage"
|
||||
_PAYLOAD_KEYS = frozenset(
|
||||
{"capability_id", "expected_backend", "expected_source"}
|
||||
)
|
||||
|
||||
|
||||
class FilesConfigurationProvider(
|
||||
ConfigurationProvider,
|
||||
InfrastructureDependencyProvider,
|
||||
):
|
||||
module_id = "files"
|
||||
capability_ids = ("files.storage",)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
settings: object | None = None,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> None:
|
||||
self._settings = runtime_settings if settings is None else settings
|
||||
self._environment = os.environ if environment is None else environment
|
||||
|
||||
def describe(self) -> ConfigurationProviderDescription:
|
||||
return ConfigurationProviderDescription(
|
||||
module_id=self.module_id,
|
||||
fragment_types=(MANAGED_STORAGE_FRAGMENT,),
|
||||
schema_refs={
|
||||
MANAGED_STORAGE_FRAGMENT: "govoplan/files/configuration/managed-storage.v1"
|
||||
},
|
||||
exported_scopes=("system",),
|
||||
)
|
||||
|
||||
def preflight(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationPreflightResult:
|
||||
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||
return ConfigurationPreflightResult(
|
||||
diagnostics=(_unsupported(fragment),),
|
||||
plan=(_blocked_plan(fragment, "Fragment type is unsupported."),),
|
||||
)
|
||||
diagnostics, binding_ref = self._binding_diagnostics(fragment, context)
|
||||
blocked = any(item.severity == "blocker" for item in diagnostics)
|
||||
return ConfigurationPreflightResult(
|
||||
diagnostics=tuple(diagnostics),
|
||||
plan=(
|
||||
ConfigurationPlanItem(
|
||||
action="blocked" if blocked else "skip",
|
||||
module_id="files",
|
||||
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||
fragment_id=fragment.fragment_id or binding_ref,
|
||||
summary=(
|
||||
"Managed storage does not match the deployment receipt."
|
||||
if blocked
|
||||
else "Deployment-owned managed storage already matches the receipt; no module state is rewritten."
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
def apply(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
supplied_data: Mapping[str, Any],
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationApplyResult:
|
||||
del supplied_data
|
||||
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||
return ConfigurationApplyResult(diagnostics=(_unsupported(fragment),))
|
||||
diagnostics, _binding_ref = self._binding_diagnostics(fragment, context)
|
||||
return ConfigurationApplyResult(
|
||||
diagnostics=tuple(
|
||||
item for item in diagnostics if item.severity == "blocker"
|
||||
)
|
||||
)
|
||||
|
||||
def export(
|
||||
self,
|
||||
selection: ConfigurationExportSelection,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationExportResult:
|
||||
del selection
|
||||
receipt = context.infrastructure_receipt
|
||||
if context.infrastructure_receipt_error:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
_receipt_error(
|
||||
context.infrastructure_receipt_error,
|
||||
object_ref="files.storage",
|
||||
),
|
||||
)
|
||||
)
|
||||
if receipt is None:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_missing",
|
||||
message="Files managed-storage export requires the deployment capability receipt.",
|
||||
module_id="files",
|
||||
object_ref="files.storage",
|
||||
resolution="Mount the installer-generated receipt before exporting deployment configuration.",
|
||||
),
|
||||
)
|
||||
)
|
||||
capability = receipt.capability("files.storage")
|
||||
if capability is None:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_missing",
|
||||
message="The deployment receipt does not declare managed file storage.",
|
||||
module_id="files",
|
||||
object_ref="files.storage",
|
||||
),
|
||||
)
|
||||
)
|
||||
return ConfigurationExportResult(
|
||||
fragments=(
|
||||
ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||
fragment_id=f"{receipt.installation_id}:files.storage",
|
||||
payload={
|
||||
"capability_id": "files.storage",
|
||||
"expected_backend": _expected_backend(capability.source),
|
||||
"expected_source": capability.source,
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def health(
|
||||
self,
|
||||
import_result: ConfigurationApplyResult,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> tuple[ConfigurationDiagnostic, ...]:
|
||||
del context
|
||||
return tuple(
|
||||
item for item in import_result.diagnostics if item.severity == "blocker"
|
||||
)
|
||||
|
||||
def infrastructure_dependencies(self) -> tuple[InfrastructureDependency, ...]:
|
||||
active_backend = _normalized_backend(
|
||||
getattr(self._settings, "file_storage_backend", "local")
|
||||
)
|
||||
dependencies = [
|
||||
InfrastructureDependency(
|
||||
capability_id="files.storage",
|
||||
module_id="files",
|
||||
dependency_type="runtime_storage_binding",
|
||||
dependency_ref=f"files-storage:{active_backend}",
|
||||
state="runtime_binding",
|
||||
scope="system",
|
||||
summary=(
|
||||
"The active Files runtime is bound to this deployment storage backend."
|
||||
),
|
||||
metrics={},
|
||||
required_action=(
|
||||
"Provision and verify the replacement backend before rebinding the Files runtime."
|
||||
),
|
||||
)
|
||||
]
|
||||
with get_database().session() as session:
|
||||
rows = session.execute(
|
||||
select(
|
||||
FileBlob.storage_backend,
|
||||
func.count(FileBlob.id),
|
||||
func.coalesce(func.sum(FileBlob.size_bytes), 0),
|
||||
)
|
||||
.group_by(FileBlob.storage_backend)
|
||||
.order_by(FileBlob.storage_backend)
|
||||
)
|
||||
for backend, blob_count, size_bytes in rows:
|
||||
normalized_backend = _normalized_backend(str(backend or "local"))
|
||||
dependencies.append(
|
||||
InfrastructureDependency(
|
||||
capability_id="files.storage",
|
||||
module_id="files",
|
||||
dependency_type="stored_blob_set",
|
||||
dependency_ref=f"file-blobs:{normalized_backend}",
|
||||
state="data_present",
|
||||
scope="all-tenants",
|
||||
summary=(
|
||||
"Persisted Files blob metadata references content in this storage backend."
|
||||
),
|
||||
metrics={
|
||||
"blob_count": int(blob_count or 0),
|
||||
"content_bytes": int(size_bytes or 0),
|
||||
},
|
||||
required_action=(
|
||||
"Copy and checksum-verify every referenced blob, switch the runtime binding, and retain rollback evidence before removing or replacing storage."
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(dependencies)
|
||||
|
||||
def _binding_diagnostics(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> tuple[list[ConfigurationDiagnostic], str]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
payload = _files_fragment_payload(fragment, diagnostics)
|
||||
if payload is None:
|
||||
return diagnostics, fragment.fragment_id or "files.storage"
|
||||
receipt_state = _files_receipt_capability(
|
||||
fragment,
|
||||
context,
|
||||
payload,
|
||||
diagnostics,
|
||||
)
|
||||
if receipt_state is None:
|
||||
return diagnostics, fragment.fragment_id or "files.storage"
|
||||
_receipt, capability, binding_ref = receipt_state
|
||||
expected_backend = _expected_backend(capability.source)
|
||||
diagnostics.extend(
|
||||
_package_expectation_diagnostics(
|
||||
payload,
|
||||
capability,
|
||||
expected_backend,
|
||||
binding_ref,
|
||||
)
|
||||
)
|
||||
active_backend = _normalized_backend(
|
||||
getattr(self._settings, "file_storage_backend", "local")
|
||||
)
|
||||
if active_backend != expected_backend:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_runtime_mismatch",
|
||||
message=(
|
||||
f"Files runtime uses {active_backend!r}, while the deployment receipt declares {expected_backend!r}."
|
||||
),
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Reconcile deployment environment and receipt before starting or importing Files configuration.",
|
||||
)
|
||||
)
|
||||
return diagnostics, binding_ref
|
||||
diagnostics.extend(
|
||||
self._runtime_storage_diagnostics(
|
||||
expected_backend,
|
||||
capability,
|
||||
binding_ref,
|
||||
)
|
||||
)
|
||||
if "files" not in capability.dependent_modules:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="info",
|
||||
code="infrastructure_consumer_not_declared",
|
||||
message="Files is active but was not selected as a receipt consumer when the deployment plan was generated.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Regenerate the deployment plan so removal-impact inventory includes Files.",
|
||||
)
|
||||
)
|
||||
return diagnostics, binding_ref
|
||||
|
||||
def _runtime_storage_diagnostics(
|
||||
self,
|
||||
expected_backend: str,
|
||||
capability: InfrastructureCapability,
|
||||
binding_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
if expected_backend == "s3":
|
||||
return self._s3_diagnostics(
|
||||
capability.endpoint,
|
||||
capability.secret_refs,
|
||||
capability.source,
|
||||
binding_ref,
|
||||
)
|
||||
root = Path(
|
||||
str(getattr(self._settings, "file_storage_local_root", "") or "")
|
||||
)
|
||||
if root.is_absolute():
|
||||
return []
|
||||
return [
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_local_root_not_durable",
|
||||
message="Receipt-bound local file storage requires an absolute deployment-managed path.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Set FILE_STORAGE_LOCAL_ROOT to the mounted persistent-volume path.",
|
||||
)
|
||||
]
|
||||
|
||||
def _s3_diagnostics(
|
||||
self,
|
||||
endpoint: Mapping[str, object],
|
||||
secret_refs: tuple[str, ...],
|
||||
source: str,
|
||||
object_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
endpoint_url = str(
|
||||
getattr(self._settings, "file_storage_s3_endpoint_url", "")
|
||||
or getattr(self._settings, "s3_endpoint_url", "")
|
||||
or ""
|
||||
).strip()
|
||||
runtime_endpoint = _redacted_endpoint(
|
||||
endpoint_url,
|
||||
default_port=int(endpoint.get("port") or 443),
|
||||
)
|
||||
receipt_endpoint = {
|
||||
key: endpoint.get(key) for key in ("scheme", "host", "port") if endpoint.get(key) is not None
|
||||
}
|
||||
if receipt_endpoint and runtime_endpoint != receipt_endpoint:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_endpoint_mismatch",
|
||||
message="Files S3 endpoint does not match the sanitized deployment receipt endpoint.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Reconcile FILE_STORAGE_S3_ENDPOINT_URL and the deployment plan before apply.",
|
||||
)
|
||||
)
|
||||
bucket = str(
|
||||
getattr(self._settings, "file_storage_s3_bucket", "")
|
||||
or getattr(self._settings, "s3_bucket", "")
|
||||
or ""
|
||||
).strip()
|
||||
if not bucket:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_bucket_missing",
|
||||
message="Files S3 storage has no configured bucket.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Set FILE_STORAGE_S3_BUCKET in the deployment environment.",
|
||||
)
|
||||
)
|
||||
for reference in secret_refs:
|
||||
variable = reference.removeprefix("env:")
|
||||
if not variable.startswith("FILE_STORAGE_"):
|
||||
continue
|
||||
if not str(self._environment.get(variable, "")).strip():
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_secret_reference_unresolved",
|
||||
message=f"Required storage secret reference {reference} is not available to the runtime.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Provide the referenced environment secret without copying its value into the package.",
|
||||
)
|
||||
)
|
||||
deployment_managed = bool(
|
||||
getattr(
|
||||
self._settings,
|
||||
"file_storage_s3_deployment_managed",
|
||||
False,
|
||||
)
|
||||
)
|
||||
endpoint_trusted = bool(
|
||||
getattr(self._settings, "file_storage_s3_endpoint_trusted", False)
|
||||
)
|
||||
if source == "installer-managed-garage" and not deployment_managed:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_management_boundary_mismatch",
|
||||
message="Garage is installer-managed in the receipt but not marked deployment-managed in Files runtime.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Set FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true for the installer-managed Garage endpoint.",
|
||||
)
|
||||
)
|
||||
if source == "operator-supplied-s3" and not endpoint_trusted:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_trust_boundary_missing",
|
||||
message="External S3 storage must be explicitly marked as a trusted deployment endpoint.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Review the endpoint and set FILE_STORAGE_S3_ENDPOINT_TRUSTED=true in deployment configuration.",
|
||||
)
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
|
||||
def _files_fragment_payload(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
diagnostics: list[ConfigurationDiagnostic],
|
||||
) -> Mapping[str, Any] | None:
|
||||
payload = fragment.payload
|
||||
if not isinstance(payload, Mapping):
|
||||
diagnostics.append(
|
||||
_invalid(fragment, "Managed-storage payload must be an object.")
|
||||
)
|
||||
return None
|
||||
unknown = sorted(set(payload) - _PAYLOAD_KEYS)
|
||||
if not unknown:
|
||||
return payload
|
||||
secret_like = any(
|
||||
marker in key.casefold()
|
||||
for key in unknown
|
||||
for marker in ("password", "secret", "token", "credential", "access_key")
|
||||
)
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code=(
|
||||
"files_configuration_secret_forbidden"
|
||||
if secret_like
|
||||
else "files_configuration_payload_invalid"
|
||||
),
|
||||
message=(
|
||||
"Managed-storage fragments accept receipt references and non-secret expectations only."
|
||||
if secret_like
|
||||
else f"Managed-storage payload contains unsupported fields: {', '.join(unknown)}."
|
||||
),
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or MANAGED_STORAGE_FRAGMENT,
|
||||
resolution="Keep storage credentials in deployment environment references, never in a configuration package.",
|
||||
)
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def _files_receipt_capability(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
payload: Mapping[str, Any],
|
||||
diagnostics: list[ConfigurationDiagnostic],
|
||||
) -> tuple[
|
||||
InfrastructureCapabilityReceipt,
|
||||
InfrastructureCapability,
|
||||
str,
|
||||
] | None:
|
||||
if context.infrastructure_receipt_error:
|
||||
diagnostics.append(
|
||||
_receipt_error(
|
||||
context.infrastructure_receipt_error,
|
||||
object_ref=fragment.fragment_id or "files.storage",
|
||||
)
|
||||
)
|
||||
return None
|
||||
receipt = context.infrastructure_receipt
|
||||
if receipt is None:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_missing",
|
||||
message="Files managed-storage configuration requires the deployment capability receipt.",
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or "files.storage",
|
||||
resolution="Mount the installer-generated receipt and rerun preflight.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
binding_ref = fragment.fragment_id or f"{receipt.installation_id}:files.storage"
|
||||
capability_id = _text(payload.get("capability_id")) or "files.storage"
|
||||
if capability_id != "files.storage":
|
||||
diagnostics.append(
|
||||
_invalid(
|
||||
fragment,
|
||||
"Files managed-storage fragments must reference capability 'files.storage'.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
capability = receipt.capability(capability_id)
|
||||
if capability is None:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_missing",
|
||||
message="The deployment receipt does not declare managed file storage.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Regenerate the receipt from a deployment profile that declares managed storage.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
state_diagnostic = _storage_capability_state_diagnostic(capability, binding_ref)
|
||||
if state_diagnostic is not None:
|
||||
diagnostics.append(state_diagnostic)
|
||||
return None
|
||||
return receipt, capability, binding_ref
|
||||
|
||||
|
||||
def _storage_capability_state_diagnostic(
|
||||
capability: InfrastructureCapability,
|
||||
binding_ref: str,
|
||||
) -> ConfigurationDiagnostic | None:
|
||||
if capability.state == "unavailable":
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_unavailable",
|
||||
message="The deployment receipt states that managed file storage is unavailable.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Select local, managed Garage, or external S3 storage in the deployment profile.",
|
||||
)
|
||||
if capability.state == "available_unconfigured":
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_unconfigured",
|
||||
message="Managed file storage is available but has not been bound by the deployment runtime.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Complete the deployment-owned storage configuration before importing Files configuration.",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _package_expectation_diagnostics(
|
||||
payload: Mapping[str, Any],
|
||||
capability: InfrastructureCapability,
|
||||
expected_backend: str,
|
||||
binding_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
package_backend = (
|
||||
_text(payload.get("expected_backend")) or expected_backend
|
||||
).casefold()
|
||||
if package_backend != expected_backend:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_backend_mismatch",
|
||||
message="The package storage backend expectation conflicts with the deployment receipt.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Use the receipt backend or regenerate the deployment package after review.",
|
||||
)
|
||||
)
|
||||
package_source = _text(payload.get("expected_source"))
|
||||
if package_source and package_source != capability.source:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_source_mismatch",
|
||||
message="The package storage source expectation conflicts with the deployment receipt.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Review the provider replacement and regenerate the package from the active receipt.",
|
||||
)
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
|
||||
def _expected_backend(source: str) -> str:
|
||||
return "local" if source == "host-local" else "s3"
|
||||
|
||||
|
||||
def _normalized_backend(value: object) -> str:
|
||||
clean = str(value or "local").strip().casefold()
|
||||
if clean in {"local", "filesystem", "fs"}:
|
||||
return "local"
|
||||
if clean in {"s3", "garage"}:
|
||||
return "s3"
|
||||
return clean
|
||||
|
||||
|
||||
def _redacted_endpoint(value: str, *, default_port: int) -> dict[str, object]:
|
||||
try:
|
||||
parsed = urlsplit(value)
|
||||
host = parsed.hostname
|
||||
if not parsed.scheme or not host:
|
||||
return {"reference": "unresolved"}
|
||||
port = parsed.port or default_port
|
||||
except ValueError:
|
||||
return {"reference": "unresolved"}
|
||||
return {"scheme": parsed.scheme, "host": host, "port": port}
|
||||
|
||||
|
||||
def _text(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _receipt_error(error: str, *, object_ref: str) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_invalid",
|
||||
message=f"The deployment capability receipt is invalid: {error}",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Repair or regenerate the deployment receipt before importing Files configuration.",
|
||||
)
|
||||
|
||||
|
||||
def _invalid(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
message: str,
|
||||
) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_configuration_payload_invalid",
|
||||
message=message,
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||
resolution="Review the Files configuration-package fragment and rerun preflight.",
|
||||
)
|
||||
|
||||
|
||||
def _unsupported(fragment: ConfigurationPackageFragment) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="fragment_type_unsupported",
|
||||
message=f"Files configuration does not support fragment type {fragment.fragment_type!r}.",
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||
)
|
||||
|
||||
|
||||
def _blocked_plan(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
summary: str,
|
||||
) -> ConfigurationPlanItem:
|
||||
return ConfigurationPlanItem(
|
||||
action="blocked",
|
||||
module_id="files",
|
||||
fragment_type=fragment.fragment_type,
|
||||
fragment_id=fragment.fragment_id,
|
||||
summary=summary,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["FILES_CONFIGURATION_CAPABILITY", "FilesConfigurationProvider"]
|
||||
@@ -4,10 +4,23 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, JSON, String, Text, UniqueConstraint, text
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
JSON,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
event,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from govoplan_core.db.base import Base, TimestampMixin
|
||||
from govoplan_files.backend.storage.provenance import source_identity_hash, source_provenance_from_metadata
|
||||
|
||||
|
||||
def new_uuid() -> str:
|
||||
@@ -33,17 +46,29 @@ class FileBlob(Base, TimestampMixin):
|
||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
protection_discriminator: Mapped[str] = mapped_column(String(320), default="plaintext", nullable=False, index=True)
|
||||
encryption_envelope_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||
storage_checksum_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
protection_discriminator: Mapped[str] = mapped_column(
|
||||
String(320), default="plaintext", nullable=False, index=True
|
||||
)
|
||||
encryption_envelope_id: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True, index=True
|
||||
)
|
||||
storage_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
storage_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
integrity_status: Mapped[str] = mapped_column(String(30), default="unchecked", nullable=False, index=True)
|
||||
integrity_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
integrity_status: Mapped[str] = mapped_column(
|
||||
String(30), default="unchecked", nullable=False, index=True
|
||||
)
|
||||
integrity_checked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
integrity_failure: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
quarantined_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
quarantined_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileIntegrityScan(Base, TimestampMixin):
|
||||
@@ -53,21 +78,35 @@ class FileIntegrityScan(Base, TimestampMixin):
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||
storage_prefix: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(30), default="pending", nullable=False, index=True)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30), default="pending", nullable=False, index=True
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
phase: Mapped[str] = mapped_column(String(30), default="blobs", nullable=False)
|
||||
verify_checksums: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
verify_checksums: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False
|
||||
)
|
||||
batch_size: Mapped[int] = mapped_column(Integer, default=100, nullable=False)
|
||||
blob_cursor: Mapped[str | None] = mapped_column(String(36), nullable=True)
|
||||
object_cursor: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
scanned_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
verified_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
quarantined_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
scanned_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
quarantined_blob_count: Mapped[int] = mapped_column(
|
||||
Integer, default=0, nullable=False
|
||||
)
|
||||
scanned_object_count: Mapped[int] = mapped_column(
|
||||
Integer, default=0, nullable=False
|
||||
)
|
||||
orphan_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
started_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
completed_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
last_error: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
|
||||
@@ -78,19 +117,35 @@ class FileIntegrityFinding(Base, TimestampMixin):
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
scan_id: Mapped[str] = mapped_column(ForeignKey("file_integrity_scans.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
scan_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_integrity_scans.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
kind: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||
state: Mapped[str] = mapped_column(String(30), default="open", nullable=False, index=True)
|
||||
state: Mapped[str] = mapped_column(
|
||||
String(30), default="open", nullable=False, index=True
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
blob_id: Mapped[str | None] = mapped_column(ForeignKey("file_blobs.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
blob_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
expected_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
observed_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
expected_checksum_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
observed_checksum_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
resolved_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
expected_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
observed_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
resolved_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileFolder(Base, TimestampMixin):
|
||||
@@ -98,14 +153,18 @@ class FileFolder(Base, TimestampMixin):
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"uq_file_folders_active_user_path",
|
||||
"tenant_id", "owner_user_id", "path",
|
||||
"tenant_id",
|
||||
"owner_user_id",
|
||||
"path",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
),
|
||||
Index(
|
||||
"uq_file_folders_active_group_path",
|
||||
"tenant_id", "owner_group_id", "path",
|
||||
"tenant_id",
|
||||
"owner_group_id",
|
||||
"path",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
@@ -115,62 +174,196 @@ class FileFolder(Base, TimestampMixin):
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class FileAsset(Base, TimestampMixin):
|
||||
__tablename__ = "file_assets"
|
||||
__table_args__ = (
|
||||
Index("ix_file_assets_user_source", "tenant_id", "owner_type", "owner_user_id", "source_identity_hash"),
|
||||
Index("ix_file_assets_group_source", "tenant_id", "owner_type", "owner_group_id", "source_identity_hash"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
current_version_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
current_version_id: Mapped[str | None] = mapped_column(
|
||||
String(36), nullable=True, index=True
|
||||
)
|
||||
display_path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||
filename: Mapped[str] = mapped_column(String(500), nullable=False, index=True)
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
retained_until: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
legal_hold: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, nullable=False, index=True
|
||||
)
|
||||
lifecycle_revision: Mapped[int] = mapped_column(
|
||||
Integer, default=1, nullable=False
|
||||
)
|
||||
lifecycle_reason: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
source_identity_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
|
||||
@event.listens_for(FileAsset, "before_insert")
|
||||
@event.listens_for(FileAsset, "before_update")
|
||||
def _refresh_asset_source_identity(_mapper: object, _connection: object, asset: FileAsset) -> None:
|
||||
# All owning metadata writes (including copy/restore) use ORM instances.
|
||||
# Direct/bulk SQL metadata writers must also update this derived column.
|
||||
asset.source_identity_hash = source_identity_hash(source_provenance_from_metadata(asset.metadata_))
|
||||
|
||||
|
||||
class FileVersion(Base, TimestampMixin):
|
||||
__tablename__ = "file_versions"
|
||||
__table_args__ = (UniqueConstraint("file_asset_id", "version_number", name="uq_file_versions_asset_number"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"file_asset_id", "version_number", name="uq_file_versions_asset_number"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
blob_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
version_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
filename_at_upload: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
display_path_at_upload: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileFormEvidenceGrant(Base, TimestampMixin):
|
||||
__tablename__ = "file_form_evidence_grants"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_file_form_evidence_grants_idempotency",
|
||||
),
|
||||
Index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
"tenant_id",
|
||||
"form_instance_id",
|
||||
"form_definition_id",
|
||||
"form_definition_revision",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
form_instance_id: Mapped[str] = mapped_column(
|
||||
String(36), nullable=False, index=True
|
||||
)
|
||||
form_definition_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
form_definition_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
token_sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
request_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
custodian_user_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="RESTRICT"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
evidence_kind: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
purpose: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30), default="issued", nullable=False, index=True
|
||||
)
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
max_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
allowed_content_types: Mapped[list[str]] = mapped_column(
|
||||
JSON, default=list, nullable=False
|
||||
)
|
||||
file_asset_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||
)
|
||||
file_version_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||
)
|
||||
uploaded_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any]] = mapped_column(
|
||||
"metadata", JSON, default=dict, nullable=False
|
||||
)
|
||||
|
||||
|
||||
class FileShare(Base, TimestampMixin):
|
||||
__tablename__ = "file_shares"
|
||||
__table_args__ = (UniqueConstraint("file_asset_id", "target_type", "target_id", "revoked_at", name="uq_file_shares_active_target"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"file_asset_id",
|
||||
"target_type",
|
||||
"target_id",
|
||||
"revoked_at",
|
||||
name="uq_file_shares_active_target",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
target_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
revoked_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
revoked_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorProfile(Base, TimestampMixin):
|
||||
@@ -181,15 +374,23 @@ class FileConnectorProfile(Base, TimestampMixin):
|
||||
|
||||
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(String(20), default="tenant", nullable=False, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(
|
||||
String(20), default="tenant", nullable=False, index=True
|
||||
)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||
endpoint_url: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
base_path: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False, index=True)
|
||||
credential_profile_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||
credential_mode: Mapped[str] = mapped_column(String(30), default="none", nullable=False)
|
||||
enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
credential_profile_id: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True, index=True
|
||||
)
|
||||
credential_mode: Mapped[str] = mapped_column(
|
||||
String(30), default="none", nullable=False
|
||||
)
|
||||
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
@@ -198,9 +399,15 @@ class FileConnectorProfile(Base, TimestampMixin):
|
||||
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
capabilities: Mapped[list[str] | None] = mapped_column(JSON, nullable=True)
|
||||
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorCredential(Base, TimestampMixin):
|
||||
@@ -211,12 +418,18 @@ class FileConnectorCredential(Base, TimestampMixin):
|
||||
|
||||
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(String(20), default="tenant", nullable=False, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(
|
||||
String(20), default="tenant", nullable=False, index=True
|
||||
)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
provider: Mapped[str | None] = mapped_column(String(50), nullable=True, index=True)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False, index=True)
|
||||
credential_mode: Mapped[str] = mapped_column(String(30), default="none", nullable=False)
|
||||
enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
credential_mode: Mapped[str] = mapped_column(
|
||||
String(30), default="none", nullable=False
|
||||
)
|
||||
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
@@ -224,15 +437,26 @@ class FileConnectorCredential(Base, TimestampMixin):
|
||||
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorPolicy(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_policies"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("tenant_id", "scope_type", "scope_id", name="uq_file_connector_policies_scope"),
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
name="uq_file_connector_policies_scope",
|
||||
),
|
||||
Index("ix_file_connector_policies_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
@@ -241,24 +465,38 @@ class FileConnectorPolicy(Base, TimestampMixin):
|
||||
scope_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
policy: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorSpace(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_spaces"
|
||||
__table_args__ = (
|
||||
Index("ix_file_connector_spaces_owner", "tenant_id", "owner_type", "owner_user_id", "owner_group_id"),
|
||||
Index(
|
||||
"ix_file_connector_spaces_owner",
|
||||
"tenant_id",
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
),
|
||||
Index(
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"tenant_id", "owner_user_id", "label",
|
||||
"tenant_id",
|
||||
"owner_user_id",
|
||||
"label",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
),
|
||||
Index(
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"tenant_id", "owner_group_id", "label",
|
||||
"tenant_id",
|
||||
"owner_group_id",
|
||||
"label",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
@@ -268,41 +506,81 @@ class FileConnectorSpace(Base, TimestampMixin):
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
connector_profile_id: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
|
||||
connector_profile_id: Mapped[str] = mapped_column(
|
||||
String(255), nullable=False, index=True
|
||||
)
|
||||
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||
library_id: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
remote_path: Mapped[str] = mapped_column(String(1000), default="", nullable=False)
|
||||
sync_mode: Mapped[str] = mapped_column(String(30), default="manual", nullable=False)
|
||||
read_only: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
is_active: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class CampaignAttachmentUse(Base, TimestampMixin):
|
||||
__tablename__ = "campaign_attachment_uses"
|
||||
__table_args__ = (UniqueConstraint("campaign_job_id", "file_version_id", "filename_used", "use_stage", name="uq_campaign_attachment_uses_job_file_stage"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"campaign_job_id",
|
||||
"file_version_id",
|
||||
"filename_used",
|
||||
"use_stage",
|
||||
name="uq_campaign_attachment_uses_job_file_stage",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
campaign_id: Mapped[str] = mapped_column(ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_version_id: Mapped[str] = mapped_column(ForeignKey("campaign_versions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_job_id: Mapped[str | None] = mapped_column(ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
campaign_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
campaign_version_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("campaign_versions.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
campaign_job_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
entry_index: Mapped[int | None] = mapped_column(Integer)
|
||||
entry_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_version_id: Mapped[str] = mapped_column(ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
file_version_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
file_blob_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
filename_used: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
use_stage: Mapped[str] = mapped_column(String(20), default="built", nullable=False, index=True)
|
||||
used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
use_stage: Mapped[str] = mapped_column(
|
||||
String(20), default="built", nullable=False, index=True
|
||||
)
|
||||
used_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
@@ -314,6 +592,7 @@ __all__ = [
|
||||
"FileConnectorProfile",
|
||||
"FileConnectorSpace",
|
||||
"FileFolder",
|
||||
"FileFormEvidenceGrant",
|
||||
"FileShare",
|
||||
"FileVersion",
|
||||
]
|
||||
|
||||
@@ -179,8 +179,34 @@ def _archive_topic(
|
||||
f"The deployment accepts ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ requests up to {request_limit}, limits actual expanded data to {expanded_limit}, "
|
||||
f"each member to {member_limit}, the archive to {max_entries:,} entries, and expansion to {max_expansion_ratio}:1. "
|
||||
f"The server-issued preview expires after {preview_minutes} minutes. Password-protected ZIP archives are supported; passwords remain request-only. "
|
||||
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers."
|
||||
"The UI uploads the archive once into bounded private temporary storage; password verification and confirmation reuse that staged upload without another transfer. "
|
||||
"Repreviewing does not extend its original lifetime. Cancel, replacing the archive and successful confirmation release the stage; an expired or missing stage requires a new upload. "
|
||||
"A requested preview may safely reupload the selected file if its stage expired; confirmation is never retried automatically. "
|
||||
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers. "
|
||||
"Archive paths are limited to 4096 UTF-8 bytes and 128 components; derived parent directories count toward the entry limit. TAR inspection checks each header's size and ratio before advancing across that member's payload. "
|
||||
"Metadata parsing and decoding run in disposable resource-limited children, including native ZIP decoding. Preview allows 120 seconds and confirmation 600 seconds; busy workers and CPU, memory, time or transport limits fail closed. Split a rejected archive or explicitly retry later; confirmation is not automatically retried. "
|
||||
"The shared blurred dialog overlay separates upload transfer, inspection, extraction/storage and final commit, without an envelope animation. "
|
||||
"Selected file/byte totals appear immediately and extraction progress uses actual server counters. Unknown progress is indeterminate; completion is shown only after the transaction succeeds. "
|
||||
"Keep the dialog open while processing; errors release the overlay and preserve the selection for review."
|
||||
),
|
||||
translations={"de": {
|
||||
"title": "Ein Archiv prüfen und entpacken",
|
||||
"summary": f"Bis zu {max_entries:,} Einträge aus ZIP- oder TAR-Archiven prüfen und gezielt entpacken, bevor verwaltete Dateien entstehen.",
|
||||
"body": (
|
||||
f"Diese Installation erlaubt ZIP, TAR, TAR.GZ, TAR.BZ2 und TAR.XZ bis {request_limit}, höchstens {expanded_limit} tatsächlich entpackte Daten, "
|
||||
f"{member_limit} pro Datei, {max_entries:,} Einträge und ein Expansionsverhältnis von {max_expansion_ratio}:1. "
|
||||
f"Die serverseitige Vorschau läuft nach {preview_minutes} Minuten ab. Passwortgeschützte ZIP-Archive werden unterstützt; Passwörter bleiben auf die Anfrage beschränkt. "
|
||||
"Die Oberfläche überträgt das Archiv einmal in begrenzten privaten Zwischenspeicher; Passwortprüfung und Bestätigung verwenden diese temporäre Kopie ohne erneute Übertragung. "
|
||||
"Eine erneute Vorschau verlängert die ursprüngliche Laufzeit nicht. Abbrechen, Archivwechsel und erfolgreicher Abschluss geben die Kopie frei; fehlt sie oder ist sie abgelaufen, muss das Archiv erneut hochgeladen werden. "
|
||||
"Bei einer angeforderten Vorschau darf die Oberfläche die ausgewählte Datei nach Ablauf erneut übertragen; die Bestätigung wird niemals automatisch wiederholt. "
|
||||
"Unsichere Pfade und besondere Dateisystemeinträge werden abgelehnt; tatsächlich gelesene Bytes werden gezählt. "
|
||||
"Archivpfade sind auf 4096 UTF-8-Bytes und 128 Komponenten begrenzt; abgeleitete übergeordnete Ordner zählen zur Eintragsgrenze. Die TAR-Prüfung kontrolliert Größe und Expansionsverhältnis bei jedem Header, bevor dessen Nutzdaten übersprungen werden. "
|
||||
"Metadatenprüfung und Dekodieren einschließlich nativem ZIP laufen in kurzlebigen ressourcenbegrenzten Kindprozessen. Für die Vorschau gelten 120 Sekunden, für die Bestätigung 600 Sekunden; ausgelastete Worker sowie CPU-, Speicher-, Zeit- oder Transportgrenzen führen zum sicheren Abbruch. Ein abgelehntes Archiv aufteilen oder später ausdrücklich erneut versuchen; die Bestätigung wird nicht automatisch wiederholt. "
|
||||
"Die gemeinsame Überlagerung des weichgezeichneten Dialogs unterscheidet Übertragung, Prüfung, Entpacken/Speichern und abschließende Transaktionsbestätigung, ohne Briefumschlaganimation. "
|
||||
"Ausgewählte Datei- und Bytezahlen erscheinen sofort; der Entpackfortschritt nutzt tatsächliche Serverzähler. Unbekannter Fortschritt bleibt unbestimmt, der Abschluss wird erst nach erfolgreicher Transaktion angezeigt. "
|
||||
"Den Dialog während der Verarbeitung geöffnet lassen; Fehler entfernen die Überlagerung und erhalten die Auswahl zur Prüfung."
|
||||
),
|
||||
}},
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "file_manager", "process_participant"),
|
||||
@@ -238,7 +264,7 @@ def _archive_topic(
|
||||
{
|
||||
"id": "archive-entry-count",
|
||||
"label": "Maximum entry count",
|
||||
"description": f"An archive may contain at most {max_entries:,} declared entries.",
|
||||
"description": f"An archive may contain at most {max_entries:,} entries, including derived parent directories.",
|
||||
"values": [f"{max_entries:,} entries"],
|
||||
},
|
||||
{
|
||||
@@ -252,6 +278,7 @@ def _archive_topic(
|
||||
"files.workflow.upload-managed-files",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.find-and-download-files",
|
||||
"files.reference.shared-storage-profile",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
@@ -0,0 +1,879 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import inspect, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import (
|
||||
DsarErasureActionRef,
|
||||
DsarExecutionResultRef,
|
||||
DsarRecordRef,
|
||||
DsarSubjectRef,
|
||||
dsar_capability_name,
|
||||
)
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
|
||||
|
||||
FILES_DSAR_CAPABILITY = dsar_capability_name("files")
|
||||
_MAX_RECORDS = 5_000
|
||||
|
||||
|
||||
class FilesDsarProvider:
|
||||
provider_id = "files"
|
||||
module_id = "files"
|
||||
|
||||
def search_subject(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
) -> Sequence[DsarRecordRef]:
|
||||
db = _session(session)
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return ()
|
||||
|
||||
records: list[DsarRecordRef] = []
|
||||
form_evidence_available = _has_table(db, FileFormEvidenceGrant)
|
||||
campaign_evidence_available = _has_table(db, CampaignAttachmentUse)
|
||||
retention_reasons: dict[str, str | None] = {}
|
||||
|
||||
def retention_reason(asset: FileAsset) -> str | None:
|
||||
if asset.id not in retention_reasons:
|
||||
retention_reasons[asset.id] = _asset_retention_reason(
|
||||
db,
|
||||
asset,
|
||||
form_evidence_available=form_evidence_available,
|
||||
campaign_evidence_available=campaign_evidence_available,
|
||||
)
|
||||
return retention_reasons[asset.id]
|
||||
|
||||
def append(record: DsarRecordRef) -> None:
|
||||
if len(records) >= _MAX_RECORDS:
|
||||
raise ValueError(
|
||||
"Files DSAR match limit exceeded; narrow the subject selectors."
|
||||
)
|
||||
records.append(record)
|
||||
|
||||
assets = _bounded_rows(
|
||||
db.query(FileAsset)
|
||||
.filter(
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
FileAsset.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileAsset.id)
|
||||
)
|
||||
for asset in assets:
|
||||
match_fields = _matching_fields(
|
||||
asset,
|
||||
subject_user_id,
|
||||
("owner_user_id", "created_by_user_id"),
|
||||
)
|
||||
asset_retention_reason = retention_reason(asset)
|
||||
append(
|
||||
_record(
|
||||
"file_asset",
|
||||
asset.id,
|
||||
"managed_file",
|
||||
asset.filename,
|
||||
{
|
||||
"match_fields": match_fields,
|
||||
"owner_type": asset.owner_type,
|
||||
"display_path": asset.display_path,
|
||||
"filename": asset.filename,
|
||||
"description": asset.description,
|
||||
"deleted_at": _iso(asset.deleted_at),
|
||||
"retained_until": _iso(asset.retained_until),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_reason": asset.lifecycle_reason,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
},
|
||||
observed_at=asset.updated_at,
|
||||
immutable=asset_retention_reason is not None,
|
||||
retention_reason=asset_retention_reason,
|
||||
source_path=f"/files?file={asset.id}",
|
||||
)
|
||||
)
|
||||
|
||||
version_rows = _bounded_rows(
|
||||
db.query(FileVersion, FileAsset)
|
||||
.join(FileAsset, FileAsset.id == FileVersion.file_asset_id)
|
||||
.filter(
|
||||
FileVersion.tenant_id == tenant_id,
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
FileVersion.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileVersion.id)
|
||||
)
|
||||
for version, asset in version_rows:
|
||||
asset_retention_reason = retention_reason(asset)
|
||||
append(
|
||||
_record(
|
||||
"file_version",
|
||||
version.id,
|
||||
"managed_file_version",
|
||||
version.filename_at_upload,
|
||||
{
|
||||
"match_fields": (
|
||||
["asset.owner_user_id"]
|
||||
if asset.owner_user_id == subject_user_id
|
||||
else []
|
||||
)
|
||||
+ (
|
||||
["created_by_user_id"]
|
||||
if version.created_by_user_id == subject_user_id
|
||||
else []
|
||||
),
|
||||
"file_asset_id": version.file_asset_id,
|
||||
"version_number": version.version_number,
|
||||
"filename_at_upload": version.filename_at_upload,
|
||||
"display_path_at_upload": version.display_path_at_upload,
|
||||
"content_type": version.content_type,
|
||||
"size_bytes": version.size_bytes,
|
||||
"checksum_sha256": version.checksum_sha256,
|
||||
"created_at": _iso(version.created_at),
|
||||
},
|
||||
observed_at=version.updated_at,
|
||||
immutable=asset_retention_reason is not None,
|
||||
retention_reason=asset_retention_reason,
|
||||
source_path=(
|
||||
f"/api/v1/files/{version.file_asset_id}/versions/"
|
||||
f"{version.id}/download"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for folder in _bounded_rows(
|
||||
db.query(FileFolder)
|
||||
.filter(
|
||||
FileFolder.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileFolder.owner_user_id == subject_user_id,
|
||||
FileFolder.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileFolder.id)
|
||||
):
|
||||
append(
|
||||
_record(
|
||||
"file_folder",
|
||||
folder.id,
|
||||
"managed_folder",
|
||||
folder.path,
|
||||
{
|
||||
"match_fields": _matching_fields(
|
||||
folder,
|
||||
subject_user_id,
|
||||
("owner_user_id", "created_by_user_id"),
|
||||
),
|
||||
"owner_type": folder.owner_type,
|
||||
"path": folder.path,
|
||||
"deleted_at": _iso(folder.deleted_at),
|
||||
},
|
||||
observed_at=folder.updated_at,
|
||||
source_path="/files",
|
||||
)
|
||||
)
|
||||
|
||||
for share in _bounded_rows(
|
||||
db.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
or_(
|
||||
(FileShare.target_type == "user")
|
||||
& (FileShare.target_id == subject_user_id),
|
||||
FileShare.created_by_user_id == subject_user_id,
|
||||
FileShare.revoked_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileShare.id)
|
||||
):
|
||||
target_matches = (
|
||||
share.target_type == "user" and share.target_id == subject_user_id
|
||||
)
|
||||
match_fields = _matching_fields(
|
||||
share,
|
||||
subject_user_id,
|
||||
("created_by_user_id", "revoked_by_user_id"),
|
||||
)
|
||||
if target_matches:
|
||||
match_fields.insert(0, "target_id")
|
||||
append(
|
||||
_record(
|
||||
"file_share",
|
||||
share.id,
|
||||
"file_access_evidence",
|
||||
f"Share for file {share.file_asset_id}",
|
||||
{
|
||||
"match_fields": match_fields,
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_is_subject": target_matches,
|
||||
"permission": share.permission,
|
||||
"expires_at": _iso(share.expires_at),
|
||||
"revoked_at": _iso(share.revoked_at),
|
||||
},
|
||||
observed_at=share.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"File-sharing history is institutional access evidence."
|
||||
),
|
||||
source_path=f"/files?file={share.file_asset_id}",
|
||||
)
|
||||
)
|
||||
|
||||
if form_evidence_available:
|
||||
evidence_rows = _bounded_rows(
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.outerjoin(
|
||||
FileAsset,
|
||||
FileAsset.id == FileFormEvidenceGrant.file_asset_id,
|
||||
)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileFormEvidenceGrant.custodian_user_id == subject_user_id,
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileFormEvidenceGrant.id)
|
||||
)
|
||||
for grant in evidence_rows:
|
||||
append(
|
||||
_record(
|
||||
"file_form_evidence",
|
||||
grant.id,
|
||||
"form_evidence",
|
||||
f"Form evidence {grant.form_definition_id}",
|
||||
{
|
||||
"match_fields": (
|
||||
["custodian_user_id"]
|
||||
if grant.custodian_user_id == subject_user_id
|
||||
else ["asset.owner_user_id"]
|
||||
),
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"form_definition_id": grant.form_definition_id,
|
||||
"form_definition_revision": grant.form_definition_revision,
|
||||
"evidence_kind": grant.evidence_kind,
|
||||
"purpose": grant.purpose,
|
||||
"status": grant.status,
|
||||
"expires_at": _iso(grant.expires_at),
|
||||
"file_asset_id": grant.file_asset_id,
|
||||
"file_version_id": grant.file_version_id,
|
||||
},
|
||||
observed_at=grant.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Submitted Form attachment evidence follows the owning "
|
||||
"process retention and cannot be erased through Files alone."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
if campaign_evidence_available:
|
||||
campaign_rows = _bounded_rows(
|
||||
db.query(CampaignAttachmentUse)
|
||||
.join(FileAsset, FileAsset.id == CampaignAttachmentUse.file_asset_id)
|
||||
.filter(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(CampaignAttachmentUse.id)
|
||||
)
|
||||
for use in campaign_rows:
|
||||
append(
|
||||
_record(
|
||||
"campaign_attachment_use",
|
||||
use.id,
|
||||
"delivery_evidence",
|
||||
use.filename_used,
|
||||
{
|
||||
"match_fields": ["asset.owner_user_id"],
|
||||
"campaign_id": use.campaign_id,
|
||||
"campaign_version_id": use.campaign_version_id,
|
||||
"campaign_job_id": use.campaign_job_id,
|
||||
"file_asset_id": use.file_asset_id,
|
||||
"file_version_id": use.file_version_id,
|
||||
"filename_used": use.filename_used,
|
||||
"checksum_sha256": use.checksum_sha256,
|
||||
"size_bytes": use.size_bytes,
|
||||
"use_stage": use.use_stage,
|
||||
"used_at": _iso(use.used_at),
|
||||
},
|
||||
observed_at=use.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Campaign attachment use is immutable delivery evidence."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
self._append_configuration_references(
|
||||
db,
|
||||
append=append,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
)
|
||||
self._append_integrity_references(
|
||||
db,
|
||||
append=append,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
)
|
||||
return tuple(records)
|
||||
|
||||
def plan_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
records: Sequence[DsarRecordRef],
|
||||
) -> Sequence[DsarErasureActionRef]:
|
||||
del session
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return ()
|
||||
actions: list[DsarErasureActionRef] = []
|
||||
for record in records:
|
||||
if record.provider_id != self.provider_id or record.module_id != self.module_id:
|
||||
raise ValueError("Files DSAR received a foreign provider record.")
|
||||
match_fields = {
|
||||
str(value) for value in record.data.get("match_fields", ())
|
||||
}
|
||||
if record.immutable_evidence:
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:retain:{record.resource_type}:{record.resource_id}",
|
||||
"retain",
|
||||
record,
|
||||
f"Retain {record.title}",
|
||||
record.retention_reason
|
||||
or "Institutional evidence must be retained.",
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
elif record.resource_type in {
|
||||
"file_asset",
|
||||
"file_version",
|
||||
"file_folder",
|
||||
}:
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:review:{record.resource_type}:{record.resource_id}",
|
||||
"manual_review",
|
||||
record,
|
||||
f"Review {record.title}",
|
||||
(
|
||||
"Managed file content, names, paths, ownership, and shared "
|
||||
"references require a case decision. Approved byte erasure "
|
||||
"must use the separate Files purge workflow."
|
||||
),
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
if (
|
||||
record.resource_type == "file_share"
|
||||
and "target_id" in match_fields
|
||||
and record.data.get("revoked_at") is None
|
||||
):
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:revoke:file_share:{record.resource_id}",
|
||||
"revoke",
|
||||
record,
|
||||
"Revoke active file share",
|
||||
"The active user-targeted share can be revoked without deleting file evidence.",
|
||||
executable=True,
|
||||
metadata={"subject_user_id": subject_user_id},
|
||||
)
|
||||
)
|
||||
for field_name in sorted(
|
||||
match_fields.intersection(_detachable_fields(record.resource_type))
|
||||
):
|
||||
actions.append(
|
||||
_action(
|
||||
(
|
||||
f"files:detach:{record.resource_type}:"
|
||||
f"{field_name}:{record.resource_id}"
|
||||
),
|
||||
"detach",
|
||||
record,
|
||||
f"Detach {field_name.replace('_', ' ')}",
|
||||
(
|
||||
"Remove the mutable subject reference while preserving the "
|
||||
"governed resource and DSAR evidence."
|
||||
),
|
||||
executable=True,
|
||||
metadata={
|
||||
"field": field_name,
|
||||
"subject_user_id": subject_user_id,
|
||||
},
|
||||
)
|
||||
)
|
||||
action_ids = [action.action_id for action in actions]
|
||||
if len(action_ids) != len(set(action_ids)):
|
||||
raise ValueError("Files DSAR produced duplicate action ids.")
|
||||
return tuple(actions)
|
||||
|
||||
def execute_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
actions: Sequence[DsarErasureActionRef],
|
||||
request_id: str,
|
||||
) -> Sequence[DsarExecutionResultRef]:
|
||||
db = _session(session)
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return tuple(
|
||||
_blocked(action, "Files requires a direct membership subject reference.")
|
||||
for action in actions
|
||||
)
|
||||
results: list[DsarExecutionResultRef] = []
|
||||
for action in actions:
|
||||
if (
|
||||
action.provider_id != self.provider_id
|
||||
or action.module_id != self.module_id
|
||||
or action.metadata.get("subject_user_id") != subject_user_id
|
||||
):
|
||||
results.append(_blocked(action, "The Files DSAR action is stale or invalid."))
|
||||
continue
|
||||
if action.action_id.startswith("files:revoke:file_share:"):
|
||||
results.append(
|
||||
_revoke_share(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
action=action,
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
elif action.action_id.startswith("files:detach:"):
|
||||
results.append(
|
||||
_detach_reference(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
action=action,
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
else:
|
||||
results.append(_blocked(action, "Files does not execute this action kind."))
|
||||
db.flush()
|
||||
return tuple(results)
|
||||
|
||||
def _append_configuration_references(
|
||||
self,
|
||||
db: Session,
|
||||
*,
|
||||
append: object,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
) -> None:
|
||||
configurations = (
|
||||
(FileConnectorProfile, "connector_profile", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorCredential, "connector_credential", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorPolicy, "connector_policy", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorSpace, "connector_space", ("owner_user_id", "created_by_user_id")),
|
||||
)
|
||||
for model, resource_type, fields in configurations:
|
||||
if not _has_table(db, model):
|
||||
continue
|
||||
conditions = [getattr(model, field) == subject_user_id for field in fields]
|
||||
query = db.query(model).filter(or_(*conditions))
|
||||
if hasattr(model, "tenant_id"):
|
||||
query = query.filter(model.tenant_id == tenant_id)
|
||||
for row in _bounded_rows(query.order_by(model.id)):
|
||||
match_fields = _matching_fields(row, subject_user_id, fields)
|
||||
data: dict[str, object] = {
|
||||
"match_fields": match_fields,
|
||||
"label": getattr(row, "label", None),
|
||||
"provider": getattr(row, "provider", None),
|
||||
}
|
||||
if isinstance(row, FileConnectorCredential):
|
||||
data["credential_mode"] = row.credential_mode
|
||||
elif isinstance(row, FileConnectorSpace):
|
||||
data.update(
|
||||
{
|
||||
"remote_path": row.remote_path,
|
||||
"sync_mode": row.sync_mode,
|
||||
"read_only": row.read_only,
|
||||
"deleted_at": _iso(row.deleted_at),
|
||||
}
|
||||
)
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
resource_type,
|
||||
row.id,
|
||||
"connector_configuration_evidence",
|
||||
getattr(row, "label", None) or resource_type.replace("_", " "),
|
||||
data,
|
||||
observed_at=row.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Connector configuration history is institutional evidence; "
|
||||
"credential secrets are excluded from the DSAR export."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def _append_integrity_references(
|
||||
self,
|
||||
db: Session,
|
||||
*,
|
||||
append: object,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
) -> None:
|
||||
if _has_table(db, FileIntegrityScan):
|
||||
for scan in _bounded_rows(
|
||||
db.query(FileIntegrityScan)
|
||||
.filter(
|
||||
FileIntegrityScan.tenant_id == tenant_id,
|
||||
FileIntegrityScan.created_by_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(FileIntegrityScan.id)
|
||||
):
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
"file_integrity_scan",
|
||||
scan.id,
|
||||
"storage_integrity_evidence",
|
||||
f"Integrity scan {scan.id}",
|
||||
{
|
||||
"match_fields": ["created_by_user_id"],
|
||||
"storage_backend": scan.storage_backend,
|
||||
"status": scan.status,
|
||||
"started_at": _iso(scan.started_at),
|
||||
"completed_at": _iso(scan.completed_at),
|
||||
},
|
||||
observed_at=scan.updated_at,
|
||||
immutable=True,
|
||||
retention_reason="Storage integrity scans are operator evidence.",
|
||||
)
|
||||
)
|
||||
if _has_table(db, FileIntegrityFinding):
|
||||
for finding in _bounded_rows(
|
||||
db.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.tenant_id == tenant_id,
|
||||
FileIntegrityFinding.resolved_by_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(FileIntegrityFinding.id)
|
||||
):
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
"file_integrity_finding",
|
||||
finding.id,
|
||||
"storage_integrity_evidence",
|
||||
f"Integrity finding {finding.kind}",
|
||||
{
|
||||
"match_fields": ["resolved_by_user_id"],
|
||||
"kind": finding.kind,
|
||||
"state": finding.state,
|
||||
"resolved_at": _iso(finding.resolved_at),
|
||||
},
|
||||
observed_at=finding.updated_at,
|
||||
immutable=True,
|
||||
retention_reason="Integrity resolution is operator evidence.",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _asset_retention_reason(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
form_evidence_available: bool,
|
||||
campaign_evidence_available: bool,
|
||||
) -> str | None:
|
||||
reasons: list[str] = []
|
||||
if asset.legal_hold:
|
||||
reasons.append("The file is under legal hold.")
|
||||
retained_until = _aware(asset.retained_until)
|
||||
if retained_until is not None and retained_until > datetime.now(timezone.utc):
|
||||
reasons.append(f"The file is retained until {retained_until.isoformat()}.")
|
||||
if form_evidence_available:
|
||||
has_form_evidence = (
|
||||
session.query(FileFormEvidenceGrant.id)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == asset.tenant_id,
|
||||
FileFormEvidenceGrant.file_asset_id == asset.id,
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
if has_form_evidence:
|
||||
reasons.append("The file is referenced by submitted Form evidence.")
|
||||
if campaign_evidence_available:
|
||||
has_campaign_evidence = (
|
||||
session.query(CampaignAttachmentUse.id)
|
||||
.filter(
|
||||
CampaignAttachmentUse.tenant_id == asset.tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == asset.id,
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
if has_campaign_evidence:
|
||||
reasons.append("The file is referenced by Campaign delivery evidence.")
|
||||
return " ".join(reasons) or None
|
||||
|
||||
|
||||
_DETACHABLE_MODELS: dict[str, tuple[type[object], frozenset[str]]] = {
|
||||
"file_asset": (FileAsset, frozenset({"created_by_user_id"})),
|
||||
"file_version": (FileVersion, frozenset({"created_by_user_id"})),
|
||||
"file_folder": (FileFolder, frozenset({"created_by_user_id"})),
|
||||
"file_share": (
|
||||
FileShare,
|
||||
frozenset({"created_by_user_id", "revoked_by_user_id"}),
|
||||
),
|
||||
"connector_profile": (
|
||||
FileConnectorProfile,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_credential": (
|
||||
FileConnectorCredential,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_policy": (
|
||||
FileConnectorPolicy,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_space": (FileConnectorSpace, frozenset({"created_by_user_id"})),
|
||||
}
|
||||
|
||||
|
||||
def _detachable_fields(resource_type: str) -> frozenset[str]:
|
||||
entry = _DETACHABLE_MODELS.get(resource_type)
|
||||
return entry[1] if entry is not None else frozenset()
|
||||
|
||||
|
||||
def _revoke_share(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
action: DsarErasureActionRef,
|
||||
request_id: str,
|
||||
) -> DsarExecutionResultRef:
|
||||
row = (
|
||||
session.query(FileShare)
|
||||
.filter(FileShare.id == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if (
|
||||
row is None
|
||||
or row.tenant_id != tenant_id
|
||||
or row.target_type != "user"
|
||||
or row.target_id != subject_user_id
|
||||
):
|
||||
return _blocked(action, "The subject-targeted file share is no longer available.")
|
||||
if row.revoked_at is not None:
|
||||
return _result(
|
||||
action,
|
||||
"unchanged",
|
||||
"The file share was already revoked.",
|
||||
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||
)
|
||||
row.revoked_at = datetime.now(timezone.utc)
|
||||
row.revoked_by_user_id = None
|
||||
return _result(
|
||||
action,
|
||||
"executed",
|
||||
"The subject-targeted file share was revoked.",
|
||||
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||
)
|
||||
|
||||
|
||||
def _detach_reference(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
action: DsarErasureActionRef,
|
||||
request_id: str,
|
||||
) -> DsarExecutionResultRef:
|
||||
entry = _DETACHABLE_MODELS.get(action.resource_type)
|
||||
field_name = str(action.metadata.get("field") or "")
|
||||
if entry is None or field_name not in entry[1]:
|
||||
return _blocked(action, "The requested Files subject reference is not detachable.")
|
||||
model = entry[0]
|
||||
row = (
|
||||
session.query(model)
|
||||
.filter(getattr(model, "id") == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if row is None or getattr(row, "tenant_id", None) != tenant_id:
|
||||
return _blocked(action, "The Files resource is no longer available.")
|
||||
current = getattr(row, field_name)
|
||||
if current is None:
|
||||
return _result(
|
||||
action,
|
||||
"unchanged",
|
||||
"The subject reference was already detached.",
|
||||
{"request_id": request_id, "field": field_name},
|
||||
)
|
||||
if current != subject_user_id:
|
||||
return _blocked(action, "The Files subject reference changed after planning.")
|
||||
setattr(row, field_name, None)
|
||||
return _result(
|
||||
action,
|
||||
"executed",
|
||||
"The mutable subject reference was detached.",
|
||||
{"request_id": request_id, "field": field_name},
|
||||
)
|
||||
|
||||
|
||||
def _subject_user_id(subject: DsarSubjectRef) -> str | None:
|
||||
candidates: list[str] = []
|
||||
if subject.membership_id:
|
||||
candidates.append(subject.membership_id)
|
||||
for key, value in subject.external_references.items():
|
||||
if key in {
|
||||
"files.user",
|
||||
"files.membership",
|
||||
"access.membership",
|
||||
"membership_id",
|
||||
}:
|
||||
candidates.append(value)
|
||||
normalized = {value.strip() for value in candidates if value.strip()}
|
||||
if len(normalized) != 1:
|
||||
return None
|
||||
return normalized.pop()
|
||||
|
||||
|
||||
def _matching_fields(
|
||||
row: object,
|
||||
subject_user_id: str,
|
||||
fields: Sequence[str],
|
||||
) -> list[str]:
|
||||
return [field for field in fields if getattr(row, field) == subject_user_id]
|
||||
|
||||
|
||||
def _record(
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
category: str,
|
||||
title: str,
|
||||
data: dict[str, object],
|
||||
*,
|
||||
observed_at: datetime | None = None,
|
||||
immutable: bool = False,
|
||||
retention_reason: str | None = None,
|
||||
source_path: str | None = None,
|
||||
) -> DsarRecordRef:
|
||||
return DsarRecordRef(
|
||||
provider_id="files",
|
||||
module_id="files",
|
||||
resource_type=resource_type,
|
||||
resource_id=resource_id,
|
||||
category=category,
|
||||
title=title,
|
||||
data=data,
|
||||
observed_at=observed_at,
|
||||
immutable_evidence=immutable,
|
||||
retention_reason=retention_reason,
|
||||
source_path=source_path,
|
||||
)
|
||||
|
||||
|
||||
def _action(
|
||||
action_id: str,
|
||||
kind: str,
|
||||
record: DsarRecordRef,
|
||||
title: str,
|
||||
rationale: str,
|
||||
*,
|
||||
executable: bool,
|
||||
metadata: dict[str, object] | None = None,
|
||||
) -> DsarErasureActionRef:
|
||||
return DsarErasureActionRef(
|
||||
action_id=action_id,
|
||||
provider_id="files",
|
||||
module_id="files",
|
||||
kind=kind, # type: ignore[arg-type]
|
||||
resource_type=record.resource_type,
|
||||
resource_id=record.resource_id,
|
||||
title=title,
|
||||
rationale=rationale,
|
||||
executable=executable,
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
|
||||
def _result(
|
||||
action: DsarErasureActionRef,
|
||||
status: str,
|
||||
summary: str,
|
||||
evidence: dict[str, object] | None = None,
|
||||
) -> DsarExecutionResultRef:
|
||||
return DsarExecutionResultRef(
|
||||
action_id=action.action_id,
|
||||
status=status, # type: ignore[arg-type]
|
||||
summary=summary,
|
||||
evidence=evidence or {},
|
||||
)
|
||||
|
||||
|
||||
def _blocked(action: DsarErasureActionRef, summary: str) -> DsarExecutionResultRef:
|
||||
return _result(action, "blocked", summary)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files DSAR provider requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
def _bounded_rows(query: object) -> list[object]:
|
||||
rows = query.limit(_MAX_RECORDS + 1).all() # type: ignore[attr-defined]
|
||||
if len(rows) > _MAX_RECORDS:
|
||||
raise ValueError("Files DSAR match limit exceeded; narrow the subject selectors.")
|
||||
return rows
|
||||
|
||||
|
||||
def _has_table(session: Session, model: type[object]) -> bool:
|
||||
return inspect(session.connection()).has_table(model.__tablename__)
|
||||
|
||||
|
||||
def _aware(value: datetime | None) -> datetime | None:
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
aware = _aware(value)
|
||||
return aware.isoformat() if aware else None
|
||||
|
||||
|
||||
__all__ = ["FILES_DSAR_CAPABILITY", "FilesDsarProvider"]
|
||||
@@ -0,0 +1,449 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.access import CAPABILITY_ACCESS_DIRECTORY, AccessDirectory
|
||||
from govoplan_core.core.form_evidence import (
|
||||
FormEvidenceContractError,
|
||||
FormEvidenceGrant,
|
||||
FormEvidenceGrantRequest,
|
||||
FormEvidenceInspection,
|
||||
FormEvidenceInspectionRequest,
|
||||
FormEvidenceState,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileFormEvidenceGrant,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.runtime import configure_runtime
|
||||
|
||||
|
||||
PROVIDER_ID = "files"
|
||||
CAPABILITY_FORM_EVIDENCE_FILES = "forms_runtime.evidence.files"
|
||||
MAX_GRANT_TTL = timedelta(minutes=15)
|
||||
|
||||
|
||||
class FilesFormEvidenceProvider:
|
||||
provider_id = PROVIDER_ID
|
||||
|
||||
def __init__(self, registry: object | None, settings: object) -> None:
|
||||
self._registry = registry
|
||||
self._settings = settings
|
||||
|
||||
def supported_kinds(self) -> Sequence[str]:
|
||||
return ("document",)
|
||||
|
||||
def create_upload_grant(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: FormEvidenceGrantRequest,
|
||||
) -> FormEvidenceGrant:
|
||||
db = _session(session)
|
||||
_assert_tenant(principal, request.tenant_id)
|
||||
if request.evidence_kind != "document":
|
||||
raise FormEvidenceContractError(
|
||||
"Files can accept only document evidence for Forms Runtime."
|
||||
)
|
||||
custodian_user_id = _custodian_user_id(request.custodian_ref)
|
||||
self._assert_active_custodian(
|
||||
tenant_id=request.tenant_id,
|
||||
user_id=custodian_user_id,
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
if request.expires_at <= now:
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence upload grant expiry must be in the future."
|
||||
)
|
||||
allowed_content_types = _content_types(request.allowed_content_types)
|
||||
configured_max = int(getattr(self._settings, "file_upload_max_bytes"))
|
||||
max_size_bytes = min(request.max_size_bytes or configured_max, configured_max)
|
||||
request_sha256 = _request_sha256(
|
||||
request,
|
||||
custodian_user_id=custodian_user_id,
|
||||
max_size_bytes=max_size_bytes,
|
||||
allowed_content_types=allowed_content_types,
|
||||
)
|
||||
existing = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.idempotency_key == request.idempotency_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is not None:
|
||||
if not secrets.compare_digest(existing.request_sha256, request_sha256):
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence grant idempotency conflict."
|
||||
)
|
||||
if (
|
||||
existing.status in {"expired", "revoked"}
|
||||
or _aware(existing.expires_at) <= now
|
||||
):
|
||||
raise FormEvidenceContractError(
|
||||
"The existing Form evidence upload grant is no longer usable; "
|
||||
"request a new grant with a new idempotency key."
|
||||
)
|
||||
return _grant_response(existing, upload_token=None, replayed=True)
|
||||
|
||||
metadata = _bounded_metadata(request.metadata)
|
||||
remaining_attachments = metadata.get("remaining_attachments")
|
||||
if isinstance(remaining_attachments, int):
|
||||
existing_attachment_ids = set(metadata.get("existing_attachment_ids", ()))
|
||||
active_grants = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||
FileFormEvidenceGrant.form_definition_id
|
||||
== request.definition_ref.object_id,
|
||||
FileFormEvidenceGrant.form_definition_revision
|
||||
== str(request.definition_ref.version),
|
||||
FileFormEvidenceGrant.status.in_(("issued", "uploaded")),
|
||||
FileFormEvidenceGrant.expires_at > now,
|
||||
)
|
||||
.with_for_update()
|
||||
.all()
|
||||
)
|
||||
outstanding = sum(
|
||||
1
|
||||
for item in active_grants
|
||||
if not item.file_asset_id
|
||||
or item.file_asset_id not in existing_attachment_ids
|
||||
)
|
||||
if outstanding >= remaining_attachments:
|
||||
raise FormEvidenceContractError(
|
||||
"This Form already has the maximum number of active attachment uploads."
|
||||
)
|
||||
|
||||
upload_token = secrets.token_urlsafe(32)
|
||||
grant = FileFormEvidenceGrant(
|
||||
tenant_id=request.tenant_id,
|
||||
form_instance_id=request.instance_id,
|
||||
form_definition_id=request.definition_ref.object_id,
|
||||
form_definition_revision=str(request.definition_ref.version),
|
||||
token_sha256=_token_sha256(upload_token),
|
||||
idempotency_key=request.idempotency_key,
|
||||
request_sha256=request_sha256,
|
||||
custodian_user_id=custodian_user_id,
|
||||
evidence_kind=request.evidence_kind,
|
||||
purpose=request.purpose,
|
||||
status="issued",
|
||||
expires_at=min(request.expires_at, now + MAX_GRANT_TTL),
|
||||
max_size_bytes=max_size_bytes,
|
||||
allowed_content_types=list(allowed_content_types),
|
||||
metadata_=metadata,
|
||||
)
|
||||
db.add(grant)
|
||||
db.flush()
|
||||
return _grant_response(grant, upload_token=upload_token, replayed=False)
|
||||
|
||||
def inspect_evidence(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: FormEvidenceInspectionRequest,
|
||||
) -> FormEvidenceInspection:
|
||||
db = _session(session)
|
||||
_assert_tenant(principal, request.tenant_id)
|
||||
now = datetime.now(UTC)
|
||||
if request.evidence.owner_module != PROVIDER_ID:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The evidence owner does not match the Files provider.",
|
||||
)
|
||||
if request.evidence.kind != "document" or not request.evidence.version:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="Files evidence requires an exact document version.",
|
||||
)
|
||||
grant = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||
FileFormEvidenceGrant.form_definition_id
|
||||
== request.definition_ref.object_id,
|
||||
FileFormEvidenceGrant.form_definition_revision
|
||||
== str(request.definition_ref.version),
|
||||
FileFormEvidenceGrant.file_asset_id == request.evidence.evidence_id,
|
||||
FileFormEvidenceGrant.file_version_id == request.evidence.version,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if grant is None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The document was not captured for this exact Form submission.",
|
||||
)
|
||||
if grant.status == "revoked":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="revoked",
|
||||
observed_at=now,
|
||||
reason="The Form evidence grant was revoked.",
|
||||
)
|
||||
if grant.status != "uploaded":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="pending",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The Form evidence upload has not completed.",
|
||||
)
|
||||
asset = db.get(FileAsset, grant.file_asset_id)
|
||||
version = db.get(FileVersion, grant.file_version_id)
|
||||
blob = db.get(FileBlob, version.blob_id) if version is not None else None
|
||||
if asset is None or version is None or blob is None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="unavailable",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The managed document cannot currently be reconstructed.",
|
||||
)
|
||||
if (
|
||||
asset.tenant_id != request.tenant_id
|
||||
or version.tenant_id != request.tenant_id
|
||||
or blob.tenant_id != request.tenant_id
|
||||
or version.file_asset_id != asset.id
|
||||
):
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document crosses an evidence ownership boundary.",
|
||||
)
|
||||
if asset.deleted_at is not None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="revoked",
|
||||
observed_at=now,
|
||||
reason="The managed document is no longer active.",
|
||||
)
|
||||
if blob.quarantined_at is not None or blob.integrity_status == "quarantined":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document failed its integrity gate.",
|
||||
)
|
||||
if blob.integrity_status != "verified":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="pending",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The managed document is awaiting integrity verification.",
|
||||
)
|
||||
if (
|
||||
not request.evidence.checksum
|
||||
or not secrets.compare_digest(
|
||||
request.evidence.checksum,
|
||||
version.checksum_sha256,
|
||||
)
|
||||
or not secrets.compare_digest(
|
||||
version.checksum_sha256,
|
||||
blob.checksum_sha256,
|
||||
)
|
||||
):
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document checksum does not match the evidence.",
|
||||
)
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="accepted",
|
||||
observed_at=now,
|
||||
metadata={
|
||||
"content_type": version.content_type,
|
||||
"size_bytes": version.size_bytes,
|
||||
"integrity_status": blob.integrity_status,
|
||||
"grant_id": grant.id,
|
||||
},
|
||||
)
|
||||
|
||||
def _assert_active_custodian(self, *, tenant_id: str, user_id: str) -> None:
|
||||
registry = self._registry
|
||||
if registry is None or not hasattr(registry, "has_capability"):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory is unavailable for Form evidence custody."
|
||||
)
|
||||
if not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory is unavailable for Form evidence custody."
|
||||
)
|
||||
directory = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||
if not isinstance(directory, AccessDirectory):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory capability is invalid."
|
||||
)
|
||||
user = directory.get_user(user_id)
|
||||
if user is None or user.tenant_id != tenant_id or user.status != "active":
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence requires an active same-tenant custodian."
|
||||
)
|
||||
|
||||
|
||||
def create_files_form_evidence_provider(
|
||||
context: ModuleContext,
|
||||
) -> FilesFormEvidenceProvider:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesFormEvidenceProvider(context.registry, context.settings)
|
||||
|
||||
|
||||
def _grant_response(
|
||||
grant: FileFormEvidenceGrant,
|
||||
*,
|
||||
upload_token: str | None,
|
||||
replayed: bool,
|
||||
) -> FormEvidenceGrant:
|
||||
return FormEvidenceGrant(
|
||||
provider_id=PROVIDER_ID,
|
||||
grant_id=grant.id,
|
||||
upload_token=upload_token,
|
||||
upload_url="/api/v1/files/form-evidence/upload",
|
||||
expires_at=_aware(grant.expires_at),
|
||||
max_size_bytes=grant.max_size_bytes,
|
||||
allowed_content_types=tuple(grant.allowed_content_types),
|
||||
replayed=replayed,
|
||||
)
|
||||
|
||||
|
||||
def _inspection(
|
||||
reference: EvidenceReference,
|
||||
*,
|
||||
state: FormEvidenceState,
|
||||
observed_at: datetime,
|
||||
retryable: bool = False,
|
||||
reason: str | None = None,
|
||||
metadata: Mapping[str, object] | None = None,
|
||||
) -> FormEvidenceInspection:
|
||||
return FormEvidenceInspection(
|
||||
provider_id=PROVIDER_ID,
|
||||
reference=reference,
|
||||
state=state,
|
||||
observed_at=observed_at,
|
||||
retryable=retryable,
|
||||
reason=reason,
|
||||
metadata=dict(metadata or {}),
|
||||
)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files Form evidence requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
def _assert_tenant(principal: object, tenant_id: str) -> None:
|
||||
principal_tenant = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not principal_tenant or principal_tenant != tenant_id:
|
||||
raise PermissionError("Form evidence cannot cross tenants.")
|
||||
|
||||
|
||||
def _custodian_user_id(value: str | None) -> str:
|
||||
clean = str(value or "").strip()
|
||||
if not clean.startswith("user:") or len(clean) <= len("user:"):
|
||||
raise FormEvidenceContractError(
|
||||
"Files Form evidence requires a user custodian."
|
||||
)
|
||||
return clean.removeprefix("user:")
|
||||
|
||||
|
||||
def _content_types(values: Sequence[str]) -> tuple[str, ...]:
|
||||
cleaned = tuple(
|
||||
dict.fromkeys(
|
||||
value.split(";", 1)[0].strip().casefold()
|
||||
for value in values
|
||||
if value.strip()
|
||||
)
|
||||
)
|
||||
if len(cleaned) > 50 or any(len(value) > 255 for value in cleaned):
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence content-type restrictions are too large."
|
||||
)
|
||||
return cleaned
|
||||
|
||||
|
||||
def _bounded_metadata(value: Mapping[str, object]) -> dict[str, object]:
|
||||
remaining = value.get("remaining_attachments")
|
||||
result: dict[str, object] = {}
|
||||
if isinstance(remaining, int) and remaining >= 0:
|
||||
result["remaining_attachments"] = remaining
|
||||
raw_ids = value.get("existing_attachment_ids")
|
||||
if isinstance(raw_ids, (list, tuple)):
|
||||
clean_ids = tuple(
|
||||
dict.fromkeys(
|
||||
item.strip()
|
||||
for item in raw_ids
|
||||
if isinstance(item, str) and item.strip() and len(item.strip()) <= 255
|
||||
)
|
||||
)
|
||||
if len(clean_ids) <= 1000:
|
||||
result["existing_attachment_ids"] = clean_ids
|
||||
return result
|
||||
|
||||
|
||||
def _request_sha256(
|
||||
request: FormEvidenceGrantRequest,
|
||||
*,
|
||||
custodian_user_id: str,
|
||||
max_size_bytes: int,
|
||||
allowed_content_types: Sequence[str],
|
||||
) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
{
|
||||
"tenant_id": request.tenant_id,
|
||||
"instance_id": request.instance_id,
|
||||
"definition_ref": request.definition_ref.to_dict(),
|
||||
"evidence_kind": request.evidence_kind,
|
||||
"purpose": request.purpose,
|
||||
"expires_at": request.expires_at.isoformat(),
|
||||
"custodian_user_id": custodian_user_id,
|
||||
"max_size_bytes": max_size_bytes,
|
||||
"allowed_content_types": list(allowed_content_types),
|
||||
"metadata": _bounded_metadata(request.metadata),
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _token_sha256(token: str) -> str:
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CAPABILITY_FORM_EVIDENCE_FILES",
|
||||
"FilesFormEvidenceProvider",
|
||||
"PROVIDER_ID",
|
||||
"create_files_form_evidence_provider",
|
||||
]
|
||||
@@ -0,0 +1,149 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Iterable
|
||||
|
||||
from govoplan_core.core.modules import DocumentationTopic, localize_documentation_topics as _localize_topics
|
||||
|
||||
|
||||
_TRANSLATIONS = {
|
||||
"files.workflow.unpack-managed-archive": {
|
||||
"title": "Ein bereits hochgeladenes Archiv entpacken",
|
||||
"summary": "Ein verwaltetes ZIP- oder TAR-Archiv auswählen und denselben Vorschau-, Passwort- und Auswahlablauf wie beim Hochladen nutzen, ohne es erneut zu übertragen.",
|
||||
"body": (
|
||||
"Wählen Sie genau eine ZIP-, TAR-, TAR.GZ-, TAR.BZ2- oder TAR.XZ-Datei in einem verwalteten Bereich und anschließend Archiv entpacken in der Aktionsleiste oder im Kontextmenü. "
|
||||
"Wählen Sie einen zugänglichen persönlichen oder Gruppen-Zielbereich und fordern Sie die Archivvorschau an; dabei entstehen noch keine Dateien. Im gemeinsamen Archivdialog prüfen Sie Einträge und gegebenenfalls das ZIP-Passwort und wählen Dateien oder Ordner aus. "
|
||||
"Beim Bestätigen liest der Server das verwaltete Quellarchiv und erstellt neue Zieldateien. Quellarchiv, Metadaten und Version bleiben unverändert; Zielkonflikte werden abgelehnt und überschreiben niemals das Quellarchiv. "
|
||||
"Lese-, Download- und Upload-Berechtigung sind gemeinsam erforderlich. Eigentum, aktuelle Freigaben, Mandant, Quellversion, Connector-Richtlinien, Integrität, Quarantäne und gegebenenfalls die Verfügbarkeit der Hüllenentschlüsselung werden beim Bestätigen erneut geprüft. "
|
||||
"Die kurzlebige Vorschau ist an Quellarchiv, Version, Person und Ziel gebunden. Geänderte oder unzugängliche Quellen verlangen eine neue Vorschau; die Version wird niemals still gewechselt. "
|
||||
"Dieselben Grenzen für Archivgröße, tatsächliche entpackte Gesamtgröße, einzelne Dateien, Anzahl der Einträge, Expansionsverhältnis und sichere Pfade wie beim Archiv-Upload gelten. Passwörter bleiben auf die Anfrage beschränkt. "
|
||||
"Wie beim Archiv-Upload entstehen gewöhnliche Zieldateien; der Passwortschutz des Archivs oder seine Speicher-Verschlüsselungshülle wird nicht automatisch auf Einträge übertragen. "
|
||||
"Während Prüfung und Entpacken überlagert die gemeinsame Ladeanzeige den weichgezeichneten Dialog, ohne Briefumschlaganimation. "
|
||||
"Die Anzahl und Gesamtgröße ausgewählter Dateien sind sofort sichtbar; der laufende Fortschritt zeigt gemessene Serverzähler statt zeitbasierter Schätzungen. "
|
||||
"Übertragung, Prüfung, Entpacken/Speichern und abschließende Transaktionsbestätigung sind getrennte Phasen; übertragene oder verarbeitete Bytes allein bedeuten noch keinen erfolgreichen Abschluss. "
|
||||
"Unbekannter Fortschritt bleibt unbestimmt. Schließen, Auswahl und Zieländerung sind während der Verarbeitung gesperrt; Fehler geben den unveränderten Prüfdialog wieder frei. "
|
||||
"Die Verarbeitung läuft synchron: Lassen Sie den beschäftigten Dialog bis zum Abschluss geöffnet. Dateien aus entfernten Connector-Bereichen müssen zuerst in den verwalteten Speicher synchronisiert werden."
|
||||
),
|
||||
},
|
||||
"files.tabular-content": {
|
||||
"title": "Verwaltete CSV- und XLSX-Versionen als gesteuerte Datenquellen verwenden",
|
||||
"summary": "Exakte autorisierte Dateiversionen für Connectors bereitstellen, ohne Files-Kontrollen zu umgehen.",
|
||||
"body": (
|
||||
"Files listet nur CSV- und XLSX-Dateien auf, die für die aktuelle Person im Mandanten sichtbar sind. Das Öffnen erfordert zusätzlich die Download-Berechtigung und eine exakte unveränderliche Versionsreferenz. Größenobergrenzen werden vor dem Speicherzugriff geprüft; Prüfsumme, Quarantäne, Verschlüsselungshülle, Löschung, Eigentum und Freigaben bleiben in Files maßgeblich. Connectors erhält Metadaten und verifizierte Bytes über die Core-Fähigkeit und importiert weder Files-Modelle noch Speicherschlüssel oder Verschlüsselungsinterna. Eine neuere aktuelle Version ersetzt eine gebundene Quellversion nie still; Connectors meldet die Änderung zur ausdrücklichen Prüfung."
|
||||
),
|
||||
},
|
||||
"files.records.exact-version-source": {
|
||||
"title": "Eine exakte Dateiversion in einer eAkte ablegen",
|
||||
"summary": "Records nach aktueller Zugriffs- und Integritätsprüfung eine unveränderliche Files-Versionsreferenz übergeben.",
|
||||
"body": (
|
||||
"Wenn Records aktiv ist, stellt Files exakte Identitäten verwalteter FileVersion-Objekte über den anbieterneutralen Aktenquellenvertrag bereit. Die Ablage prüft aktiven Mandanten, aktuelle Files-Berechtigung, Eigentums- oder Freigabezugriff, unveränderliche Versionsidentität und Integrität des verwalteten Blobs. Records erhält Dateiname, Pfad, Version, Digest, Medientyp, Größe, Schutzstatus und Aufruflink; Files bleibt Eigentümer der Bytes."
|
||||
),
|
||||
},
|
||||
"files.search.managed-content": {
|
||||
"title": "Verwaltete Dateien und Ordner durchsuchen",
|
||||
"summary": "Dateinamen, logische Pfade und Beschreibungen für die berechtigungsbewusste Plattformsuche bereitstellen.",
|
||||
"body": (
|
||||
"Wenn Search installiert ist, trägt Files verwaltete Dateien und Ordner zu dessen abgeleitetem Index bei. Jedes Ergebnis ist auf den Mandanten begrenzt und prüft vor der Ausgabe aktuelles Eigentum, Gruppenmitgliedschaft, direkte Freigaben, Ablauf, Widerruf, Löschung und Files-Berechtigungen erneut. Bestätigte Datei- und Freigabeänderungen werden über die Plattform-Event-Outbox übertragen; eine Administration kann den abgeleiteten Index neu aufbauen, ohne führende Files-Daten zu verändern."
|
||||
),
|
||||
},
|
||||
"files.forms-runtime.managed-evidence": {
|
||||
"title": "Formularanhänge als verwaltete Nachweise erfassen",
|
||||
"summary": "Kurzlebige Upload-Freigaben ausstellen und verwaltete Dateiversionen an genau eine Formulareinreichung binden.",
|
||||
"body": (
|
||||
"Fordert Forms Runtime einen Dokumentnachweis an, stellt Files einer aktiven, mandantengleichen verwahrenden Person eine zweckgebundene Freigabe aus. Das Bearer-Token wird einmal angezeigt, nur als SHA-256-Digest gespeichert, läuft spätestens nach 15 Minuten ab und kann genau eine verwaltete Dateiversion erzeugen. Bei der endgültigen Einreichung werden exaktes Formular, Freigabe, Asset, Version, Prüfsumme, Löschstatus und Files-Integritätsgrenze erneut geprüft. Öffentliche Erfassung erhält dadurch keine allgemeinen Files-Berechtigungen; ein Dokument für eine Einreichung kann nicht still für eine andere wiederverwendet werden."
|
||||
),
|
||||
},
|
||||
"files.workflow.organize-managed-files": {
|
||||
"title": "Verwaltete Dateien und Ordner organisieren",
|
||||
"summary": "Ordner anlegen und zugängliche Inhalte mit ausdrücklicher Konfliktbehandlung umbenennen, verschieben oder kopieren.",
|
||||
"body": (
|
||||
"Dokumentationsbücher stehen neben den Überschriften für Dateien, Verbindungen, Richtlinien "
|
||||
"oder Integrität und neben dem Titel des Dialogs zum Entpacken verwalteter Archive. "
|
||||
"Feldhilfe bleibt bei der Feldbezeichnung. "
|
||||
"Die Organisation bleibt in gesteuerten persönlichen oder Gruppenbereichen. Verschieben erhält die Asset-Identität; Kopieren erzeugt neue Assets und Versionen, die unveränderliche Blob-Bytes wiederverwenden. Jeder Zielkonflikt muss ausdrücklich abgelehnt, durch Umbenennen gelöst, überschrieben oder übersprungen werden. "
|
||||
"Ordner erstellen und Hochladen bleiben neben Neu laden im Kopf des Arbeitsbereichs. Nach der Auswahl von Dateien oder Ordnern öffnet Auswahl verwalten die Aktionen Verschieben, Kopieren, Umbenennen, Freigaben und Zugriffserklärungen. "
|
||||
"Löschen ist von der Organisation getrennt und benötigt weiterhin die vorhandene Bestätigung. Herunterladen und Archiv entpacken bleiben für passende Auswahlen direkt erreichbar. "
|
||||
"Verbindungen und Importe enthält die selteneren Import-, Synchronisierungs- und Connector-Bereichswerkzeuge; allein das Öffnen dieser Werkzeugauswahl importiert oder synchronisiert nichts."
|
||||
),
|
||||
},
|
||||
"files.workflow.find-and-download-files": {
|
||||
"title": "Verwaltete Dateien finden und herunterladen",
|
||||
"summary": "Zugängliche Inhalte suchen und eine aktuelle Dateiversion oder ein ZIP einer Auswahl herunterladen.",
|
||||
"body": (
|
||||
"Files kann nach logischem Pfad oder Namensmuster sortiert und durchsucht werden. Ein Download verwendet immer die zugängliche aktuelle Version; aus einer Mehrfachauswahl kann ein temporäres ZIP-Archiv erzeugt werden. Einen eigenen Dienst zur Inhaltsvorschau gibt es derzeit nicht."
|
||||
),
|
||||
},
|
||||
"files.workflow.share-managed-files": {
|
||||
"title": "Zugriff auf verwaltete Dateien steuern",
|
||||
"summary": "Direkte Lese-, Schreib- und Verwaltungsfreigaben auflisten, vergeben, ändern, befristen oder widerrufen, ohne das Eigentum zu ändern.",
|
||||
"body": (
|
||||
"Dateieigentümer und Files-Administrationen verwalten direkte Freigaben für Personen, Gruppen, den Mandanten und Campaign. Abgelaufene oder widerrufene Freigaben berechtigen sofort nicht mehr; unabhängige aktive Freigaben bleiben wirksam. Der Freigabedialog zeigt aktive und historische Einträge, und ein Widerruf ist idempotent. Die Zugriffserklärung prüft standardmäßig die angemeldete Person. Erlaubt Policy ausgewählte Fremdprüfungen, kann derselbe zentrale Dialog eine andere aktive Person im Mandanten bewerten; diese Diagnose wird im Audit protokolliert und gewährt keinen Zugriff."
|
||||
),
|
||||
},
|
||||
"files.workflow.delete-managed-files": {
|
||||
"title": "Verwaltete Dateien und Ordner löschen",
|
||||
"summary": "Zugängliche Dateien oder einen Ordnerbaum nach aktueller Richtlinie weich löschen.",
|
||||
"body": (
|
||||
"Löschen blendet ausgewählte verwaltete Assets aus, statt gespeicherte Nachweise physisch zu vernichten. Ordner werden standardmäßig einschließlich ihrer Unterordner und Dateien rekursiv gelöscht; eine nicht rekursive Anforderung scheitert bei einem nicht leeren Ordner. Eine autorisierte Wiederherstellung aktiviert dasselbe Asset mit Versionen, Provenienz, Ordnerbaum oder Connector-Bereich erneut, sofern Pfad oder Bezeichnung frei ist. Die unwiderrufliche Vernichtung ist ein eigener Administrationsablauf mit Aufbewahrungs- und Legal-Hold-Prüfungen."
|
||||
),
|
||||
},
|
||||
"files.workflow.restore-retain-and-purge": {
|
||||
"title": "Verwaltete Dateien wiederherstellen, aufbewahren und vernichten",
|
||||
"summary": "Weich gelöschte Ressourcen wiederherstellen und irreversible Löschung mit Aufbewahrung, Legal Hold, Vorschau, Freigabe und Recovery-Nachweis steuern.",
|
||||
"body": (
|
||||
"Wiederherstellen erhält ursprüngliche Asset-Identität, Versionen, Blob-Referenzen und Connector-Provenienz. Eine Aufbewahrungsadministration kann mit Begründung und Revisionsprüfung ein Aufbewahrungsdatum oder Legal Hold setzen. Eine physische Vernichtung benötigt eigene Berechtigung, aktuellen unveränderlichen Vorschau-Hash, Idempotenzschlüssel, die wörtliche Bestätigung PURGE und eine Freigabereferenz. Aktive Aufbewahrung, Legal Hold, Freigaben sowie Campaign- und Form-Nachweise blockieren sie. Zuerst werden berechtigte Datenbankeinträge entfernt und Blobs freigegeben; eine getrennte begrenzte Garbage Collection prüft unter derselben verteilten Blob-Sperre wie Uploads alle FileVersion-Referenzen erneut, bevor Bytes und Metadaten gelöscht werden. Jede irreversible Aktion erscheint im Core-Recovery-Ledger und Audit-Nachweis."
|
||||
),
|
||||
},
|
||||
"files.privacy.data-subject-requests": {
|
||||
"title": "Files-Daten in einer Datenschutzanfrage prüfen",
|
||||
"summary": "Sichere Files-Metadaten ermitteln und Entscheidungen über Aufbewahrung, Nachweise und Byte-Löschung ausdrücklich behandeln.",
|
||||
"body": (
|
||||
"Der Files-DSAR-Anbieter sucht nur im wirksamen Mandanten und verlangt eine direkte Mitgliedschaft oder namensraumbezogene Files-Benutzerreferenz. Er exportiert begrenzte Datei-, Versions-, Ordner-, Freigabe-, Nachweis-, Connector-Konfigurations- und Integritätsmetadaten ohne Dateibytes, Speicherorte, Tokens, Passwörter, Geheimnisreferenzen oder verschlüsselte Zugangsdaten. Pläne dürfen eine aktive Freigabe für die betroffene Person widerrufen oder eine veränderbare Akteursreferenz lösen. Legal Hold, aktive Aufbewahrung, Form- und Campaign-Nachweise, Konfigurationshistorie und Integritätsnachweise bleiben mit Begründung erhalten. Dateiinhalte, Eigentum, Namen und Pfade benötigen manuelle Prüfung. Freigegebene physische Löschung muss den getrennt autorisierten Files-Vernichtungs- und Blob-Garbage-Collection-Ablauf nutzen, damit DSAR keine Nachweisblocker, Freigabe, Audit- oder Recovery-Kontrollen umgeht."
|
||||
),
|
||||
},
|
||||
"files.reference.integrity-recovery-and-fail-closed-transports": {
|
||||
"title": "Files-Integrität, Recovery und ausfallsichere Connector-Transporte betreiben",
|
||||
"summary": "Datenbanknachweise, Blob-Chiffretexte und Encryption-Verwahrung als eine Recovery-Einheit sichern und jeden SDK-verwalteten Gegenpunkt binden.",
|
||||
"body": (
|
||||
"Die Archivprüfung setzt TAR-Eintrags-, Größen- und Expansionsgrenzen bei jedem Header durch, bevor die nächsten Nutzdaten durchlaufen werden. Pfade sind auf 4096 UTF-8-Bytes und 128 Komponenten begrenzt; abgeleitete Elternordner zählen zur konfigurierten Eintragsgrenze. Zu große Archive müssen vor einem erneuten Versuch verkleinert oder aufgeteilt werden; die Richtlinie wird nicht automatisch gelockert. "
|
||||
"Dauerhafter lokaler Speicher ist die betriebliche Basis. Files wird aus einem abgestimmten Datenbank-/Blob-Snapshot mit passenden Encryption-Tabellen und ursprünglichem Deployment-Hauptschlüssel wiederhergestellt; anschließend ist der begrenzte fortsetzbare Integritätsscan auszuführen und geschützter sowie ungeschützter Zugriff stichprobenartig zu prüfen. Scan- und Befundaktionen benötigen die angezeigte Revision. Geschützte Scans prüfen erst den Chiffretext, dann nach Entschlüsselung den semantischen Klartextnachweis. Unter PostgreSQL sichern lease-gebundene Core-Recovery-Absichten Objektauswirkungen ab; Abweichungen werden quarantänisiert und bleiben in Ops sichtbar. SQLite verwendet für Entwicklung eine prozesslokale Sperre und ist kein Produktions-Recovery-Profil; nach hartem Prozessverlust ist ein Integritätsscan erforderlich. Fehlende oder abweichende Blobs werden quarantänisiert, verwaiste Objekte vor einer ausdrücklich autorisierten Bereinigung zunächst nur gemeldet. Physische Vernichtung und Blob-Garbage-Collection prüfen ihre irreversiblen Wirkungen getrennt. S3-Schreibvorgänge nutzen bedingte Effekte und digestbasierte Vorwärts-Recovery; S3- und SMB-Transporte binden Wiederholungen, Umleitungen, Aliasse und erkannte Endpunkte und wenden die Richtlinie für private Netze vor jeder Verbindung erneut an. Fehlt eine verifizierbare Transportnaht, wird geschlossen abgebrochen. Destruktive Modulstilllegung entfernt Datenbanktabellen, nicht jedoch Blob-Objekte im Backend."
|
||||
),
|
||||
},
|
||||
"files.reference.snapshot-provenance-and-capabilities": {
|
||||
"title": "Snapshot-Provenienz und Files-Fähigkeiten integrieren",
|
||||
"summary": "Stabile Files-Fähigkeiten oder HTTP-Verträge verwenden und exakte Versionsnachweise bewahren, statt Files-Interna zu importieren.",
|
||||
"body": (
|
||||
"files.access erklärt Ressourcenzugriff; files.campaign_attachments friert Campaign-Eingaben auf exaktes Asset, Version, Blob, Prüfsumme und Quellrevision ein. Externe Inhalte müssen vor einer gesteuerten Nutzung importiert und ihre Provenienz auf abgeleiteten Snapshots erhalten werden. Zusammenarbeit, Provider-Synchronisierung, OAuth, externe Änderungen und fachlicher Workflow-Zustand bleiben in den jeweils verantwortlichen Modulen."
|
||||
),
|
||||
},
|
||||
"files.assurance.process-and-release-readiness": {
|
||||
"title": "Files-gestützte Prozesse und Releases absichern",
|
||||
"summary": "Einen Prozess gegen die implementierte Files-Grenze prüfen, erlaubte und abgelehnte Pfade erproben und vor Freigabe Nachweise bewahren.",
|
||||
"body": (
|
||||
"Prozessverantwortliche müssen implementierte Kontrollen von geplanten Fähigkeiten unterscheiden. Ein Release ist erst bereit, wenn Paket- und Manifestversionen übereinstimmen und repräsentative Pfade für Autorisierung, Upload-Grenzen, Konflikte, Download, Löschung, Connectoren und Recovery erprobt wurden. Wiederherstellung, Aufbewahrung, Legal Hold, gesteuerte Vernichtung, Blob-Garbage-Collection, Freigabelebenszyklus und ausdrückliches S3-Zurückschreiben erzeugen eigene Policy-, Recovery- oder Audit-Nachweise. Gewöhnliche Organisationsänderungen stützen sich weiterhin vor allem auf die Files-Änderungssequenz statt auf ein eigenes kanonisches Audit-Ereignis. Verbleibende Grenzen sind in der Prozessbewertung zu dokumentieren und Nachweisarten dürfen nicht gleichgesetzt werden."
|
||||
),
|
||||
},
|
||||
"files.reference.shared-storage-profile": {
|
||||
"title": "Files-Speicher und temporäre Archivverarbeitung betreiben",
|
||||
"summary": "Dauerhaften Speicher, privaten Archivzwischenspeicher und gemessenen Fortschritt auf die Worker-Topologie abstimmen.",
|
||||
"body": (
|
||||
"Core stellt das gemeinsame lokale/S3-Objektspeicher-Backend bereit; Files verantwortet Dateimetadaten und Objektschlüssel. Lokaler Speicher eignet sich für einen Laufzeitprozess, ein gemeinsames Host-Volume für Replikate auf demselben Host. Unabhängige Hosts benötigen einen ausdrücklich vertrauenswürdigen HTTPS-S3-kompatiblen Endpunkt. PostgreSQL, Objekte und Hauptschlüssel müssen auf denselben abgestimmten Recovery-Zeitpunkt wiederhergestellt werden. "
|
||||
"Die Archivoberfläche fordert ausdrücklich einmalige Übertragung mit Zwischenspeicherung an: Erneute Passwortprüfung und Bestätigung verwenden eine an Mandant und Person gebundene Kopie, speichern jedoch niemals Passwörter. Temporäre Archiv- und Fortschrittsdateien haben Modus 0600 unter einem privaten Verzeichnis mit Modus 0700. FILE_ARCHIVE_WORK_ROOT ersetzt den Standard im temporären Betriebssystemverzeichnis, dessen Name von Dienst-UID und Hash des konfigurierten Speicherwurzelpfads abgeleitet wird. Alle Worker desselben Hosts müssen darauf zugreifen; unabhängige Hosts benötigen gemeinsamen POSIX-Speicher mit funktionierenden flock-Sperren oder feste Host-Zuordnung für Archiv- und Fortschrittsanfragen, auch bei dauerhaftem S3-Blob-Speicher. "
|
||||
"Standardgrenzen sind 2 GiB zwischengespeicherte Archive pro Arbeitsverzeichnis, vier Kopien pro Mandant/Person und 1.800 Sekunden ab der ersten Speicherung ohne Verlängerung. Abbruch- und Wechselanfragen sowie erfolgreicher Abschluss geben Kopien frei; verlassene abgelaufene Kopien werden bei späterer Archivverarbeitung bereinigt, nicht garantiert zu einem exakten Zeitpunkt ohne weitere Aktivität. Aktive POSIX-Leases schützen benutzte Kopien auch nach Ablauf vor Verdrängung. Kopien und Fortschrittsdaten sind keine fortsetzbaren Hintergrundaufträge. "
|
||||
"Eine optionale, aktuell gehaltene Systembibliothek libarchive beschleunigt klassisches ZIPCrypto mit gespeicherten oder Deflate-komprimierten Einträgen und ASCII- oder ausdrücklich als UTF-8 markierten Namen. Fehlende Bibliotheken, AES und nicht unterstützte Kodierungen/Formate wählen vor dem Dekodieren den bisherigen Python-Leser; native Fehler führen zum sicheren Abbruch. Beide Leser laufen in einem begrenzten Kindprozess mit vollständigen Header-, Pfad- und Grenzprüfungen; der native Pfad prüft CRC und Größe zusätzlich unabhängig. Nur die Zwischenspeicherschicht schreibt Dateien unter privaten servergenerierten numerischen Namen, niemals unter Archivpfaden. "
|
||||
"Die Verarbeitung speichert jeweils einen geprüften Eintrag. Wiederverwendung des Speicher-Clients innerhalb eines Archivs und gebündelte Antwortmetadaten vermeiden Zusatzarbeit, ohne Autorisierung zwischenzuspeichern oder dauerhafte Recovery-, Integritäts- und Commit-Prüfungen pro Datei zu entfernen. Fortschritt zeigt tatsächliche Datei-/Bytezähler und eine eigene Abschlussphase statt Zeitschätzungen oder vorzeitiger Erfolgsmeldung. Ein schnellerer Decoder verspricht keine entsprechende Beschleunigung des gesamten Imports oder von S3."
|
||||
),
|
||||
},
|
||||
"files.reference.generated-artifact-store": {
|
||||
"title": "Erzeugte Modulausgaben speichern",
|
||||
"summary": "Optionalen Erzeugermodulen erlauben, Ausgaben über die Autoritätsgrenze von Files dauerhaft abzulegen.",
|
||||
"body": (
|
||||
"Die Fähigkeit files.artifact_store nimmt erzeugte Bytes mit begrenzter, nicht geheimer Provenienz entgegen, wendet Upload-Autorisierung, Eigentums-, Pfad-, Versions- und Blob-Speicherregeln von Files an und liefert anbieterneutrale Datei-/Versionsreferenzen. Idempotenz beruht auf der Quellprovenienz. Die Annahme eines Artefakts beweist weder Druck, Versand noch eine andere externe Wirkung."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def localize_documentation_topics(
|
||||
topics: Iterable[DocumentationTopic],
|
||||
) -> tuple[DocumentationTopic, ...]:
|
||||
return _localize_topics(topics, locale="de", translations=_TRANSLATIONS)
|
||||
@@ -0,0 +1,473 @@
|
||||
"""German translations for public structured documentation metadata."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
_MANAGED_ARCHIVE_GERMAN = {
|
||||
"prerequisites": ["Das ausgewählte verwaltete Archiv ist lesbar und herunterladbar; Sie dürfen in den gewählten Zielbereich hochladen."],
|
||||
"steps": [
|
||||
"Wählen Sie ein unterstütztes verwaltetes Archiv und Archiv entpacken in der Aktionsleiste oder im Kontextmenü.",
|
||||
"Wählen Sie Zielbereich und Zielordner und fordern Sie die Archivvorschau an.",
|
||||
"Prüfen Sie Einträge, geben Sie gegebenenfalls das ZIP-Passwort ein und verifizieren Sie es; wählen Sie anschließend Dateien oder Ordner zum Entpacken aus.",
|
||||
"Bestätigen Sie die ausgewählten Einträge und lassen Sie den Dialog bis zum Abschluss geöffnet. Wählen Sie bei vorhandenen Zieldateien ein anderes Ziel.",
|
||||
],
|
||||
"outcome": "Ausgewählte Einträge sind neue verwaltete Dateien; das Quellarchiv bleibt unverändert.",
|
||||
"verification": "Prüfen Sie Zielpfade und Anzahl sowie die unveränderte Version des Quellarchivs.",
|
||||
}
|
||||
|
||||
|
||||
GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'files.assurance.process-and-release-readiness': {'outcome': 'Der Prozess oder die '
|
||||
'Veröffentlichung hat eine explizite '
|
||||
'Genehmigungsaufzeichnung, die an '
|
||||
'abgestimmte Versionen, '
|
||||
'repräsentative Nachweise, bekannte '
|
||||
'Einschränkungen und eigene '
|
||||
'Restrisiken gebunden ist.',
|
||||
'prerequisites': ['Ein benannter '
|
||||
'Prozessbesitzer hat die '
|
||||
'beabsichtigten Benutzer, '
|
||||
'Datenklassifizierung, '
|
||||
'Aufbewahrungserwartungen und '
|
||||
'Integrationen definiert.',
|
||||
'Ein Kandidaten-Release wird '
|
||||
'in einer sauberen Datenbank '
|
||||
'und einer Upgrade-Kopie mit '
|
||||
'ausgerichteten Python-, '
|
||||
'Root-Paket-, WebUI-Paket- '
|
||||
'und Modulmanifestversionen '
|
||||
'installiert.',
|
||||
'Repräsentative zugelassene '
|
||||
'und verweigerte Konten, '
|
||||
'begrenzte Testdateien und '
|
||||
'ein koordiniertes '
|
||||
'Datenbank-/Blob-/Key-Wiederherstellungsset '
|
||||
'sind verfügbar.'],
|
||||
'steps': ['Vergleichen Sie die '
|
||||
'Prozessanforderungen mit der '
|
||||
'implementierten / geplanten Grenze '
|
||||
'des Handbuchs und notieren Sie jede '
|
||||
'nicht unterstützte Anforderung oder '
|
||||
'Kompensationskontrolle.',
|
||||
'Bestätigen Sie die '
|
||||
'Versionsausrichtung, wenden Sie '
|
||||
'Migrationen auf bereinigte und '
|
||||
'aktualisierte Datenbanken an und '
|
||||
'führen Sie das Repository sowie '
|
||||
'Meta-Repository-Sicherheits- und '
|
||||
'Statikanalyse-Gates aus.',
|
||||
'Übung erlaubt und verweigert '
|
||||
'persönlichen, Gruppen und teilen Sie '
|
||||
'den Zugriff mit repräsentativen '
|
||||
'Konten.',
|
||||
'Übung begrenzte Archivvorschau und '
|
||||
'Bestätigung, jede erforderliche '
|
||||
'Konfliktstrategie, Organisation, '
|
||||
'Download und weiche Löschung.',
|
||||
'Wenn Konnektoren konfiguriert sind, '
|
||||
'überprüfen Sie die '
|
||||
'Richtlinienerklärung und einen '
|
||||
'angehängten HTTP-Provider; '
|
||||
'überprüfen Sie die '
|
||||
'Installateur-eigene Garage, wenn Sie '
|
||||
'ausgewählt sind, und üben Sie dann '
|
||||
'konfigurierte S3-Retries/Aliases und '
|
||||
'SMB-Reconnect/Referral-Ziele unter '
|
||||
'der Bereitstellungsrichtlinie für '
|
||||
'das private Netzwerk aus, um zu '
|
||||
'bestätigen, dass eine inkompatible '
|
||||
'SDK-Transportnaht nicht geschlossen '
|
||||
'ist.',
|
||||
'Stellen Sie ein koordiniertes '
|
||||
'Datenbank-/Blob-/Key-Backup wieder '
|
||||
'her und vergleichen Sie '
|
||||
'repräsentative heruntergeladene '
|
||||
'Bytes mit den aufgezeichneten '
|
||||
'SHA-256-Prüfsummen.',
|
||||
'Notieren Sie die getesteten '
|
||||
'Versionen, Ergebnisse, bekannte '
|
||||
'Einschränkungen, Evidenzorte, '
|
||||
'Restrisiken, Eigentümer und '
|
||||
'Genehmigungsentscheidung.'],
|
||||
'verification': 'Ein Überprüfer kann die '
|
||||
'aufgezeichneten '
|
||||
'Berechtigungs-/Verweigerungs-, '
|
||||
'Integritäts-, Konnektor- und '
|
||||
'Wiederherstellungsüberprüfungen '
|
||||
'reproduzieren und jede nicht '
|
||||
'erfüllte Anforderung auf eine '
|
||||
'dokumentierte Einschränkung '
|
||||
'oder akzeptierte '
|
||||
'Kompensationskontrolle '
|
||||
'zurückführen.'},
|
||||
'files.forms-runtime.managed-evidence': {'limitations': ['Das erste Signaturprofil ist eine '
|
||||
'authentifizierte Bestätigung; '
|
||||
'fortgeschrittene und qualifizierte '
|
||||
'elektronische Signaturen erfordern '
|
||||
'einen separaten Anbieter.'],
|
||||
'outcome': 'Der Entwurf verweist auf eine '
|
||||
'unveränderliche verwaltete Dateiversion mit '
|
||||
'genauen Prüfsummennachweisen.',
|
||||
'prerequisites': ['Formulare Laufzeit und Dateien sind '
|
||||
'aktiviert, und das Aufnahmeprofil hat '
|
||||
'eine aktive Benutzerdepotbank für den '
|
||||
'gleichen Mandant.'],
|
||||
'steps': ['Fordern Sie eine Beweisbewilligung für '
|
||||
'Dateien von der editierbaren Formularinstanz '
|
||||
'an, laden Sie sie über die zurückgegebene '
|
||||
'einmalige Trägerroute hoch und fügen Sie dann '
|
||||
'die zurückgegebene Beweisreferenz zum Entwurf '
|
||||
'hinzu.'],
|
||||
'verification': 'Reichen Sie das Formular ein und '
|
||||
'vergewissern Sie sich, dass geänderte, '
|
||||
'gelöschte, unter Quarantäne gestellte, '
|
||||
'mieterübergreifende und '
|
||||
'gebührenübergreifende Referenzen '
|
||||
'abgelehnt werden.'},
|
||||
'files.governed-connectors-and-provenance': {'limitations': ['Die manuelle '
|
||||
'Ordner-Synchronisierung plant keine '
|
||||
'zukünftigen Runs oder mutiert, '
|
||||
'löscht, umbenennt, verschiebt oder '
|
||||
'ändert ACLs für entfernte Inhalte.',
|
||||
'Ein erfolgreicher Teildurchlauf ist '
|
||||
'keine vollständige Abdeckung, wenn '
|
||||
'die Antwort verkürzt wird oder '
|
||||
'Überprüfungsergebnisse enthält.'],
|
||||
'outcome': 'Zulässige Konnektordateien werden zu '
|
||||
'verwalteten Assets oder Versionen, '
|
||||
'während jede Ausnahme explizit und '
|
||||
'überprüfbar bleibt.',
|
||||
'prerequisites': ['Ein sichtbarer aktiver '
|
||||
'Verbindungsraum verwendet '
|
||||
'manuelle Synchronisierung und '
|
||||
'Links zu einem zugänglichen '
|
||||
'verwalteten Benutzer- oder '
|
||||
'Gruppenraum.',
|
||||
'Der Betreiber hat '
|
||||
'file:file:upload und die '
|
||||
'effektive Konnektorrichtlinie '
|
||||
'erlaubt die angeforderten Root- '
|
||||
'und Source-Pfade.'],
|
||||
'steps': ['Öffnen Sie den Ordner des verknüpften '
|
||||
'Verbinders, wählen Sie den Ordner Sync '
|
||||
'und überprüfen Sie das verwaltete Ziel.',
|
||||
'Halten Sie den sicheren Sprung '
|
||||
'standardmäßig oder wählen Sie explizit '
|
||||
'umbenennen, ablehnen oder überschreiben '
|
||||
'für nicht verwandte Zielkollisionen.',
|
||||
'Legen Sie die Auswahl der begrenzten '
|
||||
'Dateianzahl und des Unterordners fest, '
|
||||
'führen Sie die Synchronisierung aus und '
|
||||
'überprüfen Sie jedes eingegebene '
|
||||
'Elementergebnis und jede '
|
||||
'Abkürzungswarnung.',
|
||||
'Setzen Sie einen verkürzten Lauf aus '
|
||||
'einem schmaleren entfernten Ordner fort; '
|
||||
'konfigurieren Sie die Planung separat, '
|
||||
'wenn eine Hintergrundoperation '
|
||||
'erforderlich ist.'],
|
||||
'verification': 'Vergleichen Sie die '
|
||||
'Antwortzusammenfassung und die '
|
||||
'Ergebnisse pro Element mit dem '
|
||||
'Auditereignis '
|
||||
'files.connector.folder synced und '
|
||||
'prüfen Sie dann die Herkunft der '
|
||||
'erstellten, aktualisierten und '
|
||||
'unveränderten repräsentativen '
|
||||
'Assets.'},
|
||||
'files.privacy.data-subject-requests': {'limitations': ['E-Mail-, Konto- oder '
|
||||
'Identitätsselektoren allein können nicht '
|
||||
'von Dateien aufgelöst werden, da Dateien '
|
||||
'nicht im Besitz des '
|
||||
'Access-Verzeichnisses sind; geben Sie '
|
||||
'die bestätigte Mitgliedschaftsreferenz '
|
||||
'an.',
|
||||
'Der Anbieter bettet keine rohen '
|
||||
'Dateiinhalte in den JSON-Export ein und '
|
||||
'führt niemals eine physische '
|
||||
'Blob-Löschung als DSAR-Nebenwirkung '
|
||||
'durch.'],
|
||||
'outcome': 'Dateieigene Betreffreferenzen werden '
|
||||
'überprüft oder entfernt, ohne den '
|
||||
'zurückgehaltenen Inhalt oder Nachweise '
|
||||
'stillschweigend zu zerstören.',
|
||||
'prerequisites': ['Die Anfrage wurde autorisiert und '
|
||||
'enthält eine direkte '
|
||||
'Mandantenmitgliedschaft oder eine '
|
||||
'Betreffreferenz Dateien.',
|
||||
'Der Datenschutzprüfer kann den '
|
||||
'Zugriffsexport von der Löschbehörde '
|
||||
'und der Löschbehörde für Dateien '
|
||||
'unterscheiden.'],
|
||||
'steps': ['Führen Sie die Anbietersuche aus und '
|
||||
'bestätigen Sie die vollständige Abdeckung der '
|
||||
'Dateiberichte anstelle eines ausgefallenen '
|
||||
'oder abwesenden Anbieters.',
|
||||
'Überprüfen Sie Datei-/Versionsmetadaten, '
|
||||
'Gründe für die Aufbewahrung von Nachweisen und '
|
||||
'den Quellpfad für die manuelle '
|
||||
'Inhaltsüberprüfung.',
|
||||
'Generieren Sie den Löschplan und führen Sie '
|
||||
'nur die genehmigten reversiblen '
|
||||
'Aktienrücknahme- oder Subjektreferenzaktionen '
|
||||
'aus.',
|
||||
'Für die genehmigte Byte-Löschung lösen Sie '
|
||||
'jeden Lifecycle-Blocker auf und verwenden Sie '
|
||||
'Files Purge Preview, Execution und Blob '
|
||||
'Garbage Collection separat.'],
|
||||
'verification': 'Bestätigen Sie, dass jeder '
|
||||
'Dateidatensatz eine Aufbewahrungs-, '
|
||||
'Überprüfungs-, Widerrufs- oder '
|
||||
'Loslösungsdisposition hat und prüfen Sie '
|
||||
'jede separate Bereinigung durch seine '
|
||||
'Audit- und Wiederherstellungsnachweise.'},
|
||||
'files.records.exact-version-source': {'outcome': 'Records behält eine exakt geregelte Referenz, '
|
||||
'während Files Byte Authority bleibt.',
|
||||
'prerequisites': ['Die genaue Dateiversion existiert und '
|
||||
'übergibt das aktuelle Files Integrity '
|
||||
'Gate.',
|
||||
'Sie haben derzeit die Datei Lesezugriff '
|
||||
'und Records Einreichung Behörde.'],
|
||||
'steps': ['Wählen Sie die genaue verwaltete Dateiversion '
|
||||
'aus, die zum institutionellen Datensatz gehört.',
|
||||
'Wählen Sie den Zieldatensatz und geben Sie den '
|
||||
'Zugangszweck und den Anmeldegrund an.',
|
||||
'Bestätigen Sie die Einreichung; Dateien '
|
||||
'überprüft den aktuellen Zugriff erneut und löst '
|
||||
'die genaue Version auf.',
|
||||
'Öffnen Sie die Datensatzchronologie und '
|
||||
'überprüfen Sie die Versionsidentität und den '
|
||||
'SHA-256 Digest.']},
|
||||
'files.reference.integrity-recovery-and-fail-closed-transports': {'verification': 'Führen Sie '
|
||||
'nach der '
|
||||
'Wiederherstellung '
|
||||
'einen '
|
||||
'Checksummen-fähigen '
|
||||
'Integritätsscan '
|
||||
'aus, lösen Sie '
|
||||
'jedes fehlende '
|
||||
'/ korrupte '
|
||||
'Finden, '
|
||||
'genehmigen '
|
||||
'oder behalten '
|
||||
'Sie jedes '
|
||||
'gemeldete '
|
||||
'Waisenkind, '
|
||||
'inspizieren '
|
||||
'Sie die '
|
||||
'Wiederherstellungsvorgänge '
|
||||
'von Dateien in '
|
||||
'Ops, '
|
||||
'überprüfen Sie '
|
||||
'den '
|
||||
'autorisierten '
|
||||
'und '
|
||||
'verweigerten '
|
||||
'Zugriff und '
|
||||
'testen Sie '
|
||||
'konfigurierte '
|
||||
'HTTP-, S3- und '
|
||||
'SMB-Verbindungen '
|
||||
'mit ihrer '
|
||||
'aufgezeichneten '
|
||||
'Zieltopologie.'},
|
||||
'files.reference.shared-storage-profile': {'limitations': ['Installer-verwaltete Garage ist '
|
||||
'Single-Node, es sei denn, ein '
|
||||
'externer Multi-Node-Cluster wird '
|
||||
'separat betrieben.',
|
||||
'Die Anwendung erstellt oder '
|
||||
'verifiziert keine '
|
||||
'PostgreSQL/Objekt/Schlüssel-Backups.'],
|
||||
'verification': 'Führen Sie die '
|
||||
'Dateispeicher-Round-Trip-Prüfung und '
|
||||
'eine koordinierte '
|
||||
'Wiederherstellungsübung gegen die '
|
||||
'genaue Bereitstellungstopologie aus.'},
|
||||
'files.tabular-content': {'consequence_classes': {'fail_closed': 'Nicht verfügbare, '
|
||||
'überdimensionierte, unter '
|
||||
'Quarantäne gestellte oder nicht '
|
||||
'autorisierte Inhalte werden '
|
||||
'nicht analysiert oder in der '
|
||||
'Vorschau angezeigt.',
|
||||
'pin_exact_version': 'Eine Quelle behält ihre '
|
||||
'überprüfte '
|
||||
'unveränderliche '
|
||||
'Dateiversion, bis sie '
|
||||
'explizit aktualisiert '
|
||||
'wird.',
|
||||
'preserve_file_controls': 'Zugriff auf Dateien, '
|
||||
'Integrität, '
|
||||
'Verschlüsselung, '
|
||||
'Aufbewahrung und '
|
||||
'rechtliche Kontrolle '
|
||||
'bleiben '
|
||||
'autoritativ.'}},
|
||||
'files.workflow.delete-managed-files': {'limitations': ['Löschung ist eine weiche Löschung, keine '
|
||||
'harte Säuberung.',
|
||||
'Die Wiederherstellung schlägt fehl, wenn '
|
||||
'eine aktive Datei, ein Ordner oder ein '
|
||||
'Verbindungsraum bereits den Pfad oder '
|
||||
'das Label verwendet.'],
|
||||
'outcome': 'Der ausgewählte Inhalt wird aus aktiven '
|
||||
'Dateiansichten unter dem aktuellen '
|
||||
'Soft-Löschmodell ausgeblendet.',
|
||||
'prerequisites': ['Sie können die ausgewählten '
|
||||
'verwalteten Inhalte anzeigen und '
|
||||
'löschen und haben Schreib- oder '
|
||||
'Eigentümerzugriff auf jedes betroffene '
|
||||
'Asset oder jeden Ordner.',
|
||||
'Sie haben den gesamten Ordnerbaum beim '
|
||||
'rekursiven Löschen überprüft.'],
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie die Dateien '
|
||||
'oder den Ordner zum Löschen aus.',
|
||||
'Überprüfen Sie die Auswahl und, für einen '
|
||||
'Ordner, alle Inhalte darunter.',
|
||||
'Bestätigen Sie die Löschaktion.',
|
||||
'Erfrischen oder öffnen Sie den Raum und '
|
||||
'überprüfen Sie, ob die ausgewählten Pfade '
|
||||
'nicht mehr aktiv sind.'],
|
||||
'verification': 'Bestätigen Sie, dass die gelöschten '
|
||||
'Pfade nicht mehr im aktiven Raum '
|
||||
'erscheinen; behandeln Sie die Aktion '
|
||||
'nicht als physisches Löschen.'},
|
||||
'files.workflow.find-and-download-files': {'outcome': 'Die autorisierten aktuellen Dateibytes '
|
||||
'oder das generierte Archiv werden auf das '
|
||||
'lokale Gerät heruntergeladen.',
|
||||
'prerequisites': ['Sie können verwaltete Dateien im '
|
||||
'entsprechenden Bereich anzeigen und '
|
||||
'herunterladen.'],
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie den '
|
||||
'entsprechenden Personen- oder Gruppenraum '
|
||||
'aus.',
|
||||
'Navigieren Sie in Ordnern, sortieren Sie '
|
||||
'die Liste oder verwenden Sie ein Pfad- / '
|
||||
'Namensmuster, um den beabsichtigten Inhalt '
|
||||
'zu finden.',
|
||||
'Überprüfen Sie den angezeigten Besitzer, '
|
||||
'Pfad, Größe, Prüfsumme und Versionsdetails.',
|
||||
'Laden Sie eine aktuelle Dateiversion '
|
||||
'herunter oder wählen Sie mehrere Dateien '
|
||||
'aus und wählen Sie ZIP herunterladen.'],
|
||||
'verification': 'Bestätigen Sie die heruntergeladenen '
|
||||
'Namen und vergleichen Sie, wo '
|
||||
'Integrität wichtig ist, die '
|
||||
'Dateibytes mit der angezeigten '
|
||||
'Prüfsumme.'},
|
||||
'files.workflow.organize-managed-files': {'outcome': 'Der ausgewählte Inhalt hat den '
|
||||
'beabsichtigten verwalteten Eigentümer und '
|
||||
'logischen Pfad.',
|
||||
'prerequisites': ['Sie können verwaltete Dateien '
|
||||
'anzeigen und organisieren.',
|
||||
'Sie haben Schreib- oder '
|
||||
'Eigentümerzugriff auf alle von der '
|
||||
'Operation verwendeten Quellelemente '
|
||||
'und Zielbereiche; die globale '
|
||||
'Organisationsberechtigung allein '
|
||||
'gewährt keinen Ressourcenzugriff.'],
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie den '
|
||||
'persönlichen oder Gruppenraum zum '
|
||||
'Organisieren aus.',
|
||||
'Verwenden Sie Ordner erstellen im Kopf des '
|
||||
'Arbeitsbereichs oder wählen Sie Dateien und '
|
||||
'Ordner aus und öffnen Sie Auswahl verwalten '
|
||||
'zum Umbenennen, Verschieben oder Kopieren.',
|
||||
'Wählen Sie das Ziel aus und lösen Sie jeden '
|
||||
'Zielkonflikt explizit.',
|
||||
'Wenden Sie die Operation an und öffnen Sie '
|
||||
'den Zielordner erneut.'],
|
||||
'verification': 'Bestätigen Sie jeden resultierenden '
|
||||
'Pfad und Besitzer; für eine Bewegung '
|
||||
'bestätigen Sie auch, dass der alte '
|
||||
'Pfad verschwunden ist, und für eine '
|
||||
'Kopie bestätigen Sie, dass die Quelle '
|
||||
'verbleibt.'},
|
||||
'files.workflow.restore-retain-and-purge': {'limitations': ['Automatische zeitbasierte '
|
||||
'Bereinigungsplanung ist nicht '
|
||||
'enthalten; ein autorisierter '
|
||||
'handelnde Person startet Vorschau, '
|
||||
'Ausführung und Blob-Sammlung.',
|
||||
'Der Verschlüsselungsumschlag eines '
|
||||
'geschützten Blobs folgt der eigenen '
|
||||
'Aufbewahrungs- und '
|
||||
'Schlüssel-Verwahrungsrichtlinie des '
|
||||
'Verschlüsselungsmoduls.'],
|
||||
'outcome': 'Berechtigte Metadaten und nicht '
|
||||
'referenzierte Bytes werden mit separat '
|
||||
'überprüfbaren Richtlinien-, Audit- und '
|
||||
'Wiederherstellungsnachweisen gelöscht.',
|
||||
'prerequisites': ['Die handelnde Person hat die '
|
||||
'eindeutige Wiederherstellungs-, '
|
||||
'Aufbewahrungs- oder '
|
||||
'Bereinigungsberechtigung, die für '
|
||||
'die beabsichtigte Aktion '
|
||||
'erforderlich ist.',
|
||||
'Ein Bereinigungsziel ist bereits '
|
||||
'sanft gelöscht und alle abhängigen '
|
||||
'Nachweise oder Aktien wurden '
|
||||
'rechtmäßig gelöst.'],
|
||||
'steps': ['Festlegen oder Überprüfen von '
|
||||
'Aufbewahrungs- und Haltezustand unter '
|
||||
'Verwendung der angezeigten '
|
||||
'Lifecycle-Revision.',
|
||||
'Überprüfen Sie zum Wiederherstellen, ob '
|
||||
'der ursprüngliche Pfad oder das '
|
||||
'ursprüngliche Label frei ist, und stellen '
|
||||
'Sie die gleiche Ressource wieder her.',
|
||||
'Erstellen Sie zum Löschen eine begrenzte '
|
||||
'Purge-Vorschau und lösen Sie jeden '
|
||||
'benannten Blocker auf.',
|
||||
'Führen Sie die passende Vorschau mit einem '
|
||||
'idempotency-Schlüssel, PURGE-Bestätigung '
|
||||
'und Genehmigungsreferenz aus.',
|
||||
'Führen Sie eine begrenzte '
|
||||
'Blob-Garbage-Sammlung aus und prüfen Sie '
|
||||
'alle Wiederherstellungs- oder '
|
||||
'Ergebnis-unbekannten Operationen in Ops.'],
|
||||
'verification': 'Überprüfen Sie, ob die '
|
||||
'Asset-/Versionszeilen fehlen, der '
|
||||
'Blob bleibt referenziert, nicht '
|
||||
'referenzierte Bytes fehlen nach GC '
|
||||
'und jede Operation hat eine gültige '
|
||||
'Wiederherstellungsnachweiskette.'},
|
||||
'files.workflow.share-managed-files': {'limitations': ['Kampagnenziel-Zuschüsse werden '
|
||||
'normalerweise durch die '
|
||||
'Kampagnenintegration erstellt und nicht '
|
||||
'manuell im Dateidialog ausgewählt.',
|
||||
'Ein Benutzer kann den Zugriff über einen '
|
||||
'anderen aktiven direkten Zuschuss oder '
|
||||
'Besitzpfad behalten, nachdem eine Aktie '
|
||||
'widerrufen wurde.',
|
||||
'Ausgewählte benutzererklärungen erfordern '
|
||||
'die richtlinienberechtigung und sind nur '
|
||||
'diagnosen; sie geben sich nicht als '
|
||||
'benutzer aus oder gewähren dateizugriff.'],
|
||||
'outcome': 'Direkter Zugriff hat die angeforderte '
|
||||
'Berechtigung und Lebensdauer, während das '
|
||||
'verwaltete Asset seinen Eigentümer behält.',
|
||||
'prerequisites': ['Sie haben die Freigabeberechtigung für '
|
||||
'Dateien und besitzen die Datei oder '
|
||||
'verwalten Dateiräume für den aktiven '
|
||||
'Mandanten.',
|
||||
'Das beabsichtigte Benutzer-, Gruppen-, '
|
||||
'Mandant- oder Kampagnenziel existiert '
|
||||
'und ist aktiv.'],
|
||||
'steps': ['Wählen Sie eine verwaltete Datei aus, wählen '
|
||||
'Sie Shares verwalten und überprüfen Sie die '
|
||||
'effektiven und historischen direkten Zuschüsse.',
|
||||
'Wählen Sie den beabsichtigten Benutzer, die '
|
||||
'Gruppe oder den Mandanten aus und wählen Sie '
|
||||
'den Zugriff lesen, schreiben oder verwalten '
|
||||
'sowie einen optionalen Ablauf.',
|
||||
'Geben oder aktualisieren Sie die Freigabe, ohne '
|
||||
'den Dateibesitz zu ändern.',
|
||||
'Widerrufen Sie einen Zuschuss, wenn er nicht '
|
||||
'mehr benötigt wird; wiederholter Widerruf ist '
|
||||
'ein No-Op.'],
|
||||
'verification': 'Testen Sie einen beabsichtigten und einen '
|
||||
'verweigerten Pfad, verfallen oder '
|
||||
'widerrufen Sie die Gewährung und stellen '
|
||||
'Sie sicher, dass nur noch unabhängige '
|
||||
'Zugangspfade verbleiben.'}}
|
||||
|
||||
GERMAN_STRUCTURED_TRANSLATIONS["files.workflow.unpack-managed-archive"] = _MANAGED_ARCHIVE_GERMAN
|
||||
+1222
-111
File diff suppressed because it is too large
Load Diff
+147
@@ -0,0 +1,147 @@
|
||||
"""add purpose-bound Form evidence upload grants
|
||||
|
||||
Revision ID: a2b3c4d5e6f8
|
||||
Revises: f1a2b3c4d5e7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "a2b3c4d5e6f8"
|
||||
down_revision = "f1a2b3c4d5e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"file_form_evidence_grants",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("form_instance_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("form_definition_id", sa.String(length=255), nullable=False),
|
||||
sa.Column("form_definition_revision", sa.String(length=255), nullable=False),
|
||||
sa.Column("token_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||
sa.Column("request_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("custodian_user_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("evidence_kind", sa.String(length=30), nullable=False),
|
||||
sa.Column("purpose", sa.String(length=500), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("max_size_bytes", sa.Integer(), nullable=False),
|
||||
sa.Column("allowed_content_types", sa.JSON(), nullable=False),
|
||||
sa.Column("file_asset_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("file_version_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("uploaded_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["custodian_user_id"],
|
||||
["access_users.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["file_asset_id"],
|
||||
["file_assets.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["file_version_id"],
|
||||
["file_versions.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_file_form_evidence_grants_idempotency",
|
||||
),
|
||||
sa.UniqueConstraint("token_sha256"),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_tenant_id",
|
||||
"file_form_evidence_grants",
|
||||
["tenant_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_form_instance_id",
|
||||
"file_form_evidence_grants",
|
||||
["form_instance_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_custodian_user_id",
|
||||
"file_form_evidence_grants",
|
||||
["custodian_user_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_status",
|
||||
"file_form_evidence_grants",
|
||||
["status"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_expires_at",
|
||||
"file_form_evidence_grants",
|
||||
["expires_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_file_asset_id",
|
||||
"file_form_evidence_grants",
|
||||
["file_asset_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_file_version_id",
|
||||
"file_form_evidence_grants",
|
||||
["file_version_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
"file_form_evidence_grants",
|
||||
[
|
||||
"tenant_id",
|
||||
"form_instance_id",
|
||||
"form_definition_id",
|
||||
"form_definition_revision",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_file_version_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_file_asset_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_expires_at",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_status",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_custodian_user_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_form_instance_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_tenant_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_table("file_form_evidence_grants")
|
||||
@@ -0,0 +1,52 @@
|
||||
"""govern managed file retention and legal holds
|
||||
|
||||
Revision ID: a2b3c4d5e6f9
|
||||
Revises: f1a2b3c4d5e7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "a2b3c4d5e6f9"
|
||||
down_revision = "f1a2b3c4d5e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_assets") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("retained_until", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"legal_hold", sa.Boolean(), nullable=False, server_default=sa.false()
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"lifecycle_revision", sa.Integer(), nullable=False, server_default="1"
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("lifecycle_reason", sa.String(length=500), nullable=True)
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_assets_retained_until"), ["retained_until"], unique=False
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_assets_legal_hold"), ["legal_hold"], unique=False
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_assets") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_assets_legal_hold"))
|
||||
batch_op.drop_index(op.f("ix_file_assets_retained_until"))
|
||||
batch_op.drop_column("lifecycle_reason")
|
||||
batch_op.drop_column("lifecycle_revision")
|
||||
batch_op.drop_column("legal_hold")
|
||||
batch_op.drop_column("retained_until")
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
"""Index source identities without merging or discarding legacy copies.
|
||||
|
||||
Revision ID: a2b3c4d5e701
|
||||
Revises: a2b3c4d5e6f8, a2b3c4d5e6f9
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "a2b3c4d5e701"
|
||||
down_revision = ("a2b3c4d5e6f8", "a2b3c4d5e6f9")
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _digest(metadata: object) -> str | None:
|
||||
# Frozen v1 identity semantics: never import evolving application code.
|
||||
provenance = (
|
||||
metadata.get("source_provenance") if isinstance(metadata, dict) else None
|
||||
)
|
||||
if not isinstance(provenance, dict):
|
||||
return None
|
||||
|
||||
def clean(value: object) -> str | None:
|
||||
return (str(value).strip() or None) if value is not None else None
|
||||
|
||||
connector = clean(provenance.get("connector_id"))
|
||||
provider = clean(provenance.get("provider"))
|
||||
external = clean(provenance.get("external_id"))
|
||||
path = clean(provenance.get("external_path"))
|
||||
extra = (
|
||||
provenance.get("metadata")
|
||||
if isinstance(provenance.get("metadata"), dict)
|
||||
else {}
|
||||
)
|
||||
if connector and external:
|
||||
identity = ("external_id", connector, provider, external)
|
||||
elif connector and path:
|
||||
identity = (
|
||||
"external_path",
|
||||
connector,
|
||||
provider,
|
||||
clean(
|
||||
extra.get("library_id") or extra.get("profile_id") or extra.get("share")
|
||||
),
|
||||
path,
|
||||
)
|
||||
else:
|
||||
return None
|
||||
return hashlib.sha256(
|
||||
json.dumps(identity, ensure_ascii=True, separators=(",", ":")).encode("ascii")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"file_assets", sa.Column("source_identity_hash", sa.String(64), nullable=True)
|
||||
)
|
||||
table = sa.table(
|
||||
"file_assets",
|
||||
sa.column("id", sa.String),
|
||||
sa.column("metadata", sa.JSON),
|
||||
sa.column("source_identity_hash", sa.String),
|
||||
)
|
||||
connection = op.get_bind()
|
||||
cursor = None
|
||||
while True:
|
||||
statement = (
|
||||
sa.select(table.c.id, table.c.metadata).order_by(table.c.id).limit(500)
|
||||
)
|
||||
if cursor is not None:
|
||||
statement = statement.where(table.c.id > cursor)
|
||||
rows = connection.execute(statement).all()
|
||||
if not rows:
|
||||
break
|
||||
for row in rows:
|
||||
digest = _digest(row.metadata)
|
||||
if digest is not None:
|
||||
connection.execute(
|
||||
table.update()
|
||||
.where(table.c.id == row.id)
|
||||
.values(source_identity_hash=digest)
|
||||
)
|
||||
cursor = rows[-1].id
|
||||
# Deliberately non-unique: retained copies/legacy duplicates survive intact.
|
||||
# Runtime sync reports ambiguity instead of selecting a newest/winning copy.
|
||||
for owner in ("user", "group"):
|
||||
op.create_index(
|
||||
f"ix_file_assets_{owner}_source",
|
||||
"file_assets",
|
||||
["tenant_id", "owner_type", f"owner_{owner}_id", "source_identity_hash"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for owner in ("user", "group"):
|
||||
op.drop_index(f"ix_file_assets_{owner}_source", table_name="file_assets")
|
||||
with op.batch_alter_table("file_assets") as batch:
|
||||
batch.drop_column("source_identity_hash")
|
||||
@@ -0,0 +1,113 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from urllib.parse import quote
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.records import (
|
||||
RecordContractError,
|
||||
RecordSourceLocator,
|
||||
RecordSourceReference,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileBlob, FileVersion
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import get_asset_for_user
|
||||
|
||||
|
||||
CAPABILITY_RECORD_SOURCE_FILES = "records.source.files"
|
||||
|
||||
|
||||
class FilesRecordSource:
|
||||
provider_id = "files"
|
||||
|
||||
def resource_types(self) -> Sequence[str]:
|
||||
return ("file_version",)
|
||||
|
||||
def resolve(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
locator: RecordSourceLocator,
|
||||
purpose: str,
|
||||
) -> RecordSourceReference:
|
||||
if not isinstance(session, Session):
|
||||
raise RecordContractError(
|
||||
"Files record references require a database session."
|
||||
)
|
||||
tenant_id = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not tenant_id or locator.tenant_id != tenant_id:
|
||||
raise RecordContractError("Files record references cannot cross tenants.")
|
||||
if locator.source_module != "files" or locator.resource_type != "file_version":
|
||||
raise RecordContractError("Unsupported Files record source type.")
|
||||
if not str(purpose or "").strip():
|
||||
raise RecordContractError("Files record references require a purpose.")
|
||||
if not hasattr(principal, "has") or not (
|
||||
principal.has("files:file:read") or principal.has("files:file:admin")
|
||||
):
|
||||
raise RecordContractError("Current Files read permission is required.")
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(
|
||||
getattr(user, "id", "") or getattr(principal, "membership_id", "") or ""
|
||||
).strip()
|
||||
if not user_id:
|
||||
raise RecordContractError(
|
||||
"Files record references require a tenant user principal."
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=locator.resource_id,
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise RecordContractError(str(exc)) from exc
|
||||
version_query = session.query(FileVersion).filter(
|
||||
FileVersion.tenant_id == tenant_id,
|
||||
FileVersion.file_asset_id == asset.id,
|
||||
)
|
||||
revision = locator.source_revision.strip()
|
||||
version = version_query.filter(FileVersion.id == revision).one_or_none()
|
||||
if version is None:
|
||||
raise RecordContractError("The exact file version does not exist.")
|
||||
blob = session.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != tenant_id:
|
||||
raise RecordContractError(
|
||||
"The exact file version has no managed content object."
|
||||
)
|
||||
if blob.quarantined_at is not None or blob.integrity_status == "failed":
|
||||
raise RecordContractError(
|
||||
"The exact file version failed the current integrity gate."
|
||||
)
|
||||
return RecordSourceReference(
|
||||
locator=locator,
|
||||
label=version.filename_at_upload,
|
||||
authority_mode="external_authoritative",
|
||||
content_sha256=version.checksum_sha256,
|
||||
content_type=version.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
recorded_at=version.created_at,
|
||||
launch_url=(
|
||||
f"/files?fileId={quote(asset.id, safe='')}&versionId={quote(version.id, safe='')}"
|
||||
),
|
||||
metadata={
|
||||
"display_path": version.display_path_at_upload,
|
||||
"version_number": version.version_number,
|
||||
"integrity_status": blob.integrity_status,
|
||||
"protection": blob.protection_discriminator,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def create_files_record_source(_context: object) -> FilesRecordSource:
|
||||
return FilesRecordSource()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CAPABILITY_RECORD_SOURCE_FILES",
|
||||
"FilesRecordSource",
|
||||
"create_files_record_source",
|
||||
]
|
||||
@@ -1,5 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
@@ -62,7 +64,8 @@ from govoplan_files.backend.storage.connector_credential_store import (
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_imports import read_connector_file
|
||||
from govoplan_files.backend.storage.connector_imports import ConnectorRevisionConflict, read_connector_file
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
@@ -406,6 +409,8 @@ def _ensure_campaign_file_access(
|
||||
|
||||
def _http_error(exc: Exception, *, not_found: bool = False) -> HTTPException:
|
||||
code = status.HTTP_404_NOT_FOUND if not_found else status.HTTP_400_BAD_REQUEST
|
||||
if isinstance(exc, FileSourceConflict):
|
||||
code = status.HTTP_409_CONFLICT
|
||||
return HTTPException(status_code=code, detail=str(exc))
|
||||
|
||||
|
||||
@@ -860,12 +865,23 @@ def _download_connector_payload(
|
||||
path=source_path,
|
||||
max_bytes=settings.file_upload_max_bytes,
|
||||
)
|
||||
expected = str(payload.source_revision or "").strip()
|
||||
observed = {downloaded.revision}
|
||||
if profile.provider == "s3":
|
||||
observed.add(downloaded.metadata.get("etag"))
|
||||
if expected and expected not in observed:
|
||||
raise ConnectorRevisionConflict("Connector source revision changed or is unavailable; refresh the source before importing")
|
||||
provider_metadata = dict(downloaded.metadata)
|
||||
checksum_claims = {key: provider_metadata.pop(key) for key in tuple(provider_metadata) if key.startswith("checksum_")}
|
||||
provenance_metadata = {
|
||||
**provider_metadata,
|
||||
"profile_id": profile.id,
|
||||
"library_id": payload.library_id,
|
||||
"library_path": source_path,
|
||||
**downloaded.metadata,
|
||||
**payload.metadata,
|
||||
"size": len(downloaded.data),
|
||||
"acquired_sha256": hashlib.sha256(downloaded.data).hexdigest(),
|
||||
"provider_checksum_claims": checksum_claims,
|
||||
"import_annotations": dict(payload.metadata),
|
||||
}
|
||||
metadata = source_metadata(
|
||||
source_provenance={
|
||||
@@ -878,7 +894,7 @@ def _download_connector_payload(
|
||||
"external_url": downloaded.external_url,
|
||||
"metadata": provenance_metadata,
|
||||
},
|
||||
source_revision=payload.source_revision or downloaded.revision,
|
||||
source_revision=downloaded.revision,
|
||||
)
|
||||
return source_path, downloaded, metadata or {}
|
||||
|
||||
@@ -1039,6 +1055,12 @@ def _asset_response(
|
||||
created_at=asset.created_at.isoformat(),
|
||||
updated_at=asset.updated_at.isoformat(),
|
||||
deleted_at=asset.deleted_at.isoformat() if asset.deleted_at else None,
|
||||
retained_until=(
|
||||
asset.retained_until.isoformat() if asset.retained_until else None
|
||||
),
|
||||
legal_hold=asset.legal_hold,
|
||||
lifecycle_revision=asset.lifecycle_revision,
|
||||
lifecycle_reason=asset.lifecycle_reason,
|
||||
audit_relevant=asset_is_audit_relevant(session, asset),
|
||||
metadata=metadata,
|
||||
source_provenance=source_provenance_from_metadata(metadata),
|
||||
@@ -1144,6 +1166,10 @@ def _loaded_asset_response(
|
||||
created_at=asset.created_at.isoformat(),
|
||||
updated_at=asset.updated_at.isoformat(),
|
||||
deleted_at=asset.deleted_at.isoformat() if asset.deleted_at else None,
|
||||
retained_until=asset.retained_until.isoformat() if asset.retained_until else None,
|
||||
legal_hold=asset.legal_hold,
|
||||
lifecycle_revision=asset.lifecycle_revision,
|
||||
lifecycle_reason=asset.lifecycle_reason,
|
||||
audit_relevant=asset.id in sent_asset_ids,
|
||||
metadata=metadata,
|
||||
source_provenance=source_provenance_from_metadata(metadata),
|
||||
|
||||
@@ -11,21 +11,27 @@ from govoplan_files.backend.routes.connector_settings import (
|
||||
router as connector_settings_router,
|
||||
)
|
||||
from govoplan_files.backend.routes.folders import router as folders_router
|
||||
from govoplan_files.backend.routes.form_evidence import router as form_evidence_router
|
||||
from govoplan_files.backend.routes.integrity import router as integrity_router
|
||||
from govoplan_files.backend.routes.lifecycle import router as lifecycle_router
|
||||
from govoplan_files.backend.routes.listing import router as listing_router
|
||||
from govoplan_files.backend.routes.shares import router as shares_router
|
||||
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||
from govoplan_files.backend.routes.managed_archives import router as managed_archives_router
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
for workflow_router in (
|
||||
spaces_router,
|
||||
folders_router,
|
||||
form_evidence_router,
|
||||
integrity_router,
|
||||
lifecycle_router,
|
||||
listing_router,
|
||||
uploads_router,
|
||||
managed_archives_router,
|
||||
connector_settings_router,
|
||||
connector_io_router,
|
||||
connector_profiles_router,
|
||||
|
||||
@@ -16,6 +16,7 @@ from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
read_asset_version_bytes,
|
||||
soft_delete_assets,
|
||||
)
|
||||
|
||||
@@ -85,6 +86,46 @@ def download_file(
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{file_id}/versions/{version_id}/download")
|
||||
def download_file_version(
|
||||
file_id: str,
|
||||
version_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_version_bytes(session, asset, version_id)
|
||||
_audit_connector_event(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.accessed",
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
operation="download-version",
|
||||
commit=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {
|
||||
"Content-Disposition": _attachment_disposition(
|
||||
version.filename_at_upload or asset.filename
|
||||
)
|
||||
}
|
||||
return StreamingResponse(
|
||||
BytesIO(data),
|
||||
media_type=blob.content_type or "application/octet-stream",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
||||
def delete_file(
|
||||
file_id: str,
|
||||
|
||||
@@ -5,11 +5,18 @@ from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorBrowseItem,
|
||||
FileConnectorBrowseResponse,
|
||||
FileConnectorFolderSyncItemResponse,
|
||||
FileConnectorFolderSyncRequest,
|
||||
FileConnectorFolderSyncResponse,
|
||||
FileConnectorFolderSyncSummary,
|
||||
FileConnectorImportRequest,
|
||||
FileConnectorSyncResponse,
|
||||
FileConnectorWriteRequest,
|
||||
FileConnectorWriteResponse,
|
||||
FileUploadResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
@@ -21,10 +28,16 @@ from govoplan_files.backend.storage.connector_browse import (
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
from govoplan_files.backend.storage.connector_imports import (
|
||||
ConnectorImportError,
|
||||
ConnectorImportUnsupported,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_folder_sync import (
|
||||
connector_relative_path,
|
||||
discover_connector_folder,
|
||||
join_connector_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
@@ -35,8 +48,15 @@ from govoplan_files.backend.storage.connector_policy import (
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
sync_file_asset_from_source,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_spaces import (
|
||||
connector_space_owner_id,
|
||||
get_connector_space_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_writes import write_connector_file
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
@@ -45,6 +65,7 @@ from govoplan_files.backend.route_support import (
|
||||
_audit_connector_sync,
|
||||
_connector_browse_next_token,
|
||||
_connector_policy_error,
|
||||
_connector_space_policy_decision,
|
||||
_download_connector_payload,
|
||||
_ensure_campaign_file_access,
|
||||
_http_error,
|
||||
@@ -55,6 +76,365 @@ from govoplan_files.backend.route_support import (
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/sync",
|
||||
response_model=FileConnectorFolderSyncResponse,
|
||||
)
|
||||
def sync_connector_space_folder(
|
||||
space_id: str,
|
||||
payload: FileConnectorFolderSyncRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
try:
|
||||
if space.sync_mode != "manual":
|
||||
raise FileStorageError("This connector space is not configured for manual sync")
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
remote_path = join_connector_path(space.remote_path, payload.path)
|
||||
target_folder = join_connector_path(payload.target_folder)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=space.library_id,
|
||||
remote_path=remote_path,
|
||||
operation="sync",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
discovery = discover_connector_folder(
|
||||
profile,
|
||||
path=remote_path,
|
||||
library_id=space.library_id,
|
||||
recursive=payload.recursive,
|
||||
max_files=payload.max_files,
|
||||
max_depth=payload.max_depth,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (ConnectorBrowseError, FileStorageError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
owner_id = connector_space_owner_id(space)
|
||||
items: list[FileConnectorFolderSyncItemResponse] = [
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=skipped.path,
|
||||
action="skipped",
|
||||
detail=skipped.reason,
|
||||
)
|
||||
for skipped in discovery.skipped
|
||||
]
|
||||
counts = {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"skipped": len(discovery.skipped),
|
||||
"conflicts": 0,
|
||||
"policy_denied": 0,
|
||||
"failed": 0,
|
||||
}
|
||||
|
||||
for source in discovery.files:
|
||||
source_path = normalize_connector_browse_path(source.path)
|
||||
try:
|
||||
relative_path = connector_relative_path(
|
||||
space_root=space.remote_path,
|
||||
item_path=source_path,
|
||||
)
|
||||
target_path = join_connector_path(target_folder, relative_path)
|
||||
except (ConnectorBrowseError, ValueError) as exc:
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
action="failed",
|
||||
detail=str(exc),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
try:
|
||||
with session.begin_nested():
|
||||
file_payload = FileConnectorImportRequest(
|
||||
library_id=space.library_id or "",
|
||||
path=source_path,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=owner_id,
|
||||
target_path=target_path,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
source_revision=source.etag,
|
||||
metadata={
|
||||
**dict(source.metadata),
|
||||
**payload.metadata,
|
||||
"connector_space_id": space.id,
|
||||
"browse_name": source.name,
|
||||
"browse_path": source_path,
|
||||
"browse_modified_at": source.modified_at,
|
||||
"browse_etag": source.etag,
|
||||
"folder_sync": True,
|
||||
},
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, file_payload, operation="sync"
|
||||
)
|
||||
stored, sync_action, previous_version_id = (
|
||||
sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
display_path=target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
)
|
||||
_audit_connector_sync(
|
||||
session,
|
||||
principal,
|
||||
stored.asset,
|
||||
sync_action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
)
|
||||
file_response = _asset_response(
|
||||
session, stored.asset, include_shares=True
|
||||
)
|
||||
counts[sync_action] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action=sync_action,
|
||||
file=file_response,
|
||||
previous_version_id=previous_version_id,
|
||||
current_version_id=stored.version.id,
|
||||
source_revision=file_response.source_revision,
|
||||
)
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
counts["policy_denied"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action="policy_denied",
|
||||
source_revision=source.etag,
|
||||
detail=str(exc),
|
||||
policy_decision=exc.decision.to_dict(),
|
||||
)
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
detail = str(exc)
|
||||
if detail.startswith("Skipped upload target:"):
|
||||
action = "skipped"
|
||||
counts["skipped"] += 1
|
||||
elif isinstance(exc, FileSourceConflict) or detail.startswith("Target file already exists:"):
|
||||
action = "conflict"
|
||||
counts["conflicts"] += 1
|
||||
else:
|
||||
action = "failed"
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action=action,
|
||||
source_revision=source.etag,
|
||||
detail=detail,
|
||||
)
|
||||
)
|
||||
except (
|
||||
ConnectorImportError,
|
||||
UnsafeFilePathError,
|
||||
OSError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action="failed",
|
||||
source_revision=source.etag,
|
||||
detail=str(exc),
|
||||
)
|
||||
)
|
||||
|
||||
summary = FileConnectorFolderSyncSummary(
|
||||
discovered=len(discovery.files),
|
||||
**counts,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.folder_synced",
|
||||
object_type="file_connector_space",
|
||||
object_id=space.id,
|
||||
details={
|
||||
"connector_profile_id": profile.id,
|
||||
"provider": profile.provider,
|
||||
"library_id": space.library_id,
|
||||
"remote_path": remote_path,
|
||||
"target_folder": target_folder,
|
||||
"recursive": payload.recursive,
|
||||
"truncated": discovery.truncated,
|
||||
"summary": summary.model_dump(),
|
||||
"results": [
|
||||
{
|
||||
"source_path": item.source_path,
|
||||
"target_path": item.target_path,
|
||||
"action": item.action,
|
||||
"file_id": item.file.id if item.file else None,
|
||||
"current_version_id": item.current_version_id,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorFolderSyncResponse(
|
||||
connector_space_id=space.id,
|
||||
connector_profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
remote_path=remote_path,
|
||||
target_folder=target_folder,
|
||||
recursive=payload.recursive,
|
||||
truncated=discovery.truncated,
|
||||
summary=summary,
|
||||
items=items,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/write-back",
|
||||
response_model=FileConnectorWriteResponse,
|
||||
)
|
||||
def write_back_connector_file(
|
||||
space_id: str,
|
||||
payload: FileConnectorWriteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:connector:write")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if space.read_only:
|
||||
raise FileStorageError("This connector space is read-only")
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
requested_path = normalize_connector_browse_path(payload.remote_path)
|
||||
remote_path = normalize_connector_browse_path(
|
||||
"/".join(part for part in (space.remote_path, requested_path) if part)
|
||||
)
|
||||
decision = connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path=remote_path,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation="write",
|
||||
),
|
||||
profile.policy_sources,
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=payload.file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
connector_library_id = space.library_id
|
||||
content_type = blob.content_type
|
||||
# Close the read snapshot before the independent recovery transaction
|
||||
# records authority for the external effect. This avoids upgrading an
|
||||
# older SQLite read snapshot after the ledger commit.
|
||||
session.commit()
|
||||
result = write_connector_file(
|
||||
profile,
|
||||
tenant_id=principal.tenant_id,
|
||||
library_id=connector_library_id,
|
||||
remote_path=remote_path,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
expected_revision=payload.expected_revision,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.written",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={
|
||||
"file_version_id": version.id,
|
||||
"file_blob_id": blob.id,
|
||||
"checksum_sha256": blob.checksum_sha256,
|
||||
"connector_space_id": space.id,
|
||||
"connector_profile_id": profile.id,
|
||||
"remote_path": remote_path,
|
||||
"recovery_operation_id": result.recovery_operation_id,
|
||||
"recovery_status": result.status,
|
||||
"revision": result.revision,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorWriteResponse(
|
||||
recovery_operation_id=result.recovery_operation_id,
|
||||
status=result.status,
|
||||
replayed=result.replayed,
|
||||
provider=result.provider,
|
||||
remote_path=result.remote_path,
|
||||
revision=result.revision,
|
||||
checksum_sha256=result.checksum_sha256,
|
||||
size_bytes=result.size_bytes,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ConnectorBrowseError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles/{profile_id}/import", response_model=FileUploadResponse
|
||||
)
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
|
||||
from fastapi import (
|
||||
APIRouter,
|
||||
Depends,
|
||||
File as FastAPIFile,
|
||||
Header,
|
||||
HTTPException,
|
||||
UploadFile,
|
||||
status,
|
||||
)
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.events import (
|
||||
EventActorRef,
|
||||
EventObjectRef,
|
||||
EventTenantRef,
|
||||
PlatformEvent,
|
||||
emit_platform_event,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import FileFormEvidenceGrant
|
||||
from govoplan_files.backend.route_support import _read_limited_upload
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import create_file_asset
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/form-evidence/upload",
|
||||
response_model=dict[str, object],
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def upload_form_evidence(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
x_form_evidence_token: str = Header(alias="X-Form-Evidence-Token"),
|
||||
session: Session = Depends(get_session),
|
||||
) -> dict[str, object]:
|
||||
now = datetime.now(UTC)
|
||||
try:
|
||||
grant = (
|
||||
session.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.token_sha256
|
||||
== _token_sha256(x_form_evidence_token)
|
||||
)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if grant is None or grant.status != "issued" or _aware(grant.expires_at) <= now:
|
||||
raise _upload_unavailable()
|
||||
content_type = (
|
||||
str(file.content_type or "application/octet-stream")
|
||||
.split(";", 1)[0]
|
||||
.strip()
|
||||
.casefold()
|
||||
)
|
||||
if grant.allowed_content_types and content_type not in set(
|
||||
grant.allowed_content_types
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
|
||||
detail="This Form does not accept the uploaded document type.",
|
||||
)
|
||||
data = _read_limited_upload(file, max_bytes=grant.max_size_bytes)
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=grant.tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=grant.custodian_user_id,
|
||||
user_id=grant.custodian_user_id,
|
||||
filename=file.filename or "form-attachment",
|
||||
data=data,
|
||||
folder=f"Form submissions/{grant.form_instance_id}",
|
||||
content_type=content_type,
|
||||
description="Managed attachment captured through Forms Runtime.",
|
||||
metadata={
|
||||
"form_evidence": {
|
||||
"grant_id": grant.id,
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"form_definition_id": grant.form_definition_id,
|
||||
"form_definition_revision": grant.form_definition_revision,
|
||||
"purpose": grant.purpose,
|
||||
},
|
||||
"source_provenance": {
|
||||
"source_type": "form_evidence",
|
||||
"connector_id": "forms_runtime.evidence.files",
|
||||
"provider": "forms_runtime",
|
||||
"external_id": grant.id,
|
||||
"revision": grant.form_definition_revision,
|
||||
},
|
||||
},
|
||||
conflict_strategy="rename",
|
||||
is_admin=True,
|
||||
)
|
||||
grant.status = "uploaded"
|
||||
grant.file_asset_id = stored.asset.id
|
||||
grant.file_version_id = stored.version.id
|
||||
grant.uploaded_at = now
|
||||
session.add(grant)
|
||||
evidence = EvidenceReference(
|
||||
kind="document",
|
||||
owner_module="files",
|
||||
evidence_id=stored.asset.id,
|
||||
tenant_id=grant.tenant_id,
|
||||
version=stored.version.id,
|
||||
checksum=stored.version.checksum_sha256,
|
||||
source_ref=(f"form_submission:{grant.form_instance_id}:grant:{grant.id}"),
|
||||
responsible_actor_ref=f"user:{grant.custodian_user_id}",
|
||||
captured_at=now,
|
||||
)
|
||||
emit_platform_event(
|
||||
session,
|
||||
PlatformEvent(
|
||||
type="files.form_evidence.uploaded",
|
||||
module_id="files",
|
||||
payload={
|
||||
"grant_id": grant.id,
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"file_asset_id": stored.asset.id,
|
||||
"file_version_id": stored.version.id,
|
||||
"checksum_sha256": stored.version.checksum_sha256,
|
||||
"size_bytes": stored.version.size_bytes,
|
||||
},
|
||||
occurred_at=now,
|
||||
actor=EventActorRef(type="user", id=grant.custodian_user_id),
|
||||
tenant=EventTenantRef(id=grant.tenant_id),
|
||||
resource=EventObjectRef(type="file", id=stored.asset.id),
|
||||
classification="confidential",
|
||||
),
|
||||
)
|
||||
session.commit()
|
||||
except HTTPException:
|
||||
session.rollback()
|
||||
raise
|
||||
except (FileStorageError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
return {
|
||||
"grant_id": grant.id,
|
||||
"evidence": evidence.to_dict(include_inspection=True),
|
||||
}
|
||||
|
||||
|
||||
def _token_sha256(token: str) -> str:
|
||||
clean = str(token or "").strip()
|
||||
if len(clean) < 32:
|
||||
raise _upload_unavailable()
|
||||
return hashlib.sha256(clean.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _upload_unavailable() -> HTTPException:
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="The Form evidence upload is unavailable.",
|
||||
)
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = ["router", "upload_form_evidence"]
|
||||
@@ -0,0 +1,297 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_connector_space_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileAssetResponse,
|
||||
FileBlobGcRequest,
|
||||
FileBlobGcResponse,
|
||||
FileConnectorSpaceResponse,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderRestoreResponse,
|
||||
FileLifecycleUpdateRequest,
|
||||
FilePurgeExecuteRequest,
|
||||
FilePurgePreviewItem,
|
||||
FilePurgePreviewRequest,
|
||||
FilePurgePreviewResponse,
|
||||
FilePurgeResponse,
|
||||
FileRestoreResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.lifecycle import (
|
||||
execute_asset_purge,
|
||||
garbage_collect_unreferenced_blobs,
|
||||
get_asset_for_lifecycle,
|
||||
preview_asset_purge,
|
||||
restore_asset,
|
||||
restore_connector_space,
|
||||
restore_folder,
|
||||
set_asset_lifecycle,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files-lifecycle"])
|
||||
|
||||
|
||||
@router.patch("/{file_id}/lifecycle", response_model=FileAssetResponse)
|
||||
def update_file_lifecycle(
|
||||
file_id: str,
|
||||
payload: FileLifecycleUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:retention")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_lifecycle(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
set_asset_lifecycle(
|
||||
session,
|
||||
asset,
|
||||
retained_until=payload.retained_until,
|
||||
legal_hold=payload.legal_hold,
|
||||
reason=payload.reason,
|
||||
expected_revision=payload.expected_revision,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.lifecycle.updated",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={
|
||||
"retained_until": (
|
||||
asset.retained_until.isoformat() if asset.retained_until else None
|
||||
),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
"reason": asset.lifecycle_reason,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/assets/{file_id}/restore", response_model=FileRestoreResponse)
|
||||
def restore_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_lifecycle(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
changed = restore_asset(session, asset)
|
||||
if changed:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.file.restored",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={"display_path": asset.display_path},
|
||||
)
|
||||
session.commit()
|
||||
return FileRestoreResponse(restored_count=1 if changed else 0)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/folders/restore", response_model=FileFolderRestoreResponse)
|
||||
def restore_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
restored_folders, restored_files = restore_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.folder.restored",
|
||||
object_type="file_folder",
|
||||
object_id=payload.path,
|
||||
details={
|
||||
"restored_folders": restored_folders,
|
||||
"restored_files": restored_files,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderRestoreResponse(
|
||||
restored_folders=restored_folders, restored_files=restored_files
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/restore",
|
||||
response_model=FileConnectorSpaceResponse,
|
||||
)
|
||||
def restore_file_connector_space(
|
||||
space_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
space = restore_connector_space(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector_space.restored",
|
||||
object_type="file_connector_space",
|
||||
object_id=space.id,
|
||||
details={"label": space.label},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/purge/preview", response_model=FilePurgePreviewResponse)
|
||||
def preview_file_purge(
|
||||
payload: FilePurgePreviewRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
preview = preview_asset_purge(
|
||||
session, tenant_id=principal.tenant_id, file_ids=payload.file_ids
|
||||
)
|
||||
return FilePurgePreviewResponse(
|
||||
preview_sha256=preview.preview_sha256,
|
||||
eligible=preview.eligible,
|
||||
items=[
|
||||
FilePurgePreviewItem(
|
||||
file_id=item.file_id,
|
||||
filename=item.filename,
|
||||
lifecycle_revision=item.lifecycle_revision,
|
||||
deleted_at=(item.deleted_at.isoformat() if item.deleted_at else None),
|
||||
retained_until=(
|
||||
item.retained_until.isoformat() if item.retained_until else None
|
||||
),
|
||||
legal_hold=item.legal_hold,
|
||||
blockers=list(item.blockers),
|
||||
blob_ids=list(item.blob_ids),
|
||||
)
|
||||
for item in preview.items
|
||||
],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/purge/execute", response_model=FilePurgeResponse)
|
||||
def execute_file_purge(
|
||||
payload: FilePurgeExecuteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
result = execute_asset_purge(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
file_ids=payload.file_ids,
|
||||
preview_sha256=payload.preview_sha256,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
approval_reference=payload.approval_reference,
|
||||
before_commit=lambda assets: audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.files.purged",
|
||||
object_type="file_batch",
|
||||
object_id=payload.preview_sha256,
|
||||
details={
|
||||
"file_ids": [asset.id for asset in assets],
|
||||
"approval_reference": payload.approval_reference,
|
||||
"preview_sha256": payload.preview_sha256,
|
||||
},
|
||||
),
|
||||
)
|
||||
return FilePurgeResponse(
|
||||
recovery_operation_id=result.recovery_operation_id,
|
||||
status=result.status,
|
||||
replayed=result.replayed,
|
||||
purged_files=result.purged_files,
|
||||
released_blobs=result.released_blobs,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/purge/blobs", response_model=FileBlobGcResponse)
|
||||
def garbage_collect_file_blobs(
|
||||
payload: FileBlobGcRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
result = garbage_collect_unreferenced_blobs(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
limit=payload.limit,
|
||||
approval_reference=payload.approval_reference,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.blobs.garbage_collected",
|
||||
object_type="file_blob_batch",
|
||||
object_id=principal.tenant_id,
|
||||
details={
|
||||
"inspected_blobs": result.inspected_blobs,
|
||||
"deleted_blobs": result.deleted_blobs,
|
||||
"unresolved_operation_ids": list(result.unresolved_operation_ids),
|
||||
"approval_reference": payload.approval_reference,
|
||||
},
|
||||
commit=True,
|
||||
)
|
||||
return FileBlobGcResponse(
|
||||
inspected_blobs=result.inspected_blobs,
|
||||
deleted_blobs=result.deleted_blobs,
|
||||
unresolved_operation_ids=list(result.unresolved_operation_ids),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -1,7 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
@@ -14,6 +14,7 @@ from govoplan_files.backend.storage.files import (
|
||||
count_assets_for_user,
|
||||
list_assets_for_user,
|
||||
list_assets_for_user_window,
|
||||
list_recent_assets_for_user,
|
||||
)
|
||||
|
||||
|
||||
@@ -79,6 +80,7 @@ def list_files(
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
sort: Literal["path", "recent"] = "path",
|
||||
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
@@ -99,6 +101,31 @@ def list_files(
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if sort == "recent":
|
||||
if cursor:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="Recent file projections do not accept a path-order cursor.",
|
||||
)
|
||||
recent_limit = page_size or 25
|
||||
assets = list_recent_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
limit=recent_limit,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
watermark=watermark,
|
||||
)
|
||||
effective_page_size = _cursor_page_size(FILES_LIST_CURSOR_SCOPE, cursor, page_size)
|
||||
if effective_page_size is not None:
|
||||
fingerprint = _files_list_fingerprint(
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Extract an existing managed archive through the ordinary preview pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from io import BytesIO
|
||||
from collections.abc import Mapping
|
||||
import json
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, UploadFile
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, has_scope, require_scope
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_core.security.secrets import (
|
||||
TransientPayloadError,
|
||||
open_transient_payload,
|
||||
seal_transient_payload,
|
||||
)
|
||||
from govoplan_files.backend.route_support import (
|
||||
_connector_policy_error,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.schemas import ArchivePreviewResponse, FileUploadResponse
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.archives import archive_format_for_filename
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
ConnectorPolicyDenied,
|
||||
ensure_connector_policy_allows,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
effective_connector_policy_sources,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||
from govoplan_files.backend.routes.uploads import (
|
||||
confirm_archive_upload,
|
||||
preview_archive_upload,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
_PURPOSE = "files.managed-archive-preview.v1"
|
||||
|
||||
|
||||
class ManagedArchivePreviewRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
source_version_id: str = Field(min_length=1, max_length=128)
|
||||
owner_type: Literal["user", "group"] = "user"
|
||||
owner_id: str | None = Field(default=None, max_length=128)
|
||||
path: str = Field(default="", max_length=4096)
|
||||
password: str | None = Field(default=None, max_length=4096)
|
||||
|
||||
|
||||
class ManagedArchiveConfirmRequest(ManagedArchivePreviewRequest):
|
||||
preview_token: str = Field(min_length=1, max_length=32768)
|
||||
selected_paths: list[str] = Field(min_length=1, max_length=10_000)
|
||||
operation_id: str | None = Field(default=None, min_length=36, max_length=36)
|
||||
|
||||
|
||||
def _resolve_source(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
file_id: str,
|
||||
payload: ManagedArchivePreviewRequest,
|
||||
):
|
||||
for scope in ("files:file:read", "files:file:download"):
|
||||
if not has_scope(principal, scope):
|
||||
raise HTTPException(status_code=403, detail=f"Missing scope: {scope}")
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if asset.current_version_id != payload.source_version_id:
|
||||
raise FileStorageError(
|
||||
"The managed archive version changed; reload the file and preview it again"
|
||||
)
|
||||
archive_format_for_filename(asset.filename)
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
if (
|
||||
version.file_asset_id != asset.id
|
||||
or version.tenant_id != principal.tenant_id
|
||||
or blob.tenant_id != principal.tenant_id
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Managed archive version or content does not belong to this source"
|
||||
)
|
||||
if blob.size_bytes > settings.file_upload_zip_max_bytes:
|
||||
raise FileStorageError(
|
||||
"Managed archive exceeds the configured archive size limit"
|
||||
)
|
||||
provenance = source_provenance_from_metadata(asset.metadata_ or {})
|
||||
if provenance:
|
||||
source_owner = (
|
||||
asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id
|
||||
)
|
||||
sources = []
|
||||
for scope_type, scope_id in dict.fromkeys(
|
||||
(
|
||||
(asset.owner_type, source_owner),
|
||||
(payload.owner_type, target_owner),
|
||||
("user", principal.user.id),
|
||||
)
|
||||
):
|
||||
sources.extend(
|
||||
effective_connector_policy_sources(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
ensure_connector_policy_allows(
|
||||
ConnectorAccessRequest.from_provenance(provenance, operation="import"),
|
||||
sources,
|
||||
)
|
||||
# This is the same integrity/quarantine and envelope-decryption path used
|
||||
# by authorized managed-file downloads; never read raw backend bytes here.
|
||||
data, version, _blob = read_asset_bytes(session, asset)
|
||||
if version.id != payload.source_version_id:
|
||||
raise FileStorageError(
|
||||
"The managed archive version changed during the verified read; reload and preview again"
|
||||
)
|
||||
if len(data) > settings.file_upload_zip_max_bytes:
|
||||
raise FileStorageError(
|
||||
"Managed archive exceeds the configured archive size limit"
|
||||
)
|
||||
return asset, version, data, target_owner, provenance
|
||||
|
||||
|
||||
@router.post("/{file_id}/archive-preview", response_model=ArchivePreviewResponse)
|
||||
def preview_managed_archive(
|
||||
file_id: str,
|
||||
payload: ManagedArchivePreviewRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
asset, version, data, owner, _provenance = _resolve_source(
|
||||
session, principal, file_id, payload
|
||||
)
|
||||
with BytesIO(data) as source:
|
||||
preview = preview_archive_upload(
|
||||
file=UploadFile(file=source, filename=asset.filename),
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=owner,
|
||||
path=payload.path,
|
||||
campaign_id=None,
|
||||
password=payload.password,
|
||||
principal=principal,
|
||||
)
|
||||
preview.preview_token = seal_transient_payload(
|
||||
{
|
||||
"purpose": _PURPOSE,
|
||||
"file_id": asset.id,
|
||||
"version_id": version.id,
|
||||
"tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id,
|
||||
"owner_type": payload.owner_type,
|
||||
"owner_id": owner,
|
||||
"path": normalize_folder(payload.path),
|
||||
"upload_preview_token": preview.preview_token,
|
||||
}
|
||||
)
|
||||
return preview
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/{file_id}/archive-confirm", response_model=FileUploadResponse)
|
||||
def confirm_managed_archive(
|
||||
file_id: str,
|
||||
payload: ManagedArchiveConfirmRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
token = open_transient_payload(
|
||||
payload.preview_token, ttl_seconds=settings.file_archive_preview_ttl_seconds
|
||||
)
|
||||
expected = {
|
||||
"purpose": _PURPOSE,
|
||||
"file_id": file_id,
|
||||
"version_id": payload.source_version_id,
|
||||
"tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id,
|
||||
"owner_type": payload.owner_type,
|
||||
"owner_id": payload.owner_id or principal.user.id,
|
||||
"path": normalize_folder(payload.path),
|
||||
}
|
||||
if any(
|
||||
token.get(key) != value for key, value in expected.items()
|
||||
) or not isinstance(token.get("upload_preview_token"), str):
|
||||
raise FileStorageError(
|
||||
"Managed archive preview does not match this source, actor, or destination; preview again"
|
||||
)
|
||||
asset, version, data, owner, provenance = _resolve_source(
|
||||
session, principal, file_id, payload
|
||||
)
|
||||
source_provenance = dict(
|
||||
provenance or {"source_type": "managed_archive", "external_id": asset.id}
|
||||
)
|
||||
provenance_metadata = source_provenance.get("metadata")
|
||||
source_provenance["metadata"] = {
|
||||
**(
|
||||
dict(provenance_metadata)
|
||||
if isinstance(provenance_metadata, Mapping)
|
||||
else {}
|
||||
),
|
||||
"archive_source_file_id": asset.id,
|
||||
"archive_source_version_id": version.id,
|
||||
}
|
||||
with BytesIO(data) as source:
|
||||
# Reject collisions, including an archive member targeting its own
|
||||
# source. The original file/version can never be overwritten here.
|
||||
return confirm_archive_upload(
|
||||
file=UploadFile(file=source, filename=asset.filename),
|
||||
preview_token=token["upload_preview_token"],
|
||||
selected_paths_json=json.dumps(payload.selected_paths),
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=owner,
|
||||
path=payload.path,
|
||||
campaign_id=None,
|
||||
password=payload.password,
|
||||
conflict_strategy="reject",
|
||||
conflict_resolutions_json=None,
|
||||
source_provenance_json=json.dumps(source_provenance),
|
||||
source_revision=version.id,
|
||||
connector_policy_json=None,
|
||||
encryption_vault_id=None,
|
||||
operation_id=payload.operation_id,
|
||||
session=session,
|
||||
principal=principal,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, TransientPayloadError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -27,6 +27,7 @@ from govoplan_files.backend.storage.connector_spaces import (
|
||||
list_connector_spaces_for_user,
|
||||
soft_delete_connector_space,
|
||||
update_connector_space,
|
||||
validate_connector_space_write_mode,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
@@ -145,6 +146,7 @@ def create_file_connector_space(
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
validate_connector_space_write_mode(profile, read_only=payload.read_only)
|
||||
space = create_connector_space(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
@@ -156,6 +158,7 @@ def create_file_connector_space(
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
sync_mode=payload.sync_mode,
|
||||
read_only=payload.read_only,
|
||||
metadata=payload.metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
@@ -201,7 +204,11 @@ def update_file_connector_space(
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
try:
|
||||
if payload.library_id is not None or payload.remote_path is not None:
|
||||
if (
|
||||
payload.library_id is not None
|
||||
or payload.remote_path is not None
|
||||
or payload.read_only is not None
|
||||
):
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
@@ -217,6 +224,14 @@ def update_file_connector_space(
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
validate_connector_space_write_mode(
|
||||
profile,
|
||||
read_only=(
|
||||
payload.read_only
|
||||
if payload.read_only is not None
|
||||
else space.read_only
|
||||
),
|
||||
)
|
||||
update_connector_space(
|
||||
session,
|
||||
space,
|
||||
@@ -225,6 +240,7 @@ def update_file_connector_space(
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
sync_mode=payload.sync_mode,
|
||||
read_only=payload.read_only,
|
||||
is_active=payload.is_active,
|
||||
metadata=payload.metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
|
||||
@@ -3,9 +3,11 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
from contextlib import ExitStack, contextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, UploadFile
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
@@ -24,6 +26,10 @@ from govoplan_files.backend.schemas import (
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.archive_work import (
|
||||
ArchiveProgress, ArchiveWorkExpired, discard_staged_upload, read_progress,
|
||||
stage_upload, use_staged_upload,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
UnsafeFilePathError,
|
||||
normalize_folder,
|
||||
@@ -44,7 +50,7 @@ from govoplan_files.backend.storage.files import (
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_asset_list_response,
|
||||
_audit_connector_imports,
|
||||
_cleanup_temp_file,
|
||||
_connector_policy_error,
|
||||
@@ -60,6 +66,57 @@ router = APIRouter(prefix="/files", tags=["files"])
|
||||
_ARCHIVE_PREVIEW_PURPOSE = "files.archive-preview.v1"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _archive_source(file, staged_upload_id, preview_token, principal):
|
||||
if isinstance(staged_upload_id, str):
|
||||
if file is not None and hasattr(file, "file"):
|
||||
raise FileStorageError("Choose either the staged archive or a new upload")
|
||||
if not isinstance(preview_token, str) or not preview_token:
|
||||
raise FileStorageError("A valid preview token is required for a staged archive")
|
||||
payload = open_transient_payload(preview_token, ttl_seconds=settings.file_archive_preview_ttl_seconds)
|
||||
expected = {"purpose": _ARCHIVE_PREVIEW_PURPOSE, "tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id, "staged_upload_id": staged_upload_id}
|
||||
if any(payload.get(key) != value for key, value in expected.items()) or not isinstance(payload.get("filename"), str):
|
||||
raise FileStorageError("Archive preview does not match this staged upload")
|
||||
with use_staged_upload(settings, staged_upload_id, tenant_id=principal.tenant_id, user_id=principal.user.id) as path:
|
||||
token_digest = payload.get("archive_sha256")
|
||||
if not isinstance(token_digest, str) or not hmac.compare_digest(_archive_sha256(path), token_digest):
|
||||
raise FileStorageError("Archive contents changed after preview; preview it again")
|
||||
yield path, payload["filename"]
|
||||
return
|
||||
if file is None or not hasattr(file, "file"):
|
||||
raise FileStorageError("Select an archive to upload")
|
||||
filename = file.filename or "archive"
|
||||
path = _spool_limited_upload_to_temp(file, max_bytes=settings.file_upload_zip_max_bytes, suffix=_archive_suffix(filename))
|
||||
try:
|
||||
yield path, filename
|
||||
finally:
|
||||
try:
|
||||
_cleanup_temp_file(path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
@router.get("/archive-progress/{operation_id}")
|
||||
async def archive_operation_progress(operation_id: str, principal: ApiPrincipal = Depends(require_scope("files:file:upload"))):
|
||||
# Small local receipt read stays available even while sync upload workers
|
||||
# are busy. The request owns the transaction; this is not a job retry API.
|
||||
try:
|
||||
return read_progress(settings, operation_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
except ArchiveWorkExpired as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete("/archive-staging/{stage_id}", status_code=204)
|
||||
def release_staged_archive(stage_id: str, principal: ApiPrincipal = Depends(require_scope("files:file:upload"))):
|
||||
try:
|
||||
discard_staged_upload(settings, stage_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
def _archive_suffix(filename: str) -> str:
|
||||
lowered = filename.casefold()
|
||||
for suffix in (".tar.bz2", ".tar.gz", ".tar.xz", ".tbz2", ".tgz", ".txz", ".tar", ".zip"):
|
||||
@@ -135,24 +192,22 @@ def _validate_archive_preview_token(
|
||||
|
||||
@router.post("/archive-preview", response_model=ArchivePreviewResponse)
|
||||
def preview_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
file: UploadFile | None = FastAPIFile(default=None),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
password: str | None = Form(default=None),
|
||||
retain_upload: bool = Form(default=False),
|
||||
staged_upload_id: str | None = Form(default=None),
|
||||
preview_token: str | None = Form(default=None),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
sources = ExitStack()
|
||||
try:
|
||||
archive_path, filename = sources.enter_context(_archive_source(file, staged_upload_id, preview_token, principal))
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
inspection = inspect_archive(
|
||||
archive_path,
|
||||
filename=filename,
|
||||
@@ -163,6 +218,9 @@ def preview_archive_upload(
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
normalized_path = normalize_folder(path)
|
||||
stage_id = staged_upload_id if isinstance(staged_upload_id, str) else None
|
||||
if retain_upload is True and stage_id is None:
|
||||
stage_id = stage_upload(settings, archive_path, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
preview_token = seal_transient_payload(
|
||||
{
|
||||
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||
@@ -174,13 +232,21 @@ def preview_archive_upload(
|
||||
"campaign_id": campaign_id or "",
|
||||
"archive_format": archive_format,
|
||||
"archive_sha256": digest,
|
||||
**({"staged_upload_id": stage_id, "filename": filename} if stage_id else {}),
|
||||
}
|
||||
)
|
||||
expires_at = datetime.now(UTC) + timedelta(
|
||||
seconds=settings.file_archive_preview_ttl_seconds
|
||||
)
|
||||
if isinstance(staged_upload_id, str):
|
||||
# Repreviewing (for example with a password) does not renew the
|
||||
# underlying private stage's lifetime.
|
||||
expires_at = min(expires_at, datetime.fromtimestamp(
|
||||
Path(archive_path).stat().st_mtime, UTC
|
||||
) + timedelta(seconds=settings.file_archive_preview_ttl_seconds))
|
||||
return ArchivePreviewResponse(
|
||||
preview_token=preview_token,
|
||||
staged_upload_id=stage_id,
|
||||
archive_format=inspection.archive_format,
|
||||
entries=[
|
||||
ArchiveEntryResponse(
|
||||
@@ -200,16 +266,17 @@ def preview_archive_upload(
|
||||
password_verified=inspection.password_verified,
|
||||
expires_at=expires_at.isoformat(),
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
except ArchiveWorkExpired as exc:
|
||||
raise HTTPException(status_code=410, detail=str(exc)) from exc
|
||||
except (FileStorageError, TransientPayloadError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
sources.close()
|
||||
|
||||
|
||||
@router.post("/archive-confirm", response_model=FileUploadResponse)
|
||||
def confirm_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
file: UploadFile | None = FastAPIFile(default=None),
|
||||
preview_token: str = Form(...),
|
||||
selected_paths_json: str = Form(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
@@ -225,13 +292,17 @@ def confirm_archive_upload(
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
staged_upload_id: str | None = Form(default=None),
|
||||
operation_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
sources = ExitStack()
|
||||
progress = None
|
||||
committed = False
|
||||
try:
|
||||
archive_path, filename = sources.enter_context(_archive_source(file, staged_upload_id, preview_token, principal))
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json)
|
||||
if conflict_resolutions_json
|
||||
@@ -251,11 +322,6 @@ def confirm_archive_upload(
|
||||
)
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
normalized_path = normalize_folder(path)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
_validate_archive_preview_token(
|
||||
preview_token,
|
||||
@@ -268,6 +334,7 @@ def confirm_archive_upload(
|
||||
archive_format=archive_format,
|
||||
archive_sha256=digest,
|
||||
)
|
||||
progress = ArchiveProgress(settings, operation_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
extracted = extract_archive_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
@@ -289,16 +356,20 @@ def confirm_archive_upload(
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||
progress=progress,
|
||||
)
|
||||
uploaded_assets = [item.asset for item in extracted]
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
progress("finalizing", len(extracted), len(extracted), progress.value["completed_bytes"], progress.value["total_bytes"])
|
||||
session.flush()
|
||||
response = FileUploadResponse(files=_asset_list_response(session, uploaded_assets, include_shares=True))
|
||||
session.commit()
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
committed = True
|
||||
progress.finish(True)
|
||||
return response
|
||||
except ArchiveWorkExpired as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(status_code=410, detail=str(exc)) from exc
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
@@ -311,9 +382,21 @@ def confirm_archive_upload(
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
if progress and not committed:
|
||||
progress.finish(False)
|
||||
try:
|
||||
sources.close()
|
||||
except OSError:
|
||||
pass
|
||||
if committed and isinstance(staged_upload_id, str):
|
||||
try:
|
||||
discard_staged_upload(settings, staged_upload_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
except (OSError, FileStorageError):
|
||||
pass
|
||||
|
||||
|
||||
@router.post("/upload", response_model=FileUploadResponse)
|
||||
@@ -402,6 +485,8 @@ def upload_files(
|
||||
)
|
||||
uploaded_assets.append(stored.asset)
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
session.flush()
|
||||
response = FileUploadResponse(files=_asset_list_response(session, uploaded_assets, include_shares=True))
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
@@ -409,12 +494,7 @@ def upload_files(
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||
@@ -469,6 +549,8 @@ def upload_zip(
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
_audit_connector_imports(session, principal, [item.asset for item in extracted])
|
||||
session.flush()
|
||||
response = FileUploadResponse(files=_asset_list_response(session, [item.asset for item in extracted], include_shares=True))
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
@@ -479,9 +561,4 @@ def upload_zip(
|
||||
finally:
|
||||
if zip_path:
|
||||
_cleanup_temp_file(zip_path)
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, item.asset, include_shares=True)
|
||||
for item in extracted
|
||||
]
|
||||
)
|
||||
return response
|
||||
|
||||
@@ -37,6 +37,7 @@ class FileConnectorSpaceCreateRequest(BaseModel):
|
||||
library_id: str | None = None
|
||||
remote_path: str = ""
|
||||
sync_mode: Literal["manual"] = "manual"
|
||||
read_only: bool = True
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
@@ -45,6 +46,7 @@ class FileConnectorSpaceUpdateRequest(BaseModel):
|
||||
library_id: str | None = None
|
||||
remote_path: str | None = None
|
||||
sync_mode: Literal["manual"] | None = None
|
||||
read_only: bool | None = None
|
||||
is_active: bool | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
@@ -192,6 +194,10 @@ class FileAssetResponse(BaseModel):
|
||||
created_at: str
|
||||
updated_at: str
|
||||
deleted_at: str | None = None
|
||||
retained_until: str | None = None
|
||||
legal_hold: bool = False
|
||||
lifecycle_revision: int = 1
|
||||
lifecycle_reason: str | None = None
|
||||
audit_relevant: bool = False
|
||||
metadata: dict[str, Any] | None = None
|
||||
source_provenance: FileSourceProvenance | None = None
|
||||
@@ -266,6 +272,7 @@ class ArchiveEntryResponse(BaseModel):
|
||||
|
||||
class ArchivePreviewResponse(BaseModel):
|
||||
preview_token: str
|
||||
staged_upload_id: str | None = None
|
||||
archive_format: str
|
||||
entries: list[ArchiveEntryResponse] = Field(default_factory=list)
|
||||
file_count: int
|
||||
@@ -477,6 +484,7 @@ class FileConnectorProviderResponse(BaseModel):
|
||||
installed: bool
|
||||
browse_supported: bool
|
||||
import_supported: bool
|
||||
write_supported: bool = False
|
||||
optional_dependency: str | None = None
|
||||
permission_model: str
|
||||
sync_strategy: str
|
||||
@@ -523,7 +531,7 @@ class FileConnectorImportRequest(BaseModel):
|
||||
target_folder: str | None = None
|
||||
target_path: str | None = None
|
||||
campaign_id: str | None = None
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = "reject"
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename", "skip"] = "reject"
|
||||
source_revision: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@@ -535,6 +543,77 @@ class FileConnectorSyncResponse(BaseModel):
|
||||
current_version_id: str
|
||||
|
||||
|
||||
class FileConnectorFolderSyncRequest(BaseModel):
|
||||
path: str = ""
|
||||
target_folder: str | None = None
|
||||
recursive: bool = True
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename", "skip"] = "skip"
|
||||
max_files: int = Field(default=100, ge=1, le=500)
|
||||
max_depth: int = Field(default=12, ge=0, le=50)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorFolderSyncItemResponse(BaseModel):
|
||||
source_path: str
|
||||
target_path: str | None = None
|
||||
action: Literal[
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"skipped",
|
||||
"conflict",
|
||||
"policy_denied",
|
||||
"failed",
|
||||
]
|
||||
file: FileAssetResponse | None = None
|
||||
previous_version_id: str | None = None
|
||||
current_version_id: str | None = None
|
||||
source_revision: str | None = None
|
||||
detail: str | None = None
|
||||
policy_decision: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorFolderSyncSummary(BaseModel):
|
||||
discovered: int = 0
|
||||
created: int = 0
|
||||
updated: int = 0
|
||||
unchanged: int = 0
|
||||
skipped: int = 0
|
||||
conflicts: int = 0
|
||||
policy_denied: int = 0
|
||||
failed: int = 0
|
||||
|
||||
|
||||
class FileConnectorFolderSyncResponse(BaseModel):
|
||||
connector_space_id: str
|
||||
connector_profile_id: str
|
||||
provider: str
|
||||
remote_path: str
|
||||
target_folder: str
|
||||
recursive: bool
|
||||
truncated: bool = False
|
||||
summary: FileConnectorFolderSyncSummary
|
||||
items: list[FileConnectorFolderSyncItemResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorWriteRequest(BaseModel):
|
||||
file_id: str
|
||||
remote_path: str
|
||||
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||
expected_revision: str | None = Field(default=None, max_length=500)
|
||||
|
||||
|
||||
class FileConnectorWriteResponse(BaseModel):
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool = False
|
||||
provider: str
|
||||
remote_path: str
|
||||
revision: str | None = None
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
|
||||
|
||||
class BulkDeleteRequest(BaseModel):
|
||||
file_ids: list[str]
|
||||
|
||||
@@ -543,6 +622,69 @@ class BulkDeleteResponse(BaseModel):
|
||||
deleted_count: int
|
||||
|
||||
|
||||
class FileLifecycleUpdateRequest(BaseModel):
|
||||
retained_until: datetime | None = None
|
||||
legal_hold: bool = False
|
||||
reason: str = Field(min_length=1, max_length=500)
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileRestoreResponse(BaseModel):
|
||||
restored_count: int
|
||||
|
||||
|
||||
class FileFolderRestoreResponse(BaseModel):
|
||||
restored_folders: int
|
||||
restored_files: int
|
||||
|
||||
|
||||
class FilePurgePreviewRequest(BaseModel):
|
||||
file_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class FilePurgePreviewItem(BaseModel):
|
||||
file_id: str
|
||||
filename: str
|
||||
lifecycle_revision: int
|
||||
deleted_at: str | None = None
|
||||
retained_until: str | None = None
|
||||
legal_hold: bool = False
|
||||
blockers: list[str] = Field(default_factory=list)
|
||||
blob_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FilePurgePreviewResponse(BaseModel):
|
||||
preview_sha256: str
|
||||
eligible: bool
|
||||
items: list[FilePurgePreviewItem]
|
||||
|
||||
|
||||
class FilePurgeExecuteRequest(FilePurgePreviewRequest):
|
||||
preview_sha256: str = Field(min_length=64, max_length=64)
|
||||
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||
confirmation: Literal["PURGE"]
|
||||
|
||||
|
||||
class FilePurgeResponse(BaseModel):
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool = False
|
||||
purged_files: int = 0
|
||||
released_blobs: int = 0
|
||||
|
||||
|
||||
class FileBlobGcRequest(BaseModel):
|
||||
limit: int = Field(default=25, ge=1, le=100)
|
||||
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||
|
||||
|
||||
class FileBlobGcResponse(BaseModel):
|
||||
inspected_blobs: int
|
||||
deleted_blobs: int
|
||||
unresolved_operation_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ConflictResolutionRequest(BaseModel):
|
||||
target_path: str
|
||||
action: Literal["overwrite", "rename", "skip"]
|
||||
|
||||
@@ -0,0 +1,418 @@
|
||||
"""Disposable archive parsers with bounded, acknowledged member staging.
|
||||
|
||||
This is a resource boundary for server-owned parsers, not a filesystem/network
|
||||
sandbox. Only the parent owns database state, credentials and durable storage.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager, closing
|
||||
from dataclasses import asdict, replace
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import stat
|
||||
import tempfile
|
||||
|
||||
from govoplan_core.security.bounded_process import (
|
||||
ProcessBudgetError,
|
||||
ProcessLimits,
|
||||
bounded_operation_admission,
|
||||
run_bounded_operation,
|
||||
)
|
||||
from govoplan_core.security.worker_payload import (
|
||||
WorkerPayloadError, decode_worker_payload, encode_worker_payload,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
INSPECTION_LIMITS = ProcessLimits(
|
||||
wall_seconds=120, cpu_seconds=90, memory_bytes=512 * 1024 * 1024,
|
||||
input_bytes=64 * 1024 * 1024, output_bytes=64 * 1024 * 1024,
|
||||
)
|
||||
EXTRACTION_LIMITS = ProcessLimits(
|
||||
wall_seconds=600, cpu_seconds=300, memory_bytes=512 * 1024 * 1024,
|
||||
input_bytes=64 * 1024 * 1024, output_bytes=64 * 1024,
|
||||
file_bytes=50 * 1024 * 1024,
|
||||
)
|
||||
_STATUS_BYTES = 16 * 1024
|
||||
_MAX_INTEGER = 2**63 - 1
|
||||
_MAX_WAKE_BYTES = 32 * 1024
|
||||
|
||||
|
||||
def _failure(message="Archive worker returned an invalid staging record"):
|
||||
return FileStorageError(message)
|
||||
|
||||
|
||||
def _read_regular(path: Path, maximum: int, *, atomic_record: bool = False) -> bytes:
|
||||
descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
try:
|
||||
info = os.fstat(descriptor)
|
||||
allowed_links = {0, 1} if atomic_record else {1}
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_nlink not in allowed_links or info.st_size > maximum:
|
||||
raise _failure()
|
||||
except BaseException:
|
||||
os.close(descriptor)
|
||||
raise
|
||||
with os.fdopen(descriptor, "rb") as source:
|
||||
# BufferedReader.read(n) can reserve n bytes even for a tiny file. Use
|
||||
# the validated actual size, never the potentially multi-GiB policy cap.
|
||||
value = source.read(info.st_size + 1)
|
||||
after = os.fstat(source.fileno())
|
||||
if (len(value) != info.st_size or after.st_nlink not in allowed_links
|
||||
or _fingerprint(info, atomic_record=atomic_record) != _fingerprint(after, atomic_record=atomic_record)):
|
||||
raise _failure()
|
||||
return value
|
||||
|
||||
|
||||
def _fingerprint(info, *, atomic_record=False):
|
||||
# Replacing a status/ack atomically may unlink an already-open old inode and
|
||||
# update its ctime; its content, size and mtime must nevertheless be stable.
|
||||
return (info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns,
|
||||
None if atomic_record else (info.st_nlink, info.st_ctime_ns))
|
||||
|
||||
|
||||
def _write_record(directory: Path, name: str, value: object) -> None:
|
||||
# name is a server constant: neither member names nor worker DTO paths.
|
||||
payload = json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode()
|
||||
if len(payload) > _STATUS_BYTES:
|
||||
raise _failure()
|
||||
temporary = directory / f".{name}.tmp"
|
||||
try:
|
||||
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as output:
|
||||
output.write(payload)
|
||||
os.replace(temporary, directory / name)
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
finally:
|
||||
try:
|
||||
temporary.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass # The owning private-directory context retries final cleanup.
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _ack_channel(directory: Path, *, create: bool = False):
|
||||
"""One fixed private FIFO; archive names never select a channel or file."""
|
||||
path = directory / "ack"
|
||||
descriptor = None
|
||||
try:
|
||||
try:
|
||||
if create:
|
||||
os.mkfifo(path, 0o600)
|
||||
flags = os.O_RDWR if create else os.O_RDONLY
|
||||
descriptor = os.open(path, flags | os.O_NONBLOCK | os.O_NOFOLLOW)
|
||||
info = os.fstat(descriptor)
|
||||
if not stat.S_ISFIFO(info.st_mode) or info.st_nlink != 1 or stat.S_IMODE(info.st_mode) != 0o600:
|
||||
raise _failure()
|
||||
if not create:
|
||||
# Validate before blocking: a substituted regular file or FIFO
|
||||
# must not hang the open. The parent holds a RDWR handle.
|
||||
os.set_blocking(descriptor, True)
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
yield descriptor
|
||||
finally:
|
||||
if descriptor is not None:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def _send_ack(descriptor: int, sequence: int) -> None:
|
||||
try:
|
||||
# Eight bytes are within PIPE_BUF: one nonblocking write is atomic.
|
||||
if os.write(descriptor, sequence.to_bytes(8, "big")) != 8:
|
||||
raise _failure()
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
|
||||
|
||||
def _receive_ack(descriptor: int, sequence: int) -> None:
|
||||
value = b""
|
||||
while len(value) < 8:
|
||||
chunk = os.read(descriptor, 8 - len(value))
|
||||
if not chunk:
|
||||
raise _failure()
|
||||
value += chunk
|
||||
if int.from_bytes(value, "big") != sequence:
|
||||
raise _failure()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _source_stage(archive_data):
|
||||
if os.name != "posix" or not hasattr(os, "O_NOFOLLOW"):
|
||||
raise FileStorageError("Required isolated-worker resource controls are unavailable")
|
||||
with tempfile.TemporaryDirectory(prefix="govoplan-archive-worker-") as temporary:
|
||||
directory = Path(temporary)
|
||||
os.chmod(directory, 0o700)
|
||||
source_path = directory / "source"
|
||||
descriptor = os.open(source_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb") as target:
|
||||
if isinstance(archive_data, bytes):
|
||||
target.write(archive_data)
|
||||
else:
|
||||
source_descriptor = os.open(os.fspath(archive_data), os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
with os.fdopen(source_descriptor, "rb") as source:
|
||||
before = os.fstat(source.fileno())
|
||||
if not stat.S_ISREG(before.st_mode):
|
||||
raise FileStorageError("Archive source must be a regular file")
|
||||
# Never follow a growing producer to EOF. Callers apply
|
||||
# the compressed request cap before supplying this path.
|
||||
remaining = before.st_size
|
||||
while remaining:
|
||||
chunk = source.read(min(remaining, 1024 * 1024))
|
||||
if not chunk:
|
||||
raise FileStorageError("Archive source changed during inspection")
|
||||
target.write(chunk)
|
||||
remaining -= len(chunk)
|
||||
extra = source.read(1)
|
||||
after = os.fstat(source.fileno())
|
||||
if extra or _fingerprint(before) != _fingerprint(after):
|
||||
raise FileStorageError("Archive source changed during inspection")
|
||||
except OSError as exc:
|
||||
raise FileStorageError("Archive source could not be staged safely") from exc
|
||||
yield directory, source_path
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _admitted_stage(archive_data):
|
||||
# Share Core capacity with other expensive module work before copying any
|
||||
# source bytes, not just after preparation has already consumed resources.
|
||||
try:
|
||||
with bounded_operation_admission() as admission, _source_stage(archive_data) as (directory, source):
|
||||
yield directory, source, admission
|
||||
except ProcessBudgetError as exc:
|
||||
raise FileStorageError(f"Archive processing failed ({exc.code}): {exc}") from exc
|
||||
|
||||
|
||||
def _run(operation, data, *, limits, tick=None, admission=None):
|
||||
try:
|
||||
payload = encode_worker_payload(data, max_bytes=limits.input_bytes)
|
||||
wire = run_bounded_operation(operation, payload, limits=limits, cancelled=tick, admission=admission)
|
||||
result = decode_worker_payload(wire, max_bytes=limits.output_bytes)
|
||||
except ProcessBudgetError as exc:
|
||||
raise FileStorageError(f"Archive processing failed ({exc.code}): {exc}") from exc
|
||||
except WorkerPayloadError as exc:
|
||||
raise FileStorageError("Archive worker data exceeded safe transport limits") from exc
|
||||
if type(result) is not dict:
|
||||
raise _failure()
|
||||
if "error" in result:
|
||||
from govoplan_files.backend.storage.archives import ArchivePasswordError
|
||||
if type(result.get("error")) is not str or type(result.get("password_error")) is not bool:
|
||||
raise _failure()
|
||||
error_type = ArchivePasswordError if result["password_error"] else FileStorageError
|
||||
raise error_type(result["error"])
|
||||
return result
|
||||
|
||||
|
||||
def inspect_archive_isolated(archive_data, **options):
|
||||
from govoplan_files.backend.storage.archives import ArchiveEntry, ArchiveInspection, archive_format_for_filename
|
||||
archive_format_for_filename(options["filename"])
|
||||
with _admitted_stage(archive_data) as (_directory, source, admission):
|
||||
result = _run(_inspect_worker, {"source": str(source), "options": options},
|
||||
limits=INSPECTION_LIMITS, admission=admission)
|
||||
inspection = result.get("inspection")
|
||||
if type(inspection) is not dict or type(inspection.get("entries")) is not tuple:
|
||||
raise _failure()
|
||||
if len(inspection["entries"]) > options["max_entries"]:
|
||||
raise _failure()
|
||||
try:
|
||||
inspection["entries"] = tuple(ArchiveEntry(**entry) for entry in inspection["entries"])
|
||||
return ArchiveInspection(**inspection)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise _failure() from exc
|
||||
|
||||
|
||||
def _inspect_worker(payload: bytes) -> bytes:
|
||||
from govoplan_files.backend.storage.archives import ArchivePasswordError, _inspect_archive_content
|
||||
data = decode_worker_payload(payload, max_bytes=INSPECTION_LIMITS.input_bytes)
|
||||
try:
|
||||
inspection = _inspect_archive_content(data["source"], **data["options"])
|
||||
result = {"inspection": asdict(inspection)}
|
||||
except FileStorageError as exc:
|
||||
result = {"error": str(exc), "password_error": isinstance(exc, ArchivePasswordError)}
|
||||
return encode_worker_payload(result, max_bytes=INSPECTION_LIMITS.output_bytes)
|
||||
|
||||
|
||||
def extract_archive_isolated(session, *, archive_data, filename, password, selected_paths,
|
||||
max_entries, max_file_bytes, max_expanded_bytes,
|
||||
max_expansion_ratio, progress, store_options):
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
_safe_member_path, _store_archive_members, archive_format_for_filename,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import archive_storage_backend_scope
|
||||
archive_format_for_filename(filename)
|
||||
if type(max_file_bytes) is not int or not 0 < max_file_bytes <= 2 * 1024 * 1024 * 1024:
|
||||
raise FileStorageError("Isolated archive members require a positive limit of at most 2 GiB")
|
||||
limits = replace(
|
||||
EXTRACTION_LIMITS,
|
||||
memory_bytes=max(EXTRACTION_LIMITS.memory_bytes, 3 * max_file_bytes + 128 * 1024 * 1024),
|
||||
file_bytes=max(max_file_bytes, _STATUS_BYTES),
|
||||
)
|
||||
uploaded = []
|
||||
stored_bytes = 0
|
||||
sequence = 0
|
||||
seen_paths = set()
|
||||
last_extracted_bytes = 0
|
||||
last_extracted_files = 0
|
||||
expected_totals = None
|
||||
selected = None if selected_paths is None else {_safe_member_path(path) for path in selected_paths}
|
||||
if progress:
|
||||
progress("inspecting", 0, 0, 0, 0)
|
||||
with (
|
||||
_admitted_stage(archive_data) as (directory, source, admission),
|
||||
_ack_channel(directory, create=True) as acknowledgement,
|
||||
archive_storage_backend_scope(),
|
||||
):
|
||||
actual_total_limit = min(max_expanded_bytes, source.stat().st_size * max_expansion_ratio)
|
||||
|
||||
def poll():
|
||||
nonlocal stored_bytes, sequence, last_extracted_bytes, last_extracted_files, expected_totals
|
||||
try:
|
||||
record = json.loads(_read_regular(directory / "status", _STATUS_BYTES, atomic_record=True))
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
except (OSError, ValueError, RecursionError) as exc:
|
||||
raise _failure() from exc
|
||||
if type(record) is not dict or type(record.get("sequence")) is not int:
|
||||
raise _failure()
|
||||
next_sequence = record["sequence"]
|
||||
if not 0 < next_sequence <= _MAX_INTEGER or next_sequence < sequence:
|
||||
raise _failure()
|
||||
if next_sequence == sequence:
|
||||
return False
|
||||
kind = record.get("kind")
|
||||
counters = record.get("progress")
|
||||
if kind not in {"progress", "member"} or type(counters) is not list or len(counters) != 5:
|
||||
raise _failure()
|
||||
keys = {"sequence", "kind", "progress"}
|
||||
if kind == "member":
|
||||
keys |= {"path", "size", "sha256"}
|
||||
if set(record) != keys:
|
||||
raise _failure()
|
||||
phase, completed, total, completed_bytes, total_bytes = counters
|
||||
if phase != "extracting" or any(type(value) is not int for value in counters[1:]):
|
||||
raise _failure()
|
||||
if not (last_extracted_files <= completed <= total <= max_entries
|
||||
and 0 <= total_bytes <= actual_total_limit
|
||||
and last_extracted_bytes <= completed_bytes <= total_bytes):
|
||||
raise _failure()
|
||||
if expected_totals is not None and expected_totals != (total, total_bytes):
|
||||
raise _failure()
|
||||
if completed not in {len(uploaded), len(uploaded) + 1}:
|
||||
raise _failure()
|
||||
sequence = next_sequence
|
||||
last_extracted_bytes = completed_bytes
|
||||
last_extracted_files = completed
|
||||
expected_totals = total, total_bytes
|
||||
if progress:
|
||||
progress(*counters)
|
||||
if kind == "progress":
|
||||
return False
|
||||
if completed != len(uploaded) + 1 or type(record.get("path")) is not str:
|
||||
raise _failure()
|
||||
inner_path = _safe_member_path(record["path"])
|
||||
if inner_path != record["path"] or inner_path in seen_paths:
|
||||
raise _failure()
|
||||
if selected is not None and not any(inner_path == path or inner_path.startswith(path + "/") for path in selected):
|
||||
raise _failure()
|
||||
size = record.get("size")
|
||||
if type(size) is not int or not 0 <= size <= max_file_bytes or stored_bytes + size != completed_bytes:
|
||||
raise _failure()
|
||||
member_path = directory / f"member-{completed:08d}"
|
||||
try:
|
||||
data = _read_regular(member_path, max_file_bytes)
|
||||
except OSError as exc:
|
||||
raise _failure() from exc
|
||||
if len(data) != size or hashlib.sha256(data).hexdigest() != record.get("sha256"):
|
||||
raise _failure()
|
||||
# The established persistence API accepts bytes, not a stream/path.
|
||||
# Keep one member resident, with the existing transaction and quotas.
|
||||
uploaded.extend(_store_archive_members(session, members=((inner_path, data),), **store_options))
|
||||
stored_bytes += len(data)
|
||||
seen_paths.add(inner_path)
|
||||
del data
|
||||
try:
|
||||
member_path.unlink()
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
if progress:
|
||||
progress("storing", len(uploaded), total, stored_bytes, total_bytes)
|
||||
_send_ack(acknowledgement, sequence)
|
||||
return False
|
||||
|
||||
result = _run(_extract_worker, {
|
||||
"source": str(source), "directory": str(directory),
|
||||
"options": {"filename": filename, "password": password, "max_entries": max_entries,
|
||||
"max_expanded_bytes": max_expanded_bytes, "max_expansion_ratio": max_expansion_ratio},
|
||||
"selected_paths": None if selected is None else tuple(selected),
|
||||
"max_file_bytes": max_file_bytes,
|
||||
}, limits=limits, tick=poll, admission=admission)
|
||||
if type(result.get("completed")) is not int or result["completed"] != len(uploaded):
|
||||
raise _failure()
|
||||
return uploaded
|
||||
|
||||
|
||||
def _extract_worker(payload: bytes) -> bytes:
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
ArchivePasswordError, _inspect_archive_content, _read_selected_tar_members,
|
||||
_read_selected_zip_members, _selected_file_paths,
|
||||
)
|
||||
data = decode_worker_payload(payload, max_bytes=EXTRACTION_LIMITS.input_bytes)
|
||||
directory = Path(data["directory"])
|
||||
sequence = 0
|
||||
completed = 0
|
||||
|
||||
def report(phase, files, total, count, total_bytes, **member):
|
||||
nonlocal sequence
|
||||
sequence += 1
|
||||
_write_record(directory, "status", {
|
||||
"sequence": sequence, "kind": "member" if member else "progress",
|
||||
"progress": [phase, files, total, count, total_bytes], **member,
|
||||
})
|
||||
|
||||
try:
|
||||
inspection = _inspect_archive_content(data["source"], **data["options"])
|
||||
if inspection.requires_password and not inspection.password_verified:
|
||||
raise ArchivePasswordError("Archive password is required")
|
||||
selected = _selected_file_paths(inspection.entries, data["selected_paths"])
|
||||
if not selected:
|
||||
raise FileStorageError("Select at least one archive file to import")
|
||||
total_bytes = sum(entry.size_bytes for entry in inspection.entries if entry.path in selected)
|
||||
total_limit = min(data["options"]["max_expanded_bytes"],
|
||||
inspection.compressed_size_bytes * data["options"]["max_expansion_ratio"])
|
||||
arguments = {"selected_files": selected, "max_file_bytes": data["max_file_bytes"],
|
||||
"max_total_bytes": total_limit, "progress": report, "total_bytes": total_bytes}
|
||||
if inspection.archive_format == "zip":
|
||||
members = _read_selected_zip_members(data["source"], password=data["options"]["password"], **arguments)
|
||||
else:
|
||||
members = _read_selected_tar_members(data["source"], **arguments)
|
||||
actual = 0
|
||||
with closing(members), _ack_channel(directory) as acknowledgement:
|
||||
for inner_path, content in members:
|
||||
completed += 1
|
||||
actual += len(content)
|
||||
member_path = directory / f"member-{completed:08d}"
|
||||
descriptor = os.open(member_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as output:
|
||||
output.write(content)
|
||||
report("extracting", completed, len(selected), actual, total_bytes,
|
||||
path=inner_path, size=len(content), sha256=hashlib.sha256(content).hexdigest())
|
||||
del content
|
||||
# Wake Core's selector after publishing a completed member. One
|
||||
# discarded stderr byte per member fits its 64 KiB ceiling for
|
||||
# the default 10,000-entry cap; intermediate progress never
|
||||
# wakes. Non-default >32,768-member workloads retain ordinary
|
||||
# Core progress polling after the fixed wake budget is spent.
|
||||
if completed <= _MAX_WAKE_BYTES:
|
||||
os.write(2, b".")
|
||||
# Acknowledgement applies backpressure: the parent stores and
|
||||
# removes this member before we allocate/decode the next one.
|
||||
# Blocking on a bounded FIFO read avoids a fixed delay for each
|
||||
# tiny file. Core still enforces the overall wall/CPU budgets.
|
||||
_receive_ack(acknowledgement, sequence)
|
||||
result = {"completed": completed}
|
||||
except FileStorageError as exc:
|
||||
result = {"error": str(exc), "password_error": isinstance(exc, ArchivePasswordError)}
|
||||
return encode_worker_payload(result, max_bytes=EXTRACTION_LIMITS.output_bytes)
|
||||
@@ -9,7 +9,7 @@ from dataclasses import dataclass
|
||||
from io import BytesIO
|
||||
from os import PathLike
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any, BinaryIO, Iterable, Literal
|
||||
from typing import Any, BinaryIO, Callable, Iterable, Iterator, Literal
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -33,11 +33,17 @@ from govoplan_files.backend.storage.paths import (
|
||||
normalize_folder,
|
||||
normalize_logical_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.native_zip import (
|
||||
native_zip_library,
|
||||
read_native_zip_members,
|
||||
)
|
||||
|
||||
|
||||
_ARCHIVE_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:")
|
||||
ARCHIVE_UPLOAD_MAX_ENTRIES = 10_000
|
||||
ARCHIVE_MAX_PATH_BYTES = 4096
|
||||
ARCHIVE_MAX_PATH_DEPTH = 128
|
||||
# Kept for callers and documentation using the former ZIP-specific name.
|
||||
ZIP_UPLOAD_MAX_FILES = ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||
SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||
@@ -51,6 +57,8 @@ SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||
".zip",
|
||||
)
|
||||
|
||||
ArchiveProgress = Callable[[str, int, int, int, int], None]
|
||||
|
||||
|
||||
class ArchivePasswordError(FileStorageError):
|
||||
pass
|
||||
@@ -132,6 +140,22 @@ def inspect_archive(
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> ArchiveInspection:
|
||||
from govoplan_files.backend.storage.archive_workers import inspect_archive_isolated
|
||||
return inspect_archive_isolated(
|
||||
archive_data, filename=filename, password=password, max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes, max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
|
||||
|
||||
def _inspect_archive_content(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
filename: str,
|
||||
password: str | None = None,
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> ArchiveInspection:
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
compressed_size = _archive_size(archive_data)
|
||||
@@ -141,7 +165,13 @@ def inspect_archive(
|
||||
password=password,
|
||||
)
|
||||
else:
|
||||
entries = _inspect_tar(archive_data)
|
||||
entries = _inspect_tar(
|
||||
archive_data,
|
||||
compressed_size=compressed_size,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
requires_password = False
|
||||
password_verified = True
|
||||
_validate_archive_limits(
|
||||
@@ -151,14 +181,12 @@ def inspect_archive(
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
complete_entries = _with_derived_directories(entries)
|
||||
complete_entries = _with_derived_directories(entries, max_entries=max_entries)
|
||||
return ArchiveInspection(
|
||||
archive_format=archive_format,
|
||||
entries=tuple(complete_entries),
|
||||
file_count=sum(entry.kind == "file" for entry in complete_entries),
|
||||
directory_count=sum(
|
||||
entry.kind == "directory" for entry in complete_entries
|
||||
),
|
||||
directory_count=sum(entry.kind == "directory" for entry in complete_entries),
|
||||
expanded_size_bytes=sum(
|
||||
entry.size_bytes for entry in entries if entry.kind == "file"
|
||||
),
|
||||
@@ -190,53 +218,19 @@ def extract_archive_upload(
|
||||
max_file_bytes: int = 50 * 1024 * 1024,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
progress: ArchiveProgress | None = None,
|
||||
) -> list[UploadedStoredFile]:
|
||||
inspection = inspect_archive(
|
||||
archive_data,
|
||||
filename=filename,
|
||||
password=password,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
if inspection.requires_password and not inspection.password_verified:
|
||||
raise ArchivePasswordError("Archive password is required")
|
||||
selected_files = _selected_file_paths(inspection.entries, selected_paths)
|
||||
if not selected_files:
|
||||
raise FileStorageError("Select at least one archive file to import")
|
||||
actual_total_limit = min(
|
||||
max_expanded_bytes,
|
||||
inspection.compressed_size_bytes * max_expansion_ratio,
|
||||
)
|
||||
if inspection.archive_format == "zip":
|
||||
members = _read_selected_zip_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
password=password,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
else:
|
||||
members = _read_selected_tar_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
return _store_archive_members(
|
||||
session,
|
||||
members=members,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
folder=folder,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
from govoplan_files.backend.storage.archive_workers import extract_archive_isolated
|
||||
return extract_archive_isolated(
|
||||
session, archive_data=archive_data, filename=filename, password=password,
|
||||
selected_paths=selected_paths, max_entries=max_entries, max_file_bytes=max_file_bytes,
|
||||
max_expanded_bytes=max_expanded_bytes, max_expansion_ratio=max_expansion_ratio,
|
||||
progress=progress, store_options={
|
||||
"tenant_id": tenant_id, "owner_type": owner_type, "owner_id": owner_id,
|
||||
"user_id": user_id, "folder": folder, "campaign_id": campaign_id,
|
||||
"conflict_strategy": conflict_strategy, "conflict_resolutions": conflict_resolutions,
|
||||
"metadata": metadata, "is_admin": is_admin, "encryption_vault_id": encryption_vault_id,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -319,11 +313,23 @@ def _inspect_zip(
|
||||
|
||||
def _inspect_tar(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
compressed_size: int,
|
||||
max_entries: int,
|
||||
max_expanded_bytes: int,
|
||||
max_expansion_ratio: int,
|
||||
) -> list[ArchiveEntry]:
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
entries: list[ArchiveEntry] = []
|
||||
for member in archive.getmembers():
|
||||
expanded_size = 0
|
||||
for member in archive:
|
||||
# Check each header before advancing across its payload. In a
|
||||
# compressed TAR getmembers() would inflate everything first.
|
||||
if len(entries) >= max_entries:
|
||||
raise FileStorageError(
|
||||
f"Archive contains too many entries (limit {max_entries})"
|
||||
)
|
||||
path = _safe_member_path(member.name)
|
||||
if member.isdir():
|
||||
entries.append(
|
||||
@@ -334,6 +340,21 @@ def _inspect_tar(
|
||||
raise FileStorageError(
|
||||
f"Archive member {member.name!r} is not a regular file or directory"
|
||||
)
|
||||
if member.size < 0:
|
||||
raise FileStorageError("Archive member has invalid size metadata")
|
||||
expanded_size += member.size
|
||||
if expanded_size > max_expanded_bytes:
|
||||
raise FileStorageError(
|
||||
"Archive is too large after extraction "
|
||||
f"(limit {max_expanded_bytes} bytes)"
|
||||
)
|
||||
if expanded_size and (
|
||||
compressed_size <= 0
|
||||
or expanded_size > compressed_size * max_expansion_ratio
|
||||
):
|
||||
raise FileStorageError(
|
||||
f"Archive expansion ratio exceeds {max_expansion_ratio}:1"
|
||||
)
|
||||
entries.append(
|
||||
ArchiveEntry(
|
||||
path=path,
|
||||
@@ -352,21 +373,19 @@ def _zip_entry(info: zipfile.ZipInfo) -> ArchiveEntry:
|
||||
path = _safe_member_path(info.filename)
|
||||
unix_mode = (info.external_attr >> 16) & 0xFFFF
|
||||
file_type = stat.S_IFMT(unix_mode)
|
||||
if file_type and not (
|
||||
stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)
|
||||
):
|
||||
if file_type and not (stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)):
|
||||
raise FileStorageError(
|
||||
f"Archive member {info.filename!r} is not a regular file or directory"
|
||||
)
|
||||
if info.file_size < 0 or info.compress_size < 0:
|
||||
raise FileStorageError(f"Archive member {info.filename!r} has invalid size metadata")
|
||||
raise FileStorageError(
|
||||
f"Archive member {info.filename!r} has invalid size metadata"
|
||||
)
|
||||
return ArchiveEntry(
|
||||
path=path,
|
||||
kind="directory" if info.is_dir() else "file",
|
||||
size_bytes=0 if info.is_dir() else int(info.file_size),
|
||||
compressed_size_bytes=(
|
||||
None if info.is_dir() else int(info.compress_size)
|
||||
),
|
||||
compressed_size_bytes=(None if info.is_dir() else int(info.compress_size)),
|
||||
encrypted=bool(info.flag_bits & 0x1),
|
||||
)
|
||||
|
||||
@@ -388,9 +407,7 @@ def _validate_archive_limits(
|
||||
for entry in entries:
|
||||
previous_kind = seen.get(entry.path)
|
||||
if previous_kind is not None:
|
||||
raise FileStorageError(
|
||||
f"Archive contains duplicate path {entry.path!r}"
|
||||
)
|
||||
raise FileStorageError(f"Archive contains duplicate path {entry.path!r}")
|
||||
seen[entry.path] = entry.kind
|
||||
if entry.kind == "file":
|
||||
expanded_size += entry.size_bytes
|
||||
@@ -400,17 +417,17 @@ def _validate_archive_limits(
|
||||
f"(limit {max_expanded_bytes} bytes)"
|
||||
)
|
||||
if expanded_size and (
|
||||
compressed_size <= 0
|
||||
or expanded_size > compressed_size * max_expansion_ratio
|
||||
compressed_size <= 0 or expanded_size > compressed_size * max_expansion_ratio
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Archive expansion ratio exceeds "
|
||||
f"{max_expansion_ratio}:1"
|
||||
f"Archive expansion ratio exceeds {max_expansion_ratio}:1"
|
||||
)
|
||||
|
||||
|
||||
def _with_derived_directories(
|
||||
entries: list[ArchiveEntry],
|
||||
*,
|
||||
max_entries: int,
|
||||
) -> list[ArchiveEntry]:
|
||||
by_path = {entry.path: entry for entry in entries}
|
||||
for entry in entries:
|
||||
@@ -422,10 +439,13 @@ def _with_derived_directories(
|
||||
raise FileStorageError(
|
||||
f"Archive path {path!r} is both a file and a directory"
|
||||
)
|
||||
by_path.setdefault(
|
||||
path,
|
||||
ArchiveEntry(path=path, kind="directory", size_bytes=0),
|
||||
)
|
||||
if path not in by_path:
|
||||
if len(by_path) >= max_entries:
|
||||
raise FileStorageError(
|
||||
"Archive contains too many entries including parent "
|
||||
f"directories (limit {max_entries})"
|
||||
)
|
||||
by_path[path] = ArchiveEntry(path=path, kind="directory", size_bytes=0)
|
||||
parent = parent.parent
|
||||
return sorted(
|
||||
by_path.values(),
|
||||
@@ -458,9 +478,7 @@ def _selected_file_paths(
|
||||
continue
|
||||
prefix = f"{path}/"
|
||||
selected_files.update(
|
||||
file_path
|
||||
for file_path in file_paths
|
||||
if file_path.startswith(prefix)
|
||||
file_path for file_path in file_paths if file_path.startswith(prefix)
|
||||
)
|
||||
return selected_files
|
||||
|
||||
@@ -472,12 +490,45 @@ def _read_selected_zip_members(
|
||||
password: str | None,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
progress: ArchiveProgress | None = None,
|
||||
total_bytes: int = 0,
|
||||
) -> Iterator[tuple[str, bytes]]:
|
||||
total = 0
|
||||
completed = 0
|
||||
try:
|
||||
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||
for info in archive.infolist():
|
||||
infos = archive.infolist()
|
||||
# Accelerate only classic ZIPCrypto using established public C
|
||||
# decoding APIs; AES and other formats retain their existing path.
|
||||
native_compatible = (
|
||||
bool(password)
|
||||
and "\x00" not in password
|
||||
and any(info.flag_bits & 1 for info in infos)
|
||||
and all(
|
||||
not getattr(info, "wz_aes_version", None)
|
||||
and info.compress_type in {zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED}
|
||||
and (info.filename.isascii() or bool(info.flag_bits & 0x800))
|
||||
for info in infos
|
||||
)
|
||||
)
|
||||
native = native_zip_library() if native_compatible else None
|
||||
if native is not None:
|
||||
yield from read_native_zip_members(
|
||||
native,
|
||||
archive_data,
|
||||
infos_by_path={
|
||||
_safe_member_path(info.filename): info for info in infos
|
||||
},
|
||||
selected_files=selected_files,
|
||||
normalize_name=_safe_member_path,
|
||||
password=password,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
progress=progress,
|
||||
total_bytes=total_bytes,
|
||||
)
|
||||
return
|
||||
for info in infos:
|
||||
if info.is_dir():
|
||||
continue
|
||||
path = _safe_member_path(info.filename)
|
||||
@@ -492,6 +543,17 @@ def _read_selected_zip_members(
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
on_bytes=(
|
||||
lambda count: progress(
|
||||
"extracting",
|
||||
completed,
|
||||
len(selected_files),
|
||||
count,
|
||||
total_bytes,
|
||||
)
|
||||
)
|
||||
if progress
|
||||
else None,
|
||||
)
|
||||
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||
if info.flag_bits & 0x1:
|
||||
@@ -499,12 +561,17 @@ def _read_selected_zip_members(
|
||||
"Archive password is incorrect"
|
||||
) from exc
|
||||
raise
|
||||
result.append((path, data))
|
||||
completed += 1
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
yield path, data
|
||||
del data
|
||||
except (FileStorageError, ArchivePasswordError):
|
||||
raise
|
||||
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise FileStorageError("ZIP extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_selected_tar_members(
|
||||
@@ -513,12 +580,16 @@ def _read_selected_tar_members(
|
||||
selected_files: set[str],
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
progress: ArchiveProgress | None = None,
|
||||
total_bytes: int = 0,
|
||||
) -> Iterator[tuple[str, bytes]]:
|
||||
total = 0
|
||||
completed = 0
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
for member in archive.getmembers():
|
||||
# Iterate as headers are read instead of inflating the entire TAR
|
||||
# with getmembers(), then seeking backwards to inflate it again.
|
||||
for member in archive:
|
||||
if not member.isfile():
|
||||
continue
|
||||
path = _safe_member_path(member.name)
|
||||
@@ -526,9 +597,7 @@ def _read_selected_tar_members(
|
||||
continue
|
||||
source = archive.extractfile(member)
|
||||
if source is None:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} could not be read"
|
||||
)
|
||||
raise FileStorageError(f"Archive member {path!r} could not be read")
|
||||
with source:
|
||||
data, total = _read_member(
|
||||
source,
|
||||
@@ -536,13 +605,29 @@ def _read_selected_tar_members(
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
on_bytes=(
|
||||
lambda count: progress(
|
||||
"extracting",
|
||||
completed,
|
||||
len(selected_files),
|
||||
count,
|
||||
total_bytes,
|
||||
)
|
||||
)
|
||||
if progress
|
||||
else None,
|
||||
)
|
||||
result.append((path, data))
|
||||
completed += 1
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
yield path, data
|
||||
del data
|
||||
except FileStorageError:
|
||||
raise
|
||||
except (OSError, tarfile.TarError) as exc:
|
||||
raise FileStorageError("TAR extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_member(
|
||||
@@ -552,6 +637,7 @@ def _read_member(
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
current_total: int,
|
||||
on_bytes: Callable[[int], None] | None = None,
|
||||
) -> tuple[bytes, int]:
|
||||
parts: list[bytes] = []
|
||||
actual_size = 0
|
||||
@@ -565,12 +651,12 @@ def _read_member(
|
||||
break
|
||||
actual_size += len(chunk)
|
||||
if actual_size > max_file_bytes:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} exceeds per-file limit"
|
||||
)
|
||||
raise FileStorageError(f"Archive member {path!r} exceeds per-file limit")
|
||||
if current_total + actual_size > max_total_bytes:
|
||||
raise FileStorageError("Archive is too large after extraction")
|
||||
parts.append(chunk)
|
||||
if on_bytes:
|
||||
on_bytes(current_total + actual_size)
|
||||
return b"".join(parts), current_total + actual_size
|
||||
|
||||
|
||||
@@ -589,13 +675,15 @@ def _store_archive_members(
|
||||
metadata: dict[str, Any] | None,
|
||||
is_admin: bool,
|
||||
encryption_vault_id: str | None,
|
||||
progress: ArchiveProgress | None = None,
|
||||
total_files: int = 0,
|
||||
total_bytes: int = 0,
|
||||
) -> list[UploadedStoredFile]:
|
||||
uploaded: list[UploadedStoredFile] = []
|
||||
base_folder = normalize_folder(folder)
|
||||
stored_bytes = 0
|
||||
for inner_path, data in members:
|
||||
target_path = (
|
||||
f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
)
|
||||
target_path = f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
uploaded.append(
|
||||
create_file_asset(
|
||||
session,
|
||||
@@ -616,24 +704,39 @@ def _store_archive_members(
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
)
|
||||
stored_bytes += len(data)
|
||||
if progress:
|
||||
progress("storing", len(uploaded), total_files, stored_bytes, total_bytes)
|
||||
del data
|
||||
return uploaded
|
||||
|
||||
|
||||
def _safe_member_path(value: str) -> str:
|
||||
raw = str(value or "").replace("\\", "/").strip()
|
||||
if (
|
||||
not raw
|
||||
or "\x00" in raw
|
||||
or raw.startswith("/")
|
||||
or _WINDOWS_DRIVE_RE.match(raw)
|
||||
):
|
||||
try:
|
||||
oversized = (
|
||||
len(raw) > ARCHIVE_MAX_PATH_BYTES
|
||||
or len(raw.encode("utf-8")) > ARCHIVE_MAX_PATH_BYTES
|
||||
)
|
||||
except UnicodeEncodeError as exc:
|
||||
raise FileStorageError("Archive member path is not valid Unicode") from exc
|
||||
if oversized:
|
||||
raise FileStorageError(
|
||||
f"Archive member path exceeds {ARCHIVE_MAX_PATH_BYTES} UTF-8 bytes"
|
||||
)
|
||||
if not raw or "\x00" in raw or raw.startswith("/") or _WINDOWS_DRIVE_RE.match(raw):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
if any(part == ".." for part in raw.split("/")):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
try:
|
||||
return normalize_logical_path(raw.rstrip("/"))
|
||||
normalized = normalize_logical_path(raw.rstrip("/"))
|
||||
except ValueError as exc:
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}") from exc
|
||||
if normalized.count("/") + 1 > ARCHIVE_MAX_PATH_DEPTH:
|
||||
raise FileStorageError(
|
||||
f"Archive member path exceeds {ARCHIVE_MAX_PATH_DEPTH} components"
|
||||
)
|
||||
return normalized
|
||||
|
||||
|
||||
def _archive_size(archive_data: bytes | str | PathLike[str]) -> int:
|
||||
|
||||
@@ -10,6 +10,10 @@ class FileStorageError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class FileSourceConflict(FileStorageError):
|
||||
"""A source cannot be selected or revision-bound without ambiguity."""
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class UploadedStoredFile:
|
||||
asset: FileAsset
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseItem,
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorFolderSkip:
|
||||
path: str
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ConnectorFolderDiscovery:
|
||||
files: list[ConnectorBrowseItem] = field(default_factory=list)
|
||||
skipped: list[ConnectorFolderSkip] = field(default_factory=list)
|
||||
truncated: bool = False
|
||||
|
||||
|
||||
def discover_connector_folder(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
path: str,
|
||||
library_id: str | None,
|
||||
recursive: bool,
|
||||
max_files: int,
|
||||
max_depth: int,
|
||||
) -> ConnectorFolderDiscovery:
|
||||
"""Discover a bounded, deterministic set of files below one connector folder."""
|
||||
|
||||
root_path = normalize_connector_browse_path(path)
|
||||
queue: deque[tuple[str, int]] = deque([(root_path, 0)])
|
||||
visited_folders: set[str] = set()
|
||||
seen_files: set[tuple[str, str]] = set()
|
||||
result = ConnectorFolderDiscovery()
|
||||
|
||||
while queue:
|
||||
folder_path, depth = queue.popleft()
|
||||
if folder_path in visited_folders:
|
||||
continue
|
||||
visited_folders.add(folder_path)
|
||||
|
||||
continuation_token: str | None = None
|
||||
seen_tokens: set[str] = set()
|
||||
while True:
|
||||
items = browse_connector_profile(
|
||||
profile,
|
||||
path=folder_path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
for item in items:
|
||||
item_path = normalize_connector_browse_path(item.path)
|
||||
if item.kind == "file":
|
||||
identity = (str(item.external_id or ""), item_path)
|
||||
if identity in seen_files:
|
||||
continue
|
||||
if len(result.files) >= max_files:
|
||||
result.truncated = True
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path or folder_path,
|
||||
reason=f"The manual sync limit of {max_files} files was reached.",
|
||||
)
|
||||
)
|
||||
return result
|
||||
seen_files.add(identity)
|
||||
result.files.append(item)
|
||||
continue
|
||||
|
||||
if item.kind == "folder":
|
||||
if not recursive:
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason="Subfolder skipped because recursive sync is disabled.",
|
||||
)
|
||||
)
|
||||
elif depth >= max_depth:
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason=f"Subfolder skipped at the configured depth limit of {max_depth}.",
|
||||
)
|
||||
)
|
||||
else:
|
||||
queue.append((item_path, depth + 1))
|
||||
continue
|
||||
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason=f"Connector item kind {item.kind!r} cannot be synchronized as a file.",
|
||||
)
|
||||
)
|
||||
|
||||
next_token = _next_continuation_token(items)
|
||||
if not next_token or next_token in seen_tokens:
|
||||
break
|
||||
seen_tokens.add(next_token)
|
||||
continuation_token = next_token
|
||||
|
||||
result.files.sort(key=lambda item: normalize_connector_browse_path(item.path).casefold())
|
||||
result.skipped.sort(key=lambda item: (item.path.casefold(), item.reason))
|
||||
return result
|
||||
|
||||
|
||||
def connector_relative_path(*, space_root: str, item_path: str) -> str:
|
||||
root = normalize_connector_browse_path(space_root)
|
||||
item = normalize_connector_browse_path(item_path)
|
||||
if not root:
|
||||
return item
|
||||
if item == root:
|
||||
return item.rsplit("/", 1)[-1]
|
||||
prefix = f"{root}/"
|
||||
if not item.startswith(prefix):
|
||||
raise ValueError("Connector returned a file outside the linked connector-space root")
|
||||
return item[len(prefix) :]
|
||||
|
||||
|
||||
def join_connector_path(*parts: str | None) -> str:
|
||||
return normalize_connector_browse_path(
|
||||
"/".join(str(part or "").strip("/\\") for part in parts if str(part or "").strip("/\\"))
|
||||
)
|
||||
|
||||
|
||||
def _next_continuation_token(items: list[ConnectorBrowseItem]) -> str | None:
|
||||
for item in reversed(items):
|
||||
value = item.metadata.get("next_continuation_token")
|
||||
if value is not None and str(value).strip():
|
||||
return str(value).strip()
|
||||
return None
|
||||
@@ -32,6 +32,7 @@ from govoplan_files.backend.storage.connector_browse import (
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.paths import filename_from_path
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
|
||||
|
||||
class ConnectorImportError(RuntimeError):
|
||||
@@ -42,6 +43,10 @@ class ConnectorImportUnsupported(ConnectorImportError):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectorRevisionConflict(ConnectorImportError, FileSourceConflict):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorDownloadedFile:
|
||||
filename: str
|
||||
@@ -120,6 +125,14 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||
detail = detail if isinstance(detail, dict) else {}
|
||||
try:
|
||||
after = _request_json("GET", _seafile_url(profile, f"api2/repos/{repo_id}/file/detail/"), headers=headers, params={"p": file_path})
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
before_revision = _clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified"))
|
||||
after_revision = _clean(after.get("id") or after.get("mtime") or after.get("last_modified")) if isinstance(after, dict) else None
|
||||
if not before_revision or before_revision != after_revision or (_int(detail.get("size")) is not None and _int(detail.get("size")) != len(data)):
|
||||
raise ConnectorRevisionConflict("Seafile source changed or could not be revision-verified during download; refresh and retry")
|
||||
filename = filename_from_path(str(detail.get("name") or path))
|
||||
content_type = response.headers.get("content-type") or mimetypes.guess_type(filename)[0]
|
||||
external_id = f"{repo_id}:{normalize_connector_browse_path(path)}"
|
||||
@@ -127,7 +140,7 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=_clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified")),
|
||||
revision=before_revision,
|
||||
external_id=external_id,
|
||||
external_url=download_url,
|
||||
metadata={
|
||||
@@ -202,12 +215,18 @@ def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> C
|
||||
unc_path = _smb_unc_path(location, file_path)
|
||||
smbclient = _smbclient_module()
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
size = _smb_stat_size(stat_result)
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
with smbclient.open_file(unc_path, mode="rb", **kwargs) as handle:
|
||||
data = handle.read(max_bytes + 1)
|
||||
# Deny concurrent write/delete sharing while observing metadata and bytes.
|
||||
with smbclient.open_file(unc_path, mode="rb", share_access="r", **kwargs) as handle:
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
size = _smb_stat_size(stat_result)
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
data = handle.read(min(size, max_bytes) + 1 if size is not None else max_bytes + 1)
|
||||
after = smbclient.stat(unc_path, **kwargs)
|
||||
if (_smb_stat_revision(stat_result) != _smb_stat_revision(after)
|
||||
or getattr(stat_result, "st_ino", None) != getattr(after, "st_ino", None)
|
||||
or size is None or len(data) != size):
|
||||
raise ConnectorRevisionConflict("SMB source changed or could not be verified during download; refresh and retry")
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
@@ -264,16 +283,30 @@ def _download_s3_object(
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
etag: str | None,
|
||||
max_bytes: int,
|
||||
) -> tuple[Any, bytes]:
|
||||
request: dict[str, object] = {"Bucket": bucket, "Key": key}
|
||||
if version_id:
|
||||
# S3's literal "null" version is replaceable when versioning is suspended.
|
||||
if version_id and version_id != "null":
|
||||
request["VersionId"] = version_id
|
||||
elif etag:
|
||||
request["IfMatch"] = etag
|
||||
else:
|
||||
raise ConnectorRevisionConflict("S3 source has no revision or ETag for a verified download")
|
||||
try:
|
||||
response = client.get_object(**request)
|
||||
body = response.get("Body")
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
try:
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
finally:
|
||||
close = getattr(body, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
error = getattr(exc, "response", {})
|
||||
if isinstance(error, dict) and str(error.get("Error", {}).get("Code")) in {"PreconditionFailed", "412"}:
|
||||
raise ConnectorRevisionConflict("S3 source changed during conditional download; refresh and retry") from exc
|
||||
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
@@ -322,12 +355,16 @@ def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
etag=_clean(detail.get("ETag")),
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
if (_clean(response.get("VersionId")) not in (None, version_id)
|
||||
or (_clean(detail.get("ETag")) and _clean(response.get("ETag")) != _clean(detail.get("ETag")))):
|
||||
raise ConnectorRevisionConflict("S3 returned another source revision; refresh and retry")
|
||||
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
|
||||
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
|
||||
filename = filename_from_path(key)
|
||||
@@ -335,7 +372,7 @@ def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=version_id or etag or _clean(detail.get("LastModified")),
|
||||
revision=(version_id if version_id != "null" else None) or etag or _clean(detail.get("LastModified")),
|
||||
external_id=f"{bucket}:{key}",
|
||||
external_url=f"s3://{bucket}/{key}",
|
||||
metadata=_s3_download_metadata(
|
||||
|
||||
@@ -32,6 +32,7 @@ class ConnectorProviderDescriptor:
|
||||
"installed": self.installed,
|
||||
"browse_supported": self.browse_supported,
|
||||
"import_supported": self.import_supported,
|
||||
"write_supported": self.provider == "s3",
|
||||
"optional_dependency": self.optional_dependency,
|
||||
"permission_model": self.permission_model,
|
||||
"sync_strategy": self.sync_strategy,
|
||||
|
||||
@@ -14,6 +14,7 @@ from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
SYNC_MODES = {"manual"}
|
||||
WRITABLE_PROVIDERS = {"s3"}
|
||||
|
||||
|
||||
def connector_space_owner_id(space: FileConnectorSpace) -> str:
|
||||
@@ -40,6 +41,7 @@ def create_connector_space(
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
label = _normalize_label(label)
|
||||
sync_mode = _normalize_sync_mode(sync_mode)
|
||||
validate_connector_space_write_mode(profile, read_only=read_only)
|
||||
remote_path = normalize_connector_browse_path(remote_path)
|
||||
library_id = _clean_optional(library_id)
|
||||
|
||||
@@ -145,6 +147,7 @@ def update_connector_space(
|
||||
library_id: str | None = None,
|
||||
remote_path: str | None = None,
|
||||
sync_mode: str | None = None,
|
||||
read_only: bool | None = None,
|
||||
is_active: bool | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
@@ -178,6 +181,8 @@ def update_connector_space(
|
||||
space.remote_path = normalize_connector_browse_path(remote_path)
|
||||
if sync_mode is not None:
|
||||
space.sync_mode = _normalize_sync_mode(sync_mode)
|
||||
if read_only is not None:
|
||||
space.read_only = bool(read_only)
|
||||
if is_active is not None:
|
||||
space.is_active = bool(is_active)
|
||||
if metadata is not None:
|
||||
@@ -187,6 +192,17 @@ def update_connector_space(
|
||||
return space
|
||||
|
||||
|
||||
def validate_connector_space_write_mode(
|
||||
profile: ConnectorProfile, *, read_only: bool
|
||||
) -> None:
|
||||
if read_only:
|
||||
return
|
||||
if profile.provider not in WRITABLE_PROVIDERS or "write" not in profile.capabilities:
|
||||
raise FileStorageError(
|
||||
"Two-way mode requires an S3 connection with the write capability enabled"
|
||||
)
|
||||
|
||||
|
||||
def soft_delete_connector_space(
|
||||
session: Session,
|
||||
space: FileConnectorSpace,
|
||||
|
||||
@@ -0,0 +1,438 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
from typing import Any, Mapping
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
_clean,
|
||||
_s3_bucket,
|
||||
_s3_client,
|
||||
_s3_object_key,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorWriteResult:
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool
|
||||
provider: str
|
||||
remote_path: str
|
||||
revision: str | None
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _HeadProbe:
|
||||
observed: Mapping[str, Any] | None
|
||||
verified: bool
|
||||
exception_type: str | None = None
|
||||
|
||||
|
||||
def write_connector_file(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
tenant_id: str,
|
||||
library_id: str | None,
|
||||
remote_path: str,
|
||||
data: bytes,
|
||||
content_type: str | None,
|
||||
expected_revision: str | None,
|
||||
idempotency_key: str,
|
||||
) -> ConnectorWriteResult:
|
||||
if profile.provider != "s3" or "write" not in profile.capabilities:
|
||||
raise FileStorageError(
|
||||
"Remote writes require an S3 connection with the write capability enabled"
|
||||
)
|
||||
try:
|
||||
normalized_path = normalize_connector_browse_path(remote_path)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if not normalized_path:
|
||||
raise FileStorageError("Remote write requires an object path")
|
||||
try:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
key = _s3_object_key(profile, normalized_path)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if not bucket or not key:
|
||||
raise FileStorageError("S3 remote write requires a bucket and object path")
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
target_digest = hashlib.sha256(f"{bucket}:{key}".encode("utf-8")).hexdigest()
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"connector_profile_id": profile.id,
|
||||
"provider": profile.provider,
|
||||
"remote_target_sha256": target_digest,
|
||||
"content_sha256": checksum,
|
||||
"size_bytes": len(data),
|
||||
"expected_revision": expected_revision,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="connector-s3-write",
|
||||
idempotency_key=f"files-connector-write:{idempotency_key}",
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the connector space is explicitly configured for two-way writes",
|
||||
"the connection and inherited policy allow the exact remote path",
|
||||
"a conditional create or expected remote revision prevents blind overwrite",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"inspect provider metadata for the request and content digests",
|
||||
"resolve success only for an exact digest match or retry after verified absence",
|
||||
),
|
||||
verification_steps=(
|
||||
"read S3 object metadata after the write",
|
||||
"match the recorded request and content digests without downloading content",
|
||||
),
|
||||
),
|
||||
precondition_evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"content_sha256": checksum,
|
||||
"size_bytes": len(data),
|
||||
"conditional_write": True,
|
||||
},
|
||||
lease_resource_key=f"files:connector:{tenant_id}:{profile.id}:{target_digest[:40]}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_connector_object",
|
||||
resource_id=target_digest,
|
||||
metadata={
|
||||
"resources": ["postgresql", "s3-connector"],
|
||||
"provider": "s3",
|
||||
"connector_profile_id": profile.id,
|
||||
},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"This remote object is owned by another write or unresolved recovery operation"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; the remote object was not changed"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=started.status,
|
||||
replayed=True,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=None,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
operation = started.operation
|
||||
try:
|
||||
client = _s3_client(profile)
|
||||
except (ConnectorBrowseError, ConnectorBrowseUnsupported) as exc:
|
||||
operation.reject(
|
||||
summary="The S3 client was unavailable before the remote effect",
|
||||
evidence=_verified(
|
||||
{"external_effect_started": False}, exception_type=type(exc).__name__
|
||||
),
|
||||
)
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
try:
|
||||
try:
|
||||
before = _head_object(client, bucket=bucket, key=key)
|
||||
except Exception as exc:
|
||||
operation.reject(
|
||||
summary="Remote preconditions could not be inspected before the effect",
|
||||
evidence=_verified(
|
||||
{"external_effect_started": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise FileStorageError("S3 remote revision lookup failed") from exc
|
||||
expected = _normalize_revision(expected_revision)
|
||||
if before is not None and expected is None:
|
||||
operation.reject(
|
||||
summary="A blind remote overwrite was rejected",
|
||||
evidence=_verified(
|
||||
{"remote_object_exists": True, "expected_revision_supplied": False}
|
||||
),
|
||||
)
|
||||
raise FileStorageError(
|
||||
"The remote object already exists; reload it and supply its expected revision"
|
||||
)
|
||||
if expected is not None and (
|
||||
before is None or expected not in _observed_revisions(before)
|
||||
):
|
||||
operation.reject(
|
||||
summary="The expected remote revision did not match",
|
||||
evidence=_verified(
|
||||
{
|
||||
"remote_object_exists": before is not None,
|
||||
"expected_revision_matches": False,
|
||||
}
|
||||
),
|
||||
)
|
||||
raise FileStorageError("The remote object changed; reload before writing")
|
||||
params: dict[str, object] = {
|
||||
"Bucket": bucket,
|
||||
"Key": key,
|
||||
"Body": data,
|
||||
"Metadata": {
|
||||
"govoplan-sha256": checksum,
|
||||
"govoplan-request-id": started.operation_id,
|
||||
},
|
||||
}
|
||||
if content_type:
|
||||
params["ContentType"] = content_type
|
||||
if before is None:
|
||||
params["IfNoneMatch"] = "*"
|
||||
else:
|
||||
params["IfMatch"] = str(before.get("ETag") or expected_revision or "")
|
||||
try:
|
||||
client.put_object(**params)
|
||||
except Exception as exc:
|
||||
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||
if probe.verified and _matches_effect(
|
||||
probe.observed,
|
||||
checksum=checksum,
|
||||
operation_id=started.operation_id,
|
||||
):
|
||||
operation.succeed(
|
||||
evidence=_success_evidence(
|
||||
probe.observed,
|
||||
checksum=checksum,
|
||||
operation_id=started.operation_id,
|
||||
)
|
||||
)
|
||||
return _result(
|
||||
started.operation_id,
|
||||
normalized_path,
|
||||
checksum,
|
||||
len(data),
|
||||
probe.observed,
|
||||
)
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The S3 write outcome requires reconciliation",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
"exception_type": type(exc).__name__,
|
||||
},
|
||||
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=_revision(probe.observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||
if not probe.verified:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The S3 write returned but provider evidence could not be queried",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
},
|
||||
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=None,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
observed = probe.observed
|
||||
if not _matches_effect(
|
||||
observed, checksum=checksum, operation_id=started.operation_id
|
||||
):
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="The S3 write returned but exact provider evidence did not match",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
},
|
||||
failure_summary="Reconcile the remote object before another write",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=_revision(observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
operation.succeed(
|
||||
evidence=_success_evidence(
|
||||
observed, checksum=checksum, operation_id=started.operation_id
|
||||
)
|
||||
)
|
||||
return _result(
|
||||
started.operation_id,
|
||||
normalized_path,
|
||||
checksum,
|
||||
len(data),
|
||||
observed,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
def _head_object(client: Any, *, bucket: str, key: str) -> Mapping[str, Any] | None:
|
||||
try:
|
||||
response = client.head_object(Bucket=bucket, Key=key)
|
||||
except Exception as exc:
|
||||
if _is_not_found(exc):
|
||||
return None
|
||||
raise
|
||||
if not isinstance(response, Mapping):
|
||||
raise FileStorageError("S3 connector returned invalid object metadata")
|
||||
return response
|
||||
|
||||
|
||||
def _probe_head_object(client: Any, *, bucket: str, key: str) -> _HeadProbe:
|
||||
try:
|
||||
return _HeadProbe(
|
||||
observed=_head_object(client, bucket=bucket, key=key), verified=True
|
||||
)
|
||||
except Exception as exc:
|
||||
return _HeadProbe(
|
||||
observed=None,
|
||||
verified=False,
|
||||
exception_type=type(exc).__name__,
|
||||
)
|
||||
|
||||
|
||||
def _is_not_found(exc: Exception) -> bool:
|
||||
response = getattr(exc, "response", None)
|
||||
if not isinstance(response, Mapping):
|
||||
return False
|
||||
error = response.get("Error")
|
||||
code = error.get("Code") if isinstance(error, Mapping) else None
|
||||
status = response.get("ResponseMetadata")
|
||||
http_status = status.get("HTTPStatusCode") if isinstance(status, Mapping) else None
|
||||
return str(code).casefold() in {"404", "nosuchkey", "notfound"} or http_status == 404
|
||||
|
||||
|
||||
def _metadata(observed: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||
value = observed.get("Metadata") if observed else None
|
||||
return value if isinstance(value, Mapping) else {}
|
||||
|
||||
|
||||
def _matches_effect(
|
||||
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||
) -> bool:
|
||||
metadata = _metadata(observed)
|
||||
return bool(
|
||||
observed is not None
|
||||
and _clean(metadata.get("govoplan-sha256")) == checksum
|
||||
and _clean(metadata.get("govoplan-request-id")) == operation_id
|
||||
)
|
||||
|
||||
|
||||
def _revision(observed: Mapping[str, Any] | None) -> str | None:
|
||||
if observed is None:
|
||||
return None
|
||||
return _clean(observed.get("VersionId") or observed.get("ETag"))
|
||||
|
||||
|
||||
def _normalize_revision(value: str | None) -> str | None:
|
||||
cleaned = _clean(value)
|
||||
return cleaned.strip('"') if cleaned else None
|
||||
|
||||
|
||||
def _observed_revisions(observed: Mapping[str, Any]) -> set[str]:
|
||||
return {
|
||||
normalized
|
||||
for value in (observed.get("VersionId"), observed.get("ETag"))
|
||||
if (normalized := _normalize_revision(_clean(value))) is not None
|
||||
}
|
||||
|
||||
|
||||
def _success_evidence(
|
||||
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||
) -> dict[str, object]:
|
||||
return _verified(
|
||||
{
|
||||
"remote_object_present": observed is not None,
|
||||
"content_digest_matches": _clean(_metadata(observed).get("govoplan-sha256")) == checksum,
|
||||
"request_marker_matches": _clean(_metadata(observed).get("govoplan-request-id")) == operation_id,
|
||||
},
|
||||
revision=_revision(observed),
|
||||
content_sha256=checksum,
|
||||
)
|
||||
|
||||
|
||||
def _public_observation(probe: _HeadProbe) -> dict[str, object]:
|
||||
observed = probe.observed
|
||||
return {
|
||||
"probe_verified": probe.verified,
|
||||
"probe_exception_type": probe.exception_type,
|
||||
"present": observed is not None,
|
||||
"revision": _revision(observed),
|
||||
"has_content_digest": bool(_clean(_metadata(observed).get("govoplan-sha256"))),
|
||||
"has_request_marker": bool(_clean(_metadata(observed).get("govoplan-request-id"))),
|
||||
}
|
||||
|
||||
|
||||
def _verified(checks: dict[str, object], **details: object) -> dict[str, object]:
|
||||
return {"verified": True, "checks": checks, **details}
|
||||
|
||||
|
||||
def _result(
|
||||
operation_id: str,
|
||||
remote_path: str,
|
||||
checksum: str,
|
||||
size_bytes: int,
|
||||
observed: Mapping[str, Any] | None,
|
||||
) -> ConnectorWriteResult:
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=operation_id,
|
||||
status=RecoveryStatus.SUCCEEDED.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=remote_path,
|
||||
revision=_revision(observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=size_bytes,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["ConnectorWriteResult", "write_connector_file"]
|
||||
@@ -2,6 +2,8 @@ from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any, Iterable
|
||||
@@ -15,13 +17,14 @@ from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, F
|
||||
from govoplan_files.backend.runtime import get_registry, settings
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileSourceConflict, FileStorageError, UploadedStoredFile, utcnow
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||
from govoplan_files.backend.storage.provenance import source_identity as _source_identity, source_identity_hash, source_provenance_from_metadata
|
||||
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
QUARANTINED_BLOB_STATUSES,
|
||||
@@ -30,6 +33,39 @@ from govoplan_files.backend.storage.integrity import (
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
_ARCHIVE_STORAGE_BACKEND: ContextVar[dict[str, StorageBackend] | None] = ContextVar(
|
||||
"govoplan.files.archive_storage_backend", default=None
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def archive_storage_backend_scope():
|
||||
"""Reuse deployment transport only, never principals, policy or file data.
|
||||
|
||||
Lazy creation avoids opening storage for rejected or empty work. Recovery
|
||||
effects retain the backend until post-commit verification is complete.
|
||||
Nested archive work shares the same deployment transport, and finally
|
||||
restores the caller context on success, failure or generator cancellation.
|
||||
"""
|
||||
if _ARCHIVE_STORAGE_BACKEND.get() is not None:
|
||||
yield
|
||||
return
|
||||
token = _ARCHIVE_STORAGE_BACKEND.set({})
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_ARCHIVE_STORAGE_BACKEND.reset(token)
|
||||
|
||||
|
||||
def _archive_write_backend() -> StorageBackend:
|
||||
scope = _ARCHIVE_STORAGE_BACKEND.get()
|
||||
if scope is None:
|
||||
return get_storage_backend()
|
||||
if "backend" not in scope:
|
||||
scope["backend"] = get_storage_backend()
|
||||
return scope["backend"]
|
||||
|
||||
|
||||
def _campaign_access_provider() -> CampaignAccessProvider:
|
||||
registry = get_registry()
|
||||
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||
@@ -84,10 +120,11 @@ def _get_or_create_blob(
|
||||
blob = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size, FileBlob.protection_discriminator == protection_discriminator)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if blob:
|
||||
backend = get_storage_backend()
|
||||
backend = _archive_write_backend()
|
||||
repair_required = (
|
||||
blob.integrity_status in QUARANTINED_BLOB_STATUSES
|
||||
or blob.quarantined_at is not None
|
||||
@@ -161,7 +198,7 @@ def _get_or_create_blob(
|
||||
|
||||
blob_id = str(uuid4())
|
||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
|
||||
backend = get_storage_backend()
|
||||
backend = _archive_write_backend()
|
||||
recovery = begin_blob_write_recovery(
|
||||
session,
|
||||
backend=backend,
|
||||
@@ -213,7 +250,6 @@ def _get_or_create_blob(
|
||||
integrity_checked_at=utcnow(),
|
||||
)
|
||||
session.add(blob)
|
||||
session.flush()
|
||||
return blob
|
||||
|
||||
|
||||
@@ -264,7 +300,12 @@ def create_file_asset(
|
||||
logical_path = _next_available_logical_path(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, desired_path=logical_path)
|
||||
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=encryption_vault_id)
|
||||
# Assign the immutable identities before the first flush. Publishing an
|
||||
# incomplete asset and immediately updating its current version doubled
|
||||
# change events and added a redundant UPDATE for every imported member.
|
||||
version_id = str(uuid4())
|
||||
asset = FileAsset(
|
||||
id=str(uuid4()),
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_user_id=owner_id if owner_type == "user" else None,
|
||||
@@ -274,10 +315,10 @@ def create_file_asset(
|
||||
description=description,
|
||||
created_by_user_id=user_id,
|
||||
metadata_=metadata or {},
|
||||
current_version_id=version_id,
|
||||
)
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
version = FileVersion(
|
||||
id=version_id,
|
||||
tenant_id=tenant_id,
|
||||
file_asset_id=asset.id,
|
||||
blob_id=blob.id,
|
||||
@@ -289,10 +330,8 @@ def create_file_asset(
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
created_by_user_id=user_id,
|
||||
)
|
||||
session.add(version)
|
||||
session.add_all((asset, version))
|
||||
session.flush()
|
||||
asset.current_version_id = version.id
|
||||
session.add(asset)
|
||||
if campaign_id:
|
||||
share_file(session, tenant_id=tenant_id, asset=asset, target_type="campaign", target_id=campaign_id, permission="read", user_id=user_id)
|
||||
return UploadedStoredFile(asset=asset, version=version, blob=blob)
|
||||
@@ -375,14 +414,17 @@ def find_asset_by_source(
|
||||
return None
|
||||
assets = (
|
||||
_asset_query_for_owner(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id)
|
||||
.filter(FileAsset.deleted_at.is_(None))
|
||||
.order_by(FileAsset.updated_at.desc())
|
||||
.filter(FileAsset.deleted_at.is_(None), FileAsset.source_identity_hash == source_identity_hash(source_provenance))
|
||||
.limit(2)
|
||||
.all()
|
||||
)
|
||||
for asset in assets:
|
||||
if _source_identity(source_provenance_from_metadata(asset.metadata_ or {})) == wanted:
|
||||
return asset
|
||||
return None
|
||||
if len(assets) > 1:
|
||||
raise FileSourceConflict("Multiple active files have this source identity; review their provenance before synchronizing. No file was selected or merged.")
|
||||
if not assets:
|
||||
return None
|
||||
if _source_identity(source_provenance_from_metadata(assets[0].metadata_ or {})) != wanted:
|
||||
raise FileSourceConflict("The indexed file source identity does not match its provenance; synchronization was stopped.")
|
||||
return assets[0]
|
||||
|
||||
|
||||
def update_file_asset_content(
|
||||
@@ -529,6 +571,41 @@ def list_assets_for_user(
|
||||
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).all()
|
||||
|
||||
|
||||
def list_recent_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
limit: int,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileAsset]:
|
||||
"""Return a bounded recent projection through the normal access query."""
|
||||
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return (
|
||||
query.order_by(FileAsset.updated_at.desc(), FileAsset.id.asc())
|
||||
.limit(max(1, limit))
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def list_assets_for_user_window(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -731,6 +808,21 @@ def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVer
|
||||
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||
|
||||
|
||||
def read_asset_version_bytes(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
version_id: str,
|
||||
) -> tuple[bytes, FileVersion, FileBlob]:
|
||||
version = session.get(FileVersion, version_id)
|
||||
if version is None or version.file_asset_id != asset.id:
|
||||
raise FileStorageError("File version not found")
|
||||
blob = session.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != asset.tenant_id:
|
||||
raise FileStorageError("File blob not found")
|
||||
backend = get_storage_backend()
|
||||
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||
|
||||
|
||||
def share_file(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -1057,7 +1149,7 @@ def _resolution_by_path(conflict_resolutions: Iterable[FileConflictResolution] |
|
||||
|
||||
def _normalize_conflict_strategy(strategy: str | None) -> str:
|
||||
normalized = (strategy or "reject").lower().strip()
|
||||
if normalized not in {"reject", "overwrite", "rename"}:
|
||||
if normalized not in {"reject", "overwrite", "rename", "skip"}:
|
||||
raise FileStorageError("Unsupported conflict strategy")
|
||||
return normalized
|
||||
|
||||
@@ -1072,29 +1164,6 @@ def _next_version_number(session: Session, asset_id: str) -> int:
|
||||
return (int(row[0]) if row else 0) + 1
|
||||
|
||||
|
||||
def _source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||
if not provenance:
|
||||
return None
|
||||
connector_id = _clean_identity(provenance.get("connector_id"))
|
||||
provider = _clean_identity(provenance.get("provider"))
|
||||
external_id = _clean_identity(provenance.get("external_id"))
|
||||
if connector_id and external_id:
|
||||
return ("external_id", connector_id, provider, external_id)
|
||||
external_path = _clean_identity(provenance.get("external_path"))
|
||||
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||
library_id = _clean_identity(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||
if connector_id and external_path:
|
||||
return ("external_path", connector_id, provider, library_id, external_path)
|
||||
return None
|
||||
|
||||
|
||||
def _clean_identity(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _copy_asset_to_path(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
|
||||
@@ -0,0 +1,796 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Callable, Iterable
|
||||
|
||||
from sqlalchemy import func, inspect, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.connector_spaces import connector_space_owner_id
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgePreviewItem:
|
||||
file_id: str
|
||||
filename: str
|
||||
lifecycle_revision: int
|
||||
deleted_at: datetime | None
|
||||
retained_until: datetime | None
|
||||
legal_hold: bool
|
||||
blockers: tuple[str, ...]
|
||||
blob_ids: tuple[str, ...]
|
||||
|
||||
def digest_payload(self) -> dict[str, object]:
|
||||
return {
|
||||
"file_id": self.file_id,
|
||||
"filename": self.filename,
|
||||
"lifecycle_revision": self.lifecycle_revision,
|
||||
"deleted_at": _iso(self.deleted_at),
|
||||
"retained_until": _iso(self.retained_until),
|
||||
"legal_hold": self.legal_hold,
|
||||
"blockers": list(self.blockers),
|
||||
"blob_ids": list(self.blob_ids),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgePreview:
|
||||
preview_sha256: str
|
||||
items: tuple[PurgePreviewItem, ...]
|
||||
|
||||
@property
|
||||
def eligible(self) -> bool:
|
||||
return all(not item.blockers for item in self.items)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgeResult:
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool
|
||||
purged_files: int
|
||||
released_blobs: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BlobGcResult:
|
||||
inspected_blobs: int
|
||||
deleted_blobs: int
|
||||
unresolved_operation_ids: tuple[str, ...]
|
||||
|
||||
|
||||
def set_asset_lifecycle(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
retained_until: datetime | None,
|
||||
legal_hold: bool,
|
||||
reason: str,
|
||||
expected_revision: int,
|
||||
) -> FileAsset:
|
||||
if asset.lifecycle_revision != expected_revision:
|
||||
raise FileStorageError("File lifecycle settings changed; reload before saving")
|
||||
asset.retained_until = retained_until
|
||||
asset.legal_hold = bool(legal_hold)
|
||||
asset.lifecycle_reason = reason.strip()
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return asset
|
||||
|
||||
|
||||
def get_asset_for_lifecycle(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
asset_id: str,
|
||||
is_admin: bool,
|
||||
) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if asset is None or asset.tenant_id != tenant_id:
|
||||
raise FileStorageError("File not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_asset_owner_id(asset),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return asset
|
||||
|
||||
|
||||
def restore_asset(session: Session, asset: FileAsset) -> bool:
|
||||
if asset.deleted_at is None:
|
||||
return False
|
||||
collision = _asset_owner_query(session, asset).filter(
|
||||
FileAsset.id != asset.id,
|
||||
FileAsset.display_path == asset.display_path,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
"The file cannot be restored because its path is already in use"
|
||||
)
|
||||
asset.deleted_at = None
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return True
|
||||
|
||||
|
||||
def restore_folder(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
path: str,
|
||||
recursive: bool,
|
||||
is_admin: bool,
|
||||
) -> tuple[int, int]:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
normalized = normalize_folder(path)
|
||||
if not normalized:
|
||||
raise FileStorageError("Folder path is required")
|
||||
prefix = f"{normalized}/"
|
||||
folders = _folder_owner_query(
|
||||
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||
).filter(FileFolder.deleted_at.is_not(None))
|
||||
assets = _asset_owner_query_by_values(
|
||||
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||
).filter(FileAsset.deleted_at.is_not(None))
|
||||
if recursive:
|
||||
folders = folders.filter(
|
||||
(FileFolder.path == normalized) | FileFolder.path.like(f"{prefix}%")
|
||||
)
|
||||
assets = assets.filter(FileAsset.display_path.like(f"{prefix}%"))
|
||||
else:
|
||||
folders = folders.filter(FileFolder.path == normalized)
|
||||
assets = assets.filter(False)
|
||||
folder_rows = folders.order_by(FileFolder.path.asc()).all()
|
||||
asset_rows = assets.order_by(FileAsset.display_path.asc()).all()
|
||||
if not folder_rows:
|
||||
raise FileStorageError("Deleted folder not found")
|
||||
for folder in folder_rows:
|
||||
collision = _folder_owner_query(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
).filter(
|
||||
FileFolder.id != folder.id,
|
||||
FileFolder.path == folder.path,
|
||||
FileFolder.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
f"The folder cannot be restored because its path is in use: {folder.path}"
|
||||
)
|
||||
for asset in asset_rows:
|
||||
collision = _asset_owner_query(session, asset).filter(
|
||||
FileAsset.id != asset.id,
|
||||
FileAsset.display_path == asset.display_path,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
f"A file cannot be restored because its path is in use: {asset.display_path}"
|
||||
)
|
||||
for folder in folder_rows:
|
||||
folder.deleted_at = None
|
||||
session.add(folder)
|
||||
for asset in asset_rows:
|
||||
asset.deleted_at = None
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return len(folder_rows), len(asset_rows)
|
||||
|
||||
|
||||
def restore_connector_space(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
space_id: str,
|
||||
is_admin: bool,
|
||||
) -> FileConnectorSpace:
|
||||
space = session.get(FileConnectorSpace, space_id)
|
||||
if space is None or space.tenant_id != tenant_id or space.deleted_at is None:
|
||||
raise FileStorageError("Deleted connector space not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
collision = _connector_space_owner_query(session, space).filter(
|
||||
FileConnectorSpace.id != space.id,
|
||||
FileConnectorSpace.label == space.label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
"The connector space cannot be restored because its label is in use"
|
||||
)
|
||||
space.deleted_at = None
|
||||
space.is_active = True
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def preview_asset_purge(
|
||||
session: Session, *, tenant_id: str, file_ids: Iterable[str]
|
||||
) -> PurgePreview:
|
||||
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||
if not normalized_ids or len(normalized_ids) > 100:
|
||||
raise FileStorageError("Purge preview must contain between 1 and 100 files")
|
||||
assets = (
|
||||
session.query(FileAsset)
|
||||
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||
.order_by(FileAsset.id.asc())
|
||||
.all()
|
||||
)
|
||||
by_id = {asset.id: asset for asset in assets}
|
||||
items: list[PurgePreviewItem] = []
|
||||
for file_id in normalized_ids:
|
||||
asset = by_id.get(file_id)
|
||||
if asset is None:
|
||||
items.append(
|
||||
PurgePreviewItem(
|
||||
file_id=file_id,
|
||||
filename="",
|
||||
lifecycle_revision=0,
|
||||
deleted_at=None,
|
||||
retained_until=None,
|
||||
legal_hold=False,
|
||||
blockers=("not_found",),
|
||||
blob_ids=(),
|
||||
)
|
||||
)
|
||||
continue
|
||||
blockers = _asset_purge_blockers(session, asset)
|
||||
blob_ids = tuple(
|
||||
sorted(
|
||||
row[0]
|
||||
for row in session.query(FileVersion.blob_id)
|
||||
.filter(FileVersion.file_asset_id == asset.id)
|
||||
.distinct()
|
||||
.all()
|
||||
)
|
||||
)
|
||||
items.append(
|
||||
PurgePreviewItem(
|
||||
file_id=asset.id,
|
||||
filename=asset.filename,
|
||||
lifecycle_revision=asset.lifecycle_revision,
|
||||
deleted_at=asset.deleted_at,
|
||||
retained_until=asset.retained_until,
|
||||
legal_hold=asset.legal_hold,
|
||||
blockers=tuple(blockers),
|
||||
blob_ids=blob_ids,
|
||||
)
|
||||
)
|
||||
payload = [item.digest_payload() for item in items]
|
||||
digest = hashlib.sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
).hexdigest()
|
||||
return PurgePreview(preview_sha256=digest, items=tuple(items))
|
||||
|
||||
|
||||
def execute_asset_purge(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
file_ids: Iterable[str],
|
||||
preview_sha256: str,
|
||||
idempotency_key: str,
|
||||
approval_reference: str,
|
||||
before_commit: Callable[[list[FileAsset]], None] | None = None,
|
||||
) -> PurgeResult:
|
||||
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||
if not normalized_ids or len(normalized_ids) > 100:
|
||||
raise FileStorageError("Purge must contain between 1 and 100 files")
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"file_ids": normalized_ids,
|
||||
"preview_sha256": preview_sha256,
|
||||
"approval_reference": approval_reference,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="asset-hard-purge",
|
||||
idempotency_key=f"files-asset-purge:{idempotency_key}",
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.IRREVERSIBLE,
|
||||
preconditions=(
|
||||
"the actor holds the dedicated Files purge permission",
|
||||
"the signed-off preview still matches current lifecycle state",
|
||||
"every target is soft-deleted and free of retention or legal-hold blockers",
|
||||
),
|
||||
verification_steps=(
|
||||
"verify every target FileAsset and FileVersion row is absent",
|
||||
"recalculate retained FileBlob reference counts before later garbage collection",
|
||||
),
|
||||
approval_reference=approval_reference,
|
||||
),
|
||||
precondition_evidence={
|
||||
"preview_sha256": preview_sha256,
|
||||
"target_count": len(normalized_ids),
|
||||
},
|
||||
lease_resource_key=f"files:purge:{tenant_id}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_asset_batch",
|
||||
resource_id=preview_sha256,
|
||||
metadata={"resources": ["postgresql"], "bounded_target_count": len(normalized_ids)},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"Another purge or unresolved retry owns the tenant purge fence"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError("The Files recovery ledger is unavailable; nothing was purged") from exc
|
||||
if started.replayed or started.operation is None:
|
||||
return PurgeResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=started.status,
|
||||
replayed=True,
|
||||
purged_files=0,
|
||||
released_blobs=0,
|
||||
)
|
||||
operation = started.operation
|
||||
try:
|
||||
preview = preview_asset_purge(session, tenant_id=tenant_id, file_ids=normalized_ids)
|
||||
if preview.preview_sha256 != preview_sha256:
|
||||
operation.reject(
|
||||
summary="The purge preview became stale before execution",
|
||||
evidence=_verified_evidence(
|
||||
{"preview_matches": False},
|
||||
expected_preview_sha256=preview_sha256,
|
||||
observed_preview_sha256=preview.preview_sha256,
|
||||
),
|
||||
)
|
||||
raise FileStorageError("The purge preview is stale; preview again")
|
||||
if not preview.eligible:
|
||||
operation.reject(
|
||||
summary="The purge was blocked by current lifecycle policy",
|
||||
evidence=_verified_evidence(
|
||||
{"lifecycle_policy_allows": False},
|
||||
blocked_files=[
|
||||
{"file_id": item.file_id, "blockers": list(item.blockers)}
|
||||
for item in preview.items
|
||||
if item.blockers
|
||||
]
|
||||
),
|
||||
)
|
||||
raise FileStorageError("One or more files are not eligible for purge")
|
||||
assets = (
|
||||
session.query(FileAsset)
|
||||
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||
.with_for_update()
|
||||
.all()
|
||||
)
|
||||
blob_ids = sorted({blob_id for item in preview.items for blob_id in item.blob_ids})
|
||||
if before_commit is not None:
|
||||
before_commit(assets)
|
||||
session.query(FileShare).filter(
|
||||
FileShare.file_asset_id.in_(normalized_ids)
|
||||
).delete(synchronize_session=False)
|
||||
session.query(FileVersion).filter(
|
||||
FileVersion.file_asset_id.in_(normalized_ids)
|
||||
).delete(synchronize_session=False)
|
||||
for asset in assets:
|
||||
session.delete(asset)
|
||||
session.flush()
|
||||
released_blobs = 0
|
||||
for blob_id in blob_ids:
|
||||
references = int(
|
||||
session.query(func.count(FileVersion.id))
|
||||
.filter(FileVersion.blob_id == blob_id)
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
blob = session.get(FileBlob, blob_id)
|
||||
if blob is not None:
|
||||
blob.ref_count = references
|
||||
session.add(blob)
|
||||
if references == 0:
|
||||
released_blobs += 1
|
||||
operation.commit_verified_success(
|
||||
session,
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"assets_absent": True,
|
||||
"versions_absent": True,
|
||||
"blob_reference_counts_recalculated": True,
|
||||
},
|
||||
purged_file_ids=normalized_ids,
|
||||
purged_files=len(assets),
|
||||
released_blobs=released_blobs,
|
||||
preview_sha256=preview_sha256,
|
||||
),
|
||||
)
|
||||
return PurgeResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.SUCCEEDED.value,
|
||||
replayed=False,
|
||||
purged_files=len(assets),
|
||||
released_blobs=released_blobs,
|
||||
)
|
||||
except FileStorageError:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.release_unresolved()
|
||||
raise
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.reject(
|
||||
summary="The purge failed before its database transaction committed",
|
||||
evidence=_verified_evidence(
|
||||
{"database_transaction_committed": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
def garbage_collect_unreferenced_blobs(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
limit: int,
|
||||
approval_reference: str,
|
||||
) -> BlobGcResult:
|
||||
candidate_rows = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.tenant_id == tenant_id)
|
||||
.filter(
|
||||
or_(FileBlob.retained_until.is_(None), FileBlob.retained_until <= utcnow())
|
||||
)
|
||||
.filter(
|
||||
~session.query(FileVersion.id)
|
||||
.filter(FileVersion.blob_id == FileBlob.id)
|
||||
.exists()
|
||||
)
|
||||
.order_by(FileBlob.created_at.asc(), FileBlob.id.asc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
candidates = tuple(
|
||||
(row.id, row.storage_key) for row in candidate_rows
|
||||
)
|
||||
# Close the read snapshot before the independent recovery transaction takes
|
||||
# its durable SQLite/PostgreSQL write lock.
|
||||
session.commit()
|
||||
deleted = 0
|
||||
unresolved: list[str] = []
|
||||
backend = get_storage_backend()
|
||||
for candidate_id, storage_key in candidates:
|
||||
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"blob_id": candidate_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"approval_reference": approval_reference,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="blob-garbage-collection",
|
||||
idempotency_key=(
|
||||
f"files-blob-gc:{candidate_id}:{key_digest[:12]}:"
|
||||
f"{hashlib.sha256(approval_reference.encode('utf-8')).hexdigest()[:12]}"
|
||||
),
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the caller holds dedicated Files purge authority",
|
||||
"a fresh database check shows no FileVersion reference",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"verify the object is absent",
|
||||
"delete the unreferenced FileBlob row after absence is proven",
|
||||
),
|
||||
verification_steps=(
|
||||
"recheck FileVersion references while holding the blob fence",
|
||||
"probe the exact managed storage key after deletion",
|
||||
),
|
||||
approval_reference=approval_reference,
|
||||
),
|
||||
precondition_evidence={
|
||||
"blob_id": candidate_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"observed_reference_count": 0,
|
||||
},
|
||||
lease_resource_key=f"files:blob:{tenant_id}:{candidate_id}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=candidate_id,
|
||||
metadata={"resources": ["postgresql", "object-storage"], "storage_backend": backend.name},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict):
|
||||
continue
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no blob bytes were deleted"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
continue
|
||||
operation = started.operation
|
||||
try:
|
||||
blob = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.id == candidate_id, FileBlob.tenant_id == tenant_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
references = int(
|
||||
session.query(func.count(FileVersion.id))
|
||||
.filter(FileVersion.blob_id == candidate_id)
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
blob_retained = bool(
|
||||
blob is not None
|
||||
and blob.retained_until is not None
|
||||
and _as_utc(blob.retained_until) > utcnow()
|
||||
)
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.reject(
|
||||
summary="Blob eligibility could not be rechecked before deletion",
|
||||
evidence=_verified_evidence(
|
||||
{"external_effect_started": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise FileStorageError(
|
||||
"Blob eligibility could not be rechecked; no bytes were deleted"
|
||||
) from exc
|
||||
if blob is None or references or blob_retained:
|
||||
operation.reject(
|
||||
summary="Blob garbage collection was no longer eligible",
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"blob_present": blob is not None,
|
||||
"reference_count_zero": references == 0,
|
||||
"blob_retention_expired": not blob_retained,
|
||||
},
|
||||
reference_count=references,
|
||||
),
|
||||
)
|
||||
session.rollback()
|
||||
continue
|
||||
try:
|
||||
backend.delete(blob.storage_key)
|
||||
object_present = backend.exists(blob.storage_key)
|
||||
except (StorageBackendError, OSError) as exc:
|
||||
try:
|
||||
object_present = backend.exists(blob.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
object_present = None
|
||||
session.rollback()
|
||||
if object_present is True:
|
||||
operation.fail(
|
||||
summary="The unreferenced blob object was retained",
|
||||
evidence={"object_present": True, "exception_type": type(exc).__name__},
|
||||
)
|
||||
else:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The blob deletion outcome could not be verified",
|
||||
evidence={"object_present": object_present, "exception_type": type(exc).__name__},
|
||||
failure_summary="Reconcile the exact blob key before retrying garbage collection",
|
||||
)
|
||||
unresolved.append(started.operation_id)
|
||||
continue
|
||||
if object_present:
|
||||
session.rollback()
|
||||
operation.fail(
|
||||
summary="The object store did not delete the unreferenced blob",
|
||||
evidence={"object_present": True},
|
||||
)
|
||||
continue
|
||||
try:
|
||||
session.delete(blob)
|
||||
session.flush()
|
||||
operation.commit_verified_success(
|
||||
session,
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"object_absent": True,
|
||||
"database_blob_absent": True,
|
||||
"reference_count_zero": True,
|
||||
},
|
||||
blob_id=candidate_id,
|
||||
storage_key_sha256=key_digest,
|
||||
object_present=False,
|
||||
database_blob_present=False,
|
||||
reference_count=0,
|
||||
),
|
||||
)
|
||||
deleted += 1
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="The object is absent but blob metadata still requires forward recovery",
|
||||
evidence={"object_present": False, "exception_type": type(exc).__name__},
|
||||
failure_summary="Delete the FileBlob row only after rechecking all FileVersion references",
|
||||
)
|
||||
unresolved.append(started.operation_id)
|
||||
return BlobGcResult(
|
||||
inspected_blobs=len(candidates),
|
||||
deleted_blobs=deleted,
|
||||
unresolved_operation_ids=tuple(unresolved),
|
||||
)
|
||||
|
||||
|
||||
def _asset_purge_blockers(session: Session, asset: FileAsset) -> list[str]:
|
||||
blockers: list[str] = []
|
||||
if asset.deleted_at is None:
|
||||
blockers.append("not_soft_deleted")
|
||||
if asset.legal_hold:
|
||||
blockers.append("legal_hold")
|
||||
if asset.retained_until is not None and _as_utc(asset.retained_until) > utcnow():
|
||||
blockers.append("retention_active")
|
||||
if _table_exists(session, CampaignAttachmentUse.__tablename__) and session.query(
|
||||
CampaignAttachmentUse.id
|
||||
).filter(CampaignAttachmentUse.file_asset_id == asset.id).first() is not None:
|
||||
blockers.append("campaign_evidence")
|
||||
if _table_exists(session, FileFormEvidenceGrant.__tablename__) and session.query(
|
||||
FileFormEvidenceGrant.id
|
||||
).filter(FileFormEvidenceGrant.file_asset_id == asset.id).first() is not None:
|
||||
blockers.append("form_evidence")
|
||||
if session.query(FileShare.id).filter(
|
||||
FileShare.file_asset_id == asset.id,
|
||||
effective_file_share_clause(),
|
||||
).first() is not None:
|
||||
blockers.append("active_share")
|
||||
return blockers
|
||||
|
||||
|
||||
def _asset_owner_id(asset: FileAsset) -> str:
|
||||
owner_id = asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id
|
||||
if not owner_id:
|
||||
raise FileStorageError("File has no valid owner")
|
||||
return owner_id
|
||||
|
||||
|
||||
def _asset_owner_query(session: Session, asset: FileAsset):
|
||||
return _asset_owner_query_by_values(
|
||||
session,
|
||||
tenant_id=asset.tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_asset_owner_id(asset),
|
||||
)
|
||||
|
||||
|
||||
def _asset_owner_query_by_values(
|
||||
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||
):
|
||||
query = session.query(FileAsset).filter(
|
||||
FileAsset.tenant_id == tenant_id, FileAsset.owner_type == owner_type
|
||||
)
|
||||
if owner_type == "user":
|
||||
return query.filter(FileAsset.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileAsset.owner_group_id == owner_id)
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def _folder_owner_query(
|
||||
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||
):
|
||||
query = session.query(FileFolder).filter(
|
||||
FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type
|
||||
)
|
||||
if owner_type == "user":
|
||||
return query.filter(FileFolder.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileFolder.owner_group_id == owner_id)
|
||||
raise FileStorageError("Folders must be owned by a user or group")
|
||||
|
||||
|
||||
def _connector_space_owner_query(session: Session, space: FileConnectorSpace):
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == space.tenant_id,
|
||||
FileConnectorSpace.owner_type == space.owner_type,
|
||||
)
|
||||
owner_id = connector_space_owner_id(space)
|
||||
if space.owner_type == "user":
|
||||
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||
|
||||
|
||||
def _as_utc(value: datetime) -> datetime:
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
return _as_utc(value).isoformat() if value is not None else None
|
||||
|
||||
|
||||
def _table_exists(session: Session, table_name: str) -> bool:
|
||||
return bool(session.bind is not None and inspect(session.bind).has_table(table_name))
|
||||
|
||||
|
||||
def _verified_evidence(
|
||||
checks: dict[str, object], **details: object
|
||||
) -> dict[str, object]:
|
||||
return {"verified": True, "checks": checks, **details}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BlobGcResult",
|
||||
"PurgePreview",
|
||||
"PurgePreviewItem",
|
||||
"PurgeResult",
|
||||
"execute_asset_purge",
|
||||
"garbage_collect_unreferenced_blobs",
|
||||
"get_asset_for_lifecycle",
|
||||
"preview_asset_purge",
|
||||
"restore_asset",
|
||||
"restore_connector_space",
|
||||
"restore_folder",
|
||||
"set_asset_lifecycle",
|
||||
]
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Optional libarchive ZIPCrypto acceleration, without filesystem extraction.
|
||||
|
||||
Only public libarchive read APIs are bound. Callers must preflight the complete
|
||||
ZIP directory with the ordinary archive validator. Missing libraries/symbols
|
||||
select the Python path before decoding; native failures never retry a decoder.
|
||||
Public ABI reference: https://github.com/libarchive/libarchive/blob/master/libarchive/archive.h
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterator, Mapping
|
||||
import ctypes
|
||||
from ctypes.util import find_library
|
||||
from functools import lru_cache
|
||||
import os
|
||||
from os import PathLike
|
||||
import stat
|
||||
from typing import Any
|
||||
import zlib
|
||||
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_ARCHIVE_OK = 0
|
||||
_ARCHIVE_EOF = 1
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def native_zip_library() -> Any | None:
|
||||
name = find_library("archive")
|
||||
if not name:
|
||||
return None
|
||||
try:
|
||||
library = ctypes.CDLL(name)
|
||||
signatures = {
|
||||
"archive_read_new": ([], ctypes.c_void_p),
|
||||
"archive_read_support_filter_none": ([ctypes.c_void_p], ctypes.c_int),
|
||||
"archive_read_support_format_zip": ([ctypes.c_void_p], ctypes.c_int),
|
||||
"archive_read_set_format_option": (
|
||||
[ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_add_passphrase": (
|
||||
[ctypes.c_void_p, ctypes.c_char_p],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_open_memory": (
|
||||
[ctypes.c_void_p, ctypes.c_void_p, ctypes.c_size_t],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_open_filename": (
|
||||
[ctypes.c_void_p, ctypes.c_char_p, ctypes.c_size_t],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_next_header": (
|
||||
[ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_data": (
|
||||
[ctypes.c_void_p, ctypes.c_void_p, ctypes.c_size_t],
|
||||
ctypes.c_ssize_t,
|
||||
),
|
||||
"archive_read_free": ([ctypes.c_void_p], ctypes.c_int),
|
||||
"archive_entry_pathname_utf8": ([ctypes.c_void_p], ctypes.c_char_p),
|
||||
"archive_entry_size": ([ctypes.c_void_p], ctypes.c_int64),
|
||||
"archive_entry_filetype": ([ctypes.c_void_p], ctypes.c_uint),
|
||||
}
|
||||
for symbol, (arguments, result) in signatures.items():
|
||||
function = getattr(library, symbol)
|
||||
function.argtypes = arguments
|
||||
function.restype = result
|
||||
return library
|
||||
except (OSError, AttributeError):
|
||||
return None
|
||||
|
||||
|
||||
def _require_ok(status: int) -> None:
|
||||
if status != _ARCHIVE_OK:
|
||||
# Do not expose native error strings: they can contain source paths.
|
||||
raise FileStorageError(
|
||||
"Native ZIP decoding failed: incorrect password, corrupt or unsupported archive"
|
||||
)
|
||||
|
||||
|
||||
def read_native_zip_members(
|
||||
library: Any,
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
infos_by_path: Mapping[str, Any],
|
||||
selected_files: set[str],
|
||||
normalize_name: Callable[[str], str],
|
||||
password: str,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
progress: Callable[[str, int, int, int, int], None] | None = None,
|
||||
total_bytes: int = 0,
|
||||
) -> Iterator[tuple[str, bytes]]:
|
||||
# Keep memory input, password and output buffers alive until reader free.
|
||||
source_buffer = None
|
||||
password_bytes = password.encode("utf-8")
|
||||
buffer = ctypes.create_string_buffer(_READ_CHUNK_SIZE)
|
||||
archive = library.archive_read_new()
|
||||
if not archive:
|
||||
raise FileStorageError("Native ZIP reader could not be allocated")
|
||||
total = 0
|
||||
completed = 0
|
||||
seen: set[str] = set()
|
||||
try:
|
||||
_require_ok(library.archive_read_support_filter_none(archive))
|
||||
_require_ok(library.archive_read_support_format_zip(archive))
|
||||
_require_ok(
|
||||
library.archive_read_set_format_option(
|
||||
archive, b"zip", b"hdrcharset", b"UTF-8"
|
||||
)
|
||||
)
|
||||
_require_ok(library.archive_read_add_passphrase(archive, password_bytes))
|
||||
if isinstance(archive_data, bytes):
|
||||
source_buffer = ctypes.create_string_buffer(archive_data)
|
||||
_require_ok(
|
||||
library.archive_read_open_memory(
|
||||
archive, source_buffer, len(archive_data)
|
||||
)
|
||||
)
|
||||
else:
|
||||
_require_ok(
|
||||
library.archive_read_open_filename(
|
||||
archive, os.fsencode(archive_data), _READ_CHUNK_SIZE
|
||||
)
|
||||
)
|
||||
entry = ctypes.c_void_p()
|
||||
while True:
|
||||
status = library.archive_read_next_header(archive, ctypes.byref(entry))
|
||||
if status == _ARCHIVE_EOF:
|
||||
break
|
||||
_require_ok(status)
|
||||
raw_path = library.archive_entry_pathname_utf8(entry)
|
||||
if not raw_path:
|
||||
raise FileStorageError("Native ZIP entry has no path")
|
||||
try:
|
||||
path = normalize_name(raw_path.decode("utf-8"))
|
||||
except UnicodeDecodeError as exc:
|
||||
raise FileStorageError(
|
||||
"Native ZIP path does not match the inspected directory"
|
||||
) from exc
|
||||
info = infos_by_path.get(path)
|
||||
if info is None or path in seen:
|
||||
raise FileStorageError(
|
||||
"Native ZIP path does not match the inspected directory"
|
||||
)
|
||||
seen.add(path)
|
||||
directory = info.is_dir()
|
||||
file_type = library.archive_entry_filetype(entry)
|
||||
if not (stat.S_ISDIR(file_type) if directory else stat.S_ISREG(file_type)):
|
||||
raise FileStorageError(
|
||||
"Native ZIP entry type does not match the inspected directory"
|
||||
)
|
||||
if library.archive_entry_size(entry) != info.file_size:
|
||||
raise FileStorageError(
|
||||
"Native ZIP entry size does not match the inspected directory"
|
||||
)
|
||||
if directory or path not in selected_files:
|
||||
continue
|
||||
parts = []
|
||||
size = 0
|
||||
checksum = 0
|
||||
while True:
|
||||
read_size = min(
|
||||
_READ_CHUNK_SIZE,
|
||||
max_file_bytes + 1 - size,
|
||||
max_total_bytes + 1 - total,
|
||||
)
|
||||
if read_size <= 0:
|
||||
raise FileStorageError("Archive exceeds its extraction limits")
|
||||
count = library.archive_read_data(archive, buffer, read_size)
|
||||
if count < 0:
|
||||
_require_ok(count)
|
||||
if count == 0:
|
||||
break
|
||||
if count > read_size:
|
||||
raise FileStorageError("Native ZIP reader returned an invalid byte count")
|
||||
size += count
|
||||
total += count
|
||||
if size > max_file_bytes:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} exceeds per-file limit"
|
||||
)
|
||||
if total > max_total_bytes:
|
||||
raise FileStorageError("Archive is too large after extraction")
|
||||
chunk = ctypes.string_at(buffer, count)
|
||||
checksum = zlib.crc32(chunk, checksum)
|
||||
parts.append(chunk)
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
# Independent checks preserve the Python ZIP reader's CRC and size
|
||||
# guarantees even when native decoding behavior changes upstream.
|
||||
if size != info.file_size or checksum != info.CRC:
|
||||
raise FileStorageError(
|
||||
"Native ZIP member failed size or CRC verification"
|
||||
)
|
||||
completed += 1
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
yield path, b"".join(parts)
|
||||
parts.clear()
|
||||
if seen != set(infos_by_path) or completed != len(selected_files):
|
||||
raise FileStorageError(
|
||||
"Native ZIP directory does not match the inspected archive"
|
||||
)
|
||||
finally:
|
||||
library.archive_read_free(archive)
|
||||
@@ -1,12 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
SOURCE_PROVENANCE_METADATA_KEY = "source_provenance"
|
||||
SOURCE_REVISION_METADATA_KEY = "source_revision"
|
||||
|
||||
|
||||
def source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||
"""The established source identity; neither filename nor content identity."""
|
||||
if not provenance:
|
||||
return None
|
||||
def clean(value: object) -> str | None:
|
||||
return str(value).strip() or None if value is not None else None
|
||||
connector_id = clean(provenance.get("connector_id"))
|
||||
provider = clean(provenance.get("provider"))
|
||||
external_id = clean(provenance.get("external_id"))
|
||||
if connector_id and external_id:
|
||||
return ("external_id", connector_id, provider, external_id)
|
||||
external_path = clean(provenance.get("external_path"))
|
||||
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||
library_id = clean(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||
if connector_id and external_path:
|
||||
return ("external_path", connector_id, provider, library_id, external_path)
|
||||
return None
|
||||
|
||||
|
||||
def source_identity_hash(provenance: dict[str, Any] | None) -> str | None:
|
||||
identity = source_identity(provenance)
|
||||
if identity is None:
|
||||
return None
|
||||
return hashlib.sha256(json.dumps(identity, ensure_ascii=True, separators=(",", ":")).encode("ascii")).hexdigest()
|
||||
|
||||
_PROVENANCE_STRING_FIELDS = {
|
||||
"source_type",
|
||||
"connector_id",
|
||||
|
||||
@@ -1,26 +1,39 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
from threading import Lock
|
||||
from typing import Protocol
|
||||
|
||||
from sqlalchemy import event
|
||||
from sqlalchemy import event, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryOperation,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
plan_recovery_operation,
|
||||
prepare_recovery_operation,
|
||||
start_recovery_operation,
|
||||
verify_recovery_evidence_chain,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
DurableRecoveryOperation,
|
||||
DurableRecoveryStart,
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.core.runtime_coordination import (
|
||||
RuntimeIdentity,
|
||||
acquire_lease,
|
||||
process_runtime_identity,
|
||||
release_lease,
|
||||
)
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileBlob,
|
||||
@@ -38,6 +51,8 @@ from govoplan_files.backend.storage.common import FileStorageError
|
||||
_PENDING_EFFECTS_KEY = "govoplan_files_pending_recovery_effects"
|
||||
_HOOKS_INSTALLED_KEY = "govoplan_files_recovery_hooks_installed"
|
||||
_ROLLBACK_ERRORS_KEY = "govoplan_files_recovery_rollback_errors"
|
||||
_SQLITE_FENCES: set[str] = set()
|
||||
_SQLITE_FENCES_LOCK = Lock()
|
||||
|
||||
|
||||
class _PendingEffect(Protocol):
|
||||
@@ -58,6 +73,8 @@ class PendingBlobWrite:
|
||||
expected_storage_checksum_sha256: str | None = None
|
||||
expected_storage_size_bytes: int | None = None
|
||||
expected_envelope_id: str | None = None
|
||||
sqlite_fence_key: str | None = None
|
||||
restart_after_rollback: Callable[[], DurableRecoveryOperation] | None = None
|
||||
|
||||
def prepare_stored_bytes(
|
||||
self,
|
||||
@@ -72,6 +89,19 @@ class PendingBlobWrite:
|
||||
self.expected_envelope_id = envelope_id
|
||||
|
||||
def settle(self, *, committed: bool) -> None:
|
||||
if not committed and self.restart_after_rollback is not None:
|
||||
try:
|
||||
self.operation = self.restart_after_rollback()
|
||||
except Exception:
|
||||
# SQLite cannot make the pre-effect ledger row independent of
|
||||
# the caller transaction. If reconstructing the evidence row
|
||||
# also fails, still avoid leaving a known new orphan behind.
|
||||
if self.created_new:
|
||||
try:
|
||||
self.backend.delete(self.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
pass
|
||||
raise
|
||||
evidence = _blob_write_evidence(self)
|
||||
if _blob_write_complete(evidence):
|
||||
self.operation.succeed(evidence=evidence)
|
||||
@@ -219,77 +249,145 @@ def begin_blob_write_recovery(
|
||||
f"files-blob-{disposition}:{blob_id}:{state_token[:48]}"
|
||||
)
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
identity = process_runtime_identity()
|
||||
except RuntimeError as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no object was written"
|
||||
) from exc
|
||||
lease_resource_key = f"files:blob:{tenant_id}:{blob_id}"
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"blob_id": blob_id,
|
||||
"storage_key": storage_key if created_new else None,
|
||||
"storage_key_sha256": key_digest,
|
||||
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||
"semantic_size_bytes": semantic_size_bytes,
|
||||
"protection_discriminator": protection_discriminator,
|
||||
"disposition": disposition,
|
||||
}
|
||||
recovery_plan = RecoveryPlan(
|
||||
mode=(
|
||||
RecoveryMode.COMPENSATION
|
||||
if created_new
|
||||
else RecoveryMode.FORWARD_RECOVERY
|
||||
),
|
||||
preconditions=(
|
||||
"the caller has Files write authority for the target owner",
|
||||
"the storage key belongs to the tenant Files namespace",
|
||||
"the request records digests rather than file contents",
|
||||
),
|
||||
compensation_steps=(
|
||||
"verify that no FileBlob references the newly reserved key",
|
||||
"delete only that unreferenced key and verify absence",
|
||||
)
|
||||
if created_new
|
||||
else (),
|
||||
forward_recovery_steps=(
|
||||
"verify the expected bytes at the existing blob key",
|
||||
"forward-complete matching integrity metadata or quarantine the blob",
|
||||
)
|
||||
if not created_new
|
||||
else (),
|
||||
verification_steps=(
|
||||
"reload FileBlob metadata through an independent session",
|
||||
"stream and hash the managed object independently",
|
||||
),
|
||||
)
|
||||
precondition_evidence = {
|
||||
"blob_id": blob_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||
"semantic_size_bytes": semantic_size_bytes,
|
||||
"created_new": created_new,
|
||||
}
|
||||
metadata = {
|
||||
"resources": ["postgresql", "object-storage"],
|
||||
"storage_backend": backend.name,
|
||||
"durability_mode": "independent_transaction",
|
||||
}
|
||||
session_factory = get_database().SessionLocal
|
||||
sqlite_mode = session.get_bind().dialect.name == "sqlite"
|
||||
sqlite_fence_key: str | None = None
|
||||
|
||||
def start_independent(*, reconstructed_after_rollback: bool = False) -> DurableRecoveryStart:
|
||||
return begin_durable_recovery_operation(
|
||||
session_factory,
|
||||
identity=identity,
|
||||
module_id="files",
|
||||
operation_type=f"blob-{disposition}",
|
||||
idempotency_key=idempotency_key,
|
||||
request={
|
||||
"tenant_id": tenant_id,
|
||||
"blob_id": blob_id,
|
||||
"storage_key": storage_key if created_new else None,
|
||||
"storage_key_sha256": key_digest,
|
||||
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||
"semantic_size_bytes": semantic_size_bytes,
|
||||
"protection_discriminator": protection_discriminator,
|
||||
"disposition": disposition,
|
||||
},
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=(
|
||||
RecoveryMode.COMPENSATION
|
||||
if created_new
|
||||
else RecoveryMode.FORWARD_RECOVERY
|
||||
),
|
||||
preconditions=(
|
||||
"the caller has Files write authority for the target owner",
|
||||
"the storage key belongs to the tenant Files namespace",
|
||||
"the request records digests rather than file contents",
|
||||
),
|
||||
compensation_steps=(
|
||||
"verify that no FileBlob references the newly reserved key",
|
||||
"delete only that unreferenced key and verify absence",
|
||||
)
|
||||
if created_new
|
||||
else (),
|
||||
forward_recovery_steps=(
|
||||
"verify the expected bytes at the existing blob key",
|
||||
"forward-complete matching integrity metadata or quarantine the blob",
|
||||
)
|
||||
if not created_new
|
||||
else (),
|
||||
verification_steps=(
|
||||
"reload FileBlob metadata through an independent session",
|
||||
"stream and hash the managed object independently",
|
||||
),
|
||||
),
|
||||
request=request,
|
||||
recovery_plan=recovery_plan,
|
||||
precondition_evidence={
|
||||
"blob_id": blob_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||
"semantic_size_bytes": semantic_size_bytes,
|
||||
"created_new": created_new,
|
||||
**precondition_evidence,
|
||||
**(
|
||||
{
|
||||
"sqlite_caller_transaction_rolled_back": True,
|
||||
"effect_may_have_preceded_durable_intent": True,
|
||||
}
|
||||
if reconstructed_after_rollback
|
||||
else {}
|
||||
),
|
||||
},
|
||||
lease_resource_key=(
|
||||
f"files:blob:{tenant_id}:{blob_id}"
|
||||
),
|
||||
lease_resource_key=lease_resource_key,
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
metadata={
|
||||
"resources": ["postgresql", "object-storage"],
|
||||
"storage_backend": backend.name,
|
||||
**metadata,
|
||||
"durability_mode": (
|
||||
"sqlite_post_rollback_reconstruction"
|
||||
if reconstructed_after_rollback
|
||||
else "independent_transaction"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
if sqlite_mode:
|
||||
_reserve_sqlite_fence(lease_resource_key)
|
||||
sqlite_fence_key = lease_resource_key
|
||||
started = _begin_caller_transaction_recovery_operation(
|
||||
session,
|
||||
session_factory=session_factory,
|
||||
identity=identity,
|
||||
module_id="files",
|
||||
operation_type=f"blob-{disposition}",
|
||||
idempotency_key=idempotency_key,
|
||||
request=request,
|
||||
recovery_plan=recovery_plan,
|
||||
precondition_evidence={
|
||||
**precondition_evidence,
|
||||
"sqlite_caller_transaction": True,
|
||||
"reduced_crash_durability": True,
|
||||
},
|
||||
lease_resource_key=lease_resource_key,
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
metadata={
|
||||
**metadata,
|
||||
"resources": ["sqlite", "object-storage"],
|
||||
"durability_mode": "sqlite_caller_transaction",
|
||||
},
|
||||
)
|
||||
else:
|
||||
started = start_independent()
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
if sqlite_fence_key is not None:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
raise FileStorageError(
|
||||
"This managed blob is already owned by another recovery operation"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||
if sqlite_fence_key is not None:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no object was written"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
if sqlite_fence_key is not None:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
raise FileStorageError(
|
||||
"The matching Files blob operation was already completed; reload before retrying"
|
||||
)
|
||||
@@ -303,6 +401,14 @@ def begin_blob_write_recovery(
|
||||
semantic_size_bytes=semantic_size_bytes,
|
||||
protection_discriminator=protection_discriminator,
|
||||
created_new=created_new,
|
||||
sqlite_fence_key=sqlite_fence_key,
|
||||
restart_after_rollback=(
|
||||
lambda: _require_started_operation(
|
||||
start_independent(reconstructed_after_rollback=True)
|
||||
)
|
||||
if sqlite_mode
|
||||
else None
|
||||
),
|
||||
)
|
||||
_register_pending_effect(session, pending)
|
||||
return pending
|
||||
@@ -398,10 +504,17 @@ def _register_pending_effect(session: Session, effect: _PendingEffect) -> None:
|
||||
|
||||
|
||||
def _after_session_commit(session: Session) -> None:
|
||||
# SQLAlchemy also emits after_commit for a released SAVEPOINT. A Files
|
||||
# batch may open one while creating a later recovery row; settle only when
|
||||
# the outer business transaction has actually become visible.
|
||||
if session.in_nested_transaction():
|
||||
return
|
||||
_settle_pending_effects(session, committed=True)
|
||||
|
||||
|
||||
def _after_session_rollback(session: Session) -> None:
|
||||
if session.in_nested_transaction():
|
||||
return
|
||||
try:
|
||||
_settle_pending_effects(session, committed=False)
|
||||
except RecoveryGuaranteeError as exc:
|
||||
@@ -422,12 +535,139 @@ def _settle_pending_effects(session: Session, *, committed: bool) -> None:
|
||||
operation.release_unresolved()
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
sqlite_fence_key = getattr(effect, "sqlite_fence_key", None)
|
||||
if sqlite_fence_key:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
if failures:
|
||||
raise RecoveryGuaranteeError(
|
||||
f"{len(failures)} Files recovery operation(s) could not be finalized"
|
||||
) from failures[0]
|
||||
|
||||
|
||||
def _reserve_sqlite_fence(resource_key: str) -> None:
|
||||
with _SQLITE_FENCES_LOCK:
|
||||
if resource_key in _SQLITE_FENCES:
|
||||
raise RecoveryOperationBusy(
|
||||
f"Another local SQLite transaction owns {resource_key}"
|
||||
)
|
||||
_SQLITE_FENCES.add(resource_key)
|
||||
|
||||
|
||||
def _release_sqlite_fence(resource_key: str) -> None:
|
||||
with _SQLITE_FENCES_LOCK:
|
||||
_SQLITE_FENCES.discard(resource_key)
|
||||
|
||||
|
||||
def _require_started_operation(started: DurableRecoveryStart) -> DurableRecoveryOperation:
|
||||
if started.replayed or started.operation is None:
|
||||
raise RecoveryOperationStateConflict(started.operation_id, started.status)
|
||||
return started.operation
|
||||
|
||||
|
||||
def _begin_caller_transaction_recovery_operation(
|
||||
session: Session,
|
||||
*,
|
||||
session_factory: Callable[[], Session],
|
||||
identity: RuntimeIdentity,
|
||||
module_id: str,
|
||||
operation_type: str,
|
||||
idempotency_key: str,
|
||||
request: dict[str, object],
|
||||
recovery_plan: RecoveryPlan,
|
||||
precondition_evidence: dict[str, object],
|
||||
lease_resource_key: str,
|
||||
lease_ttl_seconds: int,
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
metadata: dict[str, object],
|
||||
) -> DurableRecoveryStart:
|
||||
"""Start SQLite recovery evidence inside the caller transaction.
|
||||
|
||||
SQLite permits only one writer, so opening the normal independent ledger
|
||||
transaction after an earlier member of a batch has written will deadlock
|
||||
until the busy timeout. The caller-transaction mode preserves fencing,
|
||||
request hashing, checkpoints, commit verification, and rollback
|
||||
compensation, but cannot make the intent survive a hard process loss
|
||||
before the caller commits. PostgreSQL never uses this reduced mode.
|
||||
"""
|
||||
|
||||
claim = acquire_lease(
|
||||
session,
|
||||
installation_id=identity.installation_id,
|
||||
resource_key=lease_resource_key,
|
||||
holder_node_id=identity.node_id,
|
||||
holder_incarnation=identity.incarnation,
|
||||
ttl_seconds=lease_ttl_seconds,
|
||||
metadata={
|
||||
"module_id": module_id,
|
||||
"operation_type": operation_type,
|
||||
"durability_mode": "sqlite_caller_transaction",
|
||||
},
|
||||
)
|
||||
if claim is None:
|
||||
raise RecoveryOperationBusy(
|
||||
f"Another runtime owns the recovery fence for {lease_resource_key}"
|
||||
)
|
||||
existing = session.execute(
|
||||
select(RecoveryOperation).where(
|
||||
RecoveryOperation.installation_id == identity.installation_id,
|
||||
RecoveryOperation.module_id == module_id,
|
||||
RecoveryOperation.idempotency_key == idempotency_key,
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
operation = plan_recovery_operation(
|
||||
session,
|
||||
installation_id=identity.installation_id,
|
||||
module_id=module_id,
|
||||
operation_type=operation_type,
|
||||
idempotency_key=idempotency_key,
|
||||
request=request,
|
||||
recovery_plan=recovery_plan,
|
||||
resource_type=resource_type,
|
||||
resource_id=resource_id,
|
||||
lease_claim=claim,
|
||||
metadata=metadata,
|
||||
)
|
||||
if existing is not None:
|
||||
if operation.status == RecoveryStatus.SUCCEEDED.value:
|
||||
release_lease(session, claim)
|
||||
return DurableRecoveryStart(
|
||||
operation_id=operation.id,
|
||||
status=operation.status,
|
||||
replayed=True,
|
||||
operation=None,
|
||||
)
|
||||
raise RecoveryOperationStateConflict(operation.id, operation.status)
|
||||
prepare_recovery_operation(
|
||||
session,
|
||||
operation,
|
||||
evidence=precondition_evidence,
|
||||
lease_claim=claim,
|
||||
)
|
||||
start_recovery_operation(
|
||||
session,
|
||||
operation,
|
||||
evidence={"lease_resource_key": lease_resource_key},
|
||||
lease_claim=claim,
|
||||
)
|
||||
if not verify_recovery_evidence_chain(session, operation.id):
|
||||
raise RecoveryGuaranteeError(
|
||||
"Recovery checkpoint chain verification failed before side effects"
|
||||
)
|
||||
return DurableRecoveryStart(
|
||||
operation_id=operation.id,
|
||||
status=operation.status,
|
||||
replayed=False,
|
||||
operation=DurableRecoveryOperation(
|
||||
session_factory=session_factory,
|
||||
operation_id=operation.id,
|
||||
lease_claim=claim,
|
||||
lease_ttl_seconds=lease_ttl_seconds,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _blob_write_evidence(effect: PendingBlobWrite) -> dict[str, object]:
|
||||
database_blob_present: bool | None
|
||||
database_matches: bool | None
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Isolated synthetic archive benchmark; never reads the configured Files store.
|
||||
|
||||
Run from this repository: python tests/benchmark_archive_storage.py.
|
||||
Uses the existing temporary SQLite/local-storage recovery fixture so durable
|
||||
ledger, checksum verification, commit settlement and compensation stay enabled.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import cProfile
|
||||
from contextlib import ExitStack
|
||||
from collections import Counter
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
import pstats
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from unittest.mock import patch
|
||||
import zipfile
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy import event
|
||||
|
||||
from govoplan_files.backend.change_tracking import register_files_change_tracking
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
extract_archive_upload,
|
||||
_read_selected_zip_members,
|
||||
)
|
||||
from govoplan_files.backend.storage.native_zip import native_zip_library
|
||||
from test_storage_recovery import StorageRecoveryTests
|
||||
|
||||
|
||||
def storage_probe(
|
||||
member_count: int,
|
||||
member_size: int,
|
||||
profile: bool,
|
||||
baseline_revision: str | None = None,
|
||||
):
|
||||
case = StorageRecoveryTests()
|
||||
case.setUp()
|
||||
try:
|
||||
register_files_change_tracking()
|
||||
data = BytesIO()
|
||||
rng = random.Random(20260907)
|
||||
with zipfile.ZipFile(data, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for index in range(member_count):
|
||||
archive.writestr(
|
||||
f"documents/{index:04d}.dat", rng.randbytes(member_size)
|
||||
)
|
||||
queries = []
|
||||
event.listen(
|
||||
case.engine,
|
||||
"before_cursor_execute",
|
||||
lambda conn, cursor, statement, parameters, context, executemany: (
|
||||
queries.append(statement)
|
||||
),
|
||||
)
|
||||
profiler = cProfile.Profile()
|
||||
if profile:
|
||||
profiler.enable()
|
||||
with ExitStack() as overrides:
|
||||
overrides.enter_context(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=case.backend,
|
||||
)
|
||||
)
|
||||
if baseline_revision:
|
||||
# Read an earlier local revision into this isolated process;
|
||||
# never replace shared worktree files or connect to live data.
|
||||
source = subprocess.run(
|
||||
[
|
||||
"git",
|
||||
"show",
|
||||
f"{baseline_revision}:src/govoplan_files/backend/storage/files.py",
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout
|
||||
namespace = {"__name__": "govoplan_files.backend.storage.files"}
|
||||
exec(
|
||||
compile(source, f"{baseline_revision}:files.py", "exec"), namespace
|
||||
)
|
||||
namespace.update(
|
||||
get_storage_backend=lambda: case.backend,
|
||||
_storage_backend_name=lambda: case.backend.name,
|
||||
_storage_bucket_name=lambda: "",
|
||||
)
|
||||
overrides.enter_context(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset",
|
||||
namespace["create_file_asset"],
|
||||
)
|
||||
)
|
||||
started = time.perf_counter()
|
||||
result = extract_archive_upload(
|
||||
case.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=data.getvalue(),
|
||||
filename="synthetic.zip",
|
||||
folder="imported",
|
||||
campaign_id=None,
|
||||
)
|
||||
storage_done = time.perf_counter()
|
||||
storage_queries = len(queries)
|
||||
case.session.commit()
|
||||
completed = time.perf_counter()
|
||||
if profile:
|
||||
profiler.disable()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"probe": "durable-storage",
|
||||
"storage_revision": baseline_revision or "working-tree",
|
||||
"members": len(result),
|
||||
"member_bytes": member_size,
|
||||
"extract_store_seconds": round(storage_done - started, 4),
|
||||
"settlement_seconds": round(completed - storage_done, 4),
|
||||
"total_seconds": round(completed - started, 4),
|
||||
"store_queries": storage_queries,
|
||||
"total_queries": len(queries),
|
||||
"queries_by_table": dict(
|
||||
Counter(
|
||||
(
|
||||
match.group(1)
|
||||
if (
|
||||
match := re.search(
|
||||
r"(?:FROM|INTO|UPDATE)\s+([a-z_]+)", statement
|
||||
)
|
||||
)
|
||||
else "other"
|
||||
)
|
||||
for statement in queries
|
||||
)
|
||||
),
|
||||
}
|
||||
)
|
||||
)
|
||||
if profile:
|
||||
pstats.Stats(profiler).strip_dirs().sort_stats("cumulative").print_stats(24)
|
||||
finally:
|
||||
case.doCleanups()
|
||||
|
||||
|
||||
def zipcrypto_probe(size: int):
|
||||
binary = shutil.which("zip")
|
||||
if not binary:
|
||||
print(
|
||||
json.dumps(
|
||||
{"probe": "zipcrypto", "skipped": "fixture ZIP writer unavailable"}
|
||||
)
|
||||
)
|
||||
return
|
||||
# Public fixture password only: never pass a user credential to a process.
|
||||
with tempfile.TemporaryDirectory(prefix="files-synthetic-zipcrypto-") as temporary:
|
||||
path = Path(temporary)
|
||||
content = random.Random(20260907).randbytes(size)
|
||||
(path / "synthetic.bin").write_bytes(content)
|
||||
subprocess.run(
|
||||
[binary, "-q", "-P", "fixture-only", "synthetic.zip", "synthetic.bin"],
|
||||
cwd=path,
|
||||
check=True,
|
||||
)
|
||||
for implementation in (pyzipper.AESZipFile, zipfile.ZipFile):
|
||||
started = time.perf_counter()
|
||||
with implementation(path / "synthetic.zip") as archive:
|
||||
result = archive.read("synthetic.bin", pwd=b"fixture-only")
|
||||
assert result == content
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"probe": "zipcrypto",
|
||||
"reader": implementation.__module__,
|
||||
"bytes": size,
|
||||
"seconds": round(time.perf_counter() - started, 4),
|
||||
}
|
||||
)
|
||||
)
|
||||
if native_zip_library() is not None:
|
||||
started = time.perf_counter()
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
path / "synthetic.zip",
|
||||
selected_files={"synthetic.bin"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=size,
|
||||
max_total_bytes=size,
|
||||
)
|
||||
)
|
||||
assert result == [("synthetic.bin", content)]
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"probe": "zipcrypto",
|
||||
"reader": "libarchive-with-independent-crc",
|
||||
"bytes": size,
|
||||
"seconds": round(time.perf_counter() - started, 4),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--members", type=int, default=100)
|
||||
parser.add_argument("--member-size", type=int, default=4096)
|
||||
parser.add_argument("--zipcrypto-bytes", type=int, default=4 * 1024 * 1024)
|
||||
parser.add_argument("--profile", action="store_true")
|
||||
parser.add_argument(
|
||||
"--baseline-storage-revision",
|
||||
help="Read a trusted earlier local Git revision for isolated storage comparison, without changing the worktree",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
storage_probe(
|
||||
args.members, args.member_size, args.profile, args.baseline_storage_revision
|
||||
)
|
||||
if args.zipcrypto_bytes:
|
||||
zipcrypto_probe(args.zipcrypto_bytes)
|
||||
@@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
@@ -12,7 +13,11 @@ from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_files.backend.capabilities import FilesAccessService, virtual_folder_resource_id
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_files.backend.storage.files import count_assets_for_user, list_assets_for_user
|
||||
from govoplan_files.backend.storage.files import (
|
||||
count_assets_for_user,
|
||||
list_assets_for_user,
|
||||
list_recent_assets_for_user,
|
||||
)
|
||||
|
||||
|
||||
TENANT_ID = "tenant-1"
|
||||
@@ -22,6 +27,54 @@ GROUP_ID = "group-1"
|
||||
|
||||
|
||||
class FilesAccessProviderTests(unittest.TestCase):
|
||||
def test_recent_projection_is_bounded_ordered_and_access_filtered(self) -> None:
|
||||
session = _session()
|
||||
self.addCleanup(_close_session, session)
|
||||
_seed_access_subjects(session)
|
||||
now = datetime.now(timezone.utc)
|
||||
session.add_all(
|
||||
[
|
||||
FileAsset(
|
||||
id="owned-older",
|
||||
tenant_id=TENANT_ID,
|
||||
owner_type="user",
|
||||
owner_user_id=USER_ID,
|
||||
display_path="older.pdf",
|
||||
filename="older.pdf",
|
||||
updated_at=now - timedelta(hours=2),
|
||||
),
|
||||
FileAsset(
|
||||
id="owned-newer",
|
||||
tenant_id=TENANT_ID,
|
||||
owner_type="user",
|
||||
owner_user_id=USER_ID,
|
||||
display_path="newer.pdf",
|
||||
filename="newer.pdf",
|
||||
updated_at=now - timedelta(hours=1),
|
||||
),
|
||||
FileAsset(
|
||||
id="other-newest",
|
||||
tenant_id=TENANT_ID,
|
||||
owner_type="user",
|
||||
owner_user_id=OTHER_USER_ID,
|
||||
display_path="private.pdf",
|
||||
filename="private.pdf",
|
||||
updated_at=now,
|
||||
),
|
||||
]
|
||||
)
|
||||
session.commit()
|
||||
|
||||
with patch("govoplan_files.backend.storage.files.user_group_ids", return_value=[]):
|
||||
recent = list_recent_assets_for_user(
|
||||
session,
|
||||
tenant_id=TENANT_ID,
|
||||
user_id=USER_ID,
|
||||
limit=1,
|
||||
)
|
||||
|
||||
self.assertEqual(["owned-newer"], [asset.id for asset in recent])
|
||||
|
||||
def test_file_access_provider_explains_owner_share_admin_and_missing_resources(self) -> None:
|
||||
session = _session()
|
||||
self.addCleanup(_close_session, session)
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tarfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_files.backend.storage.archives import _inspect_archive_content, _safe_member_path, inspect_archive
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from test_archives import _tar_bytes, _zip_bytes
|
||||
|
||||
|
||||
class _HeadersOnly:
|
||||
def __init__(self, count: int, size: int = 0):
|
||||
self.count = count
|
||||
self.size = size
|
||||
self.seen = 0
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def getmembers(self):
|
||||
raise AssertionError("TAR preview must not inflate every member before checking limits")
|
||||
|
||||
def __iter__(self):
|
||||
for index in range(self.count):
|
||||
member = tarfile.TarInfo(f"file-{index}.txt")
|
||||
member.size = self.size
|
||||
self.seen += 1
|
||||
yield member
|
||||
raise AssertionError("Inspection advanced beyond the rejecting header")
|
||||
|
||||
|
||||
class ArchiveInspectionBoundTests(unittest.TestCase):
|
||||
def test_tar_entry_limit_stops_at_first_excess_header(self):
|
||||
headers = _HeadersOnly(4)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", return_value=headers):
|
||||
with self.assertRaisesRegex(FileStorageError, "too many entries"):
|
||||
_inspect_archive_content(b"fixture", filename="fixture.tar.gz", max_entries=2)
|
||||
self.assertEqual(3, headers.seen)
|
||||
|
||||
def test_tar_expanded_size_rejected_before_payload_decompression(self):
|
||||
headers = _HeadersOnly(1, size=100)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", return_value=headers):
|
||||
with self.assertRaisesRegex(FileStorageError, "too large after extraction"):
|
||||
_inspect_archive_content(b"fixture", filename="fixture.tar.gz", max_expanded_bytes=10)
|
||||
self.assertEqual(1, headers.seen)
|
||||
|
||||
def test_tar_ratio_rejected_before_payload_decompression(self):
|
||||
headers = _HeadersOnly(1, size=100)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", return_value=headers):
|
||||
with self.assertRaisesRegex(FileStorageError, "expansion ratio"):
|
||||
_inspect_archive_content(b"fixture", filename="fixture.tar.gz", max_expansion_ratio=2)
|
||||
|
||||
def test_derived_directories_are_included_in_entry_limit(self):
|
||||
for filename, payload in (("fixture.zip", _zip_bytes({"a/b/c/file.txt": b"x"})), ("fixture.tar.gz", _tar_bytes({"a/b/c/file.txt": b"x"}))):
|
||||
with self.subTest(filename=filename), self.assertRaisesRegex(FileStorageError, "including parent directories"):
|
||||
inspect_archive(payload, filename=filename, max_entries=3)
|
||||
|
||||
def test_extreme_path_depth_and_byte_lengths_fail_before_deriving_directories(self):
|
||||
for path, error in (("a/" * 128 + "file.txt", "components"), ("ü" * 2049, "UTF-8 bytes")):
|
||||
with self.subTest(path_length=len(path)), self.assertRaisesRegex(FileStorageError, error):
|
||||
inspect_archive(_zip_bytes({path: b"x"}), filename="fixture.zip")
|
||||
|
||||
def test_directory_at_exact_depth_limit_accepts_trailing_separator(self):
|
||||
path = "a/" * 128
|
||||
self.assertEqual(path.rstrip("/"), _safe_member_path(path))
|
||||
|
||||
def test_invalid_unicode_member_name_has_controlled_validation_error(self):
|
||||
with self.assertRaisesRegex(FileStorageError, "not valid Unicode"):
|
||||
_safe_member_path("invalid-\udcff.txt")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,445 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import closing
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
import random
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy import event
|
||||
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_files.backend.change_tracking import register_files_change_tracking
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
_read_selected_zip_members,
|
||||
_safe_member_path,
|
||||
extract_archive_upload,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
archive_storage_backend_scope,
|
||||
_archive_write_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.native_zip import (
|
||||
native_zip_library,
|
||||
read_native_zip_members,
|
||||
)
|
||||
from test_archives import _zip_bytes, _tar_bytes, _encrypted_zip_bytes
|
||||
import test_storage_recovery as recovery_fixture
|
||||
|
||||
|
||||
def legacy_archive(entries: dict[str, bytes]) -> bytes:
|
||||
# Fixture-only known password, never an application secret.
|
||||
with tempfile.TemporaryDirectory(prefix="files-legacy-fixture-") as temporary:
|
||||
root = Path(temporary)
|
||||
for name, content in entries.items():
|
||||
path = root / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(content)
|
||||
subprocess.run(
|
||||
[shutil.which("zip"), "-q", "-P", "fixture-only", "fixture.zip", *entries],
|
||||
cwd=root,
|
||||
check=True,
|
||||
)
|
||||
return (root / "fixture.zip").read_bytes()
|
||||
|
||||
|
||||
@unittest.skipUnless(
|
||||
shutil.which("zip") and native_zip_library() is not None,
|
||||
"optional libarchive and synthetic ZIP writer required",
|
||||
)
|
||||
class NativeArchivePerformanceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.contents = {
|
||||
"first.dat": random.Random(123).randbytes(8192),
|
||||
"folder/second.txt": b"second",
|
||||
}
|
||||
self.archive = legacy_archive(self.contents)
|
||||
|
||||
def read(self, **kwargs):
|
||||
return list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=100_000,
|
||||
max_total_bytes=100_000,
|
||||
**kwargs,
|
||||
)
|
||||
)
|
||||
|
||||
def test_native_selection_and_missing_library_python_fallback_match(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives._read_member",
|
||||
side_effect=AssertionError("Python decoder must not run"),
|
||||
):
|
||||
native = self.read()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
return_value=None,
|
||||
):
|
||||
fallback = self.read()
|
||||
self.assertEqual(native, fallback)
|
||||
self.assertEqual([("first.dat", self.contents["first.dat"])], native)
|
||||
|
||||
def test_utf8_flagged_german_names_use_native_reader(self):
|
||||
# Info-ZIP stores UTF-8 names without the language flag on some hosts.
|
||||
# Set the standard UTF-8 flag in local and central headers for this
|
||||
# fixture; encryption header verification is unaffected by bit 11.
|
||||
encoded_name = "Grüße.txt".encode("utf-8")
|
||||
placeholder = "x" * len(encoded_name)
|
||||
data = bytearray(
|
||||
legacy_archive({placeholder: b"german"}).replace(
|
||||
placeholder.encode(), encoded_name
|
||||
)
|
||||
)
|
||||
for signature, offset in ((b"PK\x03\x04", 6), (b"PK\x01\x02", 8)):
|
||||
position = data.index(signature) + offset
|
||||
flags = int.from_bytes(data[position : position + 2], "little") | 0x800
|
||||
data[position : position + 2] = flags.to_bytes(2, "little")
|
||||
self.archive = bytes(data)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives._read_member",
|
||||
side_effect=AssertionError("Python decoder must not run"),
|
||||
):
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"Grüße.txt"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=100,
|
||||
max_total_bytes=100,
|
||||
)
|
||||
)
|
||||
self.assertEqual([("Grüße.txt", b"german")], result)
|
||||
|
||||
def test_wrong_password_and_corruption_never_fall_back(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives._read_member",
|
||||
side_effect=AssertionError("No fallback after native error"),
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "Native ZIP decoding failed"):
|
||||
list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="wrong",
|
||||
max_file_bytes=100_000,
|
||||
max_total_bytes=100_000,
|
||||
)
|
||||
)
|
||||
corrupted = bytearray(self.archive)
|
||||
with pyzipper.AESZipFile(BytesIO(self.archive)) as archive:
|
||||
first = archive.infolist()[0]
|
||||
payload_start = (
|
||||
first.header_offset
|
||||
+ 30
|
||||
+ len(first.filename.encode())
|
||||
+ len(first.extra)
|
||||
)
|
||||
corrupted[payload_start + first.compress_size - 1] ^= 0x40
|
||||
self.archive = bytes(corrupted)
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.read()
|
||||
|
||||
def test_independent_crc_check_rejects_mismatched_metadata(self):
|
||||
with pyzipper.AESZipFile(BytesIO(self.archive)) as archive:
|
||||
infos = {
|
||||
_safe_member_path(info.filename): info for info in archive.infolist()
|
||||
}
|
||||
infos["first.dat"].CRC ^= 1
|
||||
with self.assertRaisesRegex(FileStorageError, "CRC verification"):
|
||||
list(
|
||||
read_native_zip_members(
|
||||
native_zip_library(),
|
||||
self.archive,
|
||||
infos_by_path=infos,
|
||||
selected_files={"first.dat"},
|
||||
normalize_name=_safe_member_path,
|
||||
password="fixture-only",
|
||||
max_file_bytes=100_000,
|
||||
max_total_bytes=100_000,
|
||||
)
|
||||
)
|
||||
|
||||
def test_legacy_encoded_non_utf8_names_choose_python_before_decoding(self):
|
||||
encoded_name = "ä.txt".encode("cp437")
|
||||
placeholder = "x" * len(encoded_name)
|
||||
self.archive = legacy_archive({placeholder: b"legacy"}).replace(
|
||||
placeholder.encode(), encoded_name
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
side_effect=AssertionError("Legacy encoding must use original reader"),
|
||||
):
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"ä.txt"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=100,
|
||||
max_total_bytes=100,
|
||||
)
|
||||
)
|
||||
self.assertEqual([("ä.txt", b"legacy")], result)
|
||||
|
||||
def test_suspended_native_reader_is_freed_after_destination_failure(self):
|
||||
library = native_zip_library()
|
||||
|
||||
class LibraryProxy:
|
||||
def __getattr__(self, name):
|
||||
return getattr(library, name)
|
||||
|
||||
proxy = LibraryProxy()
|
||||
with (
|
||||
patch.object(
|
||||
proxy, "archive_read_free", wraps=library.archive_read_free
|
||||
) as close,
|
||||
patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
return_value=proxy,
|
||||
),
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "Destination failure"):
|
||||
# This unit test owns the native handle locally; public imports
|
||||
# now own it in a fresh child (covered by worker cleanup tests).
|
||||
with closing(_read_selected_zip_members(
|
||||
self.archive, selected_files=set(self.contents), password="fixture-only",
|
||||
max_file_bytes=100_000, max_total_bytes=100_000,
|
||||
)) as members:
|
||||
next(members)
|
||||
raise FileStorageError("Destination failure")
|
||||
close.assert_called_once()
|
||||
|
||||
def test_native_read_enforces_actual_member_and_total_limits(self):
|
||||
for member_limit, total_limit in ((10, 100_000), (100_000, 10)):
|
||||
with self.subTest(member_limit=member_limit, total_limit=total_limit):
|
||||
with self.assertRaises(FileStorageError):
|
||||
list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=member_limit,
|
||||
max_total_bytes=total_limit,
|
||||
)
|
||||
)
|
||||
|
||||
def test_native_emits_progress_inside_a_large_member(self):
|
||||
self.archive = legacy_archive(
|
||||
{"first.dat": random.Random(234).randbytes(3 * 1024 * 1024)}
|
||||
)
|
||||
progress = []
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=4 * 1024 * 1024,
|
||||
max_total_bytes=4 * 1024 * 1024,
|
||||
total_bytes=3 * 1024 * 1024,
|
||||
progress=lambda *args: progress.append(args),
|
||||
)
|
||||
)
|
||||
self.assertEqual(3 * 1024 * 1024, len(result[0][1]))
|
||||
self.assertTrue(
|
||||
any(item[1] == 0 and 0 < item[3] < 3 * 1024 * 1024 for item in progress)
|
||||
)
|
||||
self.assertEqual(
|
||||
("extracting", 1, 1, 3 * 1024 * 1024, 3 * 1024 * 1024), progress[-1]
|
||||
)
|
||||
|
||||
|
||||
class ArchiveStreamingTests(unittest.TestCase):
|
||||
def test_plain_zip_and_tar_store_one_member_before_reading_the_next(self):
|
||||
entries = {"one.txt": b"one", "two.txt": b"two"}
|
||||
for filename, archive in (
|
||||
("files.zip", _zip_bytes(entries)),
|
||||
("files.tar.gz", _tar_bytes(entries)),
|
||||
):
|
||||
with self.subTest(filename=filename):
|
||||
progress = []
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset",
|
||||
return_value=SimpleNamespace(asset=object()),
|
||||
) as create:
|
||||
result = extract_archive_upload(
|
||||
object(),
|
||||
tenant_id="tenant",
|
||||
owner_type="user",
|
||||
owner_id="user",
|
||||
user_id="user",
|
||||
archive_data=archive,
|
||||
filename=filename,
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
progress=lambda *args: progress.append(args),
|
||||
)
|
||||
self.assertEqual(2, len(result))
|
||||
self.assertEqual(2, create.call_count)
|
||||
stored_first = progress.index(("storing", 1, 2, 3, 6))
|
||||
extracted_second = progress.index(("extracting", 2, 2, 6, 6))
|
||||
self.assertLess(stored_first, extracted_second)
|
||||
|
||||
def test_aes_archive_keeps_existing_decoder(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
side_effect=AssertionError("AES must not use native path"),
|
||||
):
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
_encrypted_zip_bytes("correct"),
|
||||
selected_files={"secure/report.txt"},
|
||||
password="correct",
|
||||
max_file_bytes=100,
|
||||
max_total_bytes=100,
|
||||
)
|
||||
)
|
||||
self.assertEqual([("secure/report.txt", b"classified")], result)
|
||||
|
||||
def test_late_unsafe_header_prevents_every_write(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset"
|
||||
) as create:
|
||||
with self.assertRaisesRegex(FileStorageError, "Unsafe archive member"):
|
||||
extract_archive_upload(
|
||||
object(),
|
||||
tenant_id="tenant",
|
||||
owner_type="user",
|
||||
owner_id="user",
|
||||
user_id="user",
|
||||
archive_data=_zip_bytes({"good.txt": b"good", "../unsafe": b"bad"}),
|
||||
filename="unsafe.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
)
|
||||
create.assert_not_called()
|
||||
|
||||
|
||||
class ArchivePersistencePerformanceTests(unittest.TestCase):
|
||||
def test_backend_scope_is_lazy_nested_and_reset_after_exception(self):
|
||||
first_backend = object()
|
||||
second_backend = object()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
side_effect=[first_backend, second_backend],
|
||||
) as factory:
|
||||
with self.assertRaisesRegex(RuntimeError, "fixture failure"):
|
||||
with archive_storage_backend_scope():
|
||||
factory.assert_not_called()
|
||||
self.assertIs(first_backend, _archive_write_backend())
|
||||
with archive_storage_backend_scope():
|
||||
self.assertIs(first_backend, _archive_write_backend())
|
||||
factory.assert_called_once()
|
||||
raise RuntimeError("fixture failure")
|
||||
with archive_storage_backend_scope():
|
||||
self.assertIs(second_backend, _archive_write_backend())
|
||||
self.assertEqual(2, factory.call_count)
|
||||
|
||||
def test_backend_reuse_does_not_cache_tenant_identity_or_blob_dedup(self):
|
||||
fixture = recovery_fixture.StorageRecoveryTests()
|
||||
fixture.setUp()
|
||||
self.addCleanup(fixture.doCleanups)
|
||||
results = []
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=fixture.backend,
|
||||
) as factory:
|
||||
for tenant in ("tenant-one", "tenant-two"):
|
||||
results.append(
|
||||
extract_archive_upload(
|
||||
fixture.session,
|
||||
tenant_id=tenant,
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=_zip_bytes({"one.txt": b"one", "two.txt": b"two"}),
|
||||
filename="files.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
)
|
||||
)
|
||||
fixture.session.commit()
|
||||
self.assertEqual(2, factory.call_count)
|
||||
self.assertEqual(
|
||||
["tenant-one", "tenant-two"], [items[0].blob.tenant_id for items in results]
|
||||
)
|
||||
self.assertNotEqual(results[0][0].blob.id, results[1][0].blob.id)
|
||||
|
||||
def test_asset_and_version_are_published_once_without_redundant_update(self):
|
||||
fixture = recovery_fixture.StorageRecoveryTests()
|
||||
fixture.setUp()
|
||||
self.addCleanup(fixture.doCleanups)
|
||||
register_files_change_tracking()
|
||||
queries = []
|
||||
event.listen(
|
||||
fixture.engine,
|
||||
"before_cursor_execute",
|
||||
lambda conn, cursor, statement, parameters, context, executemany: (
|
||||
queries.append(statement)
|
||||
),
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=fixture.backend,
|
||||
):
|
||||
stored = extract_archive_upload(
|
||||
fixture.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=_zip_bytes({"one.txt": b"one", "two.txt": b"two"}),
|
||||
filename="files.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
)
|
||||
fixture.session.commit()
|
||||
self.assertFalse(
|
||||
any(statement.startswith("UPDATE file_assets") for statement in queries)
|
||||
)
|
||||
events = (
|
||||
fixture.session.query(ChangeSequenceEntry)
|
||||
.filter(ChangeSequenceEntry.module_id == "files")
|
||||
.all()
|
||||
)
|
||||
self.assertEqual(["created", "created"], [entry.operation for entry in events])
|
||||
self.assertTrue(
|
||||
all(item.asset.current_version_id == item.version.id for item in stored)
|
||||
)
|
||||
|
||||
def test_streamed_late_content_limit_failure_rolls_back_previous_blobs(self):
|
||||
fixture = recovery_fixture.StorageRecoveryTests()
|
||||
fixture.setUp()
|
||||
self.addCleanup(fixture.doCleanups)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=fixture.backend,
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "per-file limit"):
|
||||
extract_archive_upload(
|
||||
fixture.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=_zip_bytes(
|
||||
{"one.txt": b"one", "too-large.txt": b"x" * 200}
|
||||
),
|
||||
filename="files.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
max_file_bytes=100,
|
||||
)
|
||||
fixture.session.rollback()
|
||||
self.assertEqual([], list(fixture.backend.root.rglob("*.blob")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,315 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from govoplan_core.auth import get_api_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_core.security.secrets import open_transient_payload, seal_transient_payload
|
||||
from govoplan_files.backend.archive_work import ArchiveProgress, use_staged_upload
|
||||
from govoplan_files.backend import archive_work
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.routes import uploads
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
import test_managed_archives as managed_archive_tests
|
||||
|
||||
|
||||
class ArchiveStagingTests(unittest.TestCase):
|
||||
store_blob = managed_archive_tests.ManagedArchiveTests.store_blob
|
||||
seed_source = managed_archive_tests.ManagedArchiveTests.seed_source
|
||||
assert_no_extraction = managed_archive_tests.ManagedArchiveTests.assert_no_extraction
|
||||
|
||||
def setUp(self):
|
||||
managed_archive_tests.ManagedArchiveTests.setUp(self)
|
||||
self.directory = Path(self.enterContext(tempfile.TemporaryDirectory(prefix="archive-staging-tests-")))
|
||||
self.settings.file_archive_work_root = str(self.directory / "work")
|
||||
self.settings.file_archive_staged_per_actor = 4
|
||||
self.settings.file_archive_staged_max_bytes = 20 * 1024 * 1024
|
||||
app = FastAPI()
|
||||
app.include_router(uploads.router, prefix="/api/v1")
|
||||
app.dependency_overrides[get_api_principal] = lambda: self.principal
|
||||
app.dependency_overrides[get_session] = lambda: self.session
|
||||
self.client = self.enterContext(TestClient(app, raise_server_exceptions=False))
|
||||
|
||||
def preview(self, *, archive=None, **changes):
|
||||
return self.client.post(
|
||||
"/api/v1/files/archive-preview",
|
||||
data={"path": "extracted", "retain_upload": "true", **changes},
|
||||
files={"file": ("private-source.zip", archive or managed_archive_tests.archive_bytes(), "application/zip")},
|
||||
)
|
||||
|
||||
def staged_preview(self, preview, **changes):
|
||||
return self.client.post(
|
||||
"/api/v1/files/archive-preview",
|
||||
data={"path": "extracted", "preview_token": preview["preview_token"], "staged_upload_id": preview["staged_upload_id"], **changes},
|
||||
)
|
||||
|
||||
def confirm(self, preview, **changes):
|
||||
return self.client.post(
|
||||
"/api/v1/files/archive-confirm",
|
||||
data={"path": "extracted", "preview_token": preview["preview_token"], "staged_upload_id": preview["staged_upload_id"], "selected_paths_json": json.dumps(["folder/one.txt"]), **changes},
|
||||
)
|
||||
|
||||
def staged_path(self, preview):
|
||||
with use_staged_upload(self.settings, preview["staged_upload_id"], tenant_id="tenant-1", user_id="user-1") as filename:
|
||||
return Path(filename)
|
||||
|
||||
def progress(self, operation_id):
|
||||
return self.client.get(f"/api/v1/files/archive-progress/{operation_id}")
|
||||
|
||||
def test_preview_stages_once_password_repreview_and_confirm_reuse_upload(self):
|
||||
password = "staged secret password"
|
||||
preview = self.preview(archive=managed_archive_tests.archive_bytes(password=password))
|
||||
self.assertEqual(preview.status_code, 200, preview.text)
|
||||
self.assertTrue(preview.json()["requires_password"])
|
||||
self.assertFalse(preview.json()["password_verified"])
|
||||
stage = self.staged_path(preview.json())
|
||||
self.assert_no_extraction()
|
||||
operation_id = str(uuid4())
|
||||
with patch.object(uploads, "_spool_limited_upload_to_temp") as spool:
|
||||
wrong = self.staged_preview(preview.json(), password="incorrect")
|
||||
self.assertEqual(wrong.status_code, 400, wrong.text)
|
||||
self.assertTrue(stage.exists())
|
||||
verified = self.staged_preview(preview.json(), password=password)
|
||||
self.assertEqual(verified.status_code, 200, verified.text)
|
||||
self.assertTrue(verified.json()["password_verified"])
|
||||
self.assertEqual(verified.json()["staged_upload_id"], preview.json()["staged_upload_id"])
|
||||
confirmed = self.confirm(verified.json(), password=password, operation_id=operation_id)
|
||||
spool.assert_not_called()
|
||||
self.assertEqual(confirmed.status_code, 200, confirmed.text)
|
||||
self.assertEqual([item["display_path"] for item in confirmed.json()["files"]], ["extracted/folder/one.txt"])
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
self.assertFalse(stage.exists())
|
||||
progress = self.progress(operation_id)
|
||||
self.assertEqual(progress.status_code, 200, progress.text)
|
||||
self.assertEqual(progress.json()["status"], "complete")
|
||||
for value in (verified.text, confirmed.text, progress.text, str(self.audit.call_args_list)):
|
||||
self.assertNotIn(password, value)
|
||||
self.assertNotIn("private-source.zip", progress.text)
|
||||
self.assertNotIn("folder/one.txt", progress.text)
|
||||
payload = open_transient_payload(verified.json()["preview_token"], ttl_seconds=1800)
|
||||
self.assertNotIn(password, json.dumps(payload))
|
||||
|
||||
def test_staging_rejects_token_actor_tenant_stage_destination_and_digest_mismatch_without_writes(self):
|
||||
response = self.preview()
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
preview = response.json()
|
||||
for change in ({"preview_token": "tampered"}, {"staged_upload_id": str(uuid4())}, {"owner_id": "other-user"}, {"path": "elsewhere"}):
|
||||
with self.subTest(change=change):
|
||||
rejected = self.confirm(preview, **change)
|
||||
self.assertEqual(rejected.status_code, 400, rejected.text)
|
||||
self.assert_no_extraction()
|
||||
for field, original, foreign in (("tenant_id", "tenant-1", "other-tenant"), ("user_id", "user-1", "other-user")):
|
||||
target = self.principal if field == "tenant_id" else self.principal.user
|
||||
attribute = "tenant_id" if field == "tenant_id" else "id"
|
||||
setattr(target, attribute, foreign)
|
||||
self.assertEqual(self.confirm(preview).status_code, 400)
|
||||
setattr(target, attribute, original)
|
||||
path = self.staged_path(preview)
|
||||
path.write_bytes(b"tampered archive bytes")
|
||||
for response in (self.staged_preview(preview), self.confirm(preview)):
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
self.assertIn("contents changed", response.text)
|
||||
self.assert_no_extraction()
|
||||
self.assertTrue(path.exists())
|
||||
self.assertEqual(list(path.parent.glob("*.progress")), [])
|
||||
|
||||
def test_expired_stage_and_explicit_discard_are_bound_to_current_actor(self):
|
||||
preview = self.preview().json()
|
||||
path = self.staged_path(preview)
|
||||
self.principal.user.id = "other-user"
|
||||
self.assertEqual(self.client.delete(f"/api/v1/files/archive-staging/{preview['staged_upload_id']}").status_code, 204)
|
||||
self.assertTrue(path.exists())
|
||||
self.principal.user.id = "user-1"
|
||||
self.assertEqual(self.client.delete(f"/api/v1/files/archive-staging/{preview['staged_upload_id']}").status_code, 204)
|
||||
self.assertFalse(path.exists())
|
||||
self.assertEqual(self.confirm(preview).status_code, 410)
|
||||
preview = self.preview().json()
|
||||
path = self.staged_path(preview)
|
||||
old = time.time() - self.settings.file_archive_preview_ttl_seconds - 1
|
||||
os.utime(path, (old, old))
|
||||
self.assertEqual(self.confirm(preview).status_code, 410)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_progress_stays_running_through_commit_and_completes_only_after_commit(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
original_commit = self.session.commit
|
||||
observed = []
|
||||
|
||||
def commit():
|
||||
before = self.progress(operation_id).json()
|
||||
self.assertEqual(before["status"], "running")
|
||||
self.assertEqual(before["phase"], "finalizing")
|
||||
original_commit()
|
||||
observed.append(self.progress(operation_id).json()["status"])
|
||||
|
||||
with patch.object(self.session, "commit", side_effect=commit):
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
self.assertEqual(observed, ["running"])
|
||||
progress = self.progress(operation_id).json()
|
||||
self.assertEqual(progress["status"], "complete")
|
||||
self.assertEqual(progress["completed_files"], 1)
|
||||
self.assertEqual(progress["completed_bytes"], 3)
|
||||
|
||||
def test_commit_error_marks_progress_failed_and_preserves_stage_for_retry(self):
|
||||
preview = self.preview().json()
|
||||
stage = self.staged_path(preview)
|
||||
operation_id = str(uuid4())
|
||||
with patch.object(self.session, "commit", side_effect=FileStorageError("Commit refused")):
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "failed")
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 1)
|
||||
self.assertTrue(stage.exists())
|
||||
retry = self.confirm(preview, operation_id=str(uuid4()))
|
||||
self.assertEqual(retry.status_code, 200, retry.text)
|
||||
|
||||
def test_post_commit_stage_cleanup_failure_does_not_report_a_failed_import(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
with patch.object(uploads, "discard_staged_upload", side_effect=OSError("Temporary cleanup failure")):
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "complete")
|
||||
|
||||
def test_duplicate_operation_cannot_start_another_write_or_overwrite_progress(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
progress = ArchiveProgress(self.settings, operation_id, tenant_id="tenant-1", user_id="user-1")
|
||||
progress("extracting", 1, 5, 10, 50)
|
||||
before = self.progress(operation_id).json()
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
self.assertIn("already been used", response.text)
|
||||
self.assertEqual(self.progress(operation_id).json(), before)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_progress_and_discard_require_upload_permission_and_do_not_disclose_foreign_receipts(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
ArchiveProgress(self.settings, operation_id, tenant_id="tenant-1", user_id="user-1")
|
||||
self.principal.user.id = "other-user"
|
||||
self.assertEqual(self.progress(operation_id).status_code, 404)
|
||||
self.principal.user.id = "user-1"
|
||||
self.scopes.remove("files:file:upload")
|
||||
self.assertEqual(self.progress(operation_id).status_code, 403)
|
||||
self.assertEqual(self.client.delete(f"/api/v1/files/archive-staging/{preview['staged_upload_id']}").status_code, 403)
|
||||
self.assertTrue(self.staged_path(preview).exists())
|
||||
|
||||
def test_malformed_signed_digest_cannot_bypass_stage_binding(self):
|
||||
preview = self.preview().json()
|
||||
payload = open_transient_payload(preview["preview_token"], ttl_seconds=1800)
|
||||
payload["archive_sha256"] = None
|
||||
invalid = self.confirm(preview, preview_token=seal_transient_payload(payload))
|
||||
self.assertEqual(invalid.status_code, 400, invalid.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_progress_persistence_failure_does_not_turn_a_committed_import_into_an_error(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
original_replace = archive_work.os.replace
|
||||
|
||||
def replace(source, destination):
|
||||
# Only the optional UI progress receipt is unavailable. The private
|
||||
# worker ack is a required integrity/backpressure channel, not UI
|
||||
# progress, and must not be disabled by this module-global mock.
|
||||
if Path(source).name.startswith(".progress-"):
|
||||
raise OSError("Progress write unavailable")
|
||||
return original_replace(source, destination)
|
||||
|
||||
with patch.object(archive_work.os, "replace", side_effect=replace):
|
||||
result = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(result.status_code, 200, result.text)
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
# A failed observation write may leave an older running receipt, but
|
||||
# the confirmation response remains the authoritative outcome.
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "running")
|
||||
|
||||
def test_required_worker_ack_failure_rolls_back_and_preserves_upload_for_explicit_retry(self):
|
||||
preview = self.preview().json()
|
||||
stage = self.staged_path(preview)
|
||||
operation_id = str(uuid4())
|
||||
with patch("govoplan_files.backend.storage.archive_workers._send_ack",
|
||||
side_effect=FileStorageError("Archive worker staging channel is unavailable")):
|
||||
result = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(400, result.status_code, result.text)
|
||||
self.assertIn("staging channel is unavailable", result.text)
|
||||
self.assert_no_extraction()
|
||||
self.assertEqual("failed", self.progress(operation_id).json()["status"])
|
||||
self.assertTrue(stage.exists())
|
||||
result = self.confirm(preview, operation_id=str(uuid4()))
|
||||
self.assertEqual(200, result.status_code, result.text)
|
||||
|
||||
def test_lease_cleanup_failure_cannot_mask_a_committed_import(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
original_unlink = Path.unlink
|
||||
|
||||
def unlink(path, *args, **kwargs):
|
||||
if path.suffix == ".lease":
|
||||
raise OSError("Lease cleanup temporarily unavailable")
|
||||
return original_unlink(path, *args, **kwargs)
|
||||
|
||||
with patch.object(Path, "unlink", new=unlink):
|
||||
result = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(result.status_code, 200, result.text)
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "complete")
|
||||
|
||||
def test_invalid_operation_identifier_and_ambiguous_source_are_rejected_before_writes(self):
|
||||
preview = self.preview().json()
|
||||
response = self.confirm(preview, operation_id="../../outside")
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
ambiguous = self.client.post(
|
||||
"/api/v1/files/archive-confirm",
|
||||
data={"path": "extracted", "preview_token": preview["preview_token"], "staged_upload_id": preview["staged_upload_id"], "selected_paths_json": '["folder"]'},
|
||||
files={"file": ("other.zip", managed_archive_tests.archive_bytes(), "application/zip")},
|
||||
)
|
||||
self.assertEqual(ambiguous.status_code, 400, ambiguous.text)
|
||||
self.assertIn("either", ambiguous.text)
|
||||
self.assert_no_extraction()
|
||||
self.assertTrue(self.staged_path(preview).exists())
|
||||
|
||||
def test_staged_preview_requires_a_token_without_exposing_a_server_error(self):
|
||||
preview = self.preview().json()
|
||||
missing = self.client.post(
|
||||
"/api/v1/files/archive-preview",
|
||||
data={"path": "extracted", "staged_upload_id": preview["staged_upload_id"]},
|
||||
)
|
||||
self.assertIn(missing.status_code, (400, 422), missing.text)
|
||||
self.assertTrue(self.staged_path(preview).exists())
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_repreview_does_not_extend_the_original_stage_expiry(self):
|
||||
preview = self.preview().json()
|
||||
path = self.staged_path(preview)
|
||||
original_mtime = time.time() - 120
|
||||
os.utime(path, (original_mtime, original_mtime))
|
||||
original_deadline = path.stat().st_mtime + self.settings.file_archive_preview_ttl_seconds
|
||||
for _ in range(2):
|
||||
response = self.staged_preview(preview)
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
preview = response.json()
|
||||
returned_deadline = datetime.fromisoformat(preview["expires_at"]).timestamp()
|
||||
self.assertLessEqual(returned_deadline, original_deadline + 0.000001)
|
||||
self.assertAlmostEqual(returned_deadline, original_deadline, places=5)
|
||||
self.assertEqual(path.stat().st_mtime, original_mtime)
|
||||
self.assert_no_extraction()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,191 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from uuid import uuid4
|
||||
|
||||
from govoplan_files.backend import archive_work
|
||||
from govoplan_files.backend.archive_work import (
|
||||
ArchiveProgress,
|
||||
ArchiveWorkExpired,
|
||||
discard_staged_upload,
|
||||
read_progress,
|
||||
stage_upload,
|
||||
use_staged_upload,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
class ArchiveWorkTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.directory = Path(self.enterContext(tempfile.TemporaryDirectory(prefix="archive-work-tests-")))
|
||||
self.settings = SimpleNamespace(
|
||||
file_archive_work_root=str(self.directory / "work"),
|
||||
file_archive_preview_ttl_seconds=60,
|
||||
file_archive_staged_per_actor=2,
|
||||
file_archive_staged_max_bytes=100,
|
||||
)
|
||||
self.source = self.directory / "source.zip"
|
||||
self.source.write_bytes(b"archive bytes")
|
||||
self.actor = {"tenant_id": "tenant-1", "user_id": "user-1"}
|
||||
|
||||
def stage(self, **actor):
|
||||
return stage_upload(self.settings, str(self.source), **(self.actor | actor))
|
||||
|
||||
def use(self, stage_id, **actor):
|
||||
return use_staged_upload(self.settings, stage_id, **(self.actor | actor))
|
||||
|
||||
def progress(self, operation_id=None, **actor):
|
||||
return ArchiveProgress(self.settings, operation_id or str(uuid4()), **(self.actor | actor))
|
||||
|
||||
def test_staging_is_private_and_actor_and_tenant_bound(self):
|
||||
stage_id = self.stage()
|
||||
with self.use(stage_id) as filename:
|
||||
staged = Path(filename)
|
||||
self.assertEqual(staged.read_bytes(), self.source.read_bytes())
|
||||
self.assertEqual(staged.stat().st_mode & 0o777, 0o600)
|
||||
self.assertEqual(staged.parent.stat().st_mode & 0o777, 0o700)
|
||||
for actor in ({"user_id": "other-user"}, {"tenant_id": "other-tenant"}):
|
||||
with self.subTest(actor=actor):
|
||||
with self.assertRaises(ArchiveWorkExpired), self.use(stage_id, **actor):
|
||||
self.fail("Foreign actor obtained a staged archive")
|
||||
discard_staged_upload(self.settings, stage_id, **(self.actor | actor))
|
||||
with self.use(stage_id) as filename:
|
||||
self.assertTrue(Path(filename).is_file())
|
||||
|
||||
def test_noncanonical_identifiers_cannot_escape_the_private_root(self):
|
||||
stage_id = self.stage()
|
||||
for invalid in ("../source.zip", "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", stage_id.replace("-", ""), "", "not-a-uuid"):
|
||||
with self.subTest(invalid=invalid):
|
||||
with self.assertRaises(FileStorageError), self.use(invalid):
|
||||
self.fail("Invalid identifier accepted")
|
||||
with self.assertRaises(FileStorageError):
|
||||
discard_staged_upload(self.settings, invalid, **self.actor)
|
||||
with self.assertRaises(FileStorageError):
|
||||
read_progress(self.settings, invalid, **self.actor)
|
||||
self.assertEqual(self.source.read_bytes(), b"archive bytes")
|
||||
|
||||
def test_expired_stage_and_progress_are_not_readable(self):
|
||||
stage_id = self.stage()
|
||||
with self.use(stage_id) as filename:
|
||||
staged = Path(filename)
|
||||
progress = self.progress()
|
||||
old = time.time() - self.settings.file_archive_preview_ttl_seconds - 1
|
||||
os.utime(staged, (old, old))
|
||||
os.utime(progress.path, (old, old))
|
||||
with self.assertRaises(ArchiveWorkExpired), self.use(stage_id):
|
||||
self.fail("Expired stage accepted")
|
||||
with self.assertRaises(ArchiveWorkExpired):
|
||||
read_progress(self.settings, progress.path.stem[-36:], **self.actor)
|
||||
|
||||
def test_quota_evicts_oldest_unleased_stage_and_explicit_discard_is_idempotent(self):
|
||||
first = self.stage()
|
||||
with self.use(first) as filename:
|
||||
first_path = Path(filename)
|
||||
old = time.time() - 5
|
||||
os.utime(first_path, (old, old))
|
||||
second = self.stage()
|
||||
third = self.stage()
|
||||
self.assertFalse(first_path.exists())
|
||||
for stage_id in (second, third):
|
||||
with self.use(stage_id):
|
||||
pass
|
||||
discard_staged_upload(self.settings, third, **self.actor)
|
||||
discard_staged_upload(self.settings, third, **self.actor)
|
||||
with self.assertRaises(ArchiveWorkExpired), self.use(third):
|
||||
self.fail("Discarded archive retained")
|
||||
|
||||
def test_active_lease_blocks_duplicate_processing_discard_and_quota_eviction(self):
|
||||
self.settings.file_archive_staged_per_actor = 1
|
||||
first = self.stage()
|
||||
with self.use(first) as filename:
|
||||
with self.assertRaisesRegex(FileStorageError, "already being processed"), self.use(first):
|
||||
self.fail("Duplicate lease accepted")
|
||||
discard_staged_upload(self.settings, first, **self.actor)
|
||||
self.assertTrue(Path(filename).exists())
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.stage()
|
||||
with self.use(first):
|
||||
pass
|
||||
|
||||
def test_long_running_lease_remains_protected_after_preview_ttl(self):
|
||||
self.settings.file_archive_staged_per_actor = 1
|
||||
first = self.stage()
|
||||
with self.use(first) as filename:
|
||||
path = Path(filename)
|
||||
old = time.time() - self.settings.file_archive_preview_ttl_seconds - 1
|
||||
os.utime(path, (old, old))
|
||||
os.utime(path.with_suffix(".lease"), (old, old))
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.stage()
|
||||
self.assertTrue(path.exists(), "A live import must remain quota-accounted even after its preview TTL")
|
||||
|
||||
def test_global_byte_quota_does_not_evict_other_actor_archives(self):
|
||||
first = self.stage()
|
||||
self.settings.file_archive_staged_max_bytes = len(self.source.read_bytes())
|
||||
with self.assertRaisesRegex(FileStorageError, "storage is full"):
|
||||
self.stage(user_id="other-user")
|
||||
with self.use(first):
|
||||
pass
|
||||
|
||||
def test_progress_is_private_actor_bound_and_duplicate_operation_is_rejected(self):
|
||||
operation_id = str(uuid4())
|
||||
progress = self.progress(operation_id)
|
||||
progress("extracting", 2, 5, 20, 50)
|
||||
for actor in ({"user_id": "other-user"}, {"tenant_id": "other-tenant"}):
|
||||
with self.assertRaises(ArchiveWorkExpired):
|
||||
read_progress(self.settings, operation_id, **(self.actor | actor))
|
||||
before = read_progress(self.settings, operation_id, **self.actor)
|
||||
with self.assertRaisesRegex(FileStorageError, "already been used"):
|
||||
self.progress(operation_id)
|
||||
self.assertEqual(read_progress(self.settings, operation_id, **self.actor), before)
|
||||
self.assertEqual(set(before), {"phase", "status", "completed_files", "total_files", "completed_bytes", "total_bytes"})
|
||||
self.assertEqual(before["status"], "running")
|
||||
progress.finish(True)
|
||||
self.assertEqual(read_progress(self.settings, operation_id, **self.actor)["status"], "complete")
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.progress(operation_id)
|
||||
|
||||
def test_progress_write_failure_never_interrupts_an_import(self):
|
||||
progress = self.progress()
|
||||
with patch.object(archive_work.os, "replace", side_effect=OSError("Temporary storage unavailable")):
|
||||
progress("extracting", 1, 2, 10, 20)
|
||||
progress.finish(True)
|
||||
self.assertEqual(progress.value["status"], "complete")
|
||||
self.assertEqual(list(progress.path.parent.glob(".progress-*")), [])
|
||||
self.assertEqual(json.loads(progress.path.read_text())["status"], "running")
|
||||
|
||||
def test_missing_optional_operation_id_does_not_create_a_work_directory(self):
|
||||
progress = ArchiveProgress(self.settings, None, **self.actor)
|
||||
progress("extracting", 1, 1, 10, 10)
|
||||
progress.finish(True)
|
||||
self.assertIsNone(progress.path)
|
||||
self.assertFalse(Path(self.settings.file_archive_work_root).exists())
|
||||
|
||||
def test_work_directory_with_public_permissions_fails_closed(self):
|
||||
root = Path(self.settings.file_archive_work_root) / "v1"
|
||||
root.mkdir(parents=True, mode=0o755)
|
||||
root.chmod(0o755)
|
||||
with self.assertRaisesRegex(FileStorageError, "must be private"):
|
||||
self.stage()
|
||||
self.assertEqual(list(root.glob("*.archive")), [])
|
||||
|
||||
def test_orphan_lease_does_not_permanently_block_staging(self):
|
||||
stage_id = self.stage()
|
||||
with self.use(stage_id) as filename:
|
||||
staged = Path(filename)
|
||||
# Simulate the lock file left behind after a worker exits: no live
|
||||
# process holds its kernel lock, so a new request can reclaim it.
|
||||
staged.with_suffix(".lease").touch(mode=0o600)
|
||||
with self.use(stage_id):
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,346 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
import json
|
||||
import inspect
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import tracemalloc
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import zlib
|
||||
|
||||
from govoplan_core.security import bounded_process
|
||||
from govoplan_core.settings import settings
|
||||
from govoplan_files.backend.storage import archive_workers as workers
|
||||
from govoplan_files.backend.storage.archives import extract_archive_upload, inspect_archive
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from test_archives import _zip_bytes
|
||||
|
||||
|
||||
def _cpu_exhaustion_worker(payload: bytes) -> bytes:
|
||||
# A real imported child operation proves the Files wrapper's CPU error path;
|
||||
# the metadata tests below execute the actual archive parser, not this probe.
|
||||
while True:
|
||||
pass
|
||||
|
||||
|
||||
def _extract(payload, **options):
|
||||
return extract_archive_upload(
|
||||
object(), tenant_id="tenant", owner_type="user", owner_id="owner",
|
||||
user_id="owner", filename="fixture.zip", archive_data=payload,
|
||||
folder="destination", campaign_id=None, **options,
|
||||
)
|
||||
|
||||
|
||||
class ArchiveWorkerTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.directories = []
|
||||
self.children = []
|
||||
real_directory = tempfile.TemporaryDirectory
|
||||
real_popen = subprocess.Popen
|
||||
|
||||
def directory(*args, **kwargs):
|
||||
context = real_directory(*args, **kwargs)
|
||||
self.directories.append(Path(context.name))
|
||||
return context
|
||||
|
||||
def popen(*args, **kwargs):
|
||||
child = real_popen(*args, **kwargs)
|
||||
self.children.append(child)
|
||||
return child
|
||||
|
||||
self.enterContext(patch.object(workers.tempfile, "TemporaryDirectory", side_effect=directory))
|
||||
self.enterContext(patch.object(bounded_process.subprocess, "Popen", side_effect=popen))
|
||||
self.store = self.enterContext(patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset",
|
||||
return_value=SimpleNamespace(asset=SimpleNamespace(id="asset")),
|
||||
))
|
||||
|
||||
def tearDown(self):
|
||||
self.assertTrue(all(not directory.exists() for directory in self.directories))
|
||||
self.assertTrue(all(child.returncode is not None for child in self.children))
|
||||
|
||||
def test_parent_never_parses_metadata_or_extracts_and_one_child_handles_all_members(self):
|
||||
payload = _zip_bytes({"one.txt": b"one", "two.txt": b"two"})
|
||||
with (
|
||||
patch("govoplan_files.backend.storage.archives._inspect_archive_content", side_effect=AssertionError("parent parser")),
|
||||
patch("govoplan_files.backend.storage.archives._read_selected_zip_members", side_effect=AssertionError("parent decoder")),
|
||||
):
|
||||
self.assertEqual(2, inspect_archive(payload, filename="fixture.zip").file_count)
|
||||
self.assertEqual(2, len(_extract(payload)))
|
||||
self.assertEqual(2, len(self.children)) # one preview, one whole extraction
|
||||
self.assertEqual([b"one", b"two"], [call.kwargs["data"] for call in self.store.call_args_list])
|
||||
|
||||
def test_hostile_pax_metadata_allocation_is_confined_before_any_store(self):
|
||||
header = tarfile.TarInfo("pax")
|
||||
header.type = tarfile.XHDTYPE
|
||||
header.size = 1024 * 1024 * 1024
|
||||
payload = header.tobuf() + b"\0" * 1024
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", side_effect=AssertionError("parent TAR parser")):
|
||||
with self.assertRaisesRegex(FileStorageError, "memory_limit|Invalid TAR|could not complete safely"):
|
||||
inspect_archive(payload, filename="hostile.tar")
|
||||
self.store.assert_not_called()
|
||||
self.assertEqual(1, len(self.children))
|
||||
# A rejected archive neither consumes the admission slot permanently nor
|
||||
# prevents a later ordinary parse in the same parent process.
|
||||
self.assertEqual(1, inspect_archive(_zip_bytes({"ok": b"ok"}), filename="ok.zip").file_count)
|
||||
|
||||
def test_compressed_pax_metadata_hits_real_child_memory_limit(self):
|
||||
header = tarfile.TarInfo("pax")
|
||||
header.type = tarfile.XHDTYPE
|
||||
header.size = 1024 * 1024 * 1024
|
||||
compressor = zlib.compressobj(1, wbits=31)
|
||||
chunks = [compressor.compress(header.tobuf())]
|
||||
block = b"\0" * (1024 * 1024)
|
||||
# Build the hostile fixture incrementally: ~2.3 MiB compressed, never a
|
||||
# 512 MiB parent allocation. TAR consumes PAX metadata before ordinary
|
||||
# returned-member limits can inspect it; the child AS limit must win.
|
||||
for _ in range(512):
|
||||
chunks.append(compressor.compress(block))
|
||||
chunks.append(compressor.flush())
|
||||
payload = b"".join(chunks)
|
||||
self.assertLess(len(payload), 3 * 1024 * 1024)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", side_effect=AssertionError("parent TAR parser")):
|
||||
with self.assertRaisesRegex(FileStorageError, "memory_limit"):
|
||||
inspect_archive(payload, filename="hostile.tar.gz")
|
||||
self.store.assert_not_called()
|
||||
self.assertEqual(1, len(self.children))
|
||||
|
||||
def test_real_child_timeout_reaps_and_removes_snapshot(self):
|
||||
with patch.object(workers, "INSPECTION_LIMITS", replace(workers.INSPECTION_LIMITS, wall_seconds=0.001)):
|
||||
with self.assertRaisesRegex(FileStorageError, "timeout"):
|
||||
inspect_archive(_zip_bytes({"one": b"one"}), filename="fixture.zip")
|
||||
self.assertEqual(1, len(self.children))
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_real_child_cpu_limit_has_sanitized_files_error(self):
|
||||
with (
|
||||
patch.object(workers, "_inspect_worker", _cpu_exhaustion_worker),
|
||||
patch.object(workers, "INSPECTION_LIMITS", replace(workers.INSPECTION_LIMITS, cpu_seconds=1, wall_seconds=10)),
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "cpu_limit"):
|
||||
inspect_archive(_zip_bytes({"one": b"one"}), filename="fixture.zip")
|
||||
|
||||
def test_real_preview_output_is_bounded(self):
|
||||
with patch.object(workers, "INSPECTION_LIMITS", replace(workers.INSPECTION_LIMITS, output_bytes=256)):
|
||||
with self.assertRaisesRegex(FileStorageError, "output_limit"):
|
||||
inspect_archive(_zip_bytes({f"file-{index}": b"x" for index in range(20)}), filename="fixture.zip")
|
||||
|
||||
def test_destination_failure_kills_waiting_child_and_preserves_error(self):
|
||||
self.store.side_effect = FileStorageError("Destination failure")
|
||||
with self.assertRaisesRegex(FileStorageError, "Destination failure"):
|
||||
_extract(_zip_bytes({"one": b"one", "two": b"two"}))
|
||||
self.assertEqual(1, self.store.call_count)
|
||||
self.assertEqual(1, len(self.children))
|
||||
|
||||
def test_missing_ack_times_out_and_wrong_ack_fails_in_real_child(self):
|
||||
real_send = workers._send_ack
|
||||
for send, error in ((lambda descriptor, sequence: None, "timeout"),
|
||||
(lambda descriptor, sequence: real_send(descriptor, sequence + 1), "invalid staging record")):
|
||||
with self.subTest(error=error), patch.object(workers, "_send_ack", side_effect=send):
|
||||
with patch.object(workers, "EXTRACTION_LIMITS", replace(workers.EXTRACTION_LIMITS, wall_seconds=2)):
|
||||
with self.assertRaisesRegex(FileStorageError, error):
|
||||
_extract(_zip_bytes({"one": b"one", "two": b"two"}))
|
||||
|
||||
def test_parent_progress_exception_is_not_reclassified_as_transport_error(self):
|
||||
def progress(stage, *counters):
|
||||
if stage == "extracting":
|
||||
raise ValueError("parent progress failure")
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "parent progress failure"):
|
||||
_extract(_zip_bytes({"one": b"one"}), progress=progress)
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_source_path_snapshot_rejects_growth_and_shrink_without_spawning(self):
|
||||
# A source path is server-owned; it must still not turn into an unbounded
|
||||
# copy when another writer changes it during snapshot creation.
|
||||
for replacement in (b"fixture plus unexpected growth", b"x"):
|
||||
with self.subTest(replacement=replacement), tempfile.TemporaryDirectory() as temporary:
|
||||
source = Path(temporary) / "archive"
|
||||
source.write_bytes(b"fixture")
|
||||
real_fstat = os.fstat
|
||||
changed = False
|
||||
|
||||
def fstat(descriptor):
|
||||
nonlocal changed
|
||||
result = real_fstat(descriptor)
|
||||
if not changed:
|
||||
changed = True
|
||||
source.write_bytes(replacement)
|
||||
return result
|
||||
|
||||
with patch.object(workers.os, "fstat", side_effect=fstat):
|
||||
with self.assertRaisesRegex(FileStorageError, "source changed"):
|
||||
inspect_archive(source, filename="fixture.zip")
|
||||
self.assertEqual([], self.children)
|
||||
|
||||
def test_busy_rejects_before_source_snapshot(self):
|
||||
with patch.object(settings, "isolated_process_concurrency", 1), bounded_process.bounded_operation_admission():
|
||||
with patch.object(workers, "_source_stage", side_effect=AssertionError("must admit before copying")):
|
||||
with self.assertRaisesRegex(FileStorageError, "busy"):
|
||||
inspect_archive(b"fixture", filename="fixture.zip")
|
||||
self.assertEqual([], self.children)
|
||||
self.assertEqual([], self.directories)
|
||||
|
||||
def test_unsupported_format_rejects_before_snapshot(self):
|
||||
with self.assertRaisesRegex(FileStorageError, "Unsupported archive format"):
|
||||
inspect_archive(b"fixture", filename="fixture.exe")
|
||||
self.assertEqual([], self.directories)
|
||||
|
||||
def test_symlinked_staged_member_is_never_read_or_stored(self):
|
||||
real_read = workers._read_regular
|
||||
attacked = False
|
||||
|
||||
def read(path, maximum, **options):
|
||||
nonlocal attacked
|
||||
if path.name.startswith("member-") and not attacked:
|
||||
attacked = True
|
||||
path.unlink()
|
||||
path.symlink_to(path.parent / "source")
|
||||
return real_read(path, maximum, **options)
|
||||
|
||||
with patch.object(workers, "_read_regular", side_effect=read):
|
||||
with self.assertRaisesRegex(FileStorageError, "invalid staging record"):
|
||||
_extract(_zip_bytes({"one": b"one"}))
|
||||
self.assertTrue(attacked)
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_forged_member_records_reject_before_store(self):
|
||||
real_read = workers._read_regular
|
||||
for mutation in (
|
||||
{"path": "outside/one"}, {"path": "../escape"}, {"size": -1},
|
||||
{"sha256": "incorrect"}, {"sequence": True}, {"unexpected": True},
|
||||
{"progress": ["extracting", 1, 1, 4, 3]},
|
||||
):
|
||||
with self.subTest(mutation=mutation):
|
||||
self.store.reset_mock()
|
||||
|
||||
def read(path, maximum, **options):
|
||||
value = real_read(path, maximum, **options)
|
||||
if path.name == "status":
|
||||
record = json.loads(value)
|
||||
if record.get("kind") == "member":
|
||||
record.update(mutation)
|
||||
return json.dumps(record).encode()
|
||||
return value
|
||||
|
||||
with patch.object(workers, "_read_regular", side_effect=read):
|
||||
with self.assertRaises(FileStorageError):
|
||||
_extract(_zip_bytes({"selected/one": b"one"}), selected_paths=("selected",))
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_regressing_sequence_or_changing_totals_fail_before_second_store(self):
|
||||
real_read = workers._read_regular
|
||||
for mutation in ({"sequence": 1}, {"progress": ["extracting", 2, 3, 6, 6]}):
|
||||
with self.subTest(mutation=mutation):
|
||||
self.store.reset_mock()
|
||||
|
||||
def read(path, maximum, **options):
|
||||
value = real_read(path, maximum, **options)
|
||||
if path.name == "status":
|
||||
record = json.loads(value)
|
||||
if record.get("kind") == "member" and record["progress"][1] == 2:
|
||||
record.update(mutation)
|
||||
return json.dumps(record).encode()
|
||||
return value
|
||||
|
||||
with patch.object(workers, "_read_regular", side_effect=read):
|
||||
with self.assertRaisesRegex(FileStorageError, "invalid staging record"):
|
||||
_extract(_zip_bytes({"one": b"one", "two": b"two"}))
|
||||
self.assertEqual(1, self.store.call_count)
|
||||
|
||||
def test_record_reads_are_bounded_and_reject_non_regular_files(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
directory = Path(temporary)
|
||||
status = directory / "status"
|
||||
status.write_bytes(b"x" * (workers._STATUS_BYTES + 1))
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._read_regular(status, workers._STATUS_BYTES)
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._read_regular(directory, workers._STATUS_BYTES)
|
||||
|
||||
def test_tiny_member_read_does_not_allocate_the_configured_gibibyte_cap(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
member = Path(temporary) / "member"
|
||||
member.write_bytes(b"x")
|
||||
tracemalloc.start()
|
||||
try:
|
||||
self.assertEqual(b"x", workers._read_regular(member, 2 * 1024 * 1024 * 1024))
|
||||
_current, peak = tracemalloc.get_traced_memory()
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
self.assertLess(peak, 1024 * 1024)
|
||||
|
||||
def test_atomic_status_replacement_keeps_open_snapshot_valid_but_member_identity_is_strict(self):
|
||||
for atomic_record in (True, False):
|
||||
with self.subTest(atomic_record=atomic_record), tempfile.TemporaryDirectory() as temporary:
|
||||
status = Path(temporary) / "status"
|
||||
status.write_bytes(b"old")
|
||||
replacement = Path(temporary) / "replacement"
|
||||
replacement.write_bytes(b"new")
|
||||
real_fstat = os.fstat
|
||||
replaced = False
|
||||
|
||||
def fstat(descriptor):
|
||||
nonlocal replaced
|
||||
info = real_fstat(descriptor)
|
||||
if not replaced:
|
||||
replaced = True
|
||||
os.replace(replacement, status)
|
||||
return info
|
||||
|
||||
with patch.object(workers.os, "fstat", side_effect=fstat):
|
||||
if atomic_record:
|
||||
self.assertEqual(b"old", workers._read_regular(status, 16, atomic_record=True))
|
||||
else:
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._read_regular(status, 16)
|
||||
self.assertEqual(b"new", status.read_bytes())
|
||||
|
||||
def test_private_ack_channel_is_bounded_and_validates_type_permissions_and_sequence(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
directory = Path(temporary)
|
||||
with workers._ack_channel(directory, create=True) as writer:
|
||||
with workers._ack_channel(directory) as reader:
|
||||
workers._send_ack(writer, 7)
|
||||
workers._receive_ack(reader, 7)
|
||||
workers._send_ack(writer, 8)
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._receive_ack(reader, 9)
|
||||
with patch.object(workers.os, "write", side_effect=BlockingIOError()):
|
||||
with self.assertRaisesRegex(FileStorageError, "staging channel is unavailable"):
|
||||
workers._send_ack(writer, 10)
|
||||
ack = directory / "ack"
|
||||
os.chmod(ack, 0o644)
|
||||
with self.assertRaises(FileStorageError), workers._ack_channel(directory):
|
||||
pass
|
||||
ack.unlink()
|
||||
ack.write_bytes(b"not a FIFO")
|
||||
with self.assertRaises(FileStorageError), workers._ack_channel(directory):
|
||||
pass
|
||||
ack.unlink()
|
||||
ack.symlink_to(directory / "missing")
|
||||
with self.assertRaises(FileStorageError), workers._ack_channel(directory):
|
||||
pass
|
||||
|
||||
def test_tiny_member_import_has_no_per_member_sleep_or_polling_delay(self):
|
||||
self.assertNotIn("sleep(", inspect.getsource(workers._extract_worker))
|
||||
for count in (100, 1000):
|
||||
with self.subTest(count=count):
|
||||
payload = _zip_bytes({f"member-{index}.txt": b"x" for index in range(count)})
|
||||
started = time.monotonic()
|
||||
self.assertEqual(count, len(_extract(payload)))
|
||||
elapsed = time.monotonic() - started
|
||||
# Broad synthetic regression ceiling, not a storage-throughput
|
||||
# promise: 50 ms per member would take >=50 s for 1,000 files.
|
||||
self.assertLess(elapsed, 15)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,288 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
|
||||
from govoplan_core.core.configuration_packages import (
|
||||
ConfigurationPackageFragment,
|
||||
ConfigurationPreflightContext,
|
||||
)
|
||||
from govoplan_core.core.infrastructure_capabilities import (
|
||||
infrastructure_capability_receipt_from_mapping,
|
||||
)
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.db.session import configure_database, reset_database
|
||||
from govoplan_files.backend.configuration_provider import (
|
||||
FILES_CONFIGURATION_CAPABILITY,
|
||||
FILES_INFRASTRUCTURE_DEPENDENCY_CAPABILITY,
|
||||
FilesConfigurationProvider,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileBlob
|
||||
from govoplan_files.backend.manifest import manifest
|
||||
|
||||
|
||||
def _receipt(*, source: str = "host-local", state: str = "configured"):
|
||||
endpoint = (
|
||||
{"kind": "filesystem", "reference": "volume:files-data"}
|
||||
if source == "host-local"
|
||||
else {"scheme": "http", "host": "garage", "port": 3900}
|
||||
)
|
||||
secret_refs = (
|
||||
[]
|
||||
if source == "host-local"
|
||||
else [
|
||||
"env:FILE_STORAGE_S3_ACCESS_KEY_ID",
|
||||
"env:FILE_STORAGE_S3_SECRET_ACCESS_KEY",
|
||||
"env:GARAGE_RPC_SECRET",
|
||||
]
|
||||
)
|
||||
return infrastructure_capability_receipt_from_mapping(
|
||||
{
|
||||
"schema_version": 1,
|
||||
"installation_id": "files-provider-test",
|
||||
"profile": "evaluation",
|
||||
"capabilities": [
|
||||
{
|
||||
"id": "files.storage",
|
||||
"label": "Managed file content storage",
|
||||
"state": state,
|
||||
"source": source,
|
||||
"detail": "Deployment-owned storage binding.",
|
||||
"endpoint": endpoint,
|
||||
"secret_refs": secret_refs,
|
||||
"dependent_modules": ["files"],
|
||||
}
|
||||
],
|
||||
"post_install_tasks": [],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _local_settings(**overrides):
|
||||
values = {
|
||||
"file_storage_backend": "local",
|
||||
"file_storage_local_root": "/var/lib/govoplan/files",
|
||||
}
|
||||
values.update(overrides)
|
||||
return SimpleNamespace(**values)
|
||||
|
||||
|
||||
def _s3_settings(**overrides):
|
||||
values = {
|
||||
"file_storage_backend": "s3",
|
||||
"file_storage_s3_endpoint_url": "http://garage:3900",
|
||||
"file_storage_s3_bucket": "files",
|
||||
"file_storage_s3_deployment_managed": True,
|
||||
"file_storage_s3_endpoint_trusted": False,
|
||||
}
|
||||
values.update(overrides)
|
||||
return SimpleNamespace(**values)
|
||||
|
||||
|
||||
class FilesConfigurationProviderTests(unittest.TestCase):
|
||||
def test_provider_is_registered(self) -> None:
|
||||
self.assertIn(FILES_CONFIGURATION_CAPABILITY, manifest.capability_factories)
|
||||
self.assertIn(
|
||||
FILES_INFRASTRUCTURE_DEPENDENCY_CAPABILITY,
|
||||
manifest.capability_factories,
|
||||
)
|
||||
|
||||
def test_inventory_reports_runtime_binding_and_persisted_blob_aggregate(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="govoplan-files-inventory-") as root:
|
||||
database_path = Path(root) / "files.sqlite3"
|
||||
engine = create_engine(f"sqlite:///{database_path}")
|
||||
Base.metadata.create_all(engine, tables=(FileBlob.__table__,))
|
||||
configure_database(
|
||||
f"sqlite:///{database_path}",
|
||||
engine=engine,
|
||||
dispose_previous=True,
|
||||
)
|
||||
try:
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
FileBlob.__table__.insert(),
|
||||
[
|
||||
{
|
||||
"id": "blob-1",
|
||||
"tenant_id": "tenant-1",
|
||||
"storage_backend": "local",
|
||||
"storage_key": "tenant-1/a",
|
||||
"checksum_sha256": "a" * 64,
|
||||
"size_bytes": 7,
|
||||
"protection_discriminator": "plaintext",
|
||||
"ref_count": 1,
|
||||
"integrity_status": "unchecked",
|
||||
},
|
||||
{
|
||||
"id": "blob-2",
|
||||
"tenant_id": "tenant-1",
|
||||
"storage_backend": "local",
|
||||
"storage_key": "tenant-1/b",
|
||||
"checksum_sha256": "b" * 64,
|
||||
"size_bytes": 11,
|
||||
"protection_discriminator": "plaintext",
|
||||
"ref_count": 1,
|
||||
"integrity_status": "unchecked",
|
||||
},
|
||||
],
|
||||
)
|
||||
provider = FilesConfigurationProvider(
|
||||
settings=_local_settings(),
|
||||
environment={},
|
||||
)
|
||||
|
||||
dependencies = provider.infrastructure_dependencies()
|
||||
finally:
|
||||
reset_database()
|
||||
engine.dispose()
|
||||
|
||||
self.assertEqual(
|
||||
["runtime_storage_binding", "stored_blob_set"],
|
||||
[item.dependency_type for item in dependencies],
|
||||
)
|
||||
blob_set = dependencies[1]
|
||||
self.assertEqual(2, blob_set.metrics["blob_count"])
|
||||
self.assertEqual(18, blob_set.metrics["content_bytes"])
|
||||
|
||||
def test_matching_local_storage_is_an_idempotent_noop(self) -> None:
|
||||
provider = FilesConfigurationProvider(
|
||||
settings=_local_settings(),
|
||||
environment={},
|
||||
)
|
||||
context = ConfigurationPreflightContext(
|
||||
infrastructure_receipt=_receipt(),
|
||||
)
|
||||
fragment = ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type="managed_storage",
|
||||
fragment_id="files-storage",
|
||||
payload={
|
||||
"expected_backend": "local",
|
||||
"expected_source": "host-local",
|
||||
},
|
||||
)
|
||||
|
||||
first = provider.preflight(fragment, context)
|
||||
applied = provider.apply(fragment, {}, context)
|
||||
second = provider.preflight(fragment, context)
|
||||
|
||||
self.assertEqual("skip", first.plan[0].action)
|
||||
self.assertEqual("skip", second.plan[0].action)
|
||||
self.assertEqual((), applied.diagnostics)
|
||||
self.assertEqual({}, applied.created_refs)
|
||||
self.assertEqual({}, applied.updated_refs)
|
||||
|
||||
def test_runtime_backend_mismatch_blocks_without_rewriting_settings(self) -> None:
|
||||
settings = _local_settings()
|
||||
provider = FilesConfigurationProvider(settings=settings, environment={})
|
||||
context = ConfigurationPreflightContext(
|
||||
infrastructure_receipt=_receipt(source="installer-managed-garage"),
|
||||
)
|
||||
fragment = ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type="managed_storage",
|
||||
payload={},
|
||||
)
|
||||
|
||||
result = provider.preflight(fragment, context)
|
||||
|
||||
self.assertEqual("blocked", result.plan[0].action)
|
||||
self.assertIn(
|
||||
"files_storage_runtime_mismatch",
|
||||
{item.code for item in result.diagnostics},
|
||||
)
|
||||
self.assertEqual("local", settings.file_storage_backend)
|
||||
|
||||
def test_garage_binding_requires_only_files_secret_references(self) -> None:
|
||||
fragment = ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type="managed_storage",
|
||||
payload={},
|
||||
)
|
||||
context = ConfigurationPreflightContext(
|
||||
infrastructure_receipt=_receipt(source="installer-managed-garage"),
|
||||
)
|
||||
missing = FilesConfigurationProvider(
|
||||
settings=_s3_settings(),
|
||||
environment={},
|
||||
).preflight(fragment, context)
|
||||
available = FilesConfigurationProvider(
|
||||
settings=_s3_settings(),
|
||||
environment={
|
||||
"FILE_STORAGE_S3_ACCESS_KEY_ID": "reference-resolved",
|
||||
"FILE_STORAGE_S3_SECRET_ACCESS_KEY": "reference-resolved",
|
||||
},
|
||||
).preflight(fragment, context)
|
||||
|
||||
self.assertEqual("blocked", missing.plan[0].action)
|
||||
self.assertEqual(
|
||||
2,
|
||||
sum(
|
||||
item.code == "files_storage_secret_reference_unresolved"
|
||||
for item in missing.diagnostics
|
||||
),
|
||||
)
|
||||
self.assertEqual("skip", available.plan[0].action)
|
||||
|
||||
def test_operator_supplied_s3_requires_explicit_endpoint_trust(self) -> None:
|
||||
receipt = _receipt(source="operator-supplied-s3", state="externally_supplied")
|
||||
fragment = ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type="managed_storage",
|
||||
payload={},
|
||||
)
|
||||
context = ConfigurationPreflightContext(infrastructure_receipt=receipt)
|
||||
environment = {
|
||||
"FILE_STORAGE_S3_ACCESS_KEY_ID": "reference-resolved",
|
||||
"FILE_STORAGE_S3_SECRET_ACCESS_KEY": "reference-resolved",
|
||||
}
|
||||
|
||||
blocked = FilesConfigurationProvider(
|
||||
settings=_s3_settings(
|
||||
file_storage_s3_deployment_managed=False,
|
||||
file_storage_s3_endpoint_trusted=False,
|
||||
),
|
||||
environment=environment,
|
||||
).preflight(fragment, context)
|
||||
ready = FilesConfigurationProvider(
|
||||
settings=_s3_settings(
|
||||
file_storage_s3_deployment_managed=False,
|
||||
file_storage_s3_endpoint_trusted=True,
|
||||
),
|
||||
environment=environment,
|
||||
).preflight(fragment, context)
|
||||
|
||||
self.assertEqual("blocked", blocked.plan[0].action)
|
||||
self.assertIn(
|
||||
"files_storage_trust_boundary_missing",
|
||||
{item.code for item in blocked.diagnostics},
|
||||
)
|
||||
self.assertEqual("skip", ready.plan[0].action)
|
||||
|
||||
def test_inline_storage_secret_is_rejected(self) -> None:
|
||||
provider = FilesConfigurationProvider(
|
||||
settings=_local_settings(),
|
||||
environment={},
|
||||
)
|
||||
result = provider.preflight(
|
||||
ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type="managed_storage",
|
||||
payload={"secret_access_key": "inline"},
|
||||
),
|
||||
ConfigurationPreflightContext(infrastructure_receipt=_receipt()),
|
||||
)
|
||||
|
||||
self.assertEqual("blocked", result.plan[0].action)
|
||||
self.assertIn(
|
||||
"files_configuration_secret_forbidden",
|
||||
{item.code for item in result.diagnostics},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,303 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock, Mock, call, patch
|
||||
|
||||
from govoplan_files.backend.routes.connector_io import sync_connector_space_folder
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileAssetResponse,
|
||||
FileConnectorFolderSyncRequest,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, UploadedStoredFile
|
||||
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseItem
|
||||
from govoplan_files.backend.storage.connector_folder_sync import (
|
||||
connector_relative_path,
|
||||
discover_connector_folder,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_imports import ConnectorDownloadedFile
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicyDenied
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
def _item(kind: str, path: str, *, token: str | None = None) -> ConnectorBrowseItem:
|
||||
return ConnectorBrowseItem(
|
||||
kind=kind,
|
||||
name=path.rsplit("/", 1)[-1],
|
||||
path=path,
|
||||
external_id=f"external:{path}",
|
||||
etag=f"etag:{path}",
|
||||
metadata={"next_continuation_token": token} if token else {},
|
||||
)
|
||||
|
||||
|
||||
class ConnectorFolderDiscoveryTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.profile = ConnectorProfile(
|
||||
id="profile-1",
|
||||
label="Documents",
|
||||
provider="webdav",
|
||||
capabilities=("browse", "sync"),
|
||||
)
|
||||
|
||||
@patch("govoplan_files.backend.storage.connector_folder_sync.browse_connector_profile")
|
||||
def test_recursively_discovers_files_across_pages_in_deterministic_order(
|
||||
self, browse: Mock
|
||||
) -> None:
|
||||
browse.side_effect = [
|
||||
[_item("file", "root/z.txt", token="page-2")],
|
||||
[_item("folder", "root/nested"), _item("file", "root/a.txt")],
|
||||
[_item("file", "root/nested/b.txt")],
|
||||
]
|
||||
|
||||
result = discover_connector_folder(
|
||||
self.profile,
|
||||
path="root",
|
||||
library_id="library-1",
|
||||
recursive=True,
|
||||
max_files=10,
|
||||
max_depth=3,
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
["root/a.txt", "root/nested/b.txt", "root/z.txt"],
|
||||
[item.path for item in result.files],
|
||||
)
|
||||
self.assertFalse(result.truncated)
|
||||
self.assertEqual([], result.skipped)
|
||||
self.assertEqual(
|
||||
[
|
||||
call(
|
||||
self.profile,
|
||||
path="root",
|
||||
library_id="library-1",
|
||||
continuation_token=None,
|
||||
),
|
||||
call(
|
||||
self.profile,
|
||||
path="root",
|
||||
library_id="library-1",
|
||||
continuation_token="page-2",
|
||||
),
|
||||
call(
|
||||
self.profile,
|
||||
path="root/nested",
|
||||
library_id="library-1",
|
||||
continuation_token=None,
|
||||
),
|
||||
],
|
||||
browse.call_args_list,
|
||||
)
|
||||
|
||||
@patch("govoplan_files.backend.storage.connector_folder_sync.browse_connector_profile")
|
||||
def test_non_recursive_and_file_limit_skips_are_explicit(self, browse: Mock) -> None:
|
||||
browse.return_value = [
|
||||
_item("folder", "root/nested"),
|
||||
_item("file", "root/a.txt"),
|
||||
_item("file", "root/b.txt"),
|
||||
]
|
||||
|
||||
result = discover_connector_folder(
|
||||
self.profile,
|
||||
path="root",
|
||||
library_id=None,
|
||||
recursive=False,
|
||||
max_files=1,
|
||||
max_depth=0,
|
||||
)
|
||||
|
||||
self.assertEqual(["root/a.txt"], [item.path for item in result.files])
|
||||
self.assertTrue(result.truncated)
|
||||
self.assertEqual(
|
||||
["root/nested", "root/b.txt"],
|
||||
[item.path for item in result.skipped],
|
||||
)
|
||||
|
||||
def test_relative_path_rejects_provider_results_outside_linked_root(self) -> None:
|
||||
self.assertEqual(
|
||||
"nested/report.csv",
|
||||
connector_relative_path(
|
||||
space_root="linked/root", item_path="linked/root/nested/report.csv"
|
||||
),
|
||||
)
|
||||
with self.assertRaisesRegex(ValueError, "outside"):
|
||||
connector_relative_path(
|
||||
space_root="linked/root", item_path="another/root/report.csv"
|
||||
)
|
||||
|
||||
|
||||
class ConnectorFolderSyncRouteTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.session = MagicMock()
|
||||
self.space = SimpleNamespace(
|
||||
id="space-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="user-1",
|
||||
owner_group_id=None,
|
||||
connector_profile_id="profile-1",
|
||||
provider="webdav",
|
||||
library_id="library-1",
|
||||
remote_path="linked/root",
|
||||
sync_mode="manual",
|
||||
)
|
||||
self.profile = ConnectorProfile(
|
||||
id="profile-1",
|
||||
label="Documents",
|
||||
provider="webdav",
|
||||
capabilities=("browse", "sync"),
|
||||
)
|
||||
self.principal = SimpleNamespace(
|
||||
tenant_id="tenant-1", user=SimpleNamespace(id="user-1")
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _stored(index: int) -> UploadedStoredFile:
|
||||
return UploadedStoredFile(
|
||||
asset=SimpleNamespace(id=f"asset-{index}"),
|
||||
version=SimpleNamespace(id=f"version-{index}"),
|
||||
blob=SimpleNamespace(id=f"blob-{index}"),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _response(index: int, action: str) -> FileAssetResponse:
|
||||
now = datetime(2026, 8, 21, tzinfo=UTC).isoformat()
|
||||
return FileAssetResponse(
|
||||
id=f"asset-{index}",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
display_path=f"Imports/file-{index}.txt",
|
||||
filename=f"file-{index}.txt",
|
||||
size_bytes=index,
|
||||
checksum_sha256=str(index) * 64,
|
||||
version_id=f"version-{index}",
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
source_revision=f"revision-{action}",
|
||||
)
|
||||
|
||||
@patch("govoplan_files.backend.routes.connector_io.audit_from_principal")
|
||||
@patch("govoplan_files.backend.routes.connector_io._asset_response")
|
||||
@patch("govoplan_files.backend.routes.connector_io._audit_connector_sync")
|
||||
@patch("govoplan_files.backend.routes.connector_io.sync_file_asset_from_source")
|
||||
@patch("govoplan_files.backend.routes.connector_io._download_connector_payload")
|
||||
@patch("govoplan_files.backend.routes.connector_io.discover_connector_folder")
|
||||
@patch("govoplan_files.backend.routes.connector_io._connector_space_policy_decision")
|
||||
@patch("govoplan_files.backend.routes.connector_io._visible_connector_profile")
|
||||
@patch("govoplan_files.backend.routes.connector_io.get_connector_space_for_user")
|
||||
@patch("govoplan_files.backend.routes.connector_io._is_admin", return_value=False)
|
||||
def test_returns_reviewable_per_item_results_and_audits_the_summary(
|
||||
self,
|
||||
_is_admin: Mock,
|
||||
get_space: Mock,
|
||||
visible_profile: Mock,
|
||||
policy_decision: Mock,
|
||||
discover: Mock,
|
||||
download: Mock,
|
||||
sync_file: Mock,
|
||||
audit_sync: Mock,
|
||||
asset_response: Mock,
|
||||
audit: Mock,
|
||||
) -> None:
|
||||
get_space.return_value = self.space
|
||||
visible_profile.return_value = self.profile
|
||||
policy_decision.return_value = SimpleNamespace(allowed=True)
|
||||
sources = [_item("file", f"linked/root/file-{index}.txt") for index in range(1, 6)]
|
||||
discover.return_value = SimpleNamespace(
|
||||
files=sources,
|
||||
skipped=[SimpleNamespace(path="linked/root/ignored", reason="Unsupported item")],
|
||||
truncated=False,
|
||||
)
|
||||
downloads = [
|
||||
(
|
||||
source.path,
|
||||
ConnectorDownloadedFile(
|
||||
filename=source.name,
|
||||
data=source.name.encode(),
|
||||
revision=source.etag,
|
||||
),
|
||||
{"source_provenance": {"connector_id": "profile-1"}},
|
||||
)
|
||||
for source in sources
|
||||
]
|
||||
denied_decision = SimpleNamespace(
|
||||
reason="Path denied",
|
||||
to_dict=Mock(return_value={"allowed": False, "reason": "Path denied"}),
|
||||
)
|
||||
downloads[-1] = ConnectorPolicyDenied(denied_decision) # type: ignore[assignment]
|
||||
download.side_effect = downloads
|
||||
sync_file.side_effect = [
|
||||
(self._stored(1), "created", None),
|
||||
(self._stored(2), "updated", "previous-2"),
|
||||
(self._stored(3), "unchanged", "previous-3"),
|
||||
FileStorageError("Target file already exists: Imports/file-4.txt"),
|
||||
]
|
||||
asset_response.side_effect = [
|
||||
self._response(1, "created"),
|
||||
self._response(2, "updated"),
|
||||
self._response(3, "unchanged"),
|
||||
]
|
||||
|
||||
response = sync_connector_space_folder(
|
||||
"space-1",
|
||||
FileConnectorFolderSyncRequest(target_folder="Imports"),
|
||||
session=self.session,
|
||||
principal=self.principal, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
{
|
||||
"discovered": 5,
|
||||
"created": 1,
|
||||
"updated": 1,
|
||||
"unchanged": 1,
|
||||
"skipped": 1,
|
||||
"conflicts": 1,
|
||||
"policy_denied": 1,
|
||||
"failed": 0,
|
||||
},
|
||||
response.summary.model_dump(),
|
||||
)
|
||||
self.assertEqual(
|
||||
[
|
||||
"skipped",
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"conflict",
|
||||
"policy_denied",
|
||||
],
|
||||
[item.action for item in response.items],
|
||||
)
|
||||
self.assertEqual("Imports/file-1.txt", response.items[1].target_path)
|
||||
self.assertEqual(
|
||||
{"allowed": False, "reason": "Path denied"},
|
||||
response.items[-1].policy_decision,
|
||||
)
|
||||
self.session.commit.assert_called_once_with()
|
||||
audit_sync.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
self.session,
|
||||
self.principal,
|
||||
self._stored(index).asset,
|
||||
sync_action=action,
|
||||
previous_version_id=previous,
|
||||
)
|
||||
for index, action, previous in (
|
||||
(1, "created", None),
|
||||
(2, "updated", "previous-2"),
|
||||
(3, "unchanged", "previous-3"),
|
||||
)
|
||||
]
|
||||
)
|
||||
summary_details = audit.call_args.kwargs["details"]
|
||||
self.assertEqual(response.summary.model_dump(), summary_details["summary"])
|
||||
self.assertEqual(6, len(summary_details["results"]))
|
||||
self.assertNotIn("detail", summary_details["results"][-1])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Executable
+244
@@ -0,0 +1,244 @@
|
||||
import hashlib
|
||||
import io
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_download_connector_payload,
|
||||
_http_error,
|
||||
)
|
||||
from govoplan_files.backend.schemas import FileConnectorImportRequest
|
||||
from govoplan_files.backend.storage.connector_imports import (
|
||||
ConnectorDownloadedFile,
|
||||
ConnectorRevisionConflict,
|
||||
_read_seafile_file,
|
||||
_read_smb_file,
|
||||
_read_s3_file,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
class ConnectorProvenanceTests(unittest.TestCase):
|
||||
def acquire(self, download, *, expected=None, annotations=None):
|
||||
profile = ConnectorProfile(id="profile", label="Synthetic", provider="s3")
|
||||
request = FileConnectorImportRequest(
|
||||
library_id="bucket",
|
||||
path="source.txt",
|
||||
source_revision=expected,
|
||||
metadata=annotations or {},
|
||||
)
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.route_support.connector_policy_decision",
|
||||
return_value=SimpleNamespace(allowed=True),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.route_support.read_connector_file",
|
||||
return_value=download,
|
||||
),
|
||||
):
|
||||
return _download_connector_payload(profile, request, operation="sync")
|
||||
|
||||
def test_stale_or_unavailable_expected_revision_cannot_label_new_bytes(self):
|
||||
for revision in ("B", None):
|
||||
with (
|
||||
self.subTest(revision=revision),
|
||||
self.assertRaises(ConnectorRevisionConflict) as caught,
|
||||
):
|
||||
self.acquire(
|
||||
ConnectorDownloadedFile(
|
||||
filename="file", data=b"B", revision=revision
|
||||
),
|
||||
expected="A",
|
||||
)
|
||||
self.assertEqual(409, _http_error(caught.exception).status_code)
|
||||
|
||||
def test_actual_digest_is_computed_and_claims_and_annotations_are_not_authority(
|
||||
self,
|
||||
):
|
||||
annotations = {
|
||||
"profile_id": "forged",
|
||||
"size": 999,
|
||||
"checksum_sha256": "caller-claim",
|
||||
"acquired_sha256": "forged",
|
||||
"note": "keep this",
|
||||
}
|
||||
download = ConnectorDownloadedFile(
|
||||
filename="source.txt",
|
||||
data=b"actual",
|
||||
revision="version-B",
|
||||
metadata={"etag": "etag-B", "checksum_sha256": "provider-claim"},
|
||||
)
|
||||
_, actual, metadata = self.acquire(
|
||||
download, expected="etag-B", annotations=annotations
|
||||
)
|
||||
self.assertEqual("version-B", metadata["source_revision"])
|
||||
evidence = metadata["source_provenance"]["metadata"]
|
||||
self.assertEqual(
|
||||
hashlib.sha256(actual.data).hexdigest(), evidence["acquired_sha256"]
|
||||
)
|
||||
self.assertEqual(len(actual.data), evidence["size"])
|
||||
self.assertEqual("profile", evidence["profile_id"])
|
||||
self.assertNotIn("checksum_sha256", evidence)
|
||||
self.assertEqual(
|
||||
{"checksum_sha256": "provider-claim"}, evidence["provider_checksum_claims"]
|
||||
)
|
||||
self.assertEqual(annotations, evidence["import_annotations"])
|
||||
self.assertEqual(
|
||||
metadata,
|
||||
self.acquire(download, expected="version-B", annotations=annotations)[2],
|
||||
)
|
||||
|
||||
def test_caller_fields_stay_annotations_when_provider_omits_evidence(self):
|
||||
annotations = {
|
||||
"sha256": "caller-hash",
|
||||
"etag": "caller-etag",
|
||||
"mtime": "caller-modified-time",
|
||||
"checksum_sha256": "caller-checksum",
|
||||
"provider_checksum_claims": {"checksum_sha256": "caller-claim"},
|
||||
"connector_space_id": "browse-space",
|
||||
"browse_etag": "browse-revision",
|
||||
"folder_sync": True,
|
||||
"note": {"values": ["001", " keep spaces ", None]},
|
||||
}
|
||||
download = ConnectorDownloadedFile(filename="source.txt", data=b"actual")
|
||||
_, _, metadata = self.acquire(download, annotations=annotations)
|
||||
evidence = metadata["source_provenance"]["metadata"]
|
||||
self.assertEqual(annotations, evidence["import_annotations"])
|
||||
self.assertEqual({}, evidence["provider_checksum_claims"])
|
||||
self.assertEqual(
|
||||
hashlib.sha256(download.data).hexdigest(), evidence["acquired_sha256"]
|
||||
)
|
||||
self.assertEqual(
|
||||
{
|
||||
"profile_id", "library_id", "library_path", "size",
|
||||
"acquired_sha256", "provider_checksum_claims", "import_annotations",
|
||||
},
|
||||
set(evidence),
|
||||
)
|
||||
|
||||
def test_seafile_rechecks_revision_after_download(self):
|
||||
profile = ConnectorProfile(
|
||||
id="profile",
|
||||
label="Synthetic",
|
||||
provider="seafile",
|
||||
endpoint_url="https://example.invalid",
|
||||
)
|
||||
before = {"id": "A", "name": "source.txt", "size": 4}
|
||||
for after in ({**before, "id": "B"}, before):
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._seafile_headers",
|
||||
return_value={},
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._request_json",
|
||||
side_effect=[before, "https://example.invalid/download", after],
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports.request_connector_bytes",
|
||||
return_value=SimpleNamespace(
|
||||
status_code=200, content=b"DATA", headers={}
|
||||
),
|
||||
),
|
||||
):
|
||||
if after["id"] == "B":
|
||||
with self.assertRaises(ConnectorRevisionConflict):
|
||||
_read_seafile_file(
|
||||
profile, library_id="repo", path="source.txt", max_bytes=10
|
||||
)
|
||||
else:
|
||||
result = _read_seafile_file(
|
||||
profile, library_id="repo", path="source.txt", max_bytes=10
|
||||
)
|
||||
self.assertEqual(("A", b"DATA"), (result.revision, result.data))
|
||||
|
||||
def test_smb_observes_stable_metadata_while_write_and_delete_sharing_are_denied(
|
||||
self,
|
||||
):
|
||||
profile = ConnectorProfile(id="profile", label="Synthetic", provider="smb")
|
||||
before = SimpleNamespace(st_size=4, st_mtime_ns=10, st_mtime=1, st_ino=1)
|
||||
for after in (
|
||||
SimpleNamespace(st_size=4, st_mtime_ns=11, st_mtime=1, st_ino=1),
|
||||
before,
|
||||
):
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smb_location",
|
||||
return_value=SimpleNamespace(
|
||||
share="share", server="example.invalid", port=445
|
||||
),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smb_unc_path",
|
||||
return_value="synthetic",
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smb_client_kwargs",
|
||||
return_value={},
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smbclient_module"
|
||||
) as factory,
|
||||
):
|
||||
sdk = factory.return_value
|
||||
sdk.stat.side_effect = [before, after]
|
||||
sdk.open_file.return_value = io.BytesIO(b"DATA")
|
||||
if after is before:
|
||||
self.assertEqual(
|
||||
b"DATA",
|
||||
_read_smb_file(profile, path="source.txt", max_bytes=10).data,
|
||||
)
|
||||
else:
|
||||
with self.assertRaises(ConnectorRevisionConflict):
|
||||
_read_smb_file(profile, path="source.txt", max_bytes=10)
|
||||
self.assertEqual("r", sdk.open_file.call_args.kwargs["share_access"])
|
||||
|
||||
def test_s3_conditional_or_versioned_read_and_response_binding(self):
|
||||
profile = ConnectorProfile(id="profile", label="Synthetic", provider="s3")
|
||||
for version in (None, "version-A", "null"):
|
||||
for returned_etag in ("etag-A", "etag-B"):
|
||||
with (
|
||||
self.subTest(version=version, etag=returned_etag),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._s3_import_client"
|
||||
) as factory,
|
||||
):
|
||||
client = factory.return_value
|
||||
client.head_object.return_value = {
|
||||
"ContentLength": 4,
|
||||
"ETag": "etag-A",
|
||||
"VersionId": version,
|
||||
}
|
||||
body = io.BytesIO(b"DATA")
|
||||
client.get_object.return_value = {
|
||||
"Body": body,
|
||||
"ETag": returned_etag,
|
||||
"VersionId": version,
|
||||
}
|
||||
if returned_etag == "etag-A":
|
||||
result = _read_s3_file(
|
||||
profile,
|
||||
library_id="bucket",
|
||||
path="source.txt",
|
||||
max_bytes=10,
|
||||
)
|
||||
self.assertEqual(version if version and version != "null" else "etag-A", result.revision)
|
||||
else:
|
||||
with self.assertRaises(ConnectorRevisionConflict):
|
||||
_read_s3_file(
|
||||
profile,
|
||||
library_id="bucket",
|
||||
path="source.txt",
|
||||
max_bytes=10,
|
||||
)
|
||||
request = client.get_object.call_args.kwargs
|
||||
immutable = version and version != "null"
|
||||
self.assertEqual(version if immutable else "etag-A", request["VersionId" if immutable else "IfMatch"])
|
||||
self.assertTrue(body.closed)
|
||||
client.close.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,101 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.connector_spaces import (
|
||||
soft_delete_connector_space,
|
||||
validate_connector_space_write_mode,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorSpaceDeletionTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.session = Mock()
|
||||
self.space = SimpleNamespace(
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="owner-1",
|
||||
owner_group_id=None,
|
||||
deleted_at=None,
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
@patch("govoplan_files.backend.storage.connector_spaces.utcnow")
|
||||
@patch("govoplan_files.backend.storage.connector_spaces.ensure_owner_access")
|
||||
def test_owner_can_soft_delete_local_space_definition(
|
||||
self, ensure_owner_access: Mock, utcnow: Mock
|
||||
) -> None:
|
||||
deleted_at = object()
|
||||
utcnow.return_value = deleted_at
|
||||
|
||||
result = soft_delete_connector_space(
|
||||
self.session, self.space, user_id="owner-1"
|
||||
)
|
||||
|
||||
self.assertIs(result, self.space)
|
||||
self.assertIs(deleted_at, self.space.deleted_at)
|
||||
self.assertFalse(self.space.is_active)
|
||||
ensure_owner_access.assert_called_once_with(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="owner-1",
|
||||
user_id="owner-1",
|
||||
is_admin=False,
|
||||
)
|
||||
self.session.add.assert_called_once_with(self.space)
|
||||
self.session.flush.assert_called_once_with()
|
||||
|
||||
@patch(
|
||||
"govoplan_files.backend.storage.connector_spaces.ensure_owner_access",
|
||||
side_effect=FileStorageError("File space access denied"),
|
||||
)
|
||||
def test_inaccessible_space_is_blocked_without_mutation(
|
||||
self, _ensure_owner_access: Mock
|
||||
) -> None:
|
||||
with self.assertRaisesRegex(FileStorageError, "access denied"):
|
||||
soft_delete_connector_space(
|
||||
self.session, self.space, user_id="other-user"
|
||||
)
|
||||
|
||||
self.assertIsNone(self.space.deleted_at)
|
||||
self.assertTrue(self.space.is_active)
|
||||
self.session.add.assert_not_called()
|
||||
self.session.flush.assert_not_called()
|
||||
|
||||
def test_read_only_mode_does_not_require_remote_write_capability(self) -> None:
|
||||
profile = ConnectorProfile(
|
||||
id="webdav-read",
|
||||
label="Read only",
|
||||
provider="webdav",
|
||||
capabilities=("browse", "import"),
|
||||
)
|
||||
|
||||
validate_connector_space_write_mode(profile, read_only=True)
|
||||
|
||||
def test_two_way_mode_requires_explicit_supported_write_capability(self) -> None:
|
||||
missing_capability = ConnectorProfile(
|
||||
id="s3-read",
|
||||
label="S3 read only",
|
||||
provider="s3",
|
||||
capabilities=("browse",),
|
||||
)
|
||||
unsupported_provider = ConnectorProfile(
|
||||
id="webdav-write",
|
||||
label="WebDAV write",
|
||||
provider="webdav",
|
||||
capabilities=("write",),
|
||||
)
|
||||
|
||||
for profile in (missing_capability, unsupported_provider):
|
||||
with self.subTest(profile=profile.id):
|
||||
with self.assertRaisesRegex(FileStorageError, "requires an S3"):
|
||||
validate_connector_space_write_mode(profile, read_only=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,7 @@
|
||||
from govoplan_files.backend.manifest import get_manifest
|
||||
|
||||
|
||||
def test_static_documentation_has_complete_german_reference_copy() -> None:
|
||||
for topic in get_manifest().documentation:
|
||||
german = topic.translations.get("de", {})
|
||||
assert all(german.get(field, "").strip() for field in ("title", "summary", "body")), topic.id
|
||||
@@ -0,0 +1,468 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_access.backend.db.models import Account, Group, User
|
||||
from govoplan_core.core.dsar import DsarProvider, DsarSubjectRef
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.privacy.dsar_workflow import (
|
||||
DataSubjectRequest,
|
||||
create_data_subject_request,
|
||||
execute_data_subject_erasure,
|
||||
plan_data_subject_erasure,
|
||||
search_data_subject_request,
|
||||
)
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.dsar_provider import (
|
||||
FILES_DSAR_CAPABILITY,
|
||||
FilesDsarProvider,
|
||||
)
|
||||
from govoplan_files.backend.manifest import manifest
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, provider: FilesDsarProvider, *, files_active: bool = True) -> None:
|
||||
self.provider = provider
|
||||
self.files_active = files_active
|
||||
|
||||
def capability_names(self):
|
||||
return (FILES_DSAR_CAPABILITY,)
|
||||
|
||||
def capability_owner(self, name):
|
||||
self._assert_capability(name)
|
||||
return "files"
|
||||
|
||||
def tenant_entitlement_resolver(self):
|
||||
files_active = self.files_active
|
||||
|
||||
class _Resolver:
|
||||
@staticmethod
|
||||
def resolve(session, tenant_id):
|
||||
del session, tenant_id
|
||||
return type(
|
||||
"State",
|
||||
(),
|
||||
{"effective_modules": ("files",) if files_active else ()},
|
||||
)()
|
||||
|
||||
return _Resolver()
|
||||
|
||||
def require_tenant_capability(self, name, session, **kwargs):
|
||||
del session, kwargs
|
||||
self._assert_capability(name)
|
||||
return self.provider
|
||||
|
||||
def manifests(self):
|
||||
return (type("Manifest", (), {"id": "files"})(),)
|
||||
|
||||
@staticmethod
|
||||
def _assert_capability(name: str) -> None:
|
||||
if name != FILES_DSAR_CAPABILITY:
|
||||
raise KeyError(name)
|
||||
|
||||
|
||||
class FilesDsarProviderTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite:///:memory:", future=True)
|
||||
Base.metadata.create_all(
|
||||
bind=self.engine,
|
||||
tables=[
|
||||
Account.__table__,
|
||||
User.__table__,
|
||||
Group.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
DataSubjectRequest.__table__,
|
||||
FileBlob.__table__,
|
||||
FileAsset.__table__,
|
||||
FileVersion.__table__,
|
||||
FileFolder.__table__,
|
||||
FileFormEvidenceGrant.__table__,
|
||||
FileShare.__table__,
|
||||
FileConnectorCredential.__table__,
|
||||
FileConnectorPolicy.__table__,
|
||||
FileConnectorProfile.__table__,
|
||||
FileConnectorSpace.__table__,
|
||||
FileIntegrityScan.__table__,
|
||||
FileIntegrityFinding.__table__,
|
||||
],
|
||||
)
|
||||
self.session = sessionmaker(bind=self.engine, future=True)()
|
||||
self.account = Account(
|
||||
id="account-1",
|
||||
email="subject@example.test",
|
||||
normalized_email="subject@example.test",
|
||||
display_name="Subject",
|
||||
password_hash="not-exported",
|
||||
)
|
||||
self.user = User(
|
||||
id="membership-1",
|
||||
tenant_id="tenant-1",
|
||||
account_id=self.account.id,
|
||||
email="subject@example.test",
|
||||
display_name="Subject",
|
||||
)
|
||||
self.blob = FileBlob(
|
||||
id="blob-1",
|
||||
tenant_id="tenant-1",
|
||||
storage_backend="local",
|
||||
storage_key="private/storage/key-do-not-export",
|
||||
checksum_sha256="a" * 64,
|
||||
size_bytes=12,
|
||||
ref_count=1,
|
||||
)
|
||||
self.asset = FileAsset(
|
||||
id="asset-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id=self.user.id,
|
||||
created_by_user_id=self.user.id,
|
||||
current_version_id="version-1",
|
||||
display_path="subjects/private.txt",
|
||||
filename="private.txt",
|
||||
description="Subject-provided document",
|
||||
retained_until=datetime.now(timezone.utc) + timedelta(days=30),
|
||||
lifecycle_reason="Pending proceeding",
|
||||
metadata_={"password": "metadata-secret-do-not-export"},
|
||||
)
|
||||
self.version = FileVersion(
|
||||
id="version-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id=self.asset.id,
|
||||
blob_id=self.blob.id,
|
||||
version_number=1,
|
||||
filename_at_upload=self.asset.filename,
|
||||
display_path_at_upload=self.asset.display_path,
|
||||
content_type="text/plain",
|
||||
size_bytes=12,
|
||||
checksum_sha256="a" * 64,
|
||||
created_by_user_id=self.user.id,
|
||||
)
|
||||
self.folder = FileFolder(
|
||||
id="folder-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id=self.user.id,
|
||||
path="subjects",
|
||||
created_by_user_id=self.user.id,
|
||||
)
|
||||
self.share = FileShare(
|
||||
id="share-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id=self.asset.id,
|
||||
target_type="user",
|
||||
target_id=self.user.id,
|
||||
permission="read",
|
||||
created_by_user_id=self.user.id,
|
||||
)
|
||||
self.form_evidence = FileFormEvidenceGrant(
|
||||
id="evidence-1",
|
||||
tenant_id="tenant-1",
|
||||
form_instance_id="form-instance-1",
|
||||
form_definition_id="application",
|
||||
form_definition_revision="7",
|
||||
token_sha256="b" * 64,
|
||||
idempotency_key="evidence-key-1",
|
||||
request_sha256="c" * 64,
|
||||
custodian_user_id=self.user.id,
|
||||
evidence_kind="attachment",
|
||||
purpose="Submitted application evidence",
|
||||
status="uploaded",
|
||||
expires_at=datetime.now(timezone.utc) + timedelta(days=1),
|
||||
max_size_bytes=1024,
|
||||
allowed_content_types=["text/plain"],
|
||||
file_asset_id=self.asset.id,
|
||||
file_version_id=self.version.id,
|
||||
metadata_={},
|
||||
)
|
||||
self.credential = FileConnectorCredential(
|
||||
id="credential-1",
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
scope_id="tenant-1",
|
||||
label="Subject-created credential",
|
||||
provider="s3",
|
||||
credential_mode="database",
|
||||
username="subject-user",
|
||||
password_encrypted="encrypted-password-do-not-export",
|
||||
token_encrypted="encrypted-token-do-not-export",
|
||||
password_env="PASSWORD_ENV_DO_NOT_EXPORT",
|
||||
secret_ref="vault://do-not-export",
|
||||
created_by_user_id=self.user.id,
|
||||
updated_by_user_id=self.user.id,
|
||||
)
|
||||
self.integrity_scan = FileIntegrityScan(
|
||||
id="scan-1",
|
||||
tenant_id="tenant-1",
|
||||
storage_backend="local",
|
||||
storage_prefix="private-prefix-do-not-export",
|
||||
created_by_user_id=self.user.id,
|
||||
)
|
||||
tenant_two_asset = FileAsset(
|
||||
id="asset-tenant-2",
|
||||
tenant_id="tenant-2",
|
||||
owner_type="user",
|
||||
owner_user_id=self.user.id,
|
||||
display_path="other-tenant.txt",
|
||||
filename="other-tenant.txt",
|
||||
)
|
||||
self.session.add_all(
|
||||
[
|
||||
self.account,
|
||||
self.user,
|
||||
self.blob,
|
||||
self.asset,
|
||||
self.version,
|
||||
self.folder,
|
||||
self.share,
|
||||
self.form_evidence,
|
||||
self.credential,
|
||||
self.integrity_scan,
|
||||
tenant_two_asset,
|
||||
]
|
||||
)
|
||||
self.session.commit()
|
||||
self.provider = FilesDsarProvider()
|
||||
self.subject = DsarSubjectRef(membership_id=self.user.id)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def test_manifest_publishes_protocol_conforming_provider(self) -> None:
|
||||
provided_names = {item.name for item in manifest.provides_interfaces}
|
||||
self.assertIn(FILES_DSAR_CAPABILITY, provided_names)
|
||||
provider = manifest.capability_factories[FILES_DSAR_CAPABILITY](None)
|
||||
self.assertIsInstance(provider, DsarProvider)
|
||||
|
||||
def test_search_is_tenant_scoped_and_excludes_bytes_and_secrets(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
)
|
||||
resource_types = {record.resource_type for record in records}
|
||||
self.assertTrue(
|
||||
{
|
||||
"file_asset",
|
||||
"file_version",
|
||||
"file_folder",
|
||||
"file_share",
|
||||
"file_form_evidence",
|
||||
"connector_credential",
|
||||
"file_integrity_scan",
|
||||
}.issubset(resource_types)
|
||||
)
|
||||
serialized = repr([record.to_dict() for record in records])
|
||||
self.assertNotIn("asset-tenant-2", serialized)
|
||||
self.assertNotIn("private/storage/key-do-not-export", serialized)
|
||||
self.assertNotIn("metadata-secret-do-not-export", serialized)
|
||||
self.assertNotIn("encrypted-password-do-not-export", serialized)
|
||||
self.assertNotIn("encrypted-token-do-not-export", serialized)
|
||||
self.assertNotIn("subject-user", serialized)
|
||||
self.assertNotIn("PASSWORD_ENV_DO_NOT_EXPORT", serialized)
|
||||
self.assertNotIn("vault://do-not-export", serialized)
|
||||
self.assertNotIn("private-prefix-do-not-export", serialized)
|
||||
|
||||
def test_conflicting_direct_subject_references_fail_closed(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
membership_id=self.user.id,
|
||||
external_references={"files.user": "another-membership"},
|
||||
),
|
||||
)
|
||||
|
||||
self.assertEqual((), records)
|
||||
|
||||
def test_plan_classifies_retention_and_manual_review(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
records=records,
|
||||
)
|
||||
|
||||
kinds = {action.kind for action in actions}
|
||||
self.assertTrue({"retain", "manual_review", "revoke", "detach"}.issubset(kinds))
|
||||
asset_retention = next(
|
||||
action
|
||||
for action in actions
|
||||
if action.action_id == "files:retain:file_asset:asset-1"
|
||||
)
|
||||
self.assertIn("retained until", asset_retention.rationale)
|
||||
self.assertTrue(
|
||||
any(action.action_id == "files:revoke:file_share:share-1" for action in actions)
|
||||
)
|
||||
self.assertFalse(
|
||||
any(action.kind == "delete" and action.executable for action in actions)
|
||||
)
|
||||
|
||||
def test_execution_is_revalidated_and_idempotent(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
records=records,
|
||||
)
|
||||
executable = tuple(action for action in actions if action.executable)
|
||||
|
||||
first = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
actions=executable,
|
||||
request_id="dsar-1",
|
||||
)
|
||||
self.assertEqual({"executed"}, {result.status for result in first})
|
||||
self.assertIsNotNone(self.share.revoked_at)
|
||||
self.assertIsNone(self.asset.created_by_user_id)
|
||||
self.assertIsNone(self.version.created_by_user_id)
|
||||
self.assertIsNone(self.folder.created_by_user_id)
|
||||
self.assertIsNone(self.credential.created_by_user_id)
|
||||
self.assertIsNone(self.credential.updated_by_user_id)
|
||||
|
||||
repeated = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
actions=executable,
|
||||
request_id="dsar-1",
|
||||
)
|
||||
self.assertEqual({"unchanged"}, {result.status for result in repeated})
|
||||
|
||||
def test_execution_blocks_when_reference_changed_after_planning(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
records=records,
|
||||
)
|
||||
detach = next(
|
||||
action
|
||||
for action in actions
|
||||
if action.action_id
|
||||
== "files:detach:file_version:created_by_user_id:version-1"
|
||||
)
|
||||
self.version.created_by_user_id = "replacement-user"
|
||||
self.session.flush()
|
||||
|
||||
result = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self.subject,
|
||||
actions=(detach,),
|
||||
request_id="dsar-2",
|
||||
)
|
||||
|
||||
self.assertEqual("blocked", result[0].status)
|
||||
self.assertEqual("replacement-user", self.version.created_by_user_id)
|
||||
|
||||
def test_core_workflow_discovers_active_provider_and_skips_it_when_disabled(
|
||||
self,
|
||||
) -> None:
|
||||
request = create_data_subject_request(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
reference="DSAR-FILES-1",
|
||||
request_kind="access_and_erasure",
|
||||
subject=self.subject,
|
||||
purpose="Respond to an authorized privacy request.",
|
||||
legal_basis="Article 15 and 17 GDPR",
|
||||
due_at=None,
|
||||
requested_by_account_id="privacy-officer",
|
||||
)
|
||||
self.session.commit()
|
||||
registry = _Registry(self.provider)
|
||||
|
||||
search_data_subject_request(
|
||||
self.session,
|
||||
registry=registry,
|
||||
row=request,
|
||||
expected_revision=1,
|
||||
)
|
||||
self.assertEqual(
|
||||
"searched", request.status, request.search_result["provider_runs"]
|
||||
)
|
||||
self.assertEqual(["files"], request.coverage["covered_modules"])
|
||||
self.assertEqual([], request.coverage["modules_without_provider"])
|
||||
plan_data_subject_erasure(
|
||||
self.session,
|
||||
registry=registry,
|
||||
row=request,
|
||||
expected_revision=2,
|
||||
)
|
||||
executable_ids = [
|
||||
action["action_id"]
|
||||
for action in request.erasure_plan["actions"]
|
||||
if action["executable"]
|
||||
]
|
||||
execute_data_subject_erasure(
|
||||
self.session,
|
||||
registry=registry,
|
||||
row=request,
|
||||
expected_revision=3,
|
||||
action_ids=executable_ids,
|
||||
)
|
||||
self.assertEqual("completed", request.status)
|
||||
|
||||
disabled = create_data_subject_request(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
reference="DSAR-FILES-DISABLED",
|
||||
request_kind="access",
|
||||
subject=self.subject,
|
||||
purpose="Verify disabled-module coverage.",
|
||||
legal_basis="Article 15 GDPR",
|
||||
due_at=None,
|
||||
requested_by_account_id="privacy-officer",
|
||||
)
|
||||
search_data_subject_request(
|
||||
self.session,
|
||||
registry=_Registry(self.provider, files_active=False),
|
||||
row=disabled,
|
||||
expected_revision=1,
|
||||
)
|
||||
|
||||
self.assertEqual(0, disabled.search_result["record_count"])
|
||||
self.assertEqual(
|
||||
[FILES_DSAR_CAPABILITY],
|
||||
disabled.coverage["inactive_provider_capabilities"],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,164 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timedelta
|
||||
import unittest
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_access.backend.db.models import Account, Group, User
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.lifecycle import (
|
||||
preview_asset_purge,
|
||||
restore_asset,
|
||||
set_asset_lifecycle,
|
||||
)
|
||||
|
||||
|
||||
class FileLifecycleTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite:///:memory:", future=True)
|
||||
Base.metadata.create_all(
|
||||
bind=self.engine,
|
||||
tables=[
|
||||
Account.__table__,
|
||||
User.__table__,
|
||||
Group.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
FileBlob.__table__,
|
||||
FileAsset.__table__,
|
||||
FileVersion.__table__,
|
||||
FileFolder.__table__,
|
||||
FileFormEvidenceGrant.__table__,
|
||||
FileShare.__table__,
|
||||
],
|
||||
)
|
||||
self.session = sessionmaker(bind=self.engine, future=True)()
|
||||
self.blob = FileBlob(
|
||||
id="blob-1",
|
||||
tenant_id="tenant-1",
|
||||
storage_backend="local",
|
||||
storage_key="tenants/tenant-1/files/blob-1",
|
||||
checksum_sha256="a" * 64,
|
||||
size_bytes=1,
|
||||
ref_count=1,
|
||||
)
|
||||
self.asset = FileAsset(
|
||||
id="file-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="user-1",
|
||||
current_version_id="version-1",
|
||||
display_path="records/report.txt",
|
||||
filename="report.txt",
|
||||
deleted_at=utcnow(),
|
||||
)
|
||||
self.version = FileVersion(
|
||||
id="version-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id="file-1",
|
||||
blob_id="blob-1",
|
||||
version_number=1,
|
||||
filename_at_upload="report.txt",
|
||||
display_path_at_upload="records/report.txt",
|
||||
size_bytes=1,
|
||||
checksum_sha256="a" * 64,
|
||||
)
|
||||
self.session.add_all([self.blob, self.asset, self.version])
|
||||
self.session.commit()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def test_preview_hash_changes_with_retention_and_blocks_active_policy(self) -> None:
|
||||
first = preview_asset_purge(
|
||||
self.session, tenant_id="tenant-1", file_ids=[self.asset.id]
|
||||
)
|
||||
self.assertTrue(first.eligible)
|
||||
|
||||
set_asset_lifecycle(
|
||||
self.session,
|
||||
self.asset,
|
||||
retained_until=utcnow() + timedelta(days=7),
|
||||
legal_hold=True,
|
||||
reason="Active proceeding",
|
||||
expected_revision=1,
|
||||
)
|
||||
self.session.commit()
|
||||
second = preview_asset_purge(
|
||||
self.session, tenant_id="tenant-1", file_ids=[self.asset.id]
|
||||
)
|
||||
|
||||
self.assertFalse(second.eligible)
|
||||
self.assertNotEqual(first.preview_sha256, second.preview_sha256)
|
||||
self.assertEqual(
|
||||
{"legal_hold", "retention_active"}, set(second.items[0].blockers)
|
||||
)
|
||||
|
||||
def test_restore_preserves_version_and_provenance_and_rejects_collision(self) -> None:
|
||||
self.asset.metadata_ = {"source": {"provider": "s3", "revision": "v1"}}
|
||||
self.session.commit()
|
||||
|
||||
self.assertTrue(restore_asset(self.session, self.asset))
|
||||
self.assertEqual("version-1", self.asset.current_version_id)
|
||||
self.assertEqual("v1", self.asset.metadata_["source"]["revision"])
|
||||
self.session.commit()
|
||||
|
||||
self.asset.deleted_at = utcnow()
|
||||
collision = FileAsset(
|
||||
id="file-2",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="user-1",
|
||||
display_path=self.asset.display_path,
|
||||
filename=self.asset.filename,
|
||||
)
|
||||
self.session.add(collision)
|
||||
self.session.commit()
|
||||
with self.assertRaisesRegex(FileStorageError, "path is already in use"):
|
||||
restore_asset(self.session, self.asset)
|
||||
|
||||
def test_active_share_blocks_purge_until_revoked(self) -> None:
|
||||
share = FileShare(
|
||||
id="share-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id=self.asset.id,
|
||||
target_type="tenant",
|
||||
target_id="tenant-1",
|
||||
permission="read",
|
||||
)
|
||||
self.session.add(share)
|
||||
self.session.commit()
|
||||
|
||||
blocked = preview_asset_purge(
|
||||
self.session, tenant_id="tenant-1", file_ids=[self.asset.id]
|
||||
)
|
||||
self.assertIn("active_share", blocked.items[0].blockers)
|
||||
|
||||
share.revoked_at = utcnow()
|
||||
self.session.commit()
|
||||
allowed = preview_asset_purge(
|
||||
self.session, tenant_id="tenant-1", file_ids=[self.asset.id]
|
||||
)
|
||||
self.assertTrue(allowed.eligible)
|
||||
|
||||
def test_missing_campaign_table_is_an_optional_module_safe_path(self) -> None:
|
||||
preview = preview_asset_purge(
|
||||
self.session, tenant_id="tenant-1", file_ids=[self.asset.id]
|
||||
)
|
||||
self.assertNotIn("campaign_evidence", preview.items[0].blockers)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,333 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from io import BytesIO
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from fastapi import UploadFile
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
from starlette.datastructures import Headers
|
||||
|
||||
from govoplan_access.backend.db.models import Account, User
|
||||
from govoplan_core.core.access import (
|
||||
CAPABILITY_ACCESS_DIRECTORY,
|
||||
AccessSubjectRef,
|
||||
UserRef,
|
||||
)
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.core.form_evidence import (
|
||||
FormEvidenceContractError,
|
||||
FormEvidenceGrantRequest,
|
||||
FormEvidenceInspectionRequest,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference, InstitutionalReference
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileFormEvidenceGrant,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.form_evidence import FilesFormEvidenceProvider
|
||||
from govoplan_files.backend.routes.form_evidence import upload_form_evidence
|
||||
|
||||
|
||||
class _Directory:
|
||||
def __init__(self, user: UserRef) -> None:
|
||||
self._user = user
|
||||
|
||||
def get_account(self, account_id: str):
|
||||
del account_id
|
||||
return None
|
||||
|
||||
def get_user(self, user_id: str):
|
||||
return self._user if user_id == self._user.id else None
|
||||
|
||||
def get_users(self, user_ids):
|
||||
return {user_id: self._user for user_id in user_ids if user_id == self._user.id}
|
||||
|
||||
def users_for_tenant(self, tenant_id: str):
|
||||
return (self._user,) if tenant_id == self._user.tenant_id else ()
|
||||
|
||||
def get_group(self, group_id: str):
|
||||
del group_id
|
||||
return None
|
||||
|
||||
def get_groups(self, group_ids):
|
||||
del group_ids
|
||||
return {}
|
||||
|
||||
def groups_for_tenant(self, tenant_id: str):
|
||||
del tenant_id
|
||||
return ()
|
||||
|
||||
def groups_for_user(self, user_id: str, *, tenant_id: str):
|
||||
del user_id, tenant_id
|
||||
return ()
|
||||
|
||||
def display_label(self, subject: AccessSubjectRef):
|
||||
del subject
|
||||
return self._user.display_name
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, directory: _Directory) -> None:
|
||||
self._directory = directory
|
||||
|
||||
def has_capability(self, name: str) -> bool:
|
||||
return name == CAPABILITY_ACCESS_DIRECTORY
|
||||
|
||||
def require_capability(self, name: str):
|
||||
if not self.has_capability(name):
|
||||
raise KeyError(name)
|
||||
return self._directory
|
||||
|
||||
|
||||
class FilesFormEvidenceTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(
|
||||
self.engine,
|
||||
tables=[
|
||||
Account.__table__,
|
||||
User.__table__,
|
||||
FileBlob.__table__,
|
||||
FileAsset.__table__,
|
||||
FileVersion.__table__,
|
||||
FileFormEvidenceGrant.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
],
|
||||
)
|
||||
self.session = Session(self.engine)
|
||||
user_ref = UserRef(
|
||||
id="user-1",
|
||||
account_id="account-1",
|
||||
tenant_id="tenant-1",
|
||||
email="user@example.test",
|
||||
display_name="Evidence Custodian",
|
||||
)
|
||||
self.provider = FilesFormEvidenceProvider(
|
||||
_Registry(_Directory(user_ref)),
|
||||
SimpleNamespace(file_upload_max_bytes=2_000_000),
|
||||
)
|
||||
self.principal = SimpleNamespace(tenant_id="tenant-1")
|
||||
self.definition_ref = InstitutionalReference(
|
||||
kind="form",
|
||||
owner_module="forms",
|
||||
object_id="form-1",
|
||||
tenant_id="tenant-1",
|
||||
version="3",
|
||||
)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def request(self, **overrides: object) -> FormEvidenceGrantRequest:
|
||||
values: dict[str, object] = {
|
||||
"tenant_id": "tenant-1",
|
||||
"instance_id": "instance-1",
|
||||
"definition_ref": self.definition_ref,
|
||||
"evidence_kind": "document",
|
||||
"purpose": "supporting document",
|
||||
"idempotency_key": "grant-1",
|
||||
"expires_at": datetime.now(UTC) + timedelta(hours=1),
|
||||
"custodian_ref": "user:user-1",
|
||||
"max_size_bytes": 1_000_000,
|
||||
"allowed_content_types": ("application/pdf",),
|
||||
}
|
||||
values.update(overrides)
|
||||
return FormEvidenceGrantRequest(**values) # type: ignore[arg-type]
|
||||
|
||||
def test_grant_persists_only_token_hash_and_replays_without_secret(self) -> None:
|
||||
request = self.request()
|
||||
issued = self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=request,
|
||||
)
|
||||
row = self.session.query(FileFormEvidenceGrant).one()
|
||||
|
||||
self.assertIsNotNone(issued.upload_token)
|
||||
self.assertNotEqual(issued.upload_token, row.token_sha256)
|
||||
self.assertLessEqual(
|
||||
issued.expires_at,
|
||||
datetime.now(UTC) + timedelta(minutes=15, seconds=1),
|
||||
)
|
||||
replay = self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=request,
|
||||
)
|
||||
self.assertTrue(replay.replayed)
|
||||
self.assertIsNone(replay.upload_token)
|
||||
self.assertEqual(issued.grant_id, replay.grant_id)
|
||||
|
||||
def test_grant_capacity_counts_outstanding_uploads_but_allows_replay(self) -> None:
|
||||
request = self.request(metadata={"remaining_attachments": 1})
|
||||
self.provider.create_upload_grant(self.session, self.principal, request=request)
|
||||
|
||||
replay = self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=request,
|
||||
)
|
||||
self.assertTrue(replay.replayed)
|
||||
with self.assertRaisesRegex(
|
||||
FormEvidenceContractError,
|
||||
"maximum number of active attachment uploads",
|
||||
):
|
||||
self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=self.request(
|
||||
idempotency_key="grant-2",
|
||||
metadata={"remaining_attachments": 1},
|
||||
),
|
||||
)
|
||||
|
||||
def test_idempotency_key_cannot_be_reused_for_another_request(self) -> None:
|
||||
self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=self.request(),
|
||||
)
|
||||
with self.assertRaisesRegex(
|
||||
FormEvidenceContractError,
|
||||
"idempotency conflict",
|
||||
):
|
||||
self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=self.request(purpose="another purpose"),
|
||||
)
|
||||
|
||||
def test_inspection_accepts_only_exact_verified_managed_version(self) -> None:
|
||||
issued = self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=self.request(),
|
||||
)
|
||||
grant = self.session.get(FileFormEvidenceGrant, issued.grant_id)
|
||||
assert grant is not None
|
||||
blob = FileBlob(
|
||||
id="blob-1",
|
||||
tenant_id="tenant-1",
|
||||
storage_backend="local",
|
||||
storage_key="tenant-1/blob-1",
|
||||
checksum_sha256="a" * 64,
|
||||
size_bytes=100,
|
||||
integrity_status="verified",
|
||||
)
|
||||
asset = FileAsset(
|
||||
id="asset-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="user-1",
|
||||
current_version_id="version-1",
|
||||
display_path="Form submissions/instance-1/evidence.pdf",
|
||||
filename="evidence.pdf",
|
||||
)
|
||||
version = FileVersion(
|
||||
id="version-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id="asset-1",
|
||||
blob_id="blob-1",
|
||||
version_number=1,
|
||||
filename_at_upload="evidence.pdf",
|
||||
display_path_at_upload=asset.display_path,
|
||||
content_type="application/pdf",
|
||||
size_bytes=100,
|
||||
checksum_sha256="a" * 64,
|
||||
)
|
||||
self.session.add_all((blob, asset, version))
|
||||
grant.status = "uploaded"
|
||||
grant.file_asset_id = asset.id
|
||||
grant.file_version_id = version.id
|
||||
self.session.flush()
|
||||
reference = EvidenceReference(
|
||||
kind="document",
|
||||
owner_module="files",
|
||||
evidence_id=asset.id,
|
||||
tenant_id="tenant-1",
|
||||
version=version.id,
|
||||
checksum="a" * 64,
|
||||
)
|
||||
|
||||
accepted = self.provider.inspect_evidence(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=FormEvidenceInspectionRequest(
|
||||
tenant_id="tenant-1",
|
||||
instance_id="instance-1",
|
||||
definition_ref=self.definition_ref,
|
||||
evidence=reference,
|
||||
purpose="final submission",
|
||||
final=True,
|
||||
),
|
||||
)
|
||||
self.assertTrue(accepted.accepted)
|
||||
|
||||
rejected = self.provider.inspect_evidence(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=FormEvidenceInspectionRequest(
|
||||
tenant_id="tenant-1",
|
||||
instance_id="another-instance",
|
||||
definition_ref=self.definition_ref,
|
||||
evidence=reference,
|
||||
purpose="final submission",
|
||||
final=True,
|
||||
),
|
||||
)
|
||||
self.assertEqual("rejected", rejected.state)
|
||||
|
||||
def test_public_upload_consumes_grant_and_returns_exact_evidence(self) -> None:
|
||||
issued = self.provider.create_upload_grant(
|
||||
self.session,
|
||||
self.principal,
|
||||
request=self.request(),
|
||||
)
|
||||
assert issued.upload_token is not None
|
||||
stored = SimpleNamespace(
|
||||
asset=SimpleNamespace(id="asset-uploaded"),
|
||||
version=SimpleNamespace(
|
||||
id="version-uploaded",
|
||||
checksum_sha256="c" * 64,
|
||||
size_bytes=8,
|
||||
),
|
||||
)
|
||||
upload = UploadFile(
|
||||
filename="evidence.pdf",
|
||||
file=BytesIO(b"evidence"),
|
||||
headers=Headers({"content-type": "application/pdf"}),
|
||||
)
|
||||
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.routes.form_evidence.create_file_asset",
|
||||
return_value=stored,
|
||||
) as create,
|
||||
patch("govoplan_files.backend.routes.form_evidence.emit_platform_event"),
|
||||
):
|
||||
response = upload_form_evidence(
|
||||
file=upload,
|
||||
x_form_evidence_token=issued.upload_token,
|
||||
session=self.session,
|
||||
)
|
||||
|
||||
grant = self.session.get(FileFormEvidenceGrant, issued.grant_id)
|
||||
assert grant is not None
|
||||
self.assertEqual("uploaded", grant.status)
|
||||
self.assertEqual("asset-uploaded", grant.file_asset_id)
|
||||
self.assertEqual("version-uploaded", grant.file_version_id)
|
||||
self.assertEqual("asset-uploaded", response["evidence"]["evidence_id"])
|
||||
self.assertEqual(b"evidence", create.call_args.kwargs["data"])
|
||||
self.assertEqual("user-1", create.call_args.kwargs["owner_id"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,450 @@
|
||||
"""Managed archive routes: real authorization, preview and extraction pipeline.
|
||||
|
||||
Only external blob persistence, directory membership and audit delivery are
|
||||
isolated; source/version/share checks and destination writes use SQLite.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import tarfile
|
||||
import unittest
|
||||
import zipfile
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
import pyzipper
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.pool import StaticPool
|
||||
|
||||
from govoplan_access.backend.db.models import Account, Group, User
|
||||
from govoplan_core.auth import get_api_principal
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorPolicy,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.routes.managed_archives import router
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.files import create_file_asset
|
||||
|
||||
|
||||
def archive_bytes(entries=None, *, password=None, tar=False):
|
||||
entries = entries or {"folder/one.txt": b"one", "other/two.txt": b"two"}
|
||||
output = io.BytesIO()
|
||||
if tar:
|
||||
with tarfile.open(fileobj=output, mode="w:gz") as archive:
|
||||
for name, data in entries.items():
|
||||
member = tarfile.TarInfo(name)
|
||||
member.size = len(data)
|
||||
archive.addfile(member, io.BytesIO(data))
|
||||
else:
|
||||
archive_type = pyzipper.AESZipFile if password else zipfile.ZipFile
|
||||
options = {"encryption": pyzipper.WZ_AES} if password else {}
|
||||
with archive_type(
|
||||
output, "w", compression=zipfile.ZIP_DEFLATED, **options
|
||||
) as archive:
|
||||
if password:
|
||||
archive.setpassword(password.encode())
|
||||
for name, data in entries.items():
|
||||
archive.writestr(name, data)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
class ManagedArchiveTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine(
|
||||
"sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool
|
||||
)
|
||||
Base.metadata.create_all(
|
||||
self.engine,
|
||||
tables=[
|
||||
model.__table__
|
||||
for model in (
|
||||
Account,
|
||||
User,
|
||||
Group,
|
||||
ChangeSequenceEntry,
|
||||
FileBlob,
|
||||
FileAsset,
|
||||
FileVersion,
|
||||
FileShare,
|
||||
FileConnectorPolicy,
|
||||
)
|
||||
],
|
||||
)
|
||||
self.session = Session(self.engine, expire_on_commit=False)
|
||||
self.addCleanup(self.engine.dispose)
|
||||
self.addCleanup(self.session.close)
|
||||
self.objects = {}
|
||||
self.scopes = {"files:file:read", "files:file:download", "files:file:upload"}
|
||||
self.principal = SimpleNamespace(
|
||||
tenant_id="tenant-1",
|
||||
user=SimpleNamespace(id="user-1"),
|
||||
has=lambda scope: scope in self.scopes,
|
||||
)
|
||||
self.settings = SimpleNamespace(
|
||||
file_upload_zip_max_bytes=10 * 1024 * 1024,
|
||||
file_archive_max_entries=10_000,
|
||||
file_archive_max_expanded_bytes=20 * 1024 * 1024,
|
||||
file_archive_max_expansion_ratio=100,
|
||||
file_archive_preview_ttl_seconds=1800,
|
||||
file_upload_max_bytes=1024 * 1024,
|
||||
)
|
||||
for module in ("uploads", "managed_archives"):
|
||||
self.enterContext(
|
||||
patch(f"govoplan_files.backend.routes.{module}.settings", self.settings)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files.user_group_ids", return_value=[]
|
||||
)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files._get_or_create_blob",
|
||||
side_effect=self.store_blob,
|
||||
)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=SimpleNamespace(
|
||||
name="test", get_bytes=lambda key: self.objects[key]
|
||||
),
|
||||
)
|
||||
)
|
||||
self.audit = self.enterContext(
|
||||
patch("govoplan_files.backend.route_support.audit_from_principal")
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.route_support.asset_is_audit_relevant",
|
||||
return_value=False,
|
||||
)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.route_support._sent_campaign_asset_ids",
|
||||
return_value=set(),
|
||||
)
|
||||
)
|
||||
self.source = self.seed_source(archive_bytes())
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api/v1")
|
||||
app.dependency_overrides[get_api_principal] = lambda: self.principal
|
||||
app.dependency_overrides[get_session] = lambda: self.session
|
||||
self.client = self.enterContext(TestClient(app))
|
||||
|
||||
def store_blob(self, session, *, tenant_id, data, content_type=None, **kwargs):
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
blob = FileBlob(
|
||||
tenant_id=tenant_id,
|
||||
storage_backend="test",
|
||||
storage_key=f"objects/{checksum}",
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
content_type=content_type,
|
||||
ref_count=1,
|
||||
)
|
||||
session.add(blob)
|
||||
session.flush()
|
||||
self.objects[blob.storage_key] = data
|
||||
return blob
|
||||
|
||||
def seed_source(self, data, filename="source.zip"):
|
||||
stored = create_file_asset(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder="archives",
|
||||
)
|
||||
self.session.commit()
|
||||
return stored
|
||||
|
||||
def preview(self, **changes):
|
||||
return self.client.post(
|
||||
f"/api/v1/files/{self.source.asset.id}/archive-preview",
|
||||
json={
|
||||
"source_version_id": self.source.version.id,
|
||||
"path": "extracted",
|
||||
**changes,
|
||||
},
|
||||
)
|
||||
|
||||
def confirm(self, token, **changes):
|
||||
return self.client.post(
|
||||
f"/api/v1/files/{self.source.asset.id}/archive-confirm",
|
||||
json={
|
||||
"source_version_id": self.source.version.id,
|
||||
"path": "extracted",
|
||||
"preview_token": token,
|
||||
"selected_paths": ["folder/one.txt"],
|
||||
**changes,
|
||||
},
|
||||
)
|
||||
|
||||
def token(self):
|
||||
response = self.preview()
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
return response.json()["preview_token"]
|
||||
|
||||
def assert_no_extraction(self):
|
||||
self.assertEqual(1, self.session.query(FileAsset).count())
|
||||
self.assertIsNone(self.source.asset.deleted_at)
|
||||
self.assertEqual(self.source.version.id, self.source.asset.current_version_id)
|
||||
self.audit.assert_not_called()
|
||||
|
||||
def test_preview_is_read_only_and_confirmation_extracts_only_selected_members(self):
|
||||
token = self.token()
|
||||
self.assert_no_extraction()
|
||||
original_data = self.objects[self.source.blob.storage_key]
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
files = response.json()["files"]
|
||||
self.assertEqual(
|
||||
["extracted/folder/one.txt"], [file["display_path"] for file in files]
|
||||
)
|
||||
self.assertEqual(2, self.session.query(FileAsset).count())
|
||||
self.assertEqual(original_data, self.objects[self.source.blob.storage_key])
|
||||
self.assertEqual(self.source.version.id, self.source.asset.current_version_id)
|
||||
self.assertIsNone(self.source.asset.deleted_at)
|
||||
self.assertEqual(
|
||||
self.source.asset.id,
|
||||
files[0]["source_provenance"]["metadata"]["archive_source_file_id"],
|
||||
)
|
||||
self.assertEqual(
|
||||
self.source.version.id,
|
||||
files[0]["source_provenance"]["metadata"]["archive_source_version_id"],
|
||||
)
|
||||
self.audit.assert_called_once()
|
||||
|
||||
def test_all_three_permissions_are_required(self):
|
||||
token = self.token()
|
||||
for scope in tuple(self.scopes):
|
||||
with self.subTest(scope=scope):
|
||||
self.scopes.remove(scope)
|
||||
self.assertEqual(403, self.preview().status_code)
|
||||
self.assertEqual(403, self.confirm(token).status_code)
|
||||
self.scopes.add(scope)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_changed_source_version_is_rejected_before_read_or_write(self):
|
||||
token = self.token()
|
||||
self.source.asset.current_version_id = "replacement-version"
|
||||
self.session.commit()
|
||||
self.assertIn("version changed", self.confirm(token).text)
|
||||
self.assertEqual(400, self.preview().status_code)
|
||||
self.assertEqual(1, self.session.query(FileAsset).count())
|
||||
|
||||
def test_mismatched_version_or_blob_tenant_is_rejected(self):
|
||||
self.source.version.file_asset_id = "other-file"
|
||||
self.session.commit()
|
||||
self.assertIn("does not belong", self.preview().text)
|
||||
self.source.version.file_asset_id = self.source.asset.id
|
||||
self.source.blob.tenant_id = "other-tenant"
|
||||
self.session.commit()
|
||||
self.assertIn("does not belong", self.preview().text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_expired_preview_cannot_extract(self):
|
||||
with patch("cryptography.fernet.time.time", return_value=1):
|
||||
token = self.token()
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("expired", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_source_refresh_during_verified_read_cannot_change_preview_or_extract(self):
|
||||
token = self.token()
|
||||
replacement_version = SimpleNamespace(id="refreshed-version")
|
||||
with patch(
|
||||
"govoplan_files.backend.routes.managed_archives.read_asset_bytes",
|
||||
return_value=(
|
||||
self.objects[self.source.blob.storage_key],
|
||||
replacement_version,
|
||||
self.source.blob,
|
||||
),
|
||||
):
|
||||
preview = self.preview()
|
||||
confirmed = self.confirm(token)
|
||||
for response in (preview, confirmed):
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("version changed during the verified read", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_revoked_shared_access_cannot_reuse_preview(self):
|
||||
self.source.asset.owner_user_id = "other-user"
|
||||
share = FileShare(
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id=self.source.asset.id,
|
||||
target_type="user",
|
||||
target_id="user-1",
|
||||
permission="read",
|
||||
)
|
||||
self.session.add(share)
|
||||
self.session.commit()
|
||||
token = self.token()
|
||||
share.revoked_at = utcnow()
|
||||
self.session.commit()
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("No access", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_deleted_or_cross_tenant_sources_are_rejected(self):
|
||||
token = self.token()
|
||||
self.source.asset.deleted_at = utcnow()
|
||||
self.session.commit()
|
||||
self.assertIn("File not found", self.confirm(token).text)
|
||||
self.source.asset.deleted_at = None
|
||||
self.source.asset.tenant_id = "other-tenant"
|
||||
self.session.commit()
|
||||
self.assertIn("File not found", self.confirm(token).text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_confirmation_is_bound_to_destination_actor_version_and_file(self):
|
||||
token = self.token()
|
||||
for change in (
|
||||
{"path": "elsewhere"},
|
||||
{"owner_id": "other-user"},
|
||||
{"source_version_id": "wrong"},
|
||||
):
|
||||
with self.subTest(change=change):
|
||||
response = self.confirm(token, **change)
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("does not match", response.text)
|
||||
self.principal.user.id = "other-user"
|
||||
self.assertIn("does not match", self.confirm(token).text)
|
||||
self.principal.user.id = "user-1"
|
||||
self.assertEqual(400, self.confirm("tampered-token").status_code)
|
||||
response = self.client.post(
|
||||
"/api/v1/files/other-file/archive-confirm",
|
||||
json={
|
||||
"source_version_id": self.source.version.id,
|
||||
"path": "extracted",
|
||||
"preview_token": token,
|
||||
"selected_paths": ["folder"],
|
||||
},
|
||||
)
|
||||
self.assertIn("does not match", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_destination_ownership_is_checked_during_preview(self):
|
||||
response = self.preview(owner_id="other-user")
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("No access to this user file space", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_connector_policy_is_rechecked_at_confirmation(self):
|
||||
self.source.asset.metadata_ = {
|
||||
"source_provenance": {
|
||||
"source_type": "webdav",
|
||||
"provider": "webdav",
|
||||
"external_id": "archive",
|
||||
}
|
||||
}
|
||||
self.session.commit()
|
||||
token = self.token()
|
||||
self.session.add(
|
||||
FileConnectorPolicy(
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
scope_id="tenant-1",
|
||||
policy={"deny": {"providers": ["webdav"]}},
|
||||
)
|
||||
)
|
||||
self.session.commit()
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(403, response.status_code, response.text)
|
||||
self.assertEqual(403, self.preview().status_code)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_source_collision_rolls_back_all_destination_writes(self):
|
||||
self.source = self.seed_source(
|
||||
archive_bytes(
|
||||
{"a-first.txt": b"first", "collision.zip": b"cannot replace source"}
|
||||
),
|
||||
"collision.zip",
|
||||
)
|
||||
response = self.preview(path="archives")
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
result = self.confirm(
|
||||
response.json()["preview_token"],
|
||||
path="archives",
|
||||
selected_paths=["a-first.txt", "collision.zip"],
|
||||
)
|
||||
self.assertEqual(400, result.status_code, result.text)
|
||||
self.assertIn("already exists", result.text)
|
||||
self.assertEqual(2, self.session.query(FileAsset).count())
|
||||
self.assertEqual(self.source.version.id, self.source.asset.current_version_id)
|
||||
self.assertIsNone(self.source.asset.deleted_at)
|
||||
|
||||
def test_encrypted_zip_uses_the_same_password_and_selection_flow(self):
|
||||
self.source = self.seed_source(
|
||||
archive_bytes(password="correct horse"), "encrypted.zip"
|
||||
)
|
||||
preview = self.preview()
|
||||
self.assertTrue(preview.json()["requires_password"])
|
||||
self.assertFalse(preview.json()["password_verified"])
|
||||
token = preview.json()["preview_token"]
|
||||
self.assertEqual(400, self.confirm(token).status_code)
|
||||
self.assertEqual(400, self.preview(password="wrong").status_code)
|
||||
verified = self.preview(password="correct horse")
|
||||
self.assertTrue(verified.json()["password_verified"])
|
||||
response = self.confirm(
|
||||
verified.json()["preview_token"], password="correct horse"
|
||||
)
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
self.assertNotIn("correct horse", response.text)
|
||||
self.assertNotIn("correct horse", str(self.audit.call_args_list))
|
||||
|
||||
def test_tar_reuses_the_same_confirmation_flow(self):
|
||||
self.source = self.seed_source(archive_bytes(tar=True), "source.tar.gz")
|
||||
preview = self.preview()
|
||||
self.assertEqual("tar.gz", preview.json()["archive_format"])
|
||||
self.assertEqual(200, self.confirm(preview.json()["preview_token"]).status_code)
|
||||
|
||||
def test_unsafe_members_and_size_limits_fail_before_extraction(self):
|
||||
self.settings.file_upload_zip_max_bytes = 1
|
||||
self.assertIn("size limit", self.preview().text)
|
||||
self.settings.file_upload_zip_max_bytes = 10 * 1024 * 1024
|
||||
self.source = self.seed_source(
|
||||
archive_bytes({"../escape": b"no"}), "unsafe.zip"
|
||||
)
|
||||
self.assertIn("Unsafe archive member", self.preview().text)
|
||||
self.audit.assert_not_called()
|
||||
|
||||
def test_quarantined_or_protected_sources_use_verified_read_path(self):
|
||||
self.source.blob.quarantined_at = utcnow()
|
||||
self.session.commit()
|
||||
self.assertIn("quarantin", self.preview().text.lower())
|
||||
self.source.blob.quarantined_at = None
|
||||
self.source.blob.encryption_envelope_id = "protected-envelope"
|
||||
self.session.commit()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.content_protection.unprotect_blob_content",
|
||||
side_effect=FileStorageError("Decryption is unavailable"),
|
||||
) as decrypt:
|
||||
response = self.preview()
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("Decryption is unavailable", response.text)
|
||||
decrypt.assert_called_once()
|
||||
self.assert_no_extraction()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -4,16 +4,27 @@ import unittest
|
||||
|
||||
|
||||
STATIC_TOPIC_IDS = {
|
||||
"files.source-identity-provenance",
|
||||
"files.archive-worker-limits",
|
||||
"files.configuration-package.managed-storage",
|
||||
"files.quick-access-and-product-area",
|
||||
"files.search.managed-content",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.unpack-managed-archive",
|
||||
"files.workflow.find-and-download-files",
|
||||
"files.workflow.share-managed-files",
|
||||
"files.workflow.delete-managed-files",
|
||||
"files.workflow.restore-retain-and-purge",
|
||||
"files.privacy.data-subject-requests",
|
||||
"files.governed-connectors-and-provenance",
|
||||
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||
"files.reference.shared-storage-profile",
|
||||
"files.reference.generated-artifact-store",
|
||||
"files.reference.snapshot-provenance-and-capabilities",
|
||||
"files.records.exact-version-source",
|
||||
"files.forms-runtime.managed-evidence",
|
||||
"files.postbox.exact-version-references",
|
||||
"files.tabular-content",
|
||||
"files.assurance.process-and-release-readiness",
|
||||
}
|
||||
RUNTIME_TOPIC_IDS = {
|
||||
@@ -36,6 +47,38 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
def topic(self, topic_id: str):
|
||||
return self.topics[topic_id]
|
||||
|
||||
def test_source_provenance_and_legacy_ambiguity_are_bilingual(self):
|
||||
topic = self.topic("files.source-identity-provenance")
|
||||
self.assertEqual({"admin", "user"}, set(topic.documentation_types))
|
||||
for body in (topic.body, topic.translations["de"]["body"]):
|
||||
for field in ("source_revision", "acquired_sha256", "provider_checksum_claims", "import_annotations"):
|
||||
self.assertIn(field, body)
|
||||
self.assertIn("non-unique", topic.body)
|
||||
self.assertIn("nicht eindeutig", topic.translations["de"]["body"])
|
||||
|
||||
def test_archive_resource_boundary_is_static_and_bilingual(self):
|
||||
topic = self.topic("files.archive-worker-limits")
|
||||
self.assertEqual("available", topic.layer)
|
||||
self.assertEqual({"admin", "user"}, set(topic.documentation_types))
|
||||
self.assertIn("GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY", topic.configuration_keys)
|
||||
for body in (topic.body, topic.translations["de"]["body"]):
|
||||
for limit in ("120", "90", "512 MiB", "600", "300", "128 MiB", "64 MiB", "16 KiB", "2 GiB", "250 MiB"):
|
||||
self.assertIn(limit, body)
|
||||
self.assertIn("before copying", topic.body)
|
||||
self.assertIn("not a filesystem/network sandbox", topic.body)
|
||||
self.assertIn("keine Dateisystem-/Netzwerk-Sandbox", topic.translations["de"]["body"])
|
||||
|
||||
def test_workspace_action_locations_and_read_only_reload_are_documented_in_both_languages(self) -> None:
|
||||
topic = self.topic("files.quick-access-and-product-area")
|
||||
self.assertEqual({"user", "admin"}, set(topic.documentation_types))
|
||||
for label in ("Reload", "Create folder", "Upload", "Connections and imports", "Manage selection"):
|
||||
self.assertIn(label, topic.body)
|
||||
self.assertIn("never imports or synchronizes", topic.body)
|
||||
german = topic.translations["de"]["body"]
|
||||
for label in ("Neu laden", "Ordner erstellen", "Hochladen", "Verbindungen und Importe", "Auswahl verwalten"):
|
||||
self.assertIn(label, german)
|
||||
self.assertIn("niemals", german)
|
||||
|
||||
def test_static_topics_have_role_scope_module_and_link_contracts(self) -> None:
|
||||
self.assertEqual(STATIC_TOPIC_IDS, set(self.topics))
|
||||
self.assertEqual(
|
||||
@@ -114,9 +157,22 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
|
||||
delete = self.topic("files.workflow.delete-managed-files")
|
||||
self.assertIn("Soft-delete", delete.summary)
|
||||
self.assertIn("no self-service restore or hard-purge", delete.body)
|
||||
self.assertIn("Authorized restoration", delete.body)
|
||||
self.assertTrue(
|
||||
any("not a hard purge" in item for item in delete.metadata["limitations"])
|
||||
any("soft deletion" in item.casefold() for item in delete.metadata["limitations"])
|
||||
)
|
||||
|
||||
lifecycle = self.topic("files.workflow.restore-retain-and-purge")
|
||||
self.assertIn("legal hold", lifecycle.body)
|
||||
self.assertIn("preview hash", lifecycle.body)
|
||||
self.assertIn("recovery ledger", lifecycle.body)
|
||||
|
||||
privacy = self.topic("files.privacy.data-subject-requests")
|
||||
self.assertEqual(("admin",), privacy.documentation_types)
|
||||
self.assertIn("without raw file bytes", privacy.body)
|
||||
self.assertIn("separately authorized Files purge", privacy.body)
|
||||
self.assertTrue(
|
||||
any("membership" in item for item in privacy.metadata["limitations"])
|
||||
)
|
||||
|
||||
def test_admin_topic_covers_policy_redaction_and_atomic_credential_deletion(
|
||||
@@ -133,12 +189,23 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
"files.connectors",
|
||||
"files.connector.credentials",
|
||||
"files.connector.policy",
|
||||
"files.connector-folder-sync.remote-path",
|
||||
"files.connector-folder-sync.target-folder",
|
||||
"files.connector-folder-sync.conflict-strategy",
|
||||
"files.connector-folder-sync.max-files",
|
||||
"files.connector-folder-sync.recursive",
|
||||
},
|
||||
set(topic.metadata["help_contexts"]),
|
||||
)
|
||||
self.assertIn("deny rules win", topic.body)
|
||||
self.assertIn("redact secret values", topic.body)
|
||||
self.assertIn("same transaction", topic.body)
|
||||
self.assertIn("read-only by default", topic.body)
|
||||
self.assertIn("S3", topic.body)
|
||||
self.assertIn("manual connector-folder sync", topic.body)
|
||||
self.assertIn("Per-file savepoints", topic.body)
|
||||
self.assertIn("files.connector.folder_synced", topic.body)
|
||||
self.assertIn("Automatic remote deletion", topic.body)
|
||||
self.assertTrue(
|
||||
any(
|
||||
"non-owned external references" in item
|
||||
@@ -152,6 +219,11 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
self.assertIn(
|
||||
"/api/v1/files/connectors/credentials", {link.href for link in topic.links}
|
||||
)
|
||||
self.assertIn(
|
||||
"/api/v1/files/connector-spaces/{space_id}/sync",
|
||||
{link.href for link in topic.links},
|
||||
)
|
||||
self.assertIn("de", topic.translations)
|
||||
|
||||
def test_operator_topic_covers_recovery_and_pinned_s3_smb(self) -> None:
|
||||
topic = self.topic(
|
||||
@@ -169,9 +241,13 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
self.assertIn("bounded resumable integrity scan", topic.body)
|
||||
self.assertIn("quarantined", topic.body)
|
||||
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
|
||||
self.assertIn("Encryption envelope and wrapped-key rows", topic.metadata["recovery_unit"])
|
||||
self.assertIn(
|
||||
"Encryption envelope and wrapped-key rows", topic.metadata["recovery_unit"]
|
||||
)
|
||||
self.assertIn("lease-fenced Core recovery", topic.body)
|
||||
self.assertIn("Ops", topic.body)
|
||||
self.assertIn("Hard purge", topic.body)
|
||||
self.assertIn("S3 connector write-back", topic.body)
|
||||
self.assertTrue(topic.metadata["verification"])
|
||||
self.assertIn(
|
||||
"/api/v1/files/integrity/scans",
|
||||
@@ -195,7 +271,11 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
self.assertEqual(("admin", "user"), topic.documentation_types)
|
||||
self.assertEqual("reference", topic.metadata["kind"])
|
||||
self.assertEqual(
|
||||
["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
|
||||
[
|
||||
"files.access@0.1.6",
|
||||
"files.campaign_attachments@0.1.6",
|
||||
"forms_runtime.evidence.files@1.0.0",
|
||||
],
|
||||
topic.metadata["provided_interfaces"],
|
||||
)
|
||||
self.assertIn("revision", topic.metadata["provenance_fields"])
|
||||
@@ -213,8 +293,9 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
self.assertIn("process_owner", topic.audience)
|
||||
self.assertIn("release_manager", topic.audience)
|
||||
self.assertIn("versions align", topic.body)
|
||||
self.assertIn("Share grant, change, expiry, and revocation", topic.body)
|
||||
self.assertIn("no enforced retention or legal hold", topic.body)
|
||||
self.assertIn("share lifecycle", topic.body)
|
||||
self.assertIn("legal hold", topic.body)
|
||||
self.assertIn("S3 write-back", topic.body)
|
||||
for key in ("prerequisites", "steps", "outcome", "verification"):
|
||||
self.assertTrue(topic.metadata[key])
|
||||
self.assertTrue(
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from alembic.runtime.migration import MigrationContext
|
||||
from sqlalchemy import create_engine, inspect
|
||||
|
||||
from govoplan_access.backend.manifest import get_manifest as get_access_manifest
|
||||
from govoplan_core.db.migrations import migrate_database
|
||||
from govoplan_files.backend.manifest import get_manifest as get_files_manifest
|
||||
|
||||
|
||||
class FilesMigrationTests(unittest.TestCase):
|
||||
def test_fresh_migration_creates_form_evidence_grants_and_head(self) -> None:
|
||||
with tempfile.TemporaryDirectory(
|
||||
prefix="govoplan-files-migration-"
|
||||
) as directory:
|
||||
url = f"sqlite:///{Path(directory) / 'files.db'}"
|
||||
migrate_database(
|
||||
database_url=url,
|
||||
enabled_modules=("access", "files"),
|
||||
manifest_factories=(get_access_manifest, get_files_manifest),
|
||||
)
|
||||
engine = create_engine(url)
|
||||
try:
|
||||
self.assertIn(
|
||||
"file_form_evidence_grants",
|
||||
inspect(engine).get_table_names(),
|
||||
)
|
||||
with engine.connect() as connection:
|
||||
self.assertIn(
|
||||
"a2b3c4d5e701",
|
||||
set(MigrationContext.configure(connection).get_current_heads()),
|
||||
)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,160 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_core.core.files import PostboxFileReferenceRequest
|
||||
from govoplan_files.backend.capabilities import FilesPostboxReferenceService
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileVersion
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
class _Principal:
|
||||
def __init__(self, *, tenant_id: str = "tenant-1", scopes: set[str] | None = None):
|
||||
self.tenant_id = tenant_id
|
||||
self.user = SimpleNamespace(id="user-1")
|
||||
self._scopes = {"files:file:download"} if scopes is None else scopes
|
||||
|
||||
def has(self, scope: str) -> bool:
|
||||
return scope in self._scopes
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self, values: dict[tuple[type[object], str], object]):
|
||||
self.values = values
|
||||
|
||||
def get(self, model: type[object], resource_id: str):
|
||||
return self.values.get((model, resource_id))
|
||||
|
||||
|
||||
def _request(reference_type: str = "file_version", reference_id: str = "version-1"):
|
||||
return PostboxFileReferenceRequest(
|
||||
postbox_id="postbox-1",
|
||||
message_id="message-1",
|
||||
reference_type=reference_type,
|
||||
reference_id=reference_id,
|
||||
)
|
||||
|
||||
|
||||
def _objects():
|
||||
asset = SimpleNamespace(
|
||||
id="file-1",
|
||||
tenant_id="tenant-1",
|
||||
current_version_id="version-2",
|
||||
deleted_at=None,
|
||||
filename="current.txt",
|
||||
display_path="Evidence/current.txt",
|
||||
)
|
||||
version = SimpleNamespace(
|
||||
id="version-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id="file-1",
|
||||
blob_id="blob-1",
|
||||
filename_at_upload="evidence.txt",
|
||||
content_type="text/plain",
|
||||
size_bytes=8,
|
||||
checksum_sha256="a" * 64,
|
||||
)
|
||||
blob = SimpleNamespace(
|
||||
id="blob-1",
|
||||
tenant_id="tenant-1",
|
||||
content_type="text/plain",
|
||||
)
|
||||
return asset, version, blob
|
||||
|
||||
|
||||
class FilesPostboxReferenceTests(unittest.TestCase):
|
||||
def test_resolves_the_referenced_version_instead_of_the_current_version(self) -> None:
|
||||
asset, version, blob = _objects()
|
||||
session = _Session(
|
||||
{
|
||||
(FileAsset, asset.id): asset,
|
||||
(FileVersion, version.id): version,
|
||||
(FileBlob, blob.id): blob,
|
||||
}
|
||||
)
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.capabilities.get_asset_for_user",
|
||||
return_value=asset,
|
||||
):
|
||||
result = FilesPostboxReferenceService().resolve_postbox_references(
|
||||
session,
|
||||
_Principal(),
|
||||
tenant_id="tenant-1",
|
||||
requests=(_request(),),
|
||||
)[0]
|
||||
|
||||
self.assertTrue(result.available)
|
||||
self.assertEqual("version-1", result.file_version_id)
|
||||
self.assertEqual(
|
||||
"/api/v1/files/file-1/versions/version-1/download",
|
||||
result.download_path,
|
||||
)
|
||||
self.assertTrue(result.provenance["exact_version"])
|
||||
|
||||
def test_fails_closed_for_tenant_permission_and_file_access_mismatches(self) -> None:
|
||||
service = FilesPostboxReferenceService()
|
||||
empty = _Session({})
|
||||
|
||||
tenant_result = service.resolve_postbox_references(
|
||||
empty,
|
||||
_Principal(tenant_id="tenant-2"),
|
||||
tenant_id="tenant-1",
|
||||
requests=(_request(),),
|
||||
)[0]
|
||||
permission_result = service.resolve_postbox_references(
|
||||
empty,
|
||||
_Principal(scopes=set()),
|
||||
tenant_id="tenant-1",
|
||||
requests=(_request(),),
|
||||
)[0]
|
||||
|
||||
asset, version, blob = _objects()
|
||||
populated = _Session(
|
||||
{
|
||||
(FileAsset, asset.id): asset,
|
||||
(FileVersion, version.id): version,
|
||||
(FileBlob, blob.id): blob,
|
||||
}
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.capabilities.get_asset_for_user",
|
||||
side_effect=FileStorageError("denied"),
|
||||
):
|
||||
access_result = service.resolve_postbox_references(
|
||||
populated,
|
||||
_Principal(),
|
||||
tenant_id="tenant-1",
|
||||
requests=(_request(),),
|
||||
)[0]
|
||||
|
||||
self.assertEqual("tenant_mismatch", tenant_result.reason_code)
|
||||
self.assertEqual("download_permission_missing", permission_result.reason_code)
|
||||
self.assertEqual("file_access_denied", access_result.reason_code)
|
||||
self.assertFalse(access_result.available)
|
||||
|
||||
def test_asset_reference_does_not_drift_to_a_later_current_version(self) -> None:
|
||||
asset, version, blob = _objects()
|
||||
session = _Session(
|
||||
{
|
||||
(FileAsset, asset.id): asset,
|
||||
(FileVersion, version.id): version,
|
||||
(FileBlob, blob.id): blob,
|
||||
}
|
||||
)
|
||||
|
||||
result = FilesPostboxReferenceService().resolve_postbox_references(
|
||||
session,
|
||||
_Principal(),
|
||||
tenant_id="tenant-1",
|
||||
requests=(_request("file", asset.id),),
|
||||
)[0]
|
||||
|
||||
self.assertFalse(result.available)
|
||||
self.assertEqual("exact_version_required", result.reason_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,38 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
from govoplan_files.backend.manifest import manifest
|
||||
|
||||
|
||||
REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
class FilesQuickAccessContractTests(unittest.TestCase):
|
||||
def test_manifest_declares_exact_version_result_and_fallback(self) -> None:
|
||||
tool = manifest.frontend.quick_access_tools[0]
|
||||
|
||||
self.assertEqual("files.recent", tool.id)
|
||||
self.assertEqual(("files.file-version",), tool.returned_reference_kinds)
|
||||
self.assertEqual("files.quick_access.files", tool.help_context_id)
|
||||
self.assertEqual("/files", tool.full_page_path)
|
||||
|
||||
def test_renderer_and_page_keep_recent_selection_and_upload_bounded(self) -> None:
|
||||
renderer = (
|
||||
REPOSITORY_ROOT / "webui/src/features/files/FileQuickAccess.tsx"
|
||||
).read_text(encoding="utf-8")
|
||||
page = (
|
||||
REPOSITORY_ROOT / "webui/src/features/files/FilesPage.tsx"
|
||||
).read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn('sort: "recent", page_size: 7', renderer)
|
||||
self.assertIn('kind: "file-version"', renderer)
|
||||
self.assertIn("quickAccessLaunchState(launchContext)", renderer)
|
||||
self.assertIn('to="/files?quickAction=upload"', renderer)
|
||||
self.assertIn('parameters.get("quickAction") !== "upload"', page)
|
||||
self.assertIn('openDialog("upload"', page)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,108 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.records import RecordContractError, RecordSourceLocator
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileVersion
|
||||
from govoplan_files.backend.record_source import FilesRecordSource
|
||||
|
||||
|
||||
@dataclass
|
||||
class Principal:
|
||||
tenant_id: str = "tenant-1"
|
||||
membership_id: str = "user-1"
|
||||
user: object = field(default_factory=lambda: SimpleNamespace(id="user-1"))
|
||||
|
||||
def has(self, scope: str) -> bool:
|
||||
return scope in {"files:file:read"}
|
||||
|
||||
|
||||
class FilesRecordSourceTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.session = MagicMock(spec=Session)
|
||||
self.asset = FileAsset(
|
||||
id="asset-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="user-1",
|
||||
current_version_id="version-1",
|
||||
display_path="Evidence/Decision.pdf",
|
||||
filename="Decision.pdf",
|
||||
)
|
||||
self.blob = FileBlob(
|
||||
id="blob-1",
|
||||
tenant_id="tenant-1",
|
||||
storage_backend="local",
|
||||
storage_key="tenant-1/blob-1",
|
||||
checksum_sha256="a" * 64,
|
||||
size_bytes=1024,
|
||||
integrity_status="verified",
|
||||
)
|
||||
self.version = FileVersion(
|
||||
id="version-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id="asset-1",
|
||||
blob_id="blob-1",
|
||||
version_number=1,
|
||||
filename_at_upload="Decision.pdf",
|
||||
display_path_at_upload="Evidence/Decision.pdf",
|
||||
content_type="application/pdf",
|
||||
size_bytes=1024,
|
||||
checksum_sha256="a" * 64,
|
||||
)
|
||||
self.version.created_at = datetime.now(UTC)
|
||||
version_query = self.session.query.return_value.filter.return_value
|
||||
version_query.filter.return_value.one_or_none.return_value = self.version
|
||||
self.session.get.return_value = self.blob
|
||||
|
||||
def locator(self) -> RecordSourceLocator:
|
||||
return RecordSourceLocator(
|
||||
tenant_id="tenant-1",
|
||||
source_module="files",
|
||||
resource_type="file_version",
|
||||
resource_id="asset-1",
|
||||
source_revision="version-1",
|
||||
)
|
||||
|
||||
def test_resolves_exact_authorized_version_and_digest(self) -> None:
|
||||
with patch(
|
||||
"govoplan_files.backend.record_source.get_asset_for_user",
|
||||
return_value=self.asset,
|
||||
):
|
||||
result = FilesRecordSource().resolve(
|
||||
self.session,
|
||||
Principal(),
|
||||
locator=self.locator(),
|
||||
purpose="document decision basis",
|
||||
)
|
||||
|
||||
self.assertEqual("version-1", result.locator.source_revision)
|
||||
self.assertEqual("a" * 64, result.content_sha256)
|
||||
self.assertEqual("verified", result.metadata["integrity_status"])
|
||||
|
||||
def test_quarantined_version_fails_closed(self) -> None:
|
||||
self.blob.quarantined_at = datetime.now(UTC)
|
||||
self.session.flush()
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.record_source.get_asset_for_user",
|
||||
return_value=self.asset,
|
||||
),
|
||||
self.assertRaisesRegex(RecordContractError, "integrity gate"),
|
||||
):
|
||||
FilesRecordSource().resolve(
|
||||
self.session,
|
||||
Principal(),
|
||||
locator=self.locator(),
|
||||
purpose="document decision basis",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -7,15 +7,22 @@ from inspect import signature
|
||||
from govoplan_files.backend.router import router
|
||||
from govoplan_files.backend.routes.assets import router as assets_router
|
||||
from govoplan_files.backend.routes.connector_io import router as connector_io_router
|
||||
from govoplan_files.backend.routes.connector_profiles import router as connector_profiles_router
|
||||
from govoplan_files.backend.routes.connector_settings import router as connector_settings_router
|
||||
from govoplan_files.backend.routes.connector_profiles import (
|
||||
router as connector_profiles_router,
|
||||
)
|
||||
from govoplan_files.backend.routes.connector_settings import (
|
||||
router as connector_settings_router,
|
||||
)
|
||||
from govoplan_files.backend.routes.folders import router as folders_router
|
||||
from govoplan_files.backend.routes.form_evidence import router as form_evidence_router
|
||||
from govoplan_files.backend.routes.integrity import router as integrity_router
|
||||
from govoplan_files.backend.routes.listing import router as listing_router
|
||||
from govoplan_files.backend.routes.lifecycle import router as lifecycle_router
|
||||
from govoplan_files.backend.routes.shares import router as shares_router
|
||||
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||
from govoplan_files.backend.routes.managed_archives import router as managed_archives_router
|
||||
|
||||
|
||||
class FilesRouterContractTests(unittest.TestCase):
|
||||
@@ -31,9 +38,12 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
workflow_routers = (
|
||||
spaces_router,
|
||||
folders_router,
|
||||
form_evidence_router,
|
||||
integrity_router,
|
||||
lifecycle_router,
|
||||
listing_router,
|
||||
uploads_router,
|
||||
managed_archives_router,
|
||||
connector_settings_router,
|
||||
connector_io_router,
|
||||
connector_profiles_router,
|
||||
@@ -49,22 +59,27 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
actual = self._operation_keys(router)
|
||||
|
||||
self.assertEqual(expected, actual)
|
||||
self.assertEqual(52, len(actual))
|
||||
self.assertEqual(67, len(actual))
|
||||
self.assertFalse(
|
||||
[operation for operation, count in Counter(actual).items() if count > 1]
|
||||
)
|
||||
|
||||
def test_archive_preview_and_confirmation_routes_are_exposed(self) -> None:
|
||||
routes = {
|
||||
(tuple(sorted(route.methods or ())), route.path)
|
||||
for route in router.routes
|
||||
(tuple(sorted(route.methods or ())), route.path) for route in router.routes
|
||||
}
|
||||
|
||||
self.assertIn((("POST",), "/files/archive-preview"), routes)
|
||||
self.assertIn((("POST",), "/files/archive-confirm"), routes)
|
||||
self.assertIn((("GET",), "/files/archive-progress/{operation_id}"), routes)
|
||||
self.assertIn((("DELETE",), "/files/archive-staging/{stage_id}"), routes)
|
||||
self.assertIn((("POST",), "/files/{file_id}/archive-preview"), routes)
|
||||
self.assertIn((("POST",), "/files/{file_id}/archive-confirm"), routes)
|
||||
|
||||
def test_connector_routes_keep_existing_api_paths(self) -> None:
|
||||
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
|
||||
routes = {
|
||||
(tuple(sorted(route.methods or ())), route.path) for route in router.routes
|
||||
}
|
||||
|
||||
expected = {
|
||||
(("GET",), "/files/connectors/providers"),
|
||||
@@ -74,28 +89,57 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
(("GET",), "/files/connectors/profiles/{profile_id}/browse"),
|
||||
(("POST",), "/files/connectors/profiles/{profile_id}/import"),
|
||||
(("POST",), "/files/connectors/profiles/{profile_id}/sync"),
|
||||
(("POST",), "/files/connector-spaces/{space_id}/sync"),
|
||||
(("POST",), "/files/connector-spaces/{space_id}/write-back"),
|
||||
(("GET",), "/files/connectors/credentials"),
|
||||
(("POST",), "/files/connectors/credentials"),
|
||||
(("GET",), "/files/connector-spaces"),
|
||||
(("POST",), "/files/connector-spaces"),
|
||||
(("PATCH",), "/files/connector-spaces/{space_id}"),
|
||||
(("DELETE",), "/files/connector-spaces/{space_id}"),
|
||||
}
|
||||
|
||||
self.assertTrue(expected.issubset(routes))
|
||||
|
||||
def test_bulk_organize_routes_keep_existing_api_paths(self) -> None:
|
||||
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
|
||||
routes = {
|
||||
(tuple(sorted(route.methods or ())), route.path) for route in router.routes
|
||||
}
|
||||
|
||||
self.assertIn((("POST",), "/files/bulk-rename"), routes)
|
||||
self.assertIn((("POST",), "/files/transfer"), routes)
|
||||
|
||||
def test_share_lifecycle_routes_are_exposed(self) -> None:
|
||||
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
|
||||
routes = {
|
||||
(tuple(sorted(route.methods or ())), route.path) for route in router.routes
|
||||
}
|
||||
|
||||
self.assertIn((("GET",), "/files/{file_id}/shares"), routes)
|
||||
self.assertIn((("POST",), "/files/{file_id}/shares"), routes)
|
||||
self.assertIn((("DELETE",), "/files/{file_id}/shares/{share_id}"), routes)
|
||||
self.assertIn((("GET",), "/files/{file_id}/share-target-options"), routes)
|
||||
|
||||
def test_exact_version_download_route_is_exposed(self) -> None:
|
||||
routes = {
|
||||
(tuple(sorted(route.methods or ())), route.path) for route in router.routes
|
||||
}
|
||||
|
||||
self.assertIn(
|
||||
(("GET",), "/files/{file_id}/versions/{version_id}/download"),
|
||||
routes,
|
||||
)
|
||||
|
||||
def test_governed_lifecycle_routes_are_exposed(self) -> None:
|
||||
routes = {
|
||||
(tuple(sorted(route.methods or ())), route.path) for route in router.routes
|
||||
}
|
||||
|
||||
self.assertIn((("PATCH",), "/files/{file_id}/lifecycle"), routes)
|
||||
self.assertIn((("POST",), "/files/assets/{file_id}/restore"), routes)
|
||||
self.assertIn((("POST",), "/files/purge/preview"), routes)
|
||||
self.assertIn((("POST",), "/files/purge/execute"), routes)
|
||||
self.assertIn((("POST",), "/files/purge/blobs"), routes)
|
||||
|
||||
def test_file_listing_exposes_structured_property_filters(self) -> None:
|
||||
route = next(
|
||||
route
|
||||
@@ -106,6 +150,7 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
|
||||
self.assertIn("campaign_usage", parameters)
|
||||
self.assertIn("audit_relevant", parameters)
|
||||
self.assertIn("sort", parameters)
|
||||
self.assertIn("total", route.response_model.model_fields)
|
||||
|
||||
|
||||
|
||||
Executable
+199
@@ -0,0 +1,199 @@
|
||||
from datetime import datetime, timezone
|
||||
import importlib
|
||||
import unittest
|
||||
|
||||
from alembic.migration import MigrationContext
|
||||
from alembic.operations import Operations
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
Column,
|
||||
MetaData,
|
||||
String,
|
||||
Table,
|
||||
create_engine,
|
||||
inspect,
|
||||
select,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_access.backend.db import models as _access_models # register FK metadata
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
from govoplan_files.backend.storage.files import find_asset_by_source
|
||||
from govoplan_files.backend.storage.provenance import source_identity_hash
|
||||
|
||||
|
||||
def provenance(external="remote-1"):
|
||||
return {"connector_id": " connector ", "provider": "s3", "external_id": external}
|
||||
|
||||
|
||||
class SourceIdentityIndexTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite:///:memory:")
|
||||
for table in (
|
||||
_access_models.Account.__table__,
|
||||
_access_models.User.__table__,
|
||||
_access_models.Group.__table__,
|
||||
FileAsset.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
):
|
||||
table.create(self.engine)
|
||||
self.session = Session(self.engine)
|
||||
|
||||
def tearDown(self):
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def asset(self, name, *, tenant="tenant", owner="owner", source=None, group=False):
|
||||
row = FileAsset(
|
||||
id=name,
|
||||
tenant_id=tenant,
|
||||
owner_type="group" if group else "user",
|
||||
owner_user_id=None if group else owner,
|
||||
owner_group_id=owner if group else None,
|
||||
display_path=f"{name}.txt",
|
||||
filename=f"{name}.txt",
|
||||
metadata_={"source_provenance": source or provenance()},
|
||||
)
|
||||
self.session.add(row)
|
||||
self.session.flush()
|
||||
return row
|
||||
|
||||
def find(self, source=None, *, tenant="tenant", owner="owner", group=False):
|
||||
return find_asset_by_source(
|
||||
self.session,
|
||||
tenant_id=tenant,
|
||||
owner_type="group" if group else "user",
|
||||
owner_id=owner,
|
||||
source_provenance=source or provenance(),
|
||||
)
|
||||
|
||||
def test_index_lookup_preserves_tenant_owner_provider_and_external_identity(self):
|
||||
row = self.asset("wanted")
|
||||
self.asset("other-tenant", tenant="another")
|
||||
self.asset("other-owner", owner="another")
|
||||
self.asset("group", group=True)
|
||||
self.asset("other-provider", source={**provenance(), "provider": "webdav"})
|
||||
for index in range(100):
|
||||
self.asset(f"unrelated-{index}", source=provenance(f"remote-{index + 2}"))
|
||||
self.assertIs(row, self.find())
|
||||
self.assertIs(row, self.find()) # retry
|
||||
self.assertEqual("group", self.find(group=True).id)
|
||||
statement = (
|
||||
select(FileAsset.id)
|
||||
.where(
|
||||
FileAsset.tenant_id == "tenant",
|
||||
FileAsset.owner_type == "user",
|
||||
FileAsset.owner_user_id == "owner",
|
||||
FileAsset.source_identity_hash == source_identity_hash(provenance()),
|
||||
FileAsset.deleted_at.is_(None),
|
||||
)
|
||||
.limit(2)
|
||||
)
|
||||
sql = str(
|
||||
statement.compile(self.engine, compile_kwargs={"literal_binds": True})
|
||||
)
|
||||
plan = self.session.execute(text("EXPLAIN QUERY PLAN " + sql)).all()
|
||||
self.assertIn("ix_file_assets_user_source", str(plan))
|
||||
self.assertIn("LIMIT 2", sql)
|
||||
|
||||
def test_metadata_copy_delete_restore_and_duplicates_never_choose_a_winner(self):
|
||||
row = self.asset("original")
|
||||
copy = self.asset("copy", source=dict(row.metadata_["source_provenance"]))
|
||||
self.assertEqual(row.source_identity_hash, copy.source_identity_hash)
|
||||
with self.assertRaisesRegex(FileSourceConflict, "Multiple active"):
|
||||
self.find()
|
||||
copy.deleted_at = datetime.now(timezone.utc)
|
||||
self.session.flush()
|
||||
self.assertIs(row, self.find())
|
||||
copy.deleted_at = None
|
||||
self.session.flush()
|
||||
with self.assertRaises(FileSourceConflict):
|
||||
self.find()
|
||||
copy.metadata_ = {"source_provenance": provenance("changed")}
|
||||
self.session.flush()
|
||||
self.assertIs(row, self.find())
|
||||
self.assertIs(copy, self.find(provenance("changed")))
|
||||
copy.metadata_ = {}
|
||||
self.session.flush()
|
||||
self.assertIsNone(copy.source_identity_hash)
|
||||
self.assertIsNone(self.find(provenance("changed")))
|
||||
|
||||
def test_backfill_retains_all_legacy_records_and_metadata_in_bounded_batches(self):
|
||||
migration = importlib.import_module(
|
||||
"govoplan_files.backend.migrations.versions.a2b3c4d5e701_file_source_identity_index"
|
||||
)
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
metadata = MetaData()
|
||||
old = Table(
|
||||
"file_assets",
|
||||
metadata,
|
||||
Column("id", String, primary_key=True),
|
||||
Column("tenant_id", String),
|
||||
Column("owner_type", String),
|
||||
Column("owner_user_id", String),
|
||||
Column("owner_group_id", String),
|
||||
Column("metadata", JSON),
|
||||
)
|
||||
metadata.create_all(engine)
|
||||
rows = [
|
||||
{
|
||||
"id": f"{index:04d}",
|
||||
"tenant_id": "tenant",
|
||||
"owner_type": "user",
|
||||
"owner_user_id": "owner",
|
||||
"metadata": {
|
||||
"source_provenance": provenance(
|
||||
"duplicate" if index < 2 else str(index)
|
||||
)
|
||||
},
|
||||
}
|
||||
for index in range(503)
|
||||
]
|
||||
rows.append(
|
||||
{
|
||||
"id": "no-source",
|
||||
"tenant_id": "tenant",
|
||||
"owner_type": "user",
|
||||
"owner_user_id": "owner",
|
||||
"metadata": {"unrelated": [1, 2]},
|
||||
}
|
||||
)
|
||||
try:
|
||||
with engine.begin() as connection:
|
||||
connection.execute(old.insert(), rows)
|
||||
with Operations.context(MigrationContext.configure(connection)):
|
||||
migration.upgrade()
|
||||
new = Table("file_assets", MetaData(), autoload_with=connection)
|
||||
actual = (
|
||||
connection.execute(select(new).order_by(new.c.id)).mappings().all()
|
||||
)
|
||||
self.assertEqual(len(rows), len(actual))
|
||||
self.assertEqual(
|
||||
[row["metadata"] for row in rows],
|
||||
[row["metadata"] for row in actual],
|
||||
)
|
||||
self.assertEqual(
|
||||
actual[0]["source_identity_hash"], actual[1]["source_identity_hash"]
|
||||
)
|
||||
for row in actual:
|
||||
self.assertEqual(
|
||||
source_identity_hash(row["metadata"].get("source_provenance")),
|
||||
row["source_identity_hash"],
|
||||
)
|
||||
indexes = inspect(connection).get_indexes("file_assets")
|
||||
self.assertEqual(2, len(indexes))
|
||||
self.assertFalse(any(index["unique"] for index in indexes))
|
||||
with Operations.context(MigrationContext.configure(connection)):
|
||||
migration.downgrade()
|
||||
self.assertEqual(
|
||||
len(rows), len(connection.execute(select(old.c.id)).all())
|
||||
)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
@@ -15,6 +16,7 @@ from govoplan_core.core.recovery import (
|
||||
RecoveryOperation,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.core.runtime_coordination import (
|
||||
DistributedLease,
|
||||
RuntimeIdentity,
|
||||
@@ -23,9 +25,13 @@ from govoplan_core.core.runtime_coordination import (
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.db.session import configure_database, reset_database
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileFormEvidenceGrant,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
LocalFilesystemStorageBackend,
|
||||
@@ -34,12 +40,61 @@ from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import _get_or_create_blob
|
||||
from govoplan_files.backend.storage.integrity import cleanup_orphan_finding
|
||||
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.connector_writes import write_connector_file
|
||||
from govoplan_files.backend.storage.lifecycle import (
|
||||
execute_asset_purge,
|
||||
garbage_collect_unreferenced_blobs,
|
||||
preview_asset_purge,
|
||||
)
|
||||
|
||||
|
||||
TENANT_ID = "tenant-1"
|
||||
USER_ID = "user-1"
|
||||
|
||||
|
||||
class _MissingS3Object(Exception):
|
||||
response = {
|
||||
"Error": {"Code": "NoSuchKey"},
|
||||
"ResponseMetadata": {"HTTPStatusCode": 404},
|
||||
}
|
||||
|
||||
|
||||
class _WriteS3Client:
|
||||
def __init__(
|
||||
self, *, tamper_metadata: bool = False, fail_post_write_probe: bool = False
|
||||
) -> None:
|
||||
self.object: dict[str, object] | None = None
|
||||
self.tamper_metadata = tamper_metadata
|
||||
self.fail_post_write_probe = fail_post_write_probe
|
||||
self.put_request: dict[str, object] | None = None
|
||||
self.closed = False
|
||||
self.write_completed = False
|
||||
|
||||
def head_object(self, **_kwargs: object) -> dict[str, object]:
|
||||
if self.write_completed and self.fail_post_write_probe:
|
||||
raise RuntimeError("provider unavailable")
|
||||
if self.object is None:
|
||||
raise _MissingS3Object()
|
||||
return dict(self.object)
|
||||
|
||||
def put_object(self, **kwargs: object) -> dict[str, object]:
|
||||
self.put_request = dict(kwargs)
|
||||
metadata = dict(kwargs.get("Metadata") or {})
|
||||
if self.tamper_metadata:
|
||||
metadata["govoplan-sha256"] = "0" * 64
|
||||
self.object = {
|
||||
"ETag": '"etag-written"',
|
||||
"VersionId": "version-written",
|
||||
"Metadata": metadata,
|
||||
}
|
||||
self.write_completed = True
|
||||
return {"ETag": '"etag-written"', "VersionId": "version-written"}
|
||||
|
||||
def close(self) -> None:
|
||||
self.closed = True
|
||||
|
||||
|
||||
class StorageRecoveryTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temporary_directory = tempfile.TemporaryDirectory()
|
||||
@@ -54,10 +109,15 @@ class StorageRecoveryTests(unittest.TestCase):
|
||||
Account.__table__,
|
||||
User.__table__,
|
||||
Group.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
DistributedLease.__table__,
|
||||
RecoveryOperation.__table__,
|
||||
RecoveryCheckpoint.__table__,
|
||||
FileBlob.__table__,
|
||||
FileAsset.__table__,
|
||||
FileVersion.__table__,
|
||||
FileFormEvidenceGrant.__table__,
|
||||
FileShare.__table__,
|
||||
FileIntegrityScan.__table__,
|
||||
FileIntegrityFinding.__table__,
|
||||
],
|
||||
@@ -115,9 +175,10 @@ class StorageRecoveryTests(unittest.TestCase):
|
||||
|
||||
def put_bytes(backend_self, key, data, *, content_type=None):
|
||||
with self.Session() as evidence_session:
|
||||
operation = evidence_session.query(RecoveryOperation).one()
|
||||
operation = evidence_session.query(RecoveryOperation).one_or_none()
|
||||
observed_running_operation.append(
|
||||
operation.status == RecoveryStatus.RUNNING.value
|
||||
operation is not None
|
||||
and operation.status == RecoveryStatus.RUNNING.value
|
||||
and len(operation.request_sha256) == 64
|
||||
)
|
||||
put_bytes(key, data, content_type=content_type)
|
||||
@@ -140,7 +201,14 @@ class StorageRecoveryTests(unittest.TestCase):
|
||||
|
||||
operation = self._only_operation()
|
||||
self.assertEqual(RecoveryStatus.SUCCEEDED.value, operation.status)
|
||||
self.assertEqual([True], observed_running_operation)
|
||||
# SQLite uses an explicit caller-transaction mode because it permits
|
||||
# only one writer. The intent becomes visible with the business commit;
|
||||
# PostgreSQL retains the independent pre-effect commit guarantee.
|
||||
self.assertEqual([False], observed_running_operation)
|
||||
self.assertEqual(
|
||||
"sqlite_caller_transaction",
|
||||
operation.metadata_["durability_mode"],
|
||||
)
|
||||
self.assertTrue(self.backend.exists(blob.storage_key))
|
||||
self.assertNotIn("private-name", blob.storage_key)
|
||||
self.assertEqual(".blob", Path(blob.storage_key).suffix)
|
||||
@@ -164,10 +232,54 @@ class StorageRecoveryTests(unittest.TestCase):
|
||||
|
||||
operation = self._only_operation()
|
||||
self.assertEqual(RecoveryStatus.RECOVERED.value, operation.status)
|
||||
self.assertEqual(
|
||||
"sqlite_post_rollback_reconstruction",
|
||||
operation.metadata_["durability_mode"],
|
||||
)
|
||||
self.assertFalse(self.backend.exists(storage_key))
|
||||
with self.Session() as evidence_session:
|
||||
self.assertIsNone(evidence_session.get(FileBlob, blob.id))
|
||||
|
||||
def test_multiple_blob_writes_share_one_sqlite_business_transaction(self) -> None:
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=self.backend,
|
||||
):
|
||||
first = _get_or_create_blob(
|
||||
self.session,
|
||||
tenant_id=TENANT_ID,
|
||||
data=b"first archive member",
|
||||
filename="first.txt",
|
||||
content_type="text/plain",
|
||||
actor_id=USER_ID,
|
||||
)
|
||||
second = _get_or_create_blob(
|
||||
self.session,
|
||||
tenant_id=TENANT_ID,
|
||||
data=b"second archive member",
|
||||
filename="second.txt",
|
||||
content_type="text/plain",
|
||||
actor_id=USER_ID,
|
||||
)
|
||||
self.session.commit()
|
||||
|
||||
self.assertTrue(self.backend.exists(first.storage_key))
|
||||
self.assertTrue(self.backend.exists(second.storage_key))
|
||||
with self.Session() as evidence_session:
|
||||
operations = evidence_session.query(RecoveryOperation).all()
|
||||
self.assertEqual(2, len(operations))
|
||||
self.assertEqual(
|
||||
{RecoveryStatus.SUCCEEDED.value},
|
||||
{operation.status for operation in operations},
|
||||
)
|
||||
self.assertEqual(
|
||||
{"sqlite_caller_transaction"},
|
||||
{
|
||||
operation.metadata_["durability_mode"]
|
||||
for operation in operations
|
||||
},
|
||||
)
|
||||
|
||||
def test_post_write_tamper_is_quarantined_and_recovery_required(self) -> None:
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
@@ -314,6 +426,214 @@ class StorageRecoveryTests(unittest.TestCase):
|
||||
self.session.rollback()
|
||||
self.assertEqual(RecoveryStatus.REJECTED.value, self._only_operation().status)
|
||||
|
||||
def test_purge_is_idempotent_and_gc_deletes_only_after_reference_check(self) -> None:
|
||||
key = f"tenants/{TENANT_ID}/files/purge-me.blob"
|
||||
self.backend.put_bytes(key, b"purge-me")
|
||||
blob = FileBlob(
|
||||
id="blob-purge",
|
||||
tenant_id=TENANT_ID,
|
||||
storage_backend=self.backend.name,
|
||||
storage_key=key,
|
||||
checksum_sha256=hashlib.sha256(b"purge-me").hexdigest(),
|
||||
size_bytes=8,
|
||||
ref_count=1,
|
||||
)
|
||||
asset = FileAsset(
|
||||
id="asset-purge",
|
||||
tenant_id=TENANT_ID,
|
||||
owner_type="user",
|
||||
owner_user_id=USER_ID,
|
||||
current_version_id="version-purge",
|
||||
display_path="purge-me.txt",
|
||||
filename="purge-me.txt",
|
||||
deleted_at=datetime.now(UTC),
|
||||
)
|
||||
version = FileVersion(
|
||||
id="version-purge",
|
||||
tenant_id=TENANT_ID,
|
||||
file_asset_id=asset.id,
|
||||
blob_id=blob.id,
|
||||
version_number=1,
|
||||
filename_at_upload=asset.filename,
|
||||
display_path_at_upload=asset.display_path,
|
||||
size_bytes=blob.size_bytes,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
)
|
||||
self.session.add_all([blob, asset, version])
|
||||
self.session.commit()
|
||||
preview = preview_asset_purge(
|
||||
self.session, tenant_id=TENANT_ID, file_ids=[asset.id]
|
||||
)
|
||||
|
||||
result = execute_asset_purge(
|
||||
self.session,
|
||||
tenant_id=TENANT_ID,
|
||||
file_ids=[asset.id],
|
||||
preview_sha256=preview.preview_sha256,
|
||||
idempotency_key="purge-request-1",
|
||||
approval_reference="approval-1",
|
||||
)
|
||||
|
||||
self.assertEqual(1, result.purged_files)
|
||||
self.assertEqual(1, result.released_blobs)
|
||||
self.assertIsNone(self.session.get(FileAsset, asset.id))
|
||||
retained_blob = self.session.get(FileBlob, blob.id)
|
||||
self.assertIsNotNone(retained_blob)
|
||||
self.assertEqual(0, retained_blob.ref_count)
|
||||
self.assertTrue(self.backend.exists(key))
|
||||
|
||||
replay = execute_asset_purge(
|
||||
self.session,
|
||||
tenant_id=TENANT_ID,
|
||||
file_ids=[asset.id],
|
||||
preview_sha256=preview.preview_sha256,
|
||||
idempotency_key="purge-request-1",
|
||||
approval_reference="approval-1",
|
||||
)
|
||||
self.assertTrue(replay.replayed)
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.lifecycle.get_storage_backend",
|
||||
return_value=self.backend,
|
||||
):
|
||||
gc_result = garbage_collect_unreferenced_blobs(
|
||||
self.session,
|
||||
tenant_id=TENANT_ID,
|
||||
limit=10,
|
||||
approval_reference="approval-gc-1",
|
||||
)
|
||||
self.assertEqual(1, gc_result.deleted_blobs)
|
||||
self.assertFalse(self.backend.exists(key))
|
||||
self.assertIsNone(self.session.get(FileBlob, blob.id))
|
||||
|
||||
with self.Session() as evidence_session:
|
||||
operations = evidence_session.query(RecoveryOperation).all()
|
||||
self.assertEqual(2, len(operations))
|
||||
self.assertEqual(
|
||||
{RecoveryStatus.SUCCEEDED.value},
|
||||
{operation.status for operation in operations},
|
||||
)
|
||||
|
||||
def test_s3_write_is_conditional_verified_and_idempotent(self) -> None:
|
||||
client = _WriteS3Client()
|
||||
profile = ConnectorProfile(
|
||||
id="s3-write",
|
||||
label="S3 write",
|
||||
provider="s3",
|
||||
base_path="root",
|
||||
capabilities=("browse", "write"),
|
||||
metadata={"bucket": "files"},
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_writes._s3_client",
|
||||
return_value=client,
|
||||
):
|
||||
result = write_connector_file(
|
||||
profile,
|
||||
tenant_id=TENANT_ID,
|
||||
library_id=None,
|
||||
remote_path="out/report.txt",
|
||||
data=b"report",
|
||||
content_type="text/plain",
|
||||
expected_revision=None,
|
||||
idempotency_key="connector-write-1",
|
||||
)
|
||||
replay = write_connector_file(
|
||||
profile,
|
||||
tenant_id=TENANT_ID,
|
||||
library_id=None,
|
||||
remote_path="out/report.txt",
|
||||
data=b"report",
|
||||
content_type="text/plain",
|
||||
expected_revision=None,
|
||||
idempotency_key="connector-write-1",
|
||||
)
|
||||
|
||||
self.assertEqual(RecoveryStatus.SUCCEEDED.value, result.status)
|
||||
self.assertTrue(replay.replayed)
|
||||
self.assertEqual("*", client.put_request["IfNoneMatch"])
|
||||
self.assertEqual("root/out/report.txt", client.put_request["Key"])
|
||||
self.assertNotIn(b"report", repr(self._only_operation().metadata_).encode())
|
||||
self.assertTrue(client.closed)
|
||||
|
||||
def test_s3_write_tamper_is_visible_and_blocks_another_writer(self) -> None:
|
||||
client = _WriteS3Client(tamper_metadata=True)
|
||||
profile = ConnectorProfile(
|
||||
id="s3-write",
|
||||
label="S3 write",
|
||||
provider="s3",
|
||||
capabilities=("write",),
|
||||
metadata={"bucket": "files"},
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_writes._s3_client",
|
||||
return_value=client,
|
||||
):
|
||||
result = write_connector_file(
|
||||
profile,
|
||||
tenant_id=TENANT_ID,
|
||||
library_id=None,
|
||||
remote_path="report.txt",
|
||||
data=b"report",
|
||||
content_type="text/plain",
|
||||
expected_revision=None,
|
||||
idempotency_key="connector-write-tampered",
|
||||
)
|
||||
with self.assertRaisesRegex(FileStorageError, "unresolved recovery"):
|
||||
write_connector_file(
|
||||
profile,
|
||||
tenant_id=TENANT_ID,
|
||||
library_id=None,
|
||||
remote_path="report.txt",
|
||||
data=b"replacement",
|
||||
content_type="text/plain",
|
||||
expected_revision="version-written",
|
||||
idempotency_key="connector-write-after-tamper",
|
||||
)
|
||||
|
||||
self.assertEqual(RecoveryStatus.RECOVERY_REQUIRED.value, result.status)
|
||||
self.assertEqual(
|
||||
RecoveryStatus.RECOVERY_REQUIRED.value, self._only_operation().status
|
||||
)
|
||||
|
||||
def test_s3_write_probe_failure_is_outcome_unknown_not_confirmed_absent(self) -> None:
|
||||
client = _WriteS3Client(fail_post_write_probe=True)
|
||||
profile = ConnectorProfile(
|
||||
id="s3-write",
|
||||
label="S3 write",
|
||||
provider="s3",
|
||||
capabilities=("write",),
|
||||
metadata={"bucket": "files"},
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_writes._s3_client",
|
||||
return_value=client,
|
||||
):
|
||||
result = write_connector_file(
|
||||
profile,
|
||||
tenant_id=TENANT_ID,
|
||||
library_id=None,
|
||||
remote_path="report.txt",
|
||||
data=b"report",
|
||||
content_type="text/plain",
|
||||
expected_revision=None,
|
||||
idempotency_key="connector-write-probe-failed",
|
||||
)
|
||||
|
||||
self.assertEqual(RecoveryStatus.OUTCOME_UNKNOWN.value, result.status)
|
||||
operation = self._only_operation()
|
||||
self.assertEqual(RecoveryStatus.OUTCOME_UNKNOWN.value, operation.status)
|
||||
with self.Session() as session:
|
||||
checkpoint = (
|
||||
session.query(RecoveryCheckpoint)
|
||||
.filter(RecoveryCheckpoint.operation_id == operation.id)
|
||||
.order_by(RecoveryCheckpoint.sequence.desc())
|
||||
.first()
|
||||
)
|
||||
self.assertIsNotNone(checkpoint)
|
||||
self.assertFalse(checkpoint.evidence["observed"]["probe_verified"])
|
||||
self.assertFalse(checkpoint.evidence["observed"]["present"])
|
||||
|
||||
def _only_operation(self) -> RecoveryOperation:
|
||||
with self.Session() as session:
|
||||
operations = session.query(RecoveryOperation).all()
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.core.files import (
|
||||
ManagedTabularFileAccessError,
|
||||
ManagedTabularFileValidationError,
|
||||
)
|
||||
from govoplan_files.backend.capabilities import FilesManagedTabularFileService
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
def principal(
|
||||
tenant_id: str = "tenant-1",
|
||||
*,
|
||||
scopes: tuple[str, ...] = (
|
||||
"files:file:read",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
),
|
||||
) -> ApiPrincipal:
|
||||
return ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
tenant_id=tenant_id,
|
||||
scopes=frozenset(scopes),
|
||||
),
|
||||
account=object(),
|
||||
user=SimpleNamespace(id="user-1"),
|
||||
)
|
||||
|
||||
|
||||
class FilesManagedTabularContentTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
self.session = sessionmaker(bind=self.engine)()
|
||||
self.blob = SimpleNamespace(
|
||||
id="blob-1",
|
||||
tenant_id="tenant-1",
|
||||
storage_backend="local",
|
||||
storage_key="tenants/tenant-1/files/blob-1",
|
||||
checksum_sha256="a" * 64,
|
||||
size_bytes=8,
|
||||
content_type="text/csv",
|
||||
integrity_status="verified",
|
||||
)
|
||||
self.asset = SimpleNamespace(
|
||||
id="asset-1",
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_user_id="user-1",
|
||||
current_version_id="version-1",
|
||||
display_path="Imports/source.csv",
|
||||
filename="source.csv",
|
||||
description=None,
|
||||
updated_at=datetime(2026, 8, 21, tzinfo=UTC),
|
||||
)
|
||||
self.version = SimpleNamespace(
|
||||
id="version-1",
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id="asset-1",
|
||||
blob_id="blob-1",
|
||||
version_number=1,
|
||||
filename_at_upload="source.csv",
|
||||
display_path_at_upload="Imports/source.csv",
|
||||
content_type="text/csv",
|
||||
size_bytes=8,
|
||||
checksum_sha256="a" * 64,
|
||||
created_by_user_id="user-1",
|
||||
created_at=datetime(2026, 8, 21, tzinfo=UTC),
|
||||
)
|
||||
self.provider = FilesManagedTabularFileService()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def test_lists_and_reads_only_exact_authorized_tabular_versions(self) -> None:
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.capabilities.list_recent_assets_for_user",
|
||||
return_value=[self.asset],
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.capabilities.current_version_and_blob",
|
||||
return_value=(self.version, self.blob),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.capabilities.get_asset_for_user",
|
||||
return_value=self.asset,
|
||||
),
|
||||
patch.object(
|
||||
self.session,
|
||||
"get",
|
||||
side_effect=lambda model, object_id: {
|
||||
"version-1": self.version,
|
||||
"blob-1": self.blob,
|
||||
}.get(object_id),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.capabilities.read_asset_version_bytes",
|
||||
return_value=(b"id\n1\n", self.version, self.blob),
|
||||
) as read_bytes,
|
||||
patch("govoplan_files.backend.capabilities.audit_from_principal"),
|
||||
):
|
||||
listed = self.provider.list_tabular_files(self.session, principal())
|
||||
result = self.provider.read_tabular_file(
|
||||
self.session,
|
||||
principal(),
|
||||
file_asset_id="asset-1",
|
||||
file_version_id="version-1",
|
||||
max_bytes=100,
|
||||
)
|
||||
|
||||
self.assertEqual(("version-1",), tuple(item.file_version_id for item in listed))
|
||||
self.assertEqual(b"id\n1\n", result.payload)
|
||||
self.assertTrue(result.file.current_version)
|
||||
read_bytes.assert_called_once()
|
||||
|
||||
def test_tenant_scope_permissions_and_pre_read_size_limit_fail_closed(self) -> None:
|
||||
with patch(
|
||||
"govoplan_files.backend.capabilities.get_asset_for_user",
|
||||
side_effect=FileStorageError("File not found"),
|
||||
):
|
||||
self.assertIsNone(
|
||||
self.provider.get_tabular_file(
|
||||
self.session,
|
||||
principal("tenant-2"),
|
||||
file_asset_id="asset-1",
|
||||
)
|
||||
)
|
||||
with self.assertRaises(ManagedTabularFileAccessError):
|
||||
self.provider.list_tabular_files(
|
||||
self.session,
|
||||
principal(scopes=()),
|
||||
)
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.capabilities.get_asset_for_user",
|
||||
return_value=self.asset,
|
||||
),
|
||||
patch.object(
|
||||
self.session,
|
||||
"get",
|
||||
side_effect=lambda model, object_id: {
|
||||
"version-1": self.version,
|
||||
"blob-1": self.blob,
|
||||
}.get(object_id),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.capabilities.read_asset_version_bytes"
|
||||
) as read_bytes,
|
||||
self.assertRaises(ManagedTabularFileValidationError),
|
||||
):
|
||||
self.provider.read_tabular_file(
|
||||
self.session,
|
||||
principal(),
|
||||
file_asset_id="asset-1",
|
||||
file_version_id="version-1",
|
||||
max_bytes=4,
|
||||
)
|
||||
read_bytes.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,5 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import unittest
|
||||
|
||||
from sqlalchemy import create_engine, event
|
||||
@@ -14,6 +15,7 @@ from govoplan_files.backend.db.models import (
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFormEvidenceGrant,
|
||||
)
|
||||
from govoplan_files.backend.manifest import _tenant_summary_batch
|
||||
|
||||
@@ -32,6 +34,7 @@ class FilesTenantSummaryBatchTests(unittest.TestCase):
|
||||
FileConnectorPolicy.__table__,
|
||||
FileConnectorProfile.__table__,
|
||||
FileConnectorSpace.__table__,
|
||||
FileFormEvidenceGrant.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
],
|
||||
)
|
||||
@@ -70,6 +73,21 @@ class FilesTenantSummaryBatchTests(unittest.TestCase):
|
||||
connector_profile_id="profile-1",
|
||||
provider="webdav",
|
||||
),
|
||||
FileFormEvidenceGrant(
|
||||
id="grant-1",
|
||||
tenant_id="tenant-1",
|
||||
form_instance_id="form-instance-1",
|
||||
form_definition_id="form-1",
|
||||
form_definition_revision="1",
|
||||
token_sha256="a" * 64,
|
||||
idempotency_key="grant-1",
|
||||
request_sha256="b" * 64,
|
||||
custodian_user_id="user-1",
|
||||
evidence_kind="document",
|
||||
purpose="submission attachment",
|
||||
expires_at=datetime.now(UTC),
|
||||
max_size_bytes=1024,
|
||||
),
|
||||
]
|
||||
)
|
||||
session.commit()
|
||||
@@ -88,7 +106,7 @@ class FilesTenantSummaryBatchTests(unittest.TestCase):
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", count_query)
|
||||
|
||||
self.assertEqual(5, query_count)
|
||||
self.assertEqual(6, query_count)
|
||||
self.assertEqual(
|
||||
{
|
||||
"files": 1,
|
||||
@@ -96,6 +114,7 @@ class FilesTenantSummaryBatchTests(unittest.TestCase):
|
||||
"connector_policies": 1,
|
||||
"connector_profiles": 1,
|
||||
"connector_spaces": 1,
|
||||
"form_evidence_upload_grants": 1,
|
||||
},
|
||||
counts["tenant-1"],
|
||||
)
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
"""Response-only parity/query regression with isolated synthetic SQLite data.
|
||||
|
||||
This does not benchmark imports, mutate storage, or replace route authorization.
|
||||
The isolated schema uses string columns for external module identities instead
|
||||
of importing optional modules. All response, share-state and Campaign-use
|
||||
queries execute through the real Files ORM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import unittest
|
||||
|
||||
from sqlalchemy import Column, MetaData, String, Table, create_engine, event, inspect
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.types import NullType
|
||||
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.route_support import _asset_list_response, _asset_response
|
||||
|
||||
|
||||
class UploadResponseBatchingTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite://")
|
||||
self.addCleanup(self.engine.dispose)
|
||||
metadata = MetaData()
|
||||
for model in (
|
||||
FileBlob,
|
||||
FileAsset,
|
||||
FileVersion,
|
||||
FileShare,
|
||||
CampaignAttachmentUse,
|
||||
):
|
||||
Table(
|
||||
model.__tablename__,
|
||||
metadata,
|
||||
*(
|
||||
Column(
|
||||
column.name,
|
||||
String(36)
|
||||
if isinstance(column.type, NullType)
|
||||
else column.type,
|
||||
primary_key=column.primary_key,
|
||||
nullable=column.nullable,
|
||||
default=column.default.arg
|
||||
if column.default is not None
|
||||
else None,
|
||||
server_default=column.server_default.arg
|
||||
if column.server_default is not None
|
||||
else None,
|
||||
)
|
||||
for column in model.__table__.columns
|
||||
),
|
||||
)
|
||||
metadata.create_all(self.engine)
|
||||
now = datetime.now(UTC)
|
||||
with self.engine.begin() as connection:
|
||||
|
||||
def seed(row):
|
||||
# Core inserts into separate fixture metadata avoid resolving
|
||||
# foreign identities or changing global ORM declarations.
|
||||
values = {
|
||||
attribute.columns[0].name: getattr(row, attribute.key)
|
||||
for attribute in inspect(type(row)).column_attrs
|
||||
if attribute.key in row.__dict__
|
||||
}
|
||||
connection.execute(metadata.tables[row.__tablename__].insert(), values)
|
||||
|
||||
for index in range(64):
|
||||
asset_id = f"asset-{index:03}"
|
||||
version_id = f"version-{index:03}"
|
||||
blob_id = f"blob-{index:03}"
|
||||
checksum = f"{index:064x}"
|
||||
owner_type = "group" if index % 2 else "user"
|
||||
seed(
|
||||
FileBlob(
|
||||
id=blob_id,
|
||||
tenant_id="tenant-fixture",
|
||||
storage_backend="local",
|
||||
storage_key=f"synthetic/{blob_id}",
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=index + 10,
|
||||
content_type="text/plain",
|
||||
ref_count=1,
|
||||
)
|
||||
)
|
||||
seed(
|
||||
FileAsset(
|
||||
id=asset_id,
|
||||
tenant_id="tenant-fixture",
|
||||
owner_type=owner_type,
|
||||
owner_user_id="user-fixture" if owner_type == "user" else None,
|
||||
owner_group_id="group-fixture"
|
||||
if owner_type == "group"
|
||||
else None,
|
||||
current_version_id=version_id,
|
||||
display_path=f"imported/{index:03}.txt",
|
||||
filename=f"{index:03}.txt",
|
||||
description=f"Synthetic response {index}",
|
||||
retained_until=now + timedelta(days=365) if index % 2 else None,
|
||||
legal_hold=bool(index % 3),
|
||||
lifecycle_revision=index + 2,
|
||||
lifecycle_reason=f"Retention decision {index}",
|
||||
deleted_at=now - timedelta(days=1) if index % 7 == 0 else None,
|
||||
metadata_={
|
||||
"fixture": index,
|
||||
"source_provenance": {
|
||||
"source_type": "archive",
|
||||
"revision": f"source-{index}",
|
||||
"metadata": {
|
||||
"archive_source_file_id": "synthetic-archive"
|
||||
},
|
||||
},
|
||||
"source_revision": f"source-{index}",
|
||||
},
|
||||
)
|
||||
)
|
||||
seed(
|
||||
FileVersion(
|
||||
id=version_id,
|
||||
tenant_id="tenant-fixture",
|
||||
file_asset_id=asset_id,
|
||||
blob_id=blob_id,
|
||||
version_number=1,
|
||||
filename_at_upload=f"{index:03}.txt",
|
||||
display_path_at_upload=f"imported/{index:03}.txt",
|
||||
size_bytes=index + 10,
|
||||
checksum_sha256=checksum,
|
||||
content_type="text/plain",
|
||||
)
|
||||
)
|
||||
for rank, state in enumerate(("active", "expired", "revoked")):
|
||||
seed(
|
||||
FileShare(
|
||||
id=f"share-{index:03}-{state}",
|
||||
tenant_id="tenant-fixture",
|
||||
file_asset_id=asset_id,
|
||||
target_type="group" if rank == 0 else "user",
|
||||
target_id=f"target-{state}",
|
||||
permission="write" if rank == 0 else "read",
|
||||
created_by_user_id="user-fixture",
|
||||
created_at=now - timedelta(hours=rank + 1),
|
||||
expires_at=now - timedelta(days=1)
|
||||
if state == "expired"
|
||||
else None,
|
||||
revoked_at=now - timedelta(days=1)
|
||||
if state == "revoked"
|
||||
else None,
|
||||
revoked_by_user_id="user-fixture"
|
||||
if state == "revoked"
|
||||
else None,
|
||||
)
|
||||
)
|
||||
# Duplicate sent evidence must still produce one response;
|
||||
# built-only usage and missing usage are not audit-relevant.
|
||||
stages = (
|
||||
("sent", "sent")
|
||||
if index % 3 == 0
|
||||
else ("built",)
|
||||
if index % 3 == 1
|
||||
else ()
|
||||
)
|
||||
for use_index, stage in enumerate(stages):
|
||||
seed(
|
||||
CampaignAttachmentUse(
|
||||
id=f"use-{index:03}-{use_index}",
|
||||
tenant_id="tenant-fixture",
|
||||
campaign_id="campaign-fixture",
|
||||
campaign_version_id="campaign-version-fixture",
|
||||
campaign_job_id=f"job-{use_index}",
|
||||
file_asset_id=asset_id,
|
||||
file_version_id=version_id,
|
||||
file_blob_id=blob_id,
|
||||
filename_used=f"{index:03}.txt",
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=index + 10,
|
||||
content_type="text/plain",
|
||||
use_stage=stage,
|
||||
)
|
||||
)
|
||||
|
||||
def render(self, *, batched: bool, include_shares: bool):
|
||||
statements = []
|
||||
|
||||
def record_select(
|
||||
connection, cursor, statement, parameters, context, executemany
|
||||
):
|
||||
if statement.lstrip().upper().startswith("SELECT"):
|
||||
statements.append(statement)
|
||||
|
||||
# A fresh session ensures neither path receives preloaded versions or
|
||||
# blobs. Loading the already-authorized asset list is deliberately
|
||||
# outside the measurement; this measures response construction only.
|
||||
with Session(self.engine) as session:
|
||||
assets = session.query(FileAsset).order_by(FileAsset.id).all()
|
||||
event.listen(self.engine, "before_cursor_execute", record_select)
|
||||
try:
|
||||
responses = (
|
||||
_asset_list_response(session, assets, include_shares=include_shares)
|
||||
if batched
|
||||
else [
|
||||
_asset_response(session, asset, include_shares=include_shares)
|
||||
for asset in assets
|
||||
]
|
||||
)
|
||||
result = [response.model_dump(mode="json") for response in responses]
|
||||
finally:
|
||||
event.remove(self.engine, "before_cursor_execute", record_select)
|
||||
return result, statements
|
||||
|
||||
def test_batched_response_matches_all_metadata_lifecycle_shares_and_audit_flags(
|
||||
self,
|
||||
):
|
||||
for include_shares in (False, True):
|
||||
with self.subTest(include_shares=include_shares):
|
||||
batched, _ = self.render(batched=True, include_shares=include_shares)
|
||||
individual, _ = self.render(
|
||||
batched=False, include_shares=include_shares
|
||||
)
|
||||
self.assertEqual(individual, batched)
|
||||
self.assertEqual(64, len(batched))
|
||||
for index, item in enumerate(batched):
|
||||
self.assertEqual(f"asset-{index:03}", item["id"])
|
||||
self.assertEqual(index + 2, item["lifecycle_revision"])
|
||||
self.assertEqual(
|
||||
f"Retention decision {index}", item["lifecycle_reason"]
|
||||
)
|
||||
self.assertEqual(
|
||||
bool(index % 2), item["retained_until"] is not None
|
||||
)
|
||||
self.assertEqual(bool(index % 3), item["legal_hold"])
|
||||
self.assertEqual(index % 3 == 0, item["audit_relevant"])
|
||||
self.assertEqual(index % 7 == 0, item["deleted_at"] is not None)
|
||||
self.assertEqual(f"source-{index}", item["source_revision"])
|
||||
if include_shares:
|
||||
self.assertEqual(
|
||||
[
|
||||
f"share-{index:03}-{state}"
|
||||
for state in ("active", "expired", "revoked")
|
||||
],
|
||||
[share["id"] for share in item["shares"]],
|
||||
)
|
||||
self.assertEqual(
|
||||
[True, False, False],
|
||||
[share["active"] for share in item["shares"]],
|
||||
)
|
||||
else:
|
||||
self.assertEqual([], item["shares"])
|
||||
|
||||
def test_64_asset_response_batches_real_selects_instead_of_querying_per_item(self):
|
||||
for include_shares, per_item_queries, batch_queries in (
|
||||
(False, 192, 2),
|
||||
(True, 256, 3),
|
||||
):
|
||||
with self.subTest(include_shares=include_shares):
|
||||
individual, individual_queries = self.render(
|
||||
batched=False, include_shares=include_shares
|
||||
)
|
||||
batched, batched_queries = self.render(
|
||||
batched=True, include_shares=include_shares
|
||||
)
|
||||
self.assertEqual(individual, batched)
|
||||
self.assertEqual(per_item_queries, len(individual_queries))
|
||||
self.assertEqual(batch_queries, len(batched_queries))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+7
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.27",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
@@ -14,8 +14,13 @@
|
||||
"./styles/file-manager.css": "./src/styles/file-manager.css"
|
||||
},
|
||||
"scripts": {
|
||||
"test:reload-client": "node scripts/test-files-reload-client.mjs",
|
||||
"test:archive-client": "node scripts/test-archive-client.mjs",
|
||||
"test:managed-archive": "node scripts/test-managed-archive-structure.mjs",
|
||||
"test:connector-folder-sync": "node scripts/test-connector-folder-sync-structure.mjs",
|
||||
"test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs",
|
||||
"test:file-property-filters": "node scripts/test-file-property-filters-structure.mjs",
|
||||
"test:connector-space-removal": "node scripts/test-connector-space-removal-structure.mjs",
|
||||
"test:interface-pattern-language": "node scripts/test-interface-pattern-language.mjs"
|
||||
},
|
||||
"peerDependencies": {
|
||||
@@ -26,7 +31,7 @@
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.15"
|
||||
"@govoplan/core-webui": "^0.1.45"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
// Execute the actual Files client and Core HTTP transport. Only browser I/O
|
||||
// (fetch, XHR, timers, document cookies) and temporal context are substituted.
|
||||
// No source slicing, regex assertions, live credentials, or server is involved.
|
||||
import assert from "node:assert/strict";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import vm from "node:vm";
|
||||
|
||||
const coreRoot = new URL("../../../govoplan-core/webui/", import.meta.url);
|
||||
const require = createRequire(new URL("package.json", coreRoot));
|
||||
const { transformSync } = require("esbuild");
|
||||
const compile = (url) => transformSync(readFileSync(url, "utf8"), {
|
||||
loader: "ts", format: "cjs", target: "es2022", sourcefile: url.pathname,
|
||||
define: { "import.meta.env": "{}" }
|
||||
}).code;
|
||||
const coreCode = compile(new URL("src/api/client.ts", coreRoot));
|
||||
const filesCode = compile(new URL("../src/api/files.ts", import.meta.url));
|
||||
const settle = () => new Promise((resolve) => setImmediate(resolve));
|
||||
const plain = (value) => JSON.parse(JSON.stringify(value));
|
||||
const settings = { apiBaseUrl: "https://fixture.invalid", accessToken: "", apiKey: "" };
|
||||
const destination = { owner_type: "user", owner_id: "fixture-user", path: "imports" };
|
||||
const previewResponse = (changes = {}) => ({
|
||||
preview_token: "preview-token", staged_upload_id: "stage-one", expires_at: "2099-01-01T00:00:00Z",
|
||||
entries: [], file_count: 1, directory_count: 0, archive_format: "zip",
|
||||
requires_password: false, password_verified: true, compressed_size_bytes: 100,
|
||||
expanded_size_bytes: 10, ...changes
|
||||
});
|
||||
const json = (body, status = 200) => new Response(JSON.stringify(body), {
|
||||
status, headers: { "content-type": "application/json" }
|
||||
});
|
||||
const deferred = () => {
|
||||
let resolve;
|
||||
const promise = new Promise((complete) => { resolve = complete; });
|
||||
return { promise, resolve };
|
||||
};
|
||||
|
||||
function harness() {
|
||||
const requests = [];
|
||||
const xhrRequests = [];
|
||||
const timers = new Map();
|
||||
let timerId = 0;
|
||||
let now = Date.parse("2026-09-07T10:00:00Z");
|
||||
let onFetch = () => { throw new Error("Unexpected fetch"); };
|
||||
let onXhr = () => { throw new Error("Unexpected XHR"); };
|
||||
class ClockDate extends Date { static now() { return now; } }
|
||||
class FakeXhr {
|
||||
upload = {};
|
||||
headers = new Headers();
|
||||
open(method, url) { this.method = method; this.url = url; }
|
||||
setRequestHeader(name, value) { this.headers.set(name, value); }
|
||||
getResponseHeader(name) { return name.toLowerCase() === "content-type" ? "application/json" : null; }
|
||||
send(body) { this.body = body; xhrRequests.push(this); onXhr(this); }
|
||||
progress(loaded, total, lengthComputable = true) { this.upload.onprogress?.({ loaded, total, lengthComputable }); }
|
||||
respond(body, status = 200) { this.status = status; this.statusText = status === 200 ? "OK" : "Error"; this.responseText = JSON.stringify(body); this.onload?.(); }
|
||||
}
|
||||
const context = vm.createContext({
|
||||
File, Blob, FormData, Headers, Response, URL, URLSearchParams, AbortController,
|
||||
Date: ClockDate, crypto: { randomUUID }, XMLHttpRequest: FakeXhr,
|
||||
document: { cookie: "" }, console,
|
||||
fetch: async (url, init) => {
|
||||
const request = { url, path: new URL(url).pathname, method: init?.method ?? "GET", ...init };
|
||||
requests.push(request);
|
||||
return onFetch(request);
|
||||
},
|
||||
setTimeout: (callback, delay) => { const id = ++timerId; timers.set(id, { callback, at: now + delay }); return id; },
|
||||
clearTimeout: (id) => timers.delete(id)
|
||||
});
|
||||
function load(code, dependencies) {
|
||||
context.module = { exports: {} };
|
||||
context.exports = context.module.exports;
|
||||
context.require = (name) => {
|
||||
assert.ok(name in dependencies, `Unexpected client import: ${name}`);
|
||||
return dependencies[name];
|
||||
};
|
||||
vm.runInContext(`(function(module, exports, require) {\n${code}\n})(module, exports, require);`, context);
|
||||
return context.module.exports;
|
||||
}
|
||||
const core = load(coreCode, { "../platform/temporal": { temporalRequestHeaders: () => ({}) } });
|
||||
const api = load(filesCode, { "@govoplan/core-webui": core });
|
||||
return {
|
||||
api, core, requests, xhrRequests, timers,
|
||||
fetch(handler) { onFetch = handler; },
|
||||
xhr(handler) { onXhr = handler; },
|
||||
async advance(milliseconds) {
|
||||
const target = now + milliseconds;
|
||||
while (true) {
|
||||
const next = [...timers.entries()].filter(([, timer]) => timer.at <= target).sort((left, right) => left[1].at - right[1].at)[0];
|
||||
if (!next) break;
|
||||
now = next[1].at;
|
||||
timers.delete(next[0]);
|
||||
next[1].callback();
|
||||
await settle();
|
||||
}
|
||||
now = target;
|
||||
await settle();
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
let passed = 0;
|
||||
async function test(name, run) {
|
||||
await run();
|
||||
passed += 1;
|
||||
console.log(`PASS ${name}`);
|
||||
}
|
||||
|
||||
await test("locally expired stage is released and preview reuploads the original file", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
let previews = 0;
|
||||
env.fetch((request) => {
|
||||
if (request.method === "DELETE") return new Response(null, { status: 204 });
|
||||
assert.equal(request.path, "/api/v1/files/archive-preview");
|
||||
assert.equal(request.body.get("file"), file);
|
||||
assert.equal(request.body.has("staged_upload_id"), false);
|
||||
return json(previewResponse(++previews === 1 ? { expires_at: "2026-09-07T10:00:01Z" } : { staged_upload_id: "stage-two" }));
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, destination);
|
||||
await env.advance(1100);
|
||||
const result = await env.api.previewArchiveUpload(settings, file, destination);
|
||||
assert.equal(previews, 2);
|
||||
assert.equal(result.staged_upload_id, "stage-two");
|
||||
assert.equal(env.requests.filter((request) => request.method === "DELETE").length, 1);
|
||||
assert.equal(env.requests.filter((request) => request.path.endsWith("archive-confirm")).length, 0);
|
||||
});
|
||||
|
||||
await test("server-expired cached preview retries only the read-only preview once with file bytes", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
let calls = 0;
|
||||
env.fetch((request) => {
|
||||
assert.equal(request.path, "/api/v1/files/archive-preview");
|
||||
calls += 1;
|
||||
if (calls === 2) {
|
||||
assert.equal(request.body.get("staged_upload_id"), "stage-one");
|
||||
assert.equal(request.body.get("preview_token"), "preview-token");
|
||||
assert.equal(request.body.has("file"), false);
|
||||
return json({ detail: "Archive preview expired" }, 410);
|
||||
}
|
||||
assert.equal(request.body.get("file"), file);
|
||||
assert.equal(request.body.has("staged_upload_id"), false);
|
||||
return json(previewResponse({ staged_upload_id: calls === 1 ? "stage-one" : "stage-two" }));
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, destination);
|
||||
assert.equal((await env.api.previewArchiveUpload(settings, file, destination)).staged_upload_id, "stage-two");
|
||||
assert.equal(calls, 3, "one initial preview, one expired-stage attempt, one safe reupload");
|
||||
});
|
||||
|
||||
await test("a failed fresh reupload cannot recurse into repeated preview attempts", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
let calls = 0;
|
||||
env.fetch(() => ++calls === 1 ? json(previewResponse()) : json({ detail: "Expired" }, 410));
|
||||
await env.api.previewArchiveUpload(settings, file, destination);
|
||||
await assert.rejects(env.api.previewArchiveUpload(settings, file, destination), (error) => error instanceof env.core.ApiError && error.status === 410);
|
||||
assert.equal(calls, 3);
|
||||
});
|
||||
|
||||
await test("failed confirmation never automatically retries or falls back to a file reupload", async () => {
|
||||
for (const status of [400, 410, 500]) {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
env.fetch((request) => request.path.endsWith("archive-preview") ? json(previewResponse()) : json({ detail: "Confirmation failed" }, status));
|
||||
const preview = await env.api.previewArchiveUpload(settings, file, destination);
|
||||
await assert.rejects(env.api.confirmArchiveUpload(settings, file, { ...destination, preview_token: preview.preview_token, selected_paths: ["one.txt"] }), (error) => error instanceof env.core.ApiError && error.status === status);
|
||||
const confirmations = env.requests.filter((request) => request.path.endsWith("archive-confirm"));
|
||||
assert.equal(confirmations.length, 1);
|
||||
assert.equal(confirmations[0].body.get("staged_upload_id"), "stage-one");
|
||||
assert.equal(confirmations[0].body.has("file"), false);
|
||||
assert.equal(env.requests.length, 2);
|
||||
}
|
||||
});
|
||||
|
||||
await test("progress polling failures neither reject nor retry the authoritative confirmation POST", async () => {
|
||||
for (const failure of [404, 500, "network"]) {
|
||||
const env = harness();
|
||||
const pending = deferred();
|
||||
const events = [];
|
||||
env.fetch((request) => {
|
||||
if (request.method === "POST") return pending.promise;
|
||||
if (failure === "network") throw new TypeError("Offline telemetry");
|
||||
return json({ detail: "Telemetry unavailable" }, failure);
|
||||
});
|
||||
const confirmation = env.api.confirmManagedArchive(settings, "managed-file", {
|
||||
...destination, source_version_id: "version-one", preview_token: "token", selected_paths: ["one.txt"],
|
||||
onArchiveProgress: (event) => events.push(event)
|
||||
});
|
||||
let settled = false;
|
||||
void confirmation.then(() => { settled = true; });
|
||||
await env.advance(1200);
|
||||
assert.equal(settled, false);
|
||||
assert.equal(events.length, 0);
|
||||
assert.equal(env.requests.filter((request) => request.method === "POST").length, 1);
|
||||
const polls = env.requests.filter((request) => request.method === "GET");
|
||||
assert.ok(polls.length >= 2);
|
||||
assert.ok(polls.every((request) => request.cache === "no-store"));
|
||||
pending.resolve(json({ files: [{ id: "one", size_bytes: 12345 }, { id: "two", size_bytes: 678 }] }));
|
||||
const result = await confirmation;
|
||||
assert.equal(result.files.length, 2);
|
||||
assert.deepEqual(plain(events.at(-1)), { phase: "complete", status: "complete", completed_files: 2, total_files: 2, completed_bytes: 13023, total_bytes: 13023 });
|
||||
assert.equal(env.timers.size, 0);
|
||||
assert.ok(polls.every((request) => request.signal.aborted));
|
||||
await env.advance(2000);
|
||||
assert.equal(env.requests.filter((request) => request.method === "POST").length, 1);
|
||||
assert.equal(env.requests.filter((request) => request.method === "GET").length, polls.length);
|
||||
}
|
||||
});
|
||||
|
||||
await test("confirmation errors stop polling without fabricating a successful completion", async () => {
|
||||
const env = harness();
|
||||
const pending = deferred();
|
||||
const events = [];
|
||||
env.fetch((request) => request.method === "POST" ? pending.promise : json({ detail: "Not yet available" }, 404));
|
||||
const confirmation = env.api.confirmManagedArchive(settings, "managed-file", {
|
||||
...destination, source_version_id: "version-one", preview_token: "token", selected_paths: ["one.txt"],
|
||||
onArchiveProgress: (event) => events.push(event)
|
||||
});
|
||||
const rejected = assert.rejects(confirmation, (error) => error instanceof env.core.ApiError && error.status === 409);
|
||||
await env.advance(200);
|
||||
pending.resolve(json({ detail: "Destination conflict" }, 409));
|
||||
await rejected;
|
||||
assert.equal(events.length, 0);
|
||||
assert.equal(env.timers.size, 0);
|
||||
assert.equal(env.requests.filter((request) => request.method === "POST").length, 1);
|
||||
});
|
||||
|
||||
await test("a late in-flight poll cannot replace the authoritative successful result", async () => {
|
||||
const env = harness();
|
||||
const post = deferred();
|
||||
const poll = deferred();
|
||||
const events = [];
|
||||
env.fetch((request) => request.method === "POST" ? post.promise : poll.promise);
|
||||
const confirmation = env.api.confirmManagedArchive(settings, "managed-file", {
|
||||
...destination, source_version_id: "version-one", preview_token: "token", selected_paths: ["one.txt"],
|
||||
onArchiveProgress: (event) => events.push(event)
|
||||
});
|
||||
await env.advance(200);
|
||||
assert.equal(env.requests.filter((request) => request.method === "GET").length, 1);
|
||||
post.resolve(json({ files: [{ id: "one", size_bytes: 12345 }] }));
|
||||
await confirmation;
|
||||
poll.resolve(json({ phase: "extracting", status: "running", completed_files: 0, total_files: 1, completed_bytes: 10, total_bytes: 12345 }));
|
||||
await settle();
|
||||
assert.equal(events.length, 1);
|
||||
assert.equal(events[0].status, "complete");
|
||||
assert.equal(events[0].completed_bytes, 12345);
|
||||
assert.equal(env.timers.size, 0);
|
||||
});
|
||||
|
||||
await test("XHR preview reports measured byte counters and preserves them at transfer completion", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
const events = [];
|
||||
env.xhr((xhr) => {
|
||||
assert.equal(xhr.body.get("file"), file);
|
||||
assert.equal(xhr.withCredentials, true);
|
||||
xhr.progress(1578, 4096);
|
||||
xhr.progress(4096, 4096);
|
||||
xhr.respond(previewResponse());
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, { ...destination, onProgress: (event) => events.push(event) });
|
||||
assert.deepEqual(plain(events.slice(1)), [
|
||||
{ loaded: 1578, total: 4096, percentage: 39 },
|
||||
{ loaded: 4096, total: 4096, percentage: 100 },
|
||||
{ loaded: 4096, total: 4096, percentage: 100 }
|
||||
]);
|
||||
assert.equal(env.xhrRequests.length, 1);
|
||||
assert.equal(env.requests.length, 0);
|
||||
});
|
||||
|
||||
await test("unknown-length XHR uploads never fabricate total bytes", async () => {
|
||||
const env = harness();
|
||||
const events = [];
|
||||
env.xhr((xhr) => { xhr.progress(3917, 0, false); xhr.respond(previewResponse()); });
|
||||
await env.api.previewArchiveUpload(settings, new File(["fixture"], "fixture.zip"), { ...destination, onProgress: (event) => events.push(event) });
|
||||
assert.equal(events.at(-2).percentage, null);
|
||||
assert.equal(events.at(-1).loaded, 3917);
|
||||
assert.equal(events.at(-1).total, undefined);
|
||||
assert.equal(events.at(-1).percentage, 100);
|
||||
});
|
||||
|
||||
await test("password repreview and confirmation reuse staging without another XHR transfer", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
env.xhr((xhr) => { xhr.progress(1024, 1024); xhr.respond(previewResponse()); });
|
||||
env.fetch((request) => {
|
||||
assert.equal(request.body.get("staged_upload_id"), "stage-one");
|
||||
assert.equal(request.body.has("file"), false);
|
||||
assert.equal(request.body.get("password"), "fixture-only-password");
|
||||
return request.path.endsWith("archive-preview") ? json(previewResponse({ preview_token: "verified-token" })) : json({ files: [] });
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, { ...destination, onProgress: () => {} });
|
||||
const verified = await env.api.previewArchiveUpload(settings, file, { ...destination, password: "fixture-only-password", onProgress: () => assert.fail("Staged repreview must not claim another upload") });
|
||||
await env.api.confirmArchiveUpload(settings, file, {
|
||||
...destination, password: "fixture-only-password", preview_token: verified.preview_token, selected_paths: ["one.txt"],
|
||||
onProgress: () => assert.fail("Staged confirmation must not claim another upload")
|
||||
});
|
||||
assert.equal(env.xhrRequests.length, 1);
|
||||
assert.equal(env.requests.length, 2);
|
||||
});
|
||||
|
||||
console.log(`Archive client behavior: ${passed} tests passed using real Files and Core transport code.`);
|
||||
@@ -0,0 +1,27 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
const page = readFileSync(new URL("../src/features/files/FilesPage.tsx", import.meta.url), "utf8");
|
||||
const api = readFileSync(new URL("../src/api/files.ts", import.meta.url), "utf8");
|
||||
|
||||
assert.match(api, /connector-spaces\/\$\{encodeURIComponent\(spaceId\)\}\/sync/);
|
||||
assert.match(api, /conflict_strategy: "skip", max_files: 100, max_depth: 12/);
|
||||
assert.match(page, /syncFileConnectorSpaceFolder\(settings, space\.connector_space_id/);
|
||||
assert.match(page, /recursive: folderSyncRecursive/);
|
||||
assert.match(page, /conflict_strategy: folderSyncConflictStrategy/);
|
||||
assert.match(page, /folderSyncResult\.items\.map/);
|
||||
assert.match(page, /folderSyncActionLabel\(item\.action\)/);
|
||||
|
||||
for (const contextId of [
|
||||
"files.connector-folder-sync.remote-path",
|
||||
"files.connector-folder-sync.target-folder",
|
||||
"files.connector-folder-sync.conflict-strategy",
|
||||
"files.connector-folder-sync.max-files",
|
||||
"files.connector-folder-sync.recursive"
|
||||
]) {
|
||||
assert.ok(page.includes(`helpContextId="${contextId}"`), `missing contextual help for ${contextId}`);
|
||||
}
|
||||
|
||||
assert.doesNotMatch(page, /window\.confirm\([\s\S]*connector-folder-sync/);
|
||||
|
||||
console.log("Connector folder-sync structure checks passed.");
|
||||
@@ -0,0 +1,12 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
const source = readFileSync(new URL("../src/features/files/FilesPage.tsx", import.meta.url), "utf8");
|
||||
|
||||
assert.match(source, /deleteFileConnectorSpace\(settings, target\.connector_space_id\)/);
|
||||
assert.match(source, /activeSpaceIsConnector &&[\s\S]*setConnectorSpaceRemovalTarget\(activeSpace\)/);
|
||||
assert.match(source, /Remote files and folders remain at the provider/);
|
||||
assert.match(source, /Imported GovOPlaN files, metadata and shares remain managed/);
|
||||
assert.match(source, /connector profile, credentials and remote references are not deleted/);
|
||||
|
||||
console.log("Connector-space removal structure checks passed.");
|
||||
@@ -0,0 +1,80 @@
|
||||
// Run the owning Files read helpers and real Core request cache, replacing only network I/O.
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import vm from "node:vm";
|
||||
|
||||
const coreRoot = new URL("../../../govoplan-core/webui/", import.meta.url);
|
||||
const require = createRequire(new URL("package.json", coreRoot));
|
||||
const { transformSync } = require("esbuild");
|
||||
const compile = (url) => transformSync(readFileSync(url, "utf8"), {
|
||||
loader: "ts", format: "cjs", target: "es2022", define: { "import.meta.env": "{}" }
|
||||
}).code;
|
||||
const coreCode = compile(new URL("src/api/client.ts", coreRoot));
|
||||
const filesCode = compile(new URL("../src/api/files.ts", import.meta.url));
|
||||
const settings = { apiBaseUrl: "https://fixture.invalid", apiKey: "", accessToken: "" };
|
||||
const owner = { owner_type: "user", owner_id: "fixture-user" };
|
||||
const fresh = { cache: "no-store" };
|
||||
|
||||
function harness(respond) {
|
||||
const requests = [];
|
||||
const context = vm.createContext({
|
||||
Headers, Response, FormData, URL, URLSearchParams, AbortController, console,
|
||||
document: { cookie: "" },
|
||||
fetch: async (url, init) => {
|
||||
const request = { url: new URL(url), ...init };
|
||||
requests.push(request);
|
||||
assert.equal(init?.method ?? "GET", "GET", "Reload read helpers must never mutate");
|
||||
return new Response(JSON.stringify(respond(request)), { status: 200, headers: { "content-type": "application/json" } });
|
||||
}
|
||||
});
|
||||
function load(code, dependencies) {
|
||||
context.module = { exports: {} };
|
||||
context.exports = context.module.exports;
|
||||
context.require = (name) => {
|
||||
assert.ok(name in dependencies, `Unexpected module import: ${name}`);
|
||||
return dependencies[name];
|
||||
};
|
||||
vm.runInContext(`(function(module, exports, require) {\n${code}\n})(module, exports, require);`, context);
|
||||
return context.module.exports;
|
||||
}
|
||||
const core = load(coreCode, { "../platform/temporal": { temporalRequestHeaders: () => ({}) } });
|
||||
return { api: load(filesCode, { "@govoplan/core-webui": core }), requests };
|
||||
}
|
||||
|
||||
for (const kind of ["spaces", "connector"]) {
|
||||
const { api, requests } = harness(() => kind === "spaces" ? { spaces: [] } : { items: [], path: "source/nested" });
|
||||
const read = (options) => kind === "spaces"
|
||||
? api.listFileSpaces(settings, options)
|
||||
: api.browseFileConnectorProfile(settings, "fixture-profile", { path: "source/nested" }, options);
|
||||
await read();
|
||||
await read();
|
||||
assert.equal(requests.length, 1, "normal reads retain short-lived request reuse");
|
||||
await read(fresh);
|
||||
await read(fresh);
|
||||
assert.equal(requests.length, 3, "each explicit Reload bypasses the shared recent-response cache");
|
||||
assert.equal(requests[2].cache, "no-store");
|
||||
}
|
||||
|
||||
{
|
||||
const { api, requests } = harness(({ url }) => {
|
||||
const next = url.searchParams.has("cursor");
|
||||
if (url.pathname.endsWith("/folders")) return { folders: [], total: 0, next_cursor: next ? null : "folders-2", watermark: "start" };
|
||||
if (url.pathname.endsWith("/delta")) return { full: false, files: [], folders: [], deleted: [], has_more: url.searchParams.get("since") === "start", watermark: url.searchParams.get("since") === "start" ? "middle" : "end" };
|
||||
return { files: [], total: 0, next_cursor: next ? null : "files-2" };
|
||||
});
|
||||
await api.listManagedFileSnapshot(settings, owner, fresh);
|
||||
await api.listManagedFileSnapshot(settings, owner, fresh);
|
||||
assert.equal(requests.length, 12, "two snapshots each re-read both folder/file pages and both reconciliation pages");
|
||||
assert.ok(requests.every((request) => request.cache === "no-store"));
|
||||
}
|
||||
|
||||
{
|
||||
const { api, requests } = harness(({ url }) => ({ files: [], total: 0, next_cursor: url.searchParams.has("cursor") ? null : "filtered-2" }));
|
||||
await api.listFilesByProperties(settings, { ...owner, campaign_usage: "linked", audit_relevant: true }, fresh);
|
||||
await api.listFilesByProperties(settings, { ...owner, campaign_usage: "linked", audit_relevant: true }, fresh);
|
||||
assert.equal(requests.length, 4, "each filtered Reload must re-read every server result page");
|
||||
assert.ok(requests.every((request) => request.cache === "no-store" && request.url.searchParams.get("campaign_usage") === "linked" && request.url.searchParams.get("audit_relevant") === "true"));
|
||||
}
|
||||
|
||||
console.log("Files Reload: 4 real-client cache, read-only, pagination and filter checks passed.");
|
||||
@@ -38,13 +38,23 @@ assert.match(filesPage, /topicId: "files\.workflow\.organize-managed-files"/);
|
||||
assert.match(filesPage, /disabledReason=\{uploadBlocker\}/);
|
||||
assert.match(filesPage, /disabledReason=\{deleteBlocker\}/);
|
||||
assert.match(filesPage, /<ConfirmDialog[\s\S]*tone="danger"/);
|
||||
assert.match(filesPage, /className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen files-page"/);
|
||||
assert.match(filesPage, /<WorkspaceFrame as="main" height="viewport" surface="plain"/);
|
||||
assert.match(filesPage, /const toolbar = <WorkspaceActionBar\s+title="i18n:govoplan-files\.files\.6ce6c512"\s+titleHelp=\{<DocumentationHelpLink reference=\{FILES_WORKFLOW_DOCUMENTATION\} \/>\}\s+scope="workspace"\s+variant="collection"\s+refreshable/);
|
||||
assert.match(filesPage, /\{toolbar\}[\s\S]*className=\{`file-manager-shell/);
|
||||
assert.match(filesPage, /const selectionToolbar = <WorkspaceActionBar\s+scope="detail-pane"/);
|
||||
assert.match(filesPage, /<Dialog\s+open=\{toolsPanel !== null\}/);
|
||||
assert.match(filesPage, /destructiveActions=\{<Button variant="danger"/);
|
||||
const reload = filesPage.slice(filesPage.indexOf(" async function reloadCurrentView()"), filesPage.indexOf(" function applyManagedSpaceDelta"));
|
||||
for (const readOperation of ["listManagedFileSnapshot", "resolveFilePatterns", "listFilesByProperties", "browseFileConnectorProfile"]) assert.ok(reload.includes(readOperation));
|
||||
assert.doesNotMatch(reload, /syncConnector|importFile|uploadFile|deleteFile|resetTransientState|setCurrentFolder|clearSelection|setSearchActive/);
|
||||
assert.match(reload, /if \(!isCurrent\(\)\) return;/);
|
||||
assert.doesNotMatch(reload, /setBusy\(/, "manual reload owns only its own busy state");
|
||||
assert.doesNotMatch(filesPage, /window\.(?:alert|confirm)\(/);
|
||||
|
||||
assert.doesNotMatch(`${connector}\n${integrity}\n${filesPage}\n${moduleSource}`, /@govoplan\/(?:campaign|mail|docs)-webui|govoplan_(?:campaign|mail|docs)/);
|
||||
assert.match(moduleSource, /"files\.connectors"/);
|
||||
assert.match(moduleSource, /"files\.fileExplorer"/);
|
||||
assert.match(styles, /@media \(max-width: 1050px\)[\s\S]*\.files-page \.file-manager-shell[\s\S]*grid-template-columns: 1fr/);
|
||||
assert.match(styles, /@media \(max-width: 1100px\)[\s\S]*\.files-page \.file-manager-shell[\s\S]*grid-template-columns: 1fr/);
|
||||
assert.match(styles, /@media \(max-width: 760px\)[\s\S]*\.file-connector-profile-row[\s\S]*grid-template-columns: 1fr/);
|
||||
|
||||
for (const archetype of ["Directory/explorer", "Adaptive create/edit", "Effective-policy editor", "Dashboard widget"]) {
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import vm from "node:vm";
|
||||
|
||||
const read = (path) => readFileSync(new URL(path, import.meta.url), "utf8");
|
||||
const page = read("../src/features/files/FilesPage.tsx");
|
||||
const menu = read("../src/features/files/components/FileManagerComponents.tsx");
|
||||
const api = read("../src/api/files.ts");
|
||||
const translations = read("../src/i18n/generatedTranslations.ts");
|
||||
|
||||
assert.match(page, /selectedFiles\.length === 1 && selectedFolderPaths\.size === 0 && ARCHIVE_FILENAME_PATTERN/);
|
||||
assert.match(page, /function openManagedArchive[\s\S]*?!canUpload \|\| !canDownload[\s\S]*?setManagedArchiveFile\(file\)/);
|
||||
assert.match(page, /disabledReason=\{unpackBlocker\}/);
|
||||
assert.match(menu, /onClick=\{onUnpackArchive\} disabled=\{!canUnpackArchive\}/);
|
||||
assert.match(page, /file: File \| ManagedFile/);
|
||||
assert.match(page, /previewManagedArchive\(settings, file\.id,[\s\S]*?source_version_id: file\.version_id/);
|
||||
assert.match(page, /confirmManagedArchive\(settings, managedArchiveFile\.id,[\s\S]*?source_version_id: managedArchiveFile\.version_id/);
|
||||
assert.match(page, /archivePreview && \(archiveFile \|\| managedArchiveFile\)/);
|
||||
assert.match(page, /loadArchivePreview\(\(managedArchiveFile \|\| archiveFile\)!/);
|
||||
assert.match(page, /if \(managedArchiveFile\) \{ setArchivePreview\(null\); setSelectedArchivePaths/);
|
||||
assert.match(page, /onClose=\{\(\) => \{ if \(!busy\) closeDialog\(\); \}\}/);
|
||||
assert.match(page, /<LoadingFrame loading=\{uploadActive\}[\s\S]*?indicator="none"[\s\S]*?progress=\{operationProgressValue\}/);
|
||||
assert.match(page, /<div inert=\{uploadActive\}>/);
|
||||
assert.match(page, /onArchiveProgress: \(progress: ArchiveOperationProgress\)/);
|
||||
assert.match(page, /archiveProgress\?\.status === "complete" \? 100/);
|
||||
assert.match(page, /archiveProgress.phase !== "finalizing"[\s\S]*?serverProgressRatio < 1/);
|
||||
assert.match(menu, /closeDisabled=\{busy\}[\s\S]*?closeOnBackdrop=\{!busy\}/);
|
||||
const feedback = page.slice(page.indexOf(" const selectedArchiveBytes ="), page.indexOf(" const connectorLocationLabel ="));
|
||||
function operationFeedback(archiveProgress, uploadPhase = "unpacking", uploadProgress = null) {
|
||||
const context = { archiveProgress, uploadPhase, uploadProgress, selectedArchivePaths: new Set(["one.txt", "empty.txt"]),
|
||||
archivePreview: { entries: [{ path: "one.txt", kind: "file", size_bytes: 3 }, { path: "empty.txt", kind: "file", size_bytes: 0 }] },
|
||||
formatBytes: (value) => `${value} B`, i18nMessage: (key, values) => ({ key, values }) };
|
||||
vm.runInNewContext(`${feedback}\nglobalThis.result = { value: operationProgressValue, label: operationProgressLabel, phase: operationBusyLabel };`, context);
|
||||
return context.result;
|
||||
}
|
||||
const measured = { phase: "extracting", status: "running", completed_files: 1, total_files: 2, completed_bytes: 3, total_bytes: 6 };
|
||||
assert.equal(operationFeedback(measured).value, 50);
|
||||
assert.equal(operationFeedback({ ...measured, phase: "finalizing", completed_files: 2, completed_bytes: 6 }).value, null, "finalization does not fabricate completion");
|
||||
assert.equal(operationFeedback({ ...measured, completed_files: 2, completed_bytes: 6 }).value, null, "processed bytes alone do not mean commit success");
|
||||
assert.equal(operationFeedback({ ...measured, phase: "complete", status: "complete" }).value, 100);
|
||||
assert.equal(operationFeedback({ ...measured, total_bytes: 0, completed_bytes: 0 }).value, 50, "empty members use actual file counts");
|
||||
assert.equal(operationFeedback(null).value, null, "no measured result remains indeterminate");
|
||||
assert.equal(operationFeedback(null).label.values.total, 2, "selected totals remain available before polling responds");
|
||||
assert.equal(operationFeedback(null).label.values.bytes, "3 B");
|
||||
assert.equal(operationFeedback(null, "uploading", 24).value, 24);
|
||||
assert.equal(operationFeedback(null, "uploading", 100).value, null, "completed transfer is not completed extraction");
|
||||
assert.equal(page.split('event.dataTransfer.effectAllowed = "copyMove";').length - 1, 2);
|
||||
assert.doesNotMatch(page, /effectAllowed = "i18n:/, "browser drag/drop enums must not be translated");
|
||||
for (const name of ["previewManagedArchive", "confirmManagedArchive"]) {
|
||||
const body = api.slice(api.indexOf(`export function ${name}`)).split("\n}\n", 1)[0];
|
||||
assert.match(body, /encodeURIComponent\(fileId\)/);
|
||||
assert.match(body, /body: JSON\.stringify\(/);
|
||||
assert.doesNotMatch(body, /FormData|downloadFile|Blob|createObjectURL/);
|
||||
}
|
||||
assert.match(page, /function resetArchiveUploadState\(\) \{\s*void releaseArchivePreview\(settings, archiveFile\)/, "reset/cancel releases any temporary archive before forgetting it");
|
||||
assert.match(page, /archiveFile && archiveFile !== file\) void releaseArchivePreview/, "replacing a selected archive releases its previous temporary upload");
|
||||
for (const key of ["unpack", "preview", "select_one", "source", "expires", "extracting", "change_destination"]) {
|
||||
assert.equal(translations.split(`"i18n:govoplan-files.managed_archive.${key}":`).length - 1, 2, `${key} must have EN and DE text`);
|
||||
}
|
||||
console.log("Managed archive actions reuse the preview dialog and send source-bound JSON without browser re-upload.");
|
||||
+345
-39
@@ -10,11 +10,15 @@ import {
|
||||
type FilesManagedFileLinkTarget,
|
||||
type ReferenceOptionProvider
|
||||
} from "@govoplan/core-webui";
|
||||
export { fetchResourceAccessExplanation } from "@govoplan/core-webui";
|
||||
export {
|
||||
fetchResourceAccessExplanation,
|
||||
fetchResourceAccessExplanationSubjects
|
||||
} from "@govoplan/core-webui";
|
||||
export type {
|
||||
AccessDecisionProvenanceItem,
|
||||
ResourceAccessExplanationUser as AccessExplanationUser,
|
||||
ResourceAccessExplanationResponse
|
||||
ResourceAccessExplanationResponse,
|
||||
ResourceAccessExplanationSubjectsResponse
|
||||
} from "@govoplan/core-webui";
|
||||
|
||||
export type FileSpace = {
|
||||
@@ -291,6 +295,7 @@ export type FileConnectorProvider = {
|
||||
installed: boolean;
|
||||
browse_supported: boolean;
|
||||
import_supported: boolean;
|
||||
write_supported: boolean;
|
||||
optional_dependency?: string | null;
|
||||
permission_model: string;
|
||||
sync_strategy: string;
|
||||
@@ -339,6 +344,7 @@ export type FileConnectorSpacePayload = {
|
||||
library_id?: string | null;
|
||||
remote_path?: string;
|
||||
sync_mode?: "manual";
|
||||
read_only?: boolean;
|
||||
metadata?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
@@ -357,6 +363,10 @@ export type ManagedFile = {
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
deleted_at?: string | null;
|
||||
retained_until?: string | null;
|
||||
legal_hold?: boolean;
|
||||
lifecycle_revision?: number;
|
||||
lifecycle_reason?: string | null;
|
||||
audit_relevant: boolean;
|
||||
metadata?: Record<string, unknown> | null;
|
||||
source_provenance?: FileSourceProvenance | null;
|
||||
@@ -379,6 +389,14 @@ export type ManagedFileSnapshotResponse = {files: ManagedFile[];folders: FileFol
|
||||
export type FileSpacesResponse = {spaces: FileSpace[];};
|
||||
export type FileUploadResponse = {files: ManagedFile[];};
|
||||
export type FileUploadProgress = {loaded: number;total?: number;percentage: number | null;};
|
||||
export type ArchiveOperationProgress = {
|
||||
phase: "inspecting" | "extracting" | "storing" | "finalizing" | "complete" | "failed";
|
||||
completed_files: number;
|
||||
total_files: number;
|
||||
completed_bytes: number;
|
||||
total_bytes: number;
|
||||
status: "running" | "complete" | "failed";
|
||||
};
|
||||
export type ArchivePreviewEntry = {
|
||||
path: string;
|
||||
kind: "file" | "directory";
|
||||
@@ -387,6 +405,7 @@ export type ArchivePreviewEntry = {
|
||||
encrypted: boolean;
|
||||
};
|
||||
export type ArchivePreviewResponse = {
|
||||
staged_upload_id?: string | null;
|
||||
preview_token: string;
|
||||
archive_format: string;
|
||||
entries: ArchivePreviewEntry[];
|
||||
@@ -416,6 +435,64 @@ export type FileConnectorSyncResponse = {
|
||||
previous_version_id?: string | null;
|
||||
current_version_id: string;
|
||||
};
|
||||
export type FileConnectorFolderSyncAction = "created" | "updated" | "unchanged" | "skipped" | "conflict" | "policy_denied" | "failed";
|
||||
export type FileConnectorFolderSyncItem = {
|
||||
source_path: string;
|
||||
target_path?: string | null;
|
||||
action: FileConnectorFolderSyncAction;
|
||||
file?: ManagedFile | null;
|
||||
previous_version_id?: string | null;
|
||||
current_version_id?: string | null;
|
||||
source_revision?: string | null;
|
||||
detail?: string | null;
|
||||
policy_decision?: FileConnectorPolicyDecision | null;
|
||||
};
|
||||
export type FileConnectorFolderSyncSummary = {
|
||||
discovered: number;
|
||||
created: number;
|
||||
updated: number;
|
||||
unchanged: number;
|
||||
skipped: number;
|
||||
conflicts: number;
|
||||
policy_denied: number;
|
||||
failed: number;
|
||||
};
|
||||
export type FileConnectorFolderSyncResponse = {
|
||||
connector_space_id: string;
|
||||
connector_profile_id: string;
|
||||
provider: string;
|
||||
remote_path: string;
|
||||
target_folder: string;
|
||||
recursive: boolean;
|
||||
truncated: boolean;
|
||||
summary: FileConnectorFolderSyncSummary;
|
||||
items: FileConnectorFolderSyncItem[];
|
||||
};
|
||||
export type FileConnectorFolderSyncPayload = {
|
||||
path?: string;
|
||||
target_folder?: string | null;
|
||||
recursive?: boolean;
|
||||
conflict_strategy?: "reject" | "overwrite" | "rename" | "skip";
|
||||
max_files?: number;
|
||||
max_depth?: number;
|
||||
metadata?: Record<string, unknown>;
|
||||
};
|
||||
export type FileConnectorWritePayload = {
|
||||
file_id: string;
|
||||
remote_path: string;
|
||||
idempotency_key: string;
|
||||
expected_revision?: string | null;
|
||||
};
|
||||
export type FileConnectorWriteResponse = {
|
||||
recovery_operation_id: string;
|
||||
status: string;
|
||||
replayed: boolean;
|
||||
provider: string;
|
||||
remote_path: string;
|
||||
revision?: string | null;
|
||||
checksum_sha256: string;
|
||||
size_bytes: number;
|
||||
};
|
||||
export type FileFolder = {
|
||||
id: string;
|
||||
tenant_id: string;
|
||||
@@ -430,11 +507,53 @@ export type FileFoldersResponse = {folders: FileFolder[];cursor?: string | null;
|
||||
|
||||
const DEFAULT_MANAGED_FILE_WINDOW_SIZE = 500;
|
||||
export type FolderDeleteResponse = {deleted_folders: number;deleted_files: number;};
|
||||
export type FolderRestoreResponse = {restored_folders: number;restored_files: number;};
|
||||
export type BulkDeleteResponse = {deleted_count: number;};
|
||||
export type FileRestoreResponse = {restored_count: number;};
|
||||
export type FileLifecycleUpdatePayload = {
|
||||
retained_until?: string | null;
|
||||
legal_hold: boolean;
|
||||
reason: string;
|
||||
expected_revision: number;
|
||||
};
|
||||
export type FilePurgePreviewItem = {
|
||||
file_id: string;
|
||||
filename: string;
|
||||
lifecycle_revision: number;
|
||||
deleted_at?: string | null;
|
||||
retained_until?: string | null;
|
||||
legal_hold: boolean;
|
||||
blockers: string[];
|
||||
blob_ids: string[];
|
||||
};
|
||||
export type FilePurgePreviewResponse = {
|
||||
preview_sha256: string;
|
||||
eligible: boolean;
|
||||
items: FilePurgePreviewItem[];
|
||||
};
|
||||
export type FilePurgeExecutePayload = {
|
||||
file_ids: string[];
|
||||
preview_sha256: string;
|
||||
idempotency_key: string;
|
||||
approval_reference: string;
|
||||
confirmation: "PURGE";
|
||||
};
|
||||
export type FilePurgeResponse = {
|
||||
recovery_operation_id: string;
|
||||
status: string;
|
||||
replayed: boolean;
|
||||
purged_files: number;
|
||||
released_blobs: number;
|
||||
};
|
||||
export type FileBlobGcResponse = {
|
||||
inspected_blobs: number;
|
||||
deleted_blobs: number;
|
||||
unresolved_operation_ids: string[];
|
||||
};
|
||||
export type RenameResponse = {dry_run: boolean;items: {kind: "file" | "folder";id: string;file_id?: string | null;folder_path?: string | null;old_path: string;new_path: string;}[];};
|
||||
export type TransferResponse = {operation: "move" | "copy";files: number;folders: number;};
|
||||
export type ConflictAction = "overwrite" | "rename" | "skip";
|
||||
export type ConflictStrategy = "reject" | "overwrite" | "rename";
|
||||
export type ConflictStrategy = "reject" | "overwrite" | "rename" | "skip";
|
||||
export type ConflictResolution = {target_path: string;action: ConflictAction;new_path?: string;};
|
||||
export type PatternResolveResponse = {
|
||||
patterns: {pattern: string;matches: ManagedFile[];}[];
|
||||
@@ -442,18 +561,20 @@ export type PatternResolveResponse = {
|
||||
};
|
||||
|
||||
|
||||
export function listFileSpaces(settings: ApiSettings): Promise<FileSpacesResponse> {
|
||||
return apiFetch<FileSpacesResponse>(settings, "/api/v1/files/spaces");
|
||||
type FileReadOptions = Pick<RequestInit, "cache" | "signal">;
|
||||
|
||||
export function listFileSpaces(settings: ApiSettings, options?: FileReadOptions): Promise<FileSpacesResponse> {
|
||||
return apiFetch<FileSpacesResponse>(settings, "/api/v1/files/spaces", options);
|
||||
}
|
||||
|
||||
|
||||
export function listFolders(settings: ApiSettings, params: {owner_type: "user" | "group";owner_id: string;page_size?: number;cursor?: string | null;}): Promise<FileFoldersResponse> {
|
||||
export function listFolders(settings: ApiSettings, params: {owner_type: "user" | "group";owner_id: string;page_size?: number;cursor?: string | null;}, options?: FileReadOptions): Promise<FileFoldersResponse> {
|
||||
const search = new URLSearchParams();
|
||||
search.set("owner_type", params.owner_type);
|
||||
search.set("owner_id", params.owner_id);
|
||||
if (params.page_size) search.set("page_size", String(params.page_size));
|
||||
if (params.cursor) search.set("cursor", params.cursor);
|
||||
return apiFetch<FileFoldersResponse>(settings, `/api/v1/files/folders?${search.toString()}`);
|
||||
return apiFetch<FileFoldersResponse>(settings, `/api/v1/files/folders?${search.toString()}`, options);
|
||||
}
|
||||
|
||||
export function createFolder(
|
||||
@@ -470,13 +591,13 @@ payload: {owner_type: "user" | "group";owner_id: string;path: string;recursive?:
|
||||
return apiFetch<FolderDeleteResponse>(settings, "/api/v1/files/folders/delete", { method: "POST", body: JSON.stringify({ recursive: true, ...payload }) });
|
||||
}
|
||||
|
||||
export function listFiles(settings: ApiSettings, params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;campaign_usage?: FileCampaignUsageFilter;audit_relevant?: boolean;page_size?: number;cursor?: string | null;} = {}): Promise<FileListResponse> {
|
||||
export function listFiles(settings: ApiSettings, params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;campaign_usage?: FileCampaignUsageFilter;audit_relevant?: boolean;sort?: "path" | "recent";page_size?: number;cursor?: string | null;} = {}, options?: FileReadOptions): Promise<FileListResponse> {
|
||||
const search = new URLSearchParams();
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
if (value !== undefined && value !== null && value !== "") search.set(key, String(value));
|
||||
}
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileListResponse>(settings, `/api/v1/files${suffix}`);
|
||||
return apiFetch<FileListResponse>(settings, `/api/v1/files${suffix}`, options);
|
||||
}
|
||||
|
||||
export async function listFilesByProperties(
|
||||
@@ -488,7 +609,7 @@ params: {
|
||||
campaign_usage?: FileCampaignUsageFilter;
|
||||
audit_relevant?: boolean;
|
||||
page_size?: number;
|
||||
})
|
||||
}, options?: FileReadOptions)
|
||||
: Promise<{files: ManagedFile[];total: number;}> {
|
||||
const pageSize = params.page_size ?? DEFAULT_MANAGED_FILE_WINDOW_SIZE;
|
||||
let cursor: string | null | undefined = null;
|
||||
@@ -499,7 +620,7 @@ params: {
|
||||
...params,
|
||||
page_size: pageSize,
|
||||
cursor
|
||||
});
|
||||
}, options);
|
||||
files = files.concat(response.files);
|
||||
total = response.total;
|
||||
cursor = response.next_cursor;
|
||||
@@ -509,14 +630,14 @@ params: {
|
||||
|
||||
export function listFilesDelta(
|
||||
settings: ApiSettings,
|
||||
params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;since?: string;limit?: number;} = {})
|
||||
params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;since?: string;limit?: number;} = {}, options?: FileReadOptions)
|
||||
: Promise<FileDeltaResponse> {
|
||||
const search = new URLSearchParams();
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
if (value !== undefined && value !== null && value !== "") search.set(key, String(value));
|
||||
}
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileDeltaResponse>(settings, `/api/v1/files/delta${suffix}`);
|
||||
return apiFetch<FileDeltaResponse>(settings, `/api/v1/files/delta${suffix}`, options);
|
||||
}
|
||||
|
||||
function applyDeltaToSnapshot(
|
||||
@@ -538,7 +659,7 @@ response: FileDeltaResponse)
|
||||
|
||||
export async function listManagedFileSnapshot(
|
||||
settings: ApiSettings,
|
||||
params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page_size?: number;})
|
||||
params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page_size?: number;}, options?: FileReadOptions)
|
||||
: Promise<ManagedFileSnapshotResponse> {
|
||||
const pageSize = params.page_size ?? DEFAULT_MANAGED_FILE_WINDOW_SIZE;
|
||||
let watermark: string | null | undefined = null;
|
||||
@@ -550,7 +671,7 @@ params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page
|
||||
owner_id: params.owner_id,
|
||||
page_size: pageSize,
|
||||
cursor: folderCursor
|
||||
});
|
||||
}, options);
|
||||
watermark = watermark || response.watermark;
|
||||
folders = folders.concat(response.folders);
|
||||
folderCursor = response.next_cursor;
|
||||
@@ -565,7 +686,7 @@ params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page
|
||||
path_prefix: params.path_prefix,
|
||||
page_size: pageSize,
|
||||
cursor: fileCursor
|
||||
});
|
||||
}, options);
|
||||
watermark = watermark || response.watermark;
|
||||
files = files.concat(response.files);
|
||||
fileCursor = response.next_cursor;
|
||||
@@ -581,7 +702,7 @@ params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page
|
||||
path_prefix: params.path_prefix,
|
||||
since,
|
||||
limit: pageSize
|
||||
});
|
||||
}, options);
|
||||
const snapshot = applyDeltaToSnapshot(files, folders, response);
|
||||
files = snapshot.files;
|
||||
folders = snapshot.folders;
|
||||
@@ -626,7 +747,17 @@ options: {
|
||||
return apiFetch<FileUploadResponse>(settings, "/api/v1/files/upload", { method: "POST", body: form });
|
||||
}
|
||||
|
||||
export function previewArchiveUpload(
|
||||
const stagedArchivePreviews = new WeakMap<File, ArchivePreviewResponse>();
|
||||
|
||||
export async function releaseArchivePreview(settings: ApiSettings, file: File | null): Promise<void> {
|
||||
const preview = file ? stagedArchivePreviews.get(file) : undefined;
|
||||
if (file) stagedArchivePreviews.delete(file);
|
||||
if (preview?.staged_upload_id) {
|
||||
await apiFetch(settings, `/api/v1/files/archive-staging/${encodeURIComponent(preview.staged_upload_id)}`, { method: "DELETE" }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
export async function previewArchiveUpload(
|
||||
settings: ApiSettings,
|
||||
file: File,
|
||||
options: {
|
||||
@@ -635,16 +766,67 @@ options: {
|
||||
path?: string;
|
||||
campaign_id?: string;
|
||||
password?: string;
|
||||
onProgress?: (progress: FileUploadProgress) => void;
|
||||
})
|
||||
: Promise<ArchivePreviewResponse> {
|
||||
let staged = stagedArchivePreviews.get(file);
|
||||
if (staged && !(Date.parse(staged.expires_at) > Date.now())) {
|
||||
void releaseArchivePreview(settings, file);
|
||||
staged = undefined;
|
||||
}
|
||||
const form = new FormData();
|
||||
form.append("file", file);
|
||||
if (staged?.staged_upload_id) {
|
||||
form.append("staged_upload_id", staged.staged_upload_id);
|
||||
form.append("preview_token", staged.preview_token);
|
||||
} else {
|
||||
form.append("file", file);
|
||||
}
|
||||
form.append("retain_upload", "true");
|
||||
form.append("owner_type", options.owner_type);
|
||||
form.append("owner_id", options.owner_id);
|
||||
form.append("path", options.path ?? "");
|
||||
if (options.campaign_id) form.append("campaign_id", options.campaign_id);
|
||||
if (options.password) form.append("password", options.password);
|
||||
return apiFetch<ArchivePreviewResponse>(settings, "/api/v1/files/archive-preview", { method: "POST", body: form });
|
||||
try {
|
||||
const result = options.onProgress && !staged?.staged_upload_id
|
||||
? await uploadFilesWithProgress<ArchivePreviewResponse>(settings, form, options.onProgress, "/api/v1/files/archive-preview")
|
||||
: await apiFetch<ArchivePreviewResponse>(settings, "/api/v1/files/archive-preview", { method: "POST", body: form });
|
||||
stagedArchivePreviews.set(file, result);
|
||||
return result;
|
||||
} catch (error) {
|
||||
// Preview is read-only with respect to managed files: an expired temporary
|
||||
// stage can safely be uploaded once again. Never retry confirmation.
|
||||
if (staged?.staged_upload_id && error instanceof ApiError && error.status === 410) {
|
||||
stagedArchivePreviews.delete(file);
|
||||
return previewArchiveUpload(settings, file, options);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
type ManagedArchiveOptions = {
|
||||
source_version_id: string;
|
||||
owner_type: "user" | "group";
|
||||
owner_id: string;
|
||||
path?: string;
|
||||
password?: string;
|
||||
};
|
||||
|
||||
export function previewManagedArchive(settings: ApiSettings, fileId: string, options: ManagedArchiveOptions): Promise<ArchivePreviewResponse> {
|
||||
return apiFetch<ArchivePreviewResponse>(settings, `/api/v1/files/${encodeURIComponent(fileId)}/archive-preview`, {
|
||||
method: "POST", body: JSON.stringify(options)
|
||||
});
|
||||
}
|
||||
|
||||
export function confirmManagedArchive(settings: ApiSettings, fileId: string, options: ManagedArchiveOptions & {
|
||||
preview_token: string;
|
||||
selected_paths: string[];
|
||||
onArchiveProgress?: (progress: ArchiveOperationProgress) => void;
|
||||
}): Promise<FileUploadResponse> {
|
||||
const { onArchiveProgress, ...payload } = options;
|
||||
return runArchiveOperation(settings, (operation_id) => apiFetch<FileUploadResponse>(settings, `/api/v1/files/${encodeURIComponent(fileId)}/archive-confirm`, {
|
||||
method: "POST", body: JSON.stringify({ ...payload, operation_id })
|
||||
}), onArchiveProgress);
|
||||
}
|
||||
|
||||
export function confirmArchiveUpload(
|
||||
@@ -664,10 +846,16 @@ options: {
|
||||
source_revision?: string;
|
||||
connector_policy_sources?: FileConnectorPolicySource[];
|
||||
onProgress?: (progress: FileUploadProgress) => void;
|
||||
onArchiveProgress?: (progress: ArchiveOperationProgress) => void;
|
||||
})
|
||||
: Promise<FileUploadResponse> {
|
||||
const form = new FormData();
|
||||
form.append("file", file);
|
||||
const staged = stagedArchivePreviews.get(file);
|
||||
if (staged?.staged_upload_id && staged.preview_token === options.preview_token) {
|
||||
form.append("staged_upload_id", staged.staged_upload_id);
|
||||
} else {
|
||||
form.append("file", file);
|
||||
}
|
||||
form.append("preview_token", options.preview_token);
|
||||
form.append("selected_paths_json", JSON.stringify(options.selected_paths));
|
||||
form.append("owner_type", options.owner_type);
|
||||
@@ -680,23 +868,59 @@ options: {
|
||||
if (options.source_provenance) form.append("source_provenance_json", JSON.stringify(options.source_provenance));
|
||||
if (options.source_revision) form.append("source_revision", options.source_revision);
|
||||
if (options.connector_policy_sources?.length) form.append("connector_policy_json", JSON.stringify({ sources: options.connector_policy_sources }));
|
||||
if (options.onProgress) {
|
||||
return uploadFilesWithProgress(
|
||||
settings,
|
||||
form,
|
||||
options.onProgress,
|
||||
"/api/v1/files/archive-confirm"
|
||||
);
|
||||
}
|
||||
return apiFetch<FileUploadResponse>(settings, "/api/v1/files/archive-confirm", { method: "POST", body: form });
|
||||
return runArchiveOperation(settings, (operationId) => {
|
||||
if (operationId) form.append("operation_id", operationId);
|
||||
return options.onProgress && !form.has("staged_upload_id")
|
||||
? uploadFilesWithProgress(settings, form, options.onProgress, "/api/v1/files/archive-confirm")
|
||||
: apiFetch<FileUploadResponse>(settings, "/api/v1/files/archive-confirm", { method: "POST", body: form });
|
||||
}, options.onArchiveProgress).then((result) => {
|
||||
stagedArchivePreviews.delete(file);
|
||||
return result;
|
||||
});
|
||||
}
|
||||
|
||||
function uploadFilesWithProgress(
|
||||
async function runArchiveOperation(
|
||||
settings: ApiSettings,
|
||||
request: (operationId?: string) => Promise<FileUploadResponse>,
|
||||
onProgress?: (progress: ArchiveOperationProgress) => void
|
||||
): Promise<FileUploadResponse> {
|
||||
if (!onProgress) return request();
|
||||
const operationId = crypto.randomUUID();
|
||||
let stopped = false;
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const abort = new AbortController();
|
||||
const poll = async () => {
|
||||
try {
|
||||
const progress = await apiFetch<ArchiveOperationProgress>(settings, `/api/v1/files/archive-progress/${operationId}`, { cache: "no-store", signal: abort.signal });
|
||||
if (!stopped) onProgress(progress);
|
||||
} catch {
|
||||
// Missing/expired or temporarily unreachable telemetry never retries or
|
||||
// fails an import. Its authoritative result remains the POST response.
|
||||
} finally {
|
||||
if (!stopped) timer = setTimeout(() => void poll(), 500);
|
||||
}
|
||||
};
|
||||
timer = setTimeout(() => void poll(), 150);
|
||||
try {
|
||||
const result = await request(operationId);
|
||||
stopped = true;
|
||||
const bytes = result.files.reduce((total, file) => total + file.size_bytes, 0);
|
||||
onProgress({ phase: "complete", status: "complete", completed_files: result.files.length,
|
||||
total_files: result.files.length, completed_bytes: bytes, total_bytes: bytes });
|
||||
return result;
|
||||
} finally {
|
||||
stopped = true;
|
||||
clearTimeout(timer);
|
||||
abort.abort();
|
||||
}
|
||||
}
|
||||
|
||||
function uploadFilesWithProgress<T = FileUploadResponse>(
|
||||
settings: ApiSettings,
|
||||
form: FormData,
|
||||
onProgress: (progress: FileUploadProgress) => void,
|
||||
endpoint = "/api/v1/files/upload")
|
||||
: Promise<FileUploadResponse> {
|
||||
: Promise<T> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const xhr = new XMLHttpRequest();
|
||||
xhr.open("POST", apiUrl(settings, endpoint));
|
||||
@@ -705,13 +929,15 @@ endpoint = "/api/v1/files/upload")
|
||||
const csrf = csrfToken();
|
||||
if (csrf) xhr.setRequestHeader("X-CSRF-Token", csrf);
|
||||
|
||||
let lastProgress: FileUploadProgress = { loaded: 0, total: undefined, percentage: 0 };
|
||||
xhr.upload.onprogress = (event) => {
|
||||
const total = event.lengthComputable ? event.total : undefined;
|
||||
onProgress({
|
||||
lastProgress = {
|
||||
loaded: event.loaded,
|
||||
total,
|
||||
percentage: total && total > 0 ? Math.round(event.loaded / total * 100) : null
|
||||
});
|
||||
};
|
||||
onProgress(lastProgress);
|
||||
};
|
||||
|
||||
xhr.onerror = () => reject(new Error("i18n:govoplan-files.upload_failed_because_the_network_request_could_.360a5ab3"));
|
||||
@@ -722,13 +948,18 @@ endpoint = "/api/v1/files/upload")
|
||||
reject(new ApiError(xhr.status, xhr.statusText, responseText));
|
||||
return;
|
||||
}
|
||||
onProgress({ loaded: 1, total: 1, percentage: 100 });
|
||||
onProgress({ ...lastProgress, percentage: 100 });
|
||||
if (xhr.status === 204 || !responseText) {
|
||||
resolve({ files: [] });
|
||||
resolve({ files: [] } as T);
|
||||
return;
|
||||
}
|
||||
const contentType = xhr.getResponseHeader("content-type") || "";
|
||||
resolve(contentType.includes("application/json") ? JSON.parse(responseText) as FileUploadResponse : { files: [] });
|
||||
try {
|
||||
if (!contentType.includes("application/json")) throw new Error("Expected JSON response");
|
||||
resolve(JSON.parse(responseText) as T);
|
||||
} catch {
|
||||
reject(new ApiError(xhr.status, "Invalid JSON response", ""));
|
||||
}
|
||||
};
|
||||
|
||||
onProgress({ loaded: 0, total: undefined, percentage: 0 });
|
||||
@@ -744,6 +975,55 @@ export function bulkDeleteFiles(settings: ApiSettings, fileIds: string[]): Promi
|
||||
return apiFetch<BulkDeleteResponse>(settings, "/api/v1/files/bulk-delete", { method: "POST", body: JSON.stringify({ file_ids: fileIds }) });
|
||||
}
|
||||
|
||||
export function updateFileLifecycle(
|
||||
settings: ApiSettings,
|
||||
fileId: string,
|
||||
payload: FileLifecycleUpdatePayload)
|
||||
: Promise<ManagedFile> {
|
||||
return apiFetch<ManagedFile>(settings, `/api/v1/files/${encodeURIComponent(fileId)}/lifecycle`, {
|
||||
method: "PATCH",
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
}
|
||||
|
||||
export function restoreFile(settings: ApiSettings, fileId: string): Promise<FileRestoreResponse> {
|
||||
return apiFetch<FileRestoreResponse>(settings, `/api/v1/files/assets/${encodeURIComponent(fileId)}/restore`, { method: "POST" });
|
||||
}
|
||||
|
||||
export function restoreFolder(
|
||||
settings: ApiSettings,
|
||||
payload: {owner_type: "user" | "group";owner_id: string;path: string;recursive?: boolean;})
|
||||
: Promise<FolderRestoreResponse> {
|
||||
return apiFetch<FolderRestoreResponse>(settings, "/api/v1/files/folders/restore", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ recursive: true, ...payload })
|
||||
});
|
||||
}
|
||||
|
||||
export function previewFilePurge(settings: ApiSettings, fileIds: string[]): Promise<FilePurgePreviewResponse> {
|
||||
return apiFetch<FilePurgePreviewResponse>(settings, "/api/v1/files/purge/preview", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ file_ids: fileIds })
|
||||
});
|
||||
}
|
||||
|
||||
export function executeFilePurge(settings: ApiSettings, payload: FilePurgeExecutePayload): Promise<FilePurgeResponse> {
|
||||
return apiFetch<FilePurgeResponse>(settings, "/api/v1/files/purge/execute", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
}
|
||||
|
||||
export function garbageCollectFileBlobs(
|
||||
settings: ApiSettings,
|
||||
payload: {limit?: number;approval_reference: string;})
|
||||
: Promise<FileBlobGcResponse> {
|
||||
return apiFetch<FileBlobGcResponse>(settings, "/api/v1/files/purge/blobs", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
}
|
||||
|
||||
export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;};
|
||||
export type FileSharePayload = {
|
||||
target_type: "user" | "group" | "tenant" | "campaign";
|
||||
@@ -974,6 +1254,21 @@ export function deleteFileConnectorSpace(settings: ApiSettings, spaceId: string)
|
||||
return apiFetch<FileConnectorSpace>(settings, `/api/v1/files/connector-spaces/${encodeURIComponent(spaceId)}`, { method: "DELETE" });
|
||||
}
|
||||
|
||||
export function restoreFileConnectorSpace(settings: ApiSettings, spaceId: string): Promise<FileConnectorSpace> {
|
||||
return apiFetch<FileConnectorSpace>(settings, `/api/v1/files/connector-spaces/${encodeURIComponent(spaceId)}/restore`, { method: "POST" });
|
||||
}
|
||||
|
||||
export function writeBackFileConnectorSpace(
|
||||
settings: ApiSettings,
|
||||
spaceId: string,
|
||||
payload: FileConnectorWritePayload)
|
||||
: Promise<FileConnectorWriteResponse> {
|
||||
return apiFetch<FileConnectorWriteResponse>(settings, `/api/v1/files/connector-spaces/${encodeURIComponent(spaceId)}/write-back`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
}
|
||||
|
||||
export function getFileConnectorProfile(
|
||||
settings: ApiSettings,
|
||||
profileId: string,
|
||||
@@ -1011,7 +1306,7 @@ export function deactivateFileConnectorProfile(settings: ApiSettings, profileId:
|
||||
export function browseFileConnectorProfile(
|
||||
settings: ApiSettings,
|
||||
profileId: string,
|
||||
params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {})
|
||||
params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {}, options?: FileReadOptions)
|
||||
: Promise<FileConnectorBrowseResponse> {
|
||||
const search = new URLSearchParams();
|
||||
if (params.path) search.set("path", params.path);
|
||||
@@ -1019,7 +1314,7 @@ params: {path?: string;library_id?: string;continuation_token?: string;campaign_
|
||||
if (params.continuation_token) search.set("continuation_token", params.continuation_token);
|
||||
if (params.campaign_id) search.set("campaign_id", params.campaign_id);
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`);
|
||||
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`, options);
|
||||
}
|
||||
|
||||
export function importFileConnectorFile(
|
||||
@@ -1044,6 +1339,17 @@ payload: FileConnectorFilePayload)
|
||||
});
|
||||
}
|
||||
|
||||
export function syncFileConnectorSpaceFolder(
|
||||
settings: ApiSettings,
|
||||
spaceId: string,
|
||||
payload: FileConnectorFolderSyncPayload)
|
||||
: Promise<FileConnectorFolderSyncResponse> {
|
||||
return apiFetch<FileConnectorFolderSyncResponse>(settings, `/api/v1/files/connector-spaces/${encodeURIComponent(spaceId)}/sync`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ path: "", target_folder: "", recursive: true, conflict_strategy: "skip", max_files: 100, max_depth: 12, metadata: {}, ...payload })
|
||||
});
|
||||
}
|
||||
|
||||
export function shareFilesWithCampaign(settings: ApiSettings, fileIds: string[], campaignId: string): Promise<FileBulkShareResponse> {
|
||||
return shareFilesWithTarget(settings, fileIds, { type: "campaign", id: campaignId, label: "campaign" });
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { Edit3, Plus, RefreshCw, Save, ShieldCheck, Trash2, UserRoundKey } from "lucide-react";
|
||||
import {
|
||||
import { FormGrid,
|
||||
Button,
|
||||
Card,
|
||||
ConnectionTree,
|
||||
@@ -76,6 +76,7 @@ type ConnectorProfileDraft = {
|
||||
canBrowse: boolean;
|
||||
canImport: boolean;
|
||||
canSync: boolean;
|
||||
canWriteRemote: boolean;
|
||||
policyMode: PolicyMode;
|
||||
allowedPaths: string;
|
||||
deniedPaths: string;
|
||||
@@ -509,7 +510,8 @@ export default function FileConnectorSettingsPanel({
|
||||
const capabilities = [
|
||||
draft.canBrowse ? "browse" : "",
|
||||
draft.canImport ? "import" : "",
|
||||
draft.canSync ? "sync" : ""].
|
||||
draft.canSync ? "sync" : "",
|
||||
draft.canWriteRemote ? "write" : ""].
|
||||
filter(Boolean);
|
||||
const sharedPayload = {
|
||||
label,
|
||||
@@ -929,14 +931,14 @@ export default function FileConnectorSettingsPanel({
|
||||
<>
|
||||
<Card
|
||||
title={panelTitle}
|
||||
titleHelp={<DocumentationHelpLink reference={CONNECTOR_DOCUMENTATION} />}
|
||||
actions={
|
||||
canWrite ?
|
||||
<div className="button-row compact-actions">
|
||||
<DocumentationHelpLink reference={CONNECTOR_DOCUMENTATION} />
|
||||
<Button onClick={() => void loadProfiles()} disabled={loading || saving} disabledReason={loading ? "File connections are already loading." : saving ? "Wait for the current connector change to finish." : undefined}>{loading ? "i18n:govoplan-files.loading.b04ba49f" : "i18n:govoplan-files.reload.cce71553"}</Button>
|
||||
<Button variant="primary" onClick={startCreate} disabled={saving} disabledReason={saving ? "Wait for the current connector change to finish." : undefined}><Plus size={16} aria-hidden="true" /> i18n:govoplan-files.new_connection.ac979fe4</Button>
|
||||
</div> :
|
||||
<DocumentationHelpLink reference={CONNECTOR_DOCUMENTATION} />
|
||||
undefined
|
||||
}>
|
||||
|
||||
<LoadingFrame loading={loading} label="i18n:govoplan-files.loading_file_connections.bd68f224">
|
||||
@@ -952,18 +954,18 @@ export default function FileConnectorSettingsPanel({
|
||||
|
||||
<Card
|
||||
title={i18nMessage("i18n:govoplan-files.value_connector_policy.0bf7f53b", { value0: scopeLabel(scopeType) })}
|
||||
titleHelp={<DocumentationHelpLink reference={CONNECTOR_DOCUMENTATION} />}
|
||||
actions={canWrite ?
|
||||
<div className="button-row compact-actions">
|
||||
<DocumentationHelpLink reference={CONNECTOR_DOCUMENTATION} />
|
||||
<Button variant="primary" onClick={() => void savePolicyDraft()} disabled={saving || loading} disabledReason={saving ? "Wait for the current connector change to finish." : loading ? "Wait until the effective connector policy has loaded." : undefined}>
|
||||
<ShieldCheck size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_policy.77d67ce3"}
|
||||
</Button>
|
||||
</div> :
|
||||
<DocumentationHelpLink reference={CONNECTOR_DOCUMENTATION} />}>
|
||||
undefined}>
|
||||
|
||||
<LoadingFrame loading={loading} label="i18n:govoplan-files.loading_connector_policy.f12ab285">
|
||||
<>
|
||||
<div className="form-grid two">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8">
|
||||
<ReferenceMultiSelect
|
||||
values={lines(policyDraft.allowedConnectors)}
|
||||
@@ -1042,7 +1044,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<FormField label="i18n:govoplan-files.denied_endpoint_urls.1d8d4369">
|
||||
<textarea value={policyDraft.deniedUrls} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ deniedUrls: event.target.value })} rows={3} placeholder={"http://*\n*://legacy.example.test/*"} />
|
||||
</FormField>
|
||||
</div>
|
||||
</FormGrid>
|
||||
{policyConflicts.length ? (
|
||||
<DismissibleAlert
|
||||
tone="warning"
|
||||
@@ -1059,7 +1061,7 @@ export default function FileConnectorSettingsPanel({
|
||||
)}
|
||||
<div className="file-connector-settings-section file-connector-policy-locks">
|
||||
<ToggleSwitch label="i18n:govoplan-files.lower_connection_limits.4f9838bc" checked={policyDraft.allowLowerConnectionLimits} disabled={saving || !canWrite} onChange={(allowLowerConnectionLimits) => patchPolicyDraft({ allowLowerConnectionLimits })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.lower_credential_limits.ec2b72bc" checked={policyDraft.allowLowerCredentialLimits} disabled={saving || !canWrite} onChange={(allowLowerCredentialLimits) => patchPolicyDraft({ allowLowerCredentialLimits })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.lower_credential_limits.ec2b72bc" helpContextId="files.connector.policy" helpModuleId="files" checked={policyDraft.allowLowerCredentialLimits} disabled={saving || !canWrite} onChange={(allowLowerCredentialLimits) => patchPolicyDraft({ allowLowerCredentialLimits })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.lower_provider_limits.5816270a" checked={policyDraft.allowLowerProviderLimits} disabled={saving || !canWrite} onChange={(allowLowerProviderLimits) => patchPolicyDraft({ allowLowerProviderLimits })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.lower_path_limits.1abcccb1" checked={policyDraft.allowLowerPathLimits} disabled={saving || !canWrite} onChange={(allowLowerPathLimits) => patchPolicyDraft({ allowLowerPathLimits })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.lower_endpoint_limits.3fd781d5" checked={policyDraft.allowLowerUrlLimits} disabled={saving || !canWrite} onChange={(allowLowerUrlLimits) => patchPolicyDraft({ allowLowerUrlLimits })} />
|
||||
@@ -1075,11 +1077,11 @@ export default function FileConnectorSettingsPanel({
|
||||
</LoadingFrame>
|
||||
</Card>
|
||||
|
||||
<Dialog
|
||||
<Dialog variant="administration" size="wide"
|
||||
open={Boolean(credentialDraft)}
|
||||
title={editingCredentialId ? "i18n:govoplan-files.edit_file_credential.bc49d44f" : "i18n:govoplan-files.new_file_credential.4c061082"}
|
||||
onClose={closeCredentialDialog}
|
||||
className="admin-dialog admin-dialog-wide file-connector-dialog adaptive-config-dialog"
|
||||
className="file-connector-dialog"
|
||||
bodyClassName="adaptive-config-dialog-body"
|
||||
closeDisabled={saving}
|
||||
footer={credentialDraft ?
|
||||
@@ -1098,15 +1100,15 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Credential</h3>
|
||||
<p>Choose where this credential can be used and how it signs in.</p>
|
||||
</header>
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.credential_id.9432a6e1">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.credential_id.9432a6e1" helpContextId="files.connector.credentials" helpModuleId="files">
|
||||
<input className={!editingCredentialId && !credentialDraft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingCredentialId && !credentialDraft.id.trim() || undefined} value={credentialDraft.id} disabled={Boolean(editingCredentialId) || saving} onChange={(event) => patchCredentialDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav-credentials`} />
|
||||
</FormField>
|
||||
<FormField label="i18n:govoplan-files.label.74341e3c">
|
||||
<input className={!credentialDraft.label.trim() ? "field-input-missing" : undefined} aria-invalid={!credentialDraft.label.trim() || undefined} value={credentialDraft.label} disabled={saving} onChange={(event) => patchCredentialDraft({ label: event.target.value })} placeholder="i18n:govoplan-files.govoplan_webdav_credentials.bc696d69" />
|
||||
</FormField>
|
||||
{credentialAttachProfileId &&
|
||||
<FormField label="i18n:govoplan-files.connection_credential.178babe0">
|
||||
<FormField label="i18n:govoplan-files.connection_credential.178babe0" helpContextId="files.connector.credentials" helpModuleId="files">
|
||||
<input value={credentialAttachedProfile ? `${credentialAttachedProfile.label} (${credentialAttachedProfile.id})` : credentialAttachProfileId} disabled readOnly />
|
||||
</FormField>
|
||||
}
|
||||
@@ -1128,8 +1130,8 @@ export default function FileConnectorSettingsPanel({
|
||||
/>
|
||||
</FormField>
|
||||
</div>
|
||||
<FormField label="i18n:govoplan-files.credential_mode.23fdd899" documentation={CONNECTOR_DOCUMENTATION}>
|
||||
<select value={credentialDraft.credentialMode} disabled={saving} onChange={(event) => patchCredentialDraft({ credentialMode: event.target.value as CredentialMode })}>
|
||||
<FormField label="i18n:govoplan-files.credential_mode.23fdd899" documentation={CONNECTOR_DOCUMENTATION} helpContextId="files.connector.credentials" helpModuleId="files">
|
||||
<select data-help-context-id="files.connector.credentials" data-help-module-id="files" value={credentialDraft.credentialMode} disabled={saving} onChange={(event) => patchCredentialDraft({ credentialMode: event.target.value as CredentialMode })}>
|
||||
<option value="none">i18n:govoplan-files.none.6eef6648</option>
|
||||
<option value="anonymous">i18n:govoplan-files.anonymous.9bed5104</option>
|
||||
<option value="basic">i18n:govoplan-files.username_password.e8ba8896</option>
|
||||
@@ -1137,11 +1139,11 @@ export default function FileConnectorSettingsPanel({
|
||||
<option value="secret_ref">i18n:govoplan-files.secret_reference.04ed2221</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</div>
|
||||
</FormGrid>
|
||||
<div className="file-connector-settings-section">
|
||||
<ToggleSwitch label="i18n:govoplan-files.enabled.df174a3f" checked={credentialDraft.enabled} disabled={saving} onChange={(enabled) => patchCredentialDraft({ enabled })} />
|
||||
{editingCredentialId && <ToggleSwitch label="i18n:govoplan-files.clear_saved_password.7442260d" checked={credentialDraft.clearPassword} disabled={saving} onChange={(clearPassword) => patchCredentialDraft({ clearPassword })} />}
|
||||
{editingCredentialId && <ToggleSwitch label="i18n:govoplan-files.clear_saved_token.a9c670aa" checked={credentialDraft.clearToken} disabled={saving} onChange={(clearToken) => patchCredentialDraft({ clearToken })} />}
|
||||
{editingCredentialId && <ToggleSwitch label="i18n:govoplan-files.clear_saved_password.7442260d" helpContextId="files.connector.credentials" helpModuleId="files" checked={credentialDraft.clearPassword} disabled={saving} onChange={(clearPassword) => patchCredentialDraft({ clearPassword })} />}
|
||||
{editingCredentialId && <ToggleSwitch label="i18n:govoplan-files.clear_saved_token.a9c670aa" helpContextId="files.connector.credentials" helpModuleId="files" checked={credentialDraft.clearToken} disabled={saving} onChange={(clearToken) => patchCredentialDraft({ clearToken })} />}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -1176,11 +1178,11 @@ export default function FileConnectorSettingsPanel({
|
||||
onChange={patchCredentialValues}
|
||||
disabled={saving}
|
||||
showPassword={false} />
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.secret_reference.04ed2221">
|
||||
<input value={credentialDraft.secretRef} disabled={saving} onChange={(event) => patchCredentialDraft({ secretRef: event.target.value })} />
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.secret_reference.04ed2221" helpContextId="files.connector.credentials" helpModuleId="files">
|
||||
<input data-help-context-id="files.connector.credentials" data-help-module-id="files" value={credentialDraft.secretRef} disabled={saving} onChange={(event) => patchCredentialDraft({ secretRef: event.target.value })} />
|
||||
</FormField>
|
||||
</div>
|
||||
</FormGrid>
|
||||
</>
|
||||
}
|
||||
{credentialDraft.credentialMode === "token" &&
|
||||
@@ -1212,7 +1214,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Policy</h3>
|
||||
<p>Limit where lower levels may use this credential.</p>
|
||||
</header>
|
||||
<div className="form-grid two">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.policy.bb9cf141" documentation={CONNECTOR_DOCUMENTATION}>
|
||||
<select value={credentialDraft.policyMode} disabled={saving} onChange={(event) => patchCredentialDraft({ policyMode: event.target.value as PolicyMode })}>
|
||||
<option value="allow-provider">i18n:govoplan-files.can_use_this_credential.f4a41971</option>
|
||||
@@ -1228,7 +1230,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<FormField label="i18n:govoplan-files.denied_paths.d25e4315">
|
||||
<textarea value={credentialDraft.deniedPaths} disabled={saving} onChange={(event) => patchCredentialDraft({ deniedPaths: event.target.value })} rows={3} placeholder={"private\narchive/closed"} />
|
||||
</FormField>
|
||||
</div>
|
||||
</FormGrid>
|
||||
<AdvancedOptionsPanel title="Credential metadata JSON" summary="For diagnostics or provider-specific compatibility flags. Normal setup should not need this.">
|
||||
<FormField label="i18n:govoplan-files.metadata_json.b0e4c283" documentation={CONNECTOR_RECOVERY_DOCUMENTATION}>
|
||||
<textarea value={credentialDraft.metadataJson} disabled={saving} onChange={(event) => patchCredentialDraft({ metadataJson: event.target.value })} rows={5} spellCheck={false} />
|
||||
@@ -1239,11 +1241,11 @@ export default function FileConnectorSettingsPanel({
|
||||
}
|
||||
</Dialog>
|
||||
|
||||
<Dialog
|
||||
<Dialog variant="administration" size="wide"
|
||||
open={Boolean(draft)}
|
||||
title={editingProfileId ? "i18n:govoplan-files.edit_file_connection.78698154" : "i18n:govoplan-files.new_file_connection.2ec6eb56"}
|
||||
onClose={closeProfileDialog}
|
||||
className="admin-dialog admin-dialog-wide file-connector-dialog adaptive-config-dialog"
|
||||
className="file-connector-dialog"
|
||||
bodyClassName="adaptive-config-dialog-body"
|
||||
closeDisabled={saving}
|
||||
footer={draft ?
|
||||
@@ -1262,7 +1264,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Connection</h3>
|
||||
<p>Name the file server connection and choose the provider.</p>
|
||||
</header>
|
||||
<div className="form-grid two">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.profile_id.25961d68">
|
||||
<input className={!editingProfileId && !draft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingProfileId && !draft.id.trim() || undefined} value={draft.id} disabled={Boolean(editingProfileId) || saving} onChange={(event) => patchDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav`} />
|
||||
</FormField>
|
||||
@@ -1287,7 +1289,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<div className="field-block file-connector-settings-toggle-field">
|
||||
<ToggleSwitch label="i18n:govoplan-files.enabled.df174a3f" checked={draft.enabled} disabled={saving} onChange={(enabled) => patchDraft({ enabled })} />
|
||||
</div>
|
||||
</div>
|
||||
</FormGrid>
|
||||
</section>
|
||||
|
||||
<section className="adaptive-config-section">
|
||||
@@ -1295,15 +1297,15 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Location and credentials</h3>
|
||||
<p>Only fields relevant for the selected provider and credential mode are shown.</p>
|
||||
</header>
|
||||
<div className="form-grid two">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.endpoint_url.65aaaa45" documentation={CONNECTOR_DOCUMENTATION}>
|
||||
<input value={draft.endpointUrl} disabled={saving} onChange={(event) => patchDraft({ endpointUrl: event.target.value })} placeholder={ENDPOINT_PLACEHOLDERS[draft.provider]} />
|
||||
</FormField>
|
||||
<FormField label="i18n:govoplan-files.base_path.6a4867ca">
|
||||
<input value={draft.basePath} disabled={saving} onChange={(event) => patchDraft({ basePath: event.target.value })} placeholder="GovOPlaN" />
|
||||
</FormField>
|
||||
<FormField label="i18n:govoplan-files.credential_mode.23fdd899" documentation={CONNECTOR_DOCUMENTATION}>
|
||||
<select value={draft.credentialMode} disabled={saving} onChange={(event) => patchDraft({ credentialMode: event.target.value as CredentialMode })}>
|
||||
<FormField label="i18n:govoplan-files.credential_mode.23fdd899" documentation={CONNECTOR_DOCUMENTATION} helpContextId="files.connector.credentials" helpModuleId="files">
|
||||
<select data-help-context-id="files.connector.credentials" data-help-module-id="files" value={draft.credentialMode} disabled={saving} onChange={(event) => patchDraft({ credentialMode: event.target.value as CredentialMode })}>
|
||||
<option value="none">i18n:govoplan-files.none.6eef6648</option>
|
||||
<option value="anonymous">i18n:govoplan-files.anonymous.9bed5104</option>
|
||||
<option value="basic">i18n:govoplan-files.username_password.e8ba8896</option>
|
||||
@@ -1311,13 +1313,13 @@ export default function FileConnectorSettingsPanel({
|
||||
<option value="secret_ref">i18n:govoplan-files.secret_reference.04ed2221</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="i18n:govoplan-files.credential.8bede3ea">
|
||||
<select value={profileCredentialSelectValue} disabled={saving || profileCredentialOptions.length === 0} onChange={(event) => patchDraft({ credentialProfileId: event.target.value })}>
|
||||
<FormField label="i18n:govoplan-files.credential.8bede3ea" helpContextId="files.connector.credentials" helpModuleId="files">
|
||||
<select data-help-context-id="files.connector.credentials" data-help-module-id="files" value={profileCredentialSelectValue} disabled={saving || profileCredentialOptions.length === 0} onChange={(event) => patchDraft({ credentialProfileId: event.target.value })}>
|
||||
{profileCredentialOptions.length === 0 && <option value="">i18n:govoplan-files.no_saved_credential.30a5a951</option>}
|
||||
{profileCredentialOptions.map((credential) => <option key={credential.id} value={credential.id}>{credential.label}</option>)}
|
||||
</select>
|
||||
</FormField>
|
||||
</div>
|
||||
</FormGrid>
|
||||
<div className="button-row compact-actions">
|
||||
<Button type="button" onClick={() => patchDraft({ endpointUrl: ENDPOINT_PLACEHOLDERS[draft.provider] })} disabled={saving} disabledReason={saving ? "Wait for the current connector change to finish." : undefined}>Use provider example</Button>
|
||||
{canDiscoverProfileEndpoint &&
|
||||
@@ -1332,7 +1334,7 @@ export default function FileConnectorSettingsPanel({
|
||||
</DismissibleAlert>
|
||||
}
|
||||
<AdvancedOptionsPanel title="Protocol and compatibility options" summary="Change these only when the provider or network requires an override.">
|
||||
<div className="form-grid two">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.browse_protocol.d1f27c90">
|
||||
<select value={draft.browseProtocol} disabled={saving} onChange={(event) => patchDraft({ browseProtocol: event.target.value })}>
|
||||
<option value="">i18n:govoplan-files.native_default.ba32f7b6</option>
|
||||
@@ -1352,7 +1354,7 @@ export default function FileConnectorSettingsPanel({
|
||||
</div>
|
||||
</>
|
||||
}
|
||||
</div>
|
||||
</FormGrid>
|
||||
</AdvancedOptionsPanel>
|
||||
</section>
|
||||
|
||||
@@ -1365,8 +1367,9 @@ export default function FileConnectorSettingsPanel({
|
||||
<ToggleSwitch label="i18n:govoplan-files.browse.2f3b5c55" checked={draft.canBrowse} disabled={saving} onChange={(canBrowse) => patchDraft({ canBrowse })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.import.d6fbc9d2" checked={draft.canImport} disabled={saving} onChange={(canImport) => patchDraft({ canImport })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.sync.905f6309" checked={draft.canSync} disabled={saving} onChange={(canSync) => patchDraft({ canSync })} />
|
||||
<ToggleSwitch label="Remote write" checked={draft.canWriteRemote} disabled={saving || draft.provider !== "s3"} onChange={(canWriteRemote) => patchDraft({ canWriteRemote })} />
|
||||
</div>
|
||||
<div className="form-grid two">
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="i18n:govoplan-files.policy.bb9cf141" documentation={CONNECTOR_DOCUMENTATION}>
|
||||
<select value={draft.policyMode} disabled={saving} onChange={(event) => patchDraft({ policyMode: event.target.value as PolicyMode })}>
|
||||
<option value="allow-provider">i18n:govoplan-files.can_use_this_connection.5091a74c</option>
|
||||
@@ -1382,7 +1385,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<FormField label="i18n:govoplan-files.denied_paths.d25e4315">
|
||||
<textarea value={draft.deniedPaths} disabled={saving} onChange={(event) => patchDraft({ deniedPaths: event.target.value })} rows={3} placeholder={"private\narchive/closed"} />
|
||||
</FormField>
|
||||
</div>
|
||||
</FormGrid>
|
||||
<AdvancedOptionsPanel title="Connection metadata JSON" summary="For diagnostics or provider-specific compatibility flags. Normal setup should not need this.">
|
||||
<FormField label="i18n:govoplan-files.metadata_json.b0e4c283" documentation={CONNECTOR_RECOVERY_DOCUMENTATION}>
|
||||
<textarea value={draft.metadataJson} disabled={saving} onChange={(event) => patchDraft({ metadataJson: event.target.value })} rows={5} spellCheck={false} />
|
||||
@@ -1528,6 +1531,7 @@ function emptyDraft(scopeType: ConnectorScope): ConnectorProfileDraft {
|
||||
canBrowse: true,
|
||||
canImport: true,
|
||||
canSync: true,
|
||||
canWriteRemote: false,
|
||||
policyMode: "allow-provider",
|
||||
allowedPaths: "",
|
||||
deniedPaths: "",
|
||||
@@ -1566,6 +1570,7 @@ function draftFromProfile(profile: FileConnectorProfile, scopeType: ConnectorSco
|
||||
canBrowse: profile.capabilities.includes("browse"),
|
||||
canImport: profile.capabilities.includes("import"),
|
||||
canSync: profile.capabilities.includes("sync"),
|
||||
canWriteRemote: profile.capabilities.includes("write"),
|
||||
policyMode: denyProviders.includes("*") || denyProviders.includes(profile.provider) ? "deny-provider" : allowPaths.length > 0 ? "allowed-paths" : "allow-provider",
|
||||
allowedPaths: allowPaths.join("\n"),
|
||||
deniedPaths: denyPaths.join("\n"),
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { MetricGrid } from "@govoplan/core-webui";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import {
|
||||
AdminPageLayout,
|
||||
@@ -383,6 +384,7 @@ export default function FileIntegrityPanel({ settings, canWrite }: Props) {
|
||||
<>
|
||||
<AdminPageLayout
|
||||
title="File integrity"
|
||||
titleHelp={<DocumentationHelpLink reference={DOCUMENTATION} label="Open Files integrity documentation" />}
|
||||
description="Run bounded storage reconciliation and resolve quarantined or unreferenced objects without acting on stale operator state."
|
||||
loading={loading}
|
||||
error={error}
|
||||
@@ -400,7 +402,6 @@ export default function FileIntegrityPanel({ settings, canWrite }: Props) {
|
||||
<Button variant="primary" onClick={() => setCreateOpen(true)} disabled={!canWrite || busy}>
|
||||
<Plus size={16} /> New scan
|
||||
</Button>
|
||||
<DocumentationHelpLink reference={DOCUMENTATION} label="Open Files integrity documentation" />
|
||||
</>
|
||||
)}
|
||||
>
|
||||
@@ -424,12 +425,12 @@ export default function FileIntegrityPanel({ settings, canWrite }: Props) {
|
||||
|
||||
{selectedScan && (
|
||||
<>
|
||||
<div className="metric-grid">
|
||||
<MetricGrid>
|
||||
<MetricCard label="Verified blobs" value={selectedScan.verified_blob_count} tone="good" />
|
||||
<MetricCard label="Quarantined blobs" value={selectedScan.quarantined_blob_count} tone={selectedScan.quarantined_blob_count ? "danger" : "neutral"} />
|
||||
<MetricCard label="Orphan objects" value={selectedScan.orphan_object_count} tone={selectedScan.orphan_object_count ? "warning" : "neutral"} />
|
||||
<MetricCard label="Open findings" value={findings.filter((finding) => finding.state === "open").length} tone={findings.some((finding) => finding.state === "open") ? "warning" : "good"} />
|
||||
</div>
|
||||
</MetricGrid>
|
||||
{selectedScan.last_error && (
|
||||
<DismissibleAlert tone="warning" dismissible={false} compact>
|
||||
The last batch failed with {selectedScan.last_error}. Verify storage availability, then resume from the recorded cursor.
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import { FileText, UploadCloud, X } from "lucide-react";
|
||||
import { useCallback } from "react";
|
||||
import { Link } from "react-router";
|
||||
import {
|
||||
Button,
|
||||
DismissibleAlert,
|
||||
LoadingFrame,
|
||||
SelectionList,
|
||||
SelectionListItem,
|
||||
SelectionListItemContent,
|
||||
hasScope,
|
||||
quickAccessLaunchState,
|
||||
useDashboardWidgetData,
|
||||
type QuickAccessToolRenderContext
|
||||
} from "@govoplan/core-webui";
|
||||
import { listFiles } from "../../api/files";
|
||||
|
||||
type Props = Pick<
|
||||
QuickAccessToolRenderContext,
|
||||
"settings" | "auth" | "launchContext" | "complete" | "cancel" | "close"
|
||||
>;
|
||||
|
||||
/**
|
||||
* Bounded file selection. listFiles remains the owner-side authorization
|
||||
* check; the host receives only a versioned reference after explicit choice.
|
||||
*/
|
||||
export default function FileQuickAccess({
|
||||
settings,
|
||||
auth,
|
||||
launchContext,
|
||||
complete,
|
||||
cancel,
|
||||
close
|
||||
}: Props) {
|
||||
const load = useCallback(
|
||||
async () => (await listFiles(settings, { sort: "recent", page_size: 7 })).files,
|
||||
[settings]
|
||||
);
|
||||
const { data: files, loading, error } = useDashboardWidgetData(load, 0);
|
||||
const canUpload = hasScope(auth, "files:upload");
|
||||
|
||||
return (
|
||||
<LoadingFrame loading={loading} label="Loading recent files">
|
||||
{launchContext.activeObject ? (
|
||||
<p className="muted small-note">
|
||||
Select a file for {launchContext.activeObject.label}. Files checks
|
||||
your current access again before showing this list.
|
||||
</p>
|
||||
) : null}
|
||||
{error ? <DismissibleAlert tone="warning" resetKey={error}>{error}</DismissibleAlert> : null}
|
||||
<SelectionList variant="navigation" label="Recent authorized files">
|
||||
{(files ?? []).map((file) => (
|
||||
<SelectionListItem
|
||||
key={file.version_id}
|
||||
selected={false}
|
||||
onClick={() => complete({
|
||||
contractVersion: "1",
|
||||
outcome: "completed",
|
||||
action: "selected",
|
||||
reference: {
|
||||
ownerModule: "files",
|
||||
kind: "file-version",
|
||||
objectId: file.version_id,
|
||||
tenantId: file.tenant_id,
|
||||
label: file.filename,
|
||||
version: file.version_id,
|
||||
path: `/files?fileId=${encodeURIComponent(file.id)}&versionId=${encodeURIComponent(file.version_id)}`
|
||||
}
|
||||
})}
|
||||
>
|
||||
<SelectionListItemContent
|
||||
leading={<FileText size={16} aria-hidden="true" />}
|
||||
title={file.filename}
|
||||
description={file.display_path}
|
||||
/>
|
||||
</SelectionListItem>
|
||||
))}
|
||||
</SelectionList>
|
||||
{!loading && !error && !(files ?? []).length ? (
|
||||
<p className="muted">No authorized files are available.</p>
|
||||
) : null}
|
||||
<div className="button-row compact-actions">
|
||||
{canUpload ? (
|
||||
<Link
|
||||
className="btn btn-secondary"
|
||||
to="/files?quickAction=upload"
|
||||
state={quickAccessLaunchState(launchContext)}
|
||||
onClick={close}
|
||||
>
|
||||
<UploadCloud size={15} aria-hidden="true" /> i18n:govoplan-files.upload.8bdf057f
|
||||
</Link>
|
||||
) : null}
|
||||
<Button onClick={() => cancel("user")}><X size={15} aria-hidden="true" /> i18n:govoplan-files.cancel.77dfd213</Button>
|
||||
</div>
|
||||
</LoadingFrame>
|
||||
);
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -195,12 +195,15 @@ export function RenamePreviewList({
|
||||
|
||||
}
|
||||
|
||||
export function FileDialog({ title, onClose, children }: {title: string;onClose: () => void;children: ReactNode;}) {
|
||||
export function FileDialog({ title, titleHelp, onClose, children, busy = false }: {title: string;titleHelp?: ReactNode;onClose: () => void;children: ReactNode;busy?: boolean;}) {
|
||||
return (
|
||||
<Dialog
|
||||
open
|
||||
title={title}
|
||||
titleHelp={titleHelp}
|
||||
onClose={onClose}
|
||||
closeDisabled={busy}
|
||||
closeOnBackdrop={!busy}
|
||||
backdropClassName="file-dialog-backdrop"
|
||||
className="file-dialog"
|
||||
headerClassName="file-dialog-header"
|
||||
@@ -217,6 +220,7 @@ export function FileContextMenu({
|
||||
hasSelection,
|
||||
canCreateFolder,
|
||||
canUpload,
|
||||
canUnpackArchive,
|
||||
canDownload,
|
||||
canOrganize,
|
||||
canDelete,
|
||||
@@ -224,6 +228,7 @@ export function FileContextMenu({
|
||||
downloadLabel,
|
||||
onCreateFolder,
|
||||
onUpload,
|
||||
onUnpackArchive,
|
||||
onDownload,
|
||||
onMove,
|
||||
onCopy,
|
||||
@@ -244,13 +249,13 @@ export function FileContextMenu({
|
||||
|
||||
|
||||
|
||||
}: {menu: ContextMenuState;hasSelection: boolean;canCreateFolder: boolean;canUpload: boolean;canDownload: boolean;canOrganize: boolean;canDelete: boolean;canExplainAccess: boolean;downloadLabel: string;onCreateFolder: () => void;onUpload: () => void;onDownload: () => void;onMove: () => void;onCopy: () => void;onExplainAccess: () => void;onDelete: () => void;}) {
|
||||
}: {menu: ContextMenuState;hasSelection: boolean;canCreateFolder: boolean;canUpload: boolean;canUnpackArchive: boolean;canDownload: boolean;canOrganize: boolean;canDelete: boolean;canExplainAccess: boolean;downloadLabel: string;onCreateFolder: () => void;onUpload: () => void;onUnpackArchive: () => void;onDownload: () => void;onMove: () => void;onCopy: () => void;onExplainAccess: () => void;onDelete: () => void;}) {
|
||||
const showNewFolder = true;
|
||||
const showDelete = menu.target !== "empty";
|
||||
const viewportWidth = typeof window === "undefined" ? 1024 : window.innerWidth;
|
||||
const viewportHeight = typeof window === "undefined" ? 768 : window.innerHeight;
|
||||
const estimatedWidth = 220;
|
||||
const estimatedHeight = 260;
|
||||
const estimatedHeight = 300;
|
||||
const left = Math.max(8, Math.min(menu.x, viewportWidth - estimatedWidth - 8));
|
||||
const openUp = menu.y + estimatedHeight > viewportHeight;
|
||||
const style: CSSProperties = openUp ?
|
||||
@@ -260,11 +265,12 @@ export function FileContextMenu({
|
||||
<div className="file-context-menu" style={style} role="menu" onClick={(event) => event.stopPropagation()}>
|
||||
{showNewFolder && <button type="button" role="menuitem" onClick={onCreateFolder} disabled={!canCreateFolder}><Plus size={15} aria-hidden="true" /> i18n:govoplan-files.new_folder.a711999b</button>}
|
||||
<button type="button" role="menuitem" onClick={onUpload} disabled={!canUpload}><UploadCloud size={15} aria-hidden="true" /> i18n:govoplan-files.upload.8bdf057f</button>
|
||||
<button type="button" role="menuitem" onClick={onUnpackArchive} disabled={!canUnpackArchive}>i18n:govoplan-files.managed_archive.unpack</button>
|
||||
<button type="button" role="menuitem" onClick={onDownload} disabled={!hasSelection || !canDownload}><Download size={15} aria-hidden="true" /> {downloadLabel}</button>
|
||||
<button type="button" role="menuitem" onClick={onMove} disabled={!hasSelection || !canOrganize}><MoveRight size={15} aria-hidden="true" /> i18n:govoplan-files.move.8a74a26e</button>
|
||||
<button type="button" role="menuitem" onClick={onCopy} disabled={!hasSelection || !canOrganize}><Copy size={15} aria-hidden="true" /> i18n:govoplan-files.copy.92556c6d</button>
|
||||
<button type="button" role="menuitem" onClick={onExplainAccess} disabled={!canExplainAccess}><KeyRound size={15} aria-hidden="true" /> i18n:govoplan-files.explain_access.4d5fac37</button>
|
||||
{showDelete && <button type="button" role="menuitem" className="danger" onClick={onDelete} disabled={!canDelete}><Trash2 size={15} aria-hidden="true" /> i18n:govoplan-files.delete.f6fdbe48</button>}
|
||||
{showDelete && <button type="button" role="menuitem" className="danger" data-help-context-id="files.list" data-help-module-id="files" onClick={onDelete} disabled={!canDelete}><Trash2 size={15} aria-hidden="true" /> i18n:govoplan-files.delete.f6fdbe48</button>}
|
||||
</div>);
|
||||
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { Trash2 } from "lucide-react";
|
||||
import {
|
||||
import { DialogSection,
|
||||
Button,
|
||||
ConfirmDialog,
|
||||
DataGrid,
|
||||
@@ -193,7 +193,7 @@ export default function FileShareDialog({
|
||||
return (
|
||||
<>
|
||||
<FileDialog title={`Manage shares - ${file.filename}`} onClose={() => !busy && onClose()}>
|
||||
<div className="file-share-dialog-content">
|
||||
<DialogSection className="file-share-dialog-content">
|
||||
{error ? (
|
||||
<DismissibleAlert tone="danger" resetKey={error}>
|
||||
{error}
|
||||
@@ -268,7 +268,7 @@ export default function FileShareDialog({
|
||||
<div className="button-row compact-actions align-end">
|
||||
<Button onClick={onClose} disabled={busy}>Close</Button>
|
||||
</div>
|
||||
</div>
|
||||
</DialogSection>
|
||||
</FileDialog>
|
||||
<ConfirmDialog
|
||||
open={Boolean(revokeTarget)}
|
||||
|
||||
@@ -10,7 +10,7 @@ import {
|
||||
type FileSpace,
|
||||
type ManagedFile } from
|
||||
"../../../api/files";
|
||||
import {
|
||||
import { ActionToolbar,
|
||||
Button,
|
||||
ConfirmDialog,
|
||||
Dialog,
|
||||
@@ -407,7 +407,7 @@ export default function ManagedFileChooser({
|
||||
</aside>
|
||||
|
||||
<section className={`managed-file-chooser-browser ${mode === "folder" ? "folder-mode" : "attachment-mode"}`}>
|
||||
<div className="managed-file-chooser-toolbar">
|
||||
<ActionToolbar className="managed-file-chooser-toolbar">
|
||||
<div className="managed-file-breadcrumb" aria-label="i18n:govoplan-files.current_folder.5aeab2f0">
|
||||
<button type="button" onClick={() => openFolder(effectiveRoot)} disabled={loading}>
|
||||
<Home size={14} aria-hidden="true" /> {selectedSpace?.label || "i18n:govoplan-files.files.6ce6c512"}
|
||||
@@ -419,7 +419,7 @@ export default function ManagedFileChooser({
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</ActionToolbar>
|
||||
|
||||
{mode === "attachment" &&
|
||||
<ManagedPatternEditor
|
||||
|
||||
@@ -4,7 +4,7 @@ export type RenameMode = "prefix" | "suffix" | "replace";
|
||||
export type SortColumn = "name" | "size" | "modified";
|
||||
export type SortDirection = "asc" | "desc";
|
||||
export type TransferMode = "move" | "copy";
|
||||
export type DialogKind = "upload" | "connector-sync" | "connector-space" | "create-folder" | "rename" | "single-rename" | "transfer" | null;
|
||||
export type DialogKind = "upload" | "connector-sync" | "connector-folder-sync" | "connector-space" | "create-folder" | "rename" | "single-rename" | "transfer" | null;
|
||||
export type EntrySelectionKey = `file:${string}` | `folder:${string}`;
|
||||
export type ContextMenuState = {
|
||||
x: number;
|
||||
|
||||
@@ -2,6 +2,37 @@ import type { PlatformTranslations } from "@govoplan/core-webui";
|
||||
|
||||
export const generatedTranslations: PlatformTranslations = {
|
||||
"en": {
|
||||
"i18n:govoplan-files.tools.connections": "Connections and imports",
|
||||
"i18n:govoplan-files.tools.connections_description": "Browse linked sources or import files explicitly. Reload only refreshes the current listing; it never synchronizes or imports files.",
|
||||
"i18n:govoplan-files.tools.selection": "Manage selection",
|
||||
"i18n:govoplan-files.tools.organize": "Organize files and folders",
|
||||
"i18n:govoplan-files.tools.organize_description": "Move, copy, or rename the selected items. The next dialog lets you review the destination or preview the change.",
|
||||
"i18n:govoplan-files.tools.sharing_access": "Sharing and access",
|
||||
"i18n:govoplan-files.tools.destructive": "Removal actions",
|
||||
"i18n:govoplan-files.tools.delete_description": "Deletion requires confirmation. Retention and audit protections continue to apply.",
|
||||
"i18n:govoplan-files.tools.import_sync": "Import and synchronize",
|
||||
"i18n:govoplan-files.tools.import_sync_description": "These actions can create or update managed files. A selected remote file is synchronized explicitly; folder synchronization has its own scope and conflict review.",
|
||||
"i18n:govoplan-files.tools.spaces": "Linked file spaces",
|
||||
"i18n:govoplan-files.tools.remove_space_description": "Remove only the local link after confirmation. Remote provider files and previously imported managed files remain unchanged.",
|
||||
"i18n:govoplan-files.archive_progress.inspecting": "Inspecting the archive…",
|
||||
"i18n:govoplan-files.archive_progress.extracting": "Extracting selected archive files…",
|
||||
"i18n:govoplan-files.archive_progress.storing": "Storing extracted files…",
|
||||
"i18n:govoplan-files.archive_progress.finalizing": "Finalizing and committing changes…",
|
||||
"i18n:govoplan-files.archive_progress.processed": "{completed} of {total} files · {bytes} of {totalBytes} processed. Keep this dialog open until completion.",
|
||||
"i18n:govoplan-files.archive_progress.selected": "{total} files selected · {bytes}. Waiting for server progress; keep this dialog open.",
|
||||
"i18n:govoplan-files.archive_progress.transferred": "{percentage}% transferred to the server. Inspection and processing follow separately.",
|
||||
"i18n:govoplan-files.archive_progress.waiting": "Waiting for a measured result. Keep this dialog open; no completion percentage is available yet.",
|
||||
"i18n:govoplan-files.archive_progress.preview_expires": "Preview expires {expires}. Confirmation reuses the protected temporary archive when available and rechecks its contents and your destination.",
|
||||
"i18n:govoplan-files.managed_archive.unpack": "Unpack archive",
|
||||
"i18n:govoplan-files.managed_archive.select_one": "Select exactly one managed ZIP or TAR archive first.",
|
||||
"i18n:govoplan-files.managed_archive.download_required": "File download permission is required to unpack an existing archive.",
|
||||
"i18n:govoplan-files.managed_archive.source": "Source: {value0}. The managed archive remains unchanged; no browser download or re-upload is needed.",
|
||||
"i18n:govoplan-files.managed_archive.preview": "Preview archive",
|
||||
"i18n:govoplan-files.managed_archive.protection": "Extracted files use normal upload storage. Archive passwords and a source storage encryption envelope are not automatically applied to individual files.",
|
||||
"i18n:govoplan-files.managed_archive.inspecting": "Inspecting the managed archive…",
|
||||
"i18n:govoplan-files.managed_archive.extracting": "Extracting the selected archive files. Keep this dialog open until the operation finishes.",
|
||||
"i18n:govoplan-files.managed_archive.expires": "Preview expires {value0}. Confirmation rechecks the source version and access; existing destination files and the source archive are never overwritten.",
|
||||
"i18n:govoplan-files.managed_archive.change_destination": "Change destination",
|
||||
"i18n:govoplan-files.add_connector_space.aa6bdbd6": "Add connector space",
|
||||
"i18n:govoplan-files.add_credential_for_value.0fa9c1fe": "Add credential for {value0}",
|
||||
"i18n:govoplan-files.add_prefix.672452bc": "Add prefix",
|
||||
@@ -157,6 +188,7 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.file.2c3cafa4": "File",
|
||||
"i18n:govoplan-files.files_with_the_same_visible_name_already_exist_i.1bb487b0": "Files with the same visible name already exist in this folder.",
|
||||
"i18n:govoplan-files.files.6ce6c512": "Files",
|
||||
"i18n:govoplan-files.quick_access_description": "Recent and contextual files available to you.",
|
||||
"i18n:govoplan-files.finalizing_upload.bcce936d": "Finalizing upload…",
|
||||
"i18n:govoplan-files.find_and_replace.2c6b404b": "Find and replace",
|
||||
"i18n:govoplan-files.find.df251b06": "Find",
|
||||
@@ -164,6 +196,49 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.folder_name.b2ce023b": "Folder name",
|
||||
"i18n:govoplan-files.folder_s.10e54730": "folder(s)",
|
||||
"i18n:govoplan-files.folder.30baa249": "Folder",
|
||||
"i18n:govoplan-files.folder_sync.action.conflict": "Conflict",
|
||||
"i18n:govoplan-files.folder_sync.action.created": "Created",
|
||||
"i18n:govoplan-files.folder_sync.action.failed": "Failed",
|
||||
"i18n:govoplan-files.folder_sync.action.policy_denied": "Policy denied",
|
||||
"i18n:govoplan-files.folder_sync.action.skipped": "Skipped",
|
||||
"i18n:govoplan-files.folder_sync.action.unchanged": "Unchanged",
|
||||
"i18n:govoplan-files.folder_sync.action.updated": "Updated",
|
||||
"i18n:govoplan-files.folder_sync.blocker.loading": "Wait until the connector folder has finished loading.",
|
||||
"i18n:govoplan-files.folder_sync.blocker.permission": "File upload permission is required to synchronize a connector folder.",
|
||||
"i18n:govoplan-files.folder_sync.blocker.select_space": "Select a linked connector space first.",
|
||||
"i18n:govoplan-files.folder_sync.button": "Sync folder",
|
||||
"i18n:govoplan-files.folder_sync.conflict.help": "Skip is privacy-safe and leaves unrelated managed files untouched. Rename creates a distinct managed path; reject reports conflicts; overwrite retires a colliding unrelated target.",
|
||||
"i18n:govoplan-files.folder_sync.conflict.label": "Existing target",
|
||||
"i18n:govoplan-files.folder_sync.conflict.overwrite": "Replace unrelated target",
|
||||
"i18n:govoplan-files.folder_sync.conflict.reject": "Report as conflict",
|
||||
"i18n:govoplan-files.folder_sync.conflict.rename": "Keep both with a new name",
|
||||
"i18n:govoplan-files.folder_sync.conflict.skip": "Skip unrelated conflicts",
|
||||
"i18n:govoplan-files.folder_sync.description": "Synchronize the visible remote folder into the linked owner's managed Files space. Existing connector identities become new versions; every skip, conflict, policy denial, and transport failure remains visible below.",
|
||||
"i18n:govoplan-files.folder_sync.destination.help": "Remote relative paths are preserved in the user or group space that owns this connector link.",
|
||||
"i18n:govoplan-files.folder_sync.destination.label": "Managed destination",
|
||||
"i18n:govoplan-files.folder_sync.finished": "Folder sync finished: {value0} created, {value1} updated, {value2} unchanged, {value3} skipped, {value4} need review.",
|
||||
"i18n:govoplan-files.folder_sync.limit.help": "Bounds one interactive run. If the limit is reached, the result is marked truncated and can be continued with a narrower folder.",
|
||||
"i18n:govoplan-files.folder_sync.limit.label": "Maximum files",
|
||||
"i18n:govoplan-files.folder_sync.recursive.help": "Subfolders are traversed to a bounded depth. Disable this to synchronize files in the visible folder only.",
|
||||
"i18n:govoplan-files.folder_sync.recursive.label": "Include subfolders",
|
||||
"i18n:govoplan-files.folder_sync.remote.help": "The folder is resolved below the linked connector-space root; parent-directory traversal is rejected.",
|
||||
"i18n:govoplan-files.folder_sync.remote.label": "Remote folder",
|
||||
"i18n:govoplan-files.folder_sync.results.aria_label": "Folder sync results",
|
||||
"i18n:govoplan-files.folder_sync.results.detail": "Managed target or detail",
|
||||
"i18n:govoplan-files.folder_sync.results.result": "Result",
|
||||
"i18n:govoplan-files.folder_sync.results.source": "Source",
|
||||
"i18n:govoplan-files.folder_sync.run_again": "Run again",
|
||||
"i18n:govoplan-files.folder_sync.summary.aria_label": "Folder sync summary",
|
||||
"i18n:govoplan-files.folder_sync.summary.conflicts": "conflicts",
|
||||
"i18n:govoplan-files.folder_sync.summary.created": "created",
|
||||
"i18n:govoplan-files.folder_sync.summary.discovered": "discovered",
|
||||
"i18n:govoplan-files.folder_sync.summary.failed": "failed",
|
||||
"i18n:govoplan-files.folder_sync.summary.policy_denied": "policy denied",
|
||||
"i18n:govoplan-files.folder_sync.summary.skipped": "skipped",
|
||||
"i18n:govoplan-files.folder_sync.summary.unchanged": "unchanged",
|
||||
"i18n:govoplan-files.folder_sync.summary.updated": "updated",
|
||||
"i18n:govoplan-files.folder_sync.title": "Sync connector folder",
|
||||
"i18n:govoplan-files.folder_sync.truncated": "The file limit was reached. Select a narrower remote folder or raise the bounded maximum to continue.",
|
||||
"i18n:govoplan-files.folders_and_files.d107ac99": "Folders and files",
|
||||
"i18n:govoplan-files.folders.19adc47b": "Folders",
|
||||
"i18n:govoplan-files.gb.505a44fa": "GB",
|
||||
@@ -357,6 +432,37 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.writable.dd35487a": "Writable"
|
||||
},
|
||||
"de": {
|
||||
"i18n:govoplan-files.tools.connections": "Verbindungen und Importe",
|
||||
"i18n:govoplan-files.tools.connections_description": "Verknüpfte Quellen durchsuchen oder Dateien ausdrücklich importieren. Neu laden aktualisiert nur die aktuelle Liste; es synchronisiert oder importiert keine Dateien.",
|
||||
"i18n:govoplan-files.tools.selection": "Auswahl verwalten",
|
||||
"i18n:govoplan-files.tools.organize": "Dateien und Ordner organisieren",
|
||||
"i18n:govoplan-files.tools.organize_description": "Ausgewählte Elemente verschieben, kopieren oder umbenennen. Im nächsten Dialog prüfen Sie das Ziel oder eine Vorschau der Änderung.",
|
||||
"i18n:govoplan-files.tools.sharing_access": "Freigaben und Zugriff",
|
||||
"i18n:govoplan-files.tools.destructive": "Entfernen",
|
||||
"i18n:govoplan-files.tools.delete_description": "Das Löschen erfordert eine Bestätigung. Aufbewahrungsvorgaben und Schutz für prüfrelevante Dateien gelten weiterhin.",
|
||||
"i18n:govoplan-files.tools.import_sync": "Importieren und synchronisieren",
|
||||
"i18n:govoplan-files.tools.import_sync_description": "Diese Aktionen können verwaltete Dateien anlegen oder aktualisieren. Eine ausgewählte entfernte Datei wird ausdrücklich synchronisiert; die Ordnersynchronisierung hat einen eigenen Umfang und eine Konfliktprüfung.",
|
||||
"i18n:govoplan-files.tools.spaces": "Verknüpfte Dateibereiche",
|
||||
"i18n:govoplan-files.tools.remove_space_description": "Nach Bestätigung wird nur die lokale Verknüpfung entfernt. Dateien beim entfernten Anbieter und bereits importierte verwaltete Dateien bleiben unverändert.",
|
||||
"i18n:govoplan-files.archive_progress.inspecting": "Archiv wird geprüft…",
|
||||
"i18n:govoplan-files.archive_progress.extracting": "Ausgewählte Archivdateien werden entpackt…",
|
||||
"i18n:govoplan-files.archive_progress.storing": "Entpackte Dateien werden gespeichert…",
|
||||
"i18n:govoplan-files.archive_progress.finalizing": "Änderungen werden abgeschlossen und verbindlich gespeichert…",
|
||||
"i18n:govoplan-files.archive_progress.processed": "{completed} von {total} Dateien · {bytes} von {totalBytes} verarbeitet. Diesen Dialog bis zum Abschluss geöffnet lassen.",
|
||||
"i18n:govoplan-files.archive_progress.selected": "{total} Dateien ausgewählt · {bytes}. Serverfortschritt wird erwartet; diesen Dialog geöffnet lassen.",
|
||||
"i18n:govoplan-files.archive_progress.transferred": "{percentage}% an den Server übertragen. Prüfung und Verarbeitung folgen gesondert.",
|
||||
"i18n:govoplan-files.archive_progress.waiting": "Ein gemessenes Ergebnis wird erwartet. Diesen Dialog geöffnet lassen; ein Abschlussprozentsatz ist noch nicht verfügbar.",
|
||||
"i18n:govoplan-files.archive_progress.preview_expires": "Die Vorschau läuft am {expires} ab. Die Bestätigung verwendet das geschützte temporäre Archiv erneut, sofern verfügbar, und prüft Inhalt sowie Ziel erneut.",
|
||||
"i18n:govoplan-files.managed_archive.unpack": "Archiv entpacken",
|
||||
"i18n:govoplan-files.managed_archive.select_one": "Wählen Sie zunächst genau ein verwaltetes ZIP- oder TAR-Archiv aus.",
|
||||
"i18n:govoplan-files.managed_archive.download_required": "Zum Entpacken eines vorhandenen Archivs ist die Berechtigung zum Herunterladen erforderlich.",
|
||||
"i18n:govoplan-files.managed_archive.source": "Quelle: {value0}. Das verwaltete Archiv bleibt unverändert; Herunterladen und erneutes Hochladen im Browser sind nicht erforderlich.",
|
||||
"i18n:govoplan-files.managed_archive.preview": "Archivvorschau",
|
||||
"i18n:govoplan-files.managed_archive.protection": "Entpackte Dateien verwenden die normalen Upload-Speichereinstellungen. Archivpasswörter und eine Speicher-Verschlüsselungshülle der Quelle werden nicht automatisch auf einzelne Dateien angewendet.",
|
||||
"i18n:govoplan-files.managed_archive.inspecting": "Verwaltetes Archiv wird geprüft…",
|
||||
"i18n:govoplan-files.managed_archive.extracting": "Die ausgewählten Archivdateien werden entpackt. Lassen Sie diesen Dialog bis zum Abschluss geöffnet.",
|
||||
"i18n:govoplan-files.managed_archive.expires": "Die Vorschau läuft am {value0} ab. Beim Bestätigen werden Quellversion und Zugriff erneut geprüft. Vorhandene Zieldateien und das Quellarchiv werden niemals überschrieben.",
|
||||
"i18n:govoplan-files.managed_archive.change_destination": "Ziel ändern",
|
||||
"i18n:govoplan-files.add_connector_space.aa6bdbd6": "Add connector space",
|
||||
"i18n:govoplan-files.add_credential_for_value.0fa9c1fe": "Add credential for {value0}",
|
||||
"i18n:govoplan-files.add_prefix.672452bc": "Add prefix",
|
||||
@@ -512,6 +618,7 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.file.2c3cafa4": "Datei",
|
||||
"i18n:govoplan-files.files_with_the_same_visible_name_already_exist_i.1bb487b0": "Files with the same visible name already exist in this folder.",
|
||||
"i18n:govoplan-files.files.6ce6c512": "Dateien",
|
||||
"i18n:govoplan-files.quick_access_description": "Für Sie verfügbare aktuelle und kontextbezogene Dateien.",
|
||||
"i18n:govoplan-files.finalizing_upload.bcce936d": "Finalizing upload…",
|
||||
"i18n:govoplan-files.find_and_replace.2c6b404b": "Find and replace",
|
||||
"i18n:govoplan-files.find.df251b06": "Find",
|
||||
@@ -519,6 +626,49 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.folder_name.b2ce023b": "Ordnername",
|
||||
"i18n:govoplan-files.folder_s.10e54730": "folder(s)",
|
||||
"i18n:govoplan-files.folder.30baa249": "Ordner",
|
||||
"i18n:govoplan-files.folder_sync.action.conflict": "Konflikt",
|
||||
"i18n:govoplan-files.folder_sync.action.created": "Erstellt",
|
||||
"i18n:govoplan-files.folder_sync.action.failed": "Fehlgeschlagen",
|
||||
"i18n:govoplan-files.folder_sync.action.policy_denied": "Durch Richtlinie abgelehnt",
|
||||
"i18n:govoplan-files.folder_sync.action.skipped": "Übersprungen",
|
||||
"i18n:govoplan-files.folder_sync.action.unchanged": "Unverändert",
|
||||
"i18n:govoplan-files.folder_sync.action.updated": "Aktualisiert",
|
||||
"i18n:govoplan-files.folder_sync.blocker.loading": "Warten Sie, bis der Connector-Ordner vollständig geladen ist.",
|
||||
"i18n:govoplan-files.folder_sync.blocker.permission": "Zum Synchronisieren eines Connector-Ordners ist die Berechtigung zum Hochladen von Dateien erforderlich.",
|
||||
"i18n:govoplan-files.folder_sync.blocker.select_space": "Wählen Sie zuerst einen verknüpften Connector-Bereich aus.",
|
||||
"i18n:govoplan-files.folder_sync.button": "Ordner synchronisieren",
|
||||
"i18n:govoplan-files.folder_sync.conflict.help": "Überspringen belässt nicht zugehörige verwaltete Dateien unverändert. Umbenennen erstellt einen eigenen verwalteten Pfad; Ablehnen meldet Konflikte; Ersetzen setzt ein kollidierendes, nicht zugehöriges Ziel außer Kraft.",
|
||||
"i18n:govoplan-files.folder_sync.conflict.label": "Vorhandenes Ziel",
|
||||
"i18n:govoplan-files.folder_sync.conflict.overwrite": "Nicht zugehöriges Ziel ersetzen",
|
||||
"i18n:govoplan-files.folder_sync.conflict.reject": "Als Konflikt melden",
|
||||
"i18n:govoplan-files.folder_sync.conflict.rename": "Beide unter neuem Namen behalten",
|
||||
"i18n:govoplan-files.folder_sync.conflict.skip": "Nicht zugehörige Konflikte überspringen",
|
||||
"i18n:govoplan-files.folder_sync.description": "Synchronisiert den sichtbaren Remote-Ordner in den verwalteten Files-Bereich des verknüpften Eigentümers. Bereits bekannte Connector-Identitäten erhalten neue Versionen; Überspringungen, Konflikte, Richtlinienablehnungen und Transportfehler bleiben unten einzeln prüfbar.",
|
||||
"i18n:govoplan-files.folder_sync.destination.help": "Relative Remote-Pfade bleiben im Benutzer- oder Gruppenbereich erhalten, dem diese Connector-Verknüpfung gehört.",
|
||||
"i18n:govoplan-files.folder_sync.destination.label": "Verwaltetes Ziel",
|
||||
"i18n:govoplan-files.folder_sync.finished": "Ordnersynchronisierung abgeschlossen: {value0} erstellt, {value1} aktualisiert, {value2} unverändert, {value3} übersprungen, {value4} zu prüfen.",
|
||||
"i18n:govoplan-files.folder_sync.limit.help": "Begrenzt einen interaktiven Lauf. Wird der Grenzwert erreicht, ist das Ergebnis als abgeschnitten markiert und kann mit einem engeren Ordner fortgesetzt werden.",
|
||||
"i18n:govoplan-files.folder_sync.limit.label": "Maximale Dateien",
|
||||
"i18n:govoplan-files.folder_sync.recursive.help": "Unterordner werden nur bis zu einer begrenzten Tiefe durchlaufen. Deaktivieren Sie dies, um ausschließlich Dateien im sichtbaren Ordner zu synchronisieren.",
|
||||
"i18n:govoplan-files.folder_sync.recursive.label": "Unterordner einbeziehen",
|
||||
"i18n:govoplan-files.folder_sync.remote.help": "Der Ordner wird unterhalb der Wurzel des verknüpften Connector-Bereichs aufgelöst; übergeordnete Pfade werden abgelehnt.",
|
||||
"i18n:govoplan-files.folder_sync.remote.label": "Remote-Ordner",
|
||||
"i18n:govoplan-files.folder_sync.results.aria_label": "Ergebnisse der Ordnersynchronisierung",
|
||||
"i18n:govoplan-files.folder_sync.results.detail": "Verwaltetes Ziel oder Detail",
|
||||
"i18n:govoplan-files.folder_sync.results.result": "Ergebnis",
|
||||
"i18n:govoplan-files.folder_sync.results.source": "Quelle",
|
||||
"i18n:govoplan-files.folder_sync.run_again": "Erneut ausführen",
|
||||
"i18n:govoplan-files.folder_sync.summary.aria_label": "Zusammenfassung der Ordnersynchronisierung",
|
||||
"i18n:govoplan-files.folder_sync.summary.conflicts": "Konflikte",
|
||||
"i18n:govoplan-files.folder_sync.summary.created": "erstellt",
|
||||
"i18n:govoplan-files.folder_sync.summary.discovered": "gefunden",
|
||||
"i18n:govoplan-files.folder_sync.summary.failed": "fehlgeschlagen",
|
||||
"i18n:govoplan-files.folder_sync.summary.policy_denied": "durch Richtlinie abgelehnt",
|
||||
"i18n:govoplan-files.folder_sync.summary.skipped": "übersprungen",
|
||||
"i18n:govoplan-files.folder_sync.summary.unchanged": "unverändert",
|
||||
"i18n:govoplan-files.folder_sync.summary.updated": "aktualisiert",
|
||||
"i18n:govoplan-files.folder_sync.title": "Connector-Ordner synchronisieren",
|
||||
"i18n:govoplan-files.folder_sync.truncated": "Der Dateigrenzwert wurde erreicht. Wählen Sie zum Fortsetzen einen engeren Remote-Ordner oder erhöhen Sie den begrenzten Höchstwert.",
|
||||
"i18n:govoplan-files.folders_and_files.d107ac99": "Folders and files",
|
||||
"i18n:govoplan-files.folders.19adc47b": "Ordner",
|
||||
"i18n:govoplan-files.gb.505a44fa": "GB",
|
||||
|
||||
+18
-5
@@ -5,12 +5,15 @@ import {
|
||||
type DashboardWidgetsUiCapability,
|
||||
type FilesConnectorsUiCapability,
|
||||
type FilesFileExplorerUiCapability,
|
||||
type PlatformWebModule
|
||||
type PlatformWebModule,
|
||||
type QuickAccessToolsUiCapability
|
||||
} from "@govoplan/core-webui";
|
||||
import { generatedTranslations as productSurfaceTranslations } from "@govoplan/core-webui/outcome-product-surface-translations";
|
||||
import { FolderTree } from "./features/files/components/FileManagerComponents";
|
||||
import FileConnectorSettingsPanel from "./features/files/FileConnectorSettingsPanel";
|
||||
import ManagedFileChooser from "./features/files/components/ManagedFileChooser";
|
||||
import FileSpacesWidget from "./features/files/FileSpacesWidget";
|
||||
import FileQuickAccess from "./features/files/FileQuickAccess";
|
||||
import { listFileSpaces, listFiles, listFilesDelta, listFolders, resolveFilePatterns, shareFilesWithTarget } from "./api/files";
|
||||
import { generatedTranslations } from "./i18n/generatedTranslations";
|
||||
import "./styles/file-manager.css";
|
||||
@@ -20,8 +23,8 @@ const FileIntegrityPanel = lazy(() => import("./features/files/FileIntegrityPane
|
||||
|
||||
const fileRead = ["files:file:read"];
|
||||
const translations = {
|
||||
en: generatedTranslations.en,
|
||||
de: generatedTranslations.de
|
||||
en: { ...generatedTranslations.en, ...productSurfaceTranslations.en },
|
||||
de: { ...generatedTranslations.de, ...productSurfaceTranslations.de }
|
||||
};
|
||||
const fileDashboardWidgets: DashboardWidgetsUiCapability = {
|
||||
widgets: [
|
||||
@@ -67,6 +70,14 @@ const fileDashboardWidgets: DashboardWidgetsUiCapability = {
|
||||
}
|
||||
]
|
||||
};
|
||||
const fileQuickAccessTools: QuickAccessToolsUiCapability = {
|
||||
tools: [
|
||||
{
|
||||
id: "files.recent",
|
||||
render: (context) => createElement(FileQuickAccess, context)
|
||||
}
|
||||
]
|
||||
};
|
||||
const fileConnectorAdminSections: AdminSectionsUiCapability = {
|
||||
sections: [
|
||||
{
|
||||
@@ -129,7 +140,8 @@ export const filesModule: PlatformWebModule = {
|
||||
{ id: "files.admin.group-connectors", moduleId: "files", kind: "section", label: "Group file connections", order: 65 },
|
||||
{ id: "files.admin.user-connectors", moduleId: "files", kind: "section", label: "User file connections", order: 65 },
|
||||
{ id: "files.settings.connectors", moduleId: "files", kind: "section", label: "Personal file connections", order: 20 },
|
||||
{ id: "files.widget.spaces", moduleId: "files", kind: "section", label: "File spaces widget", order: 35 }
|
||||
{ id: "files.widget.spaces", moduleId: "files", kind: "section", label: "File spaces widget", order: 35 },
|
||||
{ id: "files.quick_access.files", moduleId: "files", kind: "quick_access", label: "Files Quick Access", order: 40 }
|
||||
],
|
||||
navItems: [{ to: "/files", label: "i18n:govoplan-files.files.6ce6c512", iconName: "folder", anyOf: fileRead, order: 40 }],
|
||||
routes: [
|
||||
@@ -150,7 +162,8 @@ export const filesModule: PlatformWebModule = {
|
||||
FileConnectorScopeManager: FileConnectorSettingsPanel
|
||||
} satisfies FilesConnectorsUiCapability,
|
||||
"admin.sections": fileConnectorAdminSections,
|
||||
"dashboard.widgets": fileDashboardWidgets
|
||||
"dashboard.widgets": fileDashboardWidgets,
|
||||
"quickAccess.tools": fileQuickAccessTools
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -113,7 +113,7 @@
|
||||
min-width: 190px;
|
||||
overflow: hidden;
|
||||
border: var(--border-line-dark);
|
||||
border-radius: 12px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-popover);
|
||||
padding: 6px;
|
||||
@@ -125,7 +125,7 @@
|
||||
gap: 9px;
|
||||
width: 100%;
|
||||
border: 0;
|
||||
border-radius: 8px;
|
||||
border-radius: var(--radius-md);
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
@@ -161,7 +161,7 @@
|
||||
max-height: min(720px, calc(100vh - 44px));
|
||||
overflow: auto;
|
||||
border: var(--border-line-dark);
|
||||
border-radius: 16px;
|
||||
border-radius: var(--radius-xl);
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-popover);
|
||||
}
|
||||
@@ -236,7 +236,7 @@
|
||||
scrollbar-gutter: stable;
|
||||
padding: 8px;
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -247,7 +247,7 @@
|
||||
width: 100%;
|
||||
min-width: 0;
|
||||
border: 0;
|
||||
border-radius: 9px;
|
||||
border-radius: var(--radius-md);
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
@@ -308,7 +308,7 @@
|
||||
gap: 1px;
|
||||
overflow: hidden;
|
||||
border: var(--border-line);
|
||||
border-radius: 6px;
|
||||
border-radius: var(--radius-compact);
|
||||
background: var(--line);
|
||||
}
|
||||
|
||||
@@ -332,13 +332,6 @@
|
||||
font-size: var(--font-size-sm);
|
||||
}
|
||||
|
||||
.archive-selection-toolbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.archive-selection-toolbar label,
|
||||
.archive-entry-row {
|
||||
display: flex;
|
||||
@@ -363,7 +356,7 @@
|
||||
overflow: auto;
|
||||
scrollbar-gutter: stable;
|
||||
border: var(--border-line);
|
||||
border-radius: 6px;
|
||||
border-radius: var(--radius-compact);
|
||||
background: var(--surface);
|
||||
}
|
||||
|
||||
@@ -437,7 +430,7 @@
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 620px) {
|
||||
@media (max-width: 680px) {
|
||||
.archive-preview-summary {
|
||||
grid-template-columns: minmax(0, 1fr);
|
||||
}
|
||||
@@ -460,7 +453,7 @@
|
||||
min-height: 320px;
|
||||
overflow: hidden;
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--panel);
|
||||
}
|
||||
|
||||
@@ -505,7 +498,7 @@
|
||||
min-height: 48px;
|
||||
width: 100%;
|
||||
border: 0;
|
||||
border-radius: 8px;
|
||||
border-radius: var(--radius-md);
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
@@ -575,7 +568,7 @@
|
||||
min-width: 0;
|
||||
padding: 10px 12px;
|
||||
border: var(--border-line);
|
||||
border-radius: 10px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -623,7 +616,7 @@
|
||||
align-items: center;
|
||||
padding: 10px 12px;
|
||||
border: var(--border-line);
|
||||
border-radius: 8px;
|
||||
border-radius: var(--radius-md);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -720,7 +713,7 @@
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 1050px) {
|
||||
@media (max-width: 1100px) {
|
||||
.files-page .file-manager-shell {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
@@ -765,7 +758,7 @@
|
||||
gap: 3px;
|
||||
padding: 12px 14px;
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -788,7 +781,7 @@
|
||||
align-items: center;
|
||||
padding: 10px;
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -826,7 +819,7 @@
|
||||
display: block;
|
||||
width: 100%;
|
||||
border: 1px dashed var(--line-dark);
|
||||
border-radius: 6px;
|
||||
border-radius: var(--radius-compact);
|
||||
background: var(--panel-soft);
|
||||
color: var(--muted);
|
||||
cursor: pointer;
|
||||
@@ -949,7 +942,7 @@
|
||||
.managed-file-entry {
|
||||
width: 100%;
|
||||
border: 1px solid transparent;
|
||||
border-radius: 9px;
|
||||
border-radius: var(--radius-md);
|
||||
background: transparent;
|
||||
color: var(--text-strong);
|
||||
text-align: left;
|
||||
@@ -1027,7 +1020,7 @@
|
||||
gap: 5px;
|
||||
align-items: center;
|
||||
border: 0;
|
||||
border-radius: 6px;
|
||||
border-radius: var(--radius-compact);
|
||||
background: transparent;
|
||||
color: var(--text-strong);
|
||||
padding: 5px 6px;
|
||||
@@ -1176,7 +1169,7 @@
|
||||
gap: 4px;
|
||||
align-items: center;
|
||||
border: 1px solid var(--success-line, var(--line));
|
||||
border-radius: 999px;
|
||||
border-radius: var(--radius-pill);
|
||||
padding: 3px 7px;
|
||||
color: var(--success, var(--muted));
|
||||
font-size: 11px;
|
||||
@@ -1263,7 +1256,79 @@
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
@media (max-width: 850px) {
|
||||
.connector-folder-sync-review {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.connector-folder-sync-summary {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(104px, 1fr));
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.connector-folder-sync-summary span {
|
||||
display: grid;
|
||||
gap: 2px;
|
||||
padding: 9px 10px;
|
||||
border: var(--border-line);
|
||||
border-radius: var(--radius-control);
|
||||
background: var(--panel-soft);
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
text-transform: capitalize;
|
||||
}
|
||||
|
||||
.connector-folder-sync-summary strong {
|
||||
color: var(--text-strong);
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
.connector-folder-sync-results {
|
||||
max-height: min(320px, 42vh);
|
||||
overflow: auto;
|
||||
border: var(--border-line);
|
||||
border-radius: var(--radius-control);
|
||||
}
|
||||
|
||||
.connector-folder-sync-result {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(180px, 1.2fr) minmax(92px, 0.45fr) minmax(220px, 1.5fr);
|
||||
gap: 12px;
|
||||
padding: 9px 11px;
|
||||
border-bottom: var(--border-line);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.connector-folder-sync-result:last-child {
|
||||
border-bottom: 0;
|
||||
}
|
||||
|
||||
.connector-folder-sync-result > * {
|
||||
min-width: 0;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.connector-folder-sync-result.is-header {
|
||||
position: sticky;
|
||||
z-index: 1;
|
||||
top: 0;
|
||||
background: var(--panel-raised, var(--panel));
|
||||
}
|
||||
|
||||
.connector-folder-sync-result.is-created strong,
|
||||
.connector-folder-sync-result.is-updated strong,
|
||||
.connector-folder-sync-result.is-unchanged strong {
|
||||
color: var(--success, var(--text-strong));
|
||||
}
|
||||
|
||||
.connector-folder-sync-result.is-conflict strong,
|
||||
.connector-folder-sync-result.is-policy_denied strong,
|
||||
.connector-folder-sync-result.is-failed strong {
|
||||
color: var(--danger, var(--text-strong));
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.managed-file-entry-head,
|
||||
.managed-file-entry {
|
||||
grid-template-columns: 22px minmax(0, 1fr) 100px;
|
||||
@@ -1276,6 +1341,11 @@
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.connector-folder-sync-result {
|
||||
grid-template-columns: 1fr;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.managed-file-chooser-dialog {
|
||||
width: calc(100vw - 20px);
|
||||
height: calc(100vh - 20px);
|
||||
|
||||
Reference in New Issue
Block a user