Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
61625fb00f | ||
|
|
13433514b9 | ||
|
|
ff84812f7f | ||
|
|
2baa8f2657 |
+164
-4
@@ -14,6 +14,19 @@ a document collaboration engine, or a records-management system.
|
||||
|
||||
## Choose a reading path
|
||||
|
||||
The Files workspace keeps **Reload → Create folder → Upload** at the top right;
|
||||
Upload is the primary everyday action. Reload re-reads the current folder with
|
||||
its active pattern/property filters and retains usable loaded content on failure.
|
||||
For connector spaces it only browses the current remote folder: it never imports
|
||||
or synchronizes content. **Connections and imports** groups explicit import,
|
||||
synchronization, and linked-space management. Opening this chooser makes no
|
||||
remote changes. Select content to use the local **Download**, applicable
|
||||
**Unpack archive**, or **Manage selection** actions. The latter groups move,
|
||||
copy, rename, sharing, and access explanations, with deletion in a separate
|
||||
section and its existing confirmation. All permissions, connector restrictions,
|
||||
conflict checks, retention rules, keyboard shortcuts, and context menus remain
|
||||
in force; unavailable actions explain their blockers.
|
||||
|
||||
| If you need to... | Start with... |
|
||||
| --- | --- |
|
||||
| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) |
|
||||
@@ -144,6 +157,15 @@ updates currently belong to connector sync.
|
||||
|
||||
### Preview and unpack archives
|
||||
|
||||
The dialog uses the shared blurred loading overlay, without an envelope
|
||||
animation. Its status distinguishes transfer, inspection, extraction/storage,
|
||||
and transaction finalization. Selected file counts and byte totals are known
|
||||
from the preview; running extraction displays actual server counters. Unknown
|
||||
progress is indeterminate, and all bytes processed does not mean the transaction
|
||||
has committed. Closing, changing the destination, and editing the selection are
|
||||
disabled while work runs. Errors release the overlay and keep the selection
|
||||
available for review; no automatic import retry is performed.
|
||||
|
||||
The UI previews ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ before writing managed
|
||||
files. Users may select individual files or complete folders. The request is
|
||||
spooled to a bounded temporary file rather than buffered wholly in memory. The
|
||||
@@ -161,10 +183,43 @@ defaults are:
|
||||
rejected;
|
||||
- actual bytes read are counted, not only archive header declarations.
|
||||
|
||||
The browser retains the selected archive and password until confirmation.
|
||||
Confirmation reuploads the archive, verifies the token and digest, repeats all
|
||||
safety checks, and commits only the selected files. No preview archive or
|
||||
password is retained server-side.
|
||||
The UI requests private temporary staging and uploads the archive once. Password
|
||||
verification, renewed preview requests and confirmation reuse that same copy;
|
||||
the password remains request-only and is never stored with it. Confirmation
|
||||
rechecks the actor/destination-bound preview token, digest and safety limits and
|
||||
commits only selected files. Repreviewing does not extend the initial staged
|
||||
copy's lifetime. Cancelling or replacing the archive requests its release;
|
||||
successful confirmation removes it. An expired or missing copy requires a fresh
|
||||
upload; when requesting another preview, the UI can safely repeat that preview
|
||||
with the original selected file. It never retries confirmation automatically.
|
||||
API clients that do not opt into `retain_upload`
|
||||
retain the original request-only preview/confirmation upload contract.
|
||||
|
||||
For an archive that is **already uploaded**, select exactly one managed archive
|
||||
and choose **Unpack archive** in the action bar or context menu. Select an
|
||||
accessible destination space and folder, then choose **Preview archive**. The
|
||||
same entry selection, folder selection, password verification and confirmation
|
||||
dialog applies. Bytes stay on the server; no browser download/re-upload is
|
||||
needed. This path resolves verified source bytes within the configured archive
|
||||
size limit before passing them into the ordinary bounded extraction pipeline.
|
||||
|
||||
Managed extraction requires `files:file:read`, `files:file:download` and
|
||||
`files:file:upload`, current source ownership/share access, and destination-owner
|
||||
access. Both stages enforce source tenant, deletion/version checks, integrity,
|
||||
quarantine, envelope decryption availability and effective source/destination
|
||||
connector restrictions. The token additionally binds the exact managed file and
|
||||
version. A changed version, revoked share, expired token or changed destination
|
||||
requires a fresh preview. No existing destination file is overwritten, including
|
||||
the source archive; choose another destination if a member collides. Failed
|
||||
extraction rolls back newly created managed assets through the existing upload
|
||||
transaction/recovery path. The source file, bytes and version are not modified.
|
||||
|
||||
New members use ordinary archive-upload destination defaults: neither ZIP
|
||||
password protection nor a source storage encryption envelope is automatically
|
||||
copied to extracted files. The resulting provenance records source file/version
|
||||
lineage without passwords. Keep the busy dialog open until the synchronous
|
||||
operation completes; this is not a resumable background job. Remote entries in
|
||||
connector spaces must first be synchronized into managed storage.
|
||||
|
||||
### Organize files and folders
|
||||
|
||||
@@ -565,6 +620,9 @@ before it can return a usable client or session.
|
||||
| `FILE_ARCHIVE_MAX_EXPANDED_BYTES` | 2 GiB | Maximum expanded archive bytes |
|
||||
| `FILE_ARCHIVE_MAX_EXPANSION_RATIO` | 100 | Maximum expanded-to-compressed ratio |
|
||||
| `FILE_ARCHIVE_PREVIEW_TTL_SECONDS` | 1,800 | Lifetime of the sealed archive preview token |
|
||||
| `FILE_ARCHIVE_WORK_ROOT` | private OS-temporary directory | Shared transient archive/progress workspace; distinct from durable blob storage |
|
||||
| `FILE_ARCHIVE_STAGED_MAX_BYTES` | 2 GiB | Total staged archive bytes admitted per work root |
|
||||
| `FILE_ARCHIVE_STAGED_PER_ACTOR` | 4 | Maximum staged archives per tenant/user; oldest inactive stages may be evicted |
|
||||
| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets |
|
||||
|
||||
The local backend is the operational baseline. It resolves every storage key
|
||||
@@ -586,6 +644,107 @@ durable shared mount only for same-host replicas. Independent hosts require the
|
||||
`shared` profile with external S3, PostgreSQL, Redis, a stable installation id,
|
||||
and one immutable module composition.
|
||||
|
||||
### Archive staging, progress and performance
|
||||
|
||||
Archive preview staging is optional in the API and requested by the UI. It is
|
||||
not durable managed storage or a background job queue. The default work root
|
||||
is below the OS temporary directory, isolated by the service uid and a hash of
|
||||
the configured local storage root. `FILE_ARCHIVE_WORK_ROOT` selects an explicit
|
||||
root. The internal directory must have mode `0700`; archive, lease and progress
|
||||
files use `0600`. Passwords are never retained. Progress records contain only
|
||||
status and file/byte counters, not member names or content.
|
||||
|
||||
All same-host API workers must see the same work directory, including across
|
||||
container mounts. For multiple hosts, provide shared POSIX storage with working
|
||||
`flock` locks, or route an archive's preview, confirmation and progress requests
|
||||
to the same host. Shared S3 blob storage alone does not share this transient
|
||||
workspace. Include upload limits, temporary disk capacity and sufficiently long
|
||||
proxy/request timeouts in deployment checks; confirmation remains synchronous.
|
||||
|
||||
The defaults permit 2 GiB of staged data per work root and four stages per
|
||||
tenant/user. A new stage can evict the actor's oldest inactive copy; active
|
||||
imports are never evicted to admit another one. Staging expires after the
|
||||
configured preview TTL (1,800 seconds by default), measured from initial
|
||||
staging and not renewed by password repreview. Cancel/replacement requests and
|
||||
successful import remove copies; expiry cleanup is opportunistic during later
|
||||
archive work, so an abandoned file is not guaranteed to disappear at its exact
|
||||
idle deadline. Active POSIX leases protect copies in use beyond TTL until the
|
||||
operation releases them; a crashed worker releases its OS lock. Do not delete
|
||||
active work directories as a cleanup shortcut.
|
||||
|
||||
For classic ZIPCrypto archives, an optional maintained system `libarchive`
|
||||
library accelerates decoding through its public read API. It applies only to
|
||||
stored/Deflate ZIP entries with ASCII or UTF-8-flagged names; AES, unsupported
|
||||
formats/encodings and a missing library use the existing Python reader chosen
|
||||
before decoding. Native corruption/password errors never trigger a fallback.
|
||||
Install/update the library through the deployment's supported package process
|
||||
and restart workers after installation. Both paths retain complete header and
|
||||
path validation, selection and actual output limits; the native path also
|
||||
independently checks size and CRC and never extracts to filesystem paths.
|
||||
|
||||
All ZIP/TAR metadata parsing (including ZIP central directories and TAR PAX
|
||||
headers) and decoding now run in fresh, credential-stripped child processes.
|
||||
Preview/password checks have 120 seconds wall time, 90 CPU seconds and 512 MiB
|
||||
address space. Confirmation uses one child for the entire archive, with 600
|
||||
seconds wall time and 300 CPU seconds. Its address-space limit is the larger of
|
||||
512 MiB or `3 × configured member limit + 128 MiB`. A member limit must be positive
|
||||
and at most 2 GiB; the default 50 MiB member limit and the existing 250 MiB request,
|
||||
10,000-entry, 2 GiB expanded and 100:1 limits are unchanged. The kernel file-size
|
||||
ceiling is `max(member limit, 16 KiB)`; actual member/cumulative bytes are checked
|
||||
independently. Metadata transport has a 64 MiB ceiling, final extraction receipts
|
||||
64 KiB, and status records 16 KiB. Raw archive/member bytes are not pipe DTOs.
|
||||
|
||||
`GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY` is shared with other isolated module work:
|
||||
one admitted operation per API/worker process by default, configurable from one
|
||||
to 16. Admission covers source snapshot preparation before starting the child,
|
||||
so a busy request does not first copy its source. Capacity is not global across
|
||||
replicas. Include each replica's address-space and temporary-disk budgets when
|
||||
sizing the deployment. Resource controls are required; there is no inline parser
|
||||
fallback. These are process resource limits, not a filesystem/network sandbox or
|
||||
an aggregate cgroup memory guarantee.
|
||||
|
||||
Each admitted operation creates a separate local OS-temporary `0700` directory
|
||||
and `0600` source snapshot, beyond the upload-once work-root quota described
|
||||
above. Plan additional capacity for one compressed source copy and at most one
|
||||
extracted member per admitted archive. Snapshot copying is bounded to the initial
|
||||
regular-file size and rejects changed sources. The staging wrapper writes only
|
||||
numbered internal filenames, never archive names. The parent rechecks selected
|
||||
logical paths, regular/no-symlink file identity, size, digest and monotonic
|
||||
progress, persists the member with the existing Files authority/transaction, and
|
||||
deletes it before acknowledging the child to decode the next member. A private
|
||||
`0600` FIFO carries one eight-byte sequence acknowledgement; completed members
|
||||
wake the parent immediately, with no fixed per-member sleep. Wake notifications
|
||||
are capped at 32 KiB; configurations allowing more than 32,768 members use Core's
|
||||
ordinary progress polling after that budget, still within the wall-time limit.
|
||||
Passwords remain request-only and are never written to these files. Handled success/error
|
||||
paths reap the child and remove the private directory. Abrupt host/parent loss
|
||||
can leave private temporary files; use the deployment's OS-temporary cleanup
|
||||
policy without deleting active work.
|
||||
|
||||
The confirmation wall budget includes time waiting for parent storage; a slow
|
||||
destination may exhaust it after earlier members were staged for persistence.
|
||||
Existing rollback and blob recovery remain authoritative. Busy capacity, missing
|
||||
controls, CPU/memory/time/output limits, or invalid staging records produce a
|
||||
controlled failure; split the archive or explicitly retry after resolving the
|
||||
cause. There is no resumable worker or automatic confirmation retry. An optional
|
||||
UI progress-write failure remains non-fatal, but a failed private worker status
|
||||
or acknowledgement channel must abort safely.
|
||||
|
||||
Verified members are read and stored one at a time, avoiding a whole expanded
|
||||
archive in memory. Storage-client reuse is confined to one archive operation;
|
||||
authorization, tenant isolation and policy are not cached. Batched response
|
||||
metadata avoids repeated lookups, while every member still follows the durable
|
||||
recovery ledger, fencing, integrity and post-commit verification path. The
|
||||
optional decoder's speedup does not imply an equal end-to-end or S3 speedup.
|
||||
Use `tests/benchmark_archive_storage.py` for isolated synthetic SQLite/local
|
||||
storage and decoder measurements; it is not a production PostgreSQL/S3 benchmark.
|
||||
|
||||
Transfer, inspection, extraction/storage and finalization remain separate
|
||||
phases. File and byte counters report actual work; an unknown amount remains
|
||||
indeterminate. Reading all bytes is not transaction success: the UI waits for
|
||||
confirmation and commit settlement. Failed or unavailable progress reporting
|
||||
does not automatically retry an import or prove it failed.
|
||||
|
||||
### Connector egress
|
||||
|
||||
Connector access to private networks is a deployment-wide decision:
|
||||
@@ -834,6 +993,7 @@ All routes below are under `/api/v1/files`.
|
||||
| --- | --- |
|
||||
| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` |
|
||||
| Upload and folders | `POST /upload`, `POST /upload-zip` (compatibility), `POST /archive-preview`, `POST /archive-confirm`, `POST /folders`, `POST /folders/delete` |
|
||||
| Existing managed archive extraction | `POST /{file_id}/archive-preview`, `POST /{file_id}/archive-confirm` (JSON source version, destination, request-only password, preview token and selected paths; read/download/upload permissions) |
|
||||
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
|
||||
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
|
||||
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
|
||||
|
||||
@@ -9,7 +9,7 @@ and consequences described here.
|
||||
| Surface | Primary task | Archetype | Consequence | Pattern evidence |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `/files` space and folder panes | Browse managed and connected content without losing location | Directory/explorer | Low for navigation; medium for exposing filenames and provenance | Full-height two-pane workspace, bounded panes, stable selection and contextual Help Center link |
|
||||
| `/files` toolbar and property filters | Find and act on the current selection | Explorer actions and local filtering | Medium for upload, move, copy, share and synchronization; high for delete | Actions remain beside the affected list, disabled controls explain permission/state/selection blockers, destructive work uses `ConfirmDialog` |
|
||||
| `/files` workspace header and selection toolbar | Find content and act at the appropriate scope | Persistent workspace actions plus local selection tools | Medium for upload, move, copy, share and synchronization; high for delete | Reload → Create folder → primary Upload stay above both panes; Connections and imports groups secondary workspace tools; Download, eligible Unpack archive and Manage selection stay list-local; deletion is separated and uses `ConfirmDialog` |
|
||||
| Upload/archive, transfer, rename and connector-import dialogs | Supply and review one bounded change | Adaptive create/edit or guided import | Medium to high because files, paths and external bytes change | Shared `Dialog`, `FileDropZone`, validation, conflict review, unsaved inputs and explicit confirmation |
|
||||
| File share dialog | Inspect and change access | Review/decision | High because another actor gains access | Shared dialog, access explanation, stable row actions and destructive confirmation |
|
||||
| System/tenant/group/user connector surfaces | Compare connections, credentials and effective policy | Administration/configuration | High because endpoints, secrets and inherited policy control external access | Shared `ConnectionTree`, adaptive forms, `ActionBlockerHint`, policy provenance and contextual admin help |
|
||||
@@ -22,6 +22,11 @@ and consequences described here.
|
||||
|
||||
- Loading, errors, success, empty results, access explanations and confirmation
|
||||
use Core components. Files does not reproduce the application shell.
|
||||
- Explicit Reload bypasses the client read cache for the current folder and all
|
||||
active filter pages, never imports/synchronizes content, and keeps usable data
|
||||
on failure. Late responses cannot replace a changed tenant, account or folder.
|
||||
- Selection and connection tools use domain-labelled shared dialogs and form
|
||||
sections; opening a tool chooser does not itself perform a remote operation.
|
||||
- A connector that comes from deployment settings remains visible but read-only;
|
||||
its action explains that bootstrap configuration and a restart are required.
|
||||
- Missing permission, target, selection, endpoint, or compatible provider is an
|
||||
@@ -52,4 +57,3 @@ The focused structural test guards these contracts, optional-module boundaries,
|
||||
confirmation, contextual help, advanced-only JSON and responsive rules. Core's
|
||||
TypeScript build, structural localization audit, module-permutation suite and
|
||||
full-product bundle check provide the integration gates.
|
||||
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.24",
|
||||
"version": "0.1.27",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "webui/src/index.ts",
|
||||
@@ -26,7 +26,7 @@
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.18"
|
||||
"@govoplan/core-webui": "^0.1.45"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
+2
-2
@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-files"
|
||||
version = "0.1.24"
|
||||
version = "0.1.27"
|
||||
description = "GovOPlaN files module with backend and WebUI integration."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = { file = "LICENSE" }
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.42",
|
||||
"govoplan-core>=0.1.46",
|
||||
"defusedxml>=0.7,<1",
|
||||
"pyzipper>=0.3.6,<1",
|
||||
"python-multipart>=0.0.31,<1",
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""Bounded, private, transient archive work shared by workers on one host.
|
||||
|
||||
This is not a detached job queue: confirmation still owns its transaction.
|
||||
Progress never contains filenames, passwords, or extracted document content.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
class ArchiveWorkExpired(FileStorageError):
|
||||
pass
|
||||
|
||||
|
||||
def _uuid(value: str) -> str:
|
||||
try:
|
||||
result = str(UUID(value))
|
||||
except (ValueError, AttributeError, TypeError) as exc:
|
||||
raise FileStorageError("Invalid archive operation identifier") from exc
|
||||
if result != value:
|
||||
raise FileStorageError("Invalid archive operation identifier")
|
||||
return result
|
||||
|
||||
|
||||
def _actor(tenant_id: str, user_id: str) -> str:
|
||||
return hashlib.sha256(json.dumps([tenant_id, user_id]).encode()).hexdigest()
|
||||
|
||||
|
||||
def _root(settings) -> Path:
|
||||
configured = getattr(settings, "file_archive_work_root", None)
|
||||
identity = hashlib.sha256(str(Path(getattr(settings, "file_storage_local_root", "runtime/files")).absolute()).encode()).hexdigest()[:16]
|
||||
base = Path(configured) if configured else Path(tempfile.gettempdir()) / f"govoplan-archive-work-{os.getuid()}-{identity}"
|
||||
root = base.absolute() / "v1"
|
||||
# All internal paths are fixed subdirectories or server-generated UUIDs.
|
||||
root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
if root.is_symlink() or root.stat().st_mode & 0o077:
|
||||
raise FileStorageError("Archive work directory must be private (mode 0700)")
|
||||
return root
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _locked(root: Path):
|
||||
# POSIX advisory locks release automatically if a worker exits. GovOPlaN's
|
||||
# server deployment targets POSIX; no credentials are passed to a process.
|
||||
import fcntl
|
||||
descriptor = os.open(root / ".lock", os.O_CREAT | os.O_RDWR, 0o600)
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX)
|
||||
yield
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def _ttl(settings) -> int:
|
||||
return settings.file_archive_preview_ttl_seconds
|
||||
|
||||
|
||||
def _lease_active(path: Path) -> bool:
|
||||
import fcntl
|
||||
try:
|
||||
descriptor = os.open(path, os.O_RDWR | os.O_NOFOLLOW)
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
try:
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
return True
|
||||
return False
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def _cleanup(root: Path, ttl: int) -> None:
|
||||
cutoff = time.time() - ttl
|
||||
for entry in root.iterdir():
|
||||
if not re.fullmatch(r"[a-f0-9]{64}-[a-f0-9-]{36}\.(archive|progress|lease)", entry.name) or entry.is_symlink():
|
||||
continue
|
||||
try:
|
||||
if entry.stat().st_mtime >= cutoff:
|
||||
continue
|
||||
lease = entry.with_suffix(".lease")
|
||||
if entry.suffix in {".archive", ".lease"} and _lease_active(lease):
|
||||
continue
|
||||
entry.unlink(missing_ok=True)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def stage_upload(settings, source: str, *, tenant_id: str, user_id: str) -> str:
|
||||
root = _root(settings)
|
||||
actor = _actor(tenant_id, user_id)
|
||||
stage_id = str(uuid4())
|
||||
target = root / f"{actor}-{stage_id}.archive"
|
||||
size = Path(source).stat().st_size
|
||||
with _locked(root):
|
||||
_cleanup(root, _ttl(settings))
|
||||
stages = [entry for entry in root.glob("*.archive") if not entry.is_symlink()]
|
||||
own = sorted((entry for entry in stages if entry.name.startswith(actor + "-")), key=lambda item: item.stat().st_mtime)
|
||||
limit = getattr(settings, "file_archive_staged_per_actor", 4)
|
||||
while len(own) >= limit:
|
||||
victim = next((entry for entry in own if not _lease_active(entry.with_suffix(".lease"))), None)
|
||||
if victim is None:
|
||||
raise FileStorageError("Too many archive imports are active; wait for one to finish")
|
||||
victim.unlink(missing_ok=True)
|
||||
own.remove(victim)
|
||||
stages.remove(victim)
|
||||
maximum = getattr(settings, "file_archive_staged_max_bytes", 2 * 1024 ** 3)
|
||||
if size + sum(entry.stat().st_size for entry in stages) > maximum:
|
||||
raise FileStorageError("Temporary archive storage is full; retry after existing previews expire")
|
||||
descriptor = os.open(target, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb") as output, open(source, "rb") as stream:
|
||||
shutil.copyfileobj(stream, output, length=1024 * 1024)
|
||||
except BaseException:
|
||||
target.unlink(missing_ok=True)
|
||||
raise
|
||||
return stage_id
|
||||
|
||||
|
||||
@contextmanager
|
||||
def use_staged_upload(settings, stage_id: str, *, tenant_id: str, user_id: str):
|
||||
import fcntl
|
||||
root = _root(settings)
|
||||
path = root / f"{_actor(tenant_id, user_id)}-{_uuid(stage_id)}.archive"
|
||||
lease = path.with_suffix(".lease")
|
||||
with _locked(root):
|
||||
_cleanup(root, _ttl(settings))
|
||||
if not path.is_file() or path.is_symlink() or path.stat().st_mtime < time.time() - _ttl(settings):
|
||||
raise ArchiveWorkExpired("Archive preview expired; select the archive and preview it again")
|
||||
descriptor = os.open(lease, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
os.close(descriptor)
|
||||
raise FileStorageError("This archive is already being processed") from exc
|
||||
try:
|
||||
yield str(path)
|
||||
finally:
|
||||
try:
|
||||
with _locked(root):
|
||||
lease.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def discard_staged_upload(settings, stage_id: str, *, tenant_id: str, user_id: str) -> None:
|
||||
root = _root(settings)
|
||||
path = root / f"{_actor(tenant_id, user_id)}-{_uuid(stage_id)}.archive"
|
||||
with _locked(root):
|
||||
if not _lease_active(path.with_suffix(".lease")):
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
|
||||
class ArchiveProgress:
|
||||
def __init__(self, settings, operation_id: str | None, *, tenant_id: str, user_id: str):
|
||||
self.path: Path | None = None
|
||||
self.value = {"phase": "inspecting", "completed_files": 0, "total_files": 0,
|
||||
"completed_bytes": 0, "total_bytes": 0, "status": "running"}
|
||||
self.last_write = 0.0
|
||||
if not isinstance(operation_id, str):
|
||||
return
|
||||
root = _root(settings)
|
||||
self.path = root / f"{_actor(tenant_id, user_id)}-{_uuid(operation_id)}.progress"
|
||||
with _locked(root):
|
||||
_cleanup(root, _ttl(settings))
|
||||
# Bounded receipts as well as bounded file bytes; keep newest256 per
|
||||
# actor, never remove a live progress record to admit another one.
|
||||
own = sorted(root.glob(f"{_actor(tenant_id, user_id)}-*.progress"), key=lambda entry: entry.stat().st_mtime)
|
||||
for old in own[:-255]:
|
||||
try:
|
||||
if json.loads(old.read_text()).get("status") != "running":
|
||||
old.unlink(missing_ok=True)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
if len(list(root.glob(f"{_actor(tenant_id, user_id)}-*.progress"))) >= 256:
|
||||
raise FileStorageError("Too many active archive operations")
|
||||
try:
|
||||
descriptor = os.open(self.path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
except FileExistsError as exc:
|
||||
raise FileStorageError("Archive operation identifier has already been used; do not resubmit it") from exc
|
||||
with os.fdopen(descriptor, "w") as target:
|
||||
json.dump(self.value, target)
|
||||
|
||||
def __call__(self, phase: str, completed_files: int, total_files: int, completed_bytes: int, total_bytes: int):
|
||||
changed_phase = self.value["phase"] != phase
|
||||
self.value.update(phase=phase, completed_files=completed_files, total_files=total_files,
|
||||
completed_bytes=completed_bytes, total_bytes=total_bytes)
|
||||
if changed_phase or time.monotonic() - self.last_write >= .2:
|
||||
self._write()
|
||||
|
||||
def finish(self, success: bool):
|
||||
self.value.update(status="complete" if success else "failed", phase="complete" if success else "failed")
|
||||
self._write()
|
||||
|
||||
def _write(self):
|
||||
if self.path is None:
|
||||
return
|
||||
temporary = None
|
||||
try:
|
||||
descriptor, temporary = tempfile.mkstemp(prefix=".progress-", dir=self.path.parent)
|
||||
with os.fdopen(descriptor, "w") as target:
|
||||
json.dump(self.value, target)
|
||||
os.replace(temporary, self.path)
|
||||
self.last_write = time.monotonic()
|
||||
except OSError:
|
||||
# Observability cannot turn a committed import into a failed
|
||||
# response or cause the browser to retry a write.
|
||||
pass
|
||||
finally:
|
||||
if temporary:
|
||||
try:
|
||||
Path(temporary).unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def read_progress(settings, operation_id: str, *, tenant_id: str, user_id: str) -> dict:
|
||||
path = _root(settings) / f"{_actor(tenant_id, user_id)}-{_uuid(operation_id)}.progress"
|
||||
try:
|
||||
if path.is_symlink() or path.stat().st_mtime < time.time() - _ttl(settings):
|
||||
raise ArchiveWorkExpired("Archive progress is not available")
|
||||
return json.loads(path.read_text())
|
||||
except (FileNotFoundError, json.JSONDecodeError) as exc:
|
||||
raise ArchiveWorkExpired("Archive progress is not available") from exc
|
||||
@@ -14,11 +14,13 @@ from sqlalchemy import (
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
event,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from govoplan_core.db.base import Base, TimestampMixin
|
||||
from govoplan_files.backend.storage.provenance import source_identity_hash, source_provenance_from_metadata
|
||||
|
||||
|
||||
def new_uuid() -> str:
|
||||
@@ -192,6 +194,10 @@ class FileFolder(Base, TimestampMixin):
|
||||
|
||||
class FileAsset(Base, TimestampMixin):
|
||||
__tablename__ = "file_assets"
|
||||
__table_args__ = (
|
||||
Index("ix_file_assets_user_source", "tenant_id", "owner_type", "owner_user_id", "source_identity_hash"),
|
||||
Index("ix_file_assets_group_source", "tenant_id", "owner_type", "owner_group_id", "source_identity_hash"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
@@ -227,6 +233,15 @@ class FileAsset(Base, TimestampMixin):
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
source_identity_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
|
||||
@event.listens_for(FileAsset, "before_insert")
|
||||
@event.listens_for(FileAsset, "before_update")
|
||||
def _refresh_asset_source_identity(_mapper: object, _connection: object, asset: FileAsset) -> None:
|
||||
# All owning metadata writes (including copy/restore) use ORM instances.
|
||||
# Direct/bulk SQL metadata writers must also update this derived column.
|
||||
asset.source_identity_hash = source_identity_hash(source_provenance_from_metadata(asset.metadata_))
|
||||
|
||||
|
||||
class FileVersion(Base, TimestampMixin):
|
||||
|
||||
@@ -179,8 +179,34 @@ def _archive_topic(
|
||||
f"The deployment accepts ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ requests up to {request_limit}, limits actual expanded data to {expanded_limit}, "
|
||||
f"each member to {member_limit}, the archive to {max_entries:,} entries, and expansion to {max_expansion_ratio}:1. "
|
||||
f"The server-issued preview expires after {preview_minutes} minutes. Password-protected ZIP archives are supported; passwords remain request-only. "
|
||||
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers."
|
||||
"The UI uploads the archive once into bounded private temporary storage; password verification and confirmation reuse that staged upload without another transfer. "
|
||||
"Repreviewing does not extend its original lifetime. Cancel, replacing the archive and successful confirmation release the stage; an expired or missing stage requires a new upload. "
|
||||
"A requested preview may safely reupload the selected file if its stage expired; confirmation is never retried automatically. "
|
||||
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers. "
|
||||
"Archive paths are limited to 4096 UTF-8 bytes and 128 components; derived parent directories count toward the entry limit. TAR inspection checks each header's size and ratio before advancing across that member's payload. "
|
||||
"Metadata parsing and decoding run in disposable resource-limited children, including native ZIP decoding. Preview allows 120 seconds and confirmation 600 seconds; busy workers and CPU, memory, time or transport limits fail closed. Split a rejected archive or explicitly retry later; confirmation is not automatically retried. "
|
||||
"The shared blurred dialog overlay separates upload transfer, inspection, extraction/storage and final commit, without an envelope animation. "
|
||||
"Selected file/byte totals appear immediately and extraction progress uses actual server counters. Unknown progress is indeterminate; completion is shown only after the transaction succeeds. "
|
||||
"Keep the dialog open while processing; errors release the overlay and preserve the selection for review."
|
||||
),
|
||||
translations={"de": {
|
||||
"title": "Ein Archiv prüfen und entpacken",
|
||||
"summary": f"Bis zu {max_entries:,} Einträge aus ZIP- oder TAR-Archiven prüfen und gezielt entpacken, bevor verwaltete Dateien entstehen.",
|
||||
"body": (
|
||||
f"Diese Installation erlaubt ZIP, TAR, TAR.GZ, TAR.BZ2 und TAR.XZ bis {request_limit}, höchstens {expanded_limit} tatsächlich entpackte Daten, "
|
||||
f"{member_limit} pro Datei, {max_entries:,} Einträge und ein Expansionsverhältnis von {max_expansion_ratio}:1. "
|
||||
f"Die serverseitige Vorschau läuft nach {preview_minutes} Minuten ab. Passwortgeschützte ZIP-Archive werden unterstützt; Passwörter bleiben auf die Anfrage beschränkt. "
|
||||
"Die Oberfläche überträgt das Archiv einmal in begrenzten privaten Zwischenspeicher; Passwortprüfung und Bestätigung verwenden diese temporäre Kopie ohne erneute Übertragung. "
|
||||
"Eine erneute Vorschau verlängert die ursprüngliche Laufzeit nicht. Abbrechen, Archivwechsel und erfolgreicher Abschluss geben die Kopie frei; fehlt sie oder ist sie abgelaufen, muss das Archiv erneut hochgeladen werden. "
|
||||
"Bei einer angeforderten Vorschau darf die Oberfläche die ausgewählte Datei nach Ablauf erneut übertragen; die Bestätigung wird niemals automatisch wiederholt. "
|
||||
"Unsichere Pfade und besondere Dateisystemeinträge werden abgelehnt; tatsächlich gelesene Bytes werden gezählt. "
|
||||
"Archivpfade sind auf 4096 UTF-8-Bytes und 128 Komponenten begrenzt; abgeleitete übergeordnete Ordner zählen zur Eintragsgrenze. Die TAR-Prüfung kontrolliert Größe und Expansionsverhältnis bei jedem Header, bevor dessen Nutzdaten übersprungen werden. "
|
||||
"Metadatenprüfung und Dekodieren einschließlich nativem ZIP laufen in kurzlebigen ressourcenbegrenzten Kindprozessen. Für die Vorschau gelten 120 Sekunden, für die Bestätigung 600 Sekunden; ausgelastete Worker sowie CPU-, Speicher-, Zeit- oder Transportgrenzen führen zum sicheren Abbruch. Ein abgelehntes Archiv aufteilen oder später ausdrücklich erneut versuchen; die Bestätigung wird nicht automatisch wiederholt. "
|
||||
"Die gemeinsame Überlagerung des weichgezeichneten Dialogs unterscheidet Übertragung, Prüfung, Entpacken/Speichern und abschließende Transaktionsbestätigung, ohne Briefumschlaganimation. "
|
||||
"Ausgewählte Datei- und Bytezahlen erscheinen sofort; der Entpackfortschritt nutzt tatsächliche Serverzähler. Unbekannter Fortschritt bleibt unbestimmt, der Abschluss wird erst nach erfolgreicher Transaktion angezeigt. "
|
||||
"Den Dialog während der Verarbeitung geöffnet lassen; Fehler entfernen die Überlagerung und erhalten die Auswahl zur Prüfung."
|
||||
),
|
||||
}},
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "file_manager", "process_participant"),
|
||||
@@ -238,7 +264,7 @@ def _archive_topic(
|
||||
{
|
||||
"id": "archive-entry-count",
|
||||
"label": "Maximum entry count",
|
||||
"description": f"An archive may contain at most {max_entries:,} declared entries.",
|
||||
"description": f"An archive may contain at most {max_entries:,} entries, including derived parent directories.",
|
||||
"values": [f"{max_entries:,} entries"],
|
||||
},
|
||||
{
|
||||
@@ -252,6 +278,7 @@ def _archive_topic(
|
||||
"files.workflow.upload-managed-files",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.find-and-download-files",
|
||||
"files.reference.shared-storage-profile",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1,12 +1,29 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
from typing import Iterable
|
||||
|
||||
from govoplan_core.core.modules import DocumentationTopic
|
||||
from govoplan_core.core.modules import DocumentationTopic, localize_documentation_topics as _localize_topics
|
||||
|
||||
|
||||
_TRANSLATIONS = {
|
||||
"files.workflow.unpack-managed-archive": {
|
||||
"title": "Ein bereits hochgeladenes Archiv entpacken",
|
||||
"summary": "Ein verwaltetes ZIP- oder TAR-Archiv auswählen und denselben Vorschau-, Passwort- und Auswahlablauf wie beim Hochladen nutzen, ohne es erneut zu übertragen.",
|
||||
"body": (
|
||||
"Wählen Sie genau eine ZIP-, TAR-, TAR.GZ-, TAR.BZ2- oder TAR.XZ-Datei in einem verwalteten Bereich und anschließend Archiv entpacken in der Aktionsleiste oder im Kontextmenü. "
|
||||
"Wählen Sie einen zugänglichen persönlichen oder Gruppen-Zielbereich und fordern Sie die Archivvorschau an; dabei entstehen noch keine Dateien. Im gemeinsamen Archivdialog prüfen Sie Einträge und gegebenenfalls das ZIP-Passwort und wählen Dateien oder Ordner aus. "
|
||||
"Beim Bestätigen liest der Server das verwaltete Quellarchiv und erstellt neue Zieldateien. Quellarchiv, Metadaten und Version bleiben unverändert; Zielkonflikte werden abgelehnt und überschreiben niemals das Quellarchiv. "
|
||||
"Lese-, Download- und Upload-Berechtigung sind gemeinsam erforderlich. Eigentum, aktuelle Freigaben, Mandant, Quellversion, Connector-Richtlinien, Integrität, Quarantäne und gegebenenfalls die Verfügbarkeit der Hüllenentschlüsselung werden beim Bestätigen erneut geprüft. "
|
||||
"Die kurzlebige Vorschau ist an Quellarchiv, Version, Person und Ziel gebunden. Geänderte oder unzugängliche Quellen verlangen eine neue Vorschau; die Version wird niemals still gewechselt. "
|
||||
"Dieselben Grenzen für Archivgröße, tatsächliche entpackte Gesamtgröße, einzelne Dateien, Anzahl der Einträge, Expansionsverhältnis und sichere Pfade wie beim Archiv-Upload gelten. Passwörter bleiben auf die Anfrage beschränkt. "
|
||||
"Wie beim Archiv-Upload entstehen gewöhnliche Zieldateien; der Passwortschutz des Archivs oder seine Speicher-Verschlüsselungshülle wird nicht automatisch auf Einträge übertragen. "
|
||||
"Während Prüfung und Entpacken überlagert die gemeinsame Ladeanzeige den weichgezeichneten Dialog, ohne Briefumschlaganimation. "
|
||||
"Die Anzahl und Gesamtgröße ausgewählter Dateien sind sofort sichtbar; der laufende Fortschritt zeigt gemessene Serverzähler statt zeitbasierter Schätzungen. "
|
||||
"Übertragung, Prüfung, Entpacken/Speichern und abschließende Transaktionsbestätigung sind getrennte Phasen; übertragene oder verarbeitete Bytes allein bedeuten noch keinen erfolgreichen Abschluss. "
|
||||
"Unbekannter Fortschritt bleibt unbestimmt. Schließen, Auswahl und Zieländerung sind während der Verarbeitung gesperrt; Fehler geben den unveränderten Prüfdialog wieder frei. "
|
||||
"Die Verarbeitung läuft synchron: Lassen Sie den beschäftigten Dialog bis zum Abschluss geöffnet. Dateien aus entfernten Connector-Bereichen müssen zuerst in den verwalteten Speicher synchronisiert werden."
|
||||
),
|
||||
},
|
||||
"files.tabular-content": {
|
||||
"title": "Verwaltete CSV- und XLSX-Versionen als gesteuerte Datenquellen verwenden",
|
||||
"summary": "Exakte autorisierte Dateiversionen für Connectors bereitstellen, ohne Files-Kontrollen zu umgehen.",
|
||||
@@ -39,7 +56,10 @@ _TRANSLATIONS = {
|
||||
"title": "Verwaltete Dateien und Ordner organisieren",
|
||||
"summary": "Ordner anlegen und zugängliche Inhalte mit ausdrücklicher Konfliktbehandlung umbenennen, verschieben oder kopieren.",
|
||||
"body": (
|
||||
"Die Organisation bleibt in gesteuerten persönlichen oder Gruppenbereichen. Verschieben erhält die Asset-Identität; Kopieren erzeugt neue Assets und Versionen, die unveränderliche Blob-Bytes wiederverwenden. Jeder Zielkonflikt muss ausdrücklich abgelehnt, durch Umbenennen gelöst, überschrieben oder übersprungen werden."
|
||||
"Die Organisation bleibt in gesteuerten persönlichen oder Gruppenbereichen. Verschieben erhält die Asset-Identität; Kopieren erzeugt neue Assets und Versionen, die unveränderliche Blob-Bytes wiederverwenden. Jeder Zielkonflikt muss ausdrücklich abgelehnt, durch Umbenennen gelöst, überschrieben oder übersprungen werden. "
|
||||
"Ordner erstellen und Hochladen bleiben neben Neu laden im Kopf des Arbeitsbereichs. Nach der Auswahl von Dateien oder Ordnern öffnet Auswahl verwalten die Aktionen Verschieben, Kopieren, Umbenennen, Freigaben und Zugriffserklärungen. "
|
||||
"Löschen ist von der Organisation getrennt und benötigt weiterhin die vorhandene Bestätigung. Herunterladen und Archiv entpacken bleiben für passende Auswahlen direkt erreichbar. "
|
||||
"Verbindungen und Importe enthält die selteneren Import-, Synchronisierungs- und Connector-Bereichswerkzeuge; allein das Öffnen dieser Werkzeugauswahl importiert oder synchronisiert nichts."
|
||||
),
|
||||
},
|
||||
"files.workflow.find-and-download-files": {
|
||||
@@ -81,6 +101,7 @@ _TRANSLATIONS = {
|
||||
"title": "Files-Integrität, Recovery und ausfallsichere Connector-Transporte betreiben",
|
||||
"summary": "Datenbanknachweise, Blob-Chiffretexte und Encryption-Verwahrung als eine Recovery-Einheit sichern und jeden SDK-verwalteten Gegenpunkt binden.",
|
||||
"body": (
|
||||
"Die Archivprüfung setzt TAR-Eintrags-, Größen- und Expansionsgrenzen bei jedem Header durch, bevor die nächsten Nutzdaten durchlaufen werden. Pfade sind auf 4096 UTF-8-Bytes und 128 Komponenten begrenzt; abgeleitete Elternordner zählen zur konfigurierten Eintragsgrenze. Zu große Archive müssen vor einem erneuten Versuch verkleinert oder aufgeteilt werden; die Richtlinie wird nicht automatisch gelockert. "
|
||||
"Dauerhafter lokaler Speicher ist die betriebliche Basis. Files wird aus einem abgestimmten Datenbank-/Blob-Snapshot mit passenden Encryption-Tabellen und ursprünglichem Deployment-Hauptschlüssel wiederhergestellt; anschließend ist der begrenzte fortsetzbare Integritätsscan auszuführen und geschützter sowie ungeschützter Zugriff stichprobenartig zu prüfen. Scan- und Befundaktionen benötigen die angezeigte Revision. Geschützte Scans prüfen erst den Chiffretext, dann nach Entschlüsselung den semantischen Klartextnachweis. Unter PostgreSQL sichern lease-gebundene Core-Recovery-Absichten Objektauswirkungen ab; Abweichungen werden quarantänisiert und bleiben in Ops sichtbar. SQLite verwendet für Entwicklung eine prozesslokale Sperre und ist kein Produktions-Recovery-Profil; nach hartem Prozessverlust ist ein Integritätsscan erforderlich. Fehlende oder abweichende Blobs werden quarantänisiert, verwaiste Objekte vor einer ausdrücklich autorisierten Bereinigung zunächst nur gemeldet. Physische Vernichtung und Blob-Garbage-Collection prüfen ihre irreversiblen Wirkungen getrennt. S3-Schreibvorgänge nutzen bedingte Effekte und digestbasierte Vorwärts-Recovery; S3- und SMB-Transporte binden Wiederholungen, Umleitungen, Aliasse und erkannte Endpunkte und wenden die Richtlinie für private Netze vor jeder Verbindung erneut an. Fehlt eine verifizierbare Transportnaht, wird geschlossen abgebrochen. Destruktive Modulstilllegung entfernt Datenbanktabellen, nicht jedoch Blob-Objekte im Backend."
|
||||
),
|
||||
},
|
||||
@@ -99,10 +120,14 @@ _TRANSLATIONS = {
|
||||
),
|
||||
},
|
||||
"files.reference.shared-storage-profile": {
|
||||
"title": "Files mit einem gemeinsamen Speicherprofil betreiben",
|
||||
"summary": "Lokalen, hostweit gemeinsamen oder S3-basierten Speicher passend zur Laufzeittopologie wählen.",
|
||||
"title": "Files-Speicher und temporäre Archivverarbeitung betreiben",
|
||||
"summary": "Dauerhaften Speicher, privaten Archivzwischenspeicher und gemessenen Fortschritt auf die Worker-Topologie abstimmen.",
|
||||
"body": (
|
||||
"Core stellt das gemeinsame lokale/S3-Objektspeicher-Backend bereit; Files verantwortet Dateimetadaten und Objektschlüssel. Lokaler Speicher eignet sich für einen Laufzeitprozess, ein gemeinsames Host-Volume für Replikate auf demselben Host. Unabhängige Hosts benötigen einen ausdrücklich vertrauenswürdigen HTTPS-S3-kompatiblen Endpunkt. PostgreSQL, Objekte und Hauptschlüssel müssen auf denselben abgestimmten Recovery-Zeitpunkt wiederhergestellt werden."
|
||||
"Core stellt das gemeinsame lokale/S3-Objektspeicher-Backend bereit; Files verantwortet Dateimetadaten und Objektschlüssel. Lokaler Speicher eignet sich für einen Laufzeitprozess, ein gemeinsames Host-Volume für Replikate auf demselben Host. Unabhängige Hosts benötigen einen ausdrücklich vertrauenswürdigen HTTPS-S3-kompatiblen Endpunkt. PostgreSQL, Objekte und Hauptschlüssel müssen auf denselben abgestimmten Recovery-Zeitpunkt wiederhergestellt werden. "
|
||||
"Die Archivoberfläche fordert ausdrücklich einmalige Übertragung mit Zwischenspeicherung an: Erneute Passwortprüfung und Bestätigung verwenden eine an Mandant und Person gebundene Kopie, speichern jedoch niemals Passwörter. Temporäre Archiv- und Fortschrittsdateien haben Modus 0600 unter einem privaten Verzeichnis mit Modus 0700. FILE_ARCHIVE_WORK_ROOT ersetzt den Standard im temporären Betriebssystemverzeichnis, dessen Name von Dienst-UID und Hash des konfigurierten Speicherwurzelpfads abgeleitet wird. Alle Worker desselben Hosts müssen darauf zugreifen; unabhängige Hosts benötigen gemeinsamen POSIX-Speicher mit funktionierenden flock-Sperren oder feste Host-Zuordnung für Archiv- und Fortschrittsanfragen, auch bei dauerhaftem S3-Blob-Speicher. "
|
||||
"Standardgrenzen sind 2 GiB zwischengespeicherte Archive pro Arbeitsverzeichnis, vier Kopien pro Mandant/Person und 1.800 Sekunden ab der ersten Speicherung ohne Verlängerung. Abbruch- und Wechselanfragen sowie erfolgreicher Abschluss geben Kopien frei; verlassene abgelaufene Kopien werden bei späterer Archivverarbeitung bereinigt, nicht garantiert zu einem exakten Zeitpunkt ohne weitere Aktivität. Aktive POSIX-Leases schützen benutzte Kopien auch nach Ablauf vor Verdrängung. Kopien und Fortschrittsdaten sind keine fortsetzbaren Hintergrundaufträge. "
|
||||
"Eine optionale, aktuell gehaltene Systembibliothek libarchive beschleunigt klassisches ZIPCrypto mit gespeicherten oder Deflate-komprimierten Einträgen und ASCII- oder ausdrücklich als UTF-8 markierten Namen. Fehlende Bibliotheken, AES und nicht unterstützte Kodierungen/Formate wählen vor dem Dekodieren den bisherigen Python-Leser; native Fehler führen zum sicheren Abbruch. Beide Leser laufen in einem begrenzten Kindprozess mit vollständigen Header-, Pfad- und Grenzprüfungen; der native Pfad prüft CRC und Größe zusätzlich unabhängig. Nur die Zwischenspeicherschicht schreibt Dateien unter privaten servergenerierten numerischen Namen, niemals unter Archivpfaden. "
|
||||
"Die Verarbeitung speichert jeweils einen geprüften Eintrag. Wiederverwendung des Speicher-Clients innerhalb eines Archivs und gebündelte Antwortmetadaten vermeiden Zusatzarbeit, ohne Autorisierung zwischenzuspeichern oder dauerhafte Recovery-, Integritäts- und Commit-Prüfungen pro Datei zu entfernen. Fortschritt zeigt tatsächliche Datei-/Bytezähler und eine eigene Abschlussphase statt Zeitschätzungen oder vorzeitiger Erfolgsmeldung. Ein schnellerer Decoder verspricht keine entsprechende Beschleunigung des gesamten Imports oder von S3."
|
||||
),
|
||||
},
|
||||
"files.reference.generated-artifact-store": {
|
||||
@@ -118,15 +143,4 @@ _TRANSLATIONS = {
|
||||
def localize_documentation_topics(
|
||||
topics: Iterable[DocumentationTopic],
|
||||
) -> tuple[DocumentationTopic, ...]:
|
||||
localized: list[DocumentationTopic] = []
|
||||
for topic in topics:
|
||||
german = _TRANSLATIONS.get(topic.id)
|
||||
if german is None:
|
||||
localized.append(topic)
|
||||
continue
|
||||
translations = {
|
||||
locale: dict(value) for locale, value in topic.translations.items()
|
||||
}
|
||||
translations["de"] = {**translations.get("de", {}), **german}
|
||||
localized.append(replace(topic, translations=translations))
|
||||
return tuple(localized)
|
||||
return _localize_topics(topics, locale="de", translations=_TRANSLATIONS)
|
||||
|
||||
@@ -5,6 +5,19 @@ from __future__ import annotations
|
||||
from typing import Any
|
||||
|
||||
|
||||
_MANAGED_ARCHIVE_GERMAN = {
|
||||
"prerequisites": ["Das ausgewählte verwaltete Archiv ist lesbar und herunterladbar; Sie dürfen in den gewählten Zielbereich hochladen."],
|
||||
"steps": [
|
||||
"Wählen Sie ein unterstütztes verwaltetes Archiv und Archiv entpacken in der Aktionsleiste oder im Kontextmenü.",
|
||||
"Wählen Sie Zielbereich und Zielordner und fordern Sie die Archivvorschau an.",
|
||||
"Prüfen Sie Einträge, geben Sie gegebenenfalls das ZIP-Passwort ein und verifizieren Sie es; wählen Sie anschließend Dateien oder Ordner zum Entpacken aus.",
|
||||
"Bestätigen Sie die ausgewählten Einträge und lassen Sie den Dialog bis zum Abschluss geöffnet. Wählen Sie bei vorhandenen Zieldateien ein anderes Ziel.",
|
||||
],
|
||||
"outcome": "Ausgewählte Einträge sind neue verwaltete Dateien; das Quellarchiv bleibt unverändert.",
|
||||
"verification": "Prüfen Sie Zielpfade und Anzahl sowie die unveränderte Version des Quellarchivs.",
|
||||
}
|
||||
|
||||
|
||||
GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'files.assurance.process-and-release-readiness': {'outcome': 'Der Prozess oder die '
|
||||
'Veröffentlichung hat eine explizite '
|
||||
'Genehmigungsaufzeichnung, die an '
|
||||
@@ -355,10 +368,10 @@ GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'files.assurance.pr
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie den '
|
||||
'persönlichen oder Gruppenraum zum '
|
||||
'Organisieren aus.',
|
||||
'Erstellen Sie die erforderlichen Zielordner '
|
||||
'oder wählen Sie die Dateien und Ordner aus, '
|
||||
'um sie umzubenennen, zu verschieben oder zu '
|
||||
'kopieren.',
|
||||
'Verwenden Sie Ordner erstellen im Kopf des '
|
||||
'Arbeitsbereichs oder wählen Sie Dateien und '
|
||||
'Ordner aus und öffnen Sie Auswahl verwalten '
|
||||
'zum Umbenennen, Verschieben oder Kopieren.',
|
||||
'Wählen Sie das Ziel aus und lösen Sie jeden '
|
||||
'Zielkonflikt explizit.',
|
||||
'Wenden Sie die Operation an und öffnen Sie '
|
||||
@@ -456,3 +469,5 @@ GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'files.assurance.pr
|
||||
'widerrufen Sie die Gewährung und stellen '
|
||||
'Sie sicher, dass nur noch unabhängige '
|
||||
'Zugangspfade verbleiben.'}}
|
||||
|
||||
GERMAN_STRUCTURED_TRANSLATIONS["files.workflow.unpack-managed-archive"] = _MANAGED_ARCHIVE_GERMAN
|
||||
|
||||
@@ -37,7 +37,9 @@ from govoplan_core.core.modules import (
|
||||
ModuleManifest,
|
||||
NavItem,
|
||||
PermissionDefinition,
|
||||
ProductAvailabilityExplanation,
|
||||
ProductAreaContribution,
|
||||
ProductSurfaceContribution,
|
||||
QuickAccessTool,
|
||||
RoleTemplate,
|
||||
)
|
||||
@@ -459,7 +461,7 @@ def _dsar_provider(context: ModuleContext) -> object:
|
||||
manifest = ModuleManifest(
|
||||
id="files",
|
||||
name="Files",
|
||||
version="0.1.24",
|
||||
version="0.1.27",
|
||||
required_capabilities=(
|
||||
CAPABILITY_AUTH_PRINCIPAL_RESOLVER,
|
||||
CAPABILITY_AUTH_PERMISSION_EVALUATOR,
|
||||
@@ -606,6 +608,31 @@ manifest = ModuleManifest(
|
||||
order=30,
|
||||
),
|
||||
),
|
||||
product_surfaces=(
|
||||
ProductSurfaceContribution(
|
||||
id="records.files",
|
||||
module_id="files",
|
||||
label="i18n:govoplan-core.product_surface.files",
|
||||
description="i18n:govoplan-core.product_surface.files_description",
|
||||
icon="folder",
|
||||
entry_path="/documents",
|
||||
route_path="/files",
|
||||
surface_ids=("files.nav.files", "files.route.files"),
|
||||
presentations=("task", "reader"),
|
||||
search_source_ids=("files.objects",),
|
||||
help_context_ids=("files.list",),
|
||||
documentation_topic_ids=("files.quick-access-and-product-area",),
|
||||
required_any=("files:file:read",),
|
||||
order=10,
|
||||
unavailable=ProductAvailabilityExplanation(
|
||||
reason="authorization",
|
||||
title="i18n:govoplan-core.product_surface.unavailable",
|
||||
description="i18n:govoplan-core.product_surface.unavailable_description",
|
||||
resolution="i18n:govoplan-core.product_surface.unavailable_resolution",
|
||||
responsible_role="i18n:govoplan-core.access_administrator",
|
||||
),
|
||||
),
|
||||
),
|
||||
quick_access_tools=(
|
||||
QuickAccessTool(
|
||||
id="files.recent",
|
||||
@@ -625,6 +652,71 @@ manifest = ModuleManifest(
|
||||
),
|
||||
),
|
||||
documentation=localize_documentation_topics((
|
||||
DocumentationTopic(
|
||||
id="files.source-identity-provenance",
|
||||
title="Verify imported bytes and resolve source ambiguity",
|
||||
summary="Bind provenance to acquired bytes and use tenant/owner-scoped source identities. All caller and browse metadata is preserved solely under import_annotations, never flattened into acquisition evidence.",
|
||||
body="A supplied source_revision is an expected revision, not a label to assign to newer bytes. A stale or unavailable expected revision returns a conflict before persistence; refresh the provider listing and retry deliberately. Stored revisions come from the acquired object, and acquired_sha256 is computed from its actual bytes. Provider checksums are retained separately as provider_checksum_claims, not presented as locally verified hashes; caller and browse annotations remain in import_annotations and cannot replace acquisition evidence. S3 reads pin a version or condition on ETag; SMB denies write/delete sharing while reading and checks stable metadata; Seafile checks the source revision before and after acquisition. Remote provider revision claims are not cryptographic proof against a dishonest provider. Source lookup uses indexed canonical provenance within tenant and owner, never filename or equal content. Migration backfills the index in bounded batches without deleting, merging or selecting a winning copy. Multiple active files with one source identity cause an explicit conflict. Review the retained files and provenance before any authorized correction; copying, soft deletion and restoration preserve identity semantics. The index is non-unique because legitimate copies remain supported. It is not a new exactly-once concurrency guarantee. Owning ORM metadata writes refresh the index; maintenance bulk SQL must update the derived source hash consistently.",
|
||||
layer="available", documentation_types=("user", "admin"), audience=("file_user", "file_admin"), order=17,
|
||||
conditions=(DocumentationCondition(required_modules=("files",), any_scopes=("files:file:read", "files:file:upload", "files:file:admin")),),
|
||||
links=(DocumentationLink(label="Files", href="/files", kind="runtime"), DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository")),
|
||||
metadata={"kind": "reference", "related_topic_ids": ["files.governed-connectors-and-provenance", "files.workflow.import-managed-snapshot"]},
|
||||
translations={"de": {
|
||||
"title": "Importierte Bytes prüfen und Quellmehrdeutigkeit klären",
|
||||
"summary": "Herkunft an gelesene Bytes binden und nach Mandant und Eigentümer indizierte Quellidentitäten nutzen. Alle Aufrufer- und Vorschauangaben bleiben ausschließlich unter import_annotations erhalten, niemals als Erwerbsnachweise auf gleicher Ebene.",
|
||||
"body": "Eine übergebene source_revision ist eine erwartete Revision, kein Etikett für neuere Bytes. Eine veraltete oder nicht prüfbare Erwartung führt vor dem Speichern zum Konflikt; die Anbieterliste aktualisieren und bewusst erneut versuchen. Gespeicherte Revisionen stammen vom gelesenen Objekt; acquired_sha256 wird aus dessen tatsächlichen Bytes berechnet. Anbieterprüfsummen bleiben getrennt als provider_checksum_claims erhalten und gelten nicht als lokal verifizierte Hashes. Angaben des Aufrufers und der Vorschau bleiben in import_annotations und ersetzen keine Erwerbsnachweise. S3 bindet den Abruf an eine Version oder ETag-Bedingung; SMB verhindert Schreib-/Löschfreigabe während des Lesens und prüft stabile Metadaten; Seafile prüft die Revision vor und nach dem Abruf. Revisionsangaben eines unehrlichen Anbieters sind kein kryptografischer Beweis. Die Quellsuche verwendet indizierte kanonische Herkunft innerhalb von Mandant und Eigentümer, niemals Dateiname oder gleichen Inhalt. Die Migration ergänzt den Index stapelweise und löscht, vereinigt oder bevorzugt keine Kopie. Mehrere aktive Dateien mit derselben Quellidentität führen zum ausdrücklichen Konflikt. Vor berechtigter Korrektur Dateien und Herkunft prüfen; Kopieren, Papierkorb und Wiederherstellen erhalten die Identitätsregeln. Der Index bleibt wegen zulässiger Kopien nicht eindeutig und verspricht keine neue Exactly-once-Garantie bei Nebenläufigkeit. ORM-Metadatenänderungen aktualisieren ihn; administratives Bulk-SQL muss den abgeleiteten Quellhash konsistent mitpflegen.",
|
||||
}},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="files.archive-worker-limits",
|
||||
title="Bound archive inspection and extraction work",
|
||||
summary="ZIP and TAR parsers run in disposable resource-limited processes while Files retains authorization and storage.",
|
||||
body=(
|
||||
"Archive preview and password verification run in a fresh child with 120 seconds wall time, 90 CPU seconds and 512 MiB address space. "
|
||||
"Confirmation uses one child for the entire archive, including metadata validation and native or Python decoding, with 600 seconds wall time and 300 CPU seconds. "
|
||||
"Its address-space ceiling is the larger of 512 MiB or three times the configured member limit plus 128 MiB; the supported member limit is positive and at most 2 GiB. "
|
||||
"The ordinary defaults remain 50 MiB per member, 250 MiB compressed request, 10,000 entries, 2 GiB expanded and 100:1 expansion. "
|
||||
"Core's GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY admits archive snapshot creation and processing together with other isolated module work; its default is one slot per API/worker process, configurable from one to 16. "
|
||||
"Budget every replica separately. A busy request fails before copying its source and must be retried explicitly; missing process controls, CPU/memory/time/output limits and invalid staging records fail closed without inline parsing. "
|
||||
"Provide OS-temporary disk capacity for one additional compressed source snapshot and one extracted member per admitted archive, beyond the upload-once staging quota. "
|
||||
"Private 0700 directories contain only server-generated 0600 filenames. The child waits for parent storage and deletion of each verified member before decoding the next. A private FIFO carries one eight-byte sequence acknowledgement; completed members wake the parent without a fixed per-file sleep. Wake bytes are capped at 32 KiB; larger non-default entry counts use ordinary progress polling thereafter. "
|
||||
"Member paths, selection, regular-file identity, size, digest and monotonic file/byte progress are rechecked by the parent. Passwords are request-only and not written to staging. "
|
||||
"Metadata transport is bounded to 64 MiB and status records to 16 KiB; the child file-size limit is the greater of the member limit or 16 KiB, with actual member and cumulative limits checked independently. "
|
||||
"Slow destination storage consumes the confirmation wall-time budget. Failures use existing transaction rollback/recovery; successful decoding alone is not a committed import, and confirmation is never retried automatically. "
|
||||
"The child has no inherited application credentials or SQL session. These are process resource controls, not a filesystem/network sandbox or an aggregate cgroup memory guarantee."
|
||||
),
|
||||
translations={"de": {
|
||||
"title": "Archivprüfung und Entpacken begrenzen",
|
||||
"summary": "ZIP- und TAR-Parser laufen in kurzlebigen ressourcenbegrenzten Prozessen; Files behält Autorisierung und Speicherung.",
|
||||
"body": (
|
||||
"Archivvorschau und Passwortprüfung laufen in einem frischen Kindprozess mit 120 Sekunden Laufzeit, 90 CPU-Sekunden und 512 MiB Adressraum. "
|
||||
"Die Bestätigung verwendet einen Kindprozess für das gesamte Archiv einschließlich Metadatenprüfung und nativem oder Python-Dekodieren, mit 600 Sekunden Laufzeit und 300 CPU-Sekunden. "
|
||||
"Seine Adressraumgrenze ist das Maximum aus 512 MiB und dem Dreifachen der konfigurierten Dateigrenze plus 128 MiB; die unterstützte Dateigrenze ist positiv und höchstens 2 GiB. "
|
||||
"Die üblichen Standardgrenzen bleiben 50 MiB pro Datei, 250 MiB komprimierte Anfrage, 10.000 Einträge, 2 GiB entpackte Daten und 100:1 Expansion. "
|
||||
"GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY von Core begrenzt Quellkopie und Verarbeitung gemeinsam mit isolierter Arbeit anderer Module: standardmäßig ein Platz pro API-/Worker-Prozess, konfigurierbar von eins bis 16. "
|
||||
"Jedes Replikat separat budgetieren. Bei Auslastung wird vor der Quellkopie abgelehnt; ein neuer Versuch muss ausdrücklich erfolgen. Fehlende Prozesskontrollen, CPU-/Speicher-/Zeit-/Ausgabegrenzen und ungültige Zwischenspeicherdaten führen zum sicheren Abbruch ohne Parsing im Elternprozess. "
|
||||
"Zusätzlich zur Quote für einmalig hochgeladene Archive benötigt jedes zugelassene Archiv im temporären Betriebssystemverzeichnis Platz für eine weitere komprimierte Quellkopie und eine entpackte Datei. "
|
||||
"Private Verzeichnisse mit Modus 0700 enthalten ausschließlich servergenerierte Dateinamen mit Modus 0600. Der Kindprozess wartet nach jeder geprüften Datei auf Speicherung und Löschung durch den Elternprozess, bevor er die nächste dekodiert. Ein privater FIFO überträgt eine acht Byte lange Sequenzbestätigung; fertige Dateien wecken den Elternprozess ohne feste Pause pro Datei. Wecksignale sind auf 32 KiB begrenzt; größere abweichend konfigurierte Eintragszahlen nutzen danach die gewöhnliche Fortschrittsabfrage. "
|
||||
"Dieser prüft Pfade, Auswahl, reguläre Dateiidentität, Größe, Prüfsumme und monotonen Datei-/Bytefortschritt erneut. Passwörter bleiben auf die Anfrage beschränkt und werden nicht zwischengespeichert. "
|
||||
"Der Metadatentransport ist auf 64 MiB, Statusdatensätze auf 16 KiB begrenzt; die Dateigrößengrenze des Kindprozesses ist das Maximum aus Dateilimit und 16 KiB. Tatsächliche Einzel- und Gesamtgrößen werden unabhängig kontrolliert. "
|
||||
"Langsamer Zielspeicher zählt zur Laufzeit der Bestätigung. Fehler nutzen bestehendes Transaktions-Rollback und Recovery; erfolgreiches Dekodieren ist kein bestätigter Import und die Bestätigung wird nie automatisch wiederholt. "
|
||||
"Der Kindprozess erbt weder Anwendungszugangsdaten noch SQL-Sitzungen. Die Kontrollen begrenzen Prozessressourcen, sind aber keine Dateisystem-/Netzwerk-Sandbox und keine aggregierte cgroup-Speichergarantie."
|
||||
),
|
||||
}},
|
||||
layer="available",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("file_user", "file_admin", "operator"),
|
||||
configuration_keys=("GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY",),
|
||||
conditions=(DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
any_scopes=("files:file:upload", "files:file:admin", "system:settings:read"),
|
||||
),),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||
),
|
||||
order=28,
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="files.tabular-content",
|
||||
title="Use managed CSV and XLSX versions as governed data sources",
|
||||
@@ -771,12 +863,17 @@ manifest = ModuleManifest(
|
||||
title="Files in Records and documents and Quick Access",
|
||||
summary="Use managed files in the Records and documents area and keep a compact file surface beside current work.",
|
||||
body=(
|
||||
"Files contributes its authorized workspace to Records and documents. When Quick Access is enabled, the owner-rendered "
|
||||
"Files contributes its authorized workspace to the stable Files destination at /documents in Records and documents. "
|
||||
"The owner route /files remains available through All available tools and as a compatible deep link. When Quick Access is enabled, the owner-rendered "
|
||||
"selector shows at most seven recently changed files and can return one exact file-version reference to the current task. "
|
||||
"Accounts with upload permission can launch the full Files upload dialog into a freshly reauthorized managed space. Opening "
|
||||
"either path causes Files to re-run its own provider, space, folder, object, and scope checks; completion and cancellation are "
|
||||
"explicit, and the full Files workspace remains available as a deep-link fallback. View and rail settings may recommend or "
|
||||
"focus this tool, but do not bypass file ownership, shares, connector policy, integrity gates, or purpose-aware access."
|
||||
"focus this tool, but do not bypass file ownership, shares, connector policy, integrity gates, or purpose-aware access. "
|
||||
"The full workspace keeps Reload, Create folder, and Upload at the top right, with Upload as the primary action. "
|
||||
"Reload refreshes the current folder and active filters; it never imports or synchronizes remote content. Failed reloads retain loaded data and report the failure. "
|
||||
"Connections and imports groups explicit synchronization and linked-space tools. The selection bar keeps Download and, for an archive, Unpack archive; "
|
||||
"Manage selection groups organization, sharing, access explanations, and a separate confirmed deletion action. Permission and connector restrictions remain enforced per action."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user", "admin"),
|
||||
@@ -801,12 +898,17 @@ manifest = ModuleManifest(
|
||||
"title": "Dateien in Akten und Dokumente sowie im Schnellzugriff",
|
||||
"summary": "Verwaltete Dateien im Produktbereich Akten und Dokumente und optional neben der aktuellen Arbeit verwenden.",
|
||||
"body": (
|
||||
"Files ordnet seinen berechtigten Arbeitsbereich Akten und Dokumente zu. Ist der Schnellzugriff aktiviert, erscheint "
|
||||
"Files ordnet seinen berechtigten Arbeitsbereich dem stabilen Produktziel Dateien unter /documents in Akten und Dokumente zu. "
|
||||
"Der Eigentümerpfad /files bleibt unter Alle verfügbaren Werkzeuge und als kompatibler Direktlink erreichbar. Ist der Schnellzugriff aktiviert, erscheint "
|
||||
"die vom Modul gerenderte Auswahl von höchstens sieben zuletzt geänderten berechtigten Dateien rechts neben der aktuellen "
|
||||
"Seite und kann einen exakten Dateiversionsverweis zurückgeben. Mit Upload-Berechtigung lässt sich der vollständige "
|
||||
"Upload-Dialog in einem erneut berechtigungsgeprüften verwalteten Bereich öffnen. Files prüft Anbieter, Bereich, Ordner, "
|
||||
"Objekt und Berechtigung bei jedem Pfad erneut. Abschluss und Abbruch sind ausdrücklich; Eigentum, Freigaben, Connector-"
|
||||
"Richtlinien, Integritätsprüfungen und zweckgebundener Zugriff bleiben maßgeblich."
|
||||
"Richtlinien, Integritätsprüfungen und zweckgebundener Zugriff bleiben maßgeblich. "
|
||||
"Der vollständige Arbeitsbereich zeigt oben rechts Neu laden, Ordner erstellen und die Hauptaktion Hochladen. "
|
||||
"Neu laden aktualisiert den aktuellen Ordner mit aktiven Filtern; es importiert oder synchronisiert niemals entfernte Inhalte. Bei Fehlern bleiben geladene Daten mit Fehlermeldung erhalten. "
|
||||
"Verbindungen und Importe bündelt ausdrückliche Synchronisierung und verknüpfte Dateibereiche. Bei der Auswahl bleiben Herunterladen und für Archive Archiv entpacken direkt sichtbar. "
|
||||
"Auswahl verwalten bündelt Organisation, Freigaben, Zugriffserklärungen und getrennt bestätigtes Löschen. Berechtigungen und Connector-Beschränkungen gelten weiterhin je Aktion."
|
||||
),
|
||||
}
|
||||
},
|
||||
@@ -957,7 +1059,10 @@ manifest = ModuleManifest(
|
||||
summary="Create folders and rename, move, or copy accessible managed content with explicit conflict handling.",
|
||||
body=(
|
||||
"Organization stays inside governed personal or group spaces. Moves preserve the asset identity, while copies create new assets and versions that reuse immutable blob bytes. "
|
||||
"Every target conflict must be rejected, renamed, overwritten, or skipped explicitly."
|
||||
"Every target conflict must be rejected, renamed, overwritten, or skipped explicitly. "
|
||||
"Create folder and Upload remain beside Reload in the workspace header. Select files or folders and open Manage selection for Move, Copy, Rename, sharing, or access explanations. "
|
||||
"Deletion is separated from organization and still requires its existing confirmation. Download and Unpack archive stay directly available for applicable selections. "
|
||||
"Connections and imports holds the less frequent import, synchronization, and connector-space tools; opening this tool chooser performs no import or synchronization."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
@@ -994,7 +1099,7 @@ manifest = ModuleManifest(
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and select the personal or group space to organize.",
|
||||
"Create the required destination folders or select the files and folders to rename, move, or copy.",
|
||||
"Use Create folder in the workspace header, or select files and folders and open Manage selection to rename, move, or copy them.",
|
||||
"Choose the destination and resolve each target conflict explicitly.",
|
||||
"Apply the operation and reopen the destination folder.",
|
||||
],
|
||||
@@ -1007,6 +1112,48 @@ manifest = ModuleManifest(
|
||||
],
|
||||
},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="files.workflow.unpack-managed-archive",
|
||||
title="Unpack an already uploaded archive",
|
||||
summary="Select a managed ZIP or TAR archive and reuse the upload preview, password and selective extraction workflow without downloading and uploading it again.",
|
||||
body=(
|
||||
"Select exactly one ZIP, TAR, TAR.GZ, TAR.BZ2 or TAR.XZ file in a managed space, then choose Unpack archive in the action bar or context menu. "
|
||||
"Choose an accessible personal or group destination and request a preview; no files are created yet. The ordinary archive dialog reviews entries, verifies any ZIP password and selects files or folders. "
|
||||
"Confirmation reads the managed source on the server and creates new destination files. The source archive, metadata and version remain unchanged; destination collisions are rejected and never overwrite the source. "
|
||||
"Read, download and upload permissions are all required. Ownership, current shares, tenant, source version, connector restrictions, integrity, quarantine and optional envelope-decryption availability are checked again at confirmation. "
|
||||
"The expiring preview is bound to the exact source/version, actor and destination. Changed or inaccessible sources require a fresh preview; the server never silently switches to another version. "
|
||||
"The same compressed-size, expanded-size, per-member, entry-count, expansion-ratio and safe-path limits apply as for uploaded archives. Passwords remain request-only. "
|
||||
"Extraction creates ordinary destination files, just as archive upload does; source archive password protection or a source storage envelope is not automatically copied to members. "
|
||||
"The shared blurred loading overlay covers the dialog during inspection and extraction, without the envelope animation. "
|
||||
"Selected file and byte totals are shown immediately; live progress displays measured server counters, not an estimated timer. "
|
||||
"Upload transfer, inspection, extraction/storage and final commit are distinct phases; transferred or processed bytes alone never mean a completed transaction. "
|
||||
"An unknown phase is indeterminate. Close, selection and destination controls remain blocked while work runs; errors restore the unchanged review dialog for inspection. "
|
||||
"Extraction is synchronous: leave the busy dialog open until completion. Remote connector-space entries must first be synchronized to managed storage."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user", "admin"),
|
||||
audience=("file_user", "file_manager", "tenant_admin", "system_admin"),
|
||||
order=42,
|
||||
conditions=(DocumentationCondition(required_modules=("files",), required_scopes=("files:file:read", "files:file:download", "files:file:upload")),),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||
),
|
||||
metadata={
|
||||
"kind": "workflow", "route": "/files", "screen": "Files",
|
||||
"help_contexts": ["files.list", "files.archive-extract"],
|
||||
"prerequisites": ["The selected managed archive is readable and downloadable, and you may upload into the selected destination."],
|
||||
"steps": [
|
||||
"Select one supported managed archive and choose Unpack archive in the action bar or context menu.",
|
||||
"Choose the destination space and folder, then request Preview archive.",
|
||||
"Review entries, enter and verify the ZIP password if required, and select the files or folders to extract.",
|
||||
"Confirm Import selected and keep the dialog open until extraction finishes; choose a different destination if files already exist.",
|
||||
],
|
||||
"outcome": "Selected members are new managed files while the source archive remains unchanged.",
|
||||
"verification": "Check destination paths and counts, and confirm the source archive still has the same version.",
|
||||
"related_topic_ids": ["files.workflow.upload-and-unpack-zip", "files.workflow.find-and-download-files", "files.reference.shared-storage-profile"],
|
||||
},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="files.workflow.find-and-download-files",
|
||||
title="Find and download managed files",
|
||||
@@ -1440,6 +1587,7 @@ manifest = ModuleManifest(
|
||||
title="Operate Files integrity, recovery, and connector transport safety",
|
||||
summary="Back up database evidence, blob ciphertext, and Encryption custody as one recovery unit, and pin every SDK-managed connector peer.",
|
||||
body=(
|
||||
"Archive inspection enforces per-header TAR entry, size and expansion-ratio limits before traversing the next payload. Paths are capped at 4096 UTF-8 bytes and 128 components; derived parent folders count toward the configured entry limit. Reduce or split oversized archives before retrying; no automatic policy relaxation is performed. "
|
||||
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the matching Encryption tables and original deployment master key, then run the bounded resumable integrity scan from Administration and verify representative protected and unprotected access paths. Each scan batch and finding action requires the revision shown to the operator, so a stale screen cannot recheck or delete after concurrent reconciliation. Protected scans verify stored ciphertext before decryption and then verify plaintext semantic evidence. On PostgreSQL, managed blob creation/repair and applied orphan cleanup commit lease-fenced Core recovery intent before object effects; success, compensation, and forward completion require independent database and object checks, while mismatch is quarantined and unresolved work remains visible in Ops. Development SQLite instead records blob intent in the caller transaction to avoid a second-writer deadlock, uses a process-local fence, verifies after commit, and reconstructs durable compensation evidence after handled rollback. A hard process loss before the SQLite caller commits can therefore leave an unrecorded object; SQLite is not a production recovery profile and operators must run an integrity scan after such a loss. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. "
|
||||
"Hard purge records irreversible intent before deleting database evidence, and blob garbage collection separately rechecks references under the shared blob lease before deleting a managed object. S3 connector write-back records digest-only forward-recovery intent before a conditional provider effect; request and content markers prove success, while mismatches remain visible in Ops and fence later writers. S3 connector pools pin every retry, redirect, discovered endpoint, and provider alias while retaining the configured TLS authority; outbound proxies and ambient credential discovery are disabled. SMB initial connections, reconnects, aliases, and DFS referrals use a Files-owned pinned transport and cache. Both apply the deployment private-network policy immediately before each socket opens and fail closed if an SDK no longer exposes the verified transport seam. Installer-owned Garage storage is supported only at the exact deployment service endpoint with its explicit trust marker. Destructive module retirement drops database tables but does not remove backend blob objects."
|
||||
),
|
||||
@@ -1686,9 +1834,15 @@ manifest = ModuleManifest(
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="files.reference.shared-storage-profile",
|
||||
title="Operate Files with shared object storage",
|
||||
summary="Choose local, host-shared, or S3-backed storage consistently with the runtime topology.",
|
||||
body="Core supplies the common local/S3 object-storage backend while Files owns file metadata and object-key semantics. Local storage is valid for one runtime process; a shared host volume supports same-host replicas; independent hosts require an explicitly trusted HTTPS S3-compatible endpoint. Restore PostgreSQL, objects, and the master key to one coordinated recovery point.",
|
||||
title="Operate Files storage and temporary archive work",
|
||||
summary="Align durable storage, private archive staging and measured progress with the worker topology.",
|
||||
body=(
|
||||
"Core supplies the common local/S3 object-storage backend while Files owns file metadata and object-key semantics. Local storage is valid for one runtime process; a shared host volume supports same-host replicas; independent hosts require an explicitly trusted HTTPS S3-compatible endpoint. Restore PostgreSQL, objects, and the master key to one coordinated recovery point. "
|
||||
"The archive UI opts into upload-once staging: password repreview and confirmation reuse a tenant/user-bound copy, but never persist passwords. Temporary archive and progress files use mode 0600 below a private 0700 directory. FILE_ARCHIVE_WORK_ROOT overrides the default OS-temporary directory derived from the service uid and configured storage-root hash. All same-host workers must share it; independent hosts need shared POSIX storage with working flock locks or sticky routing for archive and progress requests, even when durable blobs use S3. "
|
||||
"Defaults are 2 GiB of staged archives per work root, four stages per tenant/user, and 1,800 seconds from initial staging without renewal. Cancel/replacement requests and successful confirmation release copies; abandoned expired copies are cleaned opportunistically by later archive work, not on an exact idle deadline. POSIX active leases prevent eviction of an in-use copy even after its TTL; stages and progress are not resumable background jobs. "
|
||||
"A maintained optional system libarchive accelerates classic stored/deflated ZIPCrypto decoding for ASCII or UTF-8-flagged names. Missing libraries, AES and unsupported encodings/formats select the original Python reader before decoding; native errors fail closed. Both readers run in a bounded child and retain full header/path/limit checks; the native path independently verifies CRC and size. Only the staging wrapper writes files, using private server-generated numeric names, never archive paths. "
|
||||
"Extraction stores one verified member at a time. Archive-local backend-client reuse and batched response metadata reduce avoidable work without caching authorization or removing durable per-file recovery, integrity or commit verification. Progress reports actual file/byte counters and a separate finalization phase, never a timed estimate or an early success. Native decoder speed does not promise an equivalent whole-import or S3 improvement."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("admin",),
|
||||
audience=("file_admin", "operator", "system_admin"),
|
||||
@@ -1716,6 +1870,10 @@ manifest = ModuleManifest(
|
||||
"FILE_STORAGE_S3_ENDPOINT_URL",
|
||||
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
|
||||
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
|
||||
"FILE_ARCHIVE_WORK_ROOT",
|
||||
"FILE_ARCHIVE_STAGED_MAX_BYTES",
|
||||
"FILE_ARCHIVE_STAGED_PER_ACTOR",
|
||||
"FILE_ARCHIVE_PREVIEW_TTL_SECONDS",
|
||||
),
|
||||
metadata={
|
||||
"kind": "reference",
|
||||
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
"""Index source identities without merging or discarding legacy copies.
|
||||
|
||||
Revision ID: a2b3c4d5e701
|
||||
Revises: a2b3c4d5e6f8, a2b3c4d5e6f9
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "a2b3c4d5e701"
|
||||
down_revision = ("a2b3c4d5e6f8", "a2b3c4d5e6f9")
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _digest(metadata: object) -> str | None:
|
||||
# Frozen v1 identity semantics: never import evolving application code.
|
||||
provenance = (
|
||||
metadata.get("source_provenance") if isinstance(metadata, dict) else None
|
||||
)
|
||||
if not isinstance(provenance, dict):
|
||||
return None
|
||||
|
||||
def clean(value: object) -> str | None:
|
||||
return (str(value).strip() or None) if value is not None else None
|
||||
|
||||
connector = clean(provenance.get("connector_id"))
|
||||
provider = clean(provenance.get("provider"))
|
||||
external = clean(provenance.get("external_id"))
|
||||
path = clean(provenance.get("external_path"))
|
||||
extra = (
|
||||
provenance.get("metadata")
|
||||
if isinstance(provenance.get("metadata"), dict)
|
||||
else {}
|
||||
)
|
||||
if connector and external:
|
||||
identity = ("external_id", connector, provider, external)
|
||||
elif connector and path:
|
||||
identity = (
|
||||
"external_path",
|
||||
connector,
|
||||
provider,
|
||||
clean(
|
||||
extra.get("library_id") or extra.get("profile_id") or extra.get("share")
|
||||
),
|
||||
path,
|
||||
)
|
||||
else:
|
||||
return None
|
||||
return hashlib.sha256(
|
||||
json.dumps(identity, ensure_ascii=True, separators=(",", ":")).encode("ascii")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"file_assets", sa.Column("source_identity_hash", sa.String(64), nullable=True)
|
||||
)
|
||||
table = sa.table(
|
||||
"file_assets",
|
||||
sa.column("id", sa.String),
|
||||
sa.column("metadata", sa.JSON),
|
||||
sa.column("source_identity_hash", sa.String),
|
||||
)
|
||||
connection = op.get_bind()
|
||||
cursor = None
|
||||
while True:
|
||||
statement = (
|
||||
sa.select(table.c.id, table.c.metadata).order_by(table.c.id).limit(500)
|
||||
)
|
||||
if cursor is not None:
|
||||
statement = statement.where(table.c.id > cursor)
|
||||
rows = connection.execute(statement).all()
|
||||
if not rows:
|
||||
break
|
||||
for row in rows:
|
||||
digest = _digest(row.metadata)
|
||||
if digest is not None:
|
||||
connection.execute(
|
||||
table.update()
|
||||
.where(table.c.id == row.id)
|
||||
.values(source_identity_hash=digest)
|
||||
)
|
||||
cursor = rows[-1].id
|
||||
# Deliberately non-unique: retained copies/legacy duplicates survive intact.
|
||||
# Runtime sync reports ambiguity instead of selecting a newest/winning copy.
|
||||
for owner in ("user", "group"):
|
||||
op.create_index(
|
||||
f"ix_file_assets_{owner}_source",
|
||||
"file_assets",
|
||||
["tenant_id", "owner_type", f"owner_{owner}_id", "source_identity_hash"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for owner in ("user", "group"):
|
||||
op.drop_index(f"ix_file_assets_{owner}_source", table_name="file_assets")
|
||||
with op.batch_alter_table("file_assets") as batch:
|
||||
batch.drop_column("source_identity_hash")
|
||||
@@ -1,5 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
@@ -62,7 +64,8 @@ from govoplan_files.backend.storage.connector_credential_store import (
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_imports import read_connector_file
|
||||
from govoplan_files.backend.storage.connector_imports import ConnectorRevisionConflict, read_connector_file
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
@@ -406,6 +409,8 @@ def _ensure_campaign_file_access(
|
||||
|
||||
def _http_error(exc: Exception, *, not_found: bool = False) -> HTTPException:
|
||||
code = status.HTTP_404_NOT_FOUND if not_found else status.HTTP_400_BAD_REQUEST
|
||||
if isinstance(exc, FileSourceConflict):
|
||||
code = status.HTTP_409_CONFLICT
|
||||
return HTTPException(status_code=code, detail=str(exc))
|
||||
|
||||
|
||||
@@ -860,12 +865,23 @@ def _download_connector_payload(
|
||||
path=source_path,
|
||||
max_bytes=settings.file_upload_max_bytes,
|
||||
)
|
||||
expected = str(payload.source_revision or "").strip()
|
||||
observed = {downloaded.revision}
|
||||
if profile.provider == "s3":
|
||||
observed.add(downloaded.metadata.get("etag"))
|
||||
if expected and expected not in observed:
|
||||
raise ConnectorRevisionConflict("Connector source revision changed or is unavailable; refresh the source before importing")
|
||||
provider_metadata = dict(downloaded.metadata)
|
||||
checksum_claims = {key: provider_metadata.pop(key) for key in tuple(provider_metadata) if key.startswith("checksum_")}
|
||||
provenance_metadata = {
|
||||
**provider_metadata,
|
||||
"profile_id": profile.id,
|
||||
"library_id": payload.library_id,
|
||||
"library_path": source_path,
|
||||
**downloaded.metadata,
|
||||
**payload.metadata,
|
||||
"size": len(downloaded.data),
|
||||
"acquired_sha256": hashlib.sha256(downloaded.data).hexdigest(),
|
||||
"provider_checksum_claims": checksum_claims,
|
||||
"import_annotations": dict(payload.metadata),
|
||||
}
|
||||
metadata = source_metadata(
|
||||
source_provenance={
|
||||
@@ -878,7 +894,7 @@ def _download_connector_payload(
|
||||
"external_url": downloaded.external_url,
|
||||
"metadata": provenance_metadata,
|
||||
},
|
||||
source_revision=payload.source_revision or downloaded.revision,
|
||||
source_revision=downloaded.revision,
|
||||
)
|
||||
return source_path, downloaded, metadata or {}
|
||||
|
||||
@@ -1150,6 +1166,10 @@ def _loaded_asset_response(
|
||||
created_at=asset.created_at.isoformat(),
|
||||
updated_at=asset.updated_at.isoformat(),
|
||||
deleted_at=asset.deleted_at.isoformat() if asset.deleted_at else None,
|
||||
retained_until=asset.retained_until.isoformat() if asset.retained_until else None,
|
||||
legal_hold=asset.legal_hold,
|
||||
lifecycle_revision=asset.lifecycle_revision,
|
||||
lifecycle_reason=asset.lifecycle_reason,
|
||||
audit_relevant=asset.id in sent_asset_ids,
|
||||
metadata=metadata,
|
||||
source_provenance=source_provenance_from_metadata(metadata),
|
||||
|
||||
@@ -19,6 +19,7 @@ from govoplan_files.backend.routes.shares import router as shares_router
|
||||
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||
from govoplan_files.backend.routes.managed_archives import router as managed_archives_router
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -30,6 +31,7 @@ for workflow_router in (
|
||||
lifecycle_router,
|
||||
listing_router,
|
||||
uploads_router,
|
||||
managed_archives_router,
|
||||
connector_settings_router,
|
||||
connector_io_router,
|
||||
connector_profiles_router,
|
||||
|
||||
@@ -28,6 +28,7 @@ from govoplan_files.backend.storage.connector_browse import (
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
from govoplan_files.backend.storage.connector_imports import (
|
||||
ConnectorImportError,
|
||||
ConnectorImportUnsupported,
|
||||
@@ -250,7 +251,7 @@ def sync_connector_space_folder(
|
||||
if detail.startswith("Skipped upload target:"):
|
||||
action = "skipped"
|
||||
counts["skipped"] += 1
|
||||
elif detail.startswith("Target file already exists:"):
|
||||
elif isinstance(exc, FileSourceConflict) or detail.startswith("Target file already exists:"):
|
||||
action = "conflict"
|
||||
counts["conflicts"] += 1
|
||||
else:
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Extract an existing managed archive through the ordinary preview pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from io import BytesIO
|
||||
from collections.abc import Mapping
|
||||
import json
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, UploadFile
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, has_scope, require_scope
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_core.security.secrets import (
|
||||
TransientPayloadError,
|
||||
open_transient_payload,
|
||||
seal_transient_payload,
|
||||
)
|
||||
from govoplan_files.backend.route_support import (
|
||||
_connector_policy_error,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.schemas import ArchivePreviewResponse, FileUploadResponse
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.archives import archive_format_for_filename
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
ConnectorPolicyDenied,
|
||||
ensure_connector_policy_allows,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
effective_connector_policy_sources,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||
from govoplan_files.backend.routes.uploads import (
|
||||
confirm_archive_upload,
|
||||
preview_archive_upload,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
_PURPOSE = "files.managed-archive-preview.v1"
|
||||
|
||||
|
||||
class ManagedArchivePreviewRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
source_version_id: str = Field(min_length=1, max_length=128)
|
||||
owner_type: Literal["user", "group"] = "user"
|
||||
owner_id: str | None = Field(default=None, max_length=128)
|
||||
path: str = Field(default="", max_length=4096)
|
||||
password: str | None = Field(default=None, max_length=4096)
|
||||
|
||||
|
||||
class ManagedArchiveConfirmRequest(ManagedArchivePreviewRequest):
|
||||
preview_token: str = Field(min_length=1, max_length=32768)
|
||||
selected_paths: list[str] = Field(min_length=1, max_length=10_000)
|
||||
operation_id: str | None = Field(default=None, min_length=36, max_length=36)
|
||||
|
||||
|
||||
def _resolve_source(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
file_id: str,
|
||||
payload: ManagedArchivePreviewRequest,
|
||||
):
|
||||
for scope in ("files:file:read", "files:file:download"):
|
||||
if not has_scope(principal, scope):
|
||||
raise HTTPException(status_code=403, detail=f"Missing scope: {scope}")
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if asset.current_version_id != payload.source_version_id:
|
||||
raise FileStorageError(
|
||||
"The managed archive version changed; reload the file and preview it again"
|
||||
)
|
||||
archive_format_for_filename(asset.filename)
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
if (
|
||||
version.file_asset_id != asset.id
|
||||
or version.tenant_id != principal.tenant_id
|
||||
or blob.tenant_id != principal.tenant_id
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Managed archive version or content does not belong to this source"
|
||||
)
|
||||
if blob.size_bytes > settings.file_upload_zip_max_bytes:
|
||||
raise FileStorageError(
|
||||
"Managed archive exceeds the configured archive size limit"
|
||||
)
|
||||
provenance = source_provenance_from_metadata(asset.metadata_ or {})
|
||||
if provenance:
|
||||
source_owner = (
|
||||
asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id
|
||||
)
|
||||
sources = []
|
||||
for scope_type, scope_id in dict.fromkeys(
|
||||
(
|
||||
(asset.owner_type, source_owner),
|
||||
(payload.owner_type, target_owner),
|
||||
("user", principal.user.id),
|
||||
)
|
||||
):
|
||||
sources.extend(
|
||||
effective_connector_policy_sources(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
ensure_connector_policy_allows(
|
||||
ConnectorAccessRequest.from_provenance(provenance, operation="import"),
|
||||
sources,
|
||||
)
|
||||
# This is the same integrity/quarantine and envelope-decryption path used
|
||||
# by authorized managed-file downloads; never read raw backend bytes here.
|
||||
data, version, _blob = read_asset_bytes(session, asset)
|
||||
if version.id != payload.source_version_id:
|
||||
raise FileStorageError(
|
||||
"The managed archive version changed during the verified read; reload and preview again"
|
||||
)
|
||||
if len(data) > settings.file_upload_zip_max_bytes:
|
||||
raise FileStorageError(
|
||||
"Managed archive exceeds the configured archive size limit"
|
||||
)
|
||||
return asset, version, data, target_owner, provenance
|
||||
|
||||
|
||||
@router.post("/{file_id}/archive-preview", response_model=ArchivePreviewResponse)
|
||||
def preview_managed_archive(
|
||||
file_id: str,
|
||||
payload: ManagedArchivePreviewRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
asset, version, data, owner, _provenance = _resolve_source(
|
||||
session, principal, file_id, payload
|
||||
)
|
||||
with BytesIO(data) as source:
|
||||
preview = preview_archive_upload(
|
||||
file=UploadFile(file=source, filename=asset.filename),
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=owner,
|
||||
path=payload.path,
|
||||
campaign_id=None,
|
||||
password=payload.password,
|
||||
principal=principal,
|
||||
)
|
||||
preview.preview_token = seal_transient_payload(
|
||||
{
|
||||
"purpose": _PURPOSE,
|
||||
"file_id": asset.id,
|
||||
"version_id": version.id,
|
||||
"tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id,
|
||||
"owner_type": payload.owner_type,
|
||||
"owner_id": owner,
|
||||
"path": normalize_folder(payload.path),
|
||||
"upload_preview_token": preview.preview_token,
|
||||
}
|
||||
)
|
||||
return preview
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/{file_id}/archive-confirm", response_model=FileUploadResponse)
|
||||
def confirm_managed_archive(
|
||||
file_id: str,
|
||||
payload: ManagedArchiveConfirmRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
token = open_transient_payload(
|
||||
payload.preview_token, ttl_seconds=settings.file_archive_preview_ttl_seconds
|
||||
)
|
||||
expected = {
|
||||
"purpose": _PURPOSE,
|
||||
"file_id": file_id,
|
||||
"version_id": payload.source_version_id,
|
||||
"tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id,
|
||||
"owner_type": payload.owner_type,
|
||||
"owner_id": payload.owner_id or principal.user.id,
|
||||
"path": normalize_folder(payload.path),
|
||||
}
|
||||
if any(
|
||||
token.get(key) != value for key, value in expected.items()
|
||||
) or not isinstance(token.get("upload_preview_token"), str):
|
||||
raise FileStorageError(
|
||||
"Managed archive preview does not match this source, actor, or destination; preview again"
|
||||
)
|
||||
asset, version, data, owner, provenance = _resolve_source(
|
||||
session, principal, file_id, payload
|
||||
)
|
||||
source_provenance = dict(
|
||||
provenance or {"source_type": "managed_archive", "external_id": asset.id}
|
||||
)
|
||||
provenance_metadata = source_provenance.get("metadata")
|
||||
source_provenance["metadata"] = {
|
||||
**(
|
||||
dict(provenance_metadata)
|
||||
if isinstance(provenance_metadata, Mapping)
|
||||
else {}
|
||||
),
|
||||
"archive_source_file_id": asset.id,
|
||||
"archive_source_version_id": version.id,
|
||||
}
|
||||
with BytesIO(data) as source:
|
||||
# Reject collisions, including an archive member targeting its own
|
||||
# source. The original file/version can never be overwritten here.
|
||||
return confirm_archive_upload(
|
||||
file=UploadFile(file=source, filename=asset.filename),
|
||||
preview_token=token["upload_preview_token"],
|
||||
selected_paths_json=json.dumps(payload.selected_paths),
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=owner,
|
||||
path=payload.path,
|
||||
campaign_id=None,
|
||||
password=payload.password,
|
||||
conflict_strategy="reject",
|
||||
conflict_resolutions_json=None,
|
||||
source_provenance_json=json.dumps(source_provenance),
|
||||
source_revision=version.id,
|
||||
connector_policy_json=None,
|
||||
encryption_vault_id=None,
|
||||
operation_id=payload.operation_id,
|
||||
session=session,
|
||||
principal=principal,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, TransientPayloadError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -3,9 +3,11 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
from contextlib import ExitStack, contextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, UploadFile
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
@@ -24,6 +26,10 @@ from govoplan_files.backend.schemas import (
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.archive_work import (
|
||||
ArchiveProgress, ArchiveWorkExpired, discard_staged_upload, read_progress,
|
||||
stage_upload, use_staged_upload,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
UnsafeFilePathError,
|
||||
normalize_folder,
|
||||
@@ -44,7 +50,7 @@ from govoplan_files.backend.storage.files import (
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_asset_list_response,
|
||||
_audit_connector_imports,
|
||||
_cleanup_temp_file,
|
||||
_connector_policy_error,
|
||||
@@ -60,6 +66,57 @@ router = APIRouter(prefix="/files", tags=["files"])
|
||||
_ARCHIVE_PREVIEW_PURPOSE = "files.archive-preview.v1"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _archive_source(file, staged_upload_id, preview_token, principal):
|
||||
if isinstance(staged_upload_id, str):
|
||||
if file is not None and hasattr(file, "file"):
|
||||
raise FileStorageError("Choose either the staged archive or a new upload")
|
||||
if not isinstance(preview_token, str) or not preview_token:
|
||||
raise FileStorageError("A valid preview token is required for a staged archive")
|
||||
payload = open_transient_payload(preview_token, ttl_seconds=settings.file_archive_preview_ttl_seconds)
|
||||
expected = {"purpose": _ARCHIVE_PREVIEW_PURPOSE, "tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id, "staged_upload_id": staged_upload_id}
|
||||
if any(payload.get(key) != value for key, value in expected.items()) or not isinstance(payload.get("filename"), str):
|
||||
raise FileStorageError("Archive preview does not match this staged upload")
|
||||
with use_staged_upload(settings, staged_upload_id, tenant_id=principal.tenant_id, user_id=principal.user.id) as path:
|
||||
token_digest = payload.get("archive_sha256")
|
||||
if not isinstance(token_digest, str) or not hmac.compare_digest(_archive_sha256(path), token_digest):
|
||||
raise FileStorageError("Archive contents changed after preview; preview it again")
|
||||
yield path, payload["filename"]
|
||||
return
|
||||
if file is None or not hasattr(file, "file"):
|
||||
raise FileStorageError("Select an archive to upload")
|
||||
filename = file.filename or "archive"
|
||||
path = _spool_limited_upload_to_temp(file, max_bytes=settings.file_upload_zip_max_bytes, suffix=_archive_suffix(filename))
|
||||
try:
|
||||
yield path, filename
|
||||
finally:
|
||||
try:
|
||||
_cleanup_temp_file(path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
@router.get("/archive-progress/{operation_id}")
|
||||
async def archive_operation_progress(operation_id: str, principal: ApiPrincipal = Depends(require_scope("files:file:upload"))):
|
||||
# Small local receipt read stays available even while sync upload workers
|
||||
# are busy. The request owns the transaction; this is not a job retry API.
|
||||
try:
|
||||
return read_progress(settings, operation_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
except ArchiveWorkExpired as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete("/archive-staging/{stage_id}", status_code=204)
|
||||
def release_staged_archive(stage_id: str, principal: ApiPrincipal = Depends(require_scope("files:file:upload"))):
|
||||
try:
|
||||
discard_staged_upload(settings, stage_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
def _archive_suffix(filename: str) -> str:
|
||||
lowered = filename.casefold()
|
||||
for suffix in (".tar.bz2", ".tar.gz", ".tar.xz", ".tbz2", ".tgz", ".txz", ".tar", ".zip"):
|
||||
@@ -135,24 +192,22 @@ def _validate_archive_preview_token(
|
||||
|
||||
@router.post("/archive-preview", response_model=ArchivePreviewResponse)
|
||||
def preview_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
file: UploadFile | None = FastAPIFile(default=None),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
password: str | None = Form(default=None),
|
||||
retain_upload: bool = Form(default=False),
|
||||
staged_upload_id: str | None = Form(default=None),
|
||||
preview_token: str | None = Form(default=None),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
sources = ExitStack()
|
||||
try:
|
||||
archive_path, filename = sources.enter_context(_archive_source(file, staged_upload_id, preview_token, principal))
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
inspection = inspect_archive(
|
||||
archive_path,
|
||||
filename=filename,
|
||||
@@ -163,6 +218,9 @@ def preview_archive_upload(
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
normalized_path = normalize_folder(path)
|
||||
stage_id = staged_upload_id if isinstance(staged_upload_id, str) else None
|
||||
if retain_upload is True and stage_id is None:
|
||||
stage_id = stage_upload(settings, archive_path, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
preview_token = seal_transient_payload(
|
||||
{
|
||||
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||
@@ -174,13 +232,21 @@ def preview_archive_upload(
|
||||
"campaign_id": campaign_id or "",
|
||||
"archive_format": archive_format,
|
||||
"archive_sha256": digest,
|
||||
**({"staged_upload_id": stage_id, "filename": filename} if stage_id else {}),
|
||||
}
|
||||
)
|
||||
expires_at = datetime.now(UTC) + timedelta(
|
||||
seconds=settings.file_archive_preview_ttl_seconds
|
||||
)
|
||||
if isinstance(staged_upload_id, str):
|
||||
# Repreviewing (for example with a password) does not renew the
|
||||
# underlying private stage's lifetime.
|
||||
expires_at = min(expires_at, datetime.fromtimestamp(
|
||||
Path(archive_path).stat().st_mtime, UTC
|
||||
) + timedelta(seconds=settings.file_archive_preview_ttl_seconds))
|
||||
return ArchivePreviewResponse(
|
||||
preview_token=preview_token,
|
||||
staged_upload_id=stage_id,
|
||||
archive_format=inspection.archive_format,
|
||||
entries=[
|
||||
ArchiveEntryResponse(
|
||||
@@ -200,16 +266,17 @@ def preview_archive_upload(
|
||||
password_verified=inspection.password_verified,
|
||||
expires_at=expires_at.isoformat(),
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
except ArchiveWorkExpired as exc:
|
||||
raise HTTPException(status_code=410, detail=str(exc)) from exc
|
||||
except (FileStorageError, TransientPayloadError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
sources.close()
|
||||
|
||||
|
||||
@router.post("/archive-confirm", response_model=FileUploadResponse)
|
||||
def confirm_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
file: UploadFile | None = FastAPIFile(default=None),
|
||||
preview_token: str = Form(...),
|
||||
selected_paths_json: str = Form(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
@@ -225,13 +292,17 @@ def confirm_archive_upload(
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
staged_upload_id: str | None = Form(default=None),
|
||||
operation_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
sources = ExitStack()
|
||||
progress = None
|
||||
committed = False
|
||||
try:
|
||||
archive_path, filename = sources.enter_context(_archive_source(file, staged_upload_id, preview_token, principal))
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json)
|
||||
if conflict_resolutions_json
|
||||
@@ -251,11 +322,6 @@ def confirm_archive_upload(
|
||||
)
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
normalized_path = normalize_folder(path)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
_validate_archive_preview_token(
|
||||
preview_token,
|
||||
@@ -268,6 +334,7 @@ def confirm_archive_upload(
|
||||
archive_format=archive_format,
|
||||
archive_sha256=digest,
|
||||
)
|
||||
progress = ArchiveProgress(settings, operation_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
extracted = extract_archive_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
@@ -289,16 +356,20 @@ def confirm_archive_upload(
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||
progress=progress,
|
||||
)
|
||||
uploaded_assets = [item.asset for item in extracted]
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
progress("finalizing", len(extracted), len(extracted), progress.value["completed_bytes"], progress.value["total_bytes"])
|
||||
session.flush()
|
||||
response = FileUploadResponse(files=_asset_list_response(session, uploaded_assets, include_shares=True))
|
||||
session.commit()
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
committed = True
|
||||
progress.finish(True)
|
||||
return response
|
||||
except ArchiveWorkExpired as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(status_code=410, detail=str(exc)) from exc
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
@@ -311,9 +382,21 @@ def confirm_archive_upload(
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
if progress and not committed:
|
||||
progress.finish(False)
|
||||
try:
|
||||
sources.close()
|
||||
except OSError:
|
||||
pass
|
||||
if committed and isinstance(staged_upload_id, str):
|
||||
try:
|
||||
discard_staged_upload(settings, staged_upload_id, tenant_id=principal.tenant_id, user_id=principal.user.id)
|
||||
except (OSError, FileStorageError):
|
||||
pass
|
||||
|
||||
|
||||
@router.post("/upload", response_model=FileUploadResponse)
|
||||
@@ -402,6 +485,8 @@ def upload_files(
|
||||
)
|
||||
uploaded_assets.append(stored.asset)
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
session.flush()
|
||||
response = FileUploadResponse(files=_asset_list_response(session, uploaded_assets, include_shares=True))
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
@@ -409,12 +494,7 @@ def upload_files(
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||
@@ -469,6 +549,8 @@ def upload_zip(
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
_audit_connector_imports(session, principal, [item.asset for item in extracted])
|
||||
session.flush()
|
||||
response = FileUploadResponse(files=_asset_list_response(session, [item.asset for item in extracted], include_shares=True))
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
@@ -479,9 +561,4 @@ def upload_zip(
|
||||
finally:
|
||||
if zip_path:
|
||||
_cleanup_temp_file(zip_path)
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, item.asset, include_shares=True)
|
||||
for item in extracted
|
||||
]
|
||||
)
|
||||
return response
|
||||
|
||||
@@ -272,6 +272,7 @@ class ArchiveEntryResponse(BaseModel):
|
||||
|
||||
class ArchivePreviewResponse(BaseModel):
|
||||
preview_token: str
|
||||
staged_upload_id: str | None = None
|
||||
archive_format: str
|
||||
entries: list[ArchiveEntryResponse] = Field(default_factory=list)
|
||||
file_count: int
|
||||
|
||||
@@ -0,0 +1,418 @@
|
||||
"""Disposable archive parsers with bounded, acknowledged member staging.
|
||||
|
||||
This is a resource boundary for server-owned parsers, not a filesystem/network
|
||||
sandbox. Only the parent owns database state, credentials and durable storage.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager, closing
|
||||
from dataclasses import asdict, replace
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import stat
|
||||
import tempfile
|
||||
|
||||
from govoplan_core.security.bounded_process import (
|
||||
ProcessBudgetError,
|
||||
ProcessLimits,
|
||||
bounded_operation_admission,
|
||||
run_bounded_operation,
|
||||
)
|
||||
from govoplan_core.security.worker_payload import (
|
||||
WorkerPayloadError, decode_worker_payload, encode_worker_payload,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
INSPECTION_LIMITS = ProcessLimits(
|
||||
wall_seconds=120, cpu_seconds=90, memory_bytes=512 * 1024 * 1024,
|
||||
input_bytes=64 * 1024 * 1024, output_bytes=64 * 1024 * 1024,
|
||||
)
|
||||
EXTRACTION_LIMITS = ProcessLimits(
|
||||
wall_seconds=600, cpu_seconds=300, memory_bytes=512 * 1024 * 1024,
|
||||
input_bytes=64 * 1024 * 1024, output_bytes=64 * 1024,
|
||||
file_bytes=50 * 1024 * 1024,
|
||||
)
|
||||
_STATUS_BYTES = 16 * 1024
|
||||
_MAX_INTEGER = 2**63 - 1
|
||||
_MAX_WAKE_BYTES = 32 * 1024
|
||||
|
||||
|
||||
def _failure(message="Archive worker returned an invalid staging record"):
|
||||
return FileStorageError(message)
|
||||
|
||||
|
||||
def _read_regular(path: Path, maximum: int, *, atomic_record: bool = False) -> bytes:
|
||||
descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
try:
|
||||
info = os.fstat(descriptor)
|
||||
allowed_links = {0, 1} if atomic_record else {1}
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_nlink not in allowed_links or info.st_size > maximum:
|
||||
raise _failure()
|
||||
except BaseException:
|
||||
os.close(descriptor)
|
||||
raise
|
||||
with os.fdopen(descriptor, "rb") as source:
|
||||
# BufferedReader.read(n) can reserve n bytes even for a tiny file. Use
|
||||
# the validated actual size, never the potentially multi-GiB policy cap.
|
||||
value = source.read(info.st_size + 1)
|
||||
after = os.fstat(source.fileno())
|
||||
if (len(value) != info.st_size or after.st_nlink not in allowed_links
|
||||
or _fingerprint(info, atomic_record=atomic_record) != _fingerprint(after, atomic_record=atomic_record)):
|
||||
raise _failure()
|
||||
return value
|
||||
|
||||
|
||||
def _fingerprint(info, *, atomic_record=False):
|
||||
# Replacing a status/ack atomically may unlink an already-open old inode and
|
||||
# update its ctime; its content, size and mtime must nevertheless be stable.
|
||||
return (info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns,
|
||||
None if atomic_record else (info.st_nlink, info.st_ctime_ns))
|
||||
|
||||
|
||||
def _write_record(directory: Path, name: str, value: object) -> None:
|
||||
# name is a server constant: neither member names nor worker DTO paths.
|
||||
payload = json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode()
|
||||
if len(payload) > _STATUS_BYTES:
|
||||
raise _failure()
|
||||
temporary = directory / f".{name}.tmp"
|
||||
try:
|
||||
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as output:
|
||||
output.write(payload)
|
||||
os.replace(temporary, directory / name)
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
finally:
|
||||
try:
|
||||
temporary.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass # The owning private-directory context retries final cleanup.
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _ack_channel(directory: Path, *, create: bool = False):
|
||||
"""One fixed private FIFO; archive names never select a channel or file."""
|
||||
path = directory / "ack"
|
||||
descriptor = None
|
||||
try:
|
||||
try:
|
||||
if create:
|
||||
os.mkfifo(path, 0o600)
|
||||
flags = os.O_RDWR if create else os.O_RDONLY
|
||||
descriptor = os.open(path, flags | os.O_NONBLOCK | os.O_NOFOLLOW)
|
||||
info = os.fstat(descriptor)
|
||||
if not stat.S_ISFIFO(info.st_mode) or info.st_nlink != 1 or stat.S_IMODE(info.st_mode) != 0o600:
|
||||
raise _failure()
|
||||
if not create:
|
||||
# Validate before blocking: a substituted regular file or FIFO
|
||||
# must not hang the open. The parent holds a RDWR handle.
|
||||
os.set_blocking(descriptor, True)
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
yield descriptor
|
||||
finally:
|
||||
if descriptor is not None:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def _send_ack(descriptor: int, sequence: int) -> None:
|
||||
try:
|
||||
# Eight bytes are within PIPE_BUF: one nonblocking write is atomic.
|
||||
if os.write(descriptor, sequence.to_bytes(8, "big")) != 8:
|
||||
raise _failure()
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
|
||||
|
||||
def _receive_ack(descriptor: int, sequence: int) -> None:
|
||||
value = b""
|
||||
while len(value) < 8:
|
||||
chunk = os.read(descriptor, 8 - len(value))
|
||||
if not chunk:
|
||||
raise _failure()
|
||||
value += chunk
|
||||
if int.from_bytes(value, "big") != sequence:
|
||||
raise _failure()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _source_stage(archive_data):
|
||||
if os.name != "posix" or not hasattr(os, "O_NOFOLLOW"):
|
||||
raise FileStorageError("Required isolated-worker resource controls are unavailable")
|
||||
with tempfile.TemporaryDirectory(prefix="govoplan-archive-worker-") as temporary:
|
||||
directory = Path(temporary)
|
||||
os.chmod(directory, 0o700)
|
||||
source_path = directory / "source"
|
||||
descriptor = os.open(source_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb") as target:
|
||||
if isinstance(archive_data, bytes):
|
||||
target.write(archive_data)
|
||||
else:
|
||||
source_descriptor = os.open(os.fspath(archive_data), os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
with os.fdopen(source_descriptor, "rb") as source:
|
||||
before = os.fstat(source.fileno())
|
||||
if not stat.S_ISREG(before.st_mode):
|
||||
raise FileStorageError("Archive source must be a regular file")
|
||||
# Never follow a growing producer to EOF. Callers apply
|
||||
# the compressed request cap before supplying this path.
|
||||
remaining = before.st_size
|
||||
while remaining:
|
||||
chunk = source.read(min(remaining, 1024 * 1024))
|
||||
if not chunk:
|
||||
raise FileStorageError("Archive source changed during inspection")
|
||||
target.write(chunk)
|
||||
remaining -= len(chunk)
|
||||
extra = source.read(1)
|
||||
after = os.fstat(source.fileno())
|
||||
if extra or _fingerprint(before) != _fingerprint(after):
|
||||
raise FileStorageError("Archive source changed during inspection")
|
||||
except OSError as exc:
|
||||
raise FileStorageError("Archive source could not be staged safely") from exc
|
||||
yield directory, source_path
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _admitted_stage(archive_data):
|
||||
# Share Core capacity with other expensive module work before copying any
|
||||
# source bytes, not just after preparation has already consumed resources.
|
||||
try:
|
||||
with bounded_operation_admission() as admission, _source_stage(archive_data) as (directory, source):
|
||||
yield directory, source, admission
|
||||
except ProcessBudgetError as exc:
|
||||
raise FileStorageError(f"Archive processing failed ({exc.code}): {exc}") from exc
|
||||
|
||||
|
||||
def _run(operation, data, *, limits, tick=None, admission=None):
|
||||
try:
|
||||
payload = encode_worker_payload(data, max_bytes=limits.input_bytes)
|
||||
wire = run_bounded_operation(operation, payload, limits=limits, cancelled=tick, admission=admission)
|
||||
result = decode_worker_payload(wire, max_bytes=limits.output_bytes)
|
||||
except ProcessBudgetError as exc:
|
||||
raise FileStorageError(f"Archive processing failed ({exc.code}): {exc}") from exc
|
||||
except WorkerPayloadError as exc:
|
||||
raise FileStorageError("Archive worker data exceeded safe transport limits") from exc
|
||||
if type(result) is not dict:
|
||||
raise _failure()
|
||||
if "error" in result:
|
||||
from govoplan_files.backend.storage.archives import ArchivePasswordError
|
||||
if type(result.get("error")) is not str or type(result.get("password_error")) is not bool:
|
||||
raise _failure()
|
||||
error_type = ArchivePasswordError if result["password_error"] else FileStorageError
|
||||
raise error_type(result["error"])
|
||||
return result
|
||||
|
||||
|
||||
def inspect_archive_isolated(archive_data, **options):
|
||||
from govoplan_files.backend.storage.archives import ArchiveEntry, ArchiveInspection, archive_format_for_filename
|
||||
archive_format_for_filename(options["filename"])
|
||||
with _admitted_stage(archive_data) as (_directory, source, admission):
|
||||
result = _run(_inspect_worker, {"source": str(source), "options": options},
|
||||
limits=INSPECTION_LIMITS, admission=admission)
|
||||
inspection = result.get("inspection")
|
||||
if type(inspection) is not dict or type(inspection.get("entries")) is not tuple:
|
||||
raise _failure()
|
||||
if len(inspection["entries"]) > options["max_entries"]:
|
||||
raise _failure()
|
||||
try:
|
||||
inspection["entries"] = tuple(ArchiveEntry(**entry) for entry in inspection["entries"])
|
||||
return ArchiveInspection(**inspection)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise _failure() from exc
|
||||
|
||||
|
||||
def _inspect_worker(payload: bytes) -> bytes:
|
||||
from govoplan_files.backend.storage.archives import ArchivePasswordError, _inspect_archive_content
|
||||
data = decode_worker_payload(payload, max_bytes=INSPECTION_LIMITS.input_bytes)
|
||||
try:
|
||||
inspection = _inspect_archive_content(data["source"], **data["options"])
|
||||
result = {"inspection": asdict(inspection)}
|
||||
except FileStorageError as exc:
|
||||
result = {"error": str(exc), "password_error": isinstance(exc, ArchivePasswordError)}
|
||||
return encode_worker_payload(result, max_bytes=INSPECTION_LIMITS.output_bytes)
|
||||
|
||||
|
||||
def extract_archive_isolated(session, *, archive_data, filename, password, selected_paths,
|
||||
max_entries, max_file_bytes, max_expanded_bytes,
|
||||
max_expansion_ratio, progress, store_options):
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
_safe_member_path, _store_archive_members, archive_format_for_filename,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import archive_storage_backend_scope
|
||||
archive_format_for_filename(filename)
|
||||
if type(max_file_bytes) is not int or not 0 < max_file_bytes <= 2 * 1024 * 1024 * 1024:
|
||||
raise FileStorageError("Isolated archive members require a positive limit of at most 2 GiB")
|
||||
limits = replace(
|
||||
EXTRACTION_LIMITS,
|
||||
memory_bytes=max(EXTRACTION_LIMITS.memory_bytes, 3 * max_file_bytes + 128 * 1024 * 1024),
|
||||
file_bytes=max(max_file_bytes, _STATUS_BYTES),
|
||||
)
|
||||
uploaded = []
|
||||
stored_bytes = 0
|
||||
sequence = 0
|
||||
seen_paths = set()
|
||||
last_extracted_bytes = 0
|
||||
last_extracted_files = 0
|
||||
expected_totals = None
|
||||
selected = None if selected_paths is None else {_safe_member_path(path) for path in selected_paths}
|
||||
if progress:
|
||||
progress("inspecting", 0, 0, 0, 0)
|
||||
with (
|
||||
_admitted_stage(archive_data) as (directory, source, admission),
|
||||
_ack_channel(directory, create=True) as acknowledgement,
|
||||
archive_storage_backend_scope(),
|
||||
):
|
||||
actual_total_limit = min(max_expanded_bytes, source.stat().st_size * max_expansion_ratio)
|
||||
|
||||
def poll():
|
||||
nonlocal stored_bytes, sequence, last_extracted_bytes, last_extracted_files, expected_totals
|
||||
try:
|
||||
record = json.loads(_read_regular(directory / "status", _STATUS_BYTES, atomic_record=True))
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
except (OSError, ValueError, RecursionError) as exc:
|
||||
raise _failure() from exc
|
||||
if type(record) is not dict or type(record.get("sequence")) is not int:
|
||||
raise _failure()
|
||||
next_sequence = record["sequence"]
|
||||
if not 0 < next_sequence <= _MAX_INTEGER or next_sequence < sequence:
|
||||
raise _failure()
|
||||
if next_sequence == sequence:
|
||||
return False
|
||||
kind = record.get("kind")
|
||||
counters = record.get("progress")
|
||||
if kind not in {"progress", "member"} or type(counters) is not list or len(counters) != 5:
|
||||
raise _failure()
|
||||
keys = {"sequence", "kind", "progress"}
|
||||
if kind == "member":
|
||||
keys |= {"path", "size", "sha256"}
|
||||
if set(record) != keys:
|
||||
raise _failure()
|
||||
phase, completed, total, completed_bytes, total_bytes = counters
|
||||
if phase != "extracting" or any(type(value) is not int for value in counters[1:]):
|
||||
raise _failure()
|
||||
if not (last_extracted_files <= completed <= total <= max_entries
|
||||
and 0 <= total_bytes <= actual_total_limit
|
||||
and last_extracted_bytes <= completed_bytes <= total_bytes):
|
||||
raise _failure()
|
||||
if expected_totals is not None and expected_totals != (total, total_bytes):
|
||||
raise _failure()
|
||||
if completed not in {len(uploaded), len(uploaded) + 1}:
|
||||
raise _failure()
|
||||
sequence = next_sequence
|
||||
last_extracted_bytes = completed_bytes
|
||||
last_extracted_files = completed
|
||||
expected_totals = total, total_bytes
|
||||
if progress:
|
||||
progress(*counters)
|
||||
if kind == "progress":
|
||||
return False
|
||||
if completed != len(uploaded) + 1 or type(record.get("path")) is not str:
|
||||
raise _failure()
|
||||
inner_path = _safe_member_path(record["path"])
|
||||
if inner_path != record["path"] or inner_path in seen_paths:
|
||||
raise _failure()
|
||||
if selected is not None and not any(inner_path == path or inner_path.startswith(path + "/") for path in selected):
|
||||
raise _failure()
|
||||
size = record.get("size")
|
||||
if type(size) is not int or not 0 <= size <= max_file_bytes or stored_bytes + size != completed_bytes:
|
||||
raise _failure()
|
||||
member_path = directory / f"member-{completed:08d}"
|
||||
try:
|
||||
data = _read_regular(member_path, max_file_bytes)
|
||||
except OSError as exc:
|
||||
raise _failure() from exc
|
||||
if len(data) != size or hashlib.sha256(data).hexdigest() != record.get("sha256"):
|
||||
raise _failure()
|
||||
# The established persistence API accepts bytes, not a stream/path.
|
||||
# Keep one member resident, with the existing transaction and quotas.
|
||||
uploaded.extend(_store_archive_members(session, members=((inner_path, data),), **store_options))
|
||||
stored_bytes += len(data)
|
||||
seen_paths.add(inner_path)
|
||||
del data
|
||||
try:
|
||||
member_path.unlink()
|
||||
except OSError as exc:
|
||||
raise _failure("Archive worker staging channel is unavailable") from exc
|
||||
if progress:
|
||||
progress("storing", len(uploaded), total, stored_bytes, total_bytes)
|
||||
_send_ack(acknowledgement, sequence)
|
||||
return False
|
||||
|
||||
result = _run(_extract_worker, {
|
||||
"source": str(source), "directory": str(directory),
|
||||
"options": {"filename": filename, "password": password, "max_entries": max_entries,
|
||||
"max_expanded_bytes": max_expanded_bytes, "max_expansion_ratio": max_expansion_ratio},
|
||||
"selected_paths": None if selected is None else tuple(selected),
|
||||
"max_file_bytes": max_file_bytes,
|
||||
}, limits=limits, tick=poll, admission=admission)
|
||||
if type(result.get("completed")) is not int or result["completed"] != len(uploaded):
|
||||
raise _failure()
|
||||
return uploaded
|
||||
|
||||
|
||||
def _extract_worker(payload: bytes) -> bytes:
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
ArchivePasswordError, _inspect_archive_content, _read_selected_tar_members,
|
||||
_read_selected_zip_members, _selected_file_paths,
|
||||
)
|
||||
data = decode_worker_payload(payload, max_bytes=EXTRACTION_LIMITS.input_bytes)
|
||||
directory = Path(data["directory"])
|
||||
sequence = 0
|
||||
completed = 0
|
||||
|
||||
def report(phase, files, total, count, total_bytes, **member):
|
||||
nonlocal sequence
|
||||
sequence += 1
|
||||
_write_record(directory, "status", {
|
||||
"sequence": sequence, "kind": "member" if member else "progress",
|
||||
"progress": [phase, files, total, count, total_bytes], **member,
|
||||
})
|
||||
|
||||
try:
|
||||
inspection = _inspect_archive_content(data["source"], **data["options"])
|
||||
if inspection.requires_password and not inspection.password_verified:
|
||||
raise ArchivePasswordError("Archive password is required")
|
||||
selected = _selected_file_paths(inspection.entries, data["selected_paths"])
|
||||
if not selected:
|
||||
raise FileStorageError("Select at least one archive file to import")
|
||||
total_bytes = sum(entry.size_bytes for entry in inspection.entries if entry.path in selected)
|
||||
total_limit = min(data["options"]["max_expanded_bytes"],
|
||||
inspection.compressed_size_bytes * data["options"]["max_expansion_ratio"])
|
||||
arguments = {"selected_files": selected, "max_file_bytes": data["max_file_bytes"],
|
||||
"max_total_bytes": total_limit, "progress": report, "total_bytes": total_bytes}
|
||||
if inspection.archive_format == "zip":
|
||||
members = _read_selected_zip_members(data["source"], password=data["options"]["password"], **arguments)
|
||||
else:
|
||||
members = _read_selected_tar_members(data["source"], **arguments)
|
||||
actual = 0
|
||||
with closing(members), _ack_channel(directory) as acknowledgement:
|
||||
for inner_path, content in members:
|
||||
completed += 1
|
||||
actual += len(content)
|
||||
member_path = directory / f"member-{completed:08d}"
|
||||
descriptor = os.open(member_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as output:
|
||||
output.write(content)
|
||||
report("extracting", completed, len(selected), actual, total_bytes,
|
||||
path=inner_path, size=len(content), sha256=hashlib.sha256(content).hexdigest())
|
||||
del content
|
||||
# Wake Core's selector after publishing a completed member. One
|
||||
# discarded stderr byte per member fits its 64 KiB ceiling for
|
||||
# the default 10,000-entry cap; intermediate progress never
|
||||
# wakes. Non-default >32,768-member workloads retain ordinary
|
||||
# Core progress polling after the fixed wake budget is spent.
|
||||
if completed <= _MAX_WAKE_BYTES:
|
||||
os.write(2, b".")
|
||||
# Acknowledgement applies backpressure: the parent stores and
|
||||
# removes this member before we allocate/decode the next one.
|
||||
# Blocking on a bounded FIFO read avoids a fixed delay for each
|
||||
# tiny file. Core still enforces the overall wall/CPU budgets.
|
||||
_receive_ack(acknowledgement, sequence)
|
||||
result = {"completed": completed}
|
||||
except FileStorageError as exc:
|
||||
result = {"error": str(exc), "password_error": isinstance(exc, ArchivePasswordError)}
|
||||
return encode_worker_payload(result, max_bytes=EXTRACTION_LIMITS.output_bytes)
|
||||
@@ -9,7 +9,7 @@ from dataclasses import dataclass
|
||||
from io import BytesIO
|
||||
from os import PathLike
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any, BinaryIO, Iterable, Literal
|
||||
from typing import Any, BinaryIO, Callable, Iterable, Iterator, Literal
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -33,11 +33,17 @@ from govoplan_files.backend.storage.paths import (
|
||||
normalize_folder,
|
||||
normalize_logical_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.native_zip import (
|
||||
native_zip_library,
|
||||
read_native_zip_members,
|
||||
)
|
||||
|
||||
|
||||
_ARCHIVE_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:")
|
||||
ARCHIVE_UPLOAD_MAX_ENTRIES = 10_000
|
||||
ARCHIVE_MAX_PATH_BYTES = 4096
|
||||
ARCHIVE_MAX_PATH_DEPTH = 128
|
||||
# Kept for callers and documentation using the former ZIP-specific name.
|
||||
ZIP_UPLOAD_MAX_FILES = ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||
SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||
@@ -51,6 +57,8 @@ SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||
".zip",
|
||||
)
|
||||
|
||||
ArchiveProgress = Callable[[str, int, int, int, int], None]
|
||||
|
||||
|
||||
class ArchivePasswordError(FileStorageError):
|
||||
pass
|
||||
@@ -132,6 +140,22 @@ def inspect_archive(
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> ArchiveInspection:
|
||||
from govoplan_files.backend.storage.archive_workers import inspect_archive_isolated
|
||||
return inspect_archive_isolated(
|
||||
archive_data, filename=filename, password=password, max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes, max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
|
||||
|
||||
def _inspect_archive_content(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
filename: str,
|
||||
password: str | None = None,
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> ArchiveInspection:
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
compressed_size = _archive_size(archive_data)
|
||||
@@ -141,7 +165,13 @@ def inspect_archive(
|
||||
password=password,
|
||||
)
|
||||
else:
|
||||
entries = _inspect_tar(archive_data)
|
||||
entries = _inspect_tar(
|
||||
archive_data,
|
||||
compressed_size=compressed_size,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
requires_password = False
|
||||
password_verified = True
|
||||
_validate_archive_limits(
|
||||
@@ -151,14 +181,12 @@ def inspect_archive(
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
complete_entries = _with_derived_directories(entries)
|
||||
complete_entries = _with_derived_directories(entries, max_entries=max_entries)
|
||||
return ArchiveInspection(
|
||||
archive_format=archive_format,
|
||||
entries=tuple(complete_entries),
|
||||
file_count=sum(entry.kind == "file" for entry in complete_entries),
|
||||
directory_count=sum(
|
||||
entry.kind == "directory" for entry in complete_entries
|
||||
),
|
||||
directory_count=sum(entry.kind == "directory" for entry in complete_entries),
|
||||
expanded_size_bytes=sum(
|
||||
entry.size_bytes for entry in entries if entry.kind == "file"
|
||||
),
|
||||
@@ -190,53 +218,19 @@ def extract_archive_upload(
|
||||
max_file_bytes: int = 50 * 1024 * 1024,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
progress: ArchiveProgress | None = None,
|
||||
) -> list[UploadedStoredFile]:
|
||||
inspection = inspect_archive(
|
||||
archive_data,
|
||||
filename=filename,
|
||||
password=password,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
if inspection.requires_password and not inspection.password_verified:
|
||||
raise ArchivePasswordError("Archive password is required")
|
||||
selected_files = _selected_file_paths(inspection.entries, selected_paths)
|
||||
if not selected_files:
|
||||
raise FileStorageError("Select at least one archive file to import")
|
||||
actual_total_limit = min(
|
||||
max_expanded_bytes,
|
||||
inspection.compressed_size_bytes * max_expansion_ratio,
|
||||
)
|
||||
if inspection.archive_format == "zip":
|
||||
members = _read_selected_zip_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
password=password,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
else:
|
||||
members = _read_selected_tar_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
return _store_archive_members(
|
||||
session,
|
||||
members=members,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
folder=folder,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
from govoplan_files.backend.storage.archive_workers import extract_archive_isolated
|
||||
return extract_archive_isolated(
|
||||
session, archive_data=archive_data, filename=filename, password=password,
|
||||
selected_paths=selected_paths, max_entries=max_entries, max_file_bytes=max_file_bytes,
|
||||
max_expanded_bytes=max_expanded_bytes, max_expansion_ratio=max_expansion_ratio,
|
||||
progress=progress, store_options={
|
||||
"tenant_id": tenant_id, "owner_type": owner_type, "owner_id": owner_id,
|
||||
"user_id": user_id, "folder": folder, "campaign_id": campaign_id,
|
||||
"conflict_strategy": conflict_strategy, "conflict_resolutions": conflict_resolutions,
|
||||
"metadata": metadata, "is_admin": is_admin, "encryption_vault_id": encryption_vault_id,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -319,11 +313,23 @@ def _inspect_zip(
|
||||
|
||||
def _inspect_tar(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
compressed_size: int,
|
||||
max_entries: int,
|
||||
max_expanded_bytes: int,
|
||||
max_expansion_ratio: int,
|
||||
) -> list[ArchiveEntry]:
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
entries: list[ArchiveEntry] = []
|
||||
for member in archive.getmembers():
|
||||
expanded_size = 0
|
||||
for member in archive:
|
||||
# Check each header before advancing across its payload. In a
|
||||
# compressed TAR getmembers() would inflate everything first.
|
||||
if len(entries) >= max_entries:
|
||||
raise FileStorageError(
|
||||
f"Archive contains too many entries (limit {max_entries})"
|
||||
)
|
||||
path = _safe_member_path(member.name)
|
||||
if member.isdir():
|
||||
entries.append(
|
||||
@@ -334,6 +340,21 @@ def _inspect_tar(
|
||||
raise FileStorageError(
|
||||
f"Archive member {member.name!r} is not a regular file or directory"
|
||||
)
|
||||
if member.size < 0:
|
||||
raise FileStorageError("Archive member has invalid size metadata")
|
||||
expanded_size += member.size
|
||||
if expanded_size > max_expanded_bytes:
|
||||
raise FileStorageError(
|
||||
"Archive is too large after extraction "
|
||||
f"(limit {max_expanded_bytes} bytes)"
|
||||
)
|
||||
if expanded_size and (
|
||||
compressed_size <= 0
|
||||
or expanded_size > compressed_size * max_expansion_ratio
|
||||
):
|
||||
raise FileStorageError(
|
||||
f"Archive expansion ratio exceeds {max_expansion_ratio}:1"
|
||||
)
|
||||
entries.append(
|
||||
ArchiveEntry(
|
||||
path=path,
|
||||
@@ -352,21 +373,19 @@ def _zip_entry(info: zipfile.ZipInfo) -> ArchiveEntry:
|
||||
path = _safe_member_path(info.filename)
|
||||
unix_mode = (info.external_attr >> 16) & 0xFFFF
|
||||
file_type = stat.S_IFMT(unix_mode)
|
||||
if file_type and not (
|
||||
stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)
|
||||
):
|
||||
if file_type and not (stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)):
|
||||
raise FileStorageError(
|
||||
f"Archive member {info.filename!r} is not a regular file or directory"
|
||||
)
|
||||
if info.file_size < 0 or info.compress_size < 0:
|
||||
raise FileStorageError(f"Archive member {info.filename!r} has invalid size metadata")
|
||||
raise FileStorageError(
|
||||
f"Archive member {info.filename!r} has invalid size metadata"
|
||||
)
|
||||
return ArchiveEntry(
|
||||
path=path,
|
||||
kind="directory" if info.is_dir() else "file",
|
||||
size_bytes=0 if info.is_dir() else int(info.file_size),
|
||||
compressed_size_bytes=(
|
||||
None if info.is_dir() else int(info.compress_size)
|
||||
),
|
||||
compressed_size_bytes=(None if info.is_dir() else int(info.compress_size)),
|
||||
encrypted=bool(info.flag_bits & 0x1),
|
||||
)
|
||||
|
||||
@@ -388,9 +407,7 @@ def _validate_archive_limits(
|
||||
for entry in entries:
|
||||
previous_kind = seen.get(entry.path)
|
||||
if previous_kind is not None:
|
||||
raise FileStorageError(
|
||||
f"Archive contains duplicate path {entry.path!r}"
|
||||
)
|
||||
raise FileStorageError(f"Archive contains duplicate path {entry.path!r}")
|
||||
seen[entry.path] = entry.kind
|
||||
if entry.kind == "file":
|
||||
expanded_size += entry.size_bytes
|
||||
@@ -400,17 +417,17 @@ def _validate_archive_limits(
|
||||
f"(limit {max_expanded_bytes} bytes)"
|
||||
)
|
||||
if expanded_size and (
|
||||
compressed_size <= 0
|
||||
or expanded_size > compressed_size * max_expansion_ratio
|
||||
compressed_size <= 0 or expanded_size > compressed_size * max_expansion_ratio
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Archive expansion ratio exceeds "
|
||||
f"{max_expansion_ratio}:1"
|
||||
f"Archive expansion ratio exceeds {max_expansion_ratio}:1"
|
||||
)
|
||||
|
||||
|
||||
def _with_derived_directories(
|
||||
entries: list[ArchiveEntry],
|
||||
*,
|
||||
max_entries: int,
|
||||
) -> list[ArchiveEntry]:
|
||||
by_path = {entry.path: entry for entry in entries}
|
||||
for entry in entries:
|
||||
@@ -422,10 +439,13 @@ def _with_derived_directories(
|
||||
raise FileStorageError(
|
||||
f"Archive path {path!r} is both a file and a directory"
|
||||
)
|
||||
by_path.setdefault(
|
||||
path,
|
||||
ArchiveEntry(path=path, kind="directory", size_bytes=0),
|
||||
)
|
||||
if path not in by_path:
|
||||
if len(by_path) >= max_entries:
|
||||
raise FileStorageError(
|
||||
"Archive contains too many entries including parent "
|
||||
f"directories (limit {max_entries})"
|
||||
)
|
||||
by_path[path] = ArchiveEntry(path=path, kind="directory", size_bytes=0)
|
||||
parent = parent.parent
|
||||
return sorted(
|
||||
by_path.values(),
|
||||
@@ -458,9 +478,7 @@ def _selected_file_paths(
|
||||
continue
|
||||
prefix = f"{path}/"
|
||||
selected_files.update(
|
||||
file_path
|
||||
for file_path in file_paths
|
||||
if file_path.startswith(prefix)
|
||||
file_path for file_path in file_paths if file_path.startswith(prefix)
|
||||
)
|
||||
return selected_files
|
||||
|
||||
@@ -472,12 +490,45 @@ def _read_selected_zip_members(
|
||||
password: str | None,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
progress: ArchiveProgress | None = None,
|
||||
total_bytes: int = 0,
|
||||
) -> Iterator[tuple[str, bytes]]:
|
||||
total = 0
|
||||
completed = 0
|
||||
try:
|
||||
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||
for info in archive.infolist():
|
||||
infos = archive.infolist()
|
||||
# Accelerate only classic ZIPCrypto using established public C
|
||||
# decoding APIs; AES and other formats retain their existing path.
|
||||
native_compatible = (
|
||||
bool(password)
|
||||
and "\x00" not in password
|
||||
and any(info.flag_bits & 1 for info in infos)
|
||||
and all(
|
||||
not getattr(info, "wz_aes_version", None)
|
||||
and info.compress_type in {zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED}
|
||||
and (info.filename.isascii() or bool(info.flag_bits & 0x800))
|
||||
for info in infos
|
||||
)
|
||||
)
|
||||
native = native_zip_library() if native_compatible else None
|
||||
if native is not None:
|
||||
yield from read_native_zip_members(
|
||||
native,
|
||||
archive_data,
|
||||
infos_by_path={
|
||||
_safe_member_path(info.filename): info for info in infos
|
||||
},
|
||||
selected_files=selected_files,
|
||||
normalize_name=_safe_member_path,
|
||||
password=password,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
progress=progress,
|
||||
total_bytes=total_bytes,
|
||||
)
|
||||
return
|
||||
for info in infos:
|
||||
if info.is_dir():
|
||||
continue
|
||||
path = _safe_member_path(info.filename)
|
||||
@@ -492,6 +543,17 @@ def _read_selected_zip_members(
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
on_bytes=(
|
||||
lambda count: progress(
|
||||
"extracting",
|
||||
completed,
|
||||
len(selected_files),
|
||||
count,
|
||||
total_bytes,
|
||||
)
|
||||
)
|
||||
if progress
|
||||
else None,
|
||||
)
|
||||
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||
if info.flag_bits & 0x1:
|
||||
@@ -499,12 +561,17 @@ def _read_selected_zip_members(
|
||||
"Archive password is incorrect"
|
||||
) from exc
|
||||
raise
|
||||
result.append((path, data))
|
||||
completed += 1
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
yield path, data
|
||||
del data
|
||||
except (FileStorageError, ArchivePasswordError):
|
||||
raise
|
||||
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise FileStorageError("ZIP extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_selected_tar_members(
|
||||
@@ -513,12 +580,16 @@ def _read_selected_tar_members(
|
||||
selected_files: set[str],
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
progress: ArchiveProgress | None = None,
|
||||
total_bytes: int = 0,
|
||||
) -> Iterator[tuple[str, bytes]]:
|
||||
total = 0
|
||||
completed = 0
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
for member in archive.getmembers():
|
||||
# Iterate as headers are read instead of inflating the entire TAR
|
||||
# with getmembers(), then seeking backwards to inflate it again.
|
||||
for member in archive:
|
||||
if not member.isfile():
|
||||
continue
|
||||
path = _safe_member_path(member.name)
|
||||
@@ -526,9 +597,7 @@ def _read_selected_tar_members(
|
||||
continue
|
||||
source = archive.extractfile(member)
|
||||
if source is None:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} could not be read"
|
||||
)
|
||||
raise FileStorageError(f"Archive member {path!r} could not be read")
|
||||
with source:
|
||||
data, total = _read_member(
|
||||
source,
|
||||
@@ -536,13 +605,29 @@ def _read_selected_tar_members(
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
on_bytes=(
|
||||
lambda count: progress(
|
||||
"extracting",
|
||||
completed,
|
||||
len(selected_files),
|
||||
count,
|
||||
total_bytes,
|
||||
)
|
||||
)
|
||||
if progress
|
||||
else None,
|
||||
)
|
||||
result.append((path, data))
|
||||
completed += 1
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
yield path, data
|
||||
del data
|
||||
except FileStorageError:
|
||||
raise
|
||||
except (OSError, tarfile.TarError) as exc:
|
||||
raise FileStorageError("TAR extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_member(
|
||||
@@ -552,6 +637,7 @@ def _read_member(
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
current_total: int,
|
||||
on_bytes: Callable[[int], None] | None = None,
|
||||
) -> tuple[bytes, int]:
|
||||
parts: list[bytes] = []
|
||||
actual_size = 0
|
||||
@@ -565,12 +651,12 @@ def _read_member(
|
||||
break
|
||||
actual_size += len(chunk)
|
||||
if actual_size > max_file_bytes:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} exceeds per-file limit"
|
||||
)
|
||||
raise FileStorageError(f"Archive member {path!r} exceeds per-file limit")
|
||||
if current_total + actual_size > max_total_bytes:
|
||||
raise FileStorageError("Archive is too large after extraction")
|
||||
parts.append(chunk)
|
||||
if on_bytes:
|
||||
on_bytes(current_total + actual_size)
|
||||
return b"".join(parts), current_total + actual_size
|
||||
|
||||
|
||||
@@ -589,13 +675,15 @@ def _store_archive_members(
|
||||
metadata: dict[str, Any] | None,
|
||||
is_admin: bool,
|
||||
encryption_vault_id: str | None,
|
||||
progress: ArchiveProgress | None = None,
|
||||
total_files: int = 0,
|
||||
total_bytes: int = 0,
|
||||
) -> list[UploadedStoredFile]:
|
||||
uploaded: list[UploadedStoredFile] = []
|
||||
base_folder = normalize_folder(folder)
|
||||
stored_bytes = 0
|
||||
for inner_path, data in members:
|
||||
target_path = (
|
||||
f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
)
|
||||
target_path = f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
uploaded.append(
|
||||
create_file_asset(
|
||||
session,
|
||||
@@ -616,24 +704,39 @@ def _store_archive_members(
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
)
|
||||
stored_bytes += len(data)
|
||||
if progress:
|
||||
progress("storing", len(uploaded), total_files, stored_bytes, total_bytes)
|
||||
del data
|
||||
return uploaded
|
||||
|
||||
|
||||
def _safe_member_path(value: str) -> str:
|
||||
raw = str(value or "").replace("\\", "/").strip()
|
||||
if (
|
||||
not raw
|
||||
or "\x00" in raw
|
||||
or raw.startswith("/")
|
||||
or _WINDOWS_DRIVE_RE.match(raw)
|
||||
):
|
||||
try:
|
||||
oversized = (
|
||||
len(raw) > ARCHIVE_MAX_PATH_BYTES
|
||||
or len(raw.encode("utf-8")) > ARCHIVE_MAX_PATH_BYTES
|
||||
)
|
||||
except UnicodeEncodeError as exc:
|
||||
raise FileStorageError("Archive member path is not valid Unicode") from exc
|
||||
if oversized:
|
||||
raise FileStorageError(
|
||||
f"Archive member path exceeds {ARCHIVE_MAX_PATH_BYTES} UTF-8 bytes"
|
||||
)
|
||||
if not raw or "\x00" in raw or raw.startswith("/") or _WINDOWS_DRIVE_RE.match(raw):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
if any(part == ".." for part in raw.split("/")):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
try:
|
||||
return normalize_logical_path(raw.rstrip("/"))
|
||||
normalized = normalize_logical_path(raw.rstrip("/"))
|
||||
except ValueError as exc:
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}") from exc
|
||||
if normalized.count("/") + 1 > ARCHIVE_MAX_PATH_DEPTH:
|
||||
raise FileStorageError(
|
||||
f"Archive member path exceeds {ARCHIVE_MAX_PATH_DEPTH} components"
|
||||
)
|
||||
return normalized
|
||||
|
||||
|
||||
def _archive_size(archive_data: bytes | str | PathLike[str]) -> int:
|
||||
|
||||
@@ -10,6 +10,10 @@ class FileStorageError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class FileSourceConflict(FileStorageError):
|
||||
"""A source cannot be selected or revision-bound without ambiguity."""
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class UploadedStoredFile:
|
||||
asset: FileAsset
|
||||
|
||||
@@ -32,6 +32,7 @@ from govoplan_files.backend.storage.connector_browse import (
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.paths import filename_from_path
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
|
||||
|
||||
class ConnectorImportError(RuntimeError):
|
||||
@@ -42,6 +43,10 @@ class ConnectorImportUnsupported(ConnectorImportError):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectorRevisionConflict(ConnectorImportError, FileSourceConflict):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorDownloadedFile:
|
||||
filename: str
|
||||
@@ -120,6 +125,14 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||
detail = detail if isinstance(detail, dict) else {}
|
||||
try:
|
||||
after = _request_json("GET", _seafile_url(profile, f"api2/repos/{repo_id}/file/detail/"), headers=headers, params={"p": file_path})
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
before_revision = _clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified"))
|
||||
after_revision = _clean(after.get("id") or after.get("mtime") or after.get("last_modified")) if isinstance(after, dict) else None
|
||||
if not before_revision or before_revision != after_revision or (_int(detail.get("size")) is not None and _int(detail.get("size")) != len(data)):
|
||||
raise ConnectorRevisionConflict("Seafile source changed or could not be revision-verified during download; refresh and retry")
|
||||
filename = filename_from_path(str(detail.get("name") or path))
|
||||
content_type = response.headers.get("content-type") or mimetypes.guess_type(filename)[0]
|
||||
external_id = f"{repo_id}:{normalize_connector_browse_path(path)}"
|
||||
@@ -127,7 +140,7 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=_clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified")),
|
||||
revision=before_revision,
|
||||
external_id=external_id,
|
||||
external_url=download_url,
|
||||
metadata={
|
||||
@@ -202,12 +215,18 @@ def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> C
|
||||
unc_path = _smb_unc_path(location, file_path)
|
||||
smbclient = _smbclient_module()
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
size = _smb_stat_size(stat_result)
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
with smbclient.open_file(unc_path, mode="rb", **kwargs) as handle:
|
||||
data = handle.read(max_bytes + 1)
|
||||
# Deny concurrent write/delete sharing while observing metadata and bytes.
|
||||
with smbclient.open_file(unc_path, mode="rb", share_access="r", **kwargs) as handle:
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
size = _smb_stat_size(stat_result)
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
data = handle.read(min(size, max_bytes) + 1 if size is not None else max_bytes + 1)
|
||||
after = smbclient.stat(unc_path, **kwargs)
|
||||
if (_smb_stat_revision(stat_result) != _smb_stat_revision(after)
|
||||
or getattr(stat_result, "st_ino", None) != getattr(after, "st_ino", None)
|
||||
or size is None or len(data) != size):
|
||||
raise ConnectorRevisionConflict("SMB source changed or could not be verified during download; refresh and retry")
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
@@ -264,16 +283,30 @@ def _download_s3_object(
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
etag: str | None,
|
||||
max_bytes: int,
|
||||
) -> tuple[Any, bytes]:
|
||||
request: dict[str, object] = {"Bucket": bucket, "Key": key}
|
||||
if version_id:
|
||||
# S3's literal "null" version is replaceable when versioning is suspended.
|
||||
if version_id and version_id != "null":
|
||||
request["VersionId"] = version_id
|
||||
elif etag:
|
||||
request["IfMatch"] = etag
|
||||
else:
|
||||
raise ConnectorRevisionConflict("S3 source has no revision or ETag for a verified download")
|
||||
try:
|
||||
response = client.get_object(**request)
|
||||
body = response.get("Body")
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
try:
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
finally:
|
||||
close = getattr(body, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
error = getattr(exc, "response", {})
|
||||
if isinstance(error, dict) and str(error.get("Error", {}).get("Code")) in {"PreconditionFailed", "412"}:
|
||||
raise ConnectorRevisionConflict("S3 source changed during conditional download; refresh and retry") from exc
|
||||
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
@@ -322,12 +355,16 @@ def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
etag=_clean(detail.get("ETag")),
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
if (_clean(response.get("VersionId")) not in (None, version_id)
|
||||
or (_clean(detail.get("ETag")) and _clean(response.get("ETag")) != _clean(detail.get("ETag")))):
|
||||
raise ConnectorRevisionConflict("S3 returned another source revision; refresh and retry")
|
||||
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
|
||||
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
|
||||
filename = filename_from_path(key)
|
||||
@@ -335,7 +372,7 @@ def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=version_id or etag or _clean(detail.get("LastModified")),
|
||||
revision=(version_id if version_id != "null" else None) or etag or _clean(detail.get("LastModified")),
|
||||
external_id=f"{bucket}:{key}",
|
||||
external_url=f"s3://{bucket}/{key}",
|
||||
metadata=_s3_download_metadata(
|
||||
|
||||
@@ -2,6 +2,8 @@ from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any, Iterable
|
||||
@@ -15,13 +17,14 @@ from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, F
|
||||
from govoplan_files.backend.runtime import get_registry, settings
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileSourceConflict, FileStorageError, UploadedStoredFile, utcnow
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||
from govoplan_files.backend.storage.provenance import source_identity as _source_identity, source_identity_hash, source_provenance_from_metadata
|
||||
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
QUARANTINED_BLOB_STATUSES,
|
||||
@@ -30,6 +33,39 @@ from govoplan_files.backend.storage.integrity import (
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
_ARCHIVE_STORAGE_BACKEND: ContextVar[dict[str, StorageBackend] | None] = ContextVar(
|
||||
"govoplan.files.archive_storage_backend", default=None
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def archive_storage_backend_scope():
|
||||
"""Reuse deployment transport only, never principals, policy or file data.
|
||||
|
||||
Lazy creation avoids opening storage for rejected or empty work. Recovery
|
||||
effects retain the backend until post-commit verification is complete.
|
||||
Nested archive work shares the same deployment transport, and finally
|
||||
restores the caller context on success, failure or generator cancellation.
|
||||
"""
|
||||
if _ARCHIVE_STORAGE_BACKEND.get() is not None:
|
||||
yield
|
||||
return
|
||||
token = _ARCHIVE_STORAGE_BACKEND.set({})
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_ARCHIVE_STORAGE_BACKEND.reset(token)
|
||||
|
||||
|
||||
def _archive_write_backend() -> StorageBackend:
|
||||
scope = _ARCHIVE_STORAGE_BACKEND.get()
|
||||
if scope is None:
|
||||
return get_storage_backend()
|
||||
if "backend" not in scope:
|
||||
scope["backend"] = get_storage_backend()
|
||||
return scope["backend"]
|
||||
|
||||
|
||||
def _campaign_access_provider() -> CampaignAccessProvider:
|
||||
registry = get_registry()
|
||||
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||
@@ -88,7 +124,7 @@ def _get_or_create_blob(
|
||||
.one_or_none()
|
||||
)
|
||||
if blob:
|
||||
backend = get_storage_backend()
|
||||
backend = _archive_write_backend()
|
||||
repair_required = (
|
||||
blob.integrity_status in QUARANTINED_BLOB_STATUSES
|
||||
or blob.quarantined_at is not None
|
||||
@@ -162,7 +198,7 @@ def _get_or_create_blob(
|
||||
|
||||
blob_id = str(uuid4())
|
||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
|
||||
backend = get_storage_backend()
|
||||
backend = _archive_write_backend()
|
||||
recovery = begin_blob_write_recovery(
|
||||
session,
|
||||
backend=backend,
|
||||
@@ -214,7 +250,6 @@ def _get_or_create_blob(
|
||||
integrity_checked_at=utcnow(),
|
||||
)
|
||||
session.add(blob)
|
||||
session.flush()
|
||||
return blob
|
||||
|
||||
|
||||
@@ -265,7 +300,12 @@ def create_file_asset(
|
||||
logical_path = _next_available_logical_path(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, desired_path=logical_path)
|
||||
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=encryption_vault_id)
|
||||
# Assign the immutable identities before the first flush. Publishing an
|
||||
# incomplete asset and immediately updating its current version doubled
|
||||
# change events and added a redundant UPDATE for every imported member.
|
||||
version_id = str(uuid4())
|
||||
asset = FileAsset(
|
||||
id=str(uuid4()),
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_user_id=owner_id if owner_type == "user" else None,
|
||||
@@ -275,10 +315,10 @@ def create_file_asset(
|
||||
description=description,
|
||||
created_by_user_id=user_id,
|
||||
metadata_=metadata or {},
|
||||
current_version_id=version_id,
|
||||
)
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
version = FileVersion(
|
||||
id=version_id,
|
||||
tenant_id=tenant_id,
|
||||
file_asset_id=asset.id,
|
||||
blob_id=blob.id,
|
||||
@@ -290,10 +330,8 @@ def create_file_asset(
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
created_by_user_id=user_id,
|
||||
)
|
||||
session.add(version)
|
||||
session.add_all((asset, version))
|
||||
session.flush()
|
||||
asset.current_version_id = version.id
|
||||
session.add(asset)
|
||||
if campaign_id:
|
||||
share_file(session, tenant_id=tenant_id, asset=asset, target_type="campaign", target_id=campaign_id, permission="read", user_id=user_id)
|
||||
return UploadedStoredFile(asset=asset, version=version, blob=blob)
|
||||
@@ -376,14 +414,17 @@ def find_asset_by_source(
|
||||
return None
|
||||
assets = (
|
||||
_asset_query_for_owner(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id)
|
||||
.filter(FileAsset.deleted_at.is_(None))
|
||||
.order_by(FileAsset.updated_at.desc())
|
||||
.filter(FileAsset.deleted_at.is_(None), FileAsset.source_identity_hash == source_identity_hash(source_provenance))
|
||||
.limit(2)
|
||||
.all()
|
||||
)
|
||||
for asset in assets:
|
||||
if _source_identity(source_provenance_from_metadata(asset.metadata_ or {})) == wanted:
|
||||
return asset
|
||||
return None
|
||||
if len(assets) > 1:
|
||||
raise FileSourceConflict("Multiple active files have this source identity; review their provenance before synchronizing. No file was selected or merged.")
|
||||
if not assets:
|
||||
return None
|
||||
if _source_identity(source_provenance_from_metadata(assets[0].metadata_ or {})) != wanted:
|
||||
raise FileSourceConflict("The indexed file source identity does not match its provenance; synchronization was stopped.")
|
||||
return assets[0]
|
||||
|
||||
|
||||
def update_file_asset_content(
|
||||
@@ -1123,29 +1164,6 @@ def _next_version_number(session: Session, asset_id: str) -> int:
|
||||
return (int(row[0]) if row else 0) + 1
|
||||
|
||||
|
||||
def _source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||
if not provenance:
|
||||
return None
|
||||
connector_id = _clean_identity(provenance.get("connector_id"))
|
||||
provider = _clean_identity(provenance.get("provider"))
|
||||
external_id = _clean_identity(provenance.get("external_id"))
|
||||
if connector_id and external_id:
|
||||
return ("external_id", connector_id, provider, external_id)
|
||||
external_path = _clean_identity(provenance.get("external_path"))
|
||||
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||
library_id = _clean_identity(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||
if connector_id and external_path:
|
||||
return ("external_path", connector_id, provider, library_id, external_path)
|
||||
return None
|
||||
|
||||
|
||||
def _clean_identity(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _copy_asset_to_path(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Optional libarchive ZIPCrypto acceleration, without filesystem extraction.
|
||||
|
||||
Only public libarchive read APIs are bound. Callers must preflight the complete
|
||||
ZIP directory with the ordinary archive validator. Missing libraries/symbols
|
||||
select the Python path before decoding; native failures never retry a decoder.
|
||||
Public ABI reference: https://github.com/libarchive/libarchive/blob/master/libarchive/archive.h
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterator, Mapping
|
||||
import ctypes
|
||||
from ctypes.util import find_library
|
||||
from functools import lru_cache
|
||||
import os
|
||||
from os import PathLike
|
||||
import stat
|
||||
from typing import Any
|
||||
import zlib
|
||||
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_ARCHIVE_OK = 0
|
||||
_ARCHIVE_EOF = 1
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def native_zip_library() -> Any | None:
|
||||
name = find_library("archive")
|
||||
if not name:
|
||||
return None
|
||||
try:
|
||||
library = ctypes.CDLL(name)
|
||||
signatures = {
|
||||
"archive_read_new": ([], ctypes.c_void_p),
|
||||
"archive_read_support_filter_none": ([ctypes.c_void_p], ctypes.c_int),
|
||||
"archive_read_support_format_zip": ([ctypes.c_void_p], ctypes.c_int),
|
||||
"archive_read_set_format_option": (
|
||||
[ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_add_passphrase": (
|
||||
[ctypes.c_void_p, ctypes.c_char_p],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_open_memory": (
|
||||
[ctypes.c_void_p, ctypes.c_void_p, ctypes.c_size_t],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_open_filename": (
|
||||
[ctypes.c_void_p, ctypes.c_char_p, ctypes.c_size_t],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_next_header": (
|
||||
[ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)],
|
||||
ctypes.c_int,
|
||||
),
|
||||
"archive_read_data": (
|
||||
[ctypes.c_void_p, ctypes.c_void_p, ctypes.c_size_t],
|
||||
ctypes.c_ssize_t,
|
||||
),
|
||||
"archive_read_free": ([ctypes.c_void_p], ctypes.c_int),
|
||||
"archive_entry_pathname_utf8": ([ctypes.c_void_p], ctypes.c_char_p),
|
||||
"archive_entry_size": ([ctypes.c_void_p], ctypes.c_int64),
|
||||
"archive_entry_filetype": ([ctypes.c_void_p], ctypes.c_uint),
|
||||
}
|
||||
for symbol, (arguments, result) in signatures.items():
|
||||
function = getattr(library, symbol)
|
||||
function.argtypes = arguments
|
||||
function.restype = result
|
||||
return library
|
||||
except (OSError, AttributeError):
|
||||
return None
|
||||
|
||||
|
||||
def _require_ok(status: int) -> None:
|
||||
if status != _ARCHIVE_OK:
|
||||
# Do not expose native error strings: they can contain source paths.
|
||||
raise FileStorageError(
|
||||
"Native ZIP decoding failed: incorrect password, corrupt or unsupported archive"
|
||||
)
|
||||
|
||||
|
||||
def read_native_zip_members(
|
||||
library: Any,
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
infos_by_path: Mapping[str, Any],
|
||||
selected_files: set[str],
|
||||
normalize_name: Callable[[str], str],
|
||||
password: str,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
progress: Callable[[str, int, int, int, int], None] | None = None,
|
||||
total_bytes: int = 0,
|
||||
) -> Iterator[tuple[str, bytes]]:
|
||||
# Keep memory input, password and output buffers alive until reader free.
|
||||
source_buffer = None
|
||||
password_bytes = password.encode("utf-8")
|
||||
buffer = ctypes.create_string_buffer(_READ_CHUNK_SIZE)
|
||||
archive = library.archive_read_new()
|
||||
if not archive:
|
||||
raise FileStorageError("Native ZIP reader could not be allocated")
|
||||
total = 0
|
||||
completed = 0
|
||||
seen: set[str] = set()
|
||||
try:
|
||||
_require_ok(library.archive_read_support_filter_none(archive))
|
||||
_require_ok(library.archive_read_support_format_zip(archive))
|
||||
_require_ok(
|
||||
library.archive_read_set_format_option(
|
||||
archive, b"zip", b"hdrcharset", b"UTF-8"
|
||||
)
|
||||
)
|
||||
_require_ok(library.archive_read_add_passphrase(archive, password_bytes))
|
||||
if isinstance(archive_data, bytes):
|
||||
source_buffer = ctypes.create_string_buffer(archive_data)
|
||||
_require_ok(
|
||||
library.archive_read_open_memory(
|
||||
archive, source_buffer, len(archive_data)
|
||||
)
|
||||
)
|
||||
else:
|
||||
_require_ok(
|
||||
library.archive_read_open_filename(
|
||||
archive, os.fsencode(archive_data), _READ_CHUNK_SIZE
|
||||
)
|
||||
)
|
||||
entry = ctypes.c_void_p()
|
||||
while True:
|
||||
status = library.archive_read_next_header(archive, ctypes.byref(entry))
|
||||
if status == _ARCHIVE_EOF:
|
||||
break
|
||||
_require_ok(status)
|
||||
raw_path = library.archive_entry_pathname_utf8(entry)
|
||||
if not raw_path:
|
||||
raise FileStorageError("Native ZIP entry has no path")
|
||||
try:
|
||||
path = normalize_name(raw_path.decode("utf-8"))
|
||||
except UnicodeDecodeError as exc:
|
||||
raise FileStorageError(
|
||||
"Native ZIP path does not match the inspected directory"
|
||||
) from exc
|
||||
info = infos_by_path.get(path)
|
||||
if info is None or path in seen:
|
||||
raise FileStorageError(
|
||||
"Native ZIP path does not match the inspected directory"
|
||||
)
|
||||
seen.add(path)
|
||||
directory = info.is_dir()
|
||||
file_type = library.archive_entry_filetype(entry)
|
||||
if not (stat.S_ISDIR(file_type) if directory else stat.S_ISREG(file_type)):
|
||||
raise FileStorageError(
|
||||
"Native ZIP entry type does not match the inspected directory"
|
||||
)
|
||||
if library.archive_entry_size(entry) != info.file_size:
|
||||
raise FileStorageError(
|
||||
"Native ZIP entry size does not match the inspected directory"
|
||||
)
|
||||
if directory or path not in selected_files:
|
||||
continue
|
||||
parts = []
|
||||
size = 0
|
||||
checksum = 0
|
||||
while True:
|
||||
read_size = min(
|
||||
_READ_CHUNK_SIZE,
|
||||
max_file_bytes + 1 - size,
|
||||
max_total_bytes + 1 - total,
|
||||
)
|
||||
if read_size <= 0:
|
||||
raise FileStorageError("Archive exceeds its extraction limits")
|
||||
count = library.archive_read_data(archive, buffer, read_size)
|
||||
if count < 0:
|
||||
_require_ok(count)
|
||||
if count == 0:
|
||||
break
|
||||
if count > read_size:
|
||||
raise FileStorageError("Native ZIP reader returned an invalid byte count")
|
||||
size += count
|
||||
total += count
|
||||
if size > max_file_bytes:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} exceeds per-file limit"
|
||||
)
|
||||
if total > max_total_bytes:
|
||||
raise FileStorageError("Archive is too large after extraction")
|
||||
chunk = ctypes.string_at(buffer, count)
|
||||
checksum = zlib.crc32(chunk, checksum)
|
||||
parts.append(chunk)
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
# Independent checks preserve the Python ZIP reader's CRC and size
|
||||
# guarantees even when native decoding behavior changes upstream.
|
||||
if size != info.file_size or checksum != info.CRC:
|
||||
raise FileStorageError(
|
||||
"Native ZIP member failed size or CRC verification"
|
||||
)
|
||||
completed += 1
|
||||
if progress:
|
||||
progress(
|
||||
"extracting", completed, len(selected_files), total, total_bytes
|
||||
)
|
||||
yield path, b"".join(parts)
|
||||
parts.clear()
|
||||
if seen != set(infos_by_path) or completed != len(selected_files):
|
||||
raise FileStorageError(
|
||||
"Native ZIP directory does not match the inspected archive"
|
||||
)
|
||||
finally:
|
||||
library.archive_read_free(archive)
|
||||
@@ -1,12 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
SOURCE_PROVENANCE_METADATA_KEY = "source_provenance"
|
||||
SOURCE_REVISION_METADATA_KEY = "source_revision"
|
||||
|
||||
|
||||
def source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||
"""The established source identity; neither filename nor content identity."""
|
||||
if not provenance:
|
||||
return None
|
||||
def clean(value: object) -> str | None:
|
||||
return str(value).strip() or None if value is not None else None
|
||||
connector_id = clean(provenance.get("connector_id"))
|
||||
provider = clean(provenance.get("provider"))
|
||||
external_id = clean(provenance.get("external_id"))
|
||||
if connector_id and external_id:
|
||||
return ("external_id", connector_id, provider, external_id)
|
||||
external_path = clean(provenance.get("external_path"))
|
||||
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||
library_id = clean(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||
if connector_id and external_path:
|
||||
return ("external_path", connector_id, provider, library_id, external_path)
|
||||
return None
|
||||
|
||||
|
||||
def source_identity_hash(provenance: dict[str, Any] | None) -> str | None:
|
||||
identity = source_identity(provenance)
|
||||
if identity is None:
|
||||
return None
|
||||
return hashlib.sha256(json.dumps(identity, ensure_ascii=True, separators=(",", ":")).encode("ascii")).hexdigest()
|
||||
|
||||
_PROVENANCE_STRING_FIELDS = {
|
||||
"source_type",
|
||||
"connector_id",
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Isolated synthetic archive benchmark; never reads the configured Files store.
|
||||
|
||||
Run from this repository: python tests/benchmark_archive_storage.py.
|
||||
Uses the existing temporary SQLite/local-storage recovery fixture so durable
|
||||
ledger, checksum verification, commit settlement and compensation stay enabled.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import cProfile
|
||||
from contextlib import ExitStack
|
||||
from collections import Counter
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
import pstats
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from unittest.mock import patch
|
||||
import zipfile
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy import event
|
||||
|
||||
from govoplan_files.backend.change_tracking import register_files_change_tracking
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
extract_archive_upload,
|
||||
_read_selected_zip_members,
|
||||
)
|
||||
from govoplan_files.backend.storage.native_zip import native_zip_library
|
||||
from test_storage_recovery import StorageRecoveryTests
|
||||
|
||||
|
||||
def storage_probe(
|
||||
member_count: int,
|
||||
member_size: int,
|
||||
profile: bool,
|
||||
baseline_revision: str | None = None,
|
||||
):
|
||||
case = StorageRecoveryTests()
|
||||
case.setUp()
|
||||
try:
|
||||
register_files_change_tracking()
|
||||
data = BytesIO()
|
||||
rng = random.Random(20260907)
|
||||
with zipfile.ZipFile(data, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for index in range(member_count):
|
||||
archive.writestr(
|
||||
f"documents/{index:04d}.dat", rng.randbytes(member_size)
|
||||
)
|
||||
queries = []
|
||||
event.listen(
|
||||
case.engine,
|
||||
"before_cursor_execute",
|
||||
lambda conn, cursor, statement, parameters, context, executemany: (
|
||||
queries.append(statement)
|
||||
),
|
||||
)
|
||||
profiler = cProfile.Profile()
|
||||
if profile:
|
||||
profiler.enable()
|
||||
with ExitStack() as overrides:
|
||||
overrides.enter_context(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=case.backend,
|
||||
)
|
||||
)
|
||||
if baseline_revision:
|
||||
# Read an earlier local revision into this isolated process;
|
||||
# never replace shared worktree files or connect to live data.
|
||||
source = subprocess.run(
|
||||
[
|
||||
"git",
|
||||
"show",
|
||||
f"{baseline_revision}:src/govoplan_files/backend/storage/files.py",
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout
|
||||
namespace = {"__name__": "govoplan_files.backend.storage.files"}
|
||||
exec(
|
||||
compile(source, f"{baseline_revision}:files.py", "exec"), namespace
|
||||
)
|
||||
namespace.update(
|
||||
get_storage_backend=lambda: case.backend,
|
||||
_storage_backend_name=lambda: case.backend.name,
|
||||
_storage_bucket_name=lambda: "",
|
||||
)
|
||||
overrides.enter_context(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset",
|
||||
namespace["create_file_asset"],
|
||||
)
|
||||
)
|
||||
started = time.perf_counter()
|
||||
result = extract_archive_upload(
|
||||
case.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=data.getvalue(),
|
||||
filename="synthetic.zip",
|
||||
folder="imported",
|
||||
campaign_id=None,
|
||||
)
|
||||
storage_done = time.perf_counter()
|
||||
storage_queries = len(queries)
|
||||
case.session.commit()
|
||||
completed = time.perf_counter()
|
||||
if profile:
|
||||
profiler.disable()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"probe": "durable-storage",
|
||||
"storage_revision": baseline_revision or "working-tree",
|
||||
"members": len(result),
|
||||
"member_bytes": member_size,
|
||||
"extract_store_seconds": round(storage_done - started, 4),
|
||||
"settlement_seconds": round(completed - storage_done, 4),
|
||||
"total_seconds": round(completed - started, 4),
|
||||
"store_queries": storage_queries,
|
||||
"total_queries": len(queries),
|
||||
"queries_by_table": dict(
|
||||
Counter(
|
||||
(
|
||||
match.group(1)
|
||||
if (
|
||||
match := re.search(
|
||||
r"(?:FROM|INTO|UPDATE)\s+([a-z_]+)", statement
|
||||
)
|
||||
)
|
||||
else "other"
|
||||
)
|
||||
for statement in queries
|
||||
)
|
||||
),
|
||||
}
|
||||
)
|
||||
)
|
||||
if profile:
|
||||
pstats.Stats(profiler).strip_dirs().sort_stats("cumulative").print_stats(24)
|
||||
finally:
|
||||
case.doCleanups()
|
||||
|
||||
|
||||
def zipcrypto_probe(size: int):
|
||||
binary = shutil.which("zip")
|
||||
if not binary:
|
||||
print(
|
||||
json.dumps(
|
||||
{"probe": "zipcrypto", "skipped": "fixture ZIP writer unavailable"}
|
||||
)
|
||||
)
|
||||
return
|
||||
# Public fixture password only: never pass a user credential to a process.
|
||||
with tempfile.TemporaryDirectory(prefix="files-synthetic-zipcrypto-") as temporary:
|
||||
path = Path(temporary)
|
||||
content = random.Random(20260907).randbytes(size)
|
||||
(path / "synthetic.bin").write_bytes(content)
|
||||
subprocess.run(
|
||||
[binary, "-q", "-P", "fixture-only", "synthetic.zip", "synthetic.bin"],
|
||||
cwd=path,
|
||||
check=True,
|
||||
)
|
||||
for implementation in (pyzipper.AESZipFile, zipfile.ZipFile):
|
||||
started = time.perf_counter()
|
||||
with implementation(path / "synthetic.zip") as archive:
|
||||
result = archive.read("synthetic.bin", pwd=b"fixture-only")
|
||||
assert result == content
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"probe": "zipcrypto",
|
||||
"reader": implementation.__module__,
|
||||
"bytes": size,
|
||||
"seconds": round(time.perf_counter() - started, 4),
|
||||
}
|
||||
)
|
||||
)
|
||||
if native_zip_library() is not None:
|
||||
started = time.perf_counter()
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
path / "synthetic.zip",
|
||||
selected_files={"synthetic.bin"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=size,
|
||||
max_total_bytes=size,
|
||||
)
|
||||
)
|
||||
assert result == [("synthetic.bin", content)]
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"probe": "zipcrypto",
|
||||
"reader": "libarchive-with-independent-crc",
|
||||
"bytes": size,
|
||||
"seconds": round(time.perf_counter() - started, 4),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--members", type=int, default=100)
|
||||
parser.add_argument("--member-size", type=int, default=4096)
|
||||
parser.add_argument("--zipcrypto-bytes", type=int, default=4 * 1024 * 1024)
|
||||
parser.add_argument("--profile", action="store_true")
|
||||
parser.add_argument(
|
||||
"--baseline-storage-revision",
|
||||
help="Read a trusted earlier local Git revision for isolated storage comparison, without changing the worktree",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
storage_probe(
|
||||
args.members, args.member_size, args.profile, args.baseline_storage_revision
|
||||
)
|
||||
if args.zipcrypto_bytes:
|
||||
zipcrypto_probe(args.zipcrypto_bytes)
|
||||
@@ -0,0 +1,77 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tarfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_files.backend.storage.archives import _inspect_archive_content, _safe_member_path, inspect_archive
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from test_archives import _tar_bytes, _zip_bytes
|
||||
|
||||
|
||||
class _HeadersOnly:
|
||||
def __init__(self, count: int, size: int = 0):
|
||||
self.count = count
|
||||
self.size = size
|
||||
self.seen = 0
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def getmembers(self):
|
||||
raise AssertionError("TAR preview must not inflate every member before checking limits")
|
||||
|
||||
def __iter__(self):
|
||||
for index in range(self.count):
|
||||
member = tarfile.TarInfo(f"file-{index}.txt")
|
||||
member.size = self.size
|
||||
self.seen += 1
|
||||
yield member
|
||||
raise AssertionError("Inspection advanced beyond the rejecting header")
|
||||
|
||||
|
||||
class ArchiveInspectionBoundTests(unittest.TestCase):
|
||||
def test_tar_entry_limit_stops_at_first_excess_header(self):
|
||||
headers = _HeadersOnly(4)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", return_value=headers):
|
||||
with self.assertRaisesRegex(FileStorageError, "too many entries"):
|
||||
_inspect_archive_content(b"fixture", filename="fixture.tar.gz", max_entries=2)
|
||||
self.assertEqual(3, headers.seen)
|
||||
|
||||
def test_tar_expanded_size_rejected_before_payload_decompression(self):
|
||||
headers = _HeadersOnly(1, size=100)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", return_value=headers):
|
||||
with self.assertRaisesRegex(FileStorageError, "too large after extraction"):
|
||||
_inspect_archive_content(b"fixture", filename="fixture.tar.gz", max_expanded_bytes=10)
|
||||
self.assertEqual(1, headers.seen)
|
||||
|
||||
def test_tar_ratio_rejected_before_payload_decompression(self):
|
||||
headers = _HeadersOnly(1, size=100)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", return_value=headers):
|
||||
with self.assertRaisesRegex(FileStorageError, "expansion ratio"):
|
||||
_inspect_archive_content(b"fixture", filename="fixture.tar.gz", max_expansion_ratio=2)
|
||||
|
||||
def test_derived_directories_are_included_in_entry_limit(self):
|
||||
for filename, payload in (("fixture.zip", _zip_bytes({"a/b/c/file.txt": b"x"})), ("fixture.tar.gz", _tar_bytes({"a/b/c/file.txt": b"x"}))):
|
||||
with self.subTest(filename=filename), self.assertRaisesRegex(FileStorageError, "including parent directories"):
|
||||
inspect_archive(payload, filename=filename, max_entries=3)
|
||||
|
||||
def test_extreme_path_depth_and_byte_lengths_fail_before_deriving_directories(self):
|
||||
for path, error in (("a/" * 128 + "file.txt", "components"), ("ü" * 2049, "UTF-8 bytes")):
|
||||
with self.subTest(path_length=len(path)), self.assertRaisesRegex(FileStorageError, error):
|
||||
inspect_archive(_zip_bytes({path: b"x"}), filename="fixture.zip")
|
||||
|
||||
def test_directory_at_exact_depth_limit_accepts_trailing_separator(self):
|
||||
path = "a/" * 128
|
||||
self.assertEqual(path.rstrip("/"), _safe_member_path(path))
|
||||
|
||||
def test_invalid_unicode_member_name_has_controlled_validation_error(self):
|
||||
with self.assertRaisesRegex(FileStorageError, "not valid Unicode"):
|
||||
_safe_member_path("invalid-\udcff.txt")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,445 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import closing
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
import random
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy import event
|
||||
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_files.backend.change_tracking import register_files_change_tracking
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
_read_selected_zip_members,
|
||||
_safe_member_path,
|
||||
extract_archive_upload,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
archive_storage_backend_scope,
|
||||
_archive_write_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.native_zip import (
|
||||
native_zip_library,
|
||||
read_native_zip_members,
|
||||
)
|
||||
from test_archives import _zip_bytes, _tar_bytes, _encrypted_zip_bytes
|
||||
import test_storage_recovery as recovery_fixture
|
||||
|
||||
|
||||
def legacy_archive(entries: dict[str, bytes]) -> bytes:
|
||||
# Fixture-only known password, never an application secret.
|
||||
with tempfile.TemporaryDirectory(prefix="files-legacy-fixture-") as temporary:
|
||||
root = Path(temporary)
|
||||
for name, content in entries.items():
|
||||
path = root / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(content)
|
||||
subprocess.run(
|
||||
[shutil.which("zip"), "-q", "-P", "fixture-only", "fixture.zip", *entries],
|
||||
cwd=root,
|
||||
check=True,
|
||||
)
|
||||
return (root / "fixture.zip").read_bytes()
|
||||
|
||||
|
||||
@unittest.skipUnless(
|
||||
shutil.which("zip") and native_zip_library() is not None,
|
||||
"optional libarchive and synthetic ZIP writer required",
|
||||
)
|
||||
class NativeArchivePerformanceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.contents = {
|
||||
"first.dat": random.Random(123).randbytes(8192),
|
||||
"folder/second.txt": b"second",
|
||||
}
|
||||
self.archive = legacy_archive(self.contents)
|
||||
|
||||
def read(self, **kwargs):
|
||||
return list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=100_000,
|
||||
max_total_bytes=100_000,
|
||||
**kwargs,
|
||||
)
|
||||
)
|
||||
|
||||
def test_native_selection_and_missing_library_python_fallback_match(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives._read_member",
|
||||
side_effect=AssertionError("Python decoder must not run"),
|
||||
):
|
||||
native = self.read()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
return_value=None,
|
||||
):
|
||||
fallback = self.read()
|
||||
self.assertEqual(native, fallback)
|
||||
self.assertEqual([("first.dat", self.contents["first.dat"])], native)
|
||||
|
||||
def test_utf8_flagged_german_names_use_native_reader(self):
|
||||
# Info-ZIP stores UTF-8 names without the language flag on some hosts.
|
||||
# Set the standard UTF-8 flag in local and central headers for this
|
||||
# fixture; encryption header verification is unaffected by bit 11.
|
||||
encoded_name = "Grüße.txt".encode("utf-8")
|
||||
placeholder = "x" * len(encoded_name)
|
||||
data = bytearray(
|
||||
legacy_archive({placeholder: b"german"}).replace(
|
||||
placeholder.encode(), encoded_name
|
||||
)
|
||||
)
|
||||
for signature, offset in ((b"PK\x03\x04", 6), (b"PK\x01\x02", 8)):
|
||||
position = data.index(signature) + offset
|
||||
flags = int.from_bytes(data[position : position + 2], "little") | 0x800
|
||||
data[position : position + 2] = flags.to_bytes(2, "little")
|
||||
self.archive = bytes(data)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives._read_member",
|
||||
side_effect=AssertionError("Python decoder must not run"),
|
||||
):
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"Grüße.txt"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=100,
|
||||
max_total_bytes=100,
|
||||
)
|
||||
)
|
||||
self.assertEqual([("Grüße.txt", b"german")], result)
|
||||
|
||||
def test_wrong_password_and_corruption_never_fall_back(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives._read_member",
|
||||
side_effect=AssertionError("No fallback after native error"),
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "Native ZIP decoding failed"):
|
||||
list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="wrong",
|
||||
max_file_bytes=100_000,
|
||||
max_total_bytes=100_000,
|
||||
)
|
||||
)
|
||||
corrupted = bytearray(self.archive)
|
||||
with pyzipper.AESZipFile(BytesIO(self.archive)) as archive:
|
||||
first = archive.infolist()[0]
|
||||
payload_start = (
|
||||
first.header_offset
|
||||
+ 30
|
||||
+ len(first.filename.encode())
|
||||
+ len(first.extra)
|
||||
)
|
||||
corrupted[payload_start + first.compress_size - 1] ^= 0x40
|
||||
self.archive = bytes(corrupted)
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.read()
|
||||
|
||||
def test_independent_crc_check_rejects_mismatched_metadata(self):
|
||||
with pyzipper.AESZipFile(BytesIO(self.archive)) as archive:
|
||||
infos = {
|
||||
_safe_member_path(info.filename): info for info in archive.infolist()
|
||||
}
|
||||
infos["first.dat"].CRC ^= 1
|
||||
with self.assertRaisesRegex(FileStorageError, "CRC verification"):
|
||||
list(
|
||||
read_native_zip_members(
|
||||
native_zip_library(),
|
||||
self.archive,
|
||||
infos_by_path=infos,
|
||||
selected_files={"first.dat"},
|
||||
normalize_name=_safe_member_path,
|
||||
password="fixture-only",
|
||||
max_file_bytes=100_000,
|
||||
max_total_bytes=100_000,
|
||||
)
|
||||
)
|
||||
|
||||
def test_legacy_encoded_non_utf8_names_choose_python_before_decoding(self):
|
||||
encoded_name = "ä.txt".encode("cp437")
|
||||
placeholder = "x" * len(encoded_name)
|
||||
self.archive = legacy_archive({placeholder: b"legacy"}).replace(
|
||||
placeholder.encode(), encoded_name
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
side_effect=AssertionError("Legacy encoding must use original reader"),
|
||||
):
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"ä.txt"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=100,
|
||||
max_total_bytes=100,
|
||||
)
|
||||
)
|
||||
self.assertEqual([("ä.txt", b"legacy")], result)
|
||||
|
||||
def test_suspended_native_reader_is_freed_after_destination_failure(self):
|
||||
library = native_zip_library()
|
||||
|
||||
class LibraryProxy:
|
||||
def __getattr__(self, name):
|
||||
return getattr(library, name)
|
||||
|
||||
proxy = LibraryProxy()
|
||||
with (
|
||||
patch.object(
|
||||
proxy, "archive_read_free", wraps=library.archive_read_free
|
||||
) as close,
|
||||
patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
return_value=proxy,
|
||||
),
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "Destination failure"):
|
||||
# This unit test owns the native handle locally; public imports
|
||||
# now own it in a fresh child (covered by worker cleanup tests).
|
||||
with closing(_read_selected_zip_members(
|
||||
self.archive, selected_files=set(self.contents), password="fixture-only",
|
||||
max_file_bytes=100_000, max_total_bytes=100_000,
|
||||
)) as members:
|
||||
next(members)
|
||||
raise FileStorageError("Destination failure")
|
||||
close.assert_called_once()
|
||||
|
||||
def test_native_read_enforces_actual_member_and_total_limits(self):
|
||||
for member_limit, total_limit in ((10, 100_000), (100_000, 10)):
|
||||
with self.subTest(member_limit=member_limit, total_limit=total_limit):
|
||||
with self.assertRaises(FileStorageError):
|
||||
list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=member_limit,
|
||||
max_total_bytes=total_limit,
|
||||
)
|
||||
)
|
||||
|
||||
def test_native_emits_progress_inside_a_large_member(self):
|
||||
self.archive = legacy_archive(
|
||||
{"first.dat": random.Random(234).randbytes(3 * 1024 * 1024)}
|
||||
)
|
||||
progress = []
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
self.archive,
|
||||
selected_files={"first.dat"},
|
||||
password="fixture-only",
|
||||
max_file_bytes=4 * 1024 * 1024,
|
||||
max_total_bytes=4 * 1024 * 1024,
|
||||
total_bytes=3 * 1024 * 1024,
|
||||
progress=lambda *args: progress.append(args),
|
||||
)
|
||||
)
|
||||
self.assertEqual(3 * 1024 * 1024, len(result[0][1]))
|
||||
self.assertTrue(
|
||||
any(item[1] == 0 and 0 < item[3] < 3 * 1024 * 1024 for item in progress)
|
||||
)
|
||||
self.assertEqual(
|
||||
("extracting", 1, 1, 3 * 1024 * 1024, 3 * 1024 * 1024), progress[-1]
|
||||
)
|
||||
|
||||
|
||||
class ArchiveStreamingTests(unittest.TestCase):
|
||||
def test_plain_zip_and_tar_store_one_member_before_reading_the_next(self):
|
||||
entries = {"one.txt": b"one", "two.txt": b"two"}
|
||||
for filename, archive in (
|
||||
("files.zip", _zip_bytes(entries)),
|
||||
("files.tar.gz", _tar_bytes(entries)),
|
||||
):
|
||||
with self.subTest(filename=filename):
|
||||
progress = []
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset",
|
||||
return_value=SimpleNamespace(asset=object()),
|
||||
) as create:
|
||||
result = extract_archive_upload(
|
||||
object(),
|
||||
tenant_id="tenant",
|
||||
owner_type="user",
|
||||
owner_id="user",
|
||||
user_id="user",
|
||||
archive_data=archive,
|
||||
filename=filename,
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
progress=lambda *args: progress.append(args),
|
||||
)
|
||||
self.assertEqual(2, len(result))
|
||||
self.assertEqual(2, create.call_count)
|
||||
stored_first = progress.index(("storing", 1, 2, 3, 6))
|
||||
extracted_second = progress.index(("extracting", 2, 2, 6, 6))
|
||||
self.assertLess(stored_first, extracted_second)
|
||||
|
||||
def test_aes_archive_keeps_existing_decoder(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.native_zip_library",
|
||||
side_effect=AssertionError("AES must not use native path"),
|
||||
):
|
||||
result = list(
|
||||
_read_selected_zip_members(
|
||||
_encrypted_zip_bytes("correct"),
|
||||
selected_files={"secure/report.txt"},
|
||||
password="correct",
|
||||
max_file_bytes=100,
|
||||
max_total_bytes=100,
|
||||
)
|
||||
)
|
||||
self.assertEqual([("secure/report.txt", b"classified")], result)
|
||||
|
||||
def test_late_unsafe_header_prevents_every_write(self):
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset"
|
||||
) as create:
|
||||
with self.assertRaisesRegex(FileStorageError, "Unsafe archive member"):
|
||||
extract_archive_upload(
|
||||
object(),
|
||||
tenant_id="tenant",
|
||||
owner_type="user",
|
||||
owner_id="user",
|
||||
user_id="user",
|
||||
archive_data=_zip_bytes({"good.txt": b"good", "../unsafe": b"bad"}),
|
||||
filename="unsafe.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
)
|
||||
create.assert_not_called()
|
||||
|
||||
|
||||
class ArchivePersistencePerformanceTests(unittest.TestCase):
|
||||
def test_backend_scope_is_lazy_nested_and_reset_after_exception(self):
|
||||
first_backend = object()
|
||||
second_backend = object()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
side_effect=[first_backend, second_backend],
|
||||
) as factory:
|
||||
with self.assertRaisesRegex(RuntimeError, "fixture failure"):
|
||||
with archive_storage_backend_scope():
|
||||
factory.assert_not_called()
|
||||
self.assertIs(first_backend, _archive_write_backend())
|
||||
with archive_storage_backend_scope():
|
||||
self.assertIs(first_backend, _archive_write_backend())
|
||||
factory.assert_called_once()
|
||||
raise RuntimeError("fixture failure")
|
||||
with archive_storage_backend_scope():
|
||||
self.assertIs(second_backend, _archive_write_backend())
|
||||
self.assertEqual(2, factory.call_count)
|
||||
|
||||
def test_backend_reuse_does_not_cache_tenant_identity_or_blob_dedup(self):
|
||||
fixture = recovery_fixture.StorageRecoveryTests()
|
||||
fixture.setUp()
|
||||
self.addCleanup(fixture.doCleanups)
|
||||
results = []
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=fixture.backend,
|
||||
) as factory:
|
||||
for tenant in ("tenant-one", "tenant-two"):
|
||||
results.append(
|
||||
extract_archive_upload(
|
||||
fixture.session,
|
||||
tenant_id=tenant,
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=_zip_bytes({"one.txt": b"one", "two.txt": b"two"}),
|
||||
filename="files.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
)
|
||||
)
|
||||
fixture.session.commit()
|
||||
self.assertEqual(2, factory.call_count)
|
||||
self.assertEqual(
|
||||
["tenant-one", "tenant-two"], [items[0].blob.tenant_id for items in results]
|
||||
)
|
||||
self.assertNotEqual(results[0][0].blob.id, results[1][0].blob.id)
|
||||
|
||||
def test_asset_and_version_are_published_once_without_redundant_update(self):
|
||||
fixture = recovery_fixture.StorageRecoveryTests()
|
||||
fixture.setUp()
|
||||
self.addCleanup(fixture.doCleanups)
|
||||
register_files_change_tracking()
|
||||
queries = []
|
||||
event.listen(
|
||||
fixture.engine,
|
||||
"before_cursor_execute",
|
||||
lambda conn, cursor, statement, parameters, context, executemany: (
|
||||
queries.append(statement)
|
||||
),
|
||||
)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=fixture.backend,
|
||||
):
|
||||
stored = extract_archive_upload(
|
||||
fixture.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=_zip_bytes({"one.txt": b"one", "two.txt": b"two"}),
|
||||
filename="files.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
)
|
||||
fixture.session.commit()
|
||||
self.assertFalse(
|
||||
any(statement.startswith("UPDATE file_assets") for statement in queries)
|
||||
)
|
||||
events = (
|
||||
fixture.session.query(ChangeSequenceEntry)
|
||||
.filter(ChangeSequenceEntry.module_id == "files")
|
||||
.all()
|
||||
)
|
||||
self.assertEqual(["created", "created"], [entry.operation for entry in events])
|
||||
self.assertTrue(
|
||||
all(item.asset.current_version_id == item.version.id for item in stored)
|
||||
)
|
||||
|
||||
def test_streamed_late_content_limit_failure_rolls_back_previous_blobs(self):
|
||||
fixture = recovery_fixture.StorageRecoveryTests()
|
||||
fixture.setUp()
|
||||
self.addCleanup(fixture.doCleanups)
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=fixture.backend,
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "per-file limit"):
|
||||
extract_archive_upload(
|
||||
fixture.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
archive_data=_zip_bytes(
|
||||
{"one.txt": b"one", "too-large.txt": b"x" * 200}
|
||||
),
|
||||
filename="files.zip",
|
||||
folder="",
|
||||
campaign_id=None,
|
||||
max_file_bytes=100,
|
||||
)
|
||||
fixture.session.rollback()
|
||||
self.assertEqual([], list(fixture.backend.root.rglob("*.blob")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,315 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from govoplan_core.auth import get_api_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_core.security.secrets import open_transient_payload, seal_transient_payload
|
||||
from govoplan_files.backend.archive_work import ArchiveProgress, use_staged_upload
|
||||
from govoplan_files.backend import archive_work
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.routes import uploads
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
import test_managed_archives as managed_archive_tests
|
||||
|
||||
|
||||
class ArchiveStagingTests(unittest.TestCase):
|
||||
store_blob = managed_archive_tests.ManagedArchiveTests.store_blob
|
||||
seed_source = managed_archive_tests.ManagedArchiveTests.seed_source
|
||||
assert_no_extraction = managed_archive_tests.ManagedArchiveTests.assert_no_extraction
|
||||
|
||||
def setUp(self):
|
||||
managed_archive_tests.ManagedArchiveTests.setUp(self)
|
||||
self.directory = Path(self.enterContext(tempfile.TemporaryDirectory(prefix="archive-staging-tests-")))
|
||||
self.settings.file_archive_work_root = str(self.directory / "work")
|
||||
self.settings.file_archive_staged_per_actor = 4
|
||||
self.settings.file_archive_staged_max_bytes = 20 * 1024 * 1024
|
||||
app = FastAPI()
|
||||
app.include_router(uploads.router, prefix="/api/v1")
|
||||
app.dependency_overrides[get_api_principal] = lambda: self.principal
|
||||
app.dependency_overrides[get_session] = lambda: self.session
|
||||
self.client = self.enterContext(TestClient(app, raise_server_exceptions=False))
|
||||
|
||||
def preview(self, *, archive=None, **changes):
|
||||
return self.client.post(
|
||||
"/api/v1/files/archive-preview",
|
||||
data={"path": "extracted", "retain_upload": "true", **changes},
|
||||
files={"file": ("private-source.zip", archive or managed_archive_tests.archive_bytes(), "application/zip")},
|
||||
)
|
||||
|
||||
def staged_preview(self, preview, **changes):
|
||||
return self.client.post(
|
||||
"/api/v1/files/archive-preview",
|
||||
data={"path": "extracted", "preview_token": preview["preview_token"], "staged_upload_id": preview["staged_upload_id"], **changes},
|
||||
)
|
||||
|
||||
def confirm(self, preview, **changes):
|
||||
return self.client.post(
|
||||
"/api/v1/files/archive-confirm",
|
||||
data={"path": "extracted", "preview_token": preview["preview_token"], "staged_upload_id": preview["staged_upload_id"], "selected_paths_json": json.dumps(["folder/one.txt"]), **changes},
|
||||
)
|
||||
|
||||
def staged_path(self, preview):
|
||||
with use_staged_upload(self.settings, preview["staged_upload_id"], tenant_id="tenant-1", user_id="user-1") as filename:
|
||||
return Path(filename)
|
||||
|
||||
def progress(self, operation_id):
|
||||
return self.client.get(f"/api/v1/files/archive-progress/{operation_id}")
|
||||
|
||||
def test_preview_stages_once_password_repreview_and_confirm_reuse_upload(self):
|
||||
password = "staged secret password"
|
||||
preview = self.preview(archive=managed_archive_tests.archive_bytes(password=password))
|
||||
self.assertEqual(preview.status_code, 200, preview.text)
|
||||
self.assertTrue(preview.json()["requires_password"])
|
||||
self.assertFalse(preview.json()["password_verified"])
|
||||
stage = self.staged_path(preview.json())
|
||||
self.assert_no_extraction()
|
||||
operation_id = str(uuid4())
|
||||
with patch.object(uploads, "_spool_limited_upload_to_temp") as spool:
|
||||
wrong = self.staged_preview(preview.json(), password="incorrect")
|
||||
self.assertEqual(wrong.status_code, 400, wrong.text)
|
||||
self.assertTrue(stage.exists())
|
||||
verified = self.staged_preview(preview.json(), password=password)
|
||||
self.assertEqual(verified.status_code, 200, verified.text)
|
||||
self.assertTrue(verified.json()["password_verified"])
|
||||
self.assertEqual(verified.json()["staged_upload_id"], preview.json()["staged_upload_id"])
|
||||
confirmed = self.confirm(verified.json(), password=password, operation_id=operation_id)
|
||||
spool.assert_not_called()
|
||||
self.assertEqual(confirmed.status_code, 200, confirmed.text)
|
||||
self.assertEqual([item["display_path"] for item in confirmed.json()["files"]], ["extracted/folder/one.txt"])
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
self.assertFalse(stage.exists())
|
||||
progress = self.progress(operation_id)
|
||||
self.assertEqual(progress.status_code, 200, progress.text)
|
||||
self.assertEqual(progress.json()["status"], "complete")
|
||||
for value in (verified.text, confirmed.text, progress.text, str(self.audit.call_args_list)):
|
||||
self.assertNotIn(password, value)
|
||||
self.assertNotIn("private-source.zip", progress.text)
|
||||
self.assertNotIn("folder/one.txt", progress.text)
|
||||
payload = open_transient_payload(verified.json()["preview_token"], ttl_seconds=1800)
|
||||
self.assertNotIn(password, json.dumps(payload))
|
||||
|
||||
def test_staging_rejects_token_actor_tenant_stage_destination_and_digest_mismatch_without_writes(self):
|
||||
response = self.preview()
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
preview = response.json()
|
||||
for change in ({"preview_token": "tampered"}, {"staged_upload_id": str(uuid4())}, {"owner_id": "other-user"}, {"path": "elsewhere"}):
|
||||
with self.subTest(change=change):
|
||||
rejected = self.confirm(preview, **change)
|
||||
self.assertEqual(rejected.status_code, 400, rejected.text)
|
||||
self.assert_no_extraction()
|
||||
for field, original, foreign in (("tenant_id", "tenant-1", "other-tenant"), ("user_id", "user-1", "other-user")):
|
||||
target = self.principal if field == "tenant_id" else self.principal.user
|
||||
attribute = "tenant_id" if field == "tenant_id" else "id"
|
||||
setattr(target, attribute, foreign)
|
||||
self.assertEqual(self.confirm(preview).status_code, 400)
|
||||
setattr(target, attribute, original)
|
||||
path = self.staged_path(preview)
|
||||
path.write_bytes(b"tampered archive bytes")
|
||||
for response in (self.staged_preview(preview), self.confirm(preview)):
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
self.assertIn("contents changed", response.text)
|
||||
self.assert_no_extraction()
|
||||
self.assertTrue(path.exists())
|
||||
self.assertEqual(list(path.parent.glob("*.progress")), [])
|
||||
|
||||
def test_expired_stage_and_explicit_discard_are_bound_to_current_actor(self):
|
||||
preview = self.preview().json()
|
||||
path = self.staged_path(preview)
|
||||
self.principal.user.id = "other-user"
|
||||
self.assertEqual(self.client.delete(f"/api/v1/files/archive-staging/{preview['staged_upload_id']}").status_code, 204)
|
||||
self.assertTrue(path.exists())
|
||||
self.principal.user.id = "user-1"
|
||||
self.assertEqual(self.client.delete(f"/api/v1/files/archive-staging/{preview['staged_upload_id']}").status_code, 204)
|
||||
self.assertFalse(path.exists())
|
||||
self.assertEqual(self.confirm(preview).status_code, 410)
|
||||
preview = self.preview().json()
|
||||
path = self.staged_path(preview)
|
||||
old = time.time() - self.settings.file_archive_preview_ttl_seconds - 1
|
||||
os.utime(path, (old, old))
|
||||
self.assertEqual(self.confirm(preview).status_code, 410)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_progress_stays_running_through_commit_and_completes_only_after_commit(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
original_commit = self.session.commit
|
||||
observed = []
|
||||
|
||||
def commit():
|
||||
before = self.progress(operation_id).json()
|
||||
self.assertEqual(before["status"], "running")
|
||||
self.assertEqual(before["phase"], "finalizing")
|
||||
original_commit()
|
||||
observed.append(self.progress(operation_id).json()["status"])
|
||||
|
||||
with patch.object(self.session, "commit", side_effect=commit):
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
self.assertEqual(observed, ["running"])
|
||||
progress = self.progress(operation_id).json()
|
||||
self.assertEqual(progress["status"], "complete")
|
||||
self.assertEqual(progress["completed_files"], 1)
|
||||
self.assertEqual(progress["completed_bytes"], 3)
|
||||
|
||||
def test_commit_error_marks_progress_failed_and_preserves_stage_for_retry(self):
|
||||
preview = self.preview().json()
|
||||
stage = self.staged_path(preview)
|
||||
operation_id = str(uuid4())
|
||||
with patch.object(self.session, "commit", side_effect=FileStorageError("Commit refused")):
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "failed")
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 1)
|
||||
self.assertTrue(stage.exists())
|
||||
retry = self.confirm(preview, operation_id=str(uuid4()))
|
||||
self.assertEqual(retry.status_code, 200, retry.text)
|
||||
|
||||
def test_post_commit_stage_cleanup_failure_does_not_report_a_failed_import(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
with patch.object(uploads, "discard_staged_upload", side_effect=OSError("Temporary cleanup failure")):
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "complete")
|
||||
|
||||
def test_duplicate_operation_cannot_start_another_write_or_overwrite_progress(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
progress = ArchiveProgress(self.settings, operation_id, tenant_id="tenant-1", user_id="user-1")
|
||||
progress("extracting", 1, 5, 10, 50)
|
||||
before = self.progress(operation_id).json()
|
||||
response = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
self.assertIn("already been used", response.text)
|
||||
self.assertEqual(self.progress(operation_id).json(), before)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_progress_and_discard_require_upload_permission_and_do_not_disclose_foreign_receipts(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
ArchiveProgress(self.settings, operation_id, tenant_id="tenant-1", user_id="user-1")
|
||||
self.principal.user.id = "other-user"
|
||||
self.assertEqual(self.progress(operation_id).status_code, 404)
|
||||
self.principal.user.id = "user-1"
|
||||
self.scopes.remove("files:file:upload")
|
||||
self.assertEqual(self.progress(operation_id).status_code, 403)
|
||||
self.assertEqual(self.client.delete(f"/api/v1/files/archive-staging/{preview['staged_upload_id']}").status_code, 403)
|
||||
self.assertTrue(self.staged_path(preview).exists())
|
||||
|
||||
def test_malformed_signed_digest_cannot_bypass_stage_binding(self):
|
||||
preview = self.preview().json()
|
||||
payload = open_transient_payload(preview["preview_token"], ttl_seconds=1800)
|
||||
payload["archive_sha256"] = None
|
||||
invalid = self.confirm(preview, preview_token=seal_transient_payload(payload))
|
||||
self.assertEqual(invalid.status_code, 400, invalid.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_progress_persistence_failure_does_not_turn_a_committed_import_into_an_error(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
original_replace = archive_work.os.replace
|
||||
|
||||
def replace(source, destination):
|
||||
# Only the optional UI progress receipt is unavailable. The private
|
||||
# worker ack is a required integrity/backpressure channel, not UI
|
||||
# progress, and must not be disabled by this module-global mock.
|
||||
if Path(source).name.startswith(".progress-"):
|
||||
raise OSError("Progress write unavailable")
|
||||
return original_replace(source, destination)
|
||||
|
||||
with patch.object(archive_work.os, "replace", side_effect=replace):
|
||||
result = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(result.status_code, 200, result.text)
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
# A failed observation write may leave an older running receipt, but
|
||||
# the confirmation response remains the authoritative outcome.
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "running")
|
||||
|
||||
def test_required_worker_ack_failure_rolls_back_and_preserves_upload_for_explicit_retry(self):
|
||||
preview = self.preview().json()
|
||||
stage = self.staged_path(preview)
|
||||
operation_id = str(uuid4())
|
||||
with patch("govoplan_files.backend.storage.archive_workers._send_ack",
|
||||
side_effect=FileStorageError("Archive worker staging channel is unavailable")):
|
||||
result = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(400, result.status_code, result.text)
|
||||
self.assertIn("staging channel is unavailable", result.text)
|
||||
self.assert_no_extraction()
|
||||
self.assertEqual("failed", self.progress(operation_id).json()["status"])
|
||||
self.assertTrue(stage.exists())
|
||||
result = self.confirm(preview, operation_id=str(uuid4()))
|
||||
self.assertEqual(200, result.status_code, result.text)
|
||||
|
||||
def test_lease_cleanup_failure_cannot_mask_a_committed_import(self):
|
||||
preview = self.preview().json()
|
||||
operation_id = str(uuid4())
|
||||
original_unlink = Path.unlink
|
||||
|
||||
def unlink(path, *args, **kwargs):
|
||||
if path.suffix == ".lease":
|
||||
raise OSError("Lease cleanup temporarily unavailable")
|
||||
return original_unlink(path, *args, **kwargs)
|
||||
|
||||
with patch.object(Path, "unlink", new=unlink):
|
||||
result = self.confirm(preview, operation_id=operation_id)
|
||||
self.assertEqual(result.status_code, 200, result.text)
|
||||
self.assertEqual(self.session.query(FileAsset).count(), 2)
|
||||
self.assertEqual(self.progress(operation_id).json()["status"], "complete")
|
||||
|
||||
def test_invalid_operation_identifier_and_ambiguous_source_are_rejected_before_writes(self):
|
||||
preview = self.preview().json()
|
||||
response = self.confirm(preview, operation_id="../../outside")
|
||||
self.assertEqual(response.status_code, 400, response.text)
|
||||
ambiguous = self.client.post(
|
||||
"/api/v1/files/archive-confirm",
|
||||
data={"path": "extracted", "preview_token": preview["preview_token"], "staged_upload_id": preview["staged_upload_id"], "selected_paths_json": '["folder"]'},
|
||||
files={"file": ("other.zip", managed_archive_tests.archive_bytes(), "application/zip")},
|
||||
)
|
||||
self.assertEqual(ambiguous.status_code, 400, ambiguous.text)
|
||||
self.assertIn("either", ambiguous.text)
|
||||
self.assert_no_extraction()
|
||||
self.assertTrue(self.staged_path(preview).exists())
|
||||
|
||||
def test_staged_preview_requires_a_token_without_exposing_a_server_error(self):
|
||||
preview = self.preview().json()
|
||||
missing = self.client.post(
|
||||
"/api/v1/files/archive-preview",
|
||||
data={"path": "extracted", "staged_upload_id": preview["staged_upload_id"]},
|
||||
)
|
||||
self.assertIn(missing.status_code, (400, 422), missing.text)
|
||||
self.assertTrue(self.staged_path(preview).exists())
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_repreview_does_not_extend_the_original_stage_expiry(self):
|
||||
preview = self.preview().json()
|
||||
path = self.staged_path(preview)
|
||||
original_mtime = time.time() - 120
|
||||
os.utime(path, (original_mtime, original_mtime))
|
||||
original_deadline = path.stat().st_mtime + self.settings.file_archive_preview_ttl_seconds
|
||||
for _ in range(2):
|
||||
response = self.staged_preview(preview)
|
||||
self.assertEqual(response.status_code, 200, response.text)
|
||||
preview = response.json()
|
||||
returned_deadline = datetime.fromisoformat(preview["expires_at"]).timestamp()
|
||||
self.assertLessEqual(returned_deadline, original_deadline + 0.000001)
|
||||
self.assertAlmostEqual(returned_deadline, original_deadline, places=5)
|
||||
self.assertEqual(path.stat().st_mtime, original_mtime)
|
||||
self.assert_no_extraction()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,191 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from uuid import uuid4
|
||||
|
||||
from govoplan_files.backend import archive_work
|
||||
from govoplan_files.backend.archive_work import (
|
||||
ArchiveProgress,
|
||||
ArchiveWorkExpired,
|
||||
discard_staged_upload,
|
||||
read_progress,
|
||||
stage_upload,
|
||||
use_staged_upload,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
class ArchiveWorkTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.directory = Path(self.enterContext(tempfile.TemporaryDirectory(prefix="archive-work-tests-")))
|
||||
self.settings = SimpleNamespace(
|
||||
file_archive_work_root=str(self.directory / "work"),
|
||||
file_archive_preview_ttl_seconds=60,
|
||||
file_archive_staged_per_actor=2,
|
||||
file_archive_staged_max_bytes=100,
|
||||
)
|
||||
self.source = self.directory / "source.zip"
|
||||
self.source.write_bytes(b"archive bytes")
|
||||
self.actor = {"tenant_id": "tenant-1", "user_id": "user-1"}
|
||||
|
||||
def stage(self, **actor):
|
||||
return stage_upload(self.settings, str(self.source), **(self.actor | actor))
|
||||
|
||||
def use(self, stage_id, **actor):
|
||||
return use_staged_upload(self.settings, stage_id, **(self.actor | actor))
|
||||
|
||||
def progress(self, operation_id=None, **actor):
|
||||
return ArchiveProgress(self.settings, operation_id or str(uuid4()), **(self.actor | actor))
|
||||
|
||||
def test_staging_is_private_and_actor_and_tenant_bound(self):
|
||||
stage_id = self.stage()
|
||||
with self.use(stage_id) as filename:
|
||||
staged = Path(filename)
|
||||
self.assertEqual(staged.read_bytes(), self.source.read_bytes())
|
||||
self.assertEqual(staged.stat().st_mode & 0o777, 0o600)
|
||||
self.assertEqual(staged.parent.stat().st_mode & 0o777, 0o700)
|
||||
for actor in ({"user_id": "other-user"}, {"tenant_id": "other-tenant"}):
|
||||
with self.subTest(actor=actor):
|
||||
with self.assertRaises(ArchiveWorkExpired), self.use(stage_id, **actor):
|
||||
self.fail("Foreign actor obtained a staged archive")
|
||||
discard_staged_upload(self.settings, stage_id, **(self.actor | actor))
|
||||
with self.use(stage_id) as filename:
|
||||
self.assertTrue(Path(filename).is_file())
|
||||
|
||||
def test_noncanonical_identifiers_cannot_escape_the_private_root(self):
|
||||
stage_id = self.stage()
|
||||
for invalid in ("../source.zip", "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", stage_id.replace("-", ""), "", "not-a-uuid"):
|
||||
with self.subTest(invalid=invalid):
|
||||
with self.assertRaises(FileStorageError), self.use(invalid):
|
||||
self.fail("Invalid identifier accepted")
|
||||
with self.assertRaises(FileStorageError):
|
||||
discard_staged_upload(self.settings, invalid, **self.actor)
|
||||
with self.assertRaises(FileStorageError):
|
||||
read_progress(self.settings, invalid, **self.actor)
|
||||
self.assertEqual(self.source.read_bytes(), b"archive bytes")
|
||||
|
||||
def test_expired_stage_and_progress_are_not_readable(self):
|
||||
stage_id = self.stage()
|
||||
with self.use(stage_id) as filename:
|
||||
staged = Path(filename)
|
||||
progress = self.progress()
|
||||
old = time.time() - self.settings.file_archive_preview_ttl_seconds - 1
|
||||
os.utime(staged, (old, old))
|
||||
os.utime(progress.path, (old, old))
|
||||
with self.assertRaises(ArchiveWorkExpired), self.use(stage_id):
|
||||
self.fail("Expired stage accepted")
|
||||
with self.assertRaises(ArchiveWorkExpired):
|
||||
read_progress(self.settings, progress.path.stem[-36:], **self.actor)
|
||||
|
||||
def test_quota_evicts_oldest_unleased_stage_and_explicit_discard_is_idempotent(self):
|
||||
first = self.stage()
|
||||
with self.use(first) as filename:
|
||||
first_path = Path(filename)
|
||||
old = time.time() - 5
|
||||
os.utime(first_path, (old, old))
|
||||
second = self.stage()
|
||||
third = self.stage()
|
||||
self.assertFalse(first_path.exists())
|
||||
for stage_id in (second, third):
|
||||
with self.use(stage_id):
|
||||
pass
|
||||
discard_staged_upload(self.settings, third, **self.actor)
|
||||
discard_staged_upload(self.settings, third, **self.actor)
|
||||
with self.assertRaises(ArchiveWorkExpired), self.use(third):
|
||||
self.fail("Discarded archive retained")
|
||||
|
||||
def test_active_lease_blocks_duplicate_processing_discard_and_quota_eviction(self):
|
||||
self.settings.file_archive_staged_per_actor = 1
|
||||
first = self.stage()
|
||||
with self.use(first) as filename:
|
||||
with self.assertRaisesRegex(FileStorageError, "already being processed"), self.use(first):
|
||||
self.fail("Duplicate lease accepted")
|
||||
discard_staged_upload(self.settings, first, **self.actor)
|
||||
self.assertTrue(Path(filename).exists())
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.stage()
|
||||
with self.use(first):
|
||||
pass
|
||||
|
||||
def test_long_running_lease_remains_protected_after_preview_ttl(self):
|
||||
self.settings.file_archive_staged_per_actor = 1
|
||||
first = self.stage()
|
||||
with self.use(first) as filename:
|
||||
path = Path(filename)
|
||||
old = time.time() - self.settings.file_archive_preview_ttl_seconds - 1
|
||||
os.utime(path, (old, old))
|
||||
os.utime(path.with_suffix(".lease"), (old, old))
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.stage()
|
||||
self.assertTrue(path.exists(), "A live import must remain quota-accounted even after its preview TTL")
|
||||
|
||||
def test_global_byte_quota_does_not_evict_other_actor_archives(self):
|
||||
first = self.stage()
|
||||
self.settings.file_archive_staged_max_bytes = len(self.source.read_bytes())
|
||||
with self.assertRaisesRegex(FileStorageError, "storage is full"):
|
||||
self.stage(user_id="other-user")
|
||||
with self.use(first):
|
||||
pass
|
||||
|
||||
def test_progress_is_private_actor_bound_and_duplicate_operation_is_rejected(self):
|
||||
operation_id = str(uuid4())
|
||||
progress = self.progress(operation_id)
|
||||
progress("extracting", 2, 5, 20, 50)
|
||||
for actor in ({"user_id": "other-user"}, {"tenant_id": "other-tenant"}):
|
||||
with self.assertRaises(ArchiveWorkExpired):
|
||||
read_progress(self.settings, operation_id, **(self.actor | actor))
|
||||
before = read_progress(self.settings, operation_id, **self.actor)
|
||||
with self.assertRaisesRegex(FileStorageError, "already been used"):
|
||||
self.progress(operation_id)
|
||||
self.assertEqual(read_progress(self.settings, operation_id, **self.actor), before)
|
||||
self.assertEqual(set(before), {"phase", "status", "completed_files", "total_files", "completed_bytes", "total_bytes"})
|
||||
self.assertEqual(before["status"], "running")
|
||||
progress.finish(True)
|
||||
self.assertEqual(read_progress(self.settings, operation_id, **self.actor)["status"], "complete")
|
||||
with self.assertRaises(FileStorageError):
|
||||
self.progress(operation_id)
|
||||
|
||||
def test_progress_write_failure_never_interrupts_an_import(self):
|
||||
progress = self.progress()
|
||||
with patch.object(archive_work.os, "replace", side_effect=OSError("Temporary storage unavailable")):
|
||||
progress("extracting", 1, 2, 10, 20)
|
||||
progress.finish(True)
|
||||
self.assertEqual(progress.value["status"], "complete")
|
||||
self.assertEqual(list(progress.path.parent.glob(".progress-*")), [])
|
||||
self.assertEqual(json.loads(progress.path.read_text())["status"], "running")
|
||||
|
||||
def test_missing_optional_operation_id_does_not_create_a_work_directory(self):
|
||||
progress = ArchiveProgress(self.settings, None, **self.actor)
|
||||
progress("extracting", 1, 1, 10, 10)
|
||||
progress.finish(True)
|
||||
self.assertIsNone(progress.path)
|
||||
self.assertFalse(Path(self.settings.file_archive_work_root).exists())
|
||||
|
||||
def test_work_directory_with_public_permissions_fails_closed(self):
|
||||
root = Path(self.settings.file_archive_work_root) / "v1"
|
||||
root.mkdir(parents=True, mode=0o755)
|
||||
root.chmod(0o755)
|
||||
with self.assertRaisesRegex(FileStorageError, "must be private"):
|
||||
self.stage()
|
||||
self.assertEqual(list(root.glob("*.archive")), [])
|
||||
|
||||
def test_orphan_lease_does_not_permanently_block_staging(self):
|
||||
stage_id = self.stage()
|
||||
with self.use(stage_id) as filename:
|
||||
staged = Path(filename)
|
||||
# Simulate the lock file left behind after a worker exits: no live
|
||||
# process holds its kernel lock, so a new request can reclaim it.
|
||||
staged.with_suffix(".lease").touch(mode=0o600)
|
||||
with self.use(stage_id):
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,346 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
import json
|
||||
import inspect
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import tracemalloc
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import zlib
|
||||
|
||||
from govoplan_core.security import bounded_process
|
||||
from govoplan_core.settings import settings
|
||||
from govoplan_files.backend.storage import archive_workers as workers
|
||||
from govoplan_files.backend.storage.archives import extract_archive_upload, inspect_archive
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from test_archives import _zip_bytes
|
||||
|
||||
|
||||
def _cpu_exhaustion_worker(payload: bytes) -> bytes:
|
||||
# A real imported child operation proves the Files wrapper's CPU error path;
|
||||
# the metadata tests below execute the actual archive parser, not this probe.
|
||||
while True:
|
||||
pass
|
||||
|
||||
|
||||
def _extract(payload, **options):
|
||||
return extract_archive_upload(
|
||||
object(), tenant_id="tenant", owner_type="user", owner_id="owner",
|
||||
user_id="owner", filename="fixture.zip", archive_data=payload,
|
||||
folder="destination", campaign_id=None, **options,
|
||||
)
|
||||
|
||||
|
||||
class ArchiveWorkerTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.directories = []
|
||||
self.children = []
|
||||
real_directory = tempfile.TemporaryDirectory
|
||||
real_popen = subprocess.Popen
|
||||
|
||||
def directory(*args, **kwargs):
|
||||
context = real_directory(*args, **kwargs)
|
||||
self.directories.append(Path(context.name))
|
||||
return context
|
||||
|
||||
def popen(*args, **kwargs):
|
||||
child = real_popen(*args, **kwargs)
|
||||
self.children.append(child)
|
||||
return child
|
||||
|
||||
self.enterContext(patch.object(workers.tempfile, "TemporaryDirectory", side_effect=directory))
|
||||
self.enterContext(patch.object(bounded_process.subprocess, "Popen", side_effect=popen))
|
||||
self.store = self.enterContext(patch(
|
||||
"govoplan_files.backend.storage.archives.create_file_asset",
|
||||
return_value=SimpleNamespace(asset=SimpleNamespace(id="asset")),
|
||||
))
|
||||
|
||||
def tearDown(self):
|
||||
self.assertTrue(all(not directory.exists() for directory in self.directories))
|
||||
self.assertTrue(all(child.returncode is not None for child in self.children))
|
||||
|
||||
def test_parent_never_parses_metadata_or_extracts_and_one_child_handles_all_members(self):
|
||||
payload = _zip_bytes({"one.txt": b"one", "two.txt": b"two"})
|
||||
with (
|
||||
patch("govoplan_files.backend.storage.archives._inspect_archive_content", side_effect=AssertionError("parent parser")),
|
||||
patch("govoplan_files.backend.storage.archives._read_selected_zip_members", side_effect=AssertionError("parent decoder")),
|
||||
):
|
||||
self.assertEqual(2, inspect_archive(payload, filename="fixture.zip").file_count)
|
||||
self.assertEqual(2, len(_extract(payload)))
|
||||
self.assertEqual(2, len(self.children)) # one preview, one whole extraction
|
||||
self.assertEqual([b"one", b"two"], [call.kwargs["data"] for call in self.store.call_args_list])
|
||||
|
||||
def test_hostile_pax_metadata_allocation_is_confined_before_any_store(self):
|
||||
header = tarfile.TarInfo("pax")
|
||||
header.type = tarfile.XHDTYPE
|
||||
header.size = 1024 * 1024 * 1024
|
||||
payload = header.tobuf() + b"\0" * 1024
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", side_effect=AssertionError("parent TAR parser")):
|
||||
with self.assertRaisesRegex(FileStorageError, "memory_limit|Invalid TAR|could not complete safely"):
|
||||
inspect_archive(payload, filename="hostile.tar")
|
||||
self.store.assert_not_called()
|
||||
self.assertEqual(1, len(self.children))
|
||||
# A rejected archive neither consumes the admission slot permanently nor
|
||||
# prevents a later ordinary parse in the same parent process.
|
||||
self.assertEqual(1, inspect_archive(_zip_bytes({"ok": b"ok"}), filename="ok.zip").file_count)
|
||||
|
||||
def test_compressed_pax_metadata_hits_real_child_memory_limit(self):
|
||||
header = tarfile.TarInfo("pax")
|
||||
header.type = tarfile.XHDTYPE
|
||||
header.size = 1024 * 1024 * 1024
|
||||
compressor = zlib.compressobj(1, wbits=31)
|
||||
chunks = [compressor.compress(header.tobuf())]
|
||||
block = b"\0" * (1024 * 1024)
|
||||
# Build the hostile fixture incrementally: ~2.3 MiB compressed, never a
|
||||
# 512 MiB parent allocation. TAR consumes PAX metadata before ordinary
|
||||
# returned-member limits can inspect it; the child AS limit must win.
|
||||
for _ in range(512):
|
||||
chunks.append(compressor.compress(block))
|
||||
chunks.append(compressor.flush())
|
||||
payload = b"".join(chunks)
|
||||
self.assertLess(len(payload), 3 * 1024 * 1024)
|
||||
with patch("govoplan_files.backend.storage.archives._open_tar", side_effect=AssertionError("parent TAR parser")):
|
||||
with self.assertRaisesRegex(FileStorageError, "memory_limit"):
|
||||
inspect_archive(payload, filename="hostile.tar.gz")
|
||||
self.store.assert_not_called()
|
||||
self.assertEqual(1, len(self.children))
|
||||
|
||||
def test_real_child_timeout_reaps_and_removes_snapshot(self):
|
||||
with patch.object(workers, "INSPECTION_LIMITS", replace(workers.INSPECTION_LIMITS, wall_seconds=0.001)):
|
||||
with self.assertRaisesRegex(FileStorageError, "timeout"):
|
||||
inspect_archive(_zip_bytes({"one": b"one"}), filename="fixture.zip")
|
||||
self.assertEqual(1, len(self.children))
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_real_child_cpu_limit_has_sanitized_files_error(self):
|
||||
with (
|
||||
patch.object(workers, "_inspect_worker", _cpu_exhaustion_worker),
|
||||
patch.object(workers, "INSPECTION_LIMITS", replace(workers.INSPECTION_LIMITS, cpu_seconds=1, wall_seconds=10)),
|
||||
):
|
||||
with self.assertRaisesRegex(FileStorageError, "cpu_limit"):
|
||||
inspect_archive(_zip_bytes({"one": b"one"}), filename="fixture.zip")
|
||||
|
||||
def test_real_preview_output_is_bounded(self):
|
||||
with patch.object(workers, "INSPECTION_LIMITS", replace(workers.INSPECTION_LIMITS, output_bytes=256)):
|
||||
with self.assertRaisesRegex(FileStorageError, "output_limit"):
|
||||
inspect_archive(_zip_bytes({f"file-{index}": b"x" for index in range(20)}), filename="fixture.zip")
|
||||
|
||||
def test_destination_failure_kills_waiting_child_and_preserves_error(self):
|
||||
self.store.side_effect = FileStorageError("Destination failure")
|
||||
with self.assertRaisesRegex(FileStorageError, "Destination failure"):
|
||||
_extract(_zip_bytes({"one": b"one", "two": b"two"}))
|
||||
self.assertEqual(1, self.store.call_count)
|
||||
self.assertEqual(1, len(self.children))
|
||||
|
||||
def test_missing_ack_times_out_and_wrong_ack_fails_in_real_child(self):
|
||||
real_send = workers._send_ack
|
||||
for send, error in ((lambda descriptor, sequence: None, "timeout"),
|
||||
(lambda descriptor, sequence: real_send(descriptor, sequence + 1), "invalid staging record")):
|
||||
with self.subTest(error=error), patch.object(workers, "_send_ack", side_effect=send):
|
||||
with patch.object(workers, "EXTRACTION_LIMITS", replace(workers.EXTRACTION_LIMITS, wall_seconds=2)):
|
||||
with self.assertRaisesRegex(FileStorageError, error):
|
||||
_extract(_zip_bytes({"one": b"one", "two": b"two"}))
|
||||
|
||||
def test_parent_progress_exception_is_not_reclassified_as_transport_error(self):
|
||||
def progress(stage, *counters):
|
||||
if stage == "extracting":
|
||||
raise ValueError("parent progress failure")
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "parent progress failure"):
|
||||
_extract(_zip_bytes({"one": b"one"}), progress=progress)
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_source_path_snapshot_rejects_growth_and_shrink_without_spawning(self):
|
||||
# A source path is server-owned; it must still not turn into an unbounded
|
||||
# copy when another writer changes it during snapshot creation.
|
||||
for replacement in (b"fixture plus unexpected growth", b"x"):
|
||||
with self.subTest(replacement=replacement), tempfile.TemporaryDirectory() as temporary:
|
||||
source = Path(temporary) / "archive"
|
||||
source.write_bytes(b"fixture")
|
||||
real_fstat = os.fstat
|
||||
changed = False
|
||||
|
||||
def fstat(descriptor):
|
||||
nonlocal changed
|
||||
result = real_fstat(descriptor)
|
||||
if not changed:
|
||||
changed = True
|
||||
source.write_bytes(replacement)
|
||||
return result
|
||||
|
||||
with patch.object(workers.os, "fstat", side_effect=fstat):
|
||||
with self.assertRaisesRegex(FileStorageError, "source changed"):
|
||||
inspect_archive(source, filename="fixture.zip")
|
||||
self.assertEqual([], self.children)
|
||||
|
||||
def test_busy_rejects_before_source_snapshot(self):
|
||||
with patch.object(settings, "isolated_process_concurrency", 1), bounded_process.bounded_operation_admission():
|
||||
with patch.object(workers, "_source_stage", side_effect=AssertionError("must admit before copying")):
|
||||
with self.assertRaisesRegex(FileStorageError, "busy"):
|
||||
inspect_archive(b"fixture", filename="fixture.zip")
|
||||
self.assertEqual([], self.children)
|
||||
self.assertEqual([], self.directories)
|
||||
|
||||
def test_unsupported_format_rejects_before_snapshot(self):
|
||||
with self.assertRaisesRegex(FileStorageError, "Unsupported archive format"):
|
||||
inspect_archive(b"fixture", filename="fixture.exe")
|
||||
self.assertEqual([], self.directories)
|
||||
|
||||
def test_symlinked_staged_member_is_never_read_or_stored(self):
|
||||
real_read = workers._read_regular
|
||||
attacked = False
|
||||
|
||||
def read(path, maximum, **options):
|
||||
nonlocal attacked
|
||||
if path.name.startswith("member-") and not attacked:
|
||||
attacked = True
|
||||
path.unlink()
|
||||
path.symlink_to(path.parent / "source")
|
||||
return real_read(path, maximum, **options)
|
||||
|
||||
with patch.object(workers, "_read_regular", side_effect=read):
|
||||
with self.assertRaisesRegex(FileStorageError, "invalid staging record"):
|
||||
_extract(_zip_bytes({"one": b"one"}))
|
||||
self.assertTrue(attacked)
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_forged_member_records_reject_before_store(self):
|
||||
real_read = workers._read_regular
|
||||
for mutation in (
|
||||
{"path": "outside/one"}, {"path": "../escape"}, {"size": -1},
|
||||
{"sha256": "incorrect"}, {"sequence": True}, {"unexpected": True},
|
||||
{"progress": ["extracting", 1, 1, 4, 3]},
|
||||
):
|
||||
with self.subTest(mutation=mutation):
|
||||
self.store.reset_mock()
|
||||
|
||||
def read(path, maximum, **options):
|
||||
value = real_read(path, maximum, **options)
|
||||
if path.name == "status":
|
||||
record = json.loads(value)
|
||||
if record.get("kind") == "member":
|
||||
record.update(mutation)
|
||||
return json.dumps(record).encode()
|
||||
return value
|
||||
|
||||
with patch.object(workers, "_read_regular", side_effect=read):
|
||||
with self.assertRaises(FileStorageError):
|
||||
_extract(_zip_bytes({"selected/one": b"one"}), selected_paths=("selected",))
|
||||
self.store.assert_not_called()
|
||||
|
||||
def test_regressing_sequence_or_changing_totals_fail_before_second_store(self):
|
||||
real_read = workers._read_regular
|
||||
for mutation in ({"sequence": 1}, {"progress": ["extracting", 2, 3, 6, 6]}):
|
||||
with self.subTest(mutation=mutation):
|
||||
self.store.reset_mock()
|
||||
|
||||
def read(path, maximum, **options):
|
||||
value = real_read(path, maximum, **options)
|
||||
if path.name == "status":
|
||||
record = json.loads(value)
|
||||
if record.get("kind") == "member" and record["progress"][1] == 2:
|
||||
record.update(mutation)
|
||||
return json.dumps(record).encode()
|
||||
return value
|
||||
|
||||
with patch.object(workers, "_read_regular", side_effect=read):
|
||||
with self.assertRaisesRegex(FileStorageError, "invalid staging record"):
|
||||
_extract(_zip_bytes({"one": b"one", "two": b"two"}))
|
||||
self.assertEqual(1, self.store.call_count)
|
||||
|
||||
def test_record_reads_are_bounded_and_reject_non_regular_files(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
directory = Path(temporary)
|
||||
status = directory / "status"
|
||||
status.write_bytes(b"x" * (workers._STATUS_BYTES + 1))
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._read_regular(status, workers._STATUS_BYTES)
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._read_regular(directory, workers._STATUS_BYTES)
|
||||
|
||||
def test_tiny_member_read_does_not_allocate_the_configured_gibibyte_cap(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
member = Path(temporary) / "member"
|
||||
member.write_bytes(b"x")
|
||||
tracemalloc.start()
|
||||
try:
|
||||
self.assertEqual(b"x", workers._read_regular(member, 2 * 1024 * 1024 * 1024))
|
||||
_current, peak = tracemalloc.get_traced_memory()
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
self.assertLess(peak, 1024 * 1024)
|
||||
|
||||
def test_atomic_status_replacement_keeps_open_snapshot_valid_but_member_identity_is_strict(self):
|
||||
for atomic_record in (True, False):
|
||||
with self.subTest(atomic_record=atomic_record), tempfile.TemporaryDirectory() as temporary:
|
||||
status = Path(temporary) / "status"
|
||||
status.write_bytes(b"old")
|
||||
replacement = Path(temporary) / "replacement"
|
||||
replacement.write_bytes(b"new")
|
||||
real_fstat = os.fstat
|
||||
replaced = False
|
||||
|
||||
def fstat(descriptor):
|
||||
nonlocal replaced
|
||||
info = real_fstat(descriptor)
|
||||
if not replaced:
|
||||
replaced = True
|
||||
os.replace(replacement, status)
|
||||
return info
|
||||
|
||||
with patch.object(workers.os, "fstat", side_effect=fstat):
|
||||
if atomic_record:
|
||||
self.assertEqual(b"old", workers._read_regular(status, 16, atomic_record=True))
|
||||
else:
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._read_regular(status, 16)
|
||||
self.assertEqual(b"new", status.read_bytes())
|
||||
|
||||
def test_private_ack_channel_is_bounded_and_validates_type_permissions_and_sequence(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
directory = Path(temporary)
|
||||
with workers._ack_channel(directory, create=True) as writer:
|
||||
with workers._ack_channel(directory) as reader:
|
||||
workers._send_ack(writer, 7)
|
||||
workers._receive_ack(reader, 7)
|
||||
workers._send_ack(writer, 8)
|
||||
with self.assertRaises(FileStorageError):
|
||||
workers._receive_ack(reader, 9)
|
||||
with patch.object(workers.os, "write", side_effect=BlockingIOError()):
|
||||
with self.assertRaisesRegex(FileStorageError, "staging channel is unavailable"):
|
||||
workers._send_ack(writer, 10)
|
||||
ack = directory / "ack"
|
||||
os.chmod(ack, 0o644)
|
||||
with self.assertRaises(FileStorageError), workers._ack_channel(directory):
|
||||
pass
|
||||
ack.unlink()
|
||||
ack.write_bytes(b"not a FIFO")
|
||||
with self.assertRaises(FileStorageError), workers._ack_channel(directory):
|
||||
pass
|
||||
ack.unlink()
|
||||
ack.symlink_to(directory / "missing")
|
||||
with self.assertRaises(FileStorageError), workers._ack_channel(directory):
|
||||
pass
|
||||
|
||||
def test_tiny_member_import_has_no_per_member_sleep_or_polling_delay(self):
|
||||
self.assertNotIn("sleep(", inspect.getsource(workers._extract_worker))
|
||||
for count in (100, 1000):
|
||||
with self.subTest(count=count):
|
||||
payload = _zip_bytes({f"member-{index}.txt": b"x" for index in range(count)})
|
||||
started = time.monotonic()
|
||||
self.assertEqual(count, len(_extract(payload)))
|
||||
elapsed = time.monotonic() - started
|
||||
# Broad synthetic regression ceiling, not a storage-throughput
|
||||
# promise: 50 ms per member would take >=50 s for 1,000 files.
|
||||
self.assertLess(elapsed, 15)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Executable
+244
@@ -0,0 +1,244 @@
|
||||
import hashlib
|
||||
import io
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_download_connector_payload,
|
||||
_http_error,
|
||||
)
|
||||
from govoplan_files.backend.schemas import FileConnectorImportRequest
|
||||
from govoplan_files.backend.storage.connector_imports import (
|
||||
ConnectorDownloadedFile,
|
||||
ConnectorRevisionConflict,
|
||||
_read_seafile_file,
|
||||
_read_smb_file,
|
||||
_read_s3_file,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
class ConnectorProvenanceTests(unittest.TestCase):
|
||||
def acquire(self, download, *, expected=None, annotations=None):
|
||||
profile = ConnectorProfile(id="profile", label="Synthetic", provider="s3")
|
||||
request = FileConnectorImportRequest(
|
||||
library_id="bucket",
|
||||
path="source.txt",
|
||||
source_revision=expected,
|
||||
metadata=annotations or {},
|
||||
)
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.route_support.connector_policy_decision",
|
||||
return_value=SimpleNamespace(allowed=True),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.route_support.read_connector_file",
|
||||
return_value=download,
|
||||
),
|
||||
):
|
||||
return _download_connector_payload(profile, request, operation="sync")
|
||||
|
||||
def test_stale_or_unavailable_expected_revision_cannot_label_new_bytes(self):
|
||||
for revision in ("B", None):
|
||||
with (
|
||||
self.subTest(revision=revision),
|
||||
self.assertRaises(ConnectorRevisionConflict) as caught,
|
||||
):
|
||||
self.acquire(
|
||||
ConnectorDownloadedFile(
|
||||
filename="file", data=b"B", revision=revision
|
||||
),
|
||||
expected="A",
|
||||
)
|
||||
self.assertEqual(409, _http_error(caught.exception).status_code)
|
||||
|
||||
def test_actual_digest_is_computed_and_claims_and_annotations_are_not_authority(
|
||||
self,
|
||||
):
|
||||
annotations = {
|
||||
"profile_id": "forged",
|
||||
"size": 999,
|
||||
"checksum_sha256": "caller-claim",
|
||||
"acquired_sha256": "forged",
|
||||
"note": "keep this",
|
||||
}
|
||||
download = ConnectorDownloadedFile(
|
||||
filename="source.txt",
|
||||
data=b"actual",
|
||||
revision="version-B",
|
||||
metadata={"etag": "etag-B", "checksum_sha256": "provider-claim"},
|
||||
)
|
||||
_, actual, metadata = self.acquire(
|
||||
download, expected="etag-B", annotations=annotations
|
||||
)
|
||||
self.assertEqual("version-B", metadata["source_revision"])
|
||||
evidence = metadata["source_provenance"]["metadata"]
|
||||
self.assertEqual(
|
||||
hashlib.sha256(actual.data).hexdigest(), evidence["acquired_sha256"]
|
||||
)
|
||||
self.assertEqual(len(actual.data), evidence["size"])
|
||||
self.assertEqual("profile", evidence["profile_id"])
|
||||
self.assertNotIn("checksum_sha256", evidence)
|
||||
self.assertEqual(
|
||||
{"checksum_sha256": "provider-claim"}, evidence["provider_checksum_claims"]
|
||||
)
|
||||
self.assertEqual(annotations, evidence["import_annotations"])
|
||||
self.assertEqual(
|
||||
metadata,
|
||||
self.acquire(download, expected="version-B", annotations=annotations)[2],
|
||||
)
|
||||
|
||||
def test_caller_fields_stay_annotations_when_provider_omits_evidence(self):
|
||||
annotations = {
|
||||
"sha256": "caller-hash",
|
||||
"etag": "caller-etag",
|
||||
"mtime": "caller-modified-time",
|
||||
"checksum_sha256": "caller-checksum",
|
||||
"provider_checksum_claims": {"checksum_sha256": "caller-claim"},
|
||||
"connector_space_id": "browse-space",
|
||||
"browse_etag": "browse-revision",
|
||||
"folder_sync": True,
|
||||
"note": {"values": ["001", " keep spaces ", None]},
|
||||
}
|
||||
download = ConnectorDownloadedFile(filename="source.txt", data=b"actual")
|
||||
_, _, metadata = self.acquire(download, annotations=annotations)
|
||||
evidence = metadata["source_provenance"]["metadata"]
|
||||
self.assertEqual(annotations, evidence["import_annotations"])
|
||||
self.assertEqual({}, evidence["provider_checksum_claims"])
|
||||
self.assertEqual(
|
||||
hashlib.sha256(download.data).hexdigest(), evidence["acquired_sha256"]
|
||||
)
|
||||
self.assertEqual(
|
||||
{
|
||||
"profile_id", "library_id", "library_path", "size",
|
||||
"acquired_sha256", "provider_checksum_claims", "import_annotations",
|
||||
},
|
||||
set(evidence),
|
||||
)
|
||||
|
||||
def test_seafile_rechecks_revision_after_download(self):
|
||||
profile = ConnectorProfile(
|
||||
id="profile",
|
||||
label="Synthetic",
|
||||
provider="seafile",
|
||||
endpoint_url="https://example.invalid",
|
||||
)
|
||||
before = {"id": "A", "name": "source.txt", "size": 4}
|
||||
for after in ({**before, "id": "B"}, before):
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._seafile_headers",
|
||||
return_value={},
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._request_json",
|
||||
side_effect=[before, "https://example.invalid/download", after],
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports.request_connector_bytes",
|
||||
return_value=SimpleNamespace(
|
||||
status_code=200, content=b"DATA", headers={}
|
||||
),
|
||||
),
|
||||
):
|
||||
if after["id"] == "B":
|
||||
with self.assertRaises(ConnectorRevisionConflict):
|
||||
_read_seafile_file(
|
||||
profile, library_id="repo", path="source.txt", max_bytes=10
|
||||
)
|
||||
else:
|
||||
result = _read_seafile_file(
|
||||
profile, library_id="repo", path="source.txt", max_bytes=10
|
||||
)
|
||||
self.assertEqual(("A", b"DATA"), (result.revision, result.data))
|
||||
|
||||
def test_smb_observes_stable_metadata_while_write_and_delete_sharing_are_denied(
|
||||
self,
|
||||
):
|
||||
profile = ConnectorProfile(id="profile", label="Synthetic", provider="smb")
|
||||
before = SimpleNamespace(st_size=4, st_mtime_ns=10, st_mtime=1, st_ino=1)
|
||||
for after in (
|
||||
SimpleNamespace(st_size=4, st_mtime_ns=11, st_mtime=1, st_ino=1),
|
||||
before,
|
||||
):
|
||||
with (
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smb_location",
|
||||
return_value=SimpleNamespace(
|
||||
share="share", server="example.invalid", port=445
|
||||
),
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smb_unc_path",
|
||||
return_value="synthetic",
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smb_client_kwargs",
|
||||
return_value={},
|
||||
),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._smbclient_module"
|
||||
) as factory,
|
||||
):
|
||||
sdk = factory.return_value
|
||||
sdk.stat.side_effect = [before, after]
|
||||
sdk.open_file.return_value = io.BytesIO(b"DATA")
|
||||
if after is before:
|
||||
self.assertEqual(
|
||||
b"DATA",
|
||||
_read_smb_file(profile, path="source.txt", max_bytes=10).data,
|
||||
)
|
||||
else:
|
||||
with self.assertRaises(ConnectorRevisionConflict):
|
||||
_read_smb_file(profile, path="source.txt", max_bytes=10)
|
||||
self.assertEqual("r", sdk.open_file.call_args.kwargs["share_access"])
|
||||
|
||||
def test_s3_conditional_or_versioned_read_and_response_binding(self):
|
||||
profile = ConnectorProfile(id="profile", label="Synthetic", provider="s3")
|
||||
for version in (None, "version-A", "null"):
|
||||
for returned_etag in ("etag-A", "etag-B"):
|
||||
with (
|
||||
self.subTest(version=version, etag=returned_etag),
|
||||
patch(
|
||||
"govoplan_files.backend.storage.connector_imports._s3_import_client"
|
||||
) as factory,
|
||||
):
|
||||
client = factory.return_value
|
||||
client.head_object.return_value = {
|
||||
"ContentLength": 4,
|
||||
"ETag": "etag-A",
|
||||
"VersionId": version,
|
||||
}
|
||||
body = io.BytesIO(b"DATA")
|
||||
client.get_object.return_value = {
|
||||
"Body": body,
|
||||
"ETag": returned_etag,
|
||||
"VersionId": version,
|
||||
}
|
||||
if returned_etag == "etag-A":
|
||||
result = _read_s3_file(
|
||||
profile,
|
||||
library_id="bucket",
|
||||
path="source.txt",
|
||||
max_bytes=10,
|
||||
)
|
||||
self.assertEqual(version if version and version != "null" else "etag-A", result.revision)
|
||||
else:
|
||||
with self.assertRaises(ConnectorRevisionConflict):
|
||||
_read_s3_file(
|
||||
profile,
|
||||
library_id="bucket",
|
||||
path="source.txt",
|
||||
max_bytes=10,
|
||||
)
|
||||
request = client.get_object.call_args.kwargs
|
||||
immutable = version and version != "null"
|
||||
self.assertEqual(version if immutable else "etag-A", request["VersionId" if immutable else "IfMatch"])
|
||||
self.assertTrue(body.closed)
|
||||
client.close.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,450 @@
|
||||
"""Managed archive routes: real authorization, preview and extraction pipeline.
|
||||
|
||||
Only external blob persistence, directory membership and audit delivery are
|
||||
isolated; source/version/share checks and destination writes use SQLite.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import tarfile
|
||||
import unittest
|
||||
import zipfile
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
import pyzipper
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.pool import StaticPool
|
||||
|
||||
from govoplan_access.backend.db.models import Account, Group, User
|
||||
from govoplan_core.auth import get_api_principal
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorPolicy,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.routes.managed_archives import router
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.files import create_file_asset
|
||||
|
||||
|
||||
def archive_bytes(entries=None, *, password=None, tar=False):
|
||||
entries = entries or {"folder/one.txt": b"one", "other/two.txt": b"two"}
|
||||
output = io.BytesIO()
|
||||
if tar:
|
||||
with tarfile.open(fileobj=output, mode="w:gz") as archive:
|
||||
for name, data in entries.items():
|
||||
member = tarfile.TarInfo(name)
|
||||
member.size = len(data)
|
||||
archive.addfile(member, io.BytesIO(data))
|
||||
else:
|
||||
archive_type = pyzipper.AESZipFile if password else zipfile.ZipFile
|
||||
options = {"encryption": pyzipper.WZ_AES} if password else {}
|
||||
with archive_type(
|
||||
output, "w", compression=zipfile.ZIP_DEFLATED, **options
|
||||
) as archive:
|
||||
if password:
|
||||
archive.setpassword(password.encode())
|
||||
for name, data in entries.items():
|
||||
archive.writestr(name, data)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
class ManagedArchiveTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine(
|
||||
"sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool
|
||||
)
|
||||
Base.metadata.create_all(
|
||||
self.engine,
|
||||
tables=[
|
||||
model.__table__
|
||||
for model in (
|
||||
Account,
|
||||
User,
|
||||
Group,
|
||||
ChangeSequenceEntry,
|
||||
FileBlob,
|
||||
FileAsset,
|
||||
FileVersion,
|
||||
FileShare,
|
||||
FileConnectorPolicy,
|
||||
)
|
||||
],
|
||||
)
|
||||
self.session = Session(self.engine, expire_on_commit=False)
|
||||
self.addCleanup(self.engine.dispose)
|
||||
self.addCleanup(self.session.close)
|
||||
self.objects = {}
|
||||
self.scopes = {"files:file:read", "files:file:download", "files:file:upload"}
|
||||
self.principal = SimpleNamespace(
|
||||
tenant_id="tenant-1",
|
||||
user=SimpleNamespace(id="user-1"),
|
||||
has=lambda scope: scope in self.scopes,
|
||||
)
|
||||
self.settings = SimpleNamespace(
|
||||
file_upload_zip_max_bytes=10 * 1024 * 1024,
|
||||
file_archive_max_entries=10_000,
|
||||
file_archive_max_expanded_bytes=20 * 1024 * 1024,
|
||||
file_archive_max_expansion_ratio=100,
|
||||
file_archive_preview_ttl_seconds=1800,
|
||||
file_upload_max_bytes=1024 * 1024,
|
||||
)
|
||||
for module in ("uploads", "managed_archives"):
|
||||
self.enterContext(
|
||||
patch(f"govoplan_files.backend.routes.{module}.settings", self.settings)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files.user_group_ids", return_value=[]
|
||||
)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files._get_or_create_blob",
|
||||
side_effect=self.store_blob,
|
||||
)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.storage.files.get_storage_backend",
|
||||
return_value=SimpleNamespace(
|
||||
name="test", get_bytes=lambda key: self.objects[key]
|
||||
),
|
||||
)
|
||||
)
|
||||
self.audit = self.enterContext(
|
||||
patch("govoplan_files.backend.route_support.audit_from_principal")
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.route_support.asset_is_audit_relevant",
|
||||
return_value=False,
|
||||
)
|
||||
)
|
||||
self.enterContext(
|
||||
patch(
|
||||
"govoplan_files.backend.route_support._sent_campaign_asset_ids",
|
||||
return_value=set(),
|
||||
)
|
||||
)
|
||||
self.source = self.seed_source(archive_bytes())
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api/v1")
|
||||
app.dependency_overrides[get_api_principal] = lambda: self.principal
|
||||
app.dependency_overrides[get_session] = lambda: self.session
|
||||
self.client = self.enterContext(TestClient(app))
|
||||
|
||||
def store_blob(self, session, *, tenant_id, data, content_type=None, **kwargs):
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
blob = FileBlob(
|
||||
tenant_id=tenant_id,
|
||||
storage_backend="test",
|
||||
storage_key=f"objects/{checksum}",
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
content_type=content_type,
|
||||
ref_count=1,
|
||||
)
|
||||
session.add(blob)
|
||||
session.flush()
|
||||
self.objects[blob.storage_key] = data
|
||||
return blob
|
||||
|
||||
def seed_source(self, data, filename="source.zip"):
|
||||
stored = create_file_asset(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
owner_type="user",
|
||||
owner_id="user-1",
|
||||
user_id="user-1",
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder="archives",
|
||||
)
|
||||
self.session.commit()
|
||||
return stored
|
||||
|
||||
def preview(self, **changes):
|
||||
return self.client.post(
|
||||
f"/api/v1/files/{self.source.asset.id}/archive-preview",
|
||||
json={
|
||||
"source_version_id": self.source.version.id,
|
||||
"path": "extracted",
|
||||
**changes,
|
||||
},
|
||||
)
|
||||
|
||||
def confirm(self, token, **changes):
|
||||
return self.client.post(
|
||||
f"/api/v1/files/{self.source.asset.id}/archive-confirm",
|
||||
json={
|
||||
"source_version_id": self.source.version.id,
|
||||
"path": "extracted",
|
||||
"preview_token": token,
|
||||
"selected_paths": ["folder/one.txt"],
|
||||
**changes,
|
||||
},
|
||||
)
|
||||
|
||||
def token(self):
|
||||
response = self.preview()
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
return response.json()["preview_token"]
|
||||
|
||||
def assert_no_extraction(self):
|
||||
self.assertEqual(1, self.session.query(FileAsset).count())
|
||||
self.assertIsNone(self.source.asset.deleted_at)
|
||||
self.assertEqual(self.source.version.id, self.source.asset.current_version_id)
|
||||
self.audit.assert_not_called()
|
||||
|
||||
def test_preview_is_read_only_and_confirmation_extracts_only_selected_members(self):
|
||||
token = self.token()
|
||||
self.assert_no_extraction()
|
||||
original_data = self.objects[self.source.blob.storage_key]
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
files = response.json()["files"]
|
||||
self.assertEqual(
|
||||
["extracted/folder/one.txt"], [file["display_path"] for file in files]
|
||||
)
|
||||
self.assertEqual(2, self.session.query(FileAsset).count())
|
||||
self.assertEqual(original_data, self.objects[self.source.blob.storage_key])
|
||||
self.assertEqual(self.source.version.id, self.source.asset.current_version_id)
|
||||
self.assertIsNone(self.source.asset.deleted_at)
|
||||
self.assertEqual(
|
||||
self.source.asset.id,
|
||||
files[0]["source_provenance"]["metadata"]["archive_source_file_id"],
|
||||
)
|
||||
self.assertEqual(
|
||||
self.source.version.id,
|
||||
files[0]["source_provenance"]["metadata"]["archive_source_version_id"],
|
||||
)
|
||||
self.audit.assert_called_once()
|
||||
|
||||
def test_all_three_permissions_are_required(self):
|
||||
token = self.token()
|
||||
for scope in tuple(self.scopes):
|
||||
with self.subTest(scope=scope):
|
||||
self.scopes.remove(scope)
|
||||
self.assertEqual(403, self.preview().status_code)
|
||||
self.assertEqual(403, self.confirm(token).status_code)
|
||||
self.scopes.add(scope)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_changed_source_version_is_rejected_before_read_or_write(self):
|
||||
token = self.token()
|
||||
self.source.asset.current_version_id = "replacement-version"
|
||||
self.session.commit()
|
||||
self.assertIn("version changed", self.confirm(token).text)
|
||||
self.assertEqual(400, self.preview().status_code)
|
||||
self.assertEqual(1, self.session.query(FileAsset).count())
|
||||
|
||||
def test_mismatched_version_or_blob_tenant_is_rejected(self):
|
||||
self.source.version.file_asset_id = "other-file"
|
||||
self.session.commit()
|
||||
self.assertIn("does not belong", self.preview().text)
|
||||
self.source.version.file_asset_id = self.source.asset.id
|
||||
self.source.blob.tenant_id = "other-tenant"
|
||||
self.session.commit()
|
||||
self.assertIn("does not belong", self.preview().text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_expired_preview_cannot_extract(self):
|
||||
with patch("cryptography.fernet.time.time", return_value=1):
|
||||
token = self.token()
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("expired", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_source_refresh_during_verified_read_cannot_change_preview_or_extract(self):
|
||||
token = self.token()
|
||||
replacement_version = SimpleNamespace(id="refreshed-version")
|
||||
with patch(
|
||||
"govoplan_files.backend.routes.managed_archives.read_asset_bytes",
|
||||
return_value=(
|
||||
self.objects[self.source.blob.storage_key],
|
||||
replacement_version,
|
||||
self.source.blob,
|
||||
),
|
||||
):
|
||||
preview = self.preview()
|
||||
confirmed = self.confirm(token)
|
||||
for response in (preview, confirmed):
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("version changed during the verified read", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_revoked_shared_access_cannot_reuse_preview(self):
|
||||
self.source.asset.owner_user_id = "other-user"
|
||||
share = FileShare(
|
||||
tenant_id="tenant-1",
|
||||
file_asset_id=self.source.asset.id,
|
||||
target_type="user",
|
||||
target_id="user-1",
|
||||
permission="read",
|
||||
)
|
||||
self.session.add(share)
|
||||
self.session.commit()
|
||||
token = self.token()
|
||||
share.revoked_at = utcnow()
|
||||
self.session.commit()
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("No access", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_deleted_or_cross_tenant_sources_are_rejected(self):
|
||||
token = self.token()
|
||||
self.source.asset.deleted_at = utcnow()
|
||||
self.session.commit()
|
||||
self.assertIn("File not found", self.confirm(token).text)
|
||||
self.source.asset.deleted_at = None
|
||||
self.source.asset.tenant_id = "other-tenant"
|
||||
self.session.commit()
|
||||
self.assertIn("File not found", self.confirm(token).text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_confirmation_is_bound_to_destination_actor_version_and_file(self):
|
||||
token = self.token()
|
||||
for change in (
|
||||
{"path": "elsewhere"},
|
||||
{"owner_id": "other-user"},
|
||||
{"source_version_id": "wrong"},
|
||||
):
|
||||
with self.subTest(change=change):
|
||||
response = self.confirm(token, **change)
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("does not match", response.text)
|
||||
self.principal.user.id = "other-user"
|
||||
self.assertIn("does not match", self.confirm(token).text)
|
||||
self.principal.user.id = "user-1"
|
||||
self.assertEqual(400, self.confirm("tampered-token").status_code)
|
||||
response = self.client.post(
|
||||
"/api/v1/files/other-file/archive-confirm",
|
||||
json={
|
||||
"source_version_id": self.source.version.id,
|
||||
"path": "extracted",
|
||||
"preview_token": token,
|
||||
"selected_paths": ["folder"],
|
||||
},
|
||||
)
|
||||
self.assertIn("does not match", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_destination_ownership_is_checked_during_preview(self):
|
||||
response = self.preview(owner_id="other-user")
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("No access to this user file space", response.text)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_connector_policy_is_rechecked_at_confirmation(self):
|
||||
self.source.asset.metadata_ = {
|
||||
"source_provenance": {
|
||||
"source_type": "webdav",
|
||||
"provider": "webdav",
|
||||
"external_id": "archive",
|
||||
}
|
||||
}
|
||||
self.session.commit()
|
||||
token = self.token()
|
||||
self.session.add(
|
||||
FileConnectorPolicy(
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
scope_id="tenant-1",
|
||||
policy={"deny": {"providers": ["webdav"]}},
|
||||
)
|
||||
)
|
||||
self.session.commit()
|
||||
response = self.confirm(token)
|
||||
self.assertEqual(403, response.status_code, response.text)
|
||||
self.assertEqual(403, self.preview().status_code)
|
||||
self.assert_no_extraction()
|
||||
|
||||
def test_source_collision_rolls_back_all_destination_writes(self):
|
||||
self.source = self.seed_source(
|
||||
archive_bytes(
|
||||
{"a-first.txt": b"first", "collision.zip": b"cannot replace source"}
|
||||
),
|
||||
"collision.zip",
|
||||
)
|
||||
response = self.preview(path="archives")
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
result = self.confirm(
|
||||
response.json()["preview_token"],
|
||||
path="archives",
|
||||
selected_paths=["a-first.txt", "collision.zip"],
|
||||
)
|
||||
self.assertEqual(400, result.status_code, result.text)
|
||||
self.assertIn("already exists", result.text)
|
||||
self.assertEqual(2, self.session.query(FileAsset).count())
|
||||
self.assertEqual(self.source.version.id, self.source.asset.current_version_id)
|
||||
self.assertIsNone(self.source.asset.deleted_at)
|
||||
|
||||
def test_encrypted_zip_uses_the_same_password_and_selection_flow(self):
|
||||
self.source = self.seed_source(
|
||||
archive_bytes(password="correct horse"), "encrypted.zip"
|
||||
)
|
||||
preview = self.preview()
|
||||
self.assertTrue(preview.json()["requires_password"])
|
||||
self.assertFalse(preview.json()["password_verified"])
|
||||
token = preview.json()["preview_token"]
|
||||
self.assertEqual(400, self.confirm(token).status_code)
|
||||
self.assertEqual(400, self.preview(password="wrong").status_code)
|
||||
verified = self.preview(password="correct horse")
|
||||
self.assertTrue(verified.json()["password_verified"])
|
||||
response = self.confirm(
|
||||
verified.json()["preview_token"], password="correct horse"
|
||||
)
|
||||
self.assertEqual(200, response.status_code, response.text)
|
||||
self.assertNotIn("correct horse", response.text)
|
||||
self.assertNotIn("correct horse", str(self.audit.call_args_list))
|
||||
|
||||
def test_tar_reuses_the_same_confirmation_flow(self):
|
||||
self.source = self.seed_source(archive_bytes(tar=True), "source.tar.gz")
|
||||
preview = self.preview()
|
||||
self.assertEqual("tar.gz", preview.json()["archive_format"])
|
||||
self.assertEqual(200, self.confirm(preview.json()["preview_token"]).status_code)
|
||||
|
||||
def test_unsafe_members_and_size_limits_fail_before_extraction(self):
|
||||
self.settings.file_upload_zip_max_bytes = 1
|
||||
self.assertIn("size limit", self.preview().text)
|
||||
self.settings.file_upload_zip_max_bytes = 10 * 1024 * 1024
|
||||
self.source = self.seed_source(
|
||||
archive_bytes({"../escape": b"no"}), "unsafe.zip"
|
||||
)
|
||||
self.assertIn("Unsafe archive member", self.preview().text)
|
||||
self.audit.assert_not_called()
|
||||
|
||||
def test_quarantined_or_protected_sources_use_verified_read_path(self):
|
||||
self.source.blob.quarantined_at = utcnow()
|
||||
self.session.commit()
|
||||
self.assertIn("quarantin", self.preview().text.lower())
|
||||
self.source.blob.quarantined_at = None
|
||||
self.source.blob.encryption_envelope_id = "protected-envelope"
|
||||
self.session.commit()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.content_protection.unprotect_blob_content",
|
||||
side_effect=FileStorageError("Decryption is unavailable"),
|
||||
) as decrypt:
|
||||
response = self.preview()
|
||||
self.assertEqual(400, response.status_code, response.text)
|
||||
self.assertIn("Decryption is unavailable", response.text)
|
||||
decrypt.assert_called_once()
|
||||
self.assert_no_extraction()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -4,10 +4,13 @@ import unittest
|
||||
|
||||
|
||||
STATIC_TOPIC_IDS = {
|
||||
"files.source-identity-provenance",
|
||||
"files.archive-worker-limits",
|
||||
"files.configuration-package.managed-storage",
|
||||
"files.quick-access-and-product-area",
|
||||
"files.search.managed-content",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.unpack-managed-archive",
|
||||
"files.workflow.find-and-download-files",
|
||||
"files.workflow.share-managed-files",
|
||||
"files.workflow.delete-managed-files",
|
||||
@@ -44,6 +47,38 @@ class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
def topic(self, topic_id: str):
|
||||
return self.topics[topic_id]
|
||||
|
||||
def test_source_provenance_and_legacy_ambiguity_are_bilingual(self):
|
||||
topic = self.topic("files.source-identity-provenance")
|
||||
self.assertEqual({"admin", "user"}, set(topic.documentation_types))
|
||||
for body in (topic.body, topic.translations["de"]["body"]):
|
||||
for field in ("source_revision", "acquired_sha256", "provider_checksum_claims", "import_annotations"):
|
||||
self.assertIn(field, body)
|
||||
self.assertIn("non-unique", topic.body)
|
||||
self.assertIn("nicht eindeutig", topic.translations["de"]["body"])
|
||||
|
||||
def test_archive_resource_boundary_is_static_and_bilingual(self):
|
||||
topic = self.topic("files.archive-worker-limits")
|
||||
self.assertEqual("available", topic.layer)
|
||||
self.assertEqual({"admin", "user"}, set(topic.documentation_types))
|
||||
self.assertIn("GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY", topic.configuration_keys)
|
||||
for body in (topic.body, topic.translations["de"]["body"]):
|
||||
for limit in ("120", "90", "512 MiB", "600", "300", "128 MiB", "64 MiB", "16 KiB", "2 GiB", "250 MiB"):
|
||||
self.assertIn(limit, body)
|
||||
self.assertIn("before copying", topic.body)
|
||||
self.assertIn("not a filesystem/network sandbox", topic.body)
|
||||
self.assertIn("keine Dateisystem-/Netzwerk-Sandbox", topic.translations["de"]["body"])
|
||||
|
||||
def test_workspace_action_locations_and_read_only_reload_are_documented_in_both_languages(self) -> None:
|
||||
topic = self.topic("files.quick-access-and-product-area")
|
||||
self.assertEqual({"user", "admin"}, set(topic.documentation_types))
|
||||
for label in ("Reload", "Create folder", "Upload", "Connections and imports", "Manage selection"):
|
||||
self.assertIn(label, topic.body)
|
||||
self.assertIn("never imports or synchronizes", topic.body)
|
||||
german = topic.translations["de"]["body"]
|
||||
for label in ("Neu laden", "Ordner erstellen", "Hochladen", "Verbindungen und Importe", "Auswahl verwalten"):
|
||||
self.assertIn(label, german)
|
||||
self.assertIn("niemals", german)
|
||||
|
||||
def test_static_topics_have_role_scope_module_and_link_contracts(self) -> None:
|
||||
self.assertEqual(STATIC_TOPIC_IDS, set(self.topics))
|
||||
self.assertEqual(
|
||||
|
||||
@@ -31,7 +31,7 @@ class FilesMigrationTests(unittest.TestCase):
|
||||
)
|
||||
with engine.connect() as connection:
|
||||
self.assertIn(
|
||||
"a2b3c4d5e6f9",
|
||||
"a2b3c4d5e701",
|
||||
set(MigrationContext.configure(connection).get_current_heads()),
|
||||
)
|
||||
finally:
|
||||
|
||||
@@ -22,6 +22,7 @@ from govoplan_files.backend.routes.shares import router as shares_router
|
||||
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||
from govoplan_files.backend.routes.managed_archives import router as managed_archives_router
|
||||
|
||||
|
||||
class FilesRouterContractTests(unittest.TestCase):
|
||||
@@ -42,6 +43,7 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
lifecycle_router,
|
||||
listing_router,
|
||||
uploads_router,
|
||||
managed_archives_router,
|
||||
connector_settings_router,
|
||||
connector_io_router,
|
||||
connector_profiles_router,
|
||||
@@ -57,7 +59,7 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
actual = self._operation_keys(router)
|
||||
|
||||
self.assertEqual(expected, actual)
|
||||
self.assertEqual(63, len(actual))
|
||||
self.assertEqual(67, len(actual))
|
||||
self.assertFalse(
|
||||
[operation for operation, count in Counter(actual).items() if count > 1]
|
||||
)
|
||||
@@ -69,6 +71,10 @@ class FilesRouterContractTests(unittest.TestCase):
|
||||
|
||||
self.assertIn((("POST",), "/files/archive-preview"), routes)
|
||||
self.assertIn((("POST",), "/files/archive-confirm"), routes)
|
||||
self.assertIn((("GET",), "/files/archive-progress/{operation_id}"), routes)
|
||||
self.assertIn((("DELETE",), "/files/archive-staging/{stage_id}"), routes)
|
||||
self.assertIn((("POST",), "/files/{file_id}/archive-preview"), routes)
|
||||
self.assertIn((("POST",), "/files/{file_id}/archive-confirm"), routes)
|
||||
|
||||
def test_connector_routes_keep_existing_api_paths(self) -> None:
|
||||
routes = {
|
||||
|
||||
Executable
+199
@@ -0,0 +1,199 @@
|
||||
from datetime import datetime, timezone
|
||||
import importlib
|
||||
import unittest
|
||||
|
||||
from alembic.migration import MigrationContext
|
||||
from alembic.operations import Operations
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
Column,
|
||||
MetaData,
|
||||
String,
|
||||
Table,
|
||||
create_engine,
|
||||
inspect,
|
||||
select,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_access.backend.db import models as _access_models # register FK metadata
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.common import FileSourceConflict
|
||||
from govoplan_files.backend.storage.files import find_asset_by_source
|
||||
from govoplan_files.backend.storage.provenance import source_identity_hash
|
||||
|
||||
|
||||
def provenance(external="remote-1"):
|
||||
return {"connector_id": " connector ", "provider": "s3", "external_id": external}
|
||||
|
||||
|
||||
class SourceIdentityIndexTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite:///:memory:")
|
||||
for table in (
|
||||
_access_models.Account.__table__,
|
||||
_access_models.User.__table__,
|
||||
_access_models.Group.__table__,
|
||||
FileAsset.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
):
|
||||
table.create(self.engine)
|
||||
self.session = Session(self.engine)
|
||||
|
||||
def tearDown(self):
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def asset(self, name, *, tenant="tenant", owner="owner", source=None, group=False):
|
||||
row = FileAsset(
|
||||
id=name,
|
||||
tenant_id=tenant,
|
||||
owner_type="group" if group else "user",
|
||||
owner_user_id=None if group else owner,
|
||||
owner_group_id=owner if group else None,
|
||||
display_path=f"{name}.txt",
|
||||
filename=f"{name}.txt",
|
||||
metadata_={"source_provenance": source or provenance()},
|
||||
)
|
||||
self.session.add(row)
|
||||
self.session.flush()
|
||||
return row
|
||||
|
||||
def find(self, source=None, *, tenant="tenant", owner="owner", group=False):
|
||||
return find_asset_by_source(
|
||||
self.session,
|
||||
tenant_id=tenant,
|
||||
owner_type="group" if group else "user",
|
||||
owner_id=owner,
|
||||
source_provenance=source or provenance(),
|
||||
)
|
||||
|
||||
def test_index_lookup_preserves_tenant_owner_provider_and_external_identity(self):
|
||||
row = self.asset("wanted")
|
||||
self.asset("other-tenant", tenant="another")
|
||||
self.asset("other-owner", owner="another")
|
||||
self.asset("group", group=True)
|
||||
self.asset("other-provider", source={**provenance(), "provider": "webdav"})
|
||||
for index in range(100):
|
||||
self.asset(f"unrelated-{index}", source=provenance(f"remote-{index + 2}"))
|
||||
self.assertIs(row, self.find())
|
||||
self.assertIs(row, self.find()) # retry
|
||||
self.assertEqual("group", self.find(group=True).id)
|
||||
statement = (
|
||||
select(FileAsset.id)
|
||||
.where(
|
||||
FileAsset.tenant_id == "tenant",
|
||||
FileAsset.owner_type == "user",
|
||||
FileAsset.owner_user_id == "owner",
|
||||
FileAsset.source_identity_hash == source_identity_hash(provenance()),
|
||||
FileAsset.deleted_at.is_(None),
|
||||
)
|
||||
.limit(2)
|
||||
)
|
||||
sql = str(
|
||||
statement.compile(self.engine, compile_kwargs={"literal_binds": True})
|
||||
)
|
||||
plan = self.session.execute(text("EXPLAIN QUERY PLAN " + sql)).all()
|
||||
self.assertIn("ix_file_assets_user_source", str(plan))
|
||||
self.assertIn("LIMIT 2", sql)
|
||||
|
||||
def test_metadata_copy_delete_restore_and_duplicates_never_choose_a_winner(self):
|
||||
row = self.asset("original")
|
||||
copy = self.asset("copy", source=dict(row.metadata_["source_provenance"]))
|
||||
self.assertEqual(row.source_identity_hash, copy.source_identity_hash)
|
||||
with self.assertRaisesRegex(FileSourceConflict, "Multiple active"):
|
||||
self.find()
|
||||
copy.deleted_at = datetime.now(timezone.utc)
|
||||
self.session.flush()
|
||||
self.assertIs(row, self.find())
|
||||
copy.deleted_at = None
|
||||
self.session.flush()
|
||||
with self.assertRaises(FileSourceConflict):
|
||||
self.find()
|
||||
copy.metadata_ = {"source_provenance": provenance("changed")}
|
||||
self.session.flush()
|
||||
self.assertIs(row, self.find())
|
||||
self.assertIs(copy, self.find(provenance("changed")))
|
||||
copy.metadata_ = {}
|
||||
self.session.flush()
|
||||
self.assertIsNone(copy.source_identity_hash)
|
||||
self.assertIsNone(self.find(provenance("changed")))
|
||||
|
||||
def test_backfill_retains_all_legacy_records_and_metadata_in_bounded_batches(self):
|
||||
migration = importlib.import_module(
|
||||
"govoplan_files.backend.migrations.versions.a2b3c4d5e701_file_source_identity_index"
|
||||
)
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
metadata = MetaData()
|
||||
old = Table(
|
||||
"file_assets",
|
||||
metadata,
|
||||
Column("id", String, primary_key=True),
|
||||
Column("tenant_id", String),
|
||||
Column("owner_type", String),
|
||||
Column("owner_user_id", String),
|
||||
Column("owner_group_id", String),
|
||||
Column("metadata", JSON),
|
||||
)
|
||||
metadata.create_all(engine)
|
||||
rows = [
|
||||
{
|
||||
"id": f"{index:04d}",
|
||||
"tenant_id": "tenant",
|
||||
"owner_type": "user",
|
||||
"owner_user_id": "owner",
|
||||
"metadata": {
|
||||
"source_provenance": provenance(
|
||||
"duplicate" if index < 2 else str(index)
|
||||
)
|
||||
},
|
||||
}
|
||||
for index in range(503)
|
||||
]
|
||||
rows.append(
|
||||
{
|
||||
"id": "no-source",
|
||||
"tenant_id": "tenant",
|
||||
"owner_type": "user",
|
||||
"owner_user_id": "owner",
|
||||
"metadata": {"unrelated": [1, 2]},
|
||||
}
|
||||
)
|
||||
try:
|
||||
with engine.begin() as connection:
|
||||
connection.execute(old.insert(), rows)
|
||||
with Operations.context(MigrationContext.configure(connection)):
|
||||
migration.upgrade()
|
||||
new = Table("file_assets", MetaData(), autoload_with=connection)
|
||||
actual = (
|
||||
connection.execute(select(new).order_by(new.c.id)).mappings().all()
|
||||
)
|
||||
self.assertEqual(len(rows), len(actual))
|
||||
self.assertEqual(
|
||||
[row["metadata"] for row in rows],
|
||||
[row["metadata"] for row in actual],
|
||||
)
|
||||
self.assertEqual(
|
||||
actual[0]["source_identity_hash"], actual[1]["source_identity_hash"]
|
||||
)
|
||||
for row in actual:
|
||||
self.assertEqual(
|
||||
source_identity_hash(row["metadata"].get("source_provenance")),
|
||||
row["source_identity_hash"],
|
||||
)
|
||||
indexes = inspect(connection).get_indexes("file_assets")
|
||||
self.assertEqual(2, len(indexes))
|
||||
self.assertFalse(any(index["unique"] for index in indexes))
|
||||
with Operations.context(MigrationContext.configure(connection)):
|
||||
migration.downgrade()
|
||||
self.assertEqual(
|
||||
len(rows), len(connection.execute(select(old.c.id)).all())
|
||||
)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,274 @@
|
||||
"""Response-only parity/query regression with isolated synthetic SQLite data.
|
||||
|
||||
This does not benchmark imports, mutate storage, or replace route authorization.
|
||||
The isolated schema uses string columns for external module identities instead
|
||||
of importing optional modules. All response, share-state and Campaign-use
|
||||
queries execute through the real Files ORM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import unittest
|
||||
|
||||
from sqlalchemy import Column, MetaData, String, Table, create_engine, event, inspect
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.types import NullType
|
||||
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.route_support import _asset_list_response, _asset_response
|
||||
|
||||
|
||||
class UploadResponseBatchingTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite://")
|
||||
self.addCleanup(self.engine.dispose)
|
||||
metadata = MetaData()
|
||||
for model in (
|
||||
FileBlob,
|
||||
FileAsset,
|
||||
FileVersion,
|
||||
FileShare,
|
||||
CampaignAttachmentUse,
|
||||
):
|
||||
Table(
|
||||
model.__tablename__,
|
||||
metadata,
|
||||
*(
|
||||
Column(
|
||||
column.name,
|
||||
String(36)
|
||||
if isinstance(column.type, NullType)
|
||||
else column.type,
|
||||
primary_key=column.primary_key,
|
||||
nullable=column.nullable,
|
||||
default=column.default.arg
|
||||
if column.default is not None
|
||||
else None,
|
||||
server_default=column.server_default.arg
|
||||
if column.server_default is not None
|
||||
else None,
|
||||
)
|
||||
for column in model.__table__.columns
|
||||
),
|
||||
)
|
||||
metadata.create_all(self.engine)
|
||||
now = datetime.now(UTC)
|
||||
with self.engine.begin() as connection:
|
||||
|
||||
def seed(row):
|
||||
# Core inserts into separate fixture metadata avoid resolving
|
||||
# foreign identities or changing global ORM declarations.
|
||||
values = {
|
||||
attribute.columns[0].name: getattr(row, attribute.key)
|
||||
for attribute in inspect(type(row)).column_attrs
|
||||
if attribute.key in row.__dict__
|
||||
}
|
||||
connection.execute(metadata.tables[row.__tablename__].insert(), values)
|
||||
|
||||
for index in range(64):
|
||||
asset_id = f"asset-{index:03}"
|
||||
version_id = f"version-{index:03}"
|
||||
blob_id = f"blob-{index:03}"
|
||||
checksum = f"{index:064x}"
|
||||
owner_type = "group" if index % 2 else "user"
|
||||
seed(
|
||||
FileBlob(
|
||||
id=blob_id,
|
||||
tenant_id="tenant-fixture",
|
||||
storage_backend="local",
|
||||
storage_key=f"synthetic/{blob_id}",
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=index + 10,
|
||||
content_type="text/plain",
|
||||
ref_count=1,
|
||||
)
|
||||
)
|
||||
seed(
|
||||
FileAsset(
|
||||
id=asset_id,
|
||||
tenant_id="tenant-fixture",
|
||||
owner_type=owner_type,
|
||||
owner_user_id="user-fixture" if owner_type == "user" else None,
|
||||
owner_group_id="group-fixture"
|
||||
if owner_type == "group"
|
||||
else None,
|
||||
current_version_id=version_id,
|
||||
display_path=f"imported/{index:03}.txt",
|
||||
filename=f"{index:03}.txt",
|
||||
description=f"Synthetic response {index}",
|
||||
retained_until=now + timedelta(days=365) if index % 2 else None,
|
||||
legal_hold=bool(index % 3),
|
||||
lifecycle_revision=index + 2,
|
||||
lifecycle_reason=f"Retention decision {index}",
|
||||
deleted_at=now - timedelta(days=1) if index % 7 == 0 else None,
|
||||
metadata_={
|
||||
"fixture": index,
|
||||
"source_provenance": {
|
||||
"source_type": "archive",
|
||||
"revision": f"source-{index}",
|
||||
"metadata": {
|
||||
"archive_source_file_id": "synthetic-archive"
|
||||
},
|
||||
},
|
||||
"source_revision": f"source-{index}",
|
||||
},
|
||||
)
|
||||
)
|
||||
seed(
|
||||
FileVersion(
|
||||
id=version_id,
|
||||
tenant_id="tenant-fixture",
|
||||
file_asset_id=asset_id,
|
||||
blob_id=blob_id,
|
||||
version_number=1,
|
||||
filename_at_upload=f"{index:03}.txt",
|
||||
display_path_at_upload=f"imported/{index:03}.txt",
|
||||
size_bytes=index + 10,
|
||||
checksum_sha256=checksum,
|
||||
content_type="text/plain",
|
||||
)
|
||||
)
|
||||
for rank, state in enumerate(("active", "expired", "revoked")):
|
||||
seed(
|
||||
FileShare(
|
||||
id=f"share-{index:03}-{state}",
|
||||
tenant_id="tenant-fixture",
|
||||
file_asset_id=asset_id,
|
||||
target_type="group" if rank == 0 else "user",
|
||||
target_id=f"target-{state}",
|
||||
permission="write" if rank == 0 else "read",
|
||||
created_by_user_id="user-fixture",
|
||||
created_at=now - timedelta(hours=rank + 1),
|
||||
expires_at=now - timedelta(days=1)
|
||||
if state == "expired"
|
||||
else None,
|
||||
revoked_at=now - timedelta(days=1)
|
||||
if state == "revoked"
|
||||
else None,
|
||||
revoked_by_user_id="user-fixture"
|
||||
if state == "revoked"
|
||||
else None,
|
||||
)
|
||||
)
|
||||
# Duplicate sent evidence must still produce one response;
|
||||
# built-only usage and missing usage are not audit-relevant.
|
||||
stages = (
|
||||
("sent", "sent")
|
||||
if index % 3 == 0
|
||||
else ("built",)
|
||||
if index % 3 == 1
|
||||
else ()
|
||||
)
|
||||
for use_index, stage in enumerate(stages):
|
||||
seed(
|
||||
CampaignAttachmentUse(
|
||||
id=f"use-{index:03}-{use_index}",
|
||||
tenant_id="tenant-fixture",
|
||||
campaign_id="campaign-fixture",
|
||||
campaign_version_id="campaign-version-fixture",
|
||||
campaign_job_id=f"job-{use_index}",
|
||||
file_asset_id=asset_id,
|
||||
file_version_id=version_id,
|
||||
file_blob_id=blob_id,
|
||||
filename_used=f"{index:03}.txt",
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=index + 10,
|
||||
content_type="text/plain",
|
||||
use_stage=stage,
|
||||
)
|
||||
)
|
||||
|
||||
def render(self, *, batched: bool, include_shares: bool):
|
||||
statements = []
|
||||
|
||||
def record_select(
|
||||
connection, cursor, statement, parameters, context, executemany
|
||||
):
|
||||
if statement.lstrip().upper().startswith("SELECT"):
|
||||
statements.append(statement)
|
||||
|
||||
# A fresh session ensures neither path receives preloaded versions or
|
||||
# blobs. Loading the already-authorized asset list is deliberately
|
||||
# outside the measurement; this measures response construction only.
|
||||
with Session(self.engine) as session:
|
||||
assets = session.query(FileAsset).order_by(FileAsset.id).all()
|
||||
event.listen(self.engine, "before_cursor_execute", record_select)
|
||||
try:
|
||||
responses = (
|
||||
_asset_list_response(session, assets, include_shares=include_shares)
|
||||
if batched
|
||||
else [
|
||||
_asset_response(session, asset, include_shares=include_shares)
|
||||
for asset in assets
|
||||
]
|
||||
)
|
||||
result = [response.model_dump(mode="json") for response in responses]
|
||||
finally:
|
||||
event.remove(self.engine, "before_cursor_execute", record_select)
|
||||
return result, statements
|
||||
|
||||
def test_batched_response_matches_all_metadata_lifecycle_shares_and_audit_flags(
|
||||
self,
|
||||
):
|
||||
for include_shares in (False, True):
|
||||
with self.subTest(include_shares=include_shares):
|
||||
batched, _ = self.render(batched=True, include_shares=include_shares)
|
||||
individual, _ = self.render(
|
||||
batched=False, include_shares=include_shares
|
||||
)
|
||||
self.assertEqual(individual, batched)
|
||||
self.assertEqual(64, len(batched))
|
||||
for index, item in enumerate(batched):
|
||||
self.assertEqual(f"asset-{index:03}", item["id"])
|
||||
self.assertEqual(index + 2, item["lifecycle_revision"])
|
||||
self.assertEqual(
|
||||
f"Retention decision {index}", item["lifecycle_reason"]
|
||||
)
|
||||
self.assertEqual(
|
||||
bool(index % 2), item["retained_until"] is not None
|
||||
)
|
||||
self.assertEqual(bool(index % 3), item["legal_hold"])
|
||||
self.assertEqual(index % 3 == 0, item["audit_relevant"])
|
||||
self.assertEqual(index % 7 == 0, item["deleted_at"] is not None)
|
||||
self.assertEqual(f"source-{index}", item["source_revision"])
|
||||
if include_shares:
|
||||
self.assertEqual(
|
||||
[
|
||||
f"share-{index:03}-{state}"
|
||||
for state in ("active", "expired", "revoked")
|
||||
],
|
||||
[share["id"] for share in item["shares"]],
|
||||
)
|
||||
self.assertEqual(
|
||||
[True, False, False],
|
||||
[share["active"] for share in item["shares"]],
|
||||
)
|
||||
else:
|
||||
self.assertEqual([], item["shares"])
|
||||
|
||||
def test_64_asset_response_batches_real_selects_instead_of_querying_per_item(self):
|
||||
for include_shares, per_item_queries, batch_queries in (
|
||||
(False, 192, 2),
|
||||
(True, 256, 3),
|
||||
):
|
||||
with self.subTest(include_shares=include_shares):
|
||||
individual, individual_queries = self.render(
|
||||
batched=False, include_shares=include_shares
|
||||
)
|
||||
batched, batched_queries = self.render(
|
||||
batched=True, include_shares=include_shares
|
||||
)
|
||||
self.assertEqual(individual, batched)
|
||||
self.assertEqual(per_item_queries, len(individual_queries))
|
||||
self.assertEqual(batch_queries, len(batched_queries))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+5
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.24",
|
||||
"version": "0.1.27",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
@@ -14,6 +14,9 @@
|
||||
"./styles/file-manager.css": "./src/styles/file-manager.css"
|
||||
},
|
||||
"scripts": {
|
||||
"test:reload-client": "node scripts/test-files-reload-client.mjs",
|
||||
"test:archive-client": "node scripts/test-archive-client.mjs",
|
||||
"test:managed-archive": "node scripts/test-managed-archive-structure.mjs",
|
||||
"test:connector-folder-sync": "node scripts/test-connector-folder-sync-structure.mjs",
|
||||
"test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs",
|
||||
"test:file-property-filters": "node scripts/test-file-property-filters-structure.mjs",
|
||||
@@ -28,7 +31,7 @@
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.18"
|
||||
"@govoplan/core-webui": "^0.1.45"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
// Execute the actual Files client and Core HTTP transport. Only browser I/O
|
||||
// (fetch, XHR, timers, document cookies) and temporal context are substituted.
|
||||
// No source slicing, regex assertions, live credentials, or server is involved.
|
||||
import assert from "node:assert/strict";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import vm from "node:vm";
|
||||
|
||||
const coreRoot = new URL("../../../govoplan-core/webui/", import.meta.url);
|
||||
const require = createRequire(new URL("package.json", coreRoot));
|
||||
const { transformSync } = require("esbuild");
|
||||
const compile = (url) => transformSync(readFileSync(url, "utf8"), {
|
||||
loader: "ts", format: "cjs", target: "es2022", sourcefile: url.pathname,
|
||||
define: { "import.meta.env": "{}" }
|
||||
}).code;
|
||||
const coreCode = compile(new URL("src/api/client.ts", coreRoot));
|
||||
const filesCode = compile(new URL("../src/api/files.ts", import.meta.url));
|
||||
const settle = () => new Promise((resolve) => setImmediate(resolve));
|
||||
const plain = (value) => JSON.parse(JSON.stringify(value));
|
||||
const settings = { apiBaseUrl: "https://fixture.invalid", accessToken: "", apiKey: "" };
|
||||
const destination = { owner_type: "user", owner_id: "fixture-user", path: "imports" };
|
||||
const previewResponse = (changes = {}) => ({
|
||||
preview_token: "preview-token", staged_upload_id: "stage-one", expires_at: "2099-01-01T00:00:00Z",
|
||||
entries: [], file_count: 1, directory_count: 0, archive_format: "zip",
|
||||
requires_password: false, password_verified: true, compressed_size_bytes: 100,
|
||||
expanded_size_bytes: 10, ...changes
|
||||
});
|
||||
const json = (body, status = 200) => new Response(JSON.stringify(body), {
|
||||
status, headers: { "content-type": "application/json" }
|
||||
});
|
||||
const deferred = () => {
|
||||
let resolve;
|
||||
const promise = new Promise((complete) => { resolve = complete; });
|
||||
return { promise, resolve };
|
||||
};
|
||||
|
||||
function harness() {
|
||||
const requests = [];
|
||||
const xhrRequests = [];
|
||||
const timers = new Map();
|
||||
let timerId = 0;
|
||||
let now = Date.parse("2026-09-07T10:00:00Z");
|
||||
let onFetch = () => { throw new Error("Unexpected fetch"); };
|
||||
let onXhr = () => { throw new Error("Unexpected XHR"); };
|
||||
class ClockDate extends Date { static now() { return now; } }
|
||||
class FakeXhr {
|
||||
upload = {};
|
||||
headers = new Headers();
|
||||
open(method, url) { this.method = method; this.url = url; }
|
||||
setRequestHeader(name, value) { this.headers.set(name, value); }
|
||||
getResponseHeader(name) { return name.toLowerCase() === "content-type" ? "application/json" : null; }
|
||||
send(body) { this.body = body; xhrRequests.push(this); onXhr(this); }
|
||||
progress(loaded, total, lengthComputable = true) { this.upload.onprogress?.({ loaded, total, lengthComputable }); }
|
||||
respond(body, status = 200) { this.status = status; this.statusText = status === 200 ? "OK" : "Error"; this.responseText = JSON.stringify(body); this.onload?.(); }
|
||||
}
|
||||
const context = vm.createContext({
|
||||
File, Blob, FormData, Headers, Response, URL, URLSearchParams, AbortController,
|
||||
Date: ClockDate, crypto: { randomUUID }, XMLHttpRequest: FakeXhr,
|
||||
document: { cookie: "" }, console,
|
||||
fetch: async (url, init) => {
|
||||
const request = { url, path: new URL(url).pathname, method: init?.method ?? "GET", ...init };
|
||||
requests.push(request);
|
||||
return onFetch(request);
|
||||
},
|
||||
setTimeout: (callback, delay) => { const id = ++timerId; timers.set(id, { callback, at: now + delay }); return id; },
|
||||
clearTimeout: (id) => timers.delete(id)
|
||||
});
|
||||
function load(code, dependencies) {
|
||||
context.module = { exports: {} };
|
||||
context.exports = context.module.exports;
|
||||
context.require = (name) => {
|
||||
assert.ok(name in dependencies, `Unexpected client import: ${name}`);
|
||||
return dependencies[name];
|
||||
};
|
||||
vm.runInContext(`(function(module, exports, require) {\n${code}\n})(module, exports, require);`, context);
|
||||
return context.module.exports;
|
||||
}
|
||||
const core = load(coreCode, { "../platform/temporal": { temporalRequestHeaders: () => ({}) } });
|
||||
const api = load(filesCode, { "@govoplan/core-webui": core });
|
||||
return {
|
||||
api, core, requests, xhrRequests, timers,
|
||||
fetch(handler) { onFetch = handler; },
|
||||
xhr(handler) { onXhr = handler; },
|
||||
async advance(milliseconds) {
|
||||
const target = now + milliseconds;
|
||||
while (true) {
|
||||
const next = [...timers.entries()].filter(([, timer]) => timer.at <= target).sort((left, right) => left[1].at - right[1].at)[0];
|
||||
if (!next) break;
|
||||
now = next[1].at;
|
||||
timers.delete(next[0]);
|
||||
next[1].callback();
|
||||
await settle();
|
||||
}
|
||||
now = target;
|
||||
await settle();
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
let passed = 0;
|
||||
async function test(name, run) {
|
||||
await run();
|
||||
passed += 1;
|
||||
console.log(`PASS ${name}`);
|
||||
}
|
||||
|
||||
await test("locally expired stage is released and preview reuploads the original file", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
let previews = 0;
|
||||
env.fetch((request) => {
|
||||
if (request.method === "DELETE") return new Response(null, { status: 204 });
|
||||
assert.equal(request.path, "/api/v1/files/archive-preview");
|
||||
assert.equal(request.body.get("file"), file);
|
||||
assert.equal(request.body.has("staged_upload_id"), false);
|
||||
return json(previewResponse(++previews === 1 ? { expires_at: "2026-09-07T10:00:01Z" } : { staged_upload_id: "stage-two" }));
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, destination);
|
||||
await env.advance(1100);
|
||||
const result = await env.api.previewArchiveUpload(settings, file, destination);
|
||||
assert.equal(previews, 2);
|
||||
assert.equal(result.staged_upload_id, "stage-two");
|
||||
assert.equal(env.requests.filter((request) => request.method === "DELETE").length, 1);
|
||||
assert.equal(env.requests.filter((request) => request.path.endsWith("archive-confirm")).length, 0);
|
||||
});
|
||||
|
||||
await test("server-expired cached preview retries only the read-only preview once with file bytes", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
let calls = 0;
|
||||
env.fetch((request) => {
|
||||
assert.equal(request.path, "/api/v1/files/archive-preview");
|
||||
calls += 1;
|
||||
if (calls === 2) {
|
||||
assert.equal(request.body.get("staged_upload_id"), "stage-one");
|
||||
assert.equal(request.body.get("preview_token"), "preview-token");
|
||||
assert.equal(request.body.has("file"), false);
|
||||
return json({ detail: "Archive preview expired" }, 410);
|
||||
}
|
||||
assert.equal(request.body.get("file"), file);
|
||||
assert.equal(request.body.has("staged_upload_id"), false);
|
||||
return json(previewResponse({ staged_upload_id: calls === 1 ? "stage-one" : "stage-two" }));
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, destination);
|
||||
assert.equal((await env.api.previewArchiveUpload(settings, file, destination)).staged_upload_id, "stage-two");
|
||||
assert.equal(calls, 3, "one initial preview, one expired-stage attempt, one safe reupload");
|
||||
});
|
||||
|
||||
await test("a failed fresh reupload cannot recurse into repeated preview attempts", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
let calls = 0;
|
||||
env.fetch(() => ++calls === 1 ? json(previewResponse()) : json({ detail: "Expired" }, 410));
|
||||
await env.api.previewArchiveUpload(settings, file, destination);
|
||||
await assert.rejects(env.api.previewArchiveUpload(settings, file, destination), (error) => error instanceof env.core.ApiError && error.status === 410);
|
||||
assert.equal(calls, 3);
|
||||
});
|
||||
|
||||
await test("failed confirmation never automatically retries or falls back to a file reupload", async () => {
|
||||
for (const status of [400, 410, 500]) {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
env.fetch((request) => request.path.endsWith("archive-preview") ? json(previewResponse()) : json({ detail: "Confirmation failed" }, status));
|
||||
const preview = await env.api.previewArchiveUpload(settings, file, destination);
|
||||
await assert.rejects(env.api.confirmArchiveUpload(settings, file, { ...destination, preview_token: preview.preview_token, selected_paths: ["one.txt"] }), (error) => error instanceof env.core.ApiError && error.status === status);
|
||||
const confirmations = env.requests.filter((request) => request.path.endsWith("archive-confirm"));
|
||||
assert.equal(confirmations.length, 1);
|
||||
assert.equal(confirmations[0].body.get("staged_upload_id"), "stage-one");
|
||||
assert.equal(confirmations[0].body.has("file"), false);
|
||||
assert.equal(env.requests.length, 2);
|
||||
}
|
||||
});
|
||||
|
||||
await test("progress polling failures neither reject nor retry the authoritative confirmation POST", async () => {
|
||||
for (const failure of [404, 500, "network"]) {
|
||||
const env = harness();
|
||||
const pending = deferred();
|
||||
const events = [];
|
||||
env.fetch((request) => {
|
||||
if (request.method === "POST") return pending.promise;
|
||||
if (failure === "network") throw new TypeError("Offline telemetry");
|
||||
return json({ detail: "Telemetry unavailable" }, failure);
|
||||
});
|
||||
const confirmation = env.api.confirmManagedArchive(settings, "managed-file", {
|
||||
...destination, source_version_id: "version-one", preview_token: "token", selected_paths: ["one.txt"],
|
||||
onArchiveProgress: (event) => events.push(event)
|
||||
});
|
||||
let settled = false;
|
||||
void confirmation.then(() => { settled = true; });
|
||||
await env.advance(1200);
|
||||
assert.equal(settled, false);
|
||||
assert.equal(events.length, 0);
|
||||
assert.equal(env.requests.filter((request) => request.method === "POST").length, 1);
|
||||
const polls = env.requests.filter((request) => request.method === "GET");
|
||||
assert.ok(polls.length >= 2);
|
||||
assert.ok(polls.every((request) => request.cache === "no-store"));
|
||||
pending.resolve(json({ files: [{ id: "one", size_bytes: 12345 }, { id: "two", size_bytes: 678 }] }));
|
||||
const result = await confirmation;
|
||||
assert.equal(result.files.length, 2);
|
||||
assert.deepEqual(plain(events.at(-1)), { phase: "complete", status: "complete", completed_files: 2, total_files: 2, completed_bytes: 13023, total_bytes: 13023 });
|
||||
assert.equal(env.timers.size, 0);
|
||||
assert.ok(polls.every((request) => request.signal.aborted));
|
||||
await env.advance(2000);
|
||||
assert.equal(env.requests.filter((request) => request.method === "POST").length, 1);
|
||||
assert.equal(env.requests.filter((request) => request.method === "GET").length, polls.length);
|
||||
}
|
||||
});
|
||||
|
||||
await test("confirmation errors stop polling without fabricating a successful completion", async () => {
|
||||
const env = harness();
|
||||
const pending = deferred();
|
||||
const events = [];
|
||||
env.fetch((request) => request.method === "POST" ? pending.promise : json({ detail: "Not yet available" }, 404));
|
||||
const confirmation = env.api.confirmManagedArchive(settings, "managed-file", {
|
||||
...destination, source_version_id: "version-one", preview_token: "token", selected_paths: ["one.txt"],
|
||||
onArchiveProgress: (event) => events.push(event)
|
||||
});
|
||||
const rejected = assert.rejects(confirmation, (error) => error instanceof env.core.ApiError && error.status === 409);
|
||||
await env.advance(200);
|
||||
pending.resolve(json({ detail: "Destination conflict" }, 409));
|
||||
await rejected;
|
||||
assert.equal(events.length, 0);
|
||||
assert.equal(env.timers.size, 0);
|
||||
assert.equal(env.requests.filter((request) => request.method === "POST").length, 1);
|
||||
});
|
||||
|
||||
await test("a late in-flight poll cannot replace the authoritative successful result", async () => {
|
||||
const env = harness();
|
||||
const post = deferred();
|
||||
const poll = deferred();
|
||||
const events = [];
|
||||
env.fetch((request) => request.method === "POST" ? post.promise : poll.promise);
|
||||
const confirmation = env.api.confirmManagedArchive(settings, "managed-file", {
|
||||
...destination, source_version_id: "version-one", preview_token: "token", selected_paths: ["one.txt"],
|
||||
onArchiveProgress: (event) => events.push(event)
|
||||
});
|
||||
await env.advance(200);
|
||||
assert.equal(env.requests.filter((request) => request.method === "GET").length, 1);
|
||||
post.resolve(json({ files: [{ id: "one", size_bytes: 12345 }] }));
|
||||
await confirmation;
|
||||
poll.resolve(json({ phase: "extracting", status: "running", completed_files: 0, total_files: 1, completed_bytes: 10, total_bytes: 12345 }));
|
||||
await settle();
|
||||
assert.equal(events.length, 1);
|
||||
assert.equal(events[0].status, "complete");
|
||||
assert.equal(events[0].completed_bytes, 12345);
|
||||
assert.equal(env.timers.size, 0);
|
||||
});
|
||||
|
||||
await test("XHR preview reports measured byte counters and preserves them at transfer completion", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
const events = [];
|
||||
env.xhr((xhr) => {
|
||||
assert.equal(xhr.body.get("file"), file);
|
||||
assert.equal(xhr.withCredentials, true);
|
||||
xhr.progress(1578, 4096);
|
||||
xhr.progress(4096, 4096);
|
||||
xhr.respond(previewResponse());
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, { ...destination, onProgress: (event) => events.push(event) });
|
||||
assert.deepEqual(plain(events.slice(1)), [
|
||||
{ loaded: 1578, total: 4096, percentage: 39 },
|
||||
{ loaded: 4096, total: 4096, percentage: 100 },
|
||||
{ loaded: 4096, total: 4096, percentage: 100 }
|
||||
]);
|
||||
assert.equal(env.xhrRequests.length, 1);
|
||||
assert.equal(env.requests.length, 0);
|
||||
});
|
||||
|
||||
await test("unknown-length XHR uploads never fabricate total bytes", async () => {
|
||||
const env = harness();
|
||||
const events = [];
|
||||
env.xhr((xhr) => { xhr.progress(3917, 0, false); xhr.respond(previewResponse()); });
|
||||
await env.api.previewArchiveUpload(settings, new File(["fixture"], "fixture.zip"), { ...destination, onProgress: (event) => events.push(event) });
|
||||
assert.equal(events.at(-2).percentage, null);
|
||||
assert.equal(events.at(-1).loaded, 3917);
|
||||
assert.equal(events.at(-1).total, undefined);
|
||||
assert.equal(events.at(-1).percentage, 100);
|
||||
});
|
||||
|
||||
await test("password repreview and confirmation reuse staging without another XHR transfer", async () => {
|
||||
const env = harness();
|
||||
const file = new File(["archive fixture"], "fixture.zip");
|
||||
env.xhr((xhr) => { xhr.progress(1024, 1024); xhr.respond(previewResponse()); });
|
||||
env.fetch((request) => {
|
||||
assert.equal(request.body.get("staged_upload_id"), "stage-one");
|
||||
assert.equal(request.body.has("file"), false);
|
||||
assert.equal(request.body.get("password"), "fixture-only-password");
|
||||
return request.path.endsWith("archive-preview") ? json(previewResponse({ preview_token: "verified-token" })) : json({ files: [] });
|
||||
});
|
||||
await env.api.previewArchiveUpload(settings, file, { ...destination, onProgress: () => {} });
|
||||
const verified = await env.api.previewArchiveUpload(settings, file, { ...destination, password: "fixture-only-password", onProgress: () => assert.fail("Staged repreview must not claim another upload") });
|
||||
await env.api.confirmArchiveUpload(settings, file, {
|
||||
...destination, password: "fixture-only-password", preview_token: verified.preview_token, selected_paths: ["one.txt"],
|
||||
onProgress: () => assert.fail("Staged confirmation must not claim another upload")
|
||||
});
|
||||
assert.equal(env.xhrRequests.length, 1);
|
||||
assert.equal(env.requests.length, 2);
|
||||
});
|
||||
|
||||
console.log(`Archive client behavior: ${passed} tests passed using real Files and Core transport code.`);
|
||||
@@ -0,0 +1,80 @@
|
||||
// Run the owning Files read helpers and real Core request cache, replacing only network I/O.
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import vm from "node:vm";
|
||||
|
||||
const coreRoot = new URL("../../../govoplan-core/webui/", import.meta.url);
|
||||
const require = createRequire(new URL("package.json", coreRoot));
|
||||
const { transformSync } = require("esbuild");
|
||||
const compile = (url) => transformSync(readFileSync(url, "utf8"), {
|
||||
loader: "ts", format: "cjs", target: "es2022", define: { "import.meta.env": "{}" }
|
||||
}).code;
|
||||
const coreCode = compile(new URL("src/api/client.ts", coreRoot));
|
||||
const filesCode = compile(new URL("../src/api/files.ts", import.meta.url));
|
||||
const settings = { apiBaseUrl: "https://fixture.invalid", apiKey: "", accessToken: "" };
|
||||
const owner = { owner_type: "user", owner_id: "fixture-user" };
|
||||
const fresh = { cache: "no-store" };
|
||||
|
||||
function harness(respond) {
|
||||
const requests = [];
|
||||
const context = vm.createContext({
|
||||
Headers, Response, FormData, URL, URLSearchParams, AbortController, console,
|
||||
document: { cookie: "" },
|
||||
fetch: async (url, init) => {
|
||||
const request = { url: new URL(url), ...init };
|
||||
requests.push(request);
|
||||
assert.equal(init?.method ?? "GET", "GET", "Reload read helpers must never mutate");
|
||||
return new Response(JSON.stringify(respond(request)), { status: 200, headers: { "content-type": "application/json" } });
|
||||
}
|
||||
});
|
||||
function load(code, dependencies) {
|
||||
context.module = { exports: {} };
|
||||
context.exports = context.module.exports;
|
||||
context.require = (name) => {
|
||||
assert.ok(name in dependencies, `Unexpected module import: ${name}`);
|
||||
return dependencies[name];
|
||||
};
|
||||
vm.runInContext(`(function(module, exports, require) {\n${code}\n})(module, exports, require);`, context);
|
||||
return context.module.exports;
|
||||
}
|
||||
const core = load(coreCode, { "../platform/temporal": { temporalRequestHeaders: () => ({}) } });
|
||||
return { api: load(filesCode, { "@govoplan/core-webui": core }), requests };
|
||||
}
|
||||
|
||||
for (const kind of ["spaces", "connector"]) {
|
||||
const { api, requests } = harness(() => kind === "spaces" ? { spaces: [] } : { items: [], path: "source/nested" });
|
||||
const read = (options) => kind === "spaces"
|
||||
? api.listFileSpaces(settings, options)
|
||||
: api.browseFileConnectorProfile(settings, "fixture-profile", { path: "source/nested" }, options);
|
||||
await read();
|
||||
await read();
|
||||
assert.equal(requests.length, 1, "normal reads retain short-lived request reuse");
|
||||
await read(fresh);
|
||||
await read(fresh);
|
||||
assert.equal(requests.length, 3, "each explicit Reload bypasses the shared recent-response cache");
|
||||
assert.equal(requests[2].cache, "no-store");
|
||||
}
|
||||
|
||||
{
|
||||
const { api, requests } = harness(({ url }) => {
|
||||
const next = url.searchParams.has("cursor");
|
||||
if (url.pathname.endsWith("/folders")) return { folders: [], total: 0, next_cursor: next ? null : "folders-2", watermark: "start" };
|
||||
if (url.pathname.endsWith("/delta")) return { full: false, files: [], folders: [], deleted: [], has_more: url.searchParams.get("since") === "start", watermark: url.searchParams.get("since") === "start" ? "middle" : "end" };
|
||||
return { files: [], total: 0, next_cursor: next ? null : "files-2" };
|
||||
});
|
||||
await api.listManagedFileSnapshot(settings, owner, fresh);
|
||||
await api.listManagedFileSnapshot(settings, owner, fresh);
|
||||
assert.equal(requests.length, 12, "two snapshots each re-read both folder/file pages and both reconciliation pages");
|
||||
assert.ok(requests.every((request) => request.cache === "no-store"));
|
||||
}
|
||||
|
||||
{
|
||||
const { api, requests } = harness(({ url }) => ({ files: [], total: 0, next_cursor: url.searchParams.has("cursor") ? null : "filtered-2" }));
|
||||
await api.listFilesByProperties(settings, { ...owner, campaign_usage: "linked", audit_relevant: true }, fresh);
|
||||
await api.listFilesByProperties(settings, { ...owner, campaign_usage: "linked", audit_relevant: true }, fresh);
|
||||
assert.equal(requests.length, 4, "each filtered Reload must re-read every server result page");
|
||||
assert.ok(requests.every((request) => request.cache === "no-store" && request.url.searchParams.get("campaign_usage") === "linked" && request.url.searchParams.get("audit_relevant") === "true"));
|
||||
}
|
||||
|
||||
console.log("Files Reload: 4 real-client cache, read-only, pagination and filter checks passed.");
|
||||
@@ -38,7 +38,17 @@ assert.match(filesPage, /topicId: "files\.workflow\.organize-managed-files"/);
|
||||
assert.match(filesPage, /disabledReason=\{uploadBlocker\}/);
|
||||
assert.match(filesPage, /disabledReason=\{deleteBlocker\}/);
|
||||
assert.match(filesPage, /<ConfirmDialog[\s\S]*tone="danger"/);
|
||||
assert.match(filesPage, /className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen files-page"/);
|
||||
assert.match(filesPage, /<WorkspaceFrame as="main" height="viewport" surface="plain"/);
|
||||
assert.match(filesPage, /const toolbar = <WorkspaceActionBar\s+scope="workspace"\s+variant="collection"\s+refreshable/);
|
||||
assert.match(filesPage, /\{toolbar\}[\s\S]*className=\{`file-manager-shell/);
|
||||
assert.match(filesPage, /const selectionToolbar = <WorkspaceActionBar\s+scope="detail-pane"/);
|
||||
assert.match(filesPage, /<Dialog\s+open=\{toolsPanel !== null\}/);
|
||||
assert.match(filesPage, /destructiveActions=\{<Button variant="danger"/);
|
||||
const reload = filesPage.slice(filesPage.indexOf(" async function reloadCurrentView()"), filesPage.indexOf(" function applyManagedSpaceDelta"));
|
||||
for (const readOperation of ["listManagedFileSnapshot", "resolveFilePatterns", "listFilesByProperties", "browseFileConnectorProfile"]) assert.ok(reload.includes(readOperation));
|
||||
assert.doesNotMatch(reload, /syncConnector|importFile|uploadFile|deleteFile|resetTransientState|setCurrentFolder|clearSelection|setSearchActive/);
|
||||
assert.match(reload, /if \(!isCurrent\(\)\) return;/);
|
||||
assert.doesNotMatch(reload, /setBusy\(/, "manual reload owns only its own busy state");
|
||||
assert.doesNotMatch(filesPage, /window\.(?:alert|confirm)\(/);
|
||||
|
||||
assert.doesNotMatch(`${connector}\n${integrity}\n${filesPage}\n${moduleSource}`, /@govoplan\/(?:campaign|mail|docs)-webui|govoplan_(?:campaign|mail|docs)/);
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import vm from "node:vm";
|
||||
|
||||
const read = (path) => readFileSync(new URL(path, import.meta.url), "utf8");
|
||||
const page = read("../src/features/files/FilesPage.tsx");
|
||||
const menu = read("../src/features/files/components/FileManagerComponents.tsx");
|
||||
const api = read("../src/api/files.ts");
|
||||
const translations = read("../src/i18n/generatedTranslations.ts");
|
||||
|
||||
assert.match(page, /selectedFiles\.length === 1 && selectedFolderPaths\.size === 0 && ARCHIVE_FILENAME_PATTERN/);
|
||||
assert.match(page, /function openManagedArchive[\s\S]*?!canUpload \|\| !canDownload[\s\S]*?setManagedArchiveFile\(file\)/);
|
||||
assert.match(page, /disabledReason=\{unpackBlocker\}/);
|
||||
assert.match(menu, /onClick=\{onUnpackArchive\} disabled=\{!canUnpackArchive\}/);
|
||||
assert.match(page, /file: File \| ManagedFile/);
|
||||
assert.match(page, /previewManagedArchive\(settings, file\.id,[\s\S]*?source_version_id: file\.version_id/);
|
||||
assert.match(page, /confirmManagedArchive\(settings, managedArchiveFile\.id,[\s\S]*?source_version_id: managedArchiveFile\.version_id/);
|
||||
assert.match(page, /archivePreview && \(archiveFile \|\| managedArchiveFile\)/);
|
||||
assert.match(page, /loadArchivePreview\(\(managedArchiveFile \|\| archiveFile\)!/);
|
||||
assert.match(page, /if \(managedArchiveFile\) \{ setArchivePreview\(null\); setSelectedArchivePaths/);
|
||||
assert.match(page, /onClose=\{\(\) => \{ if \(!busy\) closeDialog\(\); \}\}/);
|
||||
assert.match(page, /<LoadingFrame loading=\{uploadActive\}[\s\S]*?indicator="none"[\s\S]*?progress=\{operationProgressValue\}/);
|
||||
assert.match(page, /<div inert=\{uploadActive\}>/);
|
||||
assert.match(page, /onArchiveProgress: \(progress: ArchiveOperationProgress\)/);
|
||||
assert.match(page, /archiveProgress\?\.status === "complete" \? 100/);
|
||||
assert.match(page, /archiveProgress.phase !== "finalizing"[\s\S]*?serverProgressRatio < 1/);
|
||||
assert.match(menu, /closeDisabled=\{busy\}[\s\S]*?closeOnBackdrop=\{!busy\}/);
|
||||
const feedback = page.slice(page.indexOf(" const selectedArchiveBytes ="), page.indexOf(" const connectorLocationLabel ="));
|
||||
function operationFeedback(archiveProgress, uploadPhase = "unpacking", uploadProgress = null) {
|
||||
const context = { archiveProgress, uploadPhase, uploadProgress, selectedArchivePaths: new Set(["one.txt", "empty.txt"]),
|
||||
archivePreview: { entries: [{ path: "one.txt", kind: "file", size_bytes: 3 }, { path: "empty.txt", kind: "file", size_bytes: 0 }] },
|
||||
formatBytes: (value) => `${value} B`, i18nMessage: (key, values) => ({ key, values }) };
|
||||
vm.runInNewContext(`${feedback}\nglobalThis.result = { value: operationProgressValue, label: operationProgressLabel, phase: operationBusyLabel };`, context);
|
||||
return context.result;
|
||||
}
|
||||
const measured = { phase: "extracting", status: "running", completed_files: 1, total_files: 2, completed_bytes: 3, total_bytes: 6 };
|
||||
assert.equal(operationFeedback(measured).value, 50);
|
||||
assert.equal(operationFeedback({ ...measured, phase: "finalizing", completed_files: 2, completed_bytes: 6 }).value, null, "finalization does not fabricate completion");
|
||||
assert.equal(operationFeedback({ ...measured, completed_files: 2, completed_bytes: 6 }).value, null, "processed bytes alone do not mean commit success");
|
||||
assert.equal(operationFeedback({ ...measured, phase: "complete", status: "complete" }).value, 100);
|
||||
assert.equal(operationFeedback({ ...measured, total_bytes: 0, completed_bytes: 0 }).value, 50, "empty members use actual file counts");
|
||||
assert.equal(operationFeedback(null).value, null, "no measured result remains indeterminate");
|
||||
assert.equal(operationFeedback(null).label.values.total, 2, "selected totals remain available before polling responds");
|
||||
assert.equal(operationFeedback(null).label.values.bytes, "3 B");
|
||||
assert.equal(operationFeedback(null, "uploading", 24).value, 24);
|
||||
assert.equal(operationFeedback(null, "uploading", 100).value, null, "completed transfer is not completed extraction");
|
||||
assert.equal(page.split('event.dataTransfer.effectAllowed = "copyMove";').length - 1, 2);
|
||||
assert.doesNotMatch(page, /effectAllowed = "i18n:/, "browser drag/drop enums must not be translated");
|
||||
for (const name of ["previewManagedArchive", "confirmManagedArchive"]) {
|
||||
const body = api.slice(api.indexOf(`export function ${name}`)).split("\n}\n", 1)[0];
|
||||
assert.match(body, /encodeURIComponent\(fileId\)/);
|
||||
assert.match(body, /body: JSON\.stringify\(/);
|
||||
assert.doesNotMatch(body, /FormData|downloadFile|Blob|createObjectURL/);
|
||||
}
|
||||
assert.match(page, /function resetArchiveUploadState\(\) \{\s*void releaseArchivePreview\(settings, archiveFile\)/, "reset/cancel releases any temporary archive before forgetting it");
|
||||
assert.match(page, /archiveFile && archiveFile !== file\) void releaseArchivePreview/, "replacing a selected archive releases its previous temporary upload");
|
||||
for (const key of ["unpack", "preview", "select_one", "source", "expires", "extracting", "change_destination"]) {
|
||||
assert.equal(translations.split(`"i18n:govoplan-files.managed_archive.${key}":`).length - 1, 2, `${key} must have EN and DE text`);
|
||||
}
|
||||
console.log("Managed archive actions reuse the preview dialog and send source-bound JSON without browser re-upload.");
|
||||
+157
-36
@@ -389,6 +389,14 @@ export type ManagedFileSnapshotResponse = {files: ManagedFile[];folders: FileFol
|
||||
export type FileSpacesResponse = {spaces: FileSpace[];};
|
||||
export type FileUploadResponse = {files: ManagedFile[];};
|
||||
export type FileUploadProgress = {loaded: number;total?: number;percentage: number | null;};
|
||||
export type ArchiveOperationProgress = {
|
||||
phase: "inspecting" | "extracting" | "storing" | "finalizing" | "complete" | "failed";
|
||||
completed_files: number;
|
||||
total_files: number;
|
||||
completed_bytes: number;
|
||||
total_bytes: number;
|
||||
status: "running" | "complete" | "failed";
|
||||
};
|
||||
export type ArchivePreviewEntry = {
|
||||
path: string;
|
||||
kind: "file" | "directory";
|
||||
@@ -397,6 +405,7 @@ export type ArchivePreviewEntry = {
|
||||
encrypted: boolean;
|
||||
};
|
||||
export type ArchivePreviewResponse = {
|
||||
staged_upload_id?: string | null;
|
||||
preview_token: string;
|
||||
archive_format: string;
|
||||
entries: ArchivePreviewEntry[];
|
||||
@@ -552,18 +561,20 @@ export type PatternResolveResponse = {
|
||||
};
|
||||
|
||||
|
||||
export function listFileSpaces(settings: ApiSettings): Promise<FileSpacesResponse> {
|
||||
return apiFetch<FileSpacesResponse>(settings, "/api/v1/files/spaces");
|
||||
type FileReadOptions = Pick<RequestInit, "cache" | "signal">;
|
||||
|
||||
export function listFileSpaces(settings: ApiSettings, options?: FileReadOptions): Promise<FileSpacesResponse> {
|
||||
return apiFetch<FileSpacesResponse>(settings, "/api/v1/files/spaces", options);
|
||||
}
|
||||
|
||||
|
||||
export function listFolders(settings: ApiSettings, params: {owner_type: "user" | "group";owner_id: string;page_size?: number;cursor?: string | null;}): Promise<FileFoldersResponse> {
|
||||
export function listFolders(settings: ApiSettings, params: {owner_type: "user" | "group";owner_id: string;page_size?: number;cursor?: string | null;}, options?: FileReadOptions): Promise<FileFoldersResponse> {
|
||||
const search = new URLSearchParams();
|
||||
search.set("owner_type", params.owner_type);
|
||||
search.set("owner_id", params.owner_id);
|
||||
if (params.page_size) search.set("page_size", String(params.page_size));
|
||||
if (params.cursor) search.set("cursor", params.cursor);
|
||||
return apiFetch<FileFoldersResponse>(settings, `/api/v1/files/folders?${search.toString()}`);
|
||||
return apiFetch<FileFoldersResponse>(settings, `/api/v1/files/folders?${search.toString()}`, options);
|
||||
}
|
||||
|
||||
export function createFolder(
|
||||
@@ -580,13 +591,13 @@ payload: {owner_type: "user" | "group";owner_id: string;path: string;recursive?:
|
||||
return apiFetch<FolderDeleteResponse>(settings, "/api/v1/files/folders/delete", { method: "POST", body: JSON.stringify({ recursive: true, ...payload }) });
|
||||
}
|
||||
|
||||
export function listFiles(settings: ApiSettings, params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;campaign_usage?: FileCampaignUsageFilter;audit_relevant?: boolean;sort?: "path" | "recent";page_size?: number;cursor?: string | null;} = {}): Promise<FileListResponse> {
|
||||
export function listFiles(settings: ApiSettings, params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;campaign_usage?: FileCampaignUsageFilter;audit_relevant?: boolean;sort?: "path" | "recent";page_size?: number;cursor?: string | null;} = {}, options?: FileReadOptions): Promise<FileListResponse> {
|
||||
const search = new URLSearchParams();
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
if (value !== undefined && value !== null && value !== "") search.set(key, String(value));
|
||||
}
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileListResponse>(settings, `/api/v1/files${suffix}`);
|
||||
return apiFetch<FileListResponse>(settings, `/api/v1/files${suffix}`, options);
|
||||
}
|
||||
|
||||
export async function listFilesByProperties(
|
||||
@@ -598,7 +609,7 @@ params: {
|
||||
campaign_usage?: FileCampaignUsageFilter;
|
||||
audit_relevant?: boolean;
|
||||
page_size?: number;
|
||||
})
|
||||
}, options?: FileReadOptions)
|
||||
: Promise<{files: ManagedFile[];total: number;}> {
|
||||
const pageSize = params.page_size ?? DEFAULT_MANAGED_FILE_WINDOW_SIZE;
|
||||
let cursor: string | null | undefined = null;
|
||||
@@ -609,7 +620,7 @@ params: {
|
||||
...params,
|
||||
page_size: pageSize,
|
||||
cursor
|
||||
});
|
||||
}, options);
|
||||
files = files.concat(response.files);
|
||||
total = response.total;
|
||||
cursor = response.next_cursor;
|
||||
@@ -619,14 +630,14 @@ params: {
|
||||
|
||||
export function listFilesDelta(
|
||||
settings: ApiSettings,
|
||||
params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;since?: string;limit?: number;} = {})
|
||||
params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;since?: string;limit?: number;} = {}, options?: FileReadOptions)
|
||||
: Promise<FileDeltaResponse> {
|
||||
const search = new URLSearchParams();
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
if (value !== undefined && value !== null && value !== "") search.set(key, String(value));
|
||||
}
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileDeltaResponse>(settings, `/api/v1/files/delta${suffix}`);
|
||||
return apiFetch<FileDeltaResponse>(settings, `/api/v1/files/delta${suffix}`, options);
|
||||
}
|
||||
|
||||
function applyDeltaToSnapshot(
|
||||
@@ -648,7 +659,7 @@ response: FileDeltaResponse)
|
||||
|
||||
export async function listManagedFileSnapshot(
|
||||
settings: ApiSettings,
|
||||
params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page_size?: number;})
|
||||
params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page_size?: number;}, options?: FileReadOptions)
|
||||
: Promise<ManagedFileSnapshotResponse> {
|
||||
const pageSize = params.page_size ?? DEFAULT_MANAGED_FILE_WINDOW_SIZE;
|
||||
let watermark: string | null | undefined = null;
|
||||
@@ -660,7 +671,7 @@ params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page
|
||||
owner_id: params.owner_id,
|
||||
page_size: pageSize,
|
||||
cursor: folderCursor
|
||||
});
|
||||
}, options);
|
||||
watermark = watermark || response.watermark;
|
||||
folders = folders.concat(response.folders);
|
||||
folderCursor = response.next_cursor;
|
||||
@@ -675,7 +686,7 @@ params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page
|
||||
path_prefix: params.path_prefix,
|
||||
page_size: pageSize,
|
||||
cursor: fileCursor
|
||||
});
|
||||
}, options);
|
||||
watermark = watermark || response.watermark;
|
||||
files = files.concat(response.files);
|
||||
fileCursor = response.next_cursor;
|
||||
@@ -691,7 +702,7 @@ params: {owner_type: "user" | "group";owner_id: string;path_prefix?: string;page
|
||||
path_prefix: params.path_prefix,
|
||||
since,
|
||||
limit: pageSize
|
||||
});
|
||||
}, options);
|
||||
const snapshot = applyDeltaToSnapshot(files, folders, response);
|
||||
files = snapshot.files;
|
||||
folders = snapshot.folders;
|
||||
@@ -736,7 +747,17 @@ options: {
|
||||
return apiFetch<FileUploadResponse>(settings, "/api/v1/files/upload", { method: "POST", body: form });
|
||||
}
|
||||
|
||||
export function previewArchiveUpload(
|
||||
const stagedArchivePreviews = new WeakMap<File, ArchivePreviewResponse>();
|
||||
|
||||
export async function releaseArchivePreview(settings: ApiSettings, file: File | null): Promise<void> {
|
||||
const preview = file ? stagedArchivePreviews.get(file) : undefined;
|
||||
if (file) stagedArchivePreviews.delete(file);
|
||||
if (preview?.staged_upload_id) {
|
||||
await apiFetch(settings, `/api/v1/files/archive-staging/${encodeURIComponent(preview.staged_upload_id)}`, { method: "DELETE" }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
export async function previewArchiveUpload(
|
||||
settings: ApiSettings,
|
||||
file: File,
|
||||
options: {
|
||||
@@ -745,16 +766,67 @@ options: {
|
||||
path?: string;
|
||||
campaign_id?: string;
|
||||
password?: string;
|
||||
onProgress?: (progress: FileUploadProgress) => void;
|
||||
})
|
||||
: Promise<ArchivePreviewResponse> {
|
||||
let staged = stagedArchivePreviews.get(file);
|
||||
if (staged && !(Date.parse(staged.expires_at) > Date.now())) {
|
||||
void releaseArchivePreview(settings, file);
|
||||
staged = undefined;
|
||||
}
|
||||
const form = new FormData();
|
||||
form.append("file", file);
|
||||
if (staged?.staged_upload_id) {
|
||||
form.append("staged_upload_id", staged.staged_upload_id);
|
||||
form.append("preview_token", staged.preview_token);
|
||||
} else {
|
||||
form.append("file", file);
|
||||
}
|
||||
form.append("retain_upload", "true");
|
||||
form.append("owner_type", options.owner_type);
|
||||
form.append("owner_id", options.owner_id);
|
||||
form.append("path", options.path ?? "");
|
||||
if (options.campaign_id) form.append("campaign_id", options.campaign_id);
|
||||
if (options.password) form.append("password", options.password);
|
||||
return apiFetch<ArchivePreviewResponse>(settings, "/api/v1/files/archive-preview", { method: "POST", body: form });
|
||||
try {
|
||||
const result = options.onProgress && !staged?.staged_upload_id
|
||||
? await uploadFilesWithProgress<ArchivePreviewResponse>(settings, form, options.onProgress, "/api/v1/files/archive-preview")
|
||||
: await apiFetch<ArchivePreviewResponse>(settings, "/api/v1/files/archive-preview", { method: "POST", body: form });
|
||||
stagedArchivePreviews.set(file, result);
|
||||
return result;
|
||||
} catch (error) {
|
||||
// Preview is read-only with respect to managed files: an expired temporary
|
||||
// stage can safely be uploaded once again. Never retry confirmation.
|
||||
if (staged?.staged_upload_id && error instanceof ApiError && error.status === 410) {
|
||||
stagedArchivePreviews.delete(file);
|
||||
return previewArchiveUpload(settings, file, options);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
type ManagedArchiveOptions = {
|
||||
source_version_id: string;
|
||||
owner_type: "user" | "group";
|
||||
owner_id: string;
|
||||
path?: string;
|
||||
password?: string;
|
||||
};
|
||||
|
||||
export function previewManagedArchive(settings: ApiSettings, fileId: string, options: ManagedArchiveOptions): Promise<ArchivePreviewResponse> {
|
||||
return apiFetch<ArchivePreviewResponse>(settings, `/api/v1/files/${encodeURIComponent(fileId)}/archive-preview`, {
|
||||
method: "POST", body: JSON.stringify(options)
|
||||
});
|
||||
}
|
||||
|
||||
export function confirmManagedArchive(settings: ApiSettings, fileId: string, options: ManagedArchiveOptions & {
|
||||
preview_token: string;
|
||||
selected_paths: string[];
|
||||
onArchiveProgress?: (progress: ArchiveOperationProgress) => void;
|
||||
}): Promise<FileUploadResponse> {
|
||||
const { onArchiveProgress, ...payload } = options;
|
||||
return runArchiveOperation(settings, (operation_id) => apiFetch<FileUploadResponse>(settings, `/api/v1/files/${encodeURIComponent(fileId)}/archive-confirm`, {
|
||||
method: "POST", body: JSON.stringify({ ...payload, operation_id })
|
||||
}), onArchiveProgress);
|
||||
}
|
||||
|
||||
export function confirmArchiveUpload(
|
||||
@@ -774,10 +846,16 @@ options: {
|
||||
source_revision?: string;
|
||||
connector_policy_sources?: FileConnectorPolicySource[];
|
||||
onProgress?: (progress: FileUploadProgress) => void;
|
||||
onArchiveProgress?: (progress: ArchiveOperationProgress) => void;
|
||||
})
|
||||
: Promise<FileUploadResponse> {
|
||||
const form = new FormData();
|
||||
form.append("file", file);
|
||||
const staged = stagedArchivePreviews.get(file);
|
||||
if (staged?.staged_upload_id && staged.preview_token === options.preview_token) {
|
||||
form.append("staged_upload_id", staged.staged_upload_id);
|
||||
} else {
|
||||
form.append("file", file);
|
||||
}
|
||||
form.append("preview_token", options.preview_token);
|
||||
form.append("selected_paths_json", JSON.stringify(options.selected_paths));
|
||||
form.append("owner_type", options.owner_type);
|
||||
@@ -790,23 +868,59 @@ options: {
|
||||
if (options.source_provenance) form.append("source_provenance_json", JSON.stringify(options.source_provenance));
|
||||
if (options.source_revision) form.append("source_revision", options.source_revision);
|
||||
if (options.connector_policy_sources?.length) form.append("connector_policy_json", JSON.stringify({ sources: options.connector_policy_sources }));
|
||||
if (options.onProgress) {
|
||||
return uploadFilesWithProgress(
|
||||
settings,
|
||||
form,
|
||||
options.onProgress,
|
||||
"/api/v1/files/archive-confirm"
|
||||
);
|
||||
}
|
||||
return apiFetch<FileUploadResponse>(settings, "/api/v1/files/archive-confirm", { method: "POST", body: form });
|
||||
return runArchiveOperation(settings, (operationId) => {
|
||||
if (operationId) form.append("operation_id", operationId);
|
||||
return options.onProgress && !form.has("staged_upload_id")
|
||||
? uploadFilesWithProgress(settings, form, options.onProgress, "/api/v1/files/archive-confirm")
|
||||
: apiFetch<FileUploadResponse>(settings, "/api/v1/files/archive-confirm", { method: "POST", body: form });
|
||||
}, options.onArchiveProgress).then((result) => {
|
||||
stagedArchivePreviews.delete(file);
|
||||
return result;
|
||||
});
|
||||
}
|
||||
|
||||
function uploadFilesWithProgress(
|
||||
async function runArchiveOperation(
|
||||
settings: ApiSettings,
|
||||
request: (operationId?: string) => Promise<FileUploadResponse>,
|
||||
onProgress?: (progress: ArchiveOperationProgress) => void
|
||||
): Promise<FileUploadResponse> {
|
||||
if (!onProgress) return request();
|
||||
const operationId = crypto.randomUUID();
|
||||
let stopped = false;
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const abort = new AbortController();
|
||||
const poll = async () => {
|
||||
try {
|
||||
const progress = await apiFetch<ArchiveOperationProgress>(settings, `/api/v1/files/archive-progress/${operationId}`, { cache: "no-store", signal: abort.signal });
|
||||
if (!stopped) onProgress(progress);
|
||||
} catch {
|
||||
// Missing/expired or temporarily unreachable telemetry never retries or
|
||||
// fails an import. Its authoritative result remains the POST response.
|
||||
} finally {
|
||||
if (!stopped) timer = setTimeout(() => void poll(), 500);
|
||||
}
|
||||
};
|
||||
timer = setTimeout(() => void poll(), 150);
|
||||
try {
|
||||
const result = await request(operationId);
|
||||
stopped = true;
|
||||
const bytes = result.files.reduce((total, file) => total + file.size_bytes, 0);
|
||||
onProgress({ phase: "complete", status: "complete", completed_files: result.files.length,
|
||||
total_files: result.files.length, completed_bytes: bytes, total_bytes: bytes });
|
||||
return result;
|
||||
} finally {
|
||||
stopped = true;
|
||||
clearTimeout(timer);
|
||||
abort.abort();
|
||||
}
|
||||
}
|
||||
|
||||
function uploadFilesWithProgress<T = FileUploadResponse>(
|
||||
settings: ApiSettings,
|
||||
form: FormData,
|
||||
onProgress: (progress: FileUploadProgress) => void,
|
||||
endpoint = "/api/v1/files/upload")
|
||||
: Promise<FileUploadResponse> {
|
||||
: Promise<T> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const xhr = new XMLHttpRequest();
|
||||
xhr.open("POST", apiUrl(settings, endpoint));
|
||||
@@ -815,13 +929,15 @@ endpoint = "/api/v1/files/upload")
|
||||
const csrf = csrfToken();
|
||||
if (csrf) xhr.setRequestHeader("X-CSRF-Token", csrf);
|
||||
|
||||
let lastProgress: FileUploadProgress = { loaded: 0, total: undefined, percentage: 0 };
|
||||
xhr.upload.onprogress = (event) => {
|
||||
const total = event.lengthComputable ? event.total : undefined;
|
||||
onProgress({
|
||||
lastProgress = {
|
||||
loaded: event.loaded,
|
||||
total,
|
||||
percentage: total && total > 0 ? Math.round(event.loaded / total * 100) : null
|
||||
});
|
||||
};
|
||||
onProgress(lastProgress);
|
||||
};
|
||||
|
||||
xhr.onerror = () => reject(new Error("i18n:govoplan-files.upload_failed_because_the_network_request_could_.360a5ab3"));
|
||||
@@ -832,13 +948,18 @@ endpoint = "/api/v1/files/upload")
|
||||
reject(new ApiError(xhr.status, xhr.statusText, responseText));
|
||||
return;
|
||||
}
|
||||
onProgress({ loaded: 1, total: 1, percentage: 100 });
|
||||
onProgress({ ...lastProgress, percentage: 100 });
|
||||
if (xhr.status === 204 || !responseText) {
|
||||
resolve({ files: [] });
|
||||
resolve({ files: [] } as T);
|
||||
return;
|
||||
}
|
||||
const contentType = xhr.getResponseHeader("content-type") || "";
|
||||
resolve(contentType.includes("application/json") ? JSON.parse(responseText) as FileUploadResponse : { files: [] });
|
||||
try {
|
||||
if (!contentType.includes("application/json")) throw new Error("Expected JSON response");
|
||||
resolve(JSON.parse(responseText) as T);
|
||||
} catch {
|
||||
reject(new ApiError(xhr.status, "Invalid JSON response", ""));
|
||||
}
|
||||
};
|
||||
|
||||
onProgress({ loaded: 0, total: undefined, percentage: 0 });
|
||||
@@ -1185,7 +1306,7 @@ export function deactivateFileConnectorProfile(settings: ApiSettings, profileId:
|
||||
export function browseFileConnectorProfile(
|
||||
settings: ApiSettings,
|
||||
profileId: string,
|
||||
params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {})
|
||||
params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {}, options?: FileReadOptions)
|
||||
: Promise<FileConnectorBrowseResponse> {
|
||||
const search = new URLSearchParams();
|
||||
if (params.path) search.set("path", params.path);
|
||||
@@ -1193,7 +1314,7 @@ params: {path?: string;library_id?: string;continuation_token?: string;campaign_
|
||||
if (params.continuation_token) search.set("continuation_token", params.continuation_token);
|
||||
if (params.campaign_id) search.set("campaign_id", params.campaign_id);
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`);
|
||||
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`, options);
|
||||
}
|
||||
|
||||
export function importFileConnectorFile(
|
||||
|
||||
@@ -1,17 +1,23 @@
|
||||
import { useEffect, useMemo, useRef, useState, type DragEvent as ReactDragEvent, type KeyboardEvent as ReactKeyboardEvent, type MouseEvent as ReactMouseEvent } from "react";
|
||||
import { ArrowUp, ChevronRight, Copy, Download, File, Folder, Home, KeyRound, Link2, ListFilter, MoveRight, Plus, RefreshCw, Search, Share2, Trash2, UploadCloud } from "lucide-react";
|
||||
import { Archive, ArrowUp, ChevronRight, Copy, Download, File, Folder, Home, KeyRound, Link2, ListFilter, MoveRight, Plus, RefreshCw, Search, Settings2, Share2, Trash2, UploadCloud } from "lucide-react";
|
||||
import { FormGrid, ActionToolbar,
|
||||
Button,
|
||||
ConfirmDialog,
|
||||
ContentGrid,
|
||||
Dialog,
|
||||
DocumentationHelpLink,
|
||||
DismissibleAlert,
|
||||
FieldLabel,
|
||||
FileDropZone,
|
||||
FormField,
|
||||
FormSection,
|
||||
LoadingFrame,
|
||||
LoadingIndicator,
|
||||
PasswordField,
|
||||
ResourceAccessExplanation,
|
||||
ToggleSwitch,
|
||||
WorkspaceActionBar,
|
||||
WorkspaceFrame,
|
||||
hasScope,
|
||||
usePlatformLanguage,
|
||||
type ApiSettings,
|
||||
@@ -25,6 +31,7 @@ import {
|
||||
createFileConnectorSpace,
|
||||
createFolder,
|
||||
confirmArchiveUpload,
|
||||
confirmManagedArchive,
|
||||
deleteFolder,
|
||||
deleteFileConnectorSpace,
|
||||
downloadFile,
|
||||
@@ -37,6 +44,8 @@ import {
|
||||
listFileSpaces,
|
||||
listManagedFileSnapshot,
|
||||
previewArchiveUpload,
|
||||
previewManagedArchive,
|
||||
releaseArchivePreview,
|
||||
resolveFilePatterns,
|
||||
syncFileConnectorSpaceFolder,
|
||||
syncFileConnectorFile,
|
||||
@@ -45,6 +54,7 @@ import {
|
||||
virtualFolderResourceId,
|
||||
type ArchivePreviewEntry,
|
||||
type ArchivePreviewResponse,
|
||||
type ArchiveOperationProgress,
|
||||
type ConflictResolution,
|
||||
type ConflictStrategy,
|
||||
type FileConnectorBrowseItem,
|
||||
@@ -103,7 +113,7 @@ import { useFileTreeState } from "./hooks/useFileTreeState";
|
||||
import { useFileDialogs } from "./hooks/useFileDialogs";
|
||||
import { useFileDragDropState } from "./hooks/useFileDragDropState";
|
||||
|
||||
type UploadPhase = "idle" | "uploading" | "unpacking" | "finalizing";
|
||||
type UploadPhase = "idle" | "uploading" | "inspecting" | "unpacking" | "finalizing";
|
||||
type AuditRelevantFilter = "" | "true" | "false";
|
||||
type FileAccessExplanationTarget = {
|
||||
resourceType: "file" | "folder";
|
||||
@@ -155,13 +165,26 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
const [unmatchedCount, setUnmatchedCount] = useState<number | null>(null);
|
||||
const [unpackZip, setUnpackZip] = useState(false);
|
||||
const [archiveFile, setArchiveFile] = useState<File | null>(null);
|
||||
const [managedArchiveFile, setManagedArchiveFile] = useState<ManagedFile | null>(null);
|
||||
const [archivePreview, setArchivePreview] = useState<ArchivePreviewResponse | null>(null);
|
||||
const [archivePassword, setArchivePassword] = useState("");
|
||||
const [selectedArchivePaths, setSelectedArchivePaths] = useState<Set<string>>(new Set());
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [operationBusy, setBusy] = useState(false);
|
||||
const [reloadingView, setReloadingView] = useState(false);
|
||||
const busy = operationBusy || reloadingView;
|
||||
const [viewReloadFailed, setViewReloadFailed] = useState(false);
|
||||
const reloadSequenceRef = useRef(0);
|
||||
const reloadContextRef = useRef("");
|
||||
reloadContextRef.current = JSON.stringify([
|
||||
settings.apiBaseUrl, settings.apiKey, settings.accessToken, auth.tenant?.id, auth.user.id,
|
||||
activeSpaceId, currentFolder, searchActive, searchPattern, searchCaseSensitive,
|
||||
propertyFiltersActive, campaignUsageFilter, auditRelevantFilter
|
||||
]);
|
||||
const [toolsPanel, setToolsPanel] = useState<"connections" | "selection" | null>(null);
|
||||
const [uploadActive, setUploadActive] = useState(false);
|
||||
const [uploadPhase, setUploadPhase] = useState<UploadPhase>("idle");
|
||||
const [uploadProgress, setUploadProgress] = useState<number | null>(null);
|
||||
const [archiveProgress, setArchiveProgress] = useState<ArchiveOperationProgress | null>(null);
|
||||
const [connectorProfiles, setConnectorProfiles] = useState<FileConnectorProfile[]>([]);
|
||||
const [connectorProfileId, setConnectorProfileId] = useState("");
|
||||
const [connectorLibraryId, setConnectorLibraryId] = useState<string | null>(null);
|
||||
@@ -282,6 +305,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
onOpenFolder: openFolder
|
||||
});
|
||||
const selectedFiles = useMemo(() => files.filter((file) => selectedFileIds.has(file.id)), [files, selectedFileIds]);
|
||||
const selectedArchive = selectedFiles.length === 1 && selectedFolderPaths.size === 0 && ARCHIVE_FILENAME_PATTERN.test(selectedFiles[0].filename) ? selectedFiles[0] : null;
|
||||
const shareManageableFile = useMemo(() => {
|
||||
if (!canShare || selectedFiles.length !== 1 || selectedFolderPaths.size > 0) return null;
|
||||
const file = selectedFiles[0];
|
||||
@@ -366,6 +390,81 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
}
|
||||
|
||||
async function reloadCurrentView() {
|
||||
if (busy || reloadingView || connectorSpaceLoading) return;
|
||||
const sequence = ++reloadSequenceRef.current;
|
||||
const context = reloadContextRef.current;
|
||||
const isCurrent = () => sequence === reloadSequenceRef.current && context === reloadContextRef.current;
|
||||
const readOptions = { cache: "no-store" } as const;
|
||||
setReloadingView(true);
|
||||
setViewReloadFailed(false);
|
||||
setError("");
|
||||
if (activeSpaceIsConnector) setConnectorSpaceError("");
|
||||
try {
|
||||
if (!activeSpace) {
|
||||
const response = await listFileSpaces(settings, readOptions);
|
||||
if (!isCurrent()) return;
|
||||
setSpaces(response.spaces);
|
||||
setSpacesLoaded(true);
|
||||
setActiveSpaceId(response.spaces[0]?.id || "");
|
||||
return;
|
||||
}
|
||||
if (activeSpaceIsConnector) {
|
||||
if (!activeSpace.connector_profile_id) throw new Error(translateText("i18n:govoplan-files.connector_profile_is_not_configured_for_this_spa.669f57b5"));
|
||||
const libraryId = connectorSpaceLibrary(activeSpace);
|
||||
const response = await browseFileConnectorProfile(settings, activeSpace.connector_profile_id, {
|
||||
path: connectorSpaceBrowsePath(activeSpace, currentFolder), library_id: libraryId || undefined
|
||||
}, readOptions);
|
||||
if (!isCurrent()) return;
|
||||
setConnectorSpaceItemsBySpace((current) => ({ ...current, [activeSpace.id]: response.items }));
|
||||
setConnectorSpaceLibraryBySpace((current) => ({ ...current, [activeSpace.id]: response.library_id ?? libraryId ?? null }));
|
||||
setConnectorSpaceSelectedItem((current) => current && response.items.find((item) => item.path === current.path && item.kind === current.kind) || null);
|
||||
return;
|
||||
}
|
||||
// Re-read the current projection, including active filters. None of these
|
||||
// calls imports, synchronizes, uploads, or changes a managed resource.
|
||||
// Apply only after all reads succeed so a failed reload keeps usable data.
|
||||
const owner = { owner_type: activeSpace.owner_type, owner_id: activeSpace.owner_id };
|
||||
const [snapshot, matches, properties] = await Promise.all([
|
||||
listManagedFileSnapshot(settings, owner, readOptions),
|
||||
searchActive ? resolveFilePatterns(settings, {
|
||||
...owner, patterns: [searchPattern], path_prefix: currentFolder,
|
||||
include_unmatched: true, case_sensitive: searchCaseSensitive
|
||||
}) : Promise.resolve(null),
|
||||
propertyFiltersActive ? listFilesByProperties(settings, {
|
||||
...owner, path_prefix: currentFolder, campaign_usage: campaignUsageFilter || undefined,
|
||||
audit_relevant: auditRelevantFilter ? auditRelevantFilter === "true" : undefined
|
||||
}, readOptions) : Promise.resolve(null)
|
||||
]);
|
||||
if (!isCurrent()) return;
|
||||
setFilesBySpace((current) => ({ ...current, [activeSpace.id]: snapshot.files }));
|
||||
setFoldersBySpace((current) => ({ ...current, [activeSpace.id]: snapshot.folders }));
|
||||
setFileDeltaWatermarksBySpace((current) => ({ ...current, [activeSpace.id]: snapshot.watermark || "" }));
|
||||
const fileIds = new Set(snapshot.files.map((file) => file.id));
|
||||
const folderPaths = new Set(snapshot.folders.map((folder) => folder.path));
|
||||
setSelectedFileIds((current) => new Set(Array.from(current).filter((id) => fileIds.has(id))));
|
||||
setSelectedFolderPaths((current) => new Set(Array.from(current).filter((path) =>
|
||||
folderPaths.has(path) || snapshot.files.some((file) => file.display_path.startsWith(`${path}/`))
|
||||
)));
|
||||
if (matches) {
|
||||
setSearchResults(matches.patterns.flatMap((pattern) => pattern.matches));
|
||||
setUnmatchedCount(matches.unmatched.length);
|
||||
}
|
||||
if (properties) {
|
||||
setPropertyFilterResults(properties.files);
|
||||
setPropertyFilterTotal(properties.total);
|
||||
}
|
||||
} catch (err) {
|
||||
if (!isCurrent()) return;
|
||||
setViewReloadFailed(true);
|
||||
const detail = err instanceof Error ? err.message : String(err);
|
||||
if (activeSpaceIsConnector) setConnectorSpaceError(detail);
|
||||
else setError(detail);
|
||||
} finally {
|
||||
if (sequence === reloadSequenceRef.current) setReloadingView(false);
|
||||
}
|
||||
}
|
||||
|
||||
function applyManagedSpaceDelta(spaceId: string, response: FileDeltaResponse) {
|
||||
if (response.full) {
|
||||
setFilesBySpace((current) => ({ ...current, [spaceId]: response.files }));
|
||||
@@ -450,6 +549,14 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
// A late manual reload must not publish a previous tenant/account/folder's
|
||||
// projection, or release a newer operation's busy state.
|
||||
++reloadSequenceRef.current;
|
||||
setReloadingView(false);
|
||||
setViewReloadFailed(false);
|
||||
}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken, auth.tenant?.id, auth.user.id, activeSpaceId, currentFolder]);
|
||||
|
||||
useEffect(() => {
|
||||
void loadSpaces();
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
@@ -578,7 +685,10 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
|
||||
function resetArchiveUploadState() {
|
||||
void releaseArchivePreview(settings, archiveFile);
|
||||
setArchiveProgress(null);
|
||||
setArchiveFile(null);
|
||||
setManagedArchiveFile(null);
|
||||
setArchivePreview(null);
|
||||
setArchivePassword("");
|
||||
setSelectedArchivePaths(new Set());
|
||||
@@ -590,6 +700,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
setUploadActive(false);
|
||||
setUploadPhase("idle");
|
||||
setUploadProgress(null);
|
||||
setArchiveProgress(null);
|
||||
setConnectorError("");
|
||||
setConnectorSelectedItem(null);
|
||||
setConnectorSpaceLabel("");
|
||||
@@ -600,6 +711,16 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
|
||||
function updateActiveDialogFolder(spaceId: string, folderPath: string) {
|
||||
setDialogTarget({ spaceId, folderPath: normalizeFolder(folderPath) });
|
||||
setArchivePreview(null);
|
||||
setSelectedArchivePaths(new Set());
|
||||
}
|
||||
|
||||
function openManagedArchive(file: ManagedFile, target: FileActionTarget | null) {
|
||||
if (busy || !canUpload || !canDownload || !target || isConnectorSpace(findSpace(target.spaceId)) || !ARCHIVE_FILENAME_PATTERN.test(file.filename)) return;
|
||||
setContextMenu(null);
|
||||
openDialog("upload", target);
|
||||
setManagedArchiveFile(file);
|
||||
setUnpackZip(true);
|
||||
}
|
||||
|
||||
function findSpace(spaceId: string): FileSpace | null {
|
||||
@@ -801,29 +922,43 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
|
||||
|
||||
async function loadArchivePreview(
|
||||
file: File,
|
||||
file: File | ManagedFile,
|
||||
target: FileActionTarget,
|
||||
options: { preserveSelection?: boolean } = {}
|
||||
) {
|
||||
if (busy || uploadActive) return;
|
||||
const targetSpace = findSpace(target.spaceId);
|
||||
if (!targetSpace || isConnectorSpace(targetSpace)) {
|
||||
setError(uploadRejectedReason(target));
|
||||
return;
|
||||
}
|
||||
setArchiveFile(file);
|
||||
const managed = "version_id" in file;
|
||||
if (archiveFile && archiveFile !== file) void releaseArchivePreview(settings, archiveFile);
|
||||
setArchiveFile(managed ? null : file);
|
||||
setManagedArchiveFile(managed ? file : null);
|
||||
setBusy(true);
|
||||
setUploadActive(true);
|
||||
setUploadPhase("uploading");
|
||||
setUploadPhase(managed ? "inspecting" : "uploading");
|
||||
setUploadProgress(null);
|
||||
setArchiveProgress(null);
|
||||
setError("");
|
||||
setMessage("");
|
||||
try {
|
||||
const response = await previewArchiveUpload(settings, file, {
|
||||
const previewOptions = {
|
||||
owner_type: targetSpace.owner_type,
|
||||
owner_id: targetSpace.owner_id,
|
||||
path: target.folderPath,
|
||||
password: archivePassword || undefined
|
||||
});
|
||||
};
|
||||
const response = managed
|
||||
? await previewManagedArchive(settings, file.id, { ...previewOptions, source_version_id: file.version_id })
|
||||
: await previewArchiveUpload(settings, file, {
|
||||
...previewOptions,
|
||||
onProgress: ({ percentage }) => {
|
||||
setUploadProgress(percentage);
|
||||
if (percentage !== null && percentage >= 100) setUploadPhase("inspecting");
|
||||
}
|
||||
});
|
||||
const availableFiles = new Set(
|
||||
response.entries
|
||||
.filter((entry) => entry.kind === "file")
|
||||
@@ -889,7 +1024,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
const targetSpace = target ? findSpace(target.spaceId) : null;
|
||||
if (
|
||||
busy
|
||||
|| !archiveFile
|
||||
|| (!archiveFile && !managedArchiveFile)
|
||||
|| !archivePreview
|
||||
|| !target
|
||||
|| !targetSpace
|
||||
@@ -907,25 +1042,35 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
setBusy(true);
|
||||
setUploadActive(true);
|
||||
setUploadPhase("uploading");
|
||||
setUploadProgress(0);
|
||||
setUploadPhase("inspecting");
|
||||
setUploadProgress(null);
|
||||
setArchiveProgress(null);
|
||||
setError("");
|
||||
setMessage("Uploading the archive for confirmed extraction.");
|
||||
setMessage("i18n:govoplan-files.archive_progress.inspecting");
|
||||
try {
|
||||
const response = await confirmArchiveUpload(settings, archiveFile, {
|
||||
const confirmOptions = {
|
||||
preview_token: archivePreview.preview_token,
|
||||
selected_paths: Array.from(selectedArchivePaths),
|
||||
owner_type: targetSpace.owner_type,
|
||||
owner_id: targetSpace.owner_id,
|
||||
path: target.folderPath,
|
||||
password: archivePassword || undefined,
|
||||
onArchiveProgress: (progress: ArchiveOperationProgress) => {
|
||||
setArchiveProgress(progress);
|
||||
setUploadPhase(progress.phase === "inspecting" ? "inspecting" : progress.phase === "finalizing" || progress.phase === "complete" ? "finalizing" : "unpacking");
|
||||
}
|
||||
};
|
||||
const response = managedArchiveFile
|
||||
? await confirmManagedArchive(settings, managedArchiveFile.id, { ...confirmOptions, source_version_id: managedArchiveFile.version_id })
|
||||
: await confirmArchiveUpload(settings, archiveFile!, {
|
||||
...confirmOptions,
|
||||
conflict_strategy: "reject",
|
||||
onProgress: ({ percentage }) => {
|
||||
setUploadProgress(percentage);
|
||||
if (percentage !== null && percentage >= 100) {
|
||||
setUploadPhase("unpacking");
|
||||
setMessage("Extracting the selected archive files.");
|
||||
}
|
||||
} else setUploadPhase("uploading");
|
||||
}
|
||||
});
|
||||
setUploadPhase("finalizing");
|
||||
@@ -986,6 +1131,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
setUploadActive(true);
|
||||
setUploadPhase("uploading");
|
||||
setUploadProgress(0);
|
||||
setArchiveProgress(null);
|
||||
setError("");
|
||||
setMessage(i18nMessage("i18n:govoplan-files.uploading_value_file_s.715ba963", { value0: selected.length }));
|
||||
try {
|
||||
@@ -1708,7 +1854,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
const state = { sourceSpaceId: activeSpace.id, fileIds: Array.from(sets.fileIds), folderPaths: Array.from(sets.folderPaths) };
|
||||
setInternalDrag(state);
|
||||
event.dataTransfer.effectAllowed = "i18n:govoplan-files.copymove.d0fa5904";
|
||||
event.dataTransfer.effectAllowed = "copyMove";
|
||||
event.dataTransfer.setData(INTERNAL_DRAG_TYPE, JSON.stringify(state));
|
||||
event.dataTransfer.setData("text/plain", `${state.fileIds.length + state.folderPaths.length} item(s)`);
|
||||
}
|
||||
@@ -1719,7 +1865,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
if (!folderPath) return;
|
||||
const state = { sourceSpaceId: spaceId, fileIds: [], folderPaths: [folderPath] };
|
||||
setInternalDrag(state);
|
||||
event.dataTransfer.effectAllowed = "i18n:govoplan-files.copymove.d0fa5904";
|
||||
event.dataTransfer.effectAllowed = "copyMove";
|
||||
event.dataTransfer.setData(INTERNAL_DRAG_TYPE, JSON.stringify(state));
|
||||
event.dataTransfer.setData("text/plain", folderPath);
|
||||
}
|
||||
@@ -2240,7 +2386,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
return `${label} ${sortDirection === "asc" ? "↑" : "↓"}`;
|
||||
}
|
||||
|
||||
const noticeTone = message.startsWith("i18n:govoplan-files.no_files_uploaded_all_conflicts_were_skipped") ? "warning" :
|
||||
const noticeTone = uploadActive ? "info" : message.startsWith("i18n:govoplan-files.no_files_uploaded_all_conflicts_were_skipped") ? "warning" :
|
||||
message.startsWith("i18n:govoplan-files.uploading_value_file_s") ||
|
||||
message === "i18n:govoplan-files.unpacking_zip_upload.35019691" ||
|
||||
message === "i18n:govoplan-files.finalizing_upload.bcce936d" ? "info" : "success";
|
||||
@@ -2249,6 +2395,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
activeSpaceIsConnector ? "This action writes managed storage and is unavailable in a read-only connector space." : "";
|
||||
const workingBlocker = busy ? "Wait for the current file operation to finish." : "";
|
||||
const uploadBlocker = workingBlocker || managedSpaceBlocker || (!canUpload ? "File upload permission is required." : "");
|
||||
const unpackBlocker = uploadBlocker || (!canDownload ? "i18n:govoplan-files.managed_archive.download_required" : "") || (!selectedArchive ? "i18n:govoplan-files.managed_archive.select_one" : "");
|
||||
const organizeBlocker = workingBlocker || managedSpaceBlocker || (!canOrganize ? "File organization permission is required." : "");
|
||||
const selectionBlocker = !hasSelection ? "Select at least one file or folder first." : "";
|
||||
const downloadBlocker = workingBlocker || managedSpaceBlocker || (!canDownload ? "File download permission is required." : "") || (selectedDownloadFileIds.length === 0 ? "Select at least one downloadable file first." : "");
|
||||
@@ -2268,57 +2415,61 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
["i18n:govoplan-files.folder_sync.summary.failed", folderSyncResult.summary.failed]
|
||||
] as const : [];
|
||||
|
||||
const toolbar =
|
||||
<ActionToolbar className="file-manager-toolbar" aria-label="i18n:govoplan-files.file_actions.9e1b94c5">
|
||||
<Button variant="primary" onClick={() => openDialog("upload", toolbarTarget())} disabled={Boolean(uploadBlocker)} disabledReason={uploadBlocker}><UploadCloud size={16} aria-hidden="true" /> i18n:govoplan-files.upload.8bdf057f</Button>
|
||||
<Button
|
||||
onClick={() => activeSpaceIsConnector ? void syncConnectorSpaceSelection() : void openConnectorSyncDialog(toolbarTarget())}
|
||||
disabled={Boolean(syncBlocker)}
|
||||
disabledReason={syncBlocker}>
|
||||
|
||||
<RefreshCw size={16} aria-hidden="true" /> i18n:govoplan-files.sync.905f6309
|
||||
</Button>
|
||||
{activeSpaceIsConnector &&
|
||||
<Button onClick={openConnectorFolderSyncDialog} disabled={Boolean(folderSyncBlocker)} disabledReason={folderSyncBlocker}>
|
||||
<RefreshCw size={16} aria-hidden="true" /> i18n:govoplan-files.folder_sync.button
|
||||
</Button>
|
||||
}
|
||||
<Button onClick={() => void openConnectorSpaceDialog()} disabled={busy || !canOrganize} disabledReason={workingBlocker || (!canOrganize ? "File organization permission is required to add a connector space." : "")}>
|
||||
<Link2 size={16} aria-hidden="true" /> i18n:govoplan-files.add_space.e4d674d4
|
||||
</Button>
|
||||
<Button onClick={() => void downloadSelection()} disabled={Boolean(downloadBlocker)} disabledReason={downloadBlocker}><Download size={16} aria-hidden="true" /> {downloadLabel}</Button>
|
||||
<Button onClick={() => shareManageableFile && setShareDialogFile(shareManageableFile)} disabled={Boolean(shareBlocker)} disabledReason={shareBlocker}><Share2 size={16} aria-hidden="true" /> Manage shares</Button>
|
||||
function chooseTool(action: () => void) {
|
||||
setToolsPanel(null);
|
||||
action();
|
||||
}
|
||||
|
||||
const toolbar = <WorkspaceActionBar
|
||||
scope="workspace"
|
||||
variant="collection"
|
||||
refreshable
|
||||
label="i18n:govoplan-files.file_actions.9e1b94c5"
|
||||
interfaceId="files.workspace.actions"
|
||||
helpContextId="files.list"
|
||||
helpModuleId="files"
|
||||
reloadAction={{ onReload: () => void reloadCurrentView(), loading: reloadingView, state: viewReloadFailed ? "reload-failed" : "current", disabled: busy || connectorSpaceLoading, disabledReason: workingBlocker || (connectorSpaceLoading ? "This connector space is already refreshing." : undefined) }}
|
||||
contextActions={<Button onClick={() => setToolsPanel("connections")} disabled={busy} disabledReason={workingBlocker}><Settings2 size={16} aria-hidden="true" /> i18n:govoplan-files.tools.connections</Button>}
|
||||
helpAction={<DocumentationHelpLink reference={FILES_WORKFLOW_DOCUMENTATION} />}
|
||||
createAction={<>
|
||||
<Button onClick={() => openDialog("create-folder", toolbarTarget())} disabled={Boolean(organizeBlocker)} disabledReason={organizeBlocker}><Plus size={16} aria-hidden="true" /> i18n:govoplan-files.create_folder.97bafaba</Button>
|
||||
<Button onClick={() => openTransferDialog("move")} disabled={Boolean(organizeBlocker || selectionBlocker)} disabledReason={organizeBlocker || selectionBlocker}><MoveRight size={16} aria-hidden="true" /> i18n:govoplan-files.move.76cdb950</Button>
|
||||
<Button onClick={() => openTransferDialog("copy")} disabled={Boolean(organizeBlocker || selectionBlocker)} disabledReason={organizeBlocker || selectionBlocker}><Copy size={16} aria-hidden="true" /> i18n:govoplan-files.copy.af74f7c5</Button>
|
||||
{hasSelection && <Button onClick={openRenameDialog} disabled={Boolean(organizeBlocker)} disabledReason={organizeBlocker}>{selectedEntryCount === 1 ? "i18n:govoplan-files.rename.d3f4cb89" : "i18n:govoplan-files.bulk_rename.7dcaa624"}</Button>}
|
||||
<Button onClick={() => accessExplainableTarget && void openAccessExplanation(accessExplainableTarget)} disabled={Boolean(accessExplanationBlocker)} disabledReason={accessExplanationBlocker}><KeyRound size={16} aria-hidden="true" /> i18n:govoplan-files.explain_access.4d5fac37</Button>
|
||||
<Button variant="danger" helpContextId="files.list" helpModuleId="files" onClick={() => void deleteSelected()} disabled={Boolean(deleteBlocker)} disabledReason={deleteBlocker}><Trash2 size={16} aria-hidden="true" /> i18n:govoplan-files.delete.f6fdbe48</Button>
|
||||
{activeSpaceIsConnector &&
|
||||
<Button
|
||||
variant="danger"
|
||||
onClick={() => activeSpace && setConnectorSpaceRemovalTarget(activeSpace)}
|
||||
disabled={busy || !canOrganize || !activeSpace?.connector_space_id}
|
||||
disabledReason={workingBlocker || (!canOrganize ? "File organization permission is required to remove a connector space." : !activeSpace?.connector_space_id ? "This space is not a removable connector space." : "")}>
|
||||
<Trash2 size={16} aria-hidden="true" /> Remove space
|
||||
</Button>
|
||||
}
|
||||
{activeSpaceIsConnector &&
|
||||
<Button onClick={() => activeSpace && void loadConnectorSpaceContents(activeSpace)} disabled={busy || connectorSpaceLoading || !activeSpace} disabledReason={workingBlocker || (connectorSpaceLoading ? "This connector space is already refreshing." : !activeSpace ? "Select a connector space first." : "")}>
|
||||
<RefreshCw size={16} aria-hidden="true" /> i18n:govoplan-files.refresh.56e3badc
|
||||
</Button>
|
||||
}
|
||||
<DocumentationHelpLink reference={FILES_WORKFLOW_DOCUMENTATION} />
|
||||
</ActionToolbar>;
|
||||
<Button variant="primary" onClick={() => openDialog("upload", toolbarTarget())} disabled={Boolean(uploadBlocker)} disabledReason={uploadBlocker}><UploadCloud size={16} aria-hidden="true" /> i18n:govoplan-files.upload.8bdf057f</Button>
|
||||
</>}
|
||||
/>;
|
||||
|
||||
const selectionToolbar = <WorkspaceActionBar
|
||||
scope="detail-pane"
|
||||
variant="detail"
|
||||
label="i18n:govoplan-files.tools.selection"
|
||||
contextActions={<span>{activeSpaceIsConnector ? connectorSpaceSelectedItem?.name || translateText("i18n:govoplan-files.no_file_selected.f76f1c1c") : selectedSummary}</span>}
|
||||
primaryActions={<>
|
||||
<Button onClick={() => void downloadSelection()} disabled={Boolean(downloadBlocker)} disabledReason={downloadBlocker}><Download size={16} aria-hidden="true" /> {downloadLabel}</Button>
|
||||
{selectedArchive && <Button onClick={() => openManagedArchive(selectedArchive, toolbarTarget())} disabled={Boolean(unpackBlocker)} disabledReason={unpackBlocker}><Archive size={16} aria-hidden="true" /> i18n:govoplan-files.managed_archive.unpack</Button>}
|
||||
<Button onClick={() => setToolsPanel("selection")} disabled={Boolean(workingBlocker || managedSpaceBlocker || selectionBlocker)} disabledReason={workingBlocker || managedSpaceBlocker || selectionBlocker}><Settings2 size={16} aria-hidden="true" /> i18n:govoplan-files.tools.selection</Button>
|
||||
</>}
|
||||
/>;
|
||||
|
||||
|
||||
const uploadBusyLabel = uploadPhase === "unpacking" ? "i18n:govoplan-files.unpacking_zip_archive.698095f4" : "i18n:govoplan-files.uploading_files.6536791d";
|
||||
const uploadProgressLabel = uploadPhase === "unpacking" ?
|
||||
"i18n:govoplan-files.extracting_files_on_the_server.845a3c1a" :
|
||||
uploadProgress !== null && uploadProgress >= 100 ?
|
||||
"i18n:govoplan-files.finalizing_upload.bcce936d" :
|
||||
undefined;
|
||||
const visibleUploadProgress = uploadPhase === "unpacking" ? null : uploadProgress;
|
||||
const selectedArchiveBytes = archivePreview?.entries.reduce((total, entry) => total + (entry.kind === "file" && selectedArchivePaths.has(entry.path) ? entry.size_bytes : 0), 0) ?? 0;
|
||||
const operationBusyLabel = uploadPhase === "inspecting" ? "i18n:govoplan-files.archive_progress.inspecting"
|
||||
: uploadPhase === "finalizing" ? "i18n:govoplan-files.archive_progress.finalizing"
|
||||
: uploadPhase === "unpacking" ? (archiveProgress?.phase === "storing" ? "i18n:govoplan-files.archive_progress.storing" : "i18n:govoplan-files.archive_progress.extracting")
|
||||
: "i18n:govoplan-files.uploading_files.6536791d";
|
||||
const serverProgressRatio = archiveProgress && (archiveProgress.total_bytes > 0
|
||||
? archiveProgress.completed_bytes / archiveProgress.total_bytes
|
||||
: archiveProgress.total_files > 0 ? archiveProgress.completed_files / archiveProgress.total_files : null);
|
||||
// Complete bytes/files do not imply a committed transaction. Keep finalization indeterminate.
|
||||
const operationProgressValue = archiveProgress?.status === "complete" ? 100
|
||||
: archiveProgress && archiveProgress.phase !== "finalizing" && serverProgressRatio !== null && serverProgressRatio < 1 ? serverProgressRatio * 100
|
||||
: uploadPhase === "uploading" && uploadProgress !== null && uploadProgress < 100 ? uploadProgress
|
||||
: null;
|
||||
const operationProgressLabel = archiveProgress && archiveProgress.total_files > 0
|
||||
? i18nMessage("i18n:govoplan-files.archive_progress.processed", { completed: archiveProgress.completed_files, total: archiveProgress.total_files, bytes: formatBytes(archiveProgress.completed_bytes), totalBytes: formatBytes(archiveProgress.total_bytes) })
|
||||
: uploadPhase === "uploading" && uploadProgress !== null && uploadProgress < 100
|
||||
? i18nMessage("i18n:govoplan-files.archive_progress.transferred", { percentage: Math.floor(uploadProgress) })
|
||||
: archivePreview && selectedArchivePaths.size > 0
|
||||
? i18nMessage("i18n:govoplan-files.archive_progress.selected", { total: selectedArchivePaths.size, bytes: formatBytes(selectedArchiveBytes) })
|
||||
: "i18n:govoplan-files.archive_progress.waiting";
|
||||
const connectorLocationLabel = activeConnectorProfile ?
|
||||
[activeConnectorProfile.label, connectorLibraryId, connectorPath || (connectorLibraryId ? "i18n:govoplan-files.root.e96857c5" : "")].filter(Boolean).join(" / ") :
|
||||
"i18n:govoplan-files.connector.ba358306";
|
||||
@@ -2406,7 +2557,6 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
<ActionToolbar className="connector-browser-toolbar">
|
||||
<Button onClick={browseConnectorSpaceParent} disabled={connectorSpaceParentDisabled}><ArrowUp size={15} aria-hidden="true" /> i18n:govoplan-files.up.2038bdec</Button>
|
||||
<span className="connector-browser-path" title={connectorSpaceLocationLabel}>{connectorSpaceLocationLabel}</span>
|
||||
<Button onClick={() => void loadConnectorSpaceContents(activeSpace)} disabled={busy || connectorSpaceLoading}><RefreshCw size={15} aria-hidden="true" /> i18n:govoplan-files.refresh.56e3badc</Button>
|
||||
</ActionToolbar>
|
||||
{connectorSpaceError && <p className="field-error connector-browser-error">{connectorSpaceError}</p>}
|
||||
<div className="connector-browser-list" role="list" aria-label="i18n:govoplan-files.connector_space_files.dbb0ab24">
|
||||
@@ -2462,7 +2612,49 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
const currentFolderDropActive = currentFolderDropTarget ? dropTargetKey === dropTargetId(currentFolderDropTarget) : false;
|
||||
|
||||
return (
|
||||
<div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen files-page">
|
||||
<WorkspaceFrame as="main" height="viewport" surface="plain" className="file-manager-page files-page" label="i18n:govoplan-files.files.6ce6c512" interfaceId="files.workspace" helpContextId="files.list" helpModuleId="files">
|
||||
{toolbar}
|
||||
<Dialog
|
||||
open={toolsPanel !== null}
|
||||
title={toolsPanel === "selection" ? "i18n:govoplan-files.tools.selection" : "i18n:govoplan-files.tools.connections"}
|
||||
description={toolsPanel === "selection" ? selectedSummary : "i18n:govoplan-files.tools.connections_description"}
|
||||
size="wide"
|
||||
onClose={() => setToolsPanel(null)}
|
||||
closeDisabled={busy}
|
||||
footer={<Button onClick={() => setToolsPanel(null)} disabled={busy}>i18n:govoplan-files.close.bbfa773e</Button>}
|
||||
>
|
||||
{toolsPanel === "selection" ? <ContentGrid columns={1}>
|
||||
<FormSection title="i18n:govoplan-files.tools.organize" description="i18n:govoplan-files.tools.organize_description">
|
||||
<ActionToolbar>
|
||||
<Button onClick={() => chooseTool(() => openTransferDialog("move"))} disabled={Boolean(organizeBlocker || selectionBlocker)} disabledReason={organizeBlocker || selectionBlocker}><MoveRight size={16} aria-hidden="true" /> i18n:govoplan-files.move.76cdb950</Button>
|
||||
<Button onClick={() => chooseTool(() => openTransferDialog("copy"))} disabled={Boolean(organizeBlocker || selectionBlocker)} disabledReason={organizeBlocker || selectionBlocker}><Copy size={16} aria-hidden="true" /> i18n:govoplan-files.copy.af74f7c5</Button>
|
||||
<Button onClick={() => chooseTool(openRenameDialog)} disabled={Boolean(organizeBlocker || selectionBlocker)} disabledReason={organizeBlocker || selectionBlocker}>{selectedEntryCount === 1 ? "i18n:govoplan-files.rename.d3f4cb89" : "i18n:govoplan-files.bulk_rename.7dcaa624"}</Button>
|
||||
</ActionToolbar>
|
||||
</FormSection>
|
||||
<FormSection title="i18n:govoplan-files.tools.sharing_access" variant="separated">
|
||||
<ActionToolbar>
|
||||
<Button onClick={() => chooseTool(() => { if (shareManageableFile) setShareDialogFile(shareManageableFile); })} disabled={Boolean(shareBlocker)} disabledReason={shareBlocker}><Share2 size={16} aria-hidden="true" /> Manage shares</Button>
|
||||
<Button onClick={() => chooseTool(() => { if (accessExplainableTarget) void openAccessExplanation(accessExplainableTarget); })} disabled={Boolean(accessExplanationBlocker)} disabledReason={accessExplanationBlocker}><KeyRound size={16} aria-hidden="true" /> i18n:govoplan-files.explain_access.4d5fac37</Button>
|
||||
</ActionToolbar>
|
||||
</FormSection>
|
||||
<FormSection title="i18n:govoplan-files.tools.destructive" description="i18n:govoplan-files.tools.delete_description" variant="separated">
|
||||
<WorkspaceActionBar scope="detail-pane" variant="detail" label="i18n:govoplan-files.tools.destructive" destructiveActions={<Button variant="danger" helpContextId="files.list" helpModuleId="files" onClick={() => chooseTool(() => void deleteSelected())} disabled={Boolean(deleteBlocker)} disabledReason={deleteBlocker}><Trash2 size={16} aria-hidden="true" /> i18n:govoplan-files.delete.f6fdbe48</Button>} />
|
||||
</FormSection>
|
||||
</ContentGrid> : <ContentGrid columns={1}>
|
||||
<FormSection title="i18n:govoplan-files.tools.import_sync" description="i18n:govoplan-files.tools.import_sync_description">
|
||||
<ActionToolbar>
|
||||
<Button onClick={() => chooseTool(() => { if (activeSpaceIsConnector) void syncConnectorSpaceSelection(); else void openConnectorSyncDialog(toolbarTarget()); })} disabled={Boolean(syncBlocker)} disabledReason={syncBlocker}><RefreshCw size={16} aria-hidden="true" /> i18n:govoplan-files.sync.905f6309</Button>
|
||||
{activeSpaceIsConnector && <Button onClick={() => chooseTool(openConnectorFolderSyncDialog)} disabled={Boolean(folderSyncBlocker)} disabledReason={folderSyncBlocker}><RefreshCw size={16} aria-hidden="true" /> i18n:govoplan-files.folder_sync.button</Button>}
|
||||
</ActionToolbar>
|
||||
</FormSection>
|
||||
<FormSection title="i18n:govoplan-files.tools.spaces" variant="separated">
|
||||
<ActionToolbar><Button onClick={() => chooseTool(() => void openConnectorSpaceDialog())} disabled={busy || !canOrganize} disabledReason={workingBlocker || (!canOrganize ? "File organization permission is required to add a connector space." : "")}><Link2 size={16} aria-hidden="true" /> i18n:govoplan-files.add_space.e4d674d4</Button></ActionToolbar>
|
||||
</FormSection>
|
||||
{activeSpaceIsConnector && <FormSection title="i18n:govoplan-files.tools.destructive" description="i18n:govoplan-files.tools.remove_space_description" variant="separated">
|
||||
<WorkspaceActionBar scope="detail-pane" variant="detail" label="i18n:govoplan-files.tools.destructive" destructiveActions={<Button variant="danger" onClick={() => chooseTool(() => { if (activeSpace) setConnectorSpaceRemovalTarget(activeSpace); })} disabled={busy || !canOrganize || !activeSpace?.connector_space_id} disabledReason={workingBlocker || (!canOrganize ? "File organization permission is required to remove a connector space." : !activeSpace?.connector_space_id ? "This space is not a removable connector space." : "")}><Trash2 size={16} aria-hidden="true" /> Remove space</Button>} />
|
||||
</FormSection>}
|
||||
</ContentGrid>}
|
||||
</Dialog>
|
||||
{error &&
|
||||
<DismissibleAlert tone="danger" resetKey={error} floating>{error}</DismissibleAlert>
|
||||
}
|
||||
@@ -2524,7 +2716,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
|
||||
<section className="file-list-panel" aria-label="i18n:govoplan-files.current_folder_contents.f9a24fa8">
|
||||
<div className="file-list-sticky">
|
||||
{toolbar}
|
||||
{selectionToolbar}
|
||||
<nav className="file-breadcrumbs" aria-label="i18n:govoplan-files.current_folder.5aeab2f0">
|
||||
<button type="button" className="file-breadcrumb" onClick={() => activeSpace && openFolder(activeSpace.id, "")} disabled={busy || !activeSpace}>
|
||||
<Home size={15} aria-hidden="true" /> {activeSpace?.label || "i18n:govoplan-files.files.6ce6c512"}
|
||||
@@ -2555,7 +2747,6 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
onClear={clearPropertyFilters} />
|
||||
|
||||
<div className="file-list-meta">
|
||||
<span>{activeSpaceIsConnector ? connectorSpaceSelectedItem?.kind === "file" ? connectorSpaceSelectedItem.name : "i18n:govoplan-files.remote_connector_space.d8956863" : selectedSummary}</span>
|
||||
<span>
|
||||
{activeSpaceIsConnector ? i18nMessage("i18n:govoplan-files.value_folder_s_value_file_s.76b92c8c", { value0:
|
||||
activeConnectorSpaceItems.filter((item) => item.kind === "folder" || item.kind === "library").length, value1: activeConnectorSpaceItems.filter((item) => item.kind === "file").length }) : i18nMessage("i18n:govoplan-files.value_folder_s_value_file_s.76b92c8c", { value0:
|
||||
@@ -2700,6 +2891,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
downloadLabel={downloadLabelForSets(selectedSetsForContext(contextMenu), contextMenu.spaceId ?? activeSpaceId)}
|
||||
onCreateFolder={() => openCreateFolderDialogForContext(contextMenu)}
|
||||
onUpload={() => openUploadDialogForContext(contextMenu)}
|
||||
canUnpackArchive={!busy && canUpload && canDownload && !isConnectorSpace(findSpace(contextMenu.spaceId ?? activeSpaceId)) && contextMenu.entry?.kind === "file" && ARCHIVE_FILENAME_PATTERN.test(contextMenu.entry.file.filename)}
|
||||
onUnpackArchive={() => contextMenu.entry?.kind === "file" && openManagedArchive(contextMenu.entry.file, contextActionTarget(contextMenu))}
|
||||
onDownload={() => void downloadContextSelection(contextMenu)}
|
||||
onMove={() => openTransferDialogForContext(contextMenu, "move")}
|
||||
onCopy={() => openTransferDialogForContext(contextMenu, "copy")}
|
||||
@@ -2735,14 +2928,30 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
|
||||
{dialog === "upload" &&
|
||||
<FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}>
|
||||
<FileDialog title={managedArchiveFile ? "i18n:govoplan-files.managed_archive.unpack" : i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} busy={busy || uploadActive} onClose={() => { if (!busy) closeDialog(); }}>
|
||||
<LoadingFrame loading={uploadActive} label={operationBusyLabel} indicator="none" progress={operationProgressValue} progressLabel={operationProgressLabel}>
|
||||
<div inert={uploadActive}>
|
||||
{managedArchiveFile && <p className="form-help">{i18nMessage("i18n:govoplan-files.managed_archive.source", { value0: managedArchiveFile.filename })}</p>}
|
||||
{managedArchiveFile && <p className="form-help">i18n:govoplan-files.managed_archive.protection</p>}
|
||||
{managedArchiveFile && <DocumentationHelpLink reference={{ topicId: "files.workflow.unpack-managed-archive", documentationType: "user" }} />}
|
||||
{error && <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert>}
|
||||
{!archivePreview &&
|
||||
<>
|
||||
<ToggleSwitch
|
||||
{!managedArchiveFile && <ToggleSwitch
|
||||
label="Preview and unpack archive"
|
||||
checked={unpackZip}
|
||||
onChange={setUnpackZip}
|
||||
disabled={busy} />
|
||||
disabled={busy} />}
|
||||
|
||||
{managedArchiveFile && <FormField label="i18n:govoplan-files.destination_space.92b63970">
|
||||
<select value={activeDialogSpace?.id || ""} disabled={busy} onChange={(event) => {
|
||||
updateActiveDialogFolder(event.target.value, "");
|
||||
const space = findSpace(event.target.value);
|
||||
if (space) void loadSpaceContents(space, { silent: true });
|
||||
}}>
|
||||
{spaces.filter((space) => !isConnectorSpace(space)).map((space) => <option key={space.id} value={space.id}>{space.label}</option>)}
|
||||
</select>
|
||||
</FormField>}
|
||||
|
||||
{unpackZip &&
|
||||
<p className="form-help archive-upload-help">
|
||||
@@ -2759,25 +2968,22 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
onSelect={(folderPath) => activeDialogTarget && updateActiveDialogFolder(activeDialogTarget.spaceId, folderPath)} />
|
||||
|
||||
</div>
|
||||
<FileDropZone
|
||||
{!managedArchiveFile && <FileDropZone
|
||||
disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace) || !canUpload}
|
||||
busy={uploadActive}
|
||||
progress={visibleUploadProgress}
|
||||
busyLabel={uploadBusyLabel}
|
||||
progressLabel={uploadProgressLabel}
|
||||
note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`}
|
||||
onRejectedDrop={(reason) => setError(reason === "disabled" ? uploadRejectedReason(activeDialogTarget) : "The browser did not provide readable file data for this drop. Use the file picker, or drag local files from a file manager that exposes file contents to the browser.")}
|
||||
onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} />
|
||||
onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} />}
|
||||
<div className="button-row compact-actions align-end">
|
||||
<Button onClick={closeDialog} disabled={busy}>i18n:govoplan-files.cancel.77dfd213</Button>
|
||||
{managedArchiveFile && <Button variant="primary" disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace)} onClick={() => activeDialogTarget && void loadArchivePreview(managedArchiveFile, activeDialogTarget)}>i18n:govoplan-files.managed_archive.preview</Button>}
|
||||
</div>
|
||||
</>
|
||||
}
|
||||
{archivePreview && archiveFile &&
|
||||
{archivePreview && (archiveFile || managedArchiveFile) &&
|
||||
<div className="archive-preview">
|
||||
<div className="archive-preview-summary">
|
||||
<div>
|
||||
<strong>{archiveFile.name}</strong>
|
||||
<strong>{managedArchiveFile?.filename || archiveFile?.name}</strong>
|
||||
<span>{archivePreview.archive_format.toUpperCase()} · {formatBytes(archivePreview.compressed_size_bytes)} compressed</span>
|
||||
</div>
|
||||
<div>
|
||||
@@ -2840,30 +3046,33 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
<Folder size={16} aria-hidden="true" /> :
|
||||
<File size={16} aria-hidden="true" />
|
||||
}
|
||||
<span className="archive-entry-path">{entry.path.split("/").at(-1)}</span>
|
||||
<span className="archive-entry-path">{entry.path.split("/").slice(-1)[0]}</span>
|
||||
<span className="archive-entry-size">{entry.kind === "file" ? formatBytes(entry.size_bytes) : `${selectableFiles.length} files`}</span>
|
||||
</label>);
|
||||
})}
|
||||
</div>
|
||||
|
||||
<p className="form-help">
|
||||
Preview expires {formatDate(archivePreview.expires_at)}. The original archive is uploaded again only when you confirm.
|
||||
{managedArchiveFile
|
||||
? i18nMessage("i18n:govoplan-files.managed_archive.expires", { value0: formatDate(archivePreview.expires_at) })
|
||||
: i18nMessage("i18n:govoplan-files.archive_progress.preview_expires", { expires: formatDate(archivePreview.expires_at) })}
|
||||
</p>
|
||||
|
||||
<div className="button-row compact-actions archive-preview-actions">
|
||||
<Button
|
||||
onClick={() => {
|
||||
resetArchiveUploadState();
|
||||
if (managedArchiveFile) { setArchivePreview(null); setSelectedArchivePaths(new Set()); }
|
||||
else resetArchiveUploadState();
|
||||
setError("");
|
||||
}}
|
||||
disabled={busy}>
|
||||
Choose another file
|
||||
{managedArchiveFile ? "i18n:govoplan-files.managed_archive.change_destination" : "Choose another file"}
|
||||
</Button>
|
||||
{archivePreview.requires_password &&
|
||||
<Button
|
||||
helpContextId="files.list"
|
||||
helpModuleId="files"
|
||||
onClick={() => activeDialogTarget && void loadArchivePreview(archiveFile, activeDialogTarget, { preserveSelection: true })}
|
||||
onClick={() => activeDialogTarget && void loadArchivePreview((managedArchiveFile || archiveFile)!, activeDialogTarget, { preserveSelection: true })}
|
||||
disabled={busy || !archivePassword}>
|
||||
<RefreshCw size={15} aria-hidden="true" /> Verify password
|
||||
</Button>
|
||||
@@ -2883,6 +3092,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
</LoadingFrame>
|
||||
</FileDialog>
|
||||
}
|
||||
|
||||
@@ -3243,7 +3454,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
</FileDialog>
|
||||
}
|
||||
</div>);
|
||||
</WorkspaceFrame>);
|
||||
|
||||
}
|
||||
|
||||
|
||||
@@ -195,12 +195,14 @@ export function RenamePreviewList({
|
||||
|
||||
}
|
||||
|
||||
export function FileDialog({ title, onClose, children }: {title: string;onClose: () => void;children: ReactNode;}) {
|
||||
export function FileDialog({ title, onClose, children, busy = false }: {title: string;onClose: () => void;children: ReactNode;busy?: boolean;}) {
|
||||
return (
|
||||
<Dialog
|
||||
open
|
||||
title={title}
|
||||
onClose={onClose}
|
||||
closeDisabled={busy}
|
||||
closeOnBackdrop={!busy}
|
||||
backdropClassName="file-dialog-backdrop"
|
||||
className="file-dialog"
|
||||
headerClassName="file-dialog-header"
|
||||
@@ -217,6 +219,7 @@ export function FileContextMenu({
|
||||
hasSelection,
|
||||
canCreateFolder,
|
||||
canUpload,
|
||||
canUnpackArchive,
|
||||
canDownload,
|
||||
canOrganize,
|
||||
canDelete,
|
||||
@@ -224,6 +227,7 @@ export function FileContextMenu({
|
||||
downloadLabel,
|
||||
onCreateFolder,
|
||||
onUpload,
|
||||
onUnpackArchive,
|
||||
onDownload,
|
||||
onMove,
|
||||
onCopy,
|
||||
@@ -244,13 +248,13 @@ export function FileContextMenu({
|
||||
|
||||
|
||||
|
||||
}: {menu: ContextMenuState;hasSelection: boolean;canCreateFolder: boolean;canUpload: boolean;canDownload: boolean;canOrganize: boolean;canDelete: boolean;canExplainAccess: boolean;downloadLabel: string;onCreateFolder: () => void;onUpload: () => void;onDownload: () => void;onMove: () => void;onCopy: () => void;onExplainAccess: () => void;onDelete: () => void;}) {
|
||||
}: {menu: ContextMenuState;hasSelection: boolean;canCreateFolder: boolean;canUpload: boolean;canUnpackArchive: boolean;canDownload: boolean;canOrganize: boolean;canDelete: boolean;canExplainAccess: boolean;downloadLabel: string;onCreateFolder: () => void;onUpload: () => void;onUnpackArchive: () => void;onDownload: () => void;onMove: () => void;onCopy: () => void;onExplainAccess: () => void;onDelete: () => void;}) {
|
||||
const showNewFolder = true;
|
||||
const showDelete = menu.target !== "empty";
|
||||
const viewportWidth = typeof window === "undefined" ? 1024 : window.innerWidth;
|
||||
const viewportHeight = typeof window === "undefined" ? 768 : window.innerHeight;
|
||||
const estimatedWidth = 220;
|
||||
const estimatedHeight = 260;
|
||||
const estimatedHeight = 300;
|
||||
const left = Math.max(8, Math.min(menu.x, viewportWidth - estimatedWidth - 8));
|
||||
const openUp = menu.y + estimatedHeight > viewportHeight;
|
||||
const style: CSSProperties = openUp ?
|
||||
@@ -260,6 +264,7 @@ export function FileContextMenu({
|
||||
<div className="file-context-menu" style={style} role="menu" onClick={(event) => event.stopPropagation()}>
|
||||
{showNewFolder && <button type="button" role="menuitem" onClick={onCreateFolder} disabled={!canCreateFolder}><Plus size={15} aria-hidden="true" /> i18n:govoplan-files.new_folder.a711999b</button>}
|
||||
<button type="button" role="menuitem" onClick={onUpload} disabled={!canUpload}><UploadCloud size={15} aria-hidden="true" /> i18n:govoplan-files.upload.8bdf057f</button>
|
||||
<button type="button" role="menuitem" onClick={onUnpackArchive} disabled={!canUnpackArchive}>i18n:govoplan-files.managed_archive.unpack</button>
|
||||
<button type="button" role="menuitem" onClick={onDownload} disabled={!hasSelection || !canDownload}><Download size={15} aria-hidden="true" /> {downloadLabel}</button>
|
||||
<button type="button" role="menuitem" onClick={onMove} disabled={!hasSelection || !canOrganize}><MoveRight size={15} aria-hidden="true" /> i18n:govoplan-files.move.8a74a26e</button>
|
||||
<button type="button" role="menuitem" onClick={onCopy} disabled={!hasSelection || !canOrganize}><Copy size={15} aria-hidden="true" /> i18n:govoplan-files.copy.92556c6d</button>
|
||||
|
||||
@@ -2,6 +2,37 @@ import type { PlatformTranslations } from "@govoplan/core-webui";
|
||||
|
||||
export const generatedTranslations: PlatformTranslations = {
|
||||
"en": {
|
||||
"i18n:govoplan-files.tools.connections": "Connections and imports",
|
||||
"i18n:govoplan-files.tools.connections_description": "Browse linked sources or import files explicitly. Reload only refreshes the current listing; it never synchronizes or imports files.",
|
||||
"i18n:govoplan-files.tools.selection": "Manage selection",
|
||||
"i18n:govoplan-files.tools.organize": "Organize files and folders",
|
||||
"i18n:govoplan-files.tools.organize_description": "Move, copy, or rename the selected items. The next dialog lets you review the destination or preview the change.",
|
||||
"i18n:govoplan-files.tools.sharing_access": "Sharing and access",
|
||||
"i18n:govoplan-files.tools.destructive": "Removal actions",
|
||||
"i18n:govoplan-files.tools.delete_description": "Deletion requires confirmation. Retention and audit protections continue to apply.",
|
||||
"i18n:govoplan-files.tools.import_sync": "Import and synchronize",
|
||||
"i18n:govoplan-files.tools.import_sync_description": "These actions can create or update managed files. A selected remote file is synchronized explicitly; folder synchronization has its own scope and conflict review.",
|
||||
"i18n:govoplan-files.tools.spaces": "Linked file spaces",
|
||||
"i18n:govoplan-files.tools.remove_space_description": "Remove only the local link after confirmation. Remote provider files and previously imported managed files remain unchanged.",
|
||||
"i18n:govoplan-files.archive_progress.inspecting": "Inspecting the archive…",
|
||||
"i18n:govoplan-files.archive_progress.extracting": "Extracting selected archive files…",
|
||||
"i18n:govoplan-files.archive_progress.storing": "Storing extracted files…",
|
||||
"i18n:govoplan-files.archive_progress.finalizing": "Finalizing and committing changes…",
|
||||
"i18n:govoplan-files.archive_progress.processed": "{completed} of {total} files · {bytes} of {totalBytes} processed. Keep this dialog open until completion.",
|
||||
"i18n:govoplan-files.archive_progress.selected": "{total} files selected · {bytes}. Waiting for server progress; keep this dialog open.",
|
||||
"i18n:govoplan-files.archive_progress.transferred": "{percentage}% transferred to the server. Inspection and processing follow separately.",
|
||||
"i18n:govoplan-files.archive_progress.waiting": "Waiting for a measured result. Keep this dialog open; no completion percentage is available yet.",
|
||||
"i18n:govoplan-files.archive_progress.preview_expires": "Preview expires {expires}. Confirmation reuses the protected temporary archive when available and rechecks its contents and your destination.",
|
||||
"i18n:govoplan-files.managed_archive.unpack": "Unpack archive",
|
||||
"i18n:govoplan-files.managed_archive.select_one": "Select exactly one managed ZIP or TAR archive first.",
|
||||
"i18n:govoplan-files.managed_archive.download_required": "File download permission is required to unpack an existing archive.",
|
||||
"i18n:govoplan-files.managed_archive.source": "Source: {value0}. The managed archive remains unchanged; no browser download or re-upload is needed.",
|
||||
"i18n:govoplan-files.managed_archive.preview": "Preview archive",
|
||||
"i18n:govoplan-files.managed_archive.protection": "Extracted files use normal upload storage. Archive passwords and a source storage encryption envelope are not automatically applied to individual files.",
|
||||
"i18n:govoplan-files.managed_archive.inspecting": "Inspecting the managed archive…",
|
||||
"i18n:govoplan-files.managed_archive.extracting": "Extracting the selected archive files. Keep this dialog open until the operation finishes.",
|
||||
"i18n:govoplan-files.managed_archive.expires": "Preview expires {value0}. Confirmation rechecks the source version and access; existing destination files and the source archive are never overwritten.",
|
||||
"i18n:govoplan-files.managed_archive.change_destination": "Change destination",
|
||||
"i18n:govoplan-files.add_connector_space.aa6bdbd6": "Add connector space",
|
||||
"i18n:govoplan-files.add_credential_for_value.0fa9c1fe": "Add credential for {value0}",
|
||||
"i18n:govoplan-files.add_prefix.672452bc": "Add prefix",
|
||||
@@ -401,6 +432,37 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.writable.dd35487a": "Writable"
|
||||
},
|
||||
"de": {
|
||||
"i18n:govoplan-files.tools.connections": "Verbindungen und Importe",
|
||||
"i18n:govoplan-files.tools.connections_description": "Verknüpfte Quellen durchsuchen oder Dateien ausdrücklich importieren. Neu laden aktualisiert nur die aktuelle Liste; es synchronisiert oder importiert keine Dateien.",
|
||||
"i18n:govoplan-files.tools.selection": "Auswahl verwalten",
|
||||
"i18n:govoplan-files.tools.organize": "Dateien und Ordner organisieren",
|
||||
"i18n:govoplan-files.tools.organize_description": "Ausgewählte Elemente verschieben, kopieren oder umbenennen. Im nächsten Dialog prüfen Sie das Ziel oder eine Vorschau der Änderung.",
|
||||
"i18n:govoplan-files.tools.sharing_access": "Freigaben und Zugriff",
|
||||
"i18n:govoplan-files.tools.destructive": "Entfernen",
|
||||
"i18n:govoplan-files.tools.delete_description": "Das Löschen erfordert eine Bestätigung. Aufbewahrungsvorgaben und Schutz für prüfrelevante Dateien gelten weiterhin.",
|
||||
"i18n:govoplan-files.tools.import_sync": "Importieren und synchronisieren",
|
||||
"i18n:govoplan-files.tools.import_sync_description": "Diese Aktionen können verwaltete Dateien anlegen oder aktualisieren. Eine ausgewählte entfernte Datei wird ausdrücklich synchronisiert; die Ordnersynchronisierung hat einen eigenen Umfang und eine Konfliktprüfung.",
|
||||
"i18n:govoplan-files.tools.spaces": "Verknüpfte Dateibereiche",
|
||||
"i18n:govoplan-files.tools.remove_space_description": "Nach Bestätigung wird nur die lokale Verknüpfung entfernt. Dateien beim entfernten Anbieter und bereits importierte verwaltete Dateien bleiben unverändert.",
|
||||
"i18n:govoplan-files.archive_progress.inspecting": "Archiv wird geprüft…",
|
||||
"i18n:govoplan-files.archive_progress.extracting": "Ausgewählte Archivdateien werden entpackt…",
|
||||
"i18n:govoplan-files.archive_progress.storing": "Entpackte Dateien werden gespeichert…",
|
||||
"i18n:govoplan-files.archive_progress.finalizing": "Änderungen werden abgeschlossen und verbindlich gespeichert…",
|
||||
"i18n:govoplan-files.archive_progress.processed": "{completed} von {total} Dateien · {bytes} von {totalBytes} verarbeitet. Diesen Dialog bis zum Abschluss geöffnet lassen.",
|
||||
"i18n:govoplan-files.archive_progress.selected": "{total} Dateien ausgewählt · {bytes}. Serverfortschritt wird erwartet; diesen Dialog geöffnet lassen.",
|
||||
"i18n:govoplan-files.archive_progress.transferred": "{percentage}% an den Server übertragen. Prüfung und Verarbeitung folgen gesondert.",
|
||||
"i18n:govoplan-files.archive_progress.waiting": "Ein gemessenes Ergebnis wird erwartet. Diesen Dialog geöffnet lassen; ein Abschlussprozentsatz ist noch nicht verfügbar.",
|
||||
"i18n:govoplan-files.archive_progress.preview_expires": "Die Vorschau läuft am {expires} ab. Die Bestätigung verwendet das geschützte temporäre Archiv erneut, sofern verfügbar, und prüft Inhalt sowie Ziel erneut.",
|
||||
"i18n:govoplan-files.managed_archive.unpack": "Archiv entpacken",
|
||||
"i18n:govoplan-files.managed_archive.select_one": "Wählen Sie zunächst genau ein verwaltetes ZIP- oder TAR-Archiv aus.",
|
||||
"i18n:govoplan-files.managed_archive.download_required": "Zum Entpacken eines vorhandenen Archivs ist die Berechtigung zum Herunterladen erforderlich.",
|
||||
"i18n:govoplan-files.managed_archive.source": "Quelle: {value0}. Das verwaltete Archiv bleibt unverändert; Herunterladen und erneutes Hochladen im Browser sind nicht erforderlich.",
|
||||
"i18n:govoplan-files.managed_archive.preview": "Archivvorschau",
|
||||
"i18n:govoplan-files.managed_archive.protection": "Entpackte Dateien verwenden die normalen Upload-Speichereinstellungen. Archivpasswörter und eine Speicher-Verschlüsselungshülle der Quelle werden nicht automatisch auf einzelne Dateien angewendet.",
|
||||
"i18n:govoplan-files.managed_archive.inspecting": "Verwaltetes Archiv wird geprüft…",
|
||||
"i18n:govoplan-files.managed_archive.extracting": "Die ausgewählten Archivdateien werden entpackt. Lassen Sie diesen Dialog bis zum Abschluss geöffnet.",
|
||||
"i18n:govoplan-files.managed_archive.expires": "Die Vorschau läuft am {value0} ab. Beim Bestätigen werden Quellversion und Zugriff erneut geprüft. Vorhandene Zieldateien und das Quellarchiv werden niemals überschrieben.",
|
||||
"i18n:govoplan-files.managed_archive.change_destination": "Ziel ändern",
|
||||
"i18n:govoplan-files.add_connector_space.aa6bdbd6": "Add connector space",
|
||||
"i18n:govoplan-files.add_credential_for_value.0fa9c1fe": "Add credential for {value0}",
|
||||
"i18n:govoplan-files.add_prefix.672452bc": "Add prefix",
|
||||
|
||||
+3
-2
@@ -8,6 +8,7 @@ import {
|
||||
type PlatformWebModule,
|
||||
type QuickAccessToolsUiCapability
|
||||
} from "@govoplan/core-webui";
|
||||
import { generatedTranslations as productSurfaceTranslations } from "@govoplan/core-webui/outcome-product-surface-translations";
|
||||
import { FolderTree } from "./features/files/components/FileManagerComponents";
|
||||
import FileConnectorSettingsPanel from "./features/files/FileConnectorSettingsPanel";
|
||||
import ManagedFileChooser from "./features/files/components/ManagedFileChooser";
|
||||
@@ -22,8 +23,8 @@ const FileIntegrityPanel = lazy(() => import("./features/files/FileIntegrityPane
|
||||
|
||||
const fileRead = ["files:file:read"];
|
||||
const translations = {
|
||||
en: generatedTranslations.en,
|
||||
de: generatedTranslations.de
|
||||
en: { ...generatedTranslations.en, ...productSurfaceTranslations.en },
|
||||
de: { ...generatedTranslations.de, ...productSurfaceTranslations.de }
|
||||
};
|
||||
const fileDashboardWidgets: DashboardWidgetsUiCapability = {
|
||||
widgets: [
|
||||
|
||||
Reference in New Issue
Block a user