Add safe archive preview and extraction workflows

This commit is contained in:
2026-07-31 02:48:56 +02:00
parent 159a012833
commit b752dea610
20 changed files with 1961 additions and 182 deletions
+13 -8
View File
@@ -119,11 +119,14 @@ access fails closed in both public-only and private-network deployments. It will
remain disabled until every initial connection and referral target can be remain disabled until every initial connection and referral target can be
policy-validated and pinned. policy-validated and pinned.
The S3 browse/import implementation and S3 managed-storage backend currently S3 browse/import and arbitrary external S3 managed-storage endpoints currently
fail closed before creating a `boto3` client. Botocore does not yet use the fail closed before creating a `boto3` client. Botocore does not yet use the
GovOPlaN pinned HTTP transport and may manage redirects itself, so neither an GovOPlaN pinned HTTP transport and may manage redirects itself. The only
explicit endpoint nor SDK endpoint discovery is allowed until both peer pinning exception is the deployment-owned Garage service installed at the exact
and redirect revalidation are enforced. `http://garage:3900` endpoint with
`FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true`; this flag cannot authorize another
host. External endpoints remain disabled until peer pinning and redirect
revalidation are enforced.
Destructive Files-module retirement applies the same credential lifecycle before Destructive Files-module retirement applies the same credential lifecycle before
dropping tables. Every remaining Files-owned encrypted connector secret is dropping tables. Every remaining Files-owned encrypted connector secret is
@@ -140,10 +143,12 @@ Connector and collaboration ownership boundaries are documented in
The role-adaptive user, administration, integration, and operator guide is the The role-adaptive user, administration, integration, and operator guide is the
[Files handbook](docs/FILES_HANDBOOK.md). [Files handbook](docs/FILES_HANDBOOK.md).
ZIP uploads are processed without buffering the whole archive in memory. The API Archive imports use a two-phase preview and confirmation flow for ZIP, TAR,
spools incoming ZIP request bodies to a bounded temporary file, then extracts TAR.GZ, TAR.BZ2, and TAR.XZ. Requests are spooled to bounded temporary files;
members with per-file and total extracted-size limits before storing managed the server validates paths, entry count, expanded size, and expansion ratio,
files. then returns a 30-minute tenant/user-bound preview token. Confirmation reuploads
the original archive and stores only the selected members. Password-protected
ZIP passwords remain request-only and are never included in the preview token.
Bulk rename and transfer APIs are owner-scoped: callers must provide the active Bulk rename and transfer APIs are owner-scoped: callers must provide the active
user or group file space with `owner_type` and `owner_id`. The storage layer user or group file space with `owner_type` and `owner_id`. The storage layer
+48 -27
View File
@@ -94,21 +94,29 @@ An ordinary upload does not append a new version to an existing asset. The
`overwrite` strategy retires the old asset at that path. Version-preserving `overwrite` strategy retires the old asset at that path. Version-preserving
updates currently belong to connector sync. updates currently belong to connector sync.
### Upload and unpack ZIP files ### Preview and unpack archives
The UI can unpack a ZIP upload. The request body is spooled to a bounded The UI previews ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ before writing managed
temporary file rather than buffered wholly in memory. The defaults are: files. Users may select individual files or complete folders. The request is
spooled to a bounded temporary file rather than buffered wholly in memory. The
defaults are:
- 250 MiB compressed request and total extracted data; - 250 MiB compressed request;
- 50 MiB per extracted member; - 50 MiB per extracted member;
- 1,000 non-directory members; - 2 GiB total expanded data;
- encrypted archives are rejected; - 10,000 declared entries;
- member paths are normalized and `..` traversal is rejected; - 100:1 maximum expansion ratio;
- the actual bytes read are counted, not only the ZIP header declarations. - a 30-minute preview token bound to the tenant, user, archive digest, and
destination;
- password-protected ZIP support with request-only password handling;
- traversal, duplicate paths, links, devices, and other special entries are
rejected;
- actual bytes read are counted, not only archive header declarations.
The byte limits use `FILE_UPLOAD_ZIP_MAX_BYTES` and The browser retains the selected archive and password until confirmation.
`FILE_UPLOAD_MAX_BYTES`. The 1,000-member limit is currently fixed in the Confirmation reuploads the archive, verifies the token and digest, repeats all
extraction service. safety checks, and commits only the selected files. No preview archive or
password is retained server-side.
### Organize files and folders ### Organize files and folders
@@ -399,8 +407,13 @@ the table above says live access is disabled.
| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root | | `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root |
| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root | | `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root |
| `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket | | `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket |
| `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and ZIP-member maximum | | `FILE_STORAGE_S3_DEPLOYMENT_MANAGED` | `false` | Installer-only trust marker for the exact `http://garage:3900` service; never use it for another endpoint |
| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | ZIP request and extracted-total maximum | | `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and extracted archive-member maximum |
| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | Compressed archive request maximum (legacy name retained for compatibility) |
| `FILE_ARCHIVE_MAX_ENTRIES` | 10,000 | Maximum declared archive entries |
| `FILE_ARCHIVE_MAX_EXPANDED_BYTES` | 2 GiB | Maximum expanded archive bytes |
| `FILE_ARCHIVE_MAX_EXPANSION_RATIO` | 100 | Maximum expanded-to-compressed ratio |
| `FILE_ARCHIVE_PREVIEW_TTL_SECONDS` | 1,800 | Lifetime of the sealed archive preview token |
| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets | | `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets |
The local backend is the operational baseline. It resolves every storage key The local backend is the operational baseline. It resolves every storage key
@@ -408,10 +421,14 @@ under the configured root and rejects escape attempts. Fallback roots support a
controlled storage-root migration: new writes go to the primary root while controlled storage-root migration: new writes go to the primary root while
reads can still find older objects. reads can still find older objects.
The S3 managed-storage adapter currently fails closed before creating a boto3 The S3 managed-storage adapter fails closed before creating a boto3 client for
client because the SDK cannot yet guarantee connection-time DNS/IP pinning and arbitrary external endpoints because the SDK cannot yet guarantee
redirect revalidation. Do not select `FILE_STORAGE_BACKEND=s3` for a live Files connection-time DNS/IP pinning and redirect revalidation. The supported
deployment until that boundary is implemented and the status above changes. installer may provision a deployment-owned Garage service at the exact
`http://garage:3900` endpoint and set
`FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true`. Files accepts only that exact
service-discovery endpoint and forces path-style S3 addressing. The marker is
deployment authority, not a general private-network bypass.
Multiple API replicas require the same durable blob namespace. Separate local Multiple API replicas require the same durable blob namespace. Separate local
container filesystems will produce incomplete reads. Until a pinned shared container filesystems will produce incomplete reads. Until a pinned shared
@@ -558,7 +575,7 @@ All routes below are under `/api/v1/files`.
| Area | Routes | | Area | Routes |
| --- | --- | | --- | --- |
| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` | | Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` |
| Upload and folders | `POST /upload`, `POST /upload-zip`, `POST /folders`, `POST /folders/delete` | | Upload and folders | `POST /upload`, `POST /upload-zip` (compatibility), `POST /archive-preview`, `POST /archive-confirm`, `POST /folders`, `POST /folders/delete` |
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` | | File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` | | Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` | | Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
@@ -603,8 +620,9 @@ Files baseline indiscriminately.
- Owner and share checks protect individual resources after operation-scope - Owner and share checks protect individual resources after operation-scope
checks. checks.
- Logical paths reject traversal and storage keys cannot escape the local root. - Logical paths reject traversal and storage keys cannot escape the local root.
- Upload, ZIP extraction, connector response, and S3 stream code use bounded - Upload, archive extraction, connector response, and S3 stream code use bounded
reads; encrypted ZIP archives are rejected. reads. Archive previews are sealed and short-lived; ZIP passwords are
request-only.
- Connector HTTP sockets use connection-time DNS/IP validation and pinning, - Connector HTTP sockets use connection-time DNS/IP validation and pinning,
redirects are refused, and unsafe SDK transports fail before client creation. redirects are refused, and unsafe SDK transports fail before client creation.
- Database-managed connector passwords/tokens are encrypted; responses redact - Database-managed connector passwords/tokens are encrypted; responses redact
@@ -699,12 +717,14 @@ her personal or Finance space, then the asset has the selected owner and tenant.
Given Bob is outside Finance and has no share or Files admin permission, the Given Bob is outside Finance and has no share or Files admin permission, the
same asset ID must not make the asset readable to Bob. same asset ID must not make the asset readable to Bob.
### Safe ZIP import ### Safe archive import
Given a ZIP member contains `../../secret.txt`, is encrypted, exceeds 50 MiB, Given an archive member contains `../../secret.txt`, is a link or special
or pushes actual extracted bytes above 250 MiB, unpacking must fail without a filesystem object, exceeds 50 MiB, pushes actual expanded bytes above 2 GiB, or
committed managed asset. A valid archive must preserve normalized relative paths exceeds the 100:1 expansion ratio, preview or confirmation must fail without a
below the chosen logical folder. committed managed asset. A password-protected ZIP requires the correct
request-only password. A valid confirmation must match its unexpired preview
token and preserve normalized relative paths below the chosen logical folder.
### Explicit conflicts ### Explicit conflicts
@@ -760,7 +780,7 @@ returning different content or credentials.
| Area | Implemented now | Planned or explicitly outside the current boundary | | Area | Implemented now | Planned or explicitly outside the current boundary |
| --- | --- | --- | | --- | --- | --- |
| Managed storage | Local durable-root backend, fallback read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, and dry-run-first orphan cleanup | Operational S3 after pinned SDK transport ([#34](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/34)); scheduled scan execution | | Managed storage | Local durable-root backend, fallback read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, and dry-run-first orphan cleanup | Operational S3 after pinned SDK transport ([#34](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/34)); scheduled scan execution |
| Upload | Bounded direct upload, drag-and-drop UI, ZIP spool/extract limits, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload | | Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload |
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore | | Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles | | Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) | | Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
@@ -788,7 +808,8 @@ Before releasing Files:
5. Exercise an allowed and denied owner/share path. 5. Exercise an allowed and denied owner/share path.
6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion. 6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion.
7. Exercise connector policy explanation and one pinned HTTP provider where 7. Exercise connector policy explanation and one pinned HTTP provider where
configured; verify S3/SMB still fail closed. configured; verify managed Garage if selected, and verify arbitrary external
S3 and SMB still fail closed.
8. Verify credential deletion scrubs dependents and produces audit evidence. 8. Verify credential deletion scrubs dependents and produces audit evidence.
9. Verify a campaign attachment snapshot still identifies its exact version and 9. Verify a campaign attachment snapshot still identifies its exact version and
checksum after the current file changes. checksum after the current file changes.
+1
View File
@@ -13,6 +13,7 @@ authors = [{ name = "GovOPlaN" }]
dependencies = [ dependencies = [
"govoplan-core>=0.1.9", "govoplan-core>=0.1.9",
"defusedxml>=0.7,<1", "defusedxml>=0.7,<1",
"pyzipper>=0.3.6,<1",
"python-multipart>=0.0.31,<1", "python-multipart>=0.0.31,<1",
] ]
+78 -22
View File
@@ -12,7 +12,7 @@ from govoplan_core.core.modules import (
DocumentationLink, DocumentationLink,
DocumentationTopic, DocumentationTopic,
) )
from govoplan_files.backend.storage.archives import ZIP_UPLOAD_MAX_FILES from govoplan_files.backend.storage.archives import ARCHIVE_UPLOAD_MAX_ENTRIES
from govoplan_files.backend.storage.access import user_group_ids from govoplan_files.backend.storage.access import user_group_ids
from govoplan_files.backend.storage.connector_visibility import ( from govoplan_files.backend.storage.connector_visibility import (
connector_profile_usable_for_import, connector_profile_usable_for_import,
@@ -22,6 +22,9 @@ from govoplan_files.backend.storage.connector_visibility import (
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024 _DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024 _DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES = 2 * 1024 * 1024 * 1024
_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO = 100
_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS = 30 * 60
_FILES_READ_SCOPE = "files:file:read" _FILES_READ_SCOPE = "files:file:read"
_FILES_UPLOAD_SCOPE = "files:file:upload" _FILES_UPLOAD_SCOPE = "files:file:upload"
@@ -42,11 +45,47 @@ def documentation_topics(
"file_upload_zip_max_bytes", "file_upload_zip_max_bytes",
default=_DEFAULT_ZIP_MAX_BYTES, default=_DEFAULT_ZIP_MAX_BYTES,
) )
archive_expanded_limit = _configured_positive_int(
context.settings,
"file_archive_max_expanded_bytes",
default=_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES,
)
archive_entry_limit = _configured_positive_int(
context.settings,
"file_archive_max_entries",
default=ARCHIVE_UPLOAD_MAX_ENTRIES,
)
archive_ratio_limit = _configured_positive_int(
context.settings,
"file_archive_max_expansion_ratio",
default=_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO,
)
archive_preview_ttl = _configured_positive_int(
context.settings,
"file_archive_preview_ttl_seconds",
default=_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS,
)
topics: list[DocumentationTopic] = [] topics: list[DocumentationTopic] = []
if upload_limit is not None: if upload_limit is not None:
topics.append(_upload_topic(upload_limit)) topics.append(_upload_topic(upload_limit))
if upload_limit is not None and zip_limit is not None: if (
topics.append(_zip_topic(upload_limit, zip_limit)) upload_limit is not None
and zip_limit is not None
and archive_expanded_limit is not None
and archive_entry_limit is not None
and archive_ratio_limit is not None
and archive_preview_ttl is not None
):
topics.append(
_archive_topic(
upload_limit,
zip_limit,
archive_expanded_limit,
archive_entry_limit,
archive_ratio_limit,
archive_preview_ttl,
)
)
topics.append(_connector_import_topic(context)) topics.append(_connector_import_topic(context))
return tuple(topics) return tuple(topics)
@@ -118,19 +157,29 @@ def _upload_topic(max_bytes: int) -> DocumentationTopic:
) )
def _zip_topic(max_file_bytes: int, max_zip_bytes: int) -> DocumentationTopic: def _archive_topic(
max_file_bytes: int,
max_archive_request_bytes: int,
max_expanded_bytes: int,
max_entries: int,
max_expansion_ratio: int,
preview_ttl_seconds: int,
) -> DocumentationTopic:
member_limit = _format_byte_limit(max_file_bytes) member_limit = _format_byte_limit(max_file_bytes)
archive_limit = _format_byte_limit(max_zip_bytes) request_limit = _format_byte_limit(max_archive_request_bytes)
expanded_limit = _format_byte_limit(max_expanded_bytes)
preview_minutes = max(1, preview_ttl_seconds // 60)
return DocumentationTopic( return DocumentationTopic(
id="files.workflow.upload-and-unpack-zip", id="files.workflow.upload-and-unpack-zip",
title="Upload and unpack a ZIP archive", title="Preview and unpack an archive",
summary=( summary=(
f"Safely unpack up to {ZIP_UPLOAD_MAX_FILES:,} files from a ZIP whose request and extracted total are each limited to {archive_limit}." f"Review and selectively unpack up to {max_entries:,} entries from ZIP or TAR archives before any managed file is created."
), ),
body=( body=(
f"The current deployment limits the ZIP request and actual extracted total to {archive_limit}, each member to {member_limit}, " f"The deployment accepts ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ requests up to {request_limit}, limits actual expanded data to {expanded_limit}, "
f"and the archive to {ZIP_UPLOAD_MAX_FILES:,} non-directory members. Encrypted archives and unsafe member paths are rejected. " f"each member to {member_limit}, the archive to {max_entries:,} entries, and expansion to {max_expansion_ratio}:1. "
"Actual extracted bytes are counted instead of trusting ZIP headers." f"The server-issued preview expires after {preview_minutes} minutes. Password-protected ZIP archives are supported; passwords remain request-only. "
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers."
), ),
layer="configured", layer="configured",
documentation_types=("user",), documentation_types=("user",),
@@ -161,35 +210,42 @@ def _zip_topic(max_file_bytes: int, max_zip_bytes: int) -> DocumentationTopic:
"help_contexts": ["files.list"], "help_contexts": ["files.list"],
"prerequisites": [ "prerequisites": [
"You may view and upload managed files.", "You may view and upload managed files.",
"The archive is not encrypted and fits the current configured limits.", "The archive fits the configured request, expansion, size, and entry limits.",
"The destination personal or group space grants this account write access.", "The destination personal or group space grants this account write access.",
], ],
"steps": [ "steps": [
"Open Files and choose the managed destination space and folder.", "Open Files and choose the managed destination space and folder.",
"Enable Unpack ZIP uploads, then choose or drag the ZIP archive.", "Enable Preview and unpack archive, then choose or drag one supported archive.",
"Review the discovered entries, supply a ZIP password when required, and select the files or folders to import.",
"Resolve every destination conflict explicitly.", "Resolve every destination conflict explicitly.",
"Wait for extraction and finalization to finish before leaving the page.", "Confirm the selection, then wait for extraction and finalization to finish before leaving the page.",
], ],
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.", "outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.", "verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
"constraints": [ "constraints": [
{ {
"id": "zip-request-and-total", "id": "archive-request-and-total",
"label": "Maximum ZIP request and extracted total", "label": "Maximum archive request and expanded total",
"description": f"Both the compressed request and the actual extracted total are limited to {archive_limit}.", "description": f"The compressed request is limited to {request_limit}; actual expanded data is limited to {expanded_limit}.",
"values": [archive_limit], "values": [request_limit, expanded_limit],
}, },
{ {
"id": "zip-member-size", "id": "archive-member-size",
"label": "Maximum extracted member size", "label": "Maximum extracted member size",
"description": f"Each extracted file is limited to {member_limit}.", "description": f"Each extracted file is limited to {member_limit}.",
"values": [member_limit], "values": [member_limit],
}, },
{ {
"id": "zip-member-count", "id": "archive-entry-count",
"label": "Maximum file count", "label": "Maximum entry count",
"description": f"A ZIP may contain at most {ZIP_UPLOAD_MAX_FILES:,} non-directory members.", "description": f"An archive may contain at most {max_entries:,} declared entries.",
"values": [f"{ZIP_UPLOAD_MAX_FILES:,} files"], "values": [f"{max_entries:,} entries"],
},
{
"id": "archive-expansion-ratio",
"label": "Maximum expansion ratio",
"description": f"Declared and actual output may not exceed {max_expansion_ratio} times the compressed request size.",
"values": [f"{max_expansion_ratio}:1"],
}, },
], ],
"related_topic_ids": [ "related_topic_ids": [
+12 -3
View File
@@ -509,7 +509,7 @@ manifest = ModuleManifest(
summary="Back up database evidence, blob bytes, and the encryption key as one recovery unit, and keep unsupported SDK transports fail-closed.", summary="Back up database evidence, blob bytes, and the encryption key as one recovery unit, and keep unsupported SDK transports fail-closed.",
body=( body=(
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the original master key, then run the bounded resumable integrity scan and verify representative access paths. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. " "Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the original master key, then run the bounded resumable integrity scan and verify representative access paths. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. "
"Live S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Destructive module retirement drops database tables but does not remove backend blob objects." "Arbitrary external S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Installer-owned Garage storage is supported only at the exact deployment service endpoint with its explicit trust marker. Destructive module retirement drops database tables but does not remove backend blob objects."
), ),
layer="configured", layer="configured",
documentation_types=("admin",), documentation_types=("admin",),
@@ -538,8 +538,13 @@ manifest = ModuleManifest(
"FILE_STORAGE_BACKEND", "FILE_STORAGE_BACKEND",
"FILE_STORAGE_LOCAL_ROOT", "FILE_STORAGE_LOCAL_ROOT",
"FILE_STORAGE_LOCAL_FALLBACK_ROOTS", "FILE_STORAGE_LOCAL_FALLBACK_ROOTS",
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
"FILE_UPLOAD_MAX_BYTES", "FILE_UPLOAD_MAX_BYTES",
"FILE_UPLOAD_ZIP_MAX_BYTES", "FILE_UPLOAD_ZIP_MAX_BYTES",
"FILE_ARCHIVE_MAX_ENTRIES",
"FILE_ARCHIVE_MAX_EXPANDED_BYTES",
"FILE_ARCHIVE_MAX_EXPANSION_RATIO",
"FILE_ARCHIVE_PREVIEW_TTL_SECONDS",
"MASTER_KEY_B64", "MASTER_KEY_B64",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
"GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES", "GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES",
@@ -634,6 +639,10 @@ manifest = ModuleManifest(
"FILE_STORAGE_LOCAL_ROOT", "FILE_STORAGE_LOCAL_ROOT",
"FILE_UPLOAD_MAX_BYTES", "FILE_UPLOAD_MAX_BYTES",
"FILE_UPLOAD_ZIP_MAX_BYTES", "FILE_UPLOAD_ZIP_MAX_BYTES",
"FILE_ARCHIVE_MAX_ENTRIES",
"FILE_ARCHIVE_MAX_EXPANDED_BYTES",
"FILE_ARCHIVE_MAX_EXPANSION_RATIO",
"FILE_ARCHIVE_PREVIEW_TTL_SECONDS",
"MASTER_KEY_B64", "MASTER_KEY_B64",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
), ),
@@ -651,8 +660,8 @@ manifest = ModuleManifest(
"Compare the process requirements with the handbook's implemented/planned boundary and record every unsupported requirement or compensating control.", "Compare the process requirements with the handbook's implemented/planned boundary and record every unsupported requirement or compensating control.",
"Confirm version alignment, apply migrations on clean and upgrade databases, and run the repository plus meta-repository security and static-analysis gates.", "Confirm version alignment, apply migrations on clean and upgrade databases, and run the repository plus meta-repository security and static-analysis gates.",
"Exercise allowed and denied personal, group, and share access with representative accounts.", "Exercise allowed and denied personal, group, and share access with representative accounts.",
"Exercise bounded upload and ZIP handling, every required conflict strategy, organization, download, and soft deletion.", "Exercise bounded archive preview and confirmation, every required conflict strategy, organization, download, and soft deletion.",
"Where connectors are configured, verify policy explanation and one pinned HTTP provider; verify live S3 and SMB access still fails closed.", "Where connectors are configured, verify policy explanation and one pinned HTTP provider; verify installer-owned Garage when selected, and verify arbitrary external S3 and SMB access still fails closed.",
"Restore a coordinated database/blob/key backup and compare representative downloaded bytes with their recorded SHA-256 checksums.", "Restore a coordinated database/blob/key backup and compare representative downloaded bytes with their recorded SHA-256 checksums.",
"Record the tested versions, results, known limitations, evidence locations, residual risks, owner, and approval decision.", "Record the tested versions, results, known limitations, evidence locations, residual risks, owner, and approval decision.",
], ],
+275 -2
View File
@@ -1,12 +1,22 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import hmac
import json import json
from datetime import UTC, datetime, timedelta
from typing import Literal from typing import Literal
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_core.security.secrets import (
TransientPayloadError,
open_transient_payload,
seal_transient_payload,
)
from govoplan_files.backend.schemas import ( from govoplan_files.backend.schemas import (
ArchiveEntryResponse,
ArchivePreviewResponse,
ConflictResolutionRequest, ConflictResolutionRequest,
FileUploadResponse, FileUploadResponse,
_conflict_resolutions, _conflict_resolutions,
@@ -14,8 +24,16 @@ from govoplan_files.backend.schemas import (
from govoplan_files.backend.db.models import FileAsset from govoplan_files.backend.db.models import FileAsset
from govoplan_core.db.session import get_session from govoplan_core.db.session import get_session
from govoplan_files.backend.runtime import settings from govoplan_files.backend.runtime import settings
from govoplan_files.backend.storage.paths import UnsafeFilePathError from govoplan_files.backend.storage.paths import (
from govoplan_files.backend.storage.archives import extract_zip_upload UnsafeFilePathError,
normalize_folder,
)
from govoplan_files.backend.storage.archives import (
archive_format_for_filename,
extract_archive_upload,
extract_zip_upload,
inspect_archive,
)
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_policy import ( from govoplan_files.backend.storage.connector_policy import (
ConnectorPolicyDenied, ConnectorPolicyDenied,
@@ -39,6 +57,261 @@ from govoplan_files.backend.route_support import (
) )
router = APIRouter(prefix="/files", tags=["files"]) router = APIRouter(prefix="/files", tags=["files"])
_ARCHIVE_PREVIEW_PURPOSE = "files.archive-preview.v1"
def _archive_suffix(filename: str) -> str:
lowered = filename.casefold()
for suffix in (".tar.bz2", ".tar.gz", ".tar.xz", ".tbz2", ".tgz", ".txz", ".tar", ".zip"):
if lowered.endswith(suffix):
return suffix
return ".archive"
def _archive_sha256(path: str) -> str:
digest = hashlib.sha256()
with open(path, "rb") as source:
while chunk := source.read(1024 * 1024):
digest.update(chunk)
return digest.hexdigest()
def _archive_selected_paths(value: str) -> list[str]:
parsed = json.loads(value)
if not isinstance(parsed, list) or not parsed:
raise FileStorageError("Select at least one archive file or folder")
if len(parsed) > settings.file_archive_max_entries:
raise FileStorageError(
"Archive selection exceeds the configured entry limit"
)
selected: list[str] = []
for item in parsed:
if not isinstance(item, str) or not item.strip():
raise FileStorageError("Archive selection contains an invalid path")
if len(item) > 4096:
raise FileStorageError("Archive selection path is too long")
selected.append(item)
return selected
def _validate_archive_preview_token(
token: str,
*,
tenant_id: str,
user_id: str,
owner_type: str,
owner_id: str,
path: str,
campaign_id: str | None,
archive_format: str,
archive_sha256: str,
) -> None:
payload = open_transient_payload(
token,
ttl_seconds=settings.file_archive_preview_ttl_seconds,
)
expected = {
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
"tenant_id": tenant_id,
"user_id": user_id,
"owner_type": owner_type,
"owner_id": owner_id,
"path": path,
"campaign_id": campaign_id or "",
"archive_format": archive_format,
}
if any(payload.get(key) != value for key, value in expected.items()):
raise FileStorageError(
"Archive preview does not match this upload destination"
)
token_digest = payload.get("archive_sha256")
if not isinstance(token_digest, str) or not hmac.compare_digest(
token_digest, archive_sha256
):
raise FileStorageError(
"Archive contents changed after preview; preview it again"
)
@router.post("/archive-preview", response_model=ArchivePreviewResponse)
def preview_archive_upload(
file: UploadFile = FastAPIFile(...),
owner_type: Literal["user", "group"] = Form(default="user"),
owner_id: str | None = Form(default=None),
path: str = Form(default=""),
campaign_id: str | None = Form(default=None),
password: str | None = Form(default=None),
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
):
target_owner = owner_id or principal.user.id
archive_path: str | None = None
filename = file.filename or "archive"
try:
archive_format = archive_format_for_filename(filename)
archive_path = _spool_limited_upload_to_temp(
file,
max_bytes=settings.file_upload_zip_max_bytes,
suffix=_archive_suffix(filename),
)
inspection = inspect_archive(
archive_path,
filename=filename,
password=password,
max_entries=settings.file_archive_max_entries,
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
)
digest = _archive_sha256(archive_path)
normalized_path = normalize_folder(path)
preview_token = seal_transient_payload(
{
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
"tenant_id": principal.tenant_id,
"user_id": principal.user.id,
"owner_type": owner_type,
"owner_id": target_owner,
"path": normalized_path,
"campaign_id": campaign_id or "",
"archive_format": archive_format,
"archive_sha256": digest,
}
)
expires_at = datetime.now(UTC) + timedelta(
seconds=settings.file_archive_preview_ttl_seconds
)
return ArchivePreviewResponse(
preview_token=preview_token,
archive_format=inspection.archive_format,
entries=[
ArchiveEntryResponse(
path=entry.path,
kind=entry.kind,
size_bytes=entry.size_bytes,
compressed_size_bytes=entry.compressed_size_bytes,
encrypted=entry.encrypted,
)
for entry in inspection.entries
],
file_count=inspection.file_count,
directory_count=inspection.directory_count,
expanded_size_bytes=inspection.expanded_size_bytes,
compressed_size_bytes=inspection.compressed_size_bytes,
requires_password=inspection.requires_password,
password_verified=inspection.password_verified,
expires_at=expires_at.isoformat(),
)
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
raise _http_error(exc) from exc
finally:
if archive_path:
_cleanup_temp_file(archive_path)
@router.post("/archive-confirm", response_model=FileUploadResponse)
def confirm_archive_upload(
file: UploadFile = FastAPIFile(...),
preview_token: str = Form(...),
selected_paths_json: str = Form(...),
owner_type: Literal["user", "group"] = Form(default="user"),
owner_id: str | None = Form(default=None),
path: str = Form(default=""),
campaign_id: str | None = Form(default=None),
password: str | None = Form(default=None),
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
default="reject"
),
conflict_resolutions_json: str | None = Form(default=None),
source_provenance_json: str | None = Form(default=None),
source_revision: str | None = Form(default=None),
connector_policy_json: str | None = Form(default=None),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
):
target_owner = owner_id or principal.user.id
archive_path: str | None = None
filename = file.filename or "archive"
try:
raw_resolutions = (
json.loads(conflict_resolutions_json)
if conflict_resolutions_json
else []
)
upload_resolutions = _conflict_resolutions(
[ConflictResolutionRequest(**item) for item in raw_resolutions]
)
selected_paths = _archive_selected_paths(selected_paths_json)
_enforce_connector_policy(
source_provenance_json,
connector_policy_json,
operation="import",
)
metadata = _source_metadata_from_form(
source_provenance_json, source_revision
)
archive_format = archive_format_for_filename(filename)
normalized_path = normalize_folder(path)
archive_path = _spool_limited_upload_to_temp(
file,
max_bytes=settings.file_upload_zip_max_bytes,
suffix=_archive_suffix(filename),
)
digest = _archive_sha256(archive_path)
_validate_archive_preview_token(
preview_token,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=target_owner,
path=normalized_path,
campaign_id=campaign_id,
archive_format=archive_format,
archive_sha256=digest,
)
extracted = extract_archive_upload(
session,
tenant_id=principal.tenant_id,
owner_type=owner_type,
owner_id=target_owner,
user_id=principal.user.id,
archive_data=archive_path,
filename=filename,
folder=normalized_path,
campaign_id=campaign_id,
selected_paths=selected_paths,
password=password,
conflict_strategy=conflict_strategy,
conflict_resolutions=upload_resolutions,
metadata=metadata,
is_admin=_is_admin(principal),
max_entries=settings.file_archive_max_entries,
max_file_bytes=settings.file_upload_max_bytes,
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
)
uploaded_assets = [item.asset for item in extracted]
_audit_connector_imports(session, principal, uploaded_assets)
session.commit()
return FileUploadResponse(
files=[
_asset_response(session, asset, include_shares=True)
for asset in uploaded_assets
]
)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (
FileStorageError,
TransientPayloadError,
UnsafeFilePathError,
ValueError,
json.JSONDecodeError,
) as exc:
session.rollback()
raise _http_error(exc) from exc
finally:
if archive_path:
_cleanup_temp_file(archive_path)
@router.post("/upload", response_model=FileUploadResponse) @router.post("/upload", response_model=FileUploadResponse)
+21
View File
@@ -249,6 +249,27 @@ class FileUploadResponse(BaseModel):
files: list[FileAssetResponse] files: list[FileAssetResponse]
class ArchiveEntryResponse(BaseModel):
path: str
kind: Literal["file", "directory"]
size_bytes: int
compressed_size_bytes: int | None = None
encrypted: bool = False
class ArchivePreviewResponse(BaseModel):
preview_token: str
archive_format: str
entries: list[ArchiveEntryResponse] = Field(default_factory=list)
file_count: int
directory_count: int
expanded_size_bytes: int
compressed_size_bytes: int
requires_password: bool
password_verified: bool
expires_at: str
class FileConnectorPolicySource(BaseModel): class FileConnectorPolicySource(BaseModel):
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system" scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
scope_id: str | None = None scope_id: str | None = None
+602 -77
View File
@@ -1,55 +1,91 @@
from __future__ import annotations from __future__ import annotations
import mimetypes import mimetypes
import re
import stat
import tarfile
import zipfile import zipfile
from dataclasses import dataclass
from io import BytesIO from io import BytesIO
from os import PathLike from os import PathLike
from pathlib import Path from pathlib import Path, PurePosixPath
from typing import Any, Iterable from typing import Any, BinaryIO, Iterable, Literal
import pyzipper
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from govoplan_files.backend.db.models import FileAsset from govoplan_files.backend.db.models import FileAsset
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile from govoplan_files.backend.storage.backends import (
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend StorageBackendError,
from govoplan_files.backend.storage.files import create_file_asset, current_versions_and_blobs get_storage_backend,
from govoplan_files.backend.storage.paths import filename_from_path, normalize_folder, normalize_logical_path )
from govoplan_files.backend.storage.common import (
FileConflictResolution,
FileStorageError,
UploadedStoredFile,
)
from govoplan_files.backend.storage.files import (
create_file_asset,
current_versions_and_blobs,
)
from govoplan_files.backend.storage.paths import (
filename_from_path,
normalize_folder,
normalize_logical_path,
)
_ZIP_READ_CHUNK_SIZE = 1024 * 1024 _ARCHIVE_READ_CHUNK_SIZE = 1024 * 1024
ZIP_UPLOAD_MAX_FILES = 1000 _WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:")
ARCHIVE_UPLOAD_MAX_ENTRIES = 10_000
# Kept for callers and documentation using the former ZIP-specific name.
ZIP_UPLOAD_MAX_FILES = ARCHIVE_UPLOAD_MAX_ENTRIES
SUPPORTED_ARCHIVE_SUFFIXES = (
".tar.bz2",
".tar.gz",
".tar.xz",
".tbz2",
".tgz",
".txz",
".tar",
".zip",
)
def _read_zip_member( class ArchivePasswordError(FileStorageError):
archive: zipfile.ZipFile, pass
info: zipfile.ZipInfo,
*,
max_file_bytes: int,
max_total_bytes: int,
current_total: int,
) -> tuple[bytes, int]:
parts: list[bytes] = []
actual_size = 0
with archive.open(info) as source:
while True:
read_size = min(_ZIP_READ_CHUNK_SIZE, max_file_bytes + 1 - actual_size)
chunk = source.read(read_size)
if not chunk:
break
actual_size += len(chunk)
if actual_size > max_file_bytes:
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
if current_total + actual_size > max_total_bytes:
raise FileStorageError("ZIP is too large after extraction")
parts.append(chunk)
return b"".join(parts), current_total + actual_size
def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path: str | Path) -> None: @dataclass(frozen=True, slots=True)
class ArchiveEntry:
path: str
kind: Literal["file", "directory"]
size_bytes: int
compressed_size_bytes: int | None = None
encrypted: bool = False
@dataclass(frozen=True, slots=True)
class ArchiveInspection:
archive_format: str
entries: tuple[ArchiveEntry, ...]
file_count: int
directory_count: int
expanded_size_bytes: int
compressed_size_bytes: int
requires_password: bool
password_verified: bool
def create_zip_file(
session: Session, assets: Iterable[FileAsset], output_path: str | Path
) -> None:
backend = get_storage_backend() backend = get_storage_backend()
asset_list = list(assets) asset_list = list(assets)
version_blobs = current_versions_and_blobs(session, asset_list) version_blobs = current_versions_and_blobs(session, asset_list)
with zipfile.ZipFile(output_path, mode="w", compression=zipfile.ZIP_DEFLATED) as archive: with zipfile.ZipFile(
output_path, mode="w", compression=zipfile.ZIP_DEFLATED
) as archive:
for asset in asset_list: for asset in asset_list:
_version, blob = version_blobs[asset.id] _version, blob = version_blobs[asset.id]
info = zipfile.ZipInfo(asset.display_path) info = zipfile.ZipInfo(asset.display_path)
@@ -63,6 +99,145 @@ def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path:
raise FileStorageError(str(exc)) from exc raise FileStorageError(str(exc)) from exc
def archive_format_for_filename(filename: str) -> str:
lowered = filename.strip().casefold()
if lowered.endswith(".zip"):
return "zip"
if lowered.endswith((".tar.gz", ".tgz")):
return "tar.gz"
if lowered.endswith((".tar.bz2", ".tbz2")):
return "tar.bz2"
if lowered.endswith((".tar.xz", ".txz")):
return "tar.xz"
if lowered.endswith(".tar"):
return "tar"
raise FileStorageError(
"Unsupported archive format. Use ZIP, TAR, TAR.GZ, TAR.BZ2, or TAR.XZ."
)
def is_supported_archive_filename(filename: str) -> bool:
try:
archive_format_for_filename(filename)
except FileStorageError:
return False
return True
def inspect_archive(
archive_data: bytes | str | PathLike[str],
*,
filename: str,
password: str | None = None,
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
max_expansion_ratio: int = 100,
) -> ArchiveInspection:
archive_format = archive_format_for_filename(filename)
compressed_size = _archive_size(archive_data)
if archive_format == "zip":
entries, requires_password, password_verified = _inspect_zip(
archive_data,
password=password,
)
else:
entries = _inspect_tar(archive_data)
requires_password = False
password_verified = True
_validate_archive_limits(
entries,
compressed_size=compressed_size,
max_entries=max_entries,
max_expanded_bytes=max_expanded_bytes,
max_expansion_ratio=max_expansion_ratio,
)
complete_entries = _with_derived_directories(entries)
return ArchiveInspection(
archive_format=archive_format,
entries=tuple(complete_entries),
file_count=sum(entry.kind == "file" for entry in complete_entries),
directory_count=sum(
entry.kind == "directory" for entry in complete_entries
),
expanded_size_bytes=sum(
entry.size_bytes for entry in entries if entry.kind == "file"
),
compressed_size_bytes=compressed_size,
requires_password=requires_password,
password_verified=password_verified,
)
def extract_archive_upload(
session: Session,
*,
tenant_id: str,
owner_type: str,
owner_id: str,
user_id: str,
archive_data: bytes | str | PathLike[str],
filename: str,
folder: str | None,
campaign_id: str | None,
selected_paths: Iterable[str] | None = None,
password: str | None = None,
conflict_strategy: str = "reject",
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
metadata: dict[str, Any] | None = None,
is_admin: bool = False,
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
max_file_bytes: int = 50 * 1024 * 1024,
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
max_expansion_ratio: int = 100,
) -> list[UploadedStoredFile]:
inspection = inspect_archive(
archive_data,
filename=filename,
password=password,
max_entries=max_entries,
max_expanded_bytes=max_expanded_bytes,
max_expansion_ratio=max_expansion_ratio,
)
if inspection.requires_password and not inspection.password_verified:
raise ArchivePasswordError("Archive password is required")
selected_files = _selected_file_paths(inspection.entries, selected_paths)
if not selected_files:
raise FileStorageError("Select at least one archive file to import")
actual_total_limit = min(
max_expanded_bytes,
inspection.compressed_size_bytes * max_expansion_ratio,
)
if inspection.archive_format == "zip":
members = _read_selected_zip_members(
archive_data,
selected_files=selected_files,
password=password,
max_file_bytes=max_file_bytes,
max_total_bytes=actual_total_limit,
)
else:
members = _read_selected_tar_members(
archive_data,
selected_files=selected_files,
max_file_bytes=max_file_bytes,
max_total_bytes=actual_total_limit,
)
return _store_archive_members(
session,
members=members,
tenant_id=tenant_id,
owner_type=owner_type,
owner_id=owner_id,
user_id=user_id,
folder=folder,
campaign_id=campaign_id,
conflict_strategy=conflict_strategy,
conflict_resolutions=conflict_resolutions,
metadata=metadata,
is_admin=is_admin,
)
def extract_zip_upload( def extract_zip_upload(
session: Session, session: Session,
*, *,
@@ -81,51 +256,401 @@ def extract_zip_upload(
max_file_bytes: int = 50 * 1024 * 1024, max_file_bytes: int = 50 * 1024 * 1024,
max_total_bytes: int = 250 * 1024 * 1024, max_total_bytes: int = 250 * 1024 * 1024,
) -> list[UploadedStoredFile]: ) -> list[UploadedStoredFile]:
uploaded: list[UploadedStoredFile] = [] """Backward-compatible wrapper for the original immediate ZIP endpoint."""
total = 0
base_folder = normalize_folder(folder) return extract_archive_upload(
session,
tenant_id=tenant_id,
owner_type=owner_type,
owner_id=owner_id,
user_id=user_id,
archive_data=zip_data,
filename="archive.zip",
folder=folder,
campaign_id=campaign_id,
conflict_strategy=conflict_strategy,
conflict_resolutions=conflict_resolutions,
metadata=metadata,
is_admin=is_admin,
max_entries=max_files,
max_file_bytes=max_file_bytes,
max_expanded_bytes=max_total_bytes,
)
def _inspect_zip(
archive_data: bytes | str | PathLike[str],
*,
password: str | None,
) -> tuple[list[ArchiveEntry], bool, bool]:
try: try:
source = BytesIO(zip_data) if isinstance(zip_data, bytes) else zip_data with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
with zipfile.ZipFile(source) as archive: infos = archive.infolist()
infos = [info for info in archive.infolist() if not info.is_dir()] entries = [_zip_entry(info) for info in infos]
if len(infos) > max_files: encrypted_info = next(
raise FileStorageError(f"ZIP contains too many files (limit {max_files})") (
for info in infos: info
if info.flag_bits & 0x1: for info in infos
raise FileStorageError("Encrypted ZIP uploads are not supported") if not info.is_dir() and bool(info.flag_bits & 0x1)
if info.file_size < 0: ),
raise FileStorageError("Invalid ZIP member") None,
if info.file_size > max_file_bytes: )
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit") if encrypted_info is None:
if total + info.file_size > max_total_bytes: return entries, False, True
raise FileStorageError("ZIP is too large after extraction") if not password:
inner_path = normalize_logical_path(info.filename) return entries, True, False
target_path = f"{base_folder}/{inner_path}" if base_folder else inner_path try:
data, total = _read_zip_member( with archive.open(
archive, encrypted_info, pwd=password.encode("utf-8")
info, ) as member:
max_file_bytes=max_file_bytes, member.read(1)
max_total_bytes=max_total_bytes, except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
current_total=total, raise ArchivePasswordError("Archive password is incorrect") from exc
) return entries, True, True
uploaded.append( except ArchivePasswordError:
create_file_asset( raise
session, except (OSError, ValueError, zipfile.BadZipFile) as exc:
tenant_id=tenant_id, raise FileStorageError("Invalid ZIP upload") from exc
owner_type=owner_type,
owner_id=owner_id,
user_id=user_id, def _inspect_tar(
filename=filename_from_path(inner_path), archive_data: bytes | str | PathLike[str],
data=data, ) -> list[ArchiveEntry]:
display_path=target_path, try:
content_type=mimetypes.guess_type(inner_path)[0] or "application/octet-stream", with _open_tar(archive_data) as archive:
metadata=metadata, entries: list[ArchiveEntry] = []
campaign_id=campaign_id, for member in archive.getmembers():
conflict_strategy=conflict_strategy, path = _safe_member_path(member.name)
conflict_resolutions=conflict_resolutions, if member.isdir():
is_admin=is_admin, entries.append(
ArchiveEntry(path=path, kind="directory", size_bytes=0)
)
continue
if not member.isfile():
raise FileStorageError(
f"Archive member {member.name!r} is not a regular file or directory"
)
entries.append(
ArchiveEntry(
path=path,
kind="file",
size_bytes=max(0, int(member.size)),
) )
) )
except zipfile.BadZipFile as exc: return entries
raise FileStorageError("Invalid ZIP upload") from exc except FileStorageError:
raise
except (OSError, tarfile.TarError) as exc:
raise FileStorageError("Invalid TAR upload") from exc
def _zip_entry(info: zipfile.ZipInfo) -> ArchiveEntry:
path = _safe_member_path(info.filename)
unix_mode = (info.external_attr >> 16) & 0xFFFF
file_type = stat.S_IFMT(unix_mode)
if file_type and not (
stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)
):
raise FileStorageError(
f"Archive member {info.filename!r} is not a regular file or directory"
)
if info.file_size < 0 or info.compress_size < 0:
raise FileStorageError(f"Archive member {info.filename!r} has invalid size metadata")
return ArchiveEntry(
path=path,
kind="directory" if info.is_dir() else "file",
size_bytes=0 if info.is_dir() else int(info.file_size),
compressed_size_bytes=(
None if info.is_dir() else int(info.compress_size)
),
encrypted=bool(info.flag_bits & 0x1),
)
def _validate_archive_limits(
entries: list[ArchiveEntry],
*,
compressed_size: int,
max_entries: int,
max_expanded_bytes: int,
max_expansion_ratio: int,
) -> None:
if len(entries) > max_entries:
raise FileStorageError(
f"Archive contains too many entries (limit {max_entries})"
)
seen: dict[str, str] = {}
expanded_size = 0
for entry in entries:
previous_kind = seen.get(entry.path)
if previous_kind is not None:
raise FileStorageError(
f"Archive contains duplicate path {entry.path!r}"
)
seen[entry.path] = entry.kind
if entry.kind == "file":
expanded_size += entry.size_bytes
if expanded_size > max_expanded_bytes:
raise FileStorageError(
"Archive is too large after extraction "
f"(limit {max_expanded_bytes} bytes)"
)
if expanded_size and (
compressed_size <= 0
or expanded_size > compressed_size * max_expansion_ratio
):
raise FileStorageError(
"Archive expansion ratio exceeds "
f"{max_expansion_ratio}:1"
)
def _with_derived_directories(
entries: list[ArchiveEntry],
) -> list[ArchiveEntry]:
by_path = {entry.path: entry for entry in entries}
for entry in entries:
parent = PurePosixPath(entry.path).parent
while str(parent) not in {"", "."}:
path = str(parent)
existing = by_path.get(path)
if existing and existing.kind != "directory":
raise FileStorageError(
f"Archive path {path!r} is both a file and a directory"
)
by_path.setdefault(
path,
ArchiveEntry(path=path, kind="directory", size_bytes=0),
)
parent = parent.parent
return sorted(
by_path.values(),
key=lambda entry: (
tuple(entry.path.casefold().split("/")),
entry.kind != "directory",
),
)
def _selected_file_paths(
entries: tuple[ArchiveEntry, ...],
selected_paths: Iterable[str] | None,
) -> set[str]:
file_paths = {entry.path for entry in entries if entry.kind == "file"}
if selected_paths is None:
return file_paths
known = {entry.path: entry for entry in entries}
normalized = {_safe_member_path(path) for path in selected_paths}
unknown = normalized - known.keys()
if unknown:
raise FileStorageError(
f"Archive selection contains unknown path {sorted(unknown)[0]!r}"
)
selected_files: set[str] = set()
for path in normalized:
entry = known[path]
if entry.kind == "file":
selected_files.add(path)
continue
prefix = f"{path}/"
selected_files.update(
file_path
for file_path in file_paths
if file_path.startswith(prefix)
)
return selected_files
def _read_selected_zip_members(
archive_data: bytes | str | PathLike[str],
*,
selected_files: set[str],
password: str | None,
max_file_bytes: int,
max_total_bytes: int,
) -> list[tuple[str, bytes]]:
result: list[tuple[str, bytes]] = []
total = 0
try:
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
for info in archive.infolist():
if info.is_dir():
continue
path = _safe_member_path(info.filename)
if path not in selected_files:
continue
pwd = password.encode("utf-8") if password else None
try:
with archive.open(info, pwd=pwd) as source:
data, total = _read_member(
source,
path=path,
max_file_bytes=max_file_bytes,
max_total_bytes=max_total_bytes,
current_total=total,
)
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
if info.flag_bits & 0x1:
raise ArchivePasswordError(
"Archive password is incorrect"
) from exc
raise
result.append((path, data))
except (FileStorageError, ArchivePasswordError):
raise
except (OSError, ValueError, zipfile.BadZipFile) as exc:
raise FileStorageError("ZIP extraction failed") from exc
return result
def _read_selected_tar_members(
archive_data: bytes | str | PathLike[str],
*,
selected_files: set[str],
max_file_bytes: int,
max_total_bytes: int,
) -> list[tuple[str, bytes]]:
result: list[tuple[str, bytes]] = []
total = 0
try:
with _open_tar(archive_data) as archive:
for member in archive.getmembers():
if not member.isfile():
continue
path = _safe_member_path(member.name)
if path not in selected_files:
continue
source = archive.extractfile(member)
if source is None:
raise FileStorageError(
f"Archive member {path!r} could not be read"
)
with source:
data, total = _read_member(
source,
path=path,
max_file_bytes=max_file_bytes,
max_total_bytes=max_total_bytes,
current_total=total,
)
result.append((path, data))
except FileStorageError:
raise
except (OSError, tarfile.TarError) as exc:
raise FileStorageError("TAR extraction failed") from exc
return result
def _read_member(
source: BinaryIO,
*,
path: str,
max_file_bytes: int,
max_total_bytes: int,
current_total: int,
) -> tuple[bytes, int]:
parts: list[bytes] = []
actual_size = 0
while True:
read_size = min(
_ARCHIVE_READ_CHUNK_SIZE,
max_file_bytes + 1 - actual_size,
)
chunk = source.read(read_size)
if not chunk:
break
actual_size += len(chunk)
if actual_size > max_file_bytes:
raise FileStorageError(
f"Archive member {path!r} exceeds per-file limit"
)
if current_total + actual_size > max_total_bytes:
raise FileStorageError("Archive is too large after extraction")
parts.append(chunk)
return b"".join(parts), current_total + actual_size
def _store_archive_members(
session: Session,
*,
members: Iterable[tuple[str, bytes]],
tenant_id: str,
owner_type: str,
owner_id: str,
user_id: str,
folder: str | None,
campaign_id: str | None,
conflict_strategy: str,
conflict_resolutions: Iterable[FileConflictResolution] | None,
metadata: dict[str, Any] | None,
is_admin: bool,
) -> list[UploadedStoredFile]:
uploaded: list[UploadedStoredFile] = []
base_folder = normalize_folder(folder)
for inner_path, data in members:
target_path = (
f"{base_folder}/{inner_path}" if base_folder else inner_path
)
uploaded.append(
create_file_asset(
session,
tenant_id=tenant_id,
owner_type=owner_type,
owner_id=owner_id,
user_id=user_id,
filename=filename_from_path(inner_path),
data=data,
display_path=target_path,
content_type=mimetypes.guess_type(inner_path)[0]
or "application/octet-stream",
metadata=metadata,
campaign_id=campaign_id,
conflict_strategy=conflict_strategy,
conflict_resolutions=conflict_resolutions,
is_admin=is_admin,
)
)
return uploaded return uploaded
def _safe_member_path(value: str) -> str:
raw = str(value or "").replace("\\", "/").strip()
if (
not raw
or "\x00" in raw
or raw.startswith("/")
or _WINDOWS_DRIVE_RE.match(raw)
):
raise FileStorageError(f"Unsafe archive member path {value!r}")
if any(part == ".." for part in raw.split("/")):
raise FileStorageError(f"Unsafe archive member path {value!r}")
try:
return normalize_logical_path(raw.rstrip("/"))
except ValueError as exc:
raise FileStorageError(f"Unsafe archive member path {value!r}") from exc
def _archive_size(archive_data: bytes | str | PathLike[str]) -> int:
if isinstance(archive_data, bytes):
return len(archive_data)
try:
return Path(archive_data).stat().st_size
except OSError as exc:
raise FileStorageError("Archive upload could not be read") from exc
def _archive_source(
archive_data: bytes | str | PathLike[str],
) -> BytesIO | str | PathLike[str]:
return BytesIO(archive_data) if isinstance(archive_data, bytes) else archive_data
def _open_tar(
archive_data: bytes | str | PathLike[str],
) -> tarfile.TarFile:
if isinstance(archive_data, bytes):
return tarfile.open(fileobj=BytesIO(archive_data), mode="r:*")
try:
return tarfile.open(name=archive_data, mode="r:*")
except OSError as exc:
raise FileStorageError("Archive upload could not be read") from exc
+37 -14
View File
@@ -145,28 +145,35 @@ class S3StorageBackend:
region_name: str region_name: str
access_key_id: str access_key_id: str
secret_access_key: str secret_access_key: str
deployment_managed: bool = False
name: str = "s3" name: str = "s3"
@property @property
def client(self): def client(self):
try: if self.deployment_managed:
endpoint_url = validate_unpinned_sdk_http_url( endpoint_url = _deployment_managed_garage_endpoint(self.endpoint_url)
self.endpoint_url, else:
label="File storage S3 endpoint", try:
) endpoint_url = validate_unpinned_sdk_http_url(
except OutboundHttpError as exc: self.endpoint_url,
raise StorageBackendError(str(exc)) from exc label="File storage S3 endpoint",
)
except OutboundHttpError as exc:
raise StorageBackendError(str(exc)) from exc
try: try:
import boto3 import boto3
from botocore.config import Config
except ModuleNotFoundError as exc: except ModuleNotFoundError as exc:
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
return boto3.client( options: dict[str, object] = {
"s3", "endpoint_url": endpoint_url,
endpoint_url=endpoint_url, "region_name": self.region_name,
region_name=self.region_name, "aws_access_key_id": self.access_key_id,
aws_access_key_id=self.access_key_id, "aws_secret_access_key": self.secret_access_key,
aws_secret_access_key=self.secret_access_key, }
) if self.deployment_managed:
options["config"] = Config(s3={"addressing_style": "path"})
return boto3.client("s3", **options)
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None: def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
max_bytes = response_limit("file") max_bytes = response_limit("file")
@@ -303,6 +310,15 @@ def _s3_missing_error(exc: Exception) -> bool:
return code in {"404", "NoSuchKey", "NotFound"} or status_code == 404 return code in {"404", "NoSuchKey", "NotFound"} or status_code == 404
def _deployment_managed_garage_endpoint(value: str) -> str:
endpoint = str(value or "").strip()
if endpoint != "http://garage:3900":
raise StorageBackendError(
"Deployment-managed S3 trust is restricted to http://garage:3900"
)
return endpoint
def _fallback_roots() -> tuple[Path, ...]: def _fallback_roots() -> tuple[Path, ...]:
raw = getattr(settings, "file_storage_local_fallback_roots", "") or "" raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip()) return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())
@@ -319,5 +335,12 @@ def get_storage_backend() -> StorageBackend:
region_name=settings.file_storage_s3_region or settings.s3_region, region_name=settings.file_storage_s3_region or settings.s3_region,
access_key_id=settings.file_storage_s3_access_key_id or settings.s3_access_key_id, access_key_id=settings.file_storage_s3_access_key_id or settings.s3_access_key_id,
secret_access_key=settings.file_storage_s3_secret_access_key or settings.s3_secret_access_key, secret_access_key=settings.file_storage_s3_secret_access_key or settings.s3_secret_access_key,
deployment_managed=bool(
getattr(
settings,
"file_storage_s3_deployment_managed",
False,
)
),
) )
raise StorageBackendError(f"Unsupported file storage backend: {settings.file_storage_backend}") raise StorageBackendError(f"Unsupported file storage backend: {settings.file_storage_backend}")
@@ -14,7 +14,6 @@ from sqlalchemy.orm import Session
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
from govoplan_files.backend.storage.backends import StorageBackend, get_storage_backend from govoplan_files.backend.storage.backends import StorageBackend, get_storage_backend
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
from govoplan_files.backend.storage.access import ensure_owner_access from govoplan_files.backend.storage.access import ensure_owner_access
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
+101
View File
@@ -0,0 +1,101 @@
from __future__ import annotations
import io
import hashlib
import unittest
import zipfile
from types import SimpleNamespace
from unittest.mock import patch
from fastapi import UploadFile
from govoplan_files.backend.routes.uploads import (
_validate_archive_preview_token,
preview_archive_upload,
)
from govoplan_files.backend.storage.common import FileStorageError
def _archive_bytes() -> bytes:
target = io.BytesIO()
with zipfile.ZipFile(target, "w", compression=zipfile.ZIP_DEFLATED) as archive:
archive.writestr("folder/report.txt", b"report")
return target.getvalue()
class ArchivePreviewTokenTests(unittest.TestCase):
def setUp(self) -> None:
self.settings = SimpleNamespace(
file_upload_zip_max_bytes=10 * 1024 * 1024,
file_archive_max_entries=10_000,
file_archive_max_expanded_bytes=2 * 1024 * 1024 * 1024,
file_archive_max_expansion_ratio=100,
file_archive_preview_ttl_seconds=30 * 60,
)
self.principal = SimpleNamespace(
tenant_id="tenant-1",
user=SimpleNamespace(id="user-1"),
)
def test_preview_token_is_bound_to_actor_archive_and_destination(self) -> None:
archive_data = _archive_bytes()
upload = UploadFile(
file=io.BytesIO(archive_data),
filename="monthly.zip",
)
with patch(
"govoplan_files.backend.routes.uploads.settings",
self.settings,
):
preview = preview_archive_upload(
file=upload,
owner_type="user",
owner_id="user-1",
path="incoming",
campaign_id=None,
password=None,
principal=self.principal,
)
_validate_archive_preview_token(
preview.preview_token,
tenant_id="tenant-1",
user_id="user-1",
owner_type="user",
owner_id="user-1",
path="incoming",
campaign_id=None,
archive_format="zip",
archive_sha256=hashlib.sha256(archive_data).hexdigest(),
)
with self.assertRaisesRegex(
FileStorageError, "does not match this upload destination"
):
_validate_archive_preview_token(
preview.preview_token,
tenant_id="tenant-1",
user_id="user-1",
owner_type="user",
owner_id="user-1",
path="elsewhere",
campaign_id=None,
archive_format="zip",
archive_sha256=hashlib.sha256(archive_data).hexdigest(),
)
with self.assertRaisesRegex(
FileStorageError, "contents changed"
):
_validate_archive_preview_token(
preview.preview_token,
tenant_id="tenant-1",
user_id="user-1",
owner_type="user",
owner_id="user-1",
path="incoming",
campaign_id=None,
archive_format="zip",
archive_sha256="0" * 64,
)
if __name__ == "__main__":
unittest.main()
+170
View File
@@ -0,0 +1,170 @@
from __future__ import annotations
import io
import tarfile
import unittest
import zipfile
from types import SimpleNamespace
from unittest.mock import patch
import pyzipper
from govoplan_files.backend.storage.archives import (
ArchivePasswordError,
extract_archive_upload,
inspect_archive,
)
from govoplan_files.backend.storage.common import FileStorageError
def _zip_bytes(entries: dict[str, bytes]) -> bytes:
target = io.BytesIO()
with zipfile.ZipFile(target, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for path, data in entries.items():
archive.writestr(path, data)
return target.getvalue()
def _tar_bytes(entries: dict[str, bytes], mode: str = "w:gz") -> bytes:
target = io.BytesIO()
with tarfile.open(fileobj=target, mode=mode) as archive:
for path, data in entries.items():
info = tarfile.TarInfo(path)
info.size = len(data)
archive.addfile(info, io.BytesIO(data))
return target.getvalue()
def _encrypted_zip_bytes(password: str) -> bytes:
target = io.BytesIO()
with pyzipper.AESZipFile(
target,
"w",
compression=zipfile.ZIP_DEFLATED,
encryption=pyzipper.WZ_AES,
) as archive:
archive.setpassword(password.encode("utf-8"))
archive.writestr("secure/report.txt", b"classified")
return target.getvalue()
class ArchiveInspectionTests(unittest.TestCase):
def test_zip_preview_derives_folders_and_sizes(self) -> None:
inspection = inspect_archive(
_zip_bytes(
{
"department/one.txt": b"one",
"department/nested/two.csv": b"two",
}
),
filename="monthly.zip",
)
self.assertEqual("zip", inspection.archive_format)
self.assertEqual(2, inspection.file_count)
self.assertEqual(2, inspection.directory_count)
self.assertEqual(6, inspection.expanded_size_bytes)
self.assertEqual(
[
"department",
"department/nested",
"department/nested/two.csv",
"department/one.txt",
],
[entry.path for entry in inspection.entries],
)
def test_tar_compression_variants_are_inspected(self) -> None:
for filename, mode in (
("monthly.tar", "w"),
("monthly.tar.gz", "w:gz"),
("monthly.tar.bz2", "w:bz2"),
("monthly.tar.xz", "w:xz"),
):
with self.subTest(filename=filename):
inspection = inspect_archive(
_tar_bytes({"report.txt": b"report"}, mode=mode),
filename=filename,
)
self.assertEqual(1, inspection.file_count)
self.assertEqual(filename.removeprefix("monthly."), inspection.archive_format)
def test_unsafe_member_path_is_rejected(self) -> None:
with self.assertRaisesRegex(FileStorageError, "Unsafe archive member"):
inspect_archive(
_zip_bytes({"../escape.txt": b"no"}),
filename="unsafe.zip",
)
def test_archive_bomb_ratio_is_rejected(self) -> None:
with self.assertRaisesRegex(FileStorageError, "expansion ratio"):
inspect_archive(
_zip_bytes({"zeros.bin": b"\0" * 50_000}),
filename="bomb.zip",
max_expansion_ratio=2,
)
def test_encrypted_zip_reports_and_verifies_password(self) -> None:
archive = _encrypted_zip_bytes("correct horse")
missing = inspect_archive(archive, filename="secure.zip")
self.assertTrue(missing.requires_password)
self.assertFalse(missing.password_verified)
verified = inspect_archive(
archive,
filename="secure.zip",
password="correct horse",
)
self.assertTrue(verified.requires_password)
self.assertTrue(verified.password_verified)
with self.assertRaisesRegex(ArchivePasswordError, "incorrect"):
inspect_archive(
archive,
filename="secure.zip",
password="wrong",
)
def test_folder_selection_extracts_only_descendants(self) -> None:
archive = _zip_bytes(
{
"selected/one.txt": b"one",
"selected/two.txt": b"two",
"other/three.txt": b"three",
}
)
stored = SimpleNamespace(
asset=SimpleNamespace(id="asset"),
version=SimpleNamespace(id="version"),
blob=SimpleNamespace(id="blob"),
)
with patch(
"govoplan_files.backend.storage.archives.create_file_asset",
return_value=stored,
) as create_asset:
result = extract_archive_upload(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
owner_type="user",
owner_id="user-1",
user_id="user-1",
archive_data=archive,
filename="selection.zip",
folder="imports",
campaign_id=None,
selected_paths=["selected"],
)
self.assertEqual(2, len(result))
self.assertEqual(
{"imports/selected/one.txt", "imports/selected/two.txt"},
{
call.kwargs["display_path"]
for call in create_asset.call_args_list
},
)
if __name__ == "__main__":
unittest.main()
+9 -1
View File
@@ -49,6 +49,10 @@ class FilesRuntimeDocumentationTests(unittest.TestCase):
settings = SimpleNamespace( settings = SimpleNamespace(
file_upload_max_bytes=7 * 1024 * 1024, file_upload_max_bytes=7 * 1024 * 1024,
file_upload_zip_max_bytes=19 * 1024 * 1024, file_upload_zip_max_bytes=19 * 1024 * 1024,
file_archive_max_expanded_bytes=31 * 1024 * 1024,
file_archive_max_entries=321,
file_archive_max_expansion_ratio=42,
file_archive_preview_ttl_seconds=15 * 60,
) )
with patch( with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor", "govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
@@ -73,8 +77,12 @@ class FilesRuntimeDocumentationTests(unittest.TestCase):
archive = topics["files.workflow.upload-and-unpack-zip"] archive = topics["files.workflow.upload-and-unpack-zip"]
self.assertIn("19 MiB (19,922,944 bytes)", archive.body) self.assertIn("19 MiB (19,922,944 bytes)", archive.body)
self.assertIn("31 MiB (32,505,856 bytes)", archive.body)
self.assertIn("7 MiB (7,340,032 bytes)", archive.body) self.assertIn("7 MiB (7,340,032 bytes)", archive.body)
self.assertIn("1,000", archive.body) self.assertIn("321", archive.body)
self.assertIn("42:1", archive.body)
self.assertIn("15 minutes", archive.body)
self.assertIn("passwords remain request-only", archive.body)
self.assertIn("Actual extracted bytes are counted", archive.body) self.assertIn("Actual extracted bytes are counted", archive.body)
def test_connector_task_requires_authority_and_a_visible_usable_profile( def test_connector_task_requires_authority_and_a_visible_usable_profile(
+10 -1
View File
@@ -49,11 +49,20 @@ class FilesRouterContractTests(unittest.TestCase):
actual = self._operation_keys(router) actual = self._operation_keys(router)
self.assertEqual(expected, actual) self.assertEqual(expected, actual)
self.assertEqual(50, len(actual)) self.assertEqual(52, len(actual))
self.assertFalse( self.assertFalse(
[operation for operation, count in Counter(actual).items() if count > 1] [operation for operation, count in Counter(actual).items() if count > 1]
) )
def test_archive_preview_and_confirmation_routes_are_exposed(self) -> None:
routes = {
(tuple(sorted(route.methods or ())), route.path)
for route in router.routes
}
self.assertIn((("POST",), "/files/archive-preview"), routes)
self.assertIn((("POST",), "/files/archive-confirm"), routes)
def test_connector_routes_keep_existing_api_paths(self) -> None: def test_connector_routes_keep_existing_api_paths(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes} routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
+51 -2
View File
@@ -1,19 +1,26 @@
from __future__ import annotations from __future__ import annotations
import io import io
from types import ModuleType
import sys
import unittest import unittest
from unittest.mock import MagicMock, PropertyMock, patch from unittest.mock import MagicMock, PropertyMock, patch
from govoplan_files.backend.storage.backends import S3StorageBackend, StorageBackendError from govoplan_files.backend.storage.backends import S3StorageBackend, StorageBackendError
def _backend() -> S3StorageBackend: def _backend(
*,
endpoint_url: str = "https://objects.example.test",
deployment_managed: bool = False,
) -> S3StorageBackend:
return S3StorageBackend( return S3StorageBackend(
bucket="files", bucket="files",
endpoint_url="https://objects.example.test", endpoint_url=endpoint_url,
region_name="test", region_name="test",
access_key_id="access", access_key_id="access",
secret_access_key="secret", secret_access_key="secret",
deployment_managed=deployment_managed,
) )
@@ -32,6 +39,48 @@ class S3StorageBackendTests(unittest.TestCase):
), self.assertRaisesRegex(StorageBackendError, "until that transport supports.*DNS/IP pinning"): ), self.assertRaisesRegex(StorageBackendError, "until that transport supports.*DNS/IP pinning"):
_backend().client _backend().client
def test_installer_managed_garage_uses_exact_endpoint_and_path_style(self) -> None:
boto3 = ModuleType("boto3")
boto3.client = MagicMock(return_value=object())
botocore = ModuleType("botocore")
botocore.__path__ = []
botocore_config = ModuleType("botocore.config")
class Config:
def __init__(self, **values):
self.values = values
botocore_config.Config = Config
with patch.dict(
sys.modules,
{
"boto3": boto3,
"botocore": botocore,
"botocore.config": botocore_config,
},
):
_backend(
endpoint_url="http://garage:3900",
deployment_managed=True,
).client
_args, kwargs = boto3.client.call_args
self.assertEqual("http://garage:3900", kwargs["endpoint_url"])
self.assertEqual(
{"s3": {"addressing_style": "path"}},
kwargs["config"].values,
)
def test_installer_managed_garage_rejects_any_other_endpoint(self) -> None:
with self.assertRaisesRegex(
StorageBackendError,
"restricted to http://garage:3900",
):
_backend(
endpoint_url="http://other-s3:3900",
deployment_managed=True,
).client
def test_get_bytes_rejects_declared_oversize_object_without_reading(self) -> None: def test_get_bytes_rejects_declared_oversize_object_without_reading(self) -> None:
body = MagicMock() body = MagicMock()
client = MagicMock() client = MagicMock()
+5 -5
View File
@@ -18,12 +18,12 @@
"test:file-property-filters": "node scripts/test-file-property-filters-structure.mjs" "test:file-property-filters": "node scripts/test-file-property-filters-structure.mjs"
}, },
"peerDependencies": { "peerDependencies": {
"@vitejs/plugin-react": "^4.3.4", "@vitejs/plugin-react": "^5.2.0",
"vite": "^6.0.6", "vite": "^7.3.6",
"typescript": "^5.7.2", "typescript": "^5.7.2",
"react": "^19.0.0", "react": ">=19.2.7 <20",
"react-dom": "^19.0.0", "react-dom": ">=19.2.7 <20",
"react-router-dom": ">=7.18.2 <8", "react-router": ">=8.3.0 <9",
"lucide-react": "^1.23.0", "lucide-react": "^1.23.0",
"@govoplan/core-webui": "^0.1.9" "@govoplan/core-webui": "^0.1.9"
}, },
+87 -2
View File
@@ -328,6 +328,25 @@ export type ManagedFileSnapshotResponse = {files: ManagedFile[];folders: FileFol
export type FileSpacesResponse = {spaces: FileSpace[];}; export type FileSpacesResponse = {spaces: FileSpace[];};
export type FileUploadResponse = {files: ManagedFile[];}; export type FileUploadResponse = {files: ManagedFile[];};
export type FileUploadProgress = {loaded: number;total?: number;percentage: number | null;}; export type FileUploadProgress = {loaded: number;total?: number;percentage: number | null;};
export type ArchivePreviewEntry = {
path: string;
kind: "file" | "directory";
size_bytes: number;
compressed_size_bytes?: number | null;
encrypted: boolean;
};
export type ArchivePreviewResponse = {
preview_token: string;
archive_format: string;
entries: ArchivePreviewEntry[];
file_count: number;
directory_count: number;
expanded_size_bytes: number;
compressed_size_bytes: number;
requires_password: boolean;
password_verified: boolean;
expires_at: string;
};
export type FileConnectorFilePayload = { export type FileConnectorFilePayload = {
library_id: string; library_id: string;
path: string; path: string;
@@ -556,14 +575,80 @@ options: {
return apiFetch<FileUploadResponse>(settings, "/api/v1/files/upload", { method: "POST", body: form }); return apiFetch<FileUploadResponse>(settings, "/api/v1/files/upload", { method: "POST", body: form });
} }
export function previewArchiveUpload(
settings: ApiSettings,
file: File,
options: {
owner_type: "user" | "group";
owner_id: string;
path?: string;
campaign_id?: string;
password?: string;
})
: Promise<ArchivePreviewResponse> {
const form = new FormData();
form.append("file", file);
form.append("owner_type", options.owner_type);
form.append("owner_id", options.owner_id);
form.append("path", options.path ?? "");
if (options.campaign_id) form.append("campaign_id", options.campaign_id);
if (options.password) form.append("password", options.password);
return apiFetch<ArchivePreviewResponse>(settings, "/api/v1/files/archive-preview", { method: "POST", body: form });
}
export function confirmArchiveUpload(
settings: ApiSettings,
file: File,
options: {
preview_token: string;
selected_paths: string[];
owner_type: "user" | "group";
owner_id: string;
path?: string;
campaign_id?: string;
password?: string;
conflict_strategy?: ConflictStrategy;
conflict_resolutions?: ConflictResolution[];
source_provenance?: FileSourceProvenance;
source_revision?: string;
connector_policy_sources?: FileConnectorPolicySource[];
onProgress?: (progress: FileUploadProgress) => void;
})
: Promise<FileUploadResponse> {
const form = new FormData();
form.append("file", file);
form.append("preview_token", options.preview_token);
form.append("selected_paths_json", JSON.stringify(options.selected_paths));
form.append("owner_type", options.owner_type);
form.append("owner_id", options.owner_id);
form.append("path", options.path ?? "");
if (options.campaign_id) form.append("campaign_id", options.campaign_id);
if (options.password) form.append("password", options.password);
if (options.conflict_strategy) form.append("conflict_strategy", options.conflict_strategy);
if (options.conflict_resolutions?.length) form.append("conflict_resolutions_json", JSON.stringify(options.conflict_resolutions));
if (options.source_provenance) form.append("source_provenance_json", JSON.stringify(options.source_provenance));
if (options.source_revision) form.append("source_revision", options.source_revision);
if (options.connector_policy_sources?.length) form.append("connector_policy_json", JSON.stringify({ sources: options.connector_policy_sources }));
if (options.onProgress) {
return uploadFilesWithProgress(
settings,
form,
options.onProgress,
"/api/v1/files/archive-confirm"
);
}
return apiFetch<FileUploadResponse>(settings, "/api/v1/files/archive-confirm", { method: "POST", body: form });
}
function uploadFilesWithProgress( function uploadFilesWithProgress(
settings: ApiSettings, settings: ApiSettings,
form: FormData, form: FormData,
onProgress: (progress: FileUploadProgress) => void) onProgress: (progress: FileUploadProgress) => void,
endpoint = "/api/v1/files/upload")
: Promise<FileUploadResponse> { : Promise<FileUploadResponse> {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest(); const xhr = new XMLHttpRequest();
xhr.open("POST", apiUrl(settings, "/api/v1/files/upload")); xhr.open("POST", apiUrl(settings, endpoint));
xhr.withCredentials = true; xhr.withCredentials = true;
for (const [key, value] of authHeaders(settings)) xhr.setRequestHeader(key, value); for (const [key, value] of authHeaders(settings)) xhr.setRequestHeader(key, value);
const csrf = csrfToken(); const csrf = csrfToken();
@@ -1,6 +1,6 @@
import { useCallback } from "react"; import { useCallback } from "react";
import { Folder, Network } from "lucide-react"; import { Folder, Network } from "lucide-react";
import { Link } from "react-router-dom"; import { Link } from "react-router";
import { import {
DashboardWidgetList, DashboardWidgetList,
DismissibleAlert, DismissibleAlert,
+310 -16
View File
@@ -8,6 +8,7 @@ import {
FileDropZone, FileDropZone,
FormField, FormField,
LoadingIndicator, LoadingIndicator,
PasswordField,
ResourceAccessExplanation, ResourceAccessExplanation,
ToggleSwitch, ToggleSwitch,
hasScope, hasScope,
@@ -20,6 +21,7 @@ import {
browseFileConnectorProfile, browseFileConnectorProfile,
createFileConnectorSpace, createFileConnectorSpace,
createFolder, createFolder,
confirmArchiveUpload,
deleteFolder, deleteFolder,
downloadFile, downloadFile,
downloadFilesAsZip, downloadFilesAsZip,
@@ -29,11 +31,14 @@ import {
listFileConnectorProfiles, listFileConnectorProfiles,
listFileSpaces, listFileSpaces,
listManagedFileSnapshot, listManagedFileSnapshot,
previewArchiveUpload,
resolveFilePatterns, resolveFilePatterns,
syncFileConnectorFile, syncFileConnectorFile,
transferFiles, transferFiles,
uploadFiles, uploadFiles,
virtualFolderResourceId, virtualFolderResourceId,
type ArchivePreviewEntry,
type ArchivePreviewResponse,
type ConflictResolution, type ConflictResolution,
type ConflictStrategy, type ConflictStrategy,
type FileConnectorBrowseItem, type FileConnectorBrowseItem,
@@ -101,6 +106,7 @@ type FileAccessExplanationTarget = {
const DEFAULT_FILE_LIST_ROW_HEIGHT = 58; const DEFAULT_FILE_LIST_ROW_HEIGHT = 58;
const FILE_LIST_OVERSCAN_ROWS = 8; const FILE_LIST_OVERSCAN_ROWS = 8;
const ARCHIVE_FILENAME_PATTERN = /\.(?:zip|tar|tar\.gz|tgz|tar\.bz2|tbz2|tar\.xz|txz)$/i;
export default function FilesPage({ settings, auth }: {settings: ApiSettings;auth: AuthInfo;}) { export default function FilesPage({ settings, auth }: {settings: ApiSettings;auth: AuthInfo;}) {
const canDownload = hasScope(auth, "files:download"); const canDownload = hasScope(auth, "files:download");
@@ -132,6 +138,10 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const [sortDirection, setSortDirection] = useState<SortDirection>("asc"); const [sortDirection, setSortDirection] = useState<SortDirection>("asc");
const [unmatchedCount, setUnmatchedCount] = useState<number | null>(null); const [unmatchedCount, setUnmatchedCount] = useState<number | null>(null);
const [unpackZip, setUnpackZip] = useState(false); const [unpackZip, setUnpackZip] = useState(false);
const [archiveFile, setArchiveFile] = useState<File | null>(null);
const [archivePreview, setArchivePreview] = useState<ArchivePreviewResponse | null>(null);
const [archivePassword, setArchivePassword] = useState("");
const [selectedArchivePaths, setSelectedArchivePaths] = useState<Set<string>>(new Set());
const [busy, setBusy] = useState(false); const [busy, setBusy] = useState(false);
const [uploadActive, setUploadActive] = useState(false); const [uploadActive, setUploadActive] = useState(false);
const [uploadPhase, setUploadPhase] = useState<UploadPhase>("idle"); const [uploadPhase, setUploadPhase] = useState<UploadPhase>("idle");
@@ -501,9 +511,19 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setNewFolderName(""); setNewFolderName("");
setNewFolderError(""); setNewFolderError("");
} }
if (kind === "upload") {
resetArchiveUploadState();
}
openRawDialog(kind, target); openRawDialog(kind, target);
} }
function resetArchiveUploadState() {
setArchiveFile(null);
setArchivePreview(null);
setArchivePassword("");
setSelectedArchivePaths(new Set());
}
function closeDialog() { function closeDialog() {
closeRawDialog(); closeRawDialog();
setNewFolderError(""); setNewFolderError("");
@@ -513,6 +533,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setConnectorError(""); setConnectorError("");
setConnectorSelectedItem(null); setConnectorSelectedItem(null);
setConnectorSpaceLabel(""); setConnectorSpaceLabel("");
resetArchiveUploadState();
} }
function updateActiveDialogFolder(spaceId: string, folderPath: string) { function updateActiveDialogFolder(spaceId: string, folderPath: string) {
@@ -667,6 +688,156 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
async function loadArchivePreview(
file: File,
target: FileActionTarget,
options: { preserveSelection?: boolean } = {}
) {
const targetSpace = findSpace(target.spaceId);
if (!targetSpace || isConnectorSpace(targetSpace)) {
setError(uploadRejectedReason(target));
return;
}
setArchiveFile(file);
setBusy(true);
setUploadActive(true);
setUploadPhase("uploading");
setUploadProgress(null);
setError("");
setMessage("");
try {
const response = await previewArchiveUpload(settings, file, {
owner_type: targetSpace.owner_type,
owner_id: targetSpace.owner_id,
path: target.folderPath,
password: archivePassword || undefined
});
const availableFiles = new Set(
response.entries
.filter((entry) => entry.kind === "file")
.map((entry) => entry.path)
);
setArchivePreview(response);
setSelectedArchivePaths((current) => {
if (!options.preserveSelection) return availableFiles;
const retained = new Set(
Array.from(current).filter((path) => availableFiles.has(path))
);
return retained.size > 0 ? retained : availableFiles;
});
} catch (err) {
setArchivePreview(null);
setSelectedArchivePaths(new Set());
setError(err instanceof Error ? err.message : String(err));
} finally {
setBusy(false);
setUploadActive(false);
setUploadPhase("idle");
setUploadProgress(null);
}
}
function archiveFilesForEntry(entry: ArchivePreviewEntry): string[] {
if (!archivePreview) return [];
if (entry.kind === "file") return [entry.path];
const prefix = `${entry.path}/`;
return archivePreview.entries
.filter((candidate) => candidate.kind === "file" && candidate.path.startsWith(prefix))
.map((candidate) => candidate.path);
}
function archiveEntryIsSelected(entry: ArchivePreviewEntry): boolean {
const paths = archiveFilesForEntry(entry);
return paths.length > 0 && paths.every((path) => selectedArchivePaths.has(path));
}
function toggleArchiveEntry(entry: ArchivePreviewEntry, selected: boolean) {
const paths = archiveFilesForEntry(entry);
setSelectedArchivePaths((current) => {
const next = new Set(current);
paths.forEach((path) => selected ? next.add(path) : next.delete(path));
return next;
});
}
function toggleAllArchiveFiles(selected: boolean) {
setSelectedArchivePaths(
selected && archivePreview
? new Set(
archivePreview.entries
.filter((entry) => entry.kind === "file")
.map((entry) => entry.path)
)
: new Set()
);
}
async function confirmCurrentArchive() {
const target = currentActionTarget();
const targetSpace = target ? findSpace(target.spaceId) : null;
if (
busy
|| !archiveFile
|| !archivePreview
|| !target
|| !targetSpace
|| isConnectorSpace(targetSpace)
) {
return;
}
if (selectedArchivePaths.size === 0) {
setError("Select at least one archive file to import.");
return;
}
if (archivePreview.requires_password && !archivePassword) {
setError("Enter the archive password before importing.");
return;
}
setBusy(true);
setUploadActive(true);
setUploadPhase("uploading");
setUploadProgress(0);
setError("");
setMessage("Uploading the archive for confirmed extraction.");
try {
const response = await confirmArchiveUpload(settings, archiveFile, {
preview_token: archivePreview.preview_token,
selected_paths: Array.from(selectedArchivePaths),
owner_type: targetSpace.owner_type,
owner_id: targetSpace.owner_id,
path: target.folderPath,
password: archivePassword || undefined,
conflict_strategy: "reject",
onProgress: ({ percentage }) => {
setUploadProgress(percentage);
if (percentage !== null && percentage >= 100) {
setUploadPhase("unpacking");
setMessage("Extracting the selected archive files.");
}
}
});
setUploadPhase("finalizing");
setUploadProgress(100);
const uploadedCount = response.files.length;
const destination = target.folderPath || "i18n:govoplan-files.root.e96857c5";
closeDialog();
setMessage(i18nMessage("i18n:govoplan-files.uploaded_value_file_s_into_value.d0fa052b", {
value0: uploadedCount,
value1: destination
}));
resetTransientState();
await loadSpaceContents(targetSpace, { silent: targetSpace.id !== activeSpaceId });
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
setMessage("");
} finally {
setBusy(false);
setUploadActive(false);
setUploadPhase("idle");
setUploadProgress(null);
}
}
async function handleFilesUpload(fileList: FileList | File[], options: {conflictStrategy?: ConflictStrategy;conflictResolutions?: ConflictResolution[];bypassConflictDialog?: boolean;target?: FileActionTarget;} = {}) { async function handleFilesUpload(fileList: FileList | File[], options: {conflictStrategy?: ConflictStrategy;conflictResolutions?: ConflictResolution[];bypassConflictDialog?: boolean;target?: FileActionTarget;} = {}) {
const target = options.target ?? currentActionTarget(); const target = options.target ?? currentActionTarget();
const targetSpace = target ? findSpace(target.spaceId) : null; const targetSpace = target ? findSpace(target.spaceId) : null;
@@ -676,6 +847,14 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} }
const selected = Array.from(fileList); const selected = Array.from(fileList);
if (selected.length === 0) return; if (selected.length === 0) return;
if (unpackZip) {
if (selected.length !== 1 || !ARCHIVE_FILENAME_PATTERN.test(selected[0].name)) {
setError("Choose one ZIP, TAR, TAR.GZ, TAR.BZ2, or TAR.XZ archive to preview and unpack.");
return;
}
await loadArchivePreview(selected[0], target);
return;
}
if (!options.bypassConflictDialog && !unpackZip) { if (!options.bypassConflictDialog && !unpackZip) {
const conflicts = uploadConflicts(selected, target); const conflicts = uploadConflicts(selected, target);
if (conflicts.length > 0) { if (conflicts.length > 0) {
@@ -691,7 +870,6 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
return; return;
} }
} }
const uploadsZipArchive = unpackZip && selected.some((file) => file.name.toLowerCase().endsWith(".zip"));
setBusy(true); setBusy(true);
setUploadActive(true); setUploadActive(true);
setUploadPhase("uploading"); setUploadPhase("uploading");
@@ -703,14 +881,13 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
owner_type: targetSpace.owner_type, owner_type: targetSpace.owner_type,
owner_id: targetSpace.owner_id, owner_id: targetSpace.owner_id,
path: target.folderPath, path: target.folderPath,
unpack_zip: unpackZip,
conflict_strategy: options.conflictStrategy ?? "reject", conflict_strategy: options.conflictStrategy ?? "reject",
conflict_resolutions: options.conflictResolutions, conflict_resolutions: options.conflictResolutions,
onProgress: ({ percentage }) => { onProgress: ({ percentage }) => {
setUploadProgress(percentage); setUploadProgress(percentage);
if (percentage !== null && percentage >= 100) { if (percentage !== null && percentage >= 100) {
setUploadPhase(uploadsZipArchive ? "unpacking" : "finalizing"); setUploadPhase("finalizing");
setMessage(uploadsZipArchive ? "i18n:govoplan-files.unpacking_zip_upload.35019691" : "i18n:govoplan-files.finalizing_upload.bcce936d"); setMessage("i18n:govoplan-files.finalizing_upload.bcce936d");
} }
} }
}); });
@@ -2332,6 +2509,29 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "upload" && {dialog === "upload" &&
<FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}> <FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}>
{!archivePreview &&
<>
<ToggleSwitch
label="Preview and unpack archive"
checked={unpackZip}
onChange={setUnpackZip}
disabled={busy} />
{unpackZip &&
<p className="form-help archive-upload-help">
Supports ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ. Nothing is written until you review and confirm the archive contents.
</p>
}
<div className="field-block">
<FieldLabel className="form-label" help="i18n:govoplan-files.choose_the_destination_folder_before_selecting_o.a4922d75">i18n:govoplan-files.destination_folder.f8ccb63b</FieldLabel>
<TransferFolderSelector
space={activeDialogSpace}
nodes={activeDialogTarget ? buildFolderTree(filesBySpace[activeDialogTarget.spaceId] ?? EMPTY_FILES, foldersBySpace[activeDialogTarget.spaceId] ?? EMPTY_FOLDERS) : []}
selectedFolder={activeDialogTarget?.folderPath || ""}
disabled={busy || !activeDialogSpace}
onSelect={(folderPath) => activeDialogTarget && updateActiveDialogFolder(activeDialogTarget.spaceId, folderPath)} />
</div>
<FileDropZone <FileDropZone
disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace) || !canUpload} disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace) || !canUpload}
busy={uploadActive} busy={uploadActive}
@@ -2341,21 +2541,115 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`} note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`}
onRejectedDrop={(reason) => setError(reason === "disabled" ? uploadRejectedReason(activeDialogTarget) : "The browser did not provide readable file data for this drop. Use the file picker, or drag local files from a file manager that exposes file contents to the browser.")} onRejectedDrop={(reason) => setError(reason === "disabled" ? uploadRejectedReason(activeDialogTarget) : "The browser did not provide readable file data for this drop. Use the file picker, or drag local files from a file manager that exposes file contents to the browser.")}
onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} /> onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} />
<ToggleSwitch label="i18n:govoplan-files.unpack_zip_uploads.256fead4" checked={unpackZip} onChange={setUnpackZip} disabled={busy} />
<div className="field-block">
<FieldLabel className="form-label" help="i18n:govoplan-files.choose_the_destination_folder_before_selecting_o.a4922d75">i18n:govoplan-files.destination_folder.f8ccb63b</FieldLabel>
<TransferFolderSelector
space={activeDialogSpace}
nodes={activeDialogTarget ? buildFolderTree(filesBySpace[activeDialogTarget.spaceId] ?? EMPTY_FILES, foldersBySpace[activeDialogTarget.spaceId] ?? EMPTY_FOLDERS) : []}
selectedFolder={activeDialogTarget?.folderPath || ""}
disabled={busy || !activeDialogSpace}
onSelect={(folderPath) => activeDialogTarget && updateActiveDialogFolder(activeDialogTarget.spaceId, folderPath)} />
</div>
<div className="button-row compact-actions align-end"> <div className="button-row compact-actions align-end">
<Button onClick={closeDialog} disabled={busy}>i18n:govoplan-files.cancel.77dfd213</Button> <Button onClick={closeDialog} disabled={busy}>i18n:govoplan-files.cancel.77dfd213</Button>
</div> </div>
</>
}
{archivePreview && archiveFile &&
<div className="archive-preview">
<div className="archive-preview-summary">
<div>
<strong>{archiveFile.name}</strong>
<span>{archivePreview.archive_format.toUpperCase()} · {formatBytes(archivePreview.compressed_size_bytes)} compressed</span>
</div>
<div>
<strong>{archivePreview.file_count} files</strong>
<span>{formatBytes(archivePreview.expanded_size_bytes)} expanded · {archivePreview.directory_count} folders</span>
</div>
<div>
<strong>Destination</strong>
<span>{activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}</span>
</div>
</div>
{archivePreview.requires_password &&
<FormField
label="Archive password"
help="The password stays in this dialog and is sent only while inspecting or importing this archive.">
<PasswordField
value={archivePassword}
onValueChange={setArchivePassword}
disabled={busy}
autoComplete="off" />
</FormField>
}
<div className="archive-selection-toolbar">
<label>
<input
type="checkbox"
checked={selectedArchivePaths.size === archivePreview.file_count && archivePreview.file_count > 0}
onChange={(event) => toggleAllArchiveFiles(event.target.checked)}
disabled={busy || archivePreview.file_count === 0} />
<span>Select all files</span>
</label>
<span>{selectedArchivePaths.size} of {archivePreview.file_count} selected</span>
</div>
<div className="archive-entry-list" role="list" aria-label="Archive contents">
{archivePreview.entries.map((entry) => {
const selectableFiles = archiveFilesForEntry(entry);
const depth = Math.max(0, entry.path.split("/").length - 1);
return (
<label
key={`${entry.kind}:${entry.path}`}
className={`archive-entry-row ${entry.kind === "directory" ? "is-directory" : ""}`}
style={{ paddingLeft: `${12 + depth * 18}px` }}>
<input
type="checkbox"
checked={archiveEntryIsSelected(entry)}
onChange={(event) => toggleArchiveEntry(entry, event.target.checked)}
disabled={busy || selectableFiles.length === 0} />
{entry.kind === "directory" ?
<Folder size={16} aria-hidden="true" /> :
<File size={16} aria-hidden="true" />
}
<span className="archive-entry-path">{entry.path.split("/").at(-1)}</span>
<span className="archive-entry-size">{entry.kind === "file" ? formatBytes(entry.size_bytes) : `${selectableFiles.length} files`}</span>
</label>);
})}
</div>
<p className="form-help">
Preview expires {formatDate(archivePreview.expires_at)}. The original archive is uploaded again only when you confirm.
</p>
<div className="button-row compact-actions archive-preview-actions">
<Button
onClick={() => {
resetArchiveUploadState();
setError("");
}}
disabled={busy}>
Choose another file
</Button>
{archivePreview.requires_password &&
<Button
onClick={() => activeDialogTarget && void loadArchivePreview(archiveFile, activeDialogTarget, { preserveSelection: true })}
disabled={busy || !archivePassword}>
<RefreshCw size={15} aria-hidden="true" /> Verify password
</Button>
}
<span className="archive-preview-action-spacer" />
<Button onClick={closeDialog} disabled={busy}>i18n:govoplan-files.cancel.77dfd213</Button>
<Button
variant="primary"
onClick={() => void confirmCurrentArchive()}
disabled={
busy
|| selectedArchivePaths.size === 0
|| archivePreview.requires_password && !archivePassword
}>
Import selected
</Button>
</div>
</div>
}
</FileDialog> </FileDialog>
} }
+130
View File
@@ -288,6 +288,132 @@
width: min(820px, 100%); width: min(820px, 100%);
} }
.file-dialog:has(.archive-preview) {
width: min(860px, 100%);
}
.archive-upload-help {
margin: -8px 0 0;
}
.archive-preview {
display: grid;
min-width: 0;
gap: 16px;
}
.archive-preview-summary {
display: grid;
grid-template-columns: minmax(0, 1.3fr) minmax(0, 1fr) minmax(0, 1fr);
gap: 1px;
overflow: hidden;
border: var(--border-line);
border-radius: 6px;
background: var(--line);
}
.archive-preview-summary > div {
display: grid;
min-width: 0;
gap: 4px;
padding: 11px 12px;
background: var(--panel-soft);
}
.archive-preview-summary strong,
.archive-preview-summary span {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.archive-preview-summary span {
color: var(--muted);
font-size: var(--font-size-sm);
}
.archive-selection-toolbar {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
}
.archive-selection-toolbar label,
.archive-entry-row {
display: flex;
align-items: center;
gap: 9px;
}
.archive-selection-toolbar label {
font-weight: 700;
}
.archive-selection-toolbar > span,
.archive-entry-size {
color: var(--muted);
font-size: var(--font-size-sm);
}
.archive-entry-list {
display: grid;
align-content: start;
max-height: min(380px, 42vh);
overflow: auto;
scrollbar-gutter: stable;
border: var(--border-line);
border-radius: 6px;
background: var(--surface);
}
.archive-entry-row {
min-width: 0;
min-height: 38px;
padding-top: 7px;
padding-right: 12px;
padding-bottom: 7px;
border-bottom: var(--border-line);
cursor: pointer;
}
.archive-entry-row:last-child {
border-bottom: 0;
}
.archive-entry-row:hover {
background: var(--hover-tint);
}
.archive-entry-row.is-directory {
background: var(--panel-soft);
font-weight: 700;
}
.archive-entry-row.is-directory:hover {
background: var(--primary-soft);
}
.archive-entry-path {
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.archive-entry-size {
flex: 0 0 auto;
margin-left: auto;
}
.archive-preview-actions {
align-items: center;
}
.archive-preview-action-spacer {
flex: 1 1 auto;
}
.file-dialog:has(.file-share-dialog-content) { .file-dialog:has(.file-share-dialog-content) {
width: min(1080px, 100%); width: min(1080px, 100%);
} }
@@ -312,6 +438,10 @@
} }
@media (max-width: 620px) { @media (max-width: 620px) {
.archive-preview-summary {
grid-template-columns: minmax(0, 1fr);
}
.file-share-form { .file-share-form {
grid-template-columns: minmax(0, 1fr); grid-template-columns: minmax(0, 1fr);
} }