From 1aab50178df34a79bddd80fa52ce09a88f172e45 Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Fri, 7 Aug 2026 11:31:45 +0200 Subject: [PATCH] Sync wiki from project files --- Repo-docs-FILES-HANDBOOK.md | 977 ++++++++++++++++++++++++++++++++++++ 1 file changed, 977 insertions(+) create mode 100644 Repo-docs-FILES-HANDBOOK.md diff --git a/Repo-docs-FILES-HANDBOOK.md b/Repo-docs-FILES-HANDBOOK.md new file mode 100644 index 0000000..64bef08 --- /dev/null +++ b/Repo-docs-FILES-HANDBOOK.md @@ -0,0 +1,977 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-files/docs/FILES_HANDBOOK.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# GovOPlaN Files Handbook + +This handbook describes the Files module as implemented in version `0.1.9`. +It is the operational source of truth for users, process owners, administrators, +operators, auditors, and module integrators. Statements about future behavior +are marked **planned**; an unmarked statement describes the current code. + +Files is a governed snapshot store. It owns managed file content, versions, +logical folders, shares, source provenance, and the evidence that another +GovOPlaN module used a particular file version. It can browse selected external +stores read-only and import a frozen copy. It is not a general remote filesystem, +a document collaboration engine, or a records-management system. + +## Choose a reading path + +| If you need to... | Start with... | +| --- | --- | +| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) | +| Design a governed process that uses files | [Process perspective](#process-perspective) | +| Decide who can use a file or connector | [Ownership and access](#ownership-and-access) and [Administration and policy](#administration-and-policy) | +| Operate storage, connectors, backups, or recovery | [Operator runbook](#operator-runbook) | +| Integrate another GovOPlaN module | [Capabilities and integration](#capabilities-and-integration) | +| Review evidence, deletion, or security behavior | [Security, provenance, audit, deletion, and retention](#security-provenance-audit-deletion-and-retention) | +| Verify a release or scenario | [Acceptance scenarios](#acceptance-scenarios) | +| Check whether an idea exists today | [Implemented and planned boundary](#implemented-and-planned-boundary) | + +The Files-owned interface archetypes, consequence classes, disabled-state +wording, and verification evidence are recorded in +[Files Interface Pattern Migration](INTERFACE_PATTERN_MIGRATION.md). + +## The service contract + +A managed file is a tenant-scoped logical asset with exactly one user or group +owner, a normalized path, and a current version. The current version points to a +blob record containing the storage location, SHA-256 checksum, byte size, and +content type. A protected blob also records its Encryption envelope, protection +discriminator, stored-ciphertext checksum, and stored-ciphertext size. Current +service paths append a version when connector sync finds +changed content; they do not mutate the previous version record. + +Unprotected content with the same tenant, plaintext SHA-256 checksum, size, and +protection discriminator can reuse one blob. Protected content is deduplicated +only inside the same vault/profile discriminator; ciphertext is never silently +reused across protection boundaries. Plaintext checksums remain semantic +version evidence, while download and integrity scans verify stored ciphertext +before asking Encryption to open it. Neither digest proves authorship or source +authenticity. + +The main domain objects are: + +| Object | Meaning | Lifecycle today | +| --- | --- | --- | +| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted | +| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained | +| File blob | Stored plaintext semantics plus stored-byte integrity, backend key, optional Encryption envelope, reference count, and retention timestamp | Reused only within a tenant and matching protection boundary; no automated garbage collection | +| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted | +| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet | +| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion | +| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion | +| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use | +| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted | +| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence | +| Form evidence upload grant | A one-use, hash-only bearer grant tied to an exact Form instance/revision, purpose, custodian, size, and media-type policy | Issued for at most 15 minutes, consumed by one managed upload, then retained as evidence provenance | + +## Deployment configuration packages + +Files registers the `files.configuration` capability for `managed_storage` +fragments. Managed content storage is deployment-owned: the installer selects +local persistent storage, managed Garage, or an external S3-compatible service +and mounts a validated non-secret infrastructure capability receipt. Files +does not copy that endpoint or its credentials into module-owned tables. + +Preflight compares the receipt's `files.storage` capability with the effective +runtime backend. It validates backend kind, sanitized S3 endpoint, bucket, +Garage management marker or external trust marker, an absolute persistent path +for local storage, and the presence of Files-owned `env:` secret references. +Secret values are never read into a plan, diagnostic, export, or fragment. + +When runtime and receipt agree, the plan reports `skip`: the desired binding is +already effective, and repeated apply is a no-op. A mismatch blocks import and +explains which deployment setting must be reconciled. The provider deliberately +does not mutate process environment, migrate stored objects, probe remote +storage, or reinterpret an infrastructure replacement as safe. Use the Files +integrity and Ops checks after deployment and complete migration/recovery review +before changing an active backend. + +## User tasks + +The Files page is available at `/files`. Actions appear only when the current +principal has the required permission and resource access. + +The configured Help Center projects these sections as independently authorized +tasks, so upload, ZIP import, organization, download, sharing, and deletion do +not disappear merely because an unrelated permission is absent. It states the +deployment's actual upload limits. External import appears as a task only when +the actor has the required permissions and at least one actor-visible profile +is eligible for the current fail-closed browse/import path; endpoint, path, and +item policy are still enforced when the operation runs. + +### Choose a space + +Every file user sees **My files**. Group spaces are added for active groups of +which the user is a member. The space list also exposes every tenant group to a +Files administrator; governed API operations can administer other tenant-owned +Files resources when their owner is specified. Linked connector spaces appear +beside managed spaces when they are active and visible to the user. + +A connector space is intentionally read-only. It is a view of an approved +remote location and a starting point for importing or synchronizing selected +files into managed storage; it is not a mounted write-through filesystem. + +### Upload files + +The managed-space UI supports upload and drag-and-drop. A caller chooses the +user or group owner and destination folder. The normal per-file deployment +limit defaults to 50 MiB through `FILE_UPLOAD_MAX_BYTES`. + +When a target path already exists, the operation must use one of these conflict +strategies: + +- `reject` stops instead of silently replacing content; +- `rename` selects the next available `copy` name; +- `overwrite` soft-deletes the conflicting asset and creates the new asset; +- an item-specific conflict resolution can also `skip` that item. + +An ordinary upload does not append a new version to an existing asset. The +`overwrite` strategy retires the old asset at that path. Version-preserving +updates currently belong to connector sync. + +### Preview and unpack archives + +The UI previews ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ before writing managed +files. Users may select individual files or complete folders. The request is +spooled to a bounded temporary file rather than buffered wholly in memory. The +defaults are: + +- 250 MiB compressed request; +- 50 MiB per extracted member; +- 2 GiB total expanded data; +- 10,000 declared entries; +- 100:1 maximum expansion ratio; +- a 30-minute preview token bound to the tenant, user, archive digest, and + destination; +- password-protected ZIP support with request-only password handling; +- traversal, duplicate paths, links, devices, and other special entries are + rejected; +- actual bytes read are counted, not only archive header declarations. + +The browser retains the selected archive and password until confirmation. +Confirmation reuploads the archive, verifies the token and digest, repeats all +safety checks, and commits only the selected files. No preview archive or +password is retained server-side. + +### Organize files and folders + +With `files:file:organize`, a user can: + +- create logical folders; +- rename one selection directly; +- preview and apply bulk prefix, suffix, or replacement renames; +- move or copy files and folder trees between accessible user/group spaces; +- resolve target conflicts by rejecting, renaming, overwriting, or skipping; +- use drag-and-drop for move operations in the file explorer. + +Copies create new assets and versions while reusing the immutable blob bytes. +Moves keep the asset identity and change its owner/path. All source and target +owners are validated; knowing an identifier does not bypass space membership. + +Recursive folder deletion is the default. It soft-deletes the selected folder, +its child folders, and files below it. A non-recursive delete fails when the +folder is not empty. + +### Find and download files + +Files can list by owner and path, use cursor pagination, and consume incremental +changes through a watermark. The UI supports path/name pattern search and +sorting. The pattern API can resolve campaign-style wildcard selections and can +return unmatched files. + +A user with download permission and resource access can download one current +version or create a ZIP archive from a selection. Downloads use an attachment +content disposition. The archive is generated in a temporary file and removed +after the response completes. + +There is no dedicated content-preview service in the Files API today. File +responses expose content type, size, checksum, and current version metadata. + +### Share files + +The API can grant or update a share for a user, group, the tenant, or a campaign. +The supported permissions are `read`, `write`, and `manage`. Ownership remains +unchanged. A write operation accepts a `write` or `manage` share; read/download +accepts any of the three. + +The current Files page shows campaign linkage but does not offer a general +user/group share editor. The API also has no share-revocation route yet. Treat +share revocation and a complete share-management UI as planned work. + +### Browse and import an external file + +With a visible connector profile, a user can browse the allowed remote path, +select one file, and import it into an accessible managed space. The imported +asset records the connector, provider, remote identity/path/URL, selected remote +metadata, and source revision when supplied by the provider. + +Manual sync looks for an existing managed asset with the same source identity +inside the chosen owner space: + +- no match creates a managed asset; +- identical checksum and size updates provenance and returns `unchanged`; +- changed bytes append a version and return `updated`. + +Browse, import, and sync never write, rename, or delete the remote source. + +Connector administration separates endpoint profiles, reusable credentials, +and inherited policy. Ordinary setup uses typed fields and provider discovery; +provider metadata JSON is available only under advanced compatibility options. +Read-only deployment entries explain where they must be changed, and disabled +actions identify the missing permission, target, input, or running operation. +The contextual help icon opens the configured Help Center topic when Docs is +enabled and the hosted GovOPlaN documentation otherwise. + +## Process perspective + +### Managed ingestion + +The managed-file flow is: + +1. Core authenticates the principal and evaluates the operation permission. +2. Files validates the tenant, owner, group membership, or applicable share. +3. Files normalizes the logical path and resolves conflicts explicitly. +4. Upload or connector response limits are enforced before content is retained. +5. Files calculates SHA-256 and stores or reuses a tenant blob. +6. Files creates the asset/version records and optional campaign share. +7. The database transaction commits and emits change-sequence entries. +8. Connector-originated operations also emit their connector audit evidence. + +Blob storage is not part of the database transaction. An object may therefore +be left without committed metadata after a process or database failure. The +operator integrity API scans database blobs and the tenant storage prefix in +bounded, resumable phases. It reports orphan objects before any cleanup and +never deletes them as part of a scan. + +### Governed connector import + +The connector flow separates four concerns: + +1. An administrator defines reusable credential material. +2. An administrator defines a scoped endpoint profile that may reference that + credential. +3. System, tenant, and leaf policy sources narrow the allowed profile, + credential, provider, URL, and remote path. +4. A user optionally links an allowed remote root as a user/group connector + space, then browses and imports selected content. + +Before each network operation, Files checks profile visibility, connector +policy, endpoint safety, transport support, and response size. A successful +import becomes an independent managed snapshot. Later source changes have no +effect until an explicit sync. + +### Campaign attachment evidence + +When Campaign uses Files, the integration follows a freeze-before-send model: + +1. The campaign refers to managed user/group sources and attachment patterns. +2. Files verifies access and resolves matching managed assets. +3. A prepared campaign snapshot records the exact asset, version, blob, + checksum, size, relative path, and source provenance. +4. Files materializes those bytes for the campaign build without exposing its + database models to Campaign. +5. Campaign job/entry use is recorded and later marked as sent. + +Changing the current file after preparation does not change the version already +recorded as campaign evidence. A file response is marked `audit_relevant` once +the asset has a sent campaign attachment-use record. + +### Process ownership + +| Concern | Owner | +| --- | --- | +| Authentication, tenants, RBAC evaluation, audit service, change sequence, settings, and module lifecycle | GovOPlaN Core | +| Managed assets, blobs, versions, folders, shares, connector baseline, provenance, and campaign attachment evidence | Files | +| Campaign definition, recipient data, message build/send state, and delivery policy | Campaign | +| Remote ACLs, remote source content, and upstream revision semantics | The external provider | +| Storage durability, egress policy, master key, secret environment, backup, and recovery | Deployment operator | +| Collaborative editing, comments, review, locks, and semantic document workflows | A future Documents/workflow/provider module | + +## Ownership and access + +Files applies both permission checks and resource checks. A broad operation +permission alone does not make another user's file visible. + +### Resource access + +- A user owns their personal space. +- A group member can use the group's file space. +- A Files administrator can access all Files resources in the active tenant. +- A file share can grant read or write access to a user, group, or the tenant. +- Campaign shares are resolved only in a verified campaign context and do not + become ordinary user shares. +- Folders are owned by a user or group; they are not independently shared. +- Soft-deleted resources are excluded from normal access and listing. +- Tenant identifiers are checked on every managed object lookup. + +The `files.access` capability can explain why a principal has access: resource, +owner, administrator scope, or active share. It also explains virtual folders +that exist through child assets even when there is no explicit folder row. + +Deleting an organization/access group is vetoed while it owns Files assets, +folders, connector spaces, or is the target of file shares. Reassign or remove +those relationships first. + +### Operation permissions + +| Permission | Allows | +| --- | --- | +| `files:file:read` | List and inspect accessible files, folders, spaces, and visible connectors | +| `files:file:download` | Download an accessible current version or ZIP archive | +| `files:file:upload` | Upload managed assets and import/sync selected connector files | +| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces | +| `files:file:share` | Create or update file shares | +| `files:file:delete` | Soft-delete accessible writable files and folders | +| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant | + +The `file_manager` role template grants all normal file operations except +`files:file:admin`. The `file_viewer` template grants read and download. +System and tenant settings permissions can also authorize the corresponding +connector administration endpoints. + +## Administration and policy + +### Profiles, credentials, policies, and spaces + +Keep these definitions separate: + +- a **credential** holds reusable authentication material and may be restricted + to a provider; +- a **profile** holds the endpoint, scope, base path, credential reference, + local policy, and descriptive operation capabilities; +- a **policy** restricts what a scope may configure or use; +- a **connector space** links one approved profile/library/path to one user or + group and always uses manual, read-only synchronization today. + +Profile capability values such as `browse`, `import`, and `sync` are stored and +returned, but they are descriptive today. Provider implementation and policy +checks enforce actual availability; do not use the capability list as the sole +security control. + +Profiles, credentials, and policies support `system`, `tenant`, `user`, +`group`, and `campaign` scopes. A normal user sees system and active-tenant +profiles plus leaf profiles that match the user, one of their groups, or an +accessible campaign. Disabled definitions are visible only through authorized +administrative reads. + +### Policy evaluation + +For a leaf scope, the effective source chain is: + +```text +system -> tenant -> user | group | campaign +``` + +Policy fields are: + +- connector/profile IDs; +- credential IDs; +- providers; +- external IDs; +- external path prefixes or glob patterns; +- external URL glob patterns. + +Rules use `allow` and `deny` objects. Legacy synonyms `allowlist`, `whitelist`, +`denylist`, and `blacklist` are normalized. A matching deny at any source wins. +Every allow field defined by a source must match, so lower sources can narrow an +inherited set. The effective-policy response includes the contributing source +path and applied fields for explanation. + +A parent may set `allow_lower_level_limits` for individual fields such as +`allow.providers` or `deny.external_paths`. An explicit `false` prevents a lower +scope from configuring that field. Absence does not lock the field. + +Example tenant policy: + +```json +{ + "policy": { + "allow": { + "providers": ["webdav", "nextcloud"], + "external_urls": ["https://files.example.edu/*"], + "external_paths": ["departments/finance"] + }, + "deny": { + "external_paths": ["departments/finance/private"] + }, + "allow_lower_level_limits": { + "allow.providers": true, + "deny.external_paths": true + } + } +} +``` + +Use `POST /api/v1/files/connector-policy/evaluate` for an explainable preflight. +The normal profile browse/import/sync routes perform their own policy checks; +preflight does not replace enforcement. + +### Credential rules + +Database-created passwords and tokens use Core's Fernet encryption and require +the deployment `MASTER_KEY_B64` outside development/test/local environments. +Secret values, environment variable names, and local CA paths are never returned +in connector profile responses. + +API-managed profiles and credentials cannot select process environment +variables, create an external `secret_ref`, or conceal secret-like values in +nested metadata. Deployment-owned JSON/file profiles may use `password_env` or +`token_env` only when the exact environment variable name appears in +`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`. + +Custom CA bundles must be absolute existing files in +`GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST`. TLS verification can be disabled only +in a development/test runtime. + +### Provider status + +| Provider | Current status | +| --- | --- | +| Seafile | Read-only native API browse/download-link import and manual sync implemented using the pinned HTTP transport; WebDAV opt-in supported | +| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport | +| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport | +| SMB | Read-only browse/import/manual sync implemented through a pinned smbprotocol transport for initial peers, reconnects, aliases, and DFS referral targets | +| S3 connector | Read-only bucket/prefix browse, import, and manual sync implemented through pinned botocore pools covering retries, redirects, endpoint discovery, and provider aliases | +| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned | +| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature | + +Provider descriptors are available from +`GET /api/v1/files/connectors/providers`. Use their `implemented`, `installed`, +and support fields for display. An incompatible optional SDK release fails closed +before it can return a usable client or session. + +## Operator runbook + +### Storage configuration + +| Setting | Default | Purpose | +| --- | --- | --- | +| `FILE_STORAGE_BACKEND` | `local` | Selects `local` or `s3` managed blob storage | +| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root | +| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root | +| `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket | +| `FILE_STORAGE_S3_DEPLOYMENT_MANAGED` | `false` | Installer-only trust marker for the exact `http://garage:3900` service; never use it for another endpoint | +| `FILE_STORAGE_S3_ENDPOINT_TRUSTED` | `false` | Deployment-owner acknowledgement for one clean HTTPS external S3 origin; never expose this through connector configuration | +| `GOVOPLAN_STATE_PROFILE` | `local` | Selects `local`, one-host `host-shared`, or multi-host `shared` state validation | +| `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and extracted archive-member maximum | +| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | Compressed archive request maximum (legacy name retained for compatibility) | +| `FILE_ARCHIVE_MAX_ENTRIES` | 10,000 | Maximum declared archive entries | +| `FILE_ARCHIVE_MAX_EXPANDED_BYTES` | 2 GiB | Maximum expanded archive bytes | +| `FILE_ARCHIVE_MAX_EXPANSION_RATIO` | 100 | Maximum expanded-to-compressed ratio | +| `FILE_ARCHIVE_PREVIEW_TTL_SECONDS` | 1,800 | Lifetime of the sealed archive preview token | +| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets | + +The local backend is the operational baseline. It resolves every storage key +under the configured root and rejects escape attempts. Fallback roots support a +controlled storage-root migration: new writes go to the primary root while +reads can still find older objects. + +The supported installer may provision a deployment-owned Garage service at the +exact `http://garage:3900` endpoint and set +`FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true`. An operator-selected external S3 +backend instead requires a clean HTTPS origin and +`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`. Both are deployment authority, not a +general connector or private-network bypass. Core owns the shared backend +implementation; Files owns metadata and the Files key namespace. + +Multiple API replicas require the same durable blob namespace. Separate local +container filesystems will produce incomplete reads. Use `host-shared` with one +durable shared mount only for same-host replicas. Independent hosts require the +`shared` profile with external S3, PostgreSQL, Redis, a stable installation id, +and one immutable module composition. + +### Connector egress + +Connector access to private networks is a deployment-wide decision: + +```text +GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true|false +``` + +Production-like configuration validation requires an explicit value. Public-only +mode rejects any hostname whose DNS answers include a non-public address. +Private-enabled mode still rejects link-local, multicast, unspecified, and +limited-broadcast addresses. + +The built-in HTTP transport: + +- resolves and validates every connection attempt; +- connects the socket to the exact approved address while retaining the + original hostname for HTTP Host, TLS SNI, and certificate verification; +- does not inherit proxy settings; +- refuses redirects instead of following a new peer implicitly; +- bounds structured responses to 16 MiB and file transfers to 512 MiB by + default. + +The S3 SDK adapter applies the same socket rule to every botocore pool selected +for a retry, redirect, discovered endpoint, or virtual-host bucket alias. The +original authority remains in the request and TLS SNI/certificate check. S3 +connector clients use no outbound proxy and never discover ambient AWS +credentials: configure both access and secret keys on the governed profile, or +use an anonymous profile for a public source. + +The SMB adapter owns a separate connection cache and replaces smbprotocol's TCP +factory process-wide with the stricter pinned socket. Initial peers, reconnects, +server aliases, domain-controller connections, and DFS referral targets therefore +pass the same policy at connection time. Signing is required by default; enable +SMB encryption on the profile where the server supports it. + +Override the connector response limits with +`GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` and +`GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES`. The smaller applicable limit wins +when an import is also subject to `FILE_UPLOAD_MAX_BYTES`. + +Never work around a connector pinning failure by adding a raw IP, disabling TLS, +or enabling private networks. A failure means the peer policy rejected an actual +connection destination or the installed SDK no longer exposes the verified +transport seam. The separately configured platform S3 backend is trusted only by +the deployment owner and is not selectable by a user or connector profile. + +### Backup and restore + +The database and blob namespace are one logical backup set. A usable backup must +include: + +- Files database rows, including asset/version/blob relationships, shares, + connector settings, and campaign attachment-use evidence; +- every object below `FILE_STORAGE_LOCAL_ROOT` and any still-used fallback root; + or the complete S3 bucket/prefix and version/lifecycle evidence for an S3 + backend; +- the exact `MASTER_KEY_B64` needed to decrypt retained connector credentials; +- deployment-owned connector profile files, referenced CA bundles, and secret + environment configuration where those definitions are in use. + +There is no Files backup/restore API. Use a write quiesce or coordinated +snapshots so database references and objects represent the same recovery point. +The integrity API verifies a restored set, but it does not replace a coordinated +backup. + +Operators normally use **Administration > File integrity**. The equivalent API +creates a scan with `POST /api/v1/files/integrity/scans`, then calls +`POST /api/v1/files/integrity/scans/{scan_id}/run` with the scan's current +`expected_revision` until it reports `completed`. Each call advances at most +the persisted batch size, so a stopped operator or worker can resume from the +committed blob/object cursors. Concurrent or stale actions receive `409` before +the storage backend is invoked; reload the scan and inspect the newer state. + +Findings distinguish: + +- `missing`: metadata references an absent object; +- `size_mismatch` or `checksum_mismatch`: bytes do not match immutable blob + metadata and the blob is quarantined; +- `orphan_object`: an object exists in the tenant Files prefix without a + corresponding blob row. + +Missing or corrupt blobs fail closed for ordinary downloads and Campaign +attachment materialization. After restoring the expected bytes, use the finding +`recheck` action with its current `expected_revision`. Orphan cleanup starts +with a dry-run preview and requires separate destructive confirmation. The +confirmation reuses the finding revision from that preview, rechecks that no +database reference exists, remains scoped to the scanned tenant prefix, and is +idempotent. Both applied and dry-run actions emit audit evidence. A shared +reference blocks deletion. Files currently has no legal-hold or hard-purge +model, so retention-controlled objects must not be treated as cleanup +candidates until those controls are implemented. + +### Recovery ledger for object effects + +Every managed blob creation or integrity repair starts a Core recovery +operation in an independent committed transaction before Files protects or +writes bytes. The operation records tenant/blob identifiers, an opaque object +locator or locator digest, semantic SHA-256/size evidence, the recovery mode, +and a distributed lease fence. It never records file contents, ZIP passwords, +connector credentials, or a newly uploaded filename. New object keys are opaque; +legacy filename-bearing keys remain readable but repair operations record only +their digest and recover through the blob ID. + +The Files business transaction then creates or updates the blob, version, and +asset rows. Its actual SQLAlchemy commit or rollback settles every pending +operation: + +- commit reloads the blob through an independent session and streams the object + to verify its stored-byte SHA-256 and size before recording success; +- rollback deletes only a newly reserved object after independently proving + that no `FileBlob` references it, then records verified compensation; +- a repaired existing object is forward-completed only when its identity, + envelope, semantic evidence, and stored bytes all match; +- missing or mismatched bytes quarantine a committed blob and leave the + operation `recovery_required`; an unavailable probe remains + `outcome_unknown` rather than becoming an ordinary upload failure. + +Applied orphan cleanup has its own forward-recovery operation. The database +reference check and tenant-prefix check happen before deletion; object absence +and the durable finding state are verified afterward. If the caller transaction +rolls back after deletion, Files may forward-complete only that existing +finding after rechecking that the key is still unreferenced. + +Archive preview and confirmation use bounded process-local temporary staging. +Staging is not authoritative and is removed on every handled exit; extracted +members enter the same per-blob recovery boundary as direct uploads. A hard +process loss may leave a temporary OS file for normal host temporary-file +cleanup, but cannot make that staging path a managed Files object. + +Use the Ops recovery-operation view to inspect `files` operations. Do not retry +a busy or unresolved blob blindly: first verify the FileBlob row, object hash, +integrity state, and any Encryption envelope named by the blob. Hard purge, +legal hold, and two-way remote connector mutation are not implemented yet, so +they cannot claim recovery-ledger adoption; their owning work remains tracked +separately. + +After restore: + +1. Verify the active tenant and module migration state. +2. Verify the storage backend and roots before allowing writes. +3. Verify that the original master key is available before testing connectors. +4. Download representative files and compare bytes with their recorded SHA-256. +5. Test one authorized and one unauthorized owner/share path. +6. Test a permitted pinned HTTP connector, if connectors are configured. +7. Review audit and change-sequence continuity around the recovery point. + +An inconsistent restore should fail closed for missing objects or undecryptable +credentials. Do not repair it by deleting evidence rows without an approved, +audited data-recovery decision. + +### Disable, uninstall, and retire + +Disabling a module preserves its persistent data. Ordinary uninstall is guarded +while Files tables contain persistent rows. + +Destructive retirement is separate and irreversible at the application level. +The installer records a database snapshot, then Files scrubs and audits +remaining encrypted connector material before its database tables are dropped. +Legacy external secret references are detached and identified as non-owned; +Files never calls a provider delete operation for them. + +The retirement executor drops database tables but does not delete corresponding +objects from the configured blob backend. Operators must include those objects +in the approved retention/destruction plan, report them with an integrity scan, +and explicitly approve cleanup. Validate the installer snapshot and independent +blob backup before retirement. + +### Operational signals + +Use these symptoms as routing hints: + +| Symptom | Likely boundary | +| --- | --- | +| `Stored object does not exist` | Database/blob restore mismatch, wrong root, or missing shared storage | +| `Stored secret cannot be decrypted` | Wrong or rotated `MASTER_KEY_B64` | +| Private/non-public endpoint blocked | Deployment-wide egress policy is public-only or DNS returned a forbidden answer | +| SDK peer-pinning seam is unavailable | Optional S3/SMB SDK is incompatible; keep access fail-closed and validate the supported dependency range before upgrade | +| Connector response exceeds limit | Remote payload exceeds connector or upload limit | +| Profile is not visible | Scope, disabled state, campaign access, or policy mismatch | +| Group removal is vetoed | The group still owns a file/folder/connector space or is a share target | + +## Capabilities and integration + +Other modules must integrate through Core contracts, Files capabilities, or the +public HTTP API. They must not import Files ORM models or storage helpers. + +### Provided capabilities + +| Capability | Purpose | +| --- | --- | +| `files.access` (`0.1.6`) | Explain resource access provenance for managed files, explicit folders, and virtual folders | +| `files.campaign_attachments` (`0.1.6`) | Resolve managed attachment matches, prepare frozen campaign snapshots, annotate built messages, share assets with a campaign, and record/mark exact attachment use | +| `records.source.files` (`1.0.0`) | Recheck current Files access and resolve one exact, integrity-approved managed file version for Records filing | +| `forms_runtime.evidence.files` (`1.0.0`) | Issue a one-time managed attachment grant and re-verify the exact file/version/checksum at Form submission | + +Files requires Core principal resolution and permission evaluation. Campaign is +an optional dependency; when installed, Files consumes the optional +`campaigns.access` interface to verify campaign existence and access. Missing +optional Campaign support fails explicitly rather than bypassing the check. +Records is also optional. When enabled, the source capability returns the +requested `FileVersion` identity, path snapshot, content metadata, SHA-256, +integrity/protection state, and launch link. It rejects mutable aliases, +cross-tenant requests, missing access, and quarantined or failed blobs. Records +stores the filing decision; Files continues to own the version and bytes. +Forms Runtime is optional as well. Its public or authenticated participant never +receives general Files access through this integration. Forms Runtime asks for a +purpose-bound grant, Files verifies an active same-tenant user custodian, stores +only the token digest, accepts one bounded upload, and returns an immutable +`EvidenceReference`. Draft save and final submit independently recheck the exact +Form instance/revision, grant, asset, version, checksum, deletion state, and +integrity state. An idempotent replay returns the existing grant without +reproducing its bearer secret. + +### API families + +All routes below are under `/api/v1/files`. + +| Area | Routes | +| --- | --- | +| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` | +| Upload and folders | `POST /upload`, `POST /upload-zip` (compatibility), `POST /archive-preview`, `POST /archive-confirm`, `POST /folders`, `POST /folders/delete` | +| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` | +| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` | +| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` | +| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` | +| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` | +| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` | +| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` | +| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` | +| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` | +| Incremental connector settings | `GET /connectors/settings/delta` | +| Form evidence | `POST /form-evidence/upload` with a short-lived `X-Form-Evidence-Token` issued by Forms Runtime | + +Consumers should use cursor/watermark contracts instead of assuming an +unbounded complete list. The default full-list page size is 500 and public page +sizes are capped at 1,000. + +### Integration invariants + +An integrating module should: + +- ask Files/Core for access rather than trusting a submitted file ID; +- retain an exact version/blob/checksum at every governed evidence point; +- import external content before using it in a campaign, report, workflow, or + generated document; +- preserve Files provenance when producing a derived managed snapshot; +- treat source provenance as captured context, not cryptographic attestation; +- avoid writing remote providers through the baseline connector layer; +- define a separate capability when it needs behavior beyond managed snapshots; +- tolerate Files being absent when the integration is declared optional. + +Future Postbox, Templates/Reports, BI, Documents, DMS, and workflow modules +should keep their own domain state and use Files for governed input/output +snapshots. Long-running provider sync, OAuth, remote mutation, and +provider-specific health belong in connector modules rather than expanding the +Files baseline indiscriminately. + +## Security, provenance, audit, deletion, and retention + +### Security controls implemented + +- Core authentication, tenant scoping, CSRF/API handling, and RBAC protect the + public routes. +- Owner and share checks protect individual resources after operation-scope + checks. +- Logical paths reject traversal and storage keys cannot escape the local root. +- Upload, archive extraction, connector response, and S3 stream code use bounded + reads. Archive previews are sealed and short-lived; ZIP passwords are + request-only. +- Public Form evidence uploads require a custom-header bearer grant that is + stored only as SHA-256, expires after at most 15 minutes, is bound to one + exact submission and user custodian, and can create only one managed file. +- Connector HTTP sockets use connection-time DNS/IP validation and pinning, + redirects are refused, and unsafe SDK transports fail before client creation. +- Database-managed connector passwords/tokens are encrypted; responses redact + secrets and deployment references. +- API metadata is recursively checked for secret-like values. +- Downloads use sanitized attachment filenames. +- Plaintext semantic and stored-byte SHA-256/size evidence are recorded for + every protected blob; they are identical for unprotected blobs. +- Upload and archive-confirm APIs can select an Encryption vault. Protected + writes and reads fail closed if the optional Encryption capability is absent. +- Managed-object writes and applied orphan cleanup start lease-fenced Core + recovery operations before their physical effects; terminal success and + compensation require independent database and object checks. + +The module does **not** currently provide malware scanning, content disarm and +reconstruction, a file-type allowlist, per-user quota, automatic encryption +policy assignment, client E2EE, or a dedicated preview sandbox. The optional +server-envelope profile protects selected managed blob bytes at rest but remains +server-decryptable. Deployments that require the other controls must supply +them outside Files until explicit module contracts exist. + +### Provenance + +Connector-originated assets can retain: + +- source type; +- connector/profile ID and provider; +- external ID, path, and URL; +- revision and revision label; +- observation/import timestamps when supplied; +- selected provider metadata. + +The normalized provenance and source revision are returned in file responses, +carried into campaign attachment matches, and included in connector audit +events. External metadata is provider/user input and is not a digital signature. + +### Audit and change evidence + +Files records canonical audit events for: + +- connector discovery attempts, before the attempted external I/O; +- connector imports and manual syncs; +- download/archive access to connector-originated managed files; +- immediate connector profile and credential deletion/scrubbing; +- credential scrubbing during destructive module retirement. + +Connector audit details include the managed asset/version/blob, checksum, size, +operation, source revision, and provenance where applicable. Deletion audit +details name secret/reference kinds but never the secret values. + +Assets, folders, shares, profiles, credentials, policies, and connector spaces +also feed Core's incremental change sequence for UI synchronization. A change +entry is not equivalent to a canonical audit event. Ordinary local upload, +rename, share, and soft-delete operations do not yet all emit dedicated Files +audit events. + +Campaign attachment-use records provide separate domain evidence for an exact +file version used in campaign preparation and delivery. + +### Deletion semantics + +The word "delete" has different meanings by object type: + +| Object | Current delete behavior | +| --- | --- | +| File asset | Sets `deleted_at`; content, versions, blob references, and campaign evidence remain | +| Folder | Sets `deleted_at`; recursive deletion also soft-deletes descendants | +| Connector space | Sets `deleted_at` and disables the space | +| Connector credential | Immediately disables the tombstone and clears username, encrypted password/token, environment references, legacy external reference, mode, and private metadata; dependent profiles are disabled and detached | +| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata | +| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed | +| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility | + +Credential/profile scrubbing and its audit event use the same database +transaction. If audit creation fails, the deletion rolls back. Repeating a +delete against an already scrubbed tombstone does not recreate secret evidence. + +### Retention boundary + +File versions and blobs are effectively retained indefinitely today. Although a +blob has `ref_count` and `retained_until` fields, no complete retention-policy, +legal-hold, hard-purge, or garbage-collection service enforces them. There is +also no supported user restore endpoint for soft-deleted assets/folders. +Integrity reconciliation is operator-triggered and is not a retention or +automatic garbage-collection policy. + +Do not promise erasure, timed retention, legal hold, or self-service recovery +from the current soft-delete behavior. Those require an explicit, auditable +retention/purge design that preserves campaign and other evidence references. + +## Acceptance scenarios + +These scenarios describe expected behavior at the current boundary. + +### Personal and group ownership + +Given Alice has `files:file:upload` and belongs to Finance, when she uploads to +her personal or Finance space, then the asset has the selected owner and tenant. +Given Bob is outside Finance and has no share or Files admin permission, the +same asset ID must not make the asset readable to Bob. + +### Safe archive import + +Given an archive member contains `../../secret.txt`, is a link or special +filesystem object, exceeds 50 MiB, pushes actual expanded bytes above 2 GiB, or +exceeds the 100:1 expansion ratio, preview or confirmation must fail without a +committed managed asset. A password-protected ZIP requires the correct +request-only password. A valid confirmation must match its unexpired preview +token and preserve normalized relative paths below the chosen logical folder. + +### Explicit conflicts + +Given a target path already exists, `reject` must leave it unchanged, `rename` +must choose a non-conflicting path, and `overwrite` must soft-delete the old +asset before creating the replacement. A per-item `skip` must not create that +item. + +### Policy-denied connector + +Given the tenant permits WebDAV only below `departments/finance` and denies its +`private` child, a browse/import/sync request below the denied child must return +an explainable policy denial. Directly invoking the import endpoint must not +bypass the same rule. + +### Pinned connector transport + +Given public-only mode and DNS returns any private address, the connection must +be rejected before a socket opens. Given private mode, every HTTP, S3, and SMB +connection must still use an address from the answer validated for that exact +attempt. Botocore retries, redirects, endpoint discovery, and aliases, plus SMB +reconnects and DFS referrals, must pass through the pinned factories. A changed +or unsupported SDK seam must fail before a usable client/session is returned. + +### Imported evidence and sync + +Given a permitted WebDAV file is imported, its managed response and audit event +must carry source identity, revision when available, current version, checksum, +and size. Re-syncing identical bytes must return `unchanged`; changed bytes must +create a higher version while preserving the previous version. + +### Campaign freeze + +Given a campaign snapshot selected version V1, when the managed asset later +advances to V2, the prepared/sent attachment-use evidence must still identify +V1 and its original blob/checksum. + +### Immediate credential deletion + +Given a database credential contains an encrypted password and is referenced by +two profiles, deleting it must scrub the credential, disable/detach both +profiles, record non-secret audit evidence, and publish connector-setting +changes in one transaction. If audit creation fails, no part of the deletion +may commit. + +### Recovery + +Given a coordinated database/blob backup and the original master key, restoring +it must allow representative downloads whose bytes match recorded SHA-256 +values. A missing blob or wrong key must surface an error rather than silently +returning different content or credentials. + +## Implemented and planned boundary + +| Area | Implemented now | Planned or explicitly outside the current boundary | +| --- | --- | --- | +| Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, dry-run-first orphan cleanup, and Core-ledger verification/forward recovery | Scheduled scan execution and deployment-specific S3 HA/backup automation | +| Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts, opaque new object keys, and rollback compensation | Malware scanning, quotas, type policy, resumable/chunked upload | +| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore | +| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles | +| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) | +| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected | +| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers | +| SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, and redirect/retry/referral transport-contract tests | Live topology smoke evidence, provider-specific OAuth, remote writes, and background indexing remain separate deployment or connector-module concerns | +| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors | +| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs | +| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate | +| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation | +| Preview | File metadata and attachment download | Dedicated safe content-preview service | +| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign | +| Forms Runtime | One-time managed attachment grants plus exact-version, checksum, deletion, tenant, submission, and integrity verification | Malware scanning and advanced/qualified signature providers remain separate assurance depth | +| Collaboration | Governed input/output snapshots | Co-editing, comments, review, presence, locks, and semantic document versions belong to Documents/workflow/provider modules | + +## Release and change checklist + +Before releasing Files: + +1. Keep `pyproject.toml`, root `package.json`, WebUI `package.json`, and the + module manifest version aligned. Version alignment is a release gate. +2. Run the Files Python tests and security/static-analysis gates from the + GovOPlaN meta repository. +3. Build/type-check the WebUI through the Core host application. +4. Verify database migrations on an upgrade copy and a clean database. +5. Exercise an allowed and denied owner/share path. +6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion. +7. Exercise connector policy explanation and one pinned HTTP provider where + configured; verify the deployment-managed or trusted external S3 backend if + selected, and verify one configured S3 and SMB connector while recording the + actual target topology and private-network policy. +8. Verify credential deletion scrubs dependents and produces audit evidence. +9. Verify a campaign attachment snapshot still identifies its exact version and + checksum after the current file changes. +10. Exercise a committed upload, a rolled-back upload, object tamper detection, + and applied orphan cleanup; inspect their `files` operations in Ops. +11. Exercise a Form evidence grant, token replay, wrong-submission reference, + expired token, unsupported media type, and quarantined-file rejection. +12. Update the implemented/planned table whenever a boundary changes. + +## Related documents + +- [Repository overview](../README.md) +- [Connector ownership boundary](CONNECTOR_BOUNDARY.md) +- [Connector spaces design and implementation history](CONNECTOR_SPACES.md) +- [Document collaboration boundary](DOCUMENT_COLLABORATION_BOUNDARY.md) + +Where an older planning section conflicts with current code or this handbook's +implemented/planned table, verify the code and update both documents in the same +reviewable documentation slice.