From a7b5790cae613fa4713b8c90ab41030b45681e77 Mon Sep 17 00:00:00 2001 From: zemion Date: Tue, 21 Jul 2026 16:34:14 +0200 Subject: [PATCH] Sync Repo-docs-FILES-HANDBOOK from project files --- Repo-docs-FILES-HANDBOOK.-.md | 781 ++++++++++++++++++++++++++++++++++ 1 file changed, 781 insertions(+) create mode 100644 Repo-docs-FILES-HANDBOOK.-.md diff --git a/Repo-docs-FILES-HANDBOOK.-.md b/Repo-docs-FILES-HANDBOOK.-.md new file mode 100644 index 0000000..5862f4e --- /dev/null +++ b/Repo-docs-FILES-HANDBOOK.-.md @@ -0,0 +1,781 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-files/docs/FILES_HANDBOOK.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# GovOPlaN Files Handbook + +This handbook describes the Files module as implemented in version `0.1.9`. +It is the operational source of truth for users, process owners, administrators, +operators, auditors, and module integrators. Statements about future behavior +are marked **planned**; an unmarked statement describes the current code. + +Files is a governed snapshot store. It owns managed file content, versions, +logical folders, shares, source provenance, and the evidence that another +GovOPlaN module used a particular file version. It can browse selected external +stores read-only and import a frozen copy. It is not a general remote filesystem, +a document collaboration engine, or a records-management system. + +## Choose a reading path + +| If you need to... | Start with... | +| --- | --- | +| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) | +| Design a governed process that uses files | [Process perspective](#process-perspective) | +| Decide who can use a file or connector | [Ownership and access](#ownership-and-access) and [Administration and policy](#administration-and-policy) | +| Operate storage, connectors, backups, or recovery | [Operator runbook](#operator-runbook) | +| Integrate another GovOPlaN module | [Capabilities and integration](#capabilities-and-integration) | +| Review evidence, deletion, or security behavior | [Security, provenance, audit, deletion, and retention](#security-provenance-audit-deletion-and-retention) | +| Verify a release or scenario | [Acceptance scenarios](#acceptance-scenarios) | +| Check whether an idea exists today | [Implemented and planned boundary](#implemented-and-planned-boundary) | + +## The service contract + +A managed file is a tenant-scoped logical asset with exactly one user or group +owner, a normalized path, and a current version. The current version points to a +blob record containing the storage location, SHA-256 checksum, byte size, and +content type. Current service paths append a version when connector sync finds +changed content; they do not mutate the previous version record. + +Content with the same tenant, SHA-256 checksum, and size can reuse one blob. +This is storage deduplication and integrity evidence, not proof of authorship or +source authenticity. + +The main domain objects are: + +| Object | Meaning | Lifecycle today | +| --- | --- | --- | +| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted | +| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained | +| File blob | Stored bytes plus checksum, size, backend, key, reference count, and optional retention timestamp | Reused within a tenant; no automated garbage collection | +| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted | +| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet | +| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion | +| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion | +| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use | +| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted | +| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence | + +## User tasks + +The Files page is available at `/files`. Actions appear only when the current +principal has the required permission and resource access. + +### Choose a space + +Every file user sees **My files**. Group spaces are added for active groups of +which the user is a member. The space list also exposes every tenant group to a +Files administrator; governed API operations can administer other tenant-owned +Files resources when their owner is specified. Linked connector spaces appear +beside managed spaces when they are active and visible to the user. + +A connector space is intentionally read-only. It is a view of an approved +remote location and a starting point for importing or synchronizing selected +files into managed storage; it is not a mounted write-through filesystem. + +### Upload files + +The managed-space UI supports upload and drag-and-drop. A caller chooses the +user or group owner and destination folder. The normal per-file deployment +limit defaults to 50 MiB through `FILE_UPLOAD_MAX_BYTES`. + +When a target path already exists, the operation must use one of these conflict +strategies: + +- `reject` stops instead of silently replacing content; +- `rename` selects the next available `copy` name; +- `overwrite` soft-deletes the conflicting asset and creates the new asset; +- an item-specific conflict resolution can also `skip` that item. + +An ordinary upload does not append a new version to an existing asset. The +`overwrite` strategy retires the old asset at that path. Version-preserving +updates currently belong to connector sync. + +### Upload and unpack ZIP files + +The UI can unpack a ZIP upload. The request body is spooled to a bounded +temporary file rather than buffered wholly in memory. The defaults are: + +- 250 MiB compressed request and total extracted data; +- 50 MiB per extracted member; +- 1,000 non-directory members; +- encrypted archives are rejected; +- member paths are normalized and `..` traversal is rejected; +- the actual bytes read are counted, not only the ZIP header declarations. + +The byte limits use `FILE_UPLOAD_ZIP_MAX_BYTES` and +`FILE_UPLOAD_MAX_BYTES`. The 1,000-member limit is currently fixed in the +extraction service. + +### Organize files and folders + +With `files:file:organize`, a user can: + +- create logical folders; +- rename one selection directly; +- preview and apply bulk prefix, suffix, or replacement renames; +- move or copy files and folder trees between accessible user/group spaces; +- resolve target conflicts by rejecting, renaming, overwriting, or skipping; +- use drag-and-drop for move operations in the file explorer. + +Copies create new assets and versions while reusing the immutable blob bytes. +Moves keep the asset identity and change its owner/path. All source and target +owners are validated; knowing an identifier does not bypass space membership. + +Recursive folder deletion is the default. It soft-deletes the selected folder, +its child folders, and files below it. A non-recursive delete fails when the +folder is not empty. + +### Find and download files + +Files can list by owner and path, use cursor pagination, and consume incremental +changes through a watermark. The UI supports path/name pattern search and +sorting. The pattern API can resolve campaign-style wildcard selections and can +return unmatched files. + +A user with download permission and resource access can download one current +version or create a ZIP archive from a selection. Downloads use an attachment +content disposition. The archive is generated in a temporary file and removed +after the response completes. + +There is no dedicated content-preview service in the Files API today. File +responses expose content type, size, checksum, and current version metadata. + +### Share files + +The API can grant or update a share for a user, group, the tenant, or a campaign. +The supported permissions are `read`, `write`, and `manage`. Ownership remains +unchanged. A write operation accepts a `write` or `manage` share; read/download +accepts any of the three. + +The current Files page shows campaign linkage but does not offer a general +user/group share editor. The API also has no share-revocation route yet. Treat +share revocation and a complete share-management UI as planned work. + +### Browse and import an external file + +With a visible connector profile, a user can browse the allowed remote path, +select one file, and import it into an accessible managed space. The imported +asset records the connector, provider, remote identity/path/URL, selected remote +metadata, and source revision when supplied by the provider. + +Manual sync looks for an existing managed asset with the same source identity +inside the chosen owner space: + +- no match creates a managed asset; +- identical checksum and size updates provenance and returns `unchanged`; +- changed bytes append a version and return `updated`. + +Browse, import, and sync never write, rename, or delete the remote source. + +## Process perspective + +### Managed ingestion + +The managed-file flow is: + +1. Core authenticates the principal and evaluates the operation permission. +2. Files validates the tenant, owner, group membership, or applicable share. +3. Files normalizes the logical path and resolves conflicts explicitly. +4. Upload or connector response limits are enforced before content is retained. +5. Files calculates SHA-256 and stores or reuses a tenant blob. +6. Files creates the asset/version records and optional campaign share. +7. The database transaction commits and emits change-sequence entries. +8. Connector-originated operations also emit their connector audit evidence. + +Blob storage is not part of the database transaction. An object may therefore +be left without committed metadata after a process or database failure. No +automatic orphan reconciliation exists yet; operators must account for this in +integrity checks and retention plans. + +### Governed connector import + +The connector flow separates four concerns: + +1. An administrator defines reusable credential material. +2. An administrator defines a scoped endpoint profile that may reference that + credential. +3. System, tenant, and leaf policy sources narrow the allowed profile, + credential, provider, URL, and remote path. +4. A user optionally links an allowed remote root as a user/group connector + space, then browses and imports selected content. + +Before each network operation, Files checks profile visibility, connector +policy, endpoint safety, transport support, and response size. A successful +import becomes an independent managed snapshot. Later source changes have no +effect until an explicit sync. + +### Campaign attachment evidence + +When Campaign uses Files, the integration follows a freeze-before-send model: + +1. The campaign refers to managed user/group sources and attachment patterns. +2. Files verifies access and resolves matching managed assets. +3. A prepared campaign snapshot records the exact asset, version, blob, + checksum, size, relative path, and source provenance. +4. Files materializes those bytes for the campaign build without exposing its + database models to Campaign. +5. Campaign job/entry use is recorded and later marked as sent. + +Changing the current file after preparation does not change the version already +recorded as campaign evidence. A file response is marked `audit_relevant` once +the asset has a sent campaign attachment-use record. + +### Process ownership + +| Concern | Owner | +| --- | --- | +| Authentication, tenants, RBAC evaluation, audit service, change sequence, settings, and module lifecycle | GovOPlaN Core | +| Managed assets, blobs, versions, folders, shares, connector baseline, provenance, and campaign attachment evidence | Files | +| Campaign definition, recipient data, message build/send state, and delivery policy | Campaign | +| Remote ACLs, remote source content, and upstream revision semantics | The external provider | +| Storage durability, egress policy, master key, secret environment, backup, and recovery | Deployment operator | +| Collaborative editing, comments, review, locks, and semantic document workflows | A future Documents/workflow/provider module | + +## Ownership and access + +Files applies both permission checks and resource checks. A broad operation +permission alone does not make another user's file visible. + +### Resource access + +- A user owns their personal space. +- A group member can use the group's file space. +- A Files administrator can access all Files resources in the active tenant. +- A file share can grant read or write access to a user, group, or the tenant. +- Campaign shares are resolved only in a verified campaign context and do not + become ordinary user shares. +- Folders are owned by a user or group; they are not independently shared. +- Soft-deleted resources are excluded from normal access and listing. +- Tenant identifiers are checked on every managed object lookup. + +The `files.access` capability can explain why a principal has access: resource, +owner, administrator scope, or active share. It also explains virtual folders +that exist through child assets even when there is no explicit folder row. + +Deleting an organization/access group is vetoed while it owns Files assets, +folders, connector spaces, or is the target of file shares. Reassign or remove +those relationships first. + +### Operation permissions + +| Permission | Allows | +| --- | --- | +| `files:file:read` | List and inspect accessible files, folders, spaces, and visible connectors | +| `files:file:download` | Download an accessible current version or ZIP archive | +| `files:file:upload` | Upload managed assets and import/sync selected connector files | +| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces | +| `files:file:share` | Create or update file shares | +| `files:file:delete` | Soft-delete accessible writable files and folders | +| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant | + +The `file_manager` role template grants all normal file operations except +`files:file:admin`. The `file_viewer` template grants read and download. +System and tenant settings permissions can also authorize the corresponding +connector administration endpoints. + +## Administration and policy + +### Profiles, credentials, policies, and spaces + +Keep these definitions separate: + +- a **credential** holds reusable authentication material and may be restricted + to a provider; +- a **profile** holds the endpoint, scope, base path, credential reference, + local policy, and descriptive operation capabilities; +- a **policy** restricts what a scope may configure or use; +- a **connector space** links one approved profile/library/path to one user or + group and always uses manual, read-only synchronization today. + +Profile capability values such as `browse`, `import`, and `sync` are stored and +returned, but they are descriptive today. Provider implementation and policy +checks enforce actual availability; do not use the capability list as the sole +security control. + +Profiles, credentials, and policies support `system`, `tenant`, `user`, +`group`, and `campaign` scopes. A normal user sees system and active-tenant +profiles plus leaf profiles that match the user, one of their groups, or an +accessible campaign. Disabled definitions are visible only through authorized +administrative reads. + +### Policy evaluation + +For a leaf scope, the effective source chain is: + +```text +system -> tenant -> user | group | campaign +``` + +Policy fields are: + +- connector/profile IDs; +- credential IDs; +- providers; +- external IDs; +- external path prefixes or glob patterns; +- external URL glob patterns. + +Rules use `allow` and `deny` objects. Legacy synonyms `allowlist`, `whitelist`, +`denylist`, and `blacklist` are normalized. A matching deny at any source wins. +Every allow field defined by a source must match, so lower sources can narrow an +inherited set. The effective-policy response includes the contributing source +path and applied fields for explanation. + +A parent may set `allow_lower_level_limits` for individual fields such as +`allow.providers` or `deny.external_paths`. An explicit `false` prevents a lower +scope from configuring that field. Absence does not lock the field. + +Example tenant policy: + +```json +{ + "policy": { + "allow": { + "providers": ["webdav", "nextcloud"], + "external_urls": ["https://files.example.edu/*"], + "external_paths": ["departments/finance"] + }, + "deny": { + "external_paths": ["departments/finance/private"] + }, + "allow_lower_level_limits": { + "allow.providers": true, + "deny.external_paths": true + } + } +} +``` + +Use `POST /api/v1/files/connector-policy/evaluate` for an explainable preflight. +The normal profile browse/import/sync routes perform their own policy checks; +preflight does not replace enforcement. + +### Credential rules + +Database-created passwords and tokens use Core's Fernet encryption and require +the deployment `MASTER_KEY_B64` outside development/test/local environments. +Secret values, environment variable names, and local CA paths are never returned +in connector profile responses. + +API-managed profiles and credentials cannot select process environment +variables, create an external `secret_ref`, or conceal secret-like values in +nested metadata. Deployment-owned JSON/file profiles may use `password_env` or +`token_env` only when the exact environment variable name appears in +`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`. + +Custom CA bundles must be absolute existing files in +`GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST`. TLS verification can be disabled only +in a development/test runtime. + +### Provider status + +| Provider | Current status | +| --- | --- | +| Seafile | Read-only native API browse/download-link import and manual sync implemented using the pinned HTTP transport; WebDAV opt-in supported | +| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport | +| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport | +| SMB | Browse/import code and descriptor implemented, but every live connection fails closed until initial connections and DFS referrals can be policy-validated and pinned | +| S3 connector | Browse/import code and descriptor implemented, but every live SDK connection fails closed until botocore connections, redirects, and endpoint discovery can be validated and pinned | +| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned | +| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature | + +Provider descriptors are available from +`GET /api/v1/files/connectors/providers`. Use their `implemented`, `installed`, +and support fields for display, but expect a fail-closed transport error where +the table above says live access is disabled. + +## Operator runbook + +### Storage configuration + +| Setting | Default | Purpose | +| --- | --- | --- | +| `FILE_STORAGE_BACKEND` | `local` | Selects `local` or `s3` managed blob storage | +| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root | +| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root | +| `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket | +| `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and ZIP-member maximum | +| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | ZIP request and extracted-total maximum | +| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets | + +The local backend is the operational baseline. It resolves every storage key +under the configured root and rejects escape attempts. Fallback roots support a +controlled storage-root migration: new writes go to the primary root while +reads can still find older objects. + +The S3 managed-storage adapter currently fails closed before creating a boto3 +client because the SDK cannot yet guarantee connection-time DNS/IP pinning and +redirect revalidation. Do not select `FILE_STORAGE_BACKEND=s3` for a live Files +deployment until that boundary is implemented and the status above changes. + +Multiple API replicas require the same durable blob namespace. Separate local +container filesystems will produce incomplete reads. Until a pinned shared +object-storage transport is available, use one durable shared mount or constrain +Files traffic to a deployment topology that preserves one consistent local +root. + +### Connector egress + +Connector access to private networks is a deployment-wide decision: + +```text +GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true|false +``` + +Production-like configuration validation requires an explicit value. Public-only +mode rejects any hostname whose DNS answers include a non-public address. +Private-enabled mode still rejects link-local, multicast, unspecified, and +limited-broadcast addresses. + +The built-in HTTP transport: + +- resolves and validates every connection attempt; +- connects the socket to the exact approved address while retaining the + original hostname for HTTP Host, TLS SNI, and certificate verification; +- does not inherit proxy settings; +- refuses redirects instead of following a new peer implicitly; +- bounds structured responses to 16 MiB and file transfers to 512 MiB by + default. + +Override the connector response limits with +`GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` and +`GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES`. The smaller applicable limit wins +when an import is also subject to `FILE_UPLOAD_MAX_BYTES`. + +Never work around a pinning failure by adding a raw IP, disabling TLS, or +enabling private networks. SMB may redirect through DFS, and S3 SDKs may perform +their own redirects or endpoint discovery; both remain disabled even for an IP +literal until every connection peer can be governed. + +### Backup and restore + +The database and blob namespace are one logical backup set. A usable backup must +include: + +- Files database rows, including asset/version/blob relationships, shares, + connector settings, and campaign attachment-use evidence; +- every object below `FILE_STORAGE_LOCAL_ROOT` and any still-used fallback root; +- the exact `MASTER_KEY_B64` needed to decrypt retained connector credentials; +- deployment-owned connector profile files, referenced CA bundles, and secret + environment configuration where those definitions are in use. + +There is no Files backup/restore API and no automated blob integrity or orphan +reconciliation job. Use a write quiesce or coordinated snapshots so database +references and objects represent the same recovery point. + +After restore: + +1. Verify the active tenant and module migration state. +2. Verify the storage backend and roots before allowing writes. +3. Verify that the original master key is available before testing connectors. +4. Download representative files and compare bytes with their recorded SHA-256. +5. Test one authorized and one unauthorized owner/share path. +6. Test a permitted pinned HTTP connector, if connectors are configured. +7. Review audit and change-sequence continuity around the recovery point. + +An inconsistent restore should fail closed for missing objects or undecryptable +credentials. Do not repair it by deleting evidence rows without an approved, +audited data-recovery decision. + +### Disable, uninstall, and retire + +Disabling a module preserves its persistent data. Ordinary uninstall is guarded +while Files tables contain persistent rows. + +Destructive retirement is separate and irreversible at the application level. +The installer records a database snapshot, then Files scrubs and audits +remaining encrypted connector material before its database tables are dropped. +Legacy external secret references are detached and identified as non-owned; +Files never calls a provider delete operation for them. + +The retirement executor drops database tables but does not delete corresponding +objects from the configured blob backend. Operators must include those orphaned +objects in the approved retention/destruction plan. Validate the installer +snapshot and independent blob backup before retirement. + +### Operational signals + +Use these symptoms as routing hints: + +| Symptom | Likely boundary | +| --- | --- | +| `Stored object does not exist` | Database/blob restore mismatch, wrong root, or missing shared storage | +| `Stored secret cannot be decrypted` | Wrong or rotated `MASTER_KEY_B64` | +| Private/non-public endpoint blocked | Deployment-wide egress policy is public-only or DNS returned a forbidden answer | +| SDK cannot pin redirects/referrals | Expected fail-closed S3/SMB boundary, not a transient connector outage | +| Connector response exceeds limit | Remote payload exceeds connector or upload limit | +| Profile is not visible | Scope, disabled state, campaign access, or policy mismatch | +| Group removal is vetoed | The group still owns a file/folder/connector space or is a share target | + +## Capabilities and integration + +Other modules must integrate through Core contracts, Files capabilities, or the +public HTTP API. They must not import Files ORM models or storage helpers. + +### Provided capabilities + +| Capability | Purpose | +| --- | --- | +| `files.access` (`0.1.6`) | Explain resource access provenance for managed files, explicit folders, and virtual folders | +| `files.campaign_attachments` (`0.1.6`) | Resolve managed attachment matches, prepare frozen campaign snapshots, annotate built messages, share assets with a campaign, and record/mark exact attachment use | + +Files requires Core principal resolution and permission evaluation. Campaign is +an optional dependency; when installed, Files consumes the optional +`campaigns.access` interface to verify campaign existence and access. Missing +optional Campaign support fails explicitly rather than bypassing the check. + +### API families + +All routes below are under `/api/v1/files`. + +| Area | Routes | +| --- | --- | +| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` | +| Upload and folders | `POST /upload`, `POST /upload-zip`, `POST /folders`, `POST /folders/delete` | +| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` | +| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` | +| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` | +| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` | +| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` | +| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` | +| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` | +| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` | +| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` | +| Incremental connector settings | `GET /connectors/settings/delta` | + +Consumers should use cursor/watermark contracts instead of assuming an +unbounded complete list. The default full-list page size is 500 and public page +sizes are capped at 1,000. + +### Integration invariants + +An integrating module should: + +- ask Files/Core for access rather than trusting a submitted file ID; +- retain an exact version/blob/checksum at every governed evidence point; +- import external content before using it in a campaign, report, workflow, or + generated document; +- preserve Files provenance when producing a derived managed snapshot; +- treat source provenance as captured context, not cryptographic attestation; +- avoid writing remote providers through the baseline connector layer; +- define a separate capability when it needs behavior beyond managed snapshots; +- tolerate Files being absent when the integration is declared optional. + +Future Postbox, Templates/Reports, BI, Documents, DMS, and workflow modules +should keep their own domain state and use Files for governed input/output +snapshots. Long-running provider sync, OAuth, remote mutation, and +provider-specific health belong in connector modules rather than expanding the +Files baseline indiscriminately. + +## Security, provenance, audit, deletion, and retention + +### Security controls implemented + +- Core authentication, tenant scoping, CSRF/API handling, and RBAC protect the + public routes. +- Owner and share checks protect individual resources after operation-scope + checks. +- Logical paths reject traversal and storage keys cannot escape the local root. +- Upload, ZIP extraction, connector response, and S3 stream code use bounded + reads; encrypted ZIP archives are rejected. +- Connector HTTP sockets use connection-time DNS/IP validation and pinning, + redirects are refused, and unsafe SDK transports fail before client creation. +- Database-managed connector passwords/tokens are encrypted; responses redact + secrets and deployment references. +- API metadata is recursively checked for secret-like values. +- Downloads use sanitized attachment filenames. +- SHA-256 and byte size are recorded for every blob/version. + +The module does **not** currently provide malware scanning, content disarm and +reconstruction, a file-type allowlist, per-user quota, at-rest encryption for +local blob bytes, or a dedicated preview sandbox. Deployments that require +these controls must supply them outside Files until explicit module contracts +exist. + +### Provenance + +Connector-originated assets can retain: + +- source type; +- connector/profile ID and provider; +- external ID, path, and URL; +- revision and revision label; +- observation/import timestamps when supplied; +- selected provider metadata. + +The normalized provenance and source revision are returned in file responses, +carried into campaign attachment matches, and included in connector audit +events. External metadata is provider/user input and is not a digital signature. + +### Audit and change evidence + +Files records canonical audit events for: + +- connector discovery attempts, before the attempted external I/O; +- connector imports and manual syncs; +- download/archive access to connector-originated managed files; +- immediate connector profile and credential deletion/scrubbing; +- credential scrubbing during destructive module retirement. + +Connector audit details include the managed asset/version/blob, checksum, size, +operation, source revision, and provenance where applicable. Deletion audit +details name secret/reference kinds but never the secret values. + +Assets, folders, shares, profiles, credentials, policies, and connector spaces +also feed Core's incremental change sequence for UI synchronization. A change +entry is not equivalent to a canonical audit event. Ordinary local upload, +rename, share, and soft-delete operations do not yet all emit dedicated Files +audit events. + +Campaign attachment-use records provide separate domain evidence for an exact +file version used in campaign preparation and delivery. + +### Deletion semantics + +The word "delete" has different meanings by object type: + +| Object | Current delete behavior | +| --- | --- | +| File asset | Sets `deleted_at`; content, versions, blob references, and campaign evidence remain | +| Folder | Sets `deleted_at`; recursive deletion also soft-deletes descendants | +| Connector space | Sets `deleted_at` and disables the space | +| Connector credential | Immediately disables the tombstone and clears username, encrypted password/token, environment references, legacy external reference, mode, and private metadata; dependent profiles are disabled and detached | +| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata | +| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed | +| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility | + +Credential/profile scrubbing and its audit event use the same database +transaction. If audit creation fails, the deletion rolls back. Repeating a +delete against an already scrubbed tombstone does not recreate secret evidence. + +### Retention boundary + +File versions and blobs are effectively retained indefinitely today. Although a +blob has `ref_count` and `retained_until` fields, no complete retention-policy, +legal-hold, hard-purge, or garbage-collection service enforces them. There is +also no supported user restore endpoint for soft-deleted assets/folders and no +automated orphan-object reconciliation. + +Do not promise erasure, timed retention, legal hold, or self-service recovery +from the current soft-delete behavior. Those require an explicit, auditable +retention/purge design that preserves campaign and other evidence references. + +## Acceptance scenarios + +These scenarios describe expected behavior at the current boundary. + +### Personal and group ownership + +Given Alice has `files:file:upload` and belongs to Finance, when she uploads to +her personal or Finance space, then the asset has the selected owner and tenant. +Given Bob is outside Finance and has no share or Files admin permission, the +same asset ID must not make the asset readable to Bob. + +### Safe ZIP import + +Given a ZIP member contains `../../secret.txt`, is encrypted, exceeds 50 MiB, +or pushes actual extracted bytes above 250 MiB, unpacking must fail without a +committed managed asset. A valid archive must preserve normalized relative paths +below the chosen logical folder. + +### Explicit conflicts + +Given a target path already exists, `reject` must leave it unchanged, `rename` +must choose a non-conflicting path, and `overwrite` must soft-delete the old +asset before creating the replacement. A per-item `skip` must not create that +item. + +### Policy-denied connector + +Given the tenant permits WebDAV only below `departments/finance` and denies its +`private` child, a browse/import/sync request below the denied child must return +an explainable policy denial. Directly invoking the import endpoint must not +bypass the same rule. + +### Pinned connector transport + +Given public-only mode and DNS returns any private address, the connection must +be rejected before a socket opens. Given private mode, the HTTP connection must +still use the validated address and reject redirects. SMB and S3 must fail +before their SDK clients connect until all SDK-managed peers can be pinned. + +### Imported evidence and sync + +Given a permitted WebDAV file is imported, its managed response and audit event +must carry source identity, revision when available, current version, checksum, +and size. Re-syncing identical bytes must return `unchanged`; changed bytes must +create a higher version while preserving the previous version. + +### Campaign freeze + +Given a campaign snapshot selected version V1, when the managed asset later +advances to V2, the prepared/sent attachment-use evidence must still identify +V1 and its original blob/checksum. + +### Immediate credential deletion + +Given a database credential contains an encrypted password and is referenced by +two profiles, deleting it must scrub the credential, disable/detach both +profiles, record non-secret audit evidence, and publish connector-setting +changes in one transaction. If audit creation fails, no part of the deletion +may commit. + +### Recovery + +Given a coordinated database/blob backup and the original master key, restoring +it must allow representative downloads whose bytes match recorded SHA-256 +values. A missing blob or wrong key must surface an error rather than silently +returning different content or credentials. + +## Implemented and planned boundary + +| Area | Implemented now | Planned or explicitly outside the current boundary | +| --- | --- | --- | +| Managed storage | Local durable-root backend, fallback read roots, tenant blob deduplication, checksums | Operational S3 after pinned SDK transport; automated integrity/orphan reconciliation | +| Upload | Bounded direct upload, drag-and-drop UI, ZIP spool/extract limits, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload | +| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore | +| Sharing | User/group/tenant/campaign grant or permission update through API; campaign linkage display | Share revocation endpoint and complete general share-management UI | +| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, blob GC | +| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected | +| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers | +| SMB and S3 connectors | Provider descriptors and browse/import logic | Live access only after SDK connections plus DFS referrals/redirects are validated and pinned | +| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors | +| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs | +| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate | +| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation | +| Preview | File metadata and attachment download | Dedicated safe content-preview service | +| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign | +| Collaboration | Governed input/output snapshots | Co-editing, comments, review, presence, locks, and semantic document versions belong to Documents/workflow/provider modules | + +## Release and change checklist + +Before releasing Files: + +1. Keep `pyproject.toml`, root `package.json`, WebUI `package.json`, and the + module manifest version aligned. Version alignment is a release gate. +2. Run the Files Python tests and security/static-analysis gates from the + GovOPlaN meta repository. +3. Build/type-check the WebUI through the Core host application. +4. Verify database migrations on an upgrade copy and a clean database. +5. Exercise an allowed and denied owner/share path. +6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion. +7. Exercise connector policy explanation and one pinned HTTP provider where + configured; verify S3/SMB still fail closed. +8. Verify credential deletion scrubs dependents and produces audit evidence. +9. Verify a campaign attachment snapshot still identifies its exact version and + checksum after the current file changes. +10. Update the implemented/planned table whenever a boundary changes. + +## Related documents + +- [Repository overview](../README.md) +- [Connector ownership boundary](CONNECTOR_BOUNDARY.md) +- [Connector spaces design and implementation history](CONNECTOR_SPACES.md) +- [Document collaboration boundary](DOCUMENT_COLLABORATION_BOUNDARY.md) + +Where an older planning section conflicts with current code or this handbook's +implemented/planned table, verify the code and update both documents in the same +reviewable documentation slice.