20 Commits

Author SHA1 Message Date
835eacfc5d feat: harden file sharing and integrity 2026-07-30 14:26:47 +02:00
85606d5580 Add server-backed file property filters 2026-07-30 05:22:09 +02:00
632cc6cf7d perf(files): batch tenant summaries 2026-07-30 01:15:53 +02:00
86a905a3a7 refactor(api): split files workflow routers 2026-07-29 20:11:17 +02:00
5b868272b9 security: harden connector policy and add dashboard widget 2026-07-29 14:16:29 +02:00
93eea839c8 Declare file workspace View surfaces 2026-07-28 21:04:55 +02:00
b667e7ff0e Integrate file connectors with credential envelopes 2026-07-28 19:33:01 +02:00
0fd8f6e02a docs: update GovOPlaN repository links 2026-07-27 15:46:51 +02:00
2b34f6e305 fix(files): align adaptive tasks with live access 2026-07-21 19:15:47 +02:00
58af1a20a7 security(files): filter connectors before secret resolution 2026-07-21 19:15:41 +02:00
4722161592 feat(files): surface configured handbook tasks 2026-07-21 18:54:26 +02:00
1444ba80a0 refactor(files): centralize connector visibility 2026-07-21 18:52:33 +02:00
02ef83ecee test: commit Files retirement transaction 2026-07-21 18:18:33 +02:00
1069f85796 Refactor connector profile updates 2026-07-21 18:16:55 +02:00
1401c78c8a docs: add files workflow assurance topics 2026-07-21 17:13:53 +02:00
b6109245a7 docs(files): describe current sharing permission 2026-07-21 17:07:40 +02:00
f5d40b23c2 docs(files): link lifecycle reliability backlog 2026-07-21 16:48:13 +02:00
94ea629635 docs(files): require import workflow permissions 2026-07-21 16:33:06 +02:00
6c8a8c655d docs(files): expose adaptive handbook topics 2026-07-21 16:31:49 +02:00
062ad5ddfb docs(files): add adaptive operations handbook 2026-07-21 16:27:23 +02:00
60 changed files with 11815 additions and 3716 deletions

View File

@@ -137,6 +137,8 @@ manual interoperability testing.
Connector and collaboration ownership boundaries are documented in Connector and collaboration ownership boundaries are documented in
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`. `docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
The role-adaptive user, administration, integration, and operator guide is the
[Files handbook](docs/FILES_HANDBOOK.md).
ZIP uploads are processed without buffering the whole archive in memory. The API ZIP uploads are processed without buffering the whole archive in memory. The API
spools incoming ZIP request bodies to a bounded temporary file, then extracts spools incoming ZIP request bodies to a bounded temporary file, then extracts

806
docs/FILES_HANDBOOK.md Normal file
View File

@@ -0,0 +1,806 @@
# GovOPlaN Files Handbook
This handbook describes the Files module as implemented in version `0.1.9`.
It is the operational source of truth for users, process owners, administrators,
operators, auditors, and module integrators. Statements about future behavior
are marked **planned**; an unmarked statement describes the current code.
Files is a governed snapshot store. It owns managed file content, versions,
logical folders, shares, source provenance, and the evidence that another
GovOPlaN module used a particular file version. It can browse selected external
stores read-only and import a frozen copy. It is not a general remote filesystem,
a document collaboration engine, or a records-management system.
## Choose a reading path
| If you need to... | Start with... |
| --- | --- |
| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) |
| Design a governed process that uses files | [Process perspective](#process-perspective) |
| Decide who can use a file or connector | [Ownership and access](#ownership-and-access) and [Administration and policy](#administration-and-policy) |
| Operate storage, connectors, backups, or recovery | [Operator runbook](#operator-runbook) |
| Integrate another GovOPlaN module | [Capabilities and integration](#capabilities-and-integration) |
| Review evidence, deletion, or security behavior | [Security, provenance, audit, deletion, and retention](#security-provenance-audit-deletion-and-retention) |
| Verify a release or scenario | [Acceptance scenarios](#acceptance-scenarios) |
| Check whether an idea exists today | [Implemented and planned boundary](#implemented-and-planned-boundary) |
## The service contract
A managed file is a tenant-scoped logical asset with exactly one user or group
owner, a normalized path, and a current version. The current version points to a
blob record containing the storage location, SHA-256 checksum, byte size, and
content type. Current service paths append a version when connector sync finds
changed content; they do not mutate the previous version record.
Content with the same tenant, SHA-256 checksum, and size can reuse one blob.
This is storage deduplication and integrity evidence, not proof of authorship or
source authenticity.
The main domain objects are:
| Object | Meaning | Lifecycle today |
| --- | --- | --- |
| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted |
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained |
| File blob | Stored bytes plus checksum, size, backend, key, reference count, and optional retention timestamp | Reused within a tenant; no automated garbage collection |
| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted |
| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet |
| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion |
| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion |
| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use |
| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted |
| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence |
## User tasks
The Files page is available at `/files`. Actions appear only when the current
principal has the required permission and resource access.
The configured Help Center projects these sections as independently authorized
tasks, so upload, ZIP import, organization, download, sharing, and deletion do
not disappear merely because an unrelated permission is absent. It states the
deployment's actual upload limits. External import appears as a task only when
the actor has the required permissions and at least one actor-visible profile
is eligible for the current fail-closed browse/import path; endpoint, path, and
item policy are still enforced when the operation runs.
### Choose a space
Every file user sees **My files**. Group spaces are added for active groups of
which the user is a member. The space list also exposes every tenant group to a
Files administrator; governed API operations can administer other tenant-owned
Files resources when their owner is specified. Linked connector spaces appear
beside managed spaces when they are active and visible to the user.
A connector space is intentionally read-only. It is a view of an approved
remote location and a starting point for importing or synchronizing selected
files into managed storage; it is not a mounted write-through filesystem.
### Upload files
The managed-space UI supports upload and drag-and-drop. A caller chooses the
user or group owner and destination folder. The normal per-file deployment
limit defaults to 50 MiB through `FILE_UPLOAD_MAX_BYTES`.
When a target path already exists, the operation must use one of these conflict
strategies:
- `reject` stops instead of silently replacing content;
- `rename` selects the next available `copy` name;
- `overwrite` soft-deletes the conflicting asset and creates the new asset;
- an item-specific conflict resolution can also `skip` that item.
An ordinary upload does not append a new version to an existing asset. The
`overwrite` strategy retires the old asset at that path. Version-preserving
updates currently belong to connector sync.
### Upload and unpack ZIP files
The UI can unpack a ZIP upload. The request body is spooled to a bounded
temporary file rather than buffered wholly in memory. The defaults are:
- 250 MiB compressed request and total extracted data;
- 50 MiB per extracted member;
- 1,000 non-directory members;
- encrypted archives are rejected;
- member paths are normalized and `..` traversal is rejected;
- the actual bytes read are counted, not only the ZIP header declarations.
The byte limits use `FILE_UPLOAD_ZIP_MAX_BYTES` and
`FILE_UPLOAD_MAX_BYTES`. The 1,000-member limit is currently fixed in the
extraction service.
### Organize files and folders
With `files:file:organize`, a user can:
- create logical folders;
- rename one selection directly;
- preview and apply bulk prefix, suffix, or replacement renames;
- move or copy files and folder trees between accessible user/group spaces;
- resolve target conflicts by rejecting, renaming, overwriting, or skipping;
- use drag-and-drop for move operations in the file explorer.
Copies create new assets and versions while reusing the immutable blob bytes.
Moves keep the asset identity and change its owner/path. All source and target
owners are validated; knowing an identifier does not bypass space membership.
Recursive folder deletion is the default. It soft-deletes the selected folder,
its child folders, and files below it. A non-recursive delete fails when the
folder is not empty.
### Find and download files
Files can list by owner and path, use cursor pagination, and consume incremental
changes through a watermark. The UI supports path/name pattern search and
sorting. The pattern API can resolve campaign-style wildcard selections and can
return unmatched files.
A user with download permission and resource access can download one current
version or create a ZIP archive from a selection. Downloads use an attachment
content disposition. The archive is generated in a temporary file and removed
after the response completes.
There is no dedicated content-preview service in the Files API today. File
responses expose content type, size, checksum, and current version metadata.
### Share files
The API can grant or update a share for a user, group, the tenant, or a campaign.
The supported permissions are `read`, `write`, and `manage`. Ownership remains
unchanged. A write operation accepts a `write` or `manage` share; read/download
accepts any of the three.
The current Files page shows campaign linkage but does not offer a general
user/group share editor. The API also has no share-revocation route yet. Treat
share revocation and a complete share-management UI as planned work.
### Browse and import an external file
With a visible connector profile, a user can browse the allowed remote path,
select one file, and import it into an accessible managed space. The imported
asset records the connector, provider, remote identity/path/URL, selected remote
metadata, and source revision when supplied by the provider.
Manual sync looks for an existing managed asset with the same source identity
inside the chosen owner space:
- no match creates a managed asset;
- identical checksum and size updates provenance and returns `unchanged`;
- changed bytes append a version and return `updated`.
Browse, import, and sync never write, rename, or delete the remote source.
## Process perspective
### Managed ingestion
The managed-file flow is:
1. Core authenticates the principal and evaluates the operation permission.
2. Files validates the tenant, owner, group membership, or applicable share.
3. Files normalizes the logical path and resolves conflicts explicitly.
4. Upload or connector response limits are enforced before content is retained.
5. Files calculates SHA-256 and stores or reuses a tenant blob.
6. Files creates the asset/version records and optional campaign share.
7. The database transaction commits and emits change-sequence entries.
8. Connector-originated operations also emit their connector audit evidence.
Blob storage is not part of the database transaction. An object may therefore
be left without committed metadata after a process or database failure. The
operator integrity API scans database blobs and the tenant storage prefix in
bounded, resumable phases. It reports orphan objects before any cleanup and
never deletes them as part of a scan.
### Governed connector import
The connector flow separates four concerns:
1. An administrator defines reusable credential material.
2. An administrator defines a scoped endpoint profile that may reference that
credential.
3. System, tenant, and leaf policy sources narrow the allowed profile,
credential, provider, URL, and remote path.
4. A user optionally links an allowed remote root as a user/group connector
space, then browses and imports selected content.
Before each network operation, Files checks profile visibility, connector
policy, endpoint safety, transport support, and response size. A successful
import becomes an independent managed snapshot. Later source changes have no
effect until an explicit sync.
### Campaign attachment evidence
When Campaign uses Files, the integration follows a freeze-before-send model:
1. The campaign refers to managed user/group sources and attachment patterns.
2. Files verifies access and resolves matching managed assets.
3. A prepared campaign snapshot records the exact asset, version, blob,
checksum, size, relative path, and source provenance.
4. Files materializes those bytes for the campaign build without exposing its
database models to Campaign.
5. Campaign job/entry use is recorded and later marked as sent.
Changing the current file after preparation does not change the version already
recorded as campaign evidence. A file response is marked `audit_relevant` once
the asset has a sent campaign attachment-use record.
### Process ownership
| Concern | Owner |
| --- | --- |
| Authentication, tenants, RBAC evaluation, audit service, change sequence, settings, and module lifecycle | GovOPlaN Core |
| Managed assets, blobs, versions, folders, shares, connector baseline, provenance, and campaign attachment evidence | Files |
| Campaign definition, recipient data, message build/send state, and delivery policy | Campaign |
| Remote ACLs, remote source content, and upstream revision semantics | The external provider |
| Storage durability, egress policy, master key, secret environment, backup, and recovery | Deployment operator |
| Collaborative editing, comments, review, locks, and semantic document workflows | A future Documents/workflow/provider module |
## Ownership and access
Files applies both permission checks and resource checks. A broad operation
permission alone does not make another user's file visible.
### Resource access
- A user owns their personal space.
- A group member can use the group's file space.
- A Files administrator can access all Files resources in the active tenant.
- A file share can grant read or write access to a user, group, or the tenant.
- Campaign shares are resolved only in a verified campaign context and do not
become ordinary user shares.
- Folders are owned by a user or group; they are not independently shared.
- Soft-deleted resources are excluded from normal access and listing.
- Tenant identifiers are checked on every managed object lookup.
The `files.access` capability can explain why a principal has access: resource,
owner, administrator scope, or active share. It also explains virtual folders
that exist through child assets even when there is no explicit folder row.
Deleting an organization/access group is vetoed while it owns Files assets,
folders, connector spaces, or is the target of file shares. Reassign or remove
those relationships first.
### Operation permissions
| Permission | Allows |
| --- | --- |
| `files:file:read` | List and inspect accessible files, folders, spaces, and visible connectors |
| `files:file:download` | Download an accessible current version or ZIP archive |
| `files:file:upload` | Upload managed assets and import/sync selected connector files |
| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces |
| `files:file:share` | Create or update file shares |
| `files:file:delete` | Soft-delete accessible writable files and folders |
| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant |
The `file_manager` role template grants all normal file operations except
`files:file:admin`. The `file_viewer` template grants read and download.
System and tenant settings permissions can also authorize the corresponding
connector administration endpoints.
## Administration and policy
### Profiles, credentials, policies, and spaces
Keep these definitions separate:
- a **credential** holds reusable authentication material and may be restricted
to a provider;
- a **profile** holds the endpoint, scope, base path, credential reference,
local policy, and descriptive operation capabilities;
- a **policy** restricts what a scope may configure or use;
- a **connector space** links one approved profile/library/path to one user or
group and always uses manual, read-only synchronization today.
Profile capability values such as `browse`, `import`, and `sync` are stored and
returned, but they are descriptive today. Provider implementation and policy
checks enforce actual availability; do not use the capability list as the sole
security control.
Profiles, credentials, and policies support `system`, `tenant`, `user`,
`group`, and `campaign` scopes. A normal user sees system and active-tenant
profiles plus leaf profiles that match the user, one of their groups, or an
accessible campaign. Disabled definitions are visible only through authorized
administrative reads.
### Policy evaluation
For a leaf scope, the effective source chain is:
```text
system -> tenant -> user | group | campaign
```
Policy fields are:
- connector/profile IDs;
- credential IDs;
- providers;
- external IDs;
- external path prefixes or glob patterns;
- external URL glob patterns.
Rules use `allow` and `deny` objects. Legacy synonyms `allowlist`, `whitelist`,
`denylist`, and `blacklist` are normalized. A matching deny at any source wins.
Every allow field defined by a source must match, so lower sources can narrow an
inherited set. The effective-policy response includes the contributing source
path and applied fields for explanation.
A parent may set `allow_lower_level_limits` for individual fields such as
`allow.providers` or `deny.external_paths`. An explicit `false` prevents a lower
scope from configuring that field. Absence does not lock the field.
Example tenant policy:
```json
{
"policy": {
"allow": {
"providers": ["webdav", "nextcloud"],
"external_urls": ["https://files.example.edu/*"],
"external_paths": ["departments/finance"]
},
"deny": {
"external_paths": ["departments/finance/private"]
},
"allow_lower_level_limits": {
"allow.providers": true,
"deny.external_paths": true
}
}
}
```
Use `POST /api/v1/files/connector-policy/evaluate` for an explainable preflight.
The normal profile browse/import/sync routes perform their own policy checks;
preflight does not replace enforcement.
### Credential rules
Database-created passwords and tokens use Core's Fernet encryption and require
the deployment `MASTER_KEY_B64` outside development/test/local environments.
Secret values, environment variable names, and local CA paths are never returned
in connector profile responses.
API-managed profiles and credentials cannot select process environment
variables, create an external `secret_ref`, or conceal secret-like values in
nested metadata. Deployment-owned JSON/file profiles may use `password_env` or
`token_env` only when the exact environment variable name appears in
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`.
Custom CA bundles must be absolute existing files in
`GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST`. TLS verification can be disabled only
in a development/test runtime.
### Provider status
| Provider | Current status |
| --- | --- |
| Seafile | Read-only native API browse/download-link import and manual sync implemented using the pinned HTTP transport; WebDAV opt-in supported |
| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport |
| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport |
| SMB | Browse/import code and descriptor implemented, but every live connection fails closed until initial connections and DFS referrals can be policy-validated and pinned |
| S3 connector | Browse/import code and descriptor implemented, but every live SDK connection fails closed until botocore connections, redirects, and endpoint discovery can be validated and pinned |
| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned |
| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature |
Provider descriptors are available from
`GET /api/v1/files/connectors/providers`. Use their `implemented`, `installed`,
and support fields for display, but expect a fail-closed transport error where
the table above says live access is disabled.
## Operator runbook
### Storage configuration
| Setting | Default | Purpose |
| --- | --- | --- |
| `FILE_STORAGE_BACKEND` | `local` | Selects `local` or `s3` managed blob storage |
| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root |
| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root |
| `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket |
| `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and ZIP-member maximum |
| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | ZIP request and extracted-total maximum |
| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets |
The local backend is the operational baseline. It resolves every storage key
under the configured root and rejects escape attempts. Fallback roots support a
controlled storage-root migration: new writes go to the primary root while
reads can still find older objects.
The S3 managed-storage adapter currently fails closed before creating a boto3
client because the SDK cannot yet guarantee connection-time DNS/IP pinning and
redirect revalidation. Do not select `FILE_STORAGE_BACKEND=s3` for a live Files
deployment until that boundary is implemented and the status above changes.
Multiple API replicas require the same durable blob namespace. Separate local
container filesystems will produce incomplete reads. Until a pinned shared
object-storage transport is available, use one durable shared mount or constrain
Files traffic to a deployment topology that preserves one consistent local
root.
### Connector egress
Connector access to private networks is a deployment-wide decision:
```text
GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true|false
```
Production-like configuration validation requires an explicit value. Public-only
mode rejects any hostname whose DNS answers include a non-public address.
Private-enabled mode still rejects link-local, multicast, unspecified, and
limited-broadcast addresses.
The built-in HTTP transport:
- resolves and validates every connection attempt;
- connects the socket to the exact approved address while retaining the
original hostname for HTTP Host, TLS SNI, and certificate verification;
- does not inherit proxy settings;
- refuses redirects instead of following a new peer implicitly;
- bounds structured responses to 16 MiB and file transfers to 512 MiB by
default.
Override the connector response limits with
`GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` and
`GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES`. The smaller applicable limit wins
when an import is also subject to `FILE_UPLOAD_MAX_BYTES`.
Never work around a pinning failure by adding a raw IP, disabling TLS, or
enabling private networks. SMB may redirect through DFS, and S3 SDKs may perform
their own redirects or endpoint discovery; both remain disabled even for an IP
literal until every connection peer can be governed.
### Backup and restore
The database and blob namespace are one logical backup set. A usable backup must
include:
- Files database rows, including asset/version/blob relationships, shares,
connector settings, and campaign attachment-use evidence;
- every object below `FILE_STORAGE_LOCAL_ROOT` and any still-used fallback root;
- the exact `MASTER_KEY_B64` needed to decrypt retained connector credentials;
- deployment-owned connector profile files, referenced CA bundles, and secret
environment configuration where those definitions are in use.
There is no Files backup/restore API. Use a write quiesce or coordinated
snapshots so database references and objects represent the same recovery point.
The integrity API verifies a restored set, but it does not replace a coordinated
backup.
Create a scan with `POST /api/v1/files/integrity/scans`, then call
`POST /api/v1/files/integrity/scans/{scan_id}/run` until it reports
`completed`. Each call advances at most the persisted batch size, so a stopped
operator or worker can resume from the committed blob/object cursors.
Findings distinguish:
- `missing`: metadata references an absent object;
- `size_mismatch` or `checksum_mismatch`: bytes do not match immutable blob
metadata and the blob is quarantined;
- `orphan_object`: an object exists in the tenant Files prefix without a
corresponding blob row.
Missing or corrupt blobs fail closed for ordinary downloads and Campaign
attachment materialization. After restoring the expected bytes, use the finding
`recheck` action first in dry-run mode and then apply it. Orphan cleanup is also
dry-run by default, rechecks that no database reference exists, remains scoped
to the scanned tenant prefix, and is idempotent. Both applied and dry-run
actions emit audit evidence.
After restore:
1. Verify the active tenant and module migration state.
2. Verify the storage backend and roots before allowing writes.
3. Verify that the original master key is available before testing connectors.
4. Download representative files and compare bytes with their recorded SHA-256.
5. Test one authorized and one unauthorized owner/share path.
6. Test a permitted pinned HTTP connector, if connectors are configured.
7. Review audit and change-sequence continuity around the recovery point.
An inconsistent restore should fail closed for missing objects or undecryptable
credentials. Do not repair it by deleting evidence rows without an approved,
audited data-recovery decision.
### Disable, uninstall, and retire
Disabling a module preserves its persistent data. Ordinary uninstall is guarded
while Files tables contain persistent rows.
Destructive retirement is separate and irreversible at the application level.
The installer records a database snapshot, then Files scrubs and audits
remaining encrypted connector material before its database tables are dropped.
Legacy external secret references are detached and identified as non-owned;
Files never calls a provider delete operation for them.
The retirement executor drops database tables but does not delete corresponding
objects from the configured blob backend. Operators must include those objects
in the approved retention/destruction plan, report them with an integrity scan,
and explicitly approve cleanup. Validate the installer snapshot and independent
blob backup before retirement.
### Operational signals
Use these symptoms as routing hints:
| Symptom | Likely boundary |
| --- | --- |
| `Stored object does not exist` | Database/blob restore mismatch, wrong root, or missing shared storage |
| `Stored secret cannot be decrypted` | Wrong or rotated `MASTER_KEY_B64` |
| Private/non-public endpoint blocked | Deployment-wide egress policy is public-only or DNS returned a forbidden answer |
| SDK cannot pin redirects/referrals | Expected fail-closed S3/SMB boundary, not a transient connector outage |
| Connector response exceeds limit | Remote payload exceeds connector or upload limit |
| Profile is not visible | Scope, disabled state, campaign access, or policy mismatch |
| Group removal is vetoed | The group still owns a file/folder/connector space or is a share target |
## Capabilities and integration
Other modules must integrate through Core contracts, Files capabilities, or the
public HTTP API. They must not import Files ORM models or storage helpers.
### Provided capabilities
| Capability | Purpose |
| --- | --- |
| `files.access` (`0.1.6`) | Explain resource access provenance for managed files, explicit folders, and virtual folders |
| `files.campaign_attachments` (`0.1.6`) | Resolve managed attachment matches, prepare frozen campaign snapshots, annotate built messages, share assets with a campaign, and record/mark exact attachment use |
Files requires Core principal resolution and permission evaluation. Campaign is
an optional dependency; when installed, Files consumes the optional
`campaigns.access` interface to verify campaign existence and access. Missing
optional Campaign support fails explicitly rather than bypassing the check.
### API families
All routes below are under `/api/v1/files`.
| Area | Routes |
| --- | --- |
| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` |
| Upload and folders | `POST /upload`, `POST /upload-zip`, `POST /folders`, `POST /folders/delete` |
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` |
| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` |
| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` |
| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` |
| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` |
| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` |
| Incremental connector settings | `GET /connectors/settings/delta` |
Consumers should use cursor/watermark contracts instead of assuming an
unbounded complete list. The default full-list page size is 500 and public page
sizes are capped at 1,000.
### Integration invariants
An integrating module should:
- ask Files/Core for access rather than trusting a submitted file ID;
- retain an exact version/blob/checksum at every governed evidence point;
- import external content before using it in a campaign, report, workflow, or
generated document;
- preserve Files provenance when producing a derived managed snapshot;
- treat source provenance as captured context, not cryptographic attestation;
- avoid writing remote providers through the baseline connector layer;
- define a separate capability when it needs behavior beyond managed snapshots;
- tolerate Files being absent when the integration is declared optional.
Future Postbox, Templates/Reports, BI, Documents, DMS, and workflow modules
should keep their own domain state and use Files for governed input/output
snapshots. Long-running provider sync, OAuth, remote mutation, and
provider-specific health belong in connector modules rather than expanding the
Files baseline indiscriminately.
## Security, provenance, audit, deletion, and retention
### Security controls implemented
- Core authentication, tenant scoping, CSRF/API handling, and RBAC protect the
public routes.
- Owner and share checks protect individual resources after operation-scope
checks.
- Logical paths reject traversal and storage keys cannot escape the local root.
- Upload, ZIP extraction, connector response, and S3 stream code use bounded
reads; encrypted ZIP archives are rejected.
- Connector HTTP sockets use connection-time DNS/IP validation and pinning,
redirects are refused, and unsafe SDK transports fail before client creation.
- Database-managed connector passwords/tokens are encrypted; responses redact
secrets and deployment references.
- API metadata is recursively checked for secret-like values.
- Downloads use sanitized attachment filenames.
- SHA-256 and byte size are recorded for every blob/version.
The module does **not** currently provide malware scanning, content disarm and
reconstruction, a file-type allowlist, per-user quota, at-rest encryption for
local blob bytes, or a dedicated preview sandbox. Deployments that require
these controls must supply them outside Files until explicit module contracts
exist.
### Provenance
Connector-originated assets can retain:
- source type;
- connector/profile ID and provider;
- external ID, path, and URL;
- revision and revision label;
- observation/import timestamps when supplied;
- selected provider metadata.
The normalized provenance and source revision are returned in file responses,
carried into campaign attachment matches, and included in connector audit
events. External metadata is provider/user input and is not a digital signature.
### Audit and change evidence
Files records canonical audit events for:
- connector discovery attempts, before the attempted external I/O;
- connector imports and manual syncs;
- download/archive access to connector-originated managed files;
- immediate connector profile and credential deletion/scrubbing;
- credential scrubbing during destructive module retirement.
Connector audit details include the managed asset/version/blob, checksum, size,
operation, source revision, and provenance where applicable. Deletion audit
details name secret/reference kinds but never the secret values.
Assets, folders, shares, profiles, credentials, policies, and connector spaces
also feed Core's incremental change sequence for UI synchronization. A change
entry is not equivalent to a canonical audit event. Ordinary local upload,
rename, share, and soft-delete operations do not yet all emit dedicated Files
audit events.
Campaign attachment-use records provide separate domain evidence for an exact
file version used in campaign preparation and delivery.
### Deletion semantics
The word "delete" has different meanings by object type:
| Object | Current delete behavior |
| --- | --- |
| File asset | Sets `deleted_at`; content, versions, blob references, and campaign evidence remain |
| Folder | Sets `deleted_at`; recursive deletion also soft-deletes descendants |
| Connector space | Sets `deleted_at` and disables the space |
| Connector credential | Immediately disables the tombstone and clears username, encrypted password/token, environment references, legacy external reference, mode, and private metadata; dependent profiles are disabled and detached |
| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata |
| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed |
| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility |
Credential/profile scrubbing and its audit event use the same database
transaction. If audit creation fails, the deletion rolls back. Repeating a
delete against an already scrubbed tombstone does not recreate secret evidence.
### Retention boundary
File versions and blobs are effectively retained indefinitely today. Although a
blob has `ref_count` and `retained_until` fields, no complete retention-policy,
legal-hold, hard-purge, or garbage-collection service enforces them. There is
also no supported user restore endpoint for soft-deleted assets/folders.
Integrity reconciliation is operator-triggered and is not a retention or
automatic garbage-collection policy.
Do not promise erasure, timed retention, legal hold, or self-service recovery
from the current soft-delete behavior. Those require an explicit, auditable
retention/purge design that preserves campaign and other evidence references.
## Acceptance scenarios
These scenarios describe expected behavior at the current boundary.
### Personal and group ownership
Given Alice has `files:file:upload` and belongs to Finance, when she uploads to
her personal or Finance space, then the asset has the selected owner and tenant.
Given Bob is outside Finance and has no share or Files admin permission, the
same asset ID must not make the asset readable to Bob.
### Safe ZIP import
Given a ZIP member contains `../../secret.txt`, is encrypted, exceeds 50 MiB,
or pushes actual extracted bytes above 250 MiB, unpacking must fail without a
committed managed asset. A valid archive must preserve normalized relative paths
below the chosen logical folder.
### Explicit conflicts
Given a target path already exists, `reject` must leave it unchanged, `rename`
must choose a non-conflicting path, and `overwrite` must soft-delete the old
asset before creating the replacement. A per-item `skip` must not create that
item.
### Policy-denied connector
Given the tenant permits WebDAV only below `departments/finance` and denies its
`private` child, a browse/import/sync request below the denied child must return
an explainable policy denial. Directly invoking the import endpoint must not
bypass the same rule.
### Pinned connector transport
Given public-only mode and DNS returns any private address, the connection must
be rejected before a socket opens. Given private mode, the HTTP connection must
still use the validated address and reject redirects. SMB and S3 must fail
before their SDK clients connect until all SDK-managed peers can be pinned.
### Imported evidence and sync
Given a permitted WebDAV file is imported, its managed response and audit event
must carry source identity, revision when available, current version, checksum,
and size. Re-syncing identical bytes must return `unchanged`; changed bytes must
create a higher version while preserving the previous version.
### Campaign freeze
Given a campaign snapshot selected version V1, when the managed asset later
advances to V2, the prepared/sent attachment-use evidence must still identify
V1 and its original blob/checksum.
### Immediate credential deletion
Given a database credential contains an encrypted password and is referenced by
two profiles, deleting it must scrub the credential, disable/detach both
profiles, record non-secret audit evidence, and publish connector-setting
changes in one transaction. If audit creation fails, no part of the deletion
may commit.
### Recovery
Given a coordinated database/blob backup and the original master key, restoring
it must allow representative downloads whose bytes match recorded SHA-256
values. A missing blob or wrong key must surface an error rather than silently
returning different content or credentials.
## Implemented and planned boundary
| Area | Implemented now | Planned or explicitly outside the current boundary |
| --- | --- | --- |
| Managed storage | Local durable-root backend, fallback read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, and dry-run-first orphan cleanup | Operational S3 after pinned SDK transport ([#34](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/34)); scheduled scan execution |
| Upload | Bounded direct upload, drag-and-drop UI, ZIP spool/extract limits, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload |
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected |
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers |
| SMB and S3 connectors | Provider descriptors and browse/import logic | Live access only after SDK connections plus DFS referrals/redirects are validated and pinned ([SMB #35](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/35), [S3 #34](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/34)) |
| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors |
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs |
| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate |
| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation |
| Preview | File metadata and attachment download | Dedicated safe content-preview service |
| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign |
| Collaboration | Governed input/output snapshots | Co-editing, comments, review, presence, locks, and semantic document versions belong to Documents/workflow/provider modules |
## Release and change checklist
Before releasing Files:
1. Keep `pyproject.toml`, root `package.json`, WebUI `package.json`, and the
module manifest version aligned. Version alignment is a release gate.
2. Run the Files Python tests and security/static-analysis gates from the
GovOPlaN meta repository.
3. Build/type-check the WebUI through the Core host application.
4. Verify database migrations on an upgrade copy and a clean database.
5. Exercise an allowed and denied owner/share path.
6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion.
7. Exercise connector policy explanation and one pinned HTTP provider where
configured; verify S3/SMB still fail closed.
8. Verify credential deletion scrubs dependents and produces audit evidence.
9. Verify a campaign attachment snapshot still identifies its exact version and
checksum after the current file changes.
10. Update the implemented/planned table whenever a boundary changes.
## Related documents
- [Repository overview](../README.md)
- [Connector ownership boundary](CONNECTOR_BOUNDARY.md)
- [Connector spaces design and implementation history](CONNECTOR_SPACES.md)
- [Document collaboration boundary](DOCUMENT_COLLABORATION_BOUNDARY.md)
Where an older planning section conflicts with current code or this handbook's
implemented/planned table, verify the code and update both documents in the same
reviewable documentation slice.

View File

@@ -22,6 +22,7 @@ from govoplan_files.backend.storage.campaign_attachments import (
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
from govoplan_files.backend.storage.files import current_version_and_blob from govoplan_files.backend.storage.files import current_version_and_blob
from govoplan_files.backend.storage.paths import normalize_folder from govoplan_files.backend.storage.paths import normalize_folder
from govoplan_files.backend.storage.share_state import effective_file_share_clause
VIRTUAL_FOLDER_RESOURCE_PREFIX = "virtual-folder:v1" VIRTUAL_FOLDER_RESOURCE_PREFIX = "virtual-folder:v1"
@@ -96,7 +97,7 @@ class FilesAccessService(FileAccessProvider):
.filter( .filter(
FileShare.tenant_id == asset.tenant_id, FileShare.tenant_id == asset.tenant_id,
FileShare.file_asset_id == asset.id, FileShare.file_asset_id == asset.id,
FileShare.revoked_at.is_(None), effective_file_share_clause(),
FileShare.permission.in_(sorted(permission_values)), FileShare.permission.in_(sorted(permission_values)),
or_( or_(
(FileShare.target_type == "user") & (FileShare.target_id == principal.membership_id), (FileShare.target_type == "user") & (FileShare.target_id == principal.membership_id),

View File

@@ -117,7 +117,15 @@ def _record_share_visibility_change(session: OrmSession, share: FileShare) -> No
return return
if not state.pending and not has_attr_changes( if not state.pending and not has_attr_changes(
state, state,
("file_asset_id", "target_type", "target_id", "permission", "revoked_at"), (
"file_asset_id",
"target_type",
"target_id",
"permission",
"expires_at",
"revoked_at",
"revoked_by_user_id",
),
): ):
return return
_ensure_id(share) _ensure_id(share)
@@ -136,7 +144,9 @@ def _record_share_visibility_change(session: OrmSession, share: FileShare) -> No
"share_target_type": share.target_type, "share_target_type": share.target_type,
"share_target_id": share.target_id, "share_target_id": share.target_id,
"share_permission": share.permission, "share_permission": share.permission,
"share_expires_at": _isoformat(share.expires_at),
"share_revoked_at": _isoformat(share.revoked_at), "share_revoked_at": _isoformat(share.revoked_at),
"share_revoked_by_user_id": share.revoked_by_user_id,
}, },
) )

View File

@@ -28,6 +28,55 @@ class FileBlob(Base, TimestampMixin):
content_type: Mapped[str | None] = mapped_column(String(255)) content_type: Mapped[str | None] = mapped_column(String(255))
ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False) ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
integrity_status: Mapped[str] = mapped_column(String(30), default="unchecked", nullable=False, index=True)
integrity_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
integrity_failure: Mapped[str | None] = mapped_column(String(100), nullable=True)
quarantined_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
class FileIntegrityScan(Base, TimestampMixin):
__tablename__ = "file_integrity_scans"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
storage_prefix: Mapped[str] = mapped_column(String(1000), nullable=False)
status: Mapped[str] = mapped_column(String(30), default="pending", nullable=False, index=True)
phase: Mapped[str] = mapped_column(String(30), default="blobs", nullable=False)
verify_checksums: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
batch_size: Mapped[int] = mapped_column(Integer, default=100, nullable=False)
blob_cursor: Mapped[str | None] = mapped_column(String(36), nullable=True)
object_cursor: Mapped[str | None] = mapped_column(String(1000), nullable=True)
scanned_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
verified_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
quarantined_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
scanned_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
orphan_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(String(255), nullable=True)
class FileIntegrityFinding(Base, TimestampMixin):
__tablename__ = "file_integrity_findings"
__table_args__ = (
Index("ix_file_integrity_findings_scan_state", "scan_id", "state"),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
scan_id: Mapped[str] = mapped_column(ForeignKey("file_integrity_scans.id", ondelete="CASCADE"), nullable=False, index=True)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
kind: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
state: Mapped[str] = mapped_column(String(30), default="open", nullable=False, index=True)
blob_id: Mapped[str | None] = mapped_column(ForeignKey("file_blobs.id", ondelete="SET NULL"), nullable=True, index=True)
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
expected_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
observed_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
expected_checksum_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
observed_checksum_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
resolved_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
class FileFolder(Base, TimestampMixin): class FileFolder(Base, TimestampMixin):
@@ -105,7 +154,9 @@ class FileShare(Base, TimestampMixin):
target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False) permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False)
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True) created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True) revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
revoked_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True)
class FileConnectorProfile(Base, TimestampMixin): class FileConnectorProfile(Base, TimestampMixin):

View File

@@ -0,0 +1,461 @@
from __future__ import annotations
from sqlalchemy.orm import Session
from govoplan_core.core.campaigns import (
CAPABILITY_CAMPAIGNS_ACCESS,
CampaignAccessProvider,
)
from govoplan_core.core.modules import (
DocumentationCondition,
DocumentationContext,
DocumentationLink,
DocumentationTopic,
)
from govoplan_files.backend.storage.archives import ZIP_UPLOAD_MAX_FILES
from govoplan_files.backend.storage.access import user_group_ids
from govoplan_files.backend.storage.connector_visibility import (
connector_profile_usable_for_import,
visible_connector_profiles_for_actor,
)
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
_FILES_READ_SCOPE = "files:file:read"
_FILES_UPLOAD_SCOPE = "files:file:upload"
def documentation_topics(
context: DocumentationContext,
) -> tuple[DocumentationTopic, ...]:
if context.documentation_type != "user":
return ()
upload_limit = _configured_positive_int(
context.settings,
"file_upload_max_bytes",
default=_DEFAULT_UPLOAD_MAX_BYTES,
)
zip_limit = _configured_positive_int(
context.settings,
"file_upload_zip_max_bytes",
default=_DEFAULT_ZIP_MAX_BYTES,
)
topics: list[DocumentationTopic] = []
if upload_limit is not None:
topics.append(_upload_topic(upload_limit))
if upload_limit is not None and zip_limit is not None:
topics.append(_zip_topic(upload_limit, zip_limit))
topics.append(_connector_import_topic(context))
return tuple(topics)
def _upload_topic(max_bytes: int) -> DocumentationTopic:
limit = _format_byte_limit(max_bytes)
return DocumentationTopic(
id="files.workflow.upload-managed-files",
title="Upload managed files",
summary=f"Upload files to a personal or accessible group space; each uploaded file may contain at most {limit}.",
body=(
f"The current deployment accepts at most {limit} for each ordinary upload. "
"A name conflict is never resolved silently: reject stops the upload, rename chooses a copy name, and overwrite retires the old asset before creating a new one."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=39,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(
label="Files handbook",
href="govoplan-files/docs/FILES_HANDBOOK.md",
kind="repository",
),
),
related_modules=("campaigns",),
unlocks=(
"Users and connected processes can place governed content in managed storage.",
),
source_module_id="files",
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and upload managed files.",
"The destination personal or group space grants this account write access; upload permission alone does not grant access to every space.",
],
"steps": [
"Open Files and choose My files or an accessible group space.",
"Open the intended destination folder and choose Upload, or drag files into the file list.",
"For every name conflict, explicitly reject, rename, overwrite, or skip the affected item.",
"Wait for the upload to finish, then open the resulting file details.",
],
"outcome": "Each accepted file is stored as a governed managed asset in the selected space.",
"verification": "Confirm the owner, logical path, size, checksum, and current version in Files.",
"constraints": [
{
"id": "ordinary-upload-size",
"label": "Maximum size per file",
"description": f"The current safe upload limit is {limit} per file.",
"values": [limit],
},
],
"related_topic_ids": [
"files.workflow.upload-and-unpack-zip",
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
],
},
)
def _zip_topic(max_file_bytes: int, max_zip_bytes: int) -> DocumentationTopic:
member_limit = _format_byte_limit(max_file_bytes)
archive_limit = _format_byte_limit(max_zip_bytes)
return DocumentationTopic(
id="files.workflow.upload-and-unpack-zip",
title="Upload and unpack a ZIP archive",
summary=(
f"Safely unpack up to {ZIP_UPLOAD_MAX_FILES:,} files from a ZIP whose request and extracted total are each limited to {archive_limit}."
),
body=(
f"The current deployment limits the ZIP request and actual extracted total to {archive_limit}, each member to {member_limit}, "
f"and the archive to {ZIP_UPLOAD_MAX_FILES:,} non-directory members. Encrypted archives and unsafe member paths are rejected. "
"Actual extracted bytes are counted instead of trusting ZIP headers."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=40,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(
label="Files handbook",
href="govoplan-files/docs/FILES_HANDBOOK.md",
kind="repository",
),
),
unlocks=(
"A bounded archive can become governed managed files without trusting archive paths or size declarations.",
),
source_module_id="files",
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and upload managed files.",
"The archive is not encrypted and fits the current configured limits.",
"The destination personal or group space grants this account write access.",
],
"steps": [
"Open Files and choose the managed destination space and folder.",
"Enable Unpack ZIP uploads, then choose or drag the ZIP archive.",
"Resolve every destination conflict explicitly.",
"Wait for extraction and finalization to finish before leaving the page.",
],
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
"constraints": [
{
"id": "zip-request-and-total",
"label": "Maximum ZIP request and extracted total",
"description": f"Both the compressed request and the actual extracted total are limited to {archive_limit}.",
"values": [archive_limit],
},
{
"id": "zip-member-size",
"label": "Maximum extracted member size",
"description": f"Each extracted file is limited to {member_limit}.",
"values": [member_limit],
},
{
"id": "zip-member-count",
"label": "Maximum file count",
"description": f"A ZIP may contain at most {ZIP_UPLOAD_MAX_FILES:,} non-directory members.",
"values": [f"{ZIP_UPLOAD_MAX_FILES:,} files"],
},
],
"related_topic_ids": [
"files.workflow.upload-managed-files",
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
],
},
)
def _connector_import_topic(context: DocumentationContext) -> DocumentationTopic:
principal = context.principal
if not _has_all_scopes(principal, (_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE)):
return _connector_import_limitation(
"Connector import is not available to this account because both permission to view Files and permission to upload managed files are required."
)
tenant_id = _safe_text_attribute(principal, "tenant_id")
user_id = _safe_text_attribute(getattr(principal, "user", None), "id")
session = context.session
if not tenant_id or not user_id or not isinstance(session, Session):
return _connector_import_limitation(
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
)
try:
member_group_ids = _actor_group_ids(
session,
principal=principal,
tenant_id=tenant_id,
user_id=user_id,
include_admin_groups=False,
)
connector_group_ids = (
_actor_group_ids(
session,
principal=principal,
tenant_id=tenant_id,
user_id=user_id,
include_admin_groups=True,
)
if _has_all_scopes(principal, ("files:file:admin",))
else member_group_ids
)
profiles = visible_connector_profiles_for_actor(
session,
tenant_id=tenant_id,
user_id=user_id,
group_ids=connector_group_ids,
settings=context.settings,
campaign_visible=_campaign_visibility(
context,
session=session,
tenant_id=tenant_id,
user_id=user_id,
group_ids=member_group_ids,
),
include_effective_policy=True,
)
usable_profiles = tuple(
profile
for profile in profiles
if connector_profile_usable_for_import(profile)
)
except Exception:
return _connector_import_limitation(
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
)
if not usable_profiles:
return _connector_import_limitation(
"No connection visible to this account is currently eligible to offer an enabled, credential-ready, policy-allowed browse/import path through a pinning-safe provider. Ask a Files administrator to configure or authorize one."
)
return DocumentationTopic(
id="files.workflow.import-managed-snapshot",
title="Import an external file as a governed snapshot",
summary="Browse an authorized connection read-only and import one selected file into managed storage as a frozen, traceable snapshot.",
body=(
"At least one connection visible to this account is currently eligible to offer the governed browse/import path. "
"The selected remote path and item are re-authorized when used, and browse, import, and sync never mutate the remote source. "
"The managed snapshot stays unchanged until an explicit manual sync."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "campaign_manager", "report_author"),
order=41,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(
label="Files handbook",
href="govoplan-files/docs/FILES_HANDBOOK.md",
kind="repository",
),
),
related_modules=("campaigns",),
unlocks=(
"Campaigns, reports, and workflows can consume a managed snapshot with stable source evidence.",
),
source_module_id="files",
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list", "files.connector-import"],
"prerequisites": [
"You may view and upload managed files.",
"At least one enabled, credential-ready, policy-allowed connection using a pinning-safe provider is visible to this account.",
"The selected remote path and item must pass their operation-time policy checks.",
"The managed destination space grants this account write access.",
],
"steps": [
"Open Files and choose a managed destination space.",
"Choose Sync from connection, select an available connection, and browse to the permitted remote file.",
"Import the selected file and resolve any destination conflict explicitly.",
"Review the managed file's source and current-version details before using it in another task.",
],
"current_configuration": [
"At least one actor-visible connection is eligible to offer a safe browse/import operation; the selected endpoint, path, and item are still checked when used.",
"Every selected remote path and item is re-authorized at operation time.",
],
"outcome": "The external content is a tenant-managed snapshot with a checksum, exact version, and recorded source context.",
"verification": "Reopen the managed file and confirm its recorded source context, source revision when available, checksum, and current version.",
"related_topic_ids": [
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
"files.reference.snapshot-provenance-and-capabilities",
],
},
)
def _connector_import_limitation(message: str) -> DocumentationTopic:
return DocumentationTopic(
id="files.connector-import-unavailable",
title="External file import is not currently available",
summary=message,
body=(
f"{message} Files never exposes connection endpoints, storage paths, credential references, or raw connector policies in user documentation."
),
layer="available",
documentation_types=("user",),
order=41,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(DocumentationLink(label="Files", href="/files", kind="runtime"),),
source_module_id="files",
metadata={
"kind": "reference",
"screen": "Files",
"help_contexts": ["files.list", "files.connector-import"],
"limitations": [message],
},
)
def _configured_positive_int(
settings: object | None, name: str, *, default: int
) -> int | None:
raw_value = getattr(settings, name, default) if settings is not None else default
try:
value = int(raw_value)
except (TypeError, ValueError):
return None
return value if value > 0 else None
def _format_byte_limit(value: int) -> str:
units = ((1024**3, "GiB"), (1024**2, "MiB"), (1024, "KiB"))
for divisor, label in units:
if value % divisor == 0:
return f"{value // divisor:,} {label} ({value:,} bytes)"
return f"{value:,} bytes"
def _has_all_scopes(principal: object | None, scopes: tuple[str, ...]) -> bool:
checker = getattr(principal, "has", None)
if callable(checker):
try:
return all(bool(checker(scope)) for scope in scopes)
except Exception:
return False
granted = {str(scope) for scope in getattr(principal, "scopes", ())}
return all(scope in granted for scope in scopes)
def _safe_text_attribute(value: object | None, name: str) -> str:
try:
result = getattr(value, name, "")
except Exception:
return ""
return str(result or "")
def _principal_group_ids(principal: object) -> tuple[str, ...]:
try:
values = getattr(principal, "group_ids", ())
except Exception:
return ()
return tuple(str(value) for value in values if str(value))
def _actor_group_ids(
session: Session,
*,
principal: object,
tenant_id: str,
user_id: str,
include_admin_groups: bool,
) -> tuple[str, ...]:
try:
values = user_group_ids(
session,
tenant_id=tenant_id,
user_id=user_id,
include_admin_groups=include_admin_groups,
)
except Exception:
return _principal_group_ids(principal)
return tuple(str(value) for value in values if str(value))
def _campaign_visibility(
context: DocumentationContext,
*,
session: Session,
tenant_id: str,
user_id: str,
group_ids: tuple[str, ...],
):
principal = context.principal
registry = context.registry
if not _has_all_scopes(principal, ("campaigns:campaign:read",)):
return None
try:
if not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
return None
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
except Exception:
return None
if not isinstance(capability, CampaignAccessProvider):
return None
def campaign_visible(campaign_id: str) -> bool:
try:
return capability.campaign_exists(
session, tenant_id=tenant_id, campaign_id=campaign_id
) and capability.can_read_campaign(
session,
tenant_id=tenant_id,
campaign_id=campaign_id,
user_id=user_id,
group_ids=group_ids,
tenant_admin=_has_all_scopes(principal, ("tenant:*",)),
)
except Exception:
return False
return campaign_visible

View File

@@ -13,6 +13,7 @@ from govoplan_core.core.modules import (
DocumentationLink, DocumentationLink,
DocumentationTopic, DocumentationTopic,
FrontendModule, FrontendModule,
FrontendRoute,
MigrationSpec, MigrationSpec,
ModuleContext, ModuleContext,
ModuleInterfaceProvider, ModuleInterfaceProvider,
@@ -22,15 +23,19 @@ from govoplan_core.core.modules import (
PermissionDefinition, PermissionDefinition,
RoleTemplate, RoleTemplate,
) )
from govoplan_core.core.views import ViewSurface
from govoplan_core.db.base import Base from govoplan_core.db.base import Base
from govoplan_files.backend.change_tracking import register_files_change_tracking from govoplan_files.backend.change_tracking import register_files_change_tracking
from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata
from govoplan_files.backend.documentation import documentation_topics
register_files_change_tracking() register_files_change_tracking()
_files_table_retirement_provider = drop_table_retirement_provider( _files_table_retirement_provider = drop_table_retirement_provider(
file_models.FileBlob, file_models.FileBlob,
file_models.FileIntegrityScan,
file_models.FileIntegrityFinding,
file_models.FileFolder, file_models.FileFolder,
file_models.FileAsset, file_models.FileAsset,
file_models.FileVersion, file_models.FileVersion,
@@ -97,7 +102,7 @@ PERMISSIONS = (
_permission("files:file:download", "Download files", "Download managed files and generated archives."), _permission("files:file:download", "Download files", "Download managed files and generated archives."),
_permission("files:file:upload", "Upload files", "Upload new managed file versions."), _permission("files:file:upload", "Upload files", "Upload new managed file versions."),
_permission("files:file:organize", "Organize files", "Create folders, rename, move or copy managed files."), _permission("files:file:organize", "Organize files", "Create folders, rename, move or copy managed files."),
_permission("files:file:share", "Share files", "Grant or revoke managed file shares."), _permission("files:file:share", "Share files", "List, grant, update, expire, and revoke managed file shares."),
_permission("files:file:delete", "Delete files", "Delete or hide managed files and folders where policy allows it."), _permission("files:file:delete", "Delete files", "Delete or hide managed files and folders where policy allows it."),
_permission("files:file:admin", "Administer file spaces", "Administer all file spaces in the tenant."), _permission("files:file:admin", "Administer file spaces", "Administer all file spaces in the tenant."),
) )
@@ -137,6 +142,43 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
} }
def _tenant_summary_batch(session, tenant_ids) -> dict[str, dict[str, int]]:
from sqlalchemy import func
from govoplan_files.backend.db.models import FileAsset, FileConnectorCredential, FileConnectorPolicy, FileConnectorProfile, FileConnectorSpace
ids = tuple(dict.fromkeys(str(tenant_id) for tenant_id in tenant_ids if tenant_id))
if not ids:
return {}
counts: dict[str, dict[str, int]] = {
tenant_id: {
"files": 0,
"connector_credentials": 0,
"connector_policies": 0,
"connector_profiles": 0,
"connector_spaces": 0,
}
for tenant_id in ids
}
models = (
("files", FileAsset),
("connector_credentials", FileConnectorCredential),
("connector_policies", FileConnectorPolicy),
("connector_profiles", FileConnectorProfile),
("connector_spaces", FileConnectorSpace),
)
for count_key, model in models:
rows = (
session.query(model.tenant_id, func.count(model.id))
.filter(model.tenant_id.in_(ids))
.group_by(model.tenant_id)
.all()
)
for tenant_id, count in rows:
counts[tenant_id][count_key] = int(count)
return counts
def _veto_group_delete(session, tenant_id: str, group_id: str) -> None: def _veto_group_delete(session, tenant_id: str, group_id: str) -> None:
from govoplan_files.backend.db.models import FileAsset, FileConnectorSpace, FileFolder, FileShare from govoplan_files.backend.db.models import FileAsset, FileConnectorSpace, FileFolder, FileShare
@@ -186,84 +228,452 @@ manifest = ModuleManifest(
route_factory=_files_router, route_factory=_files_router,
role_templates=ROLE_TEMPLATES, role_templates=ROLE_TEMPLATES,
tenant_summary_providers=(_tenant_summary,), tenant_summary_providers=(_tenant_summary,),
tenant_summary_batch_providers=(_tenant_summary_batch,),
delete_veto_providers={"group": (_veto_group_delete,)}, delete_veto_providers={"group": (_veto_group_delete,)},
nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),), nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),),
frontend=FrontendModule( frontend=FrontendModule(
module_id="files", module_id="files",
package_name="@govoplan/files-webui", package_name="@govoplan/files-webui",
routes=(
FrontendRoute(
path="/files",
component="FilesPage",
required_any=("files:file:read",),
order=40,
),
),
nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),), nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),),
view_surfaces=(
ViewSurface(id="files.admin.system-connectors", module_id="files", kind="section", label="System file connections", order=75),
ViewSurface(id="files.admin.tenant-connectors", module_id="files", kind="section", label="Tenant file connections", order=65),
ViewSurface(id="files.admin.group-connectors", module_id="files", kind="section", label="Group file connections", order=65),
ViewSurface(id="files.admin.user-connectors", module_id="files", kind="section", label="User file connections", order=65),
ViewSurface(id="files.settings.connectors", module_id="files", kind="section", label="Personal file connections", order=20),
ViewSurface(id="files.widget.spaces", module_id="files", kind="section", label="File spaces widget", order=35),
),
), ),
documentation=( documentation=(
DocumentationTopic( DocumentationTopic(
id="files.governed-connectors-and-provenance", id="files.workflow.organize-managed-files",
title="Governed file connectors and source provenance", title="Organize managed files and folders",
summary="Connector profiles control who may browse an external source, which destinations are permitted, and how imported files retain verifiable source context.", summary="Create folders and rename, move, or copy accessible managed content with explicit conflict handling.",
body=( body=(
"Users select only connector profiles visible in their current scope and import remote content into managed Files storage before another module uses it. " "Organization stays inside governed personal or group spaces. Moves preserve the asset identity, while copies create new assets and versions that reuse immutable blob bytes. "
"Administrators define profiles, separate credential references, and ordered system/tenant/owner policies; deny rules win and profile responses never expose secrets. " "Every target conflict must be rejected, renamed, overwritten, or skipped explicitly."
"Deleting a database-managed profile or credential immediately scrubs Files-owned encrypted material and records a non-secret audit event in the same transaction; legacy non-owned external references are detached without provider calls. " ),
"Operators control private-network access deployment-wide and must keep every remote connection pinned to a policy-validated DNS/IP answer. " layer="configured",
"The built-in HTTP transport pins each connection and refuses redirects. Live S3 and SMB SDK access fails closed until S3 redirects and SMB DFS referrals can be revalidated and pinned. " documentation_types=("user",),
"Successful imports store the connector id, provider, remote path and identity, source revision, and selected metadata as provenance on the managed file and its audit events." audience=("file_user", "file_manager", "process_participant"),
order=42,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:organize"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Move or copy API", href="/api/v1/files/transfer", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
unlocks=("Managed content can be placed at stable logical paths without bypassing space access.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and organize managed files.",
"You have write or owner access to every source item and destination space used by the operation; the global organize permission alone does not grant resource access.",
],
"steps": [
"Open Files and select the personal or group space to organize.",
"Create the required destination folders or select the files and folders to rename, move, or copy.",
"Choose the destination and resolve each target conflict explicitly.",
"Apply the operation and reopen the destination folder.",
],
"outcome": "The selected content has the intended governed owner and logical path.",
"verification": "Confirm each resulting path and owner; for a move, also confirm the old path is gone, and for a copy, confirm the source remains.",
"related_topic_ids": [
"files.workflow.upload-managed-files",
"files.workflow.find-and-download-files",
"files.workflow.delete-managed-files",
],
},
),
DocumentationTopic(
id="files.workflow.find-and-download-files",
title="Find and download managed files",
summary="Search accessible managed content and download a current file version or a ZIP archive of a selection.",
body=(
"Files can be sorted and searched by logical path or name pattern. A download always uses the accessible current version; a multi-file selection can be generated as a temporary ZIP archive. "
"There is no dedicated content-preview service yet."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=43,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:download"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
unlocks=("Authorized users can retrieve governed current versions without gaining organization or sharing authority.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": ["You may view and download managed files in the relevant space."],
"steps": [
"Open Files and select the relevant personal or group space.",
"Navigate folders, sort the list, or use a path/name pattern to find the intended content.",
"Review the displayed owner, path, size, checksum, and version details.",
"Download one current file version, or select several files and choose Download ZIP.",
],
"outcome": "The authorized current file bytes or generated archive are downloaded to the local device.",
"verification": "Confirm the downloaded names and, where integrity matters, compare the file bytes with the displayed checksum.",
"related_topic_ids": [
"files.workflow.organize-managed-files",
"files.reference.snapshot-provenance-and-capabilities",
],
},
),
DocumentationTopic(
id="files.workflow.share-managed-files",
title="Manage access to managed files",
summary="List, grant, update, expire, or revoke direct read, write, and manage access without changing ownership.",
body=(
"File owners and file administrators can manage direct shares for users, groups, the tenant, and Campaign. Expired and revoked grants stop authorizing access immediately while independent active grants remain effective. "
"The Files share dialog lists active and historical grants, and revocation is idempotent."
),
layer="available",
documentation_types=("user",),
audience=("file_manager", "process_participant"),
order=44,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:share"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="List and manage shares", href="/api/v1/files/{file_id}/shares", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("campaigns",),
unlocks=("A supporting process can grant governed file access without changing file ownership.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You have the Files share permission and own the file, or administer file spaces for the active tenant.",
"The intended user, group, tenant, or Campaign target exists and is active.",
],
"steps": [
"Select one managed file, choose Manage shares, and review the effective and historical direct grants.",
"Select the intended user, group, or tenant and choose read, write, or manage access plus an optional expiry.",
"Grant or update the share without changing file ownership.",
"Revoke a grant when it is no longer needed; repeated revocation is a no-op.",
],
"limitations": [
"Campaign-target grants are normally created by the Campaign integration rather than selected manually in the Files dialog.",
"A user may retain access through another active direct grant or ownership path after one share is revoked.",
],
"outcome": "Direct access has the requested permission and lifetime while the managed asset keeps its owner.",
"verification": "Test one intended and one denied path, then expire or revoke the grant and verify that only independent access paths remain.",
"related_topic_ids": [
"files.workflow.find-and-download-files",
"files.assurance.process-and-release-readiness",
],
},
),
DocumentationTopic(
id="files.workflow.delete-managed-files",
title="Delete managed files and folders",
summary="Soft-delete accessible managed files or a folder tree where current policy allows it.",
body=(
"Deletion hides the selected managed assets rather than hard-purging their stored evidence. Folder deletion is recursive by default and includes child folders and files; a non-recursive request fails for a non-empty folder. "
"There is no self-service restore or hard-purge workflow today."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=45,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:delete"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
unlocks=("Authorized users can remove obsolete content from active file views while preserving the current soft-delete boundary.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and delete the selected managed content and have write or owner access to every affected asset or folder.",
"You have reviewed the complete folder tree when deleting recursively.",
],
"steps": [
"Open Files and select the files or folder to delete.",
"Review the selection and, for a folder, all content below it.",
"Confirm the delete action.",
"Refresh or reopen the space and verify that the selected paths are no longer active.",
],
"limitations": [
"Deletion is soft deletion, not a hard purge.",
"There is no self-service restore or hard-purge workflow.",
],
"outcome": "The selected content is hidden from active Files views under the current soft-delete model.",
"verification": "Confirm the deleted paths no longer appear in the active space; do not treat the action as physical erasure.",
"related_topic_ids": [
"files.workflow.organize-managed-files",
"files.assurance.process-and-release-readiness",
],
},
),
DocumentationTopic(
id="files.governed-connectors-and-provenance",
title="Govern file connections and credential deletion",
summary="Keep endpoint profiles, reusable credentials, and inherited connector policy separate, and understand what DELETE removes immediately.",
body=(
"System, tenant, and one user/group/campaign leaf form the effective policy chain: deny rules win and every configured allow rule must match. "
"Responses redact secret values and deployment references. Deleting a database-managed credential or profile immediately scrubs Files-owned encrypted material and private metadata in the same transaction as a non-secret audit event; dependent profiles are disabled, while legacy non-owned references are only detached and audited."
),
layer="configured",
documentation_types=("admin",),
audience=("file_admin", "tenant_admin", "system_admin", "security_auditor"),
order=50,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(
"files:file:admin",
"admin:settings:read",
"admin:settings:write",
"system:settings:read",
"system:settings:write",
),
),
),
links=(
DocumentationLink(label="Tenant connector administration", href="/admin?section=tenant-file-connectors", kind="runtime"),
DocumentationLink(label="System connector administration", href="/admin?section=system-file-connectors", kind="runtime"),
DocumentationLink(label="Tenant connector policy API", href="/api/v1/files/connectors/policies/tenant", kind="api"),
DocumentationLink(label="Connector credentials API", href="/api/v1/files/connectors/credentials", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("access", "audit", "mail"),
unlocks=("Scoped, explainable external-file access without exposing credentials to consuming modules.",),
configuration_keys=(
"GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON",
"GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST",
"GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST",
"MASTER_KEY_B64",
),
metadata={
"kind": "reference",
"route": "/admin?section=tenant-file-connectors",
"screen": "File connections",
"section": "Profiles, credentials, and effective connector policy",
"security_invariants": [
"New API-managed external secret references fail closed until Files can prove ownership and provider-side deletion.",
"Deletion and destructive retirement scrub Files-owned encrypted connector material before completion and emit non-secret audit evidence.",
"Legacy non-owned external references are detached and audited, never sent to an arbitrary provider delete operation.",
],
"related_topic_ids": [
"files.workflow.import-managed-snapshot",
"files.reference.integrity-recovery-and-fail-closed-transports",
"mail.profiles-and-policy",
],
},
),
DocumentationTopic(
id="files.reference.integrity-recovery-and-fail-closed-transports",
title="Operate Files integrity, recovery, and connector transport safety",
summary="Back up database evidence, blob bytes, and the encryption key as one recovery unit, and keep unsupported SDK transports fail-closed.",
body=(
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the original master key, then run the bounded resumable integrity scan and verify representative access paths. Missing or mismatched blobs are quarantined; orphan objects are reported before dry-run-first, explicitly authorized cleanup. "
"Live S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Destructive module retirement drops database tables but does not remove backend blob objects."
),
layer="configured",
documentation_types=("admin",),
audience=("operator", "system_admin", "security_auditor"),
order=51,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(
"files:file:admin",
"admin:settings:read",
"system:settings:read",
"system:audit:read",
),
),
),
links=(
DocumentationLink(label="System file connections", href="/admin?section=system-file-connectors", kind="runtime"),
DocumentationLink(label="Connector provider status", href="/api/v1/files/connectors/providers", kind="api"),
DocumentationLink(label="Create an integrity scan", href="/api/v1/files/integrity/scans", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("audit", "ops"),
unlocks=("Recoverable managed-file evidence without weakening outbound peer validation.",),
configuration_keys=(
"FILE_STORAGE_BACKEND",
"FILE_STORAGE_LOCAL_ROOT",
"FILE_STORAGE_LOCAL_FALLBACK_ROOTS",
"FILE_UPLOAD_MAX_BYTES",
"FILE_UPLOAD_ZIP_MAX_BYTES",
"MASTER_KEY_B64",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
"GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES",
"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES",
),
metadata={
"kind": "reference",
"route": "/admin?section=system-file-connectors",
"screen": "System file connections and deployment operations",
"section": "Storage integrity, backup/recovery, and fail-closed transports",
"recovery_unit": ["Files database rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"],
"verification": "After restore, complete a checksum-enabled integrity scan, resolve every missing/corrupt finding, approve or retain every reported orphan, verify authorized and denied access, and test one permitted pinned HTTP connector.",
"related_topic_ids": [
"files.governed-connectors-and-provenance",
"files.reference.snapshot-provenance-and-capabilities",
],
},
),
DocumentationTopic(
id="files.reference.snapshot-provenance-and-capabilities",
title="Integrate through managed snapshots and Files capabilities",
summary="Other modules consume stable Files capabilities or HTTP contracts and retain exact version evidence instead of importing Files internals.",
body=(
"Use files.access to explain resource access and files.campaign_attachments to freeze campaign inputs at an exact asset, version, blob, checksum, and source revision. "
"Import external content before a governed use, preserve provenance on derived snapshots, and keep collaboration, provider sync, OAuth, remote mutation, and domain workflow state in their owning modules."
),
layer="available",
documentation_types=("admin", "user"),
audience=("module_integrator", "file_admin", "campaign_admin", "process_designer"),
order=52,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=("files:file:read", "files:file:upload", "files:file:admin"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files API", href="/api/v1/files", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("campaigns", "docs"),
unlocks=("Campaign, report, template, workflow, and document modules can exchange governed input/output snapshots.",),
metadata={
"kind": "reference",
"route": "/files",
"screen": "Files and module integration",
"section": "Managed snapshot provenance and capability boundaries",
"provided_interfaces": ["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
"provenance_fields": ["connector_id", "provider", "external_id", "external_path", "revision", "metadata"],
"related_topic_ids": [
"files.workflow.import-managed-snapshot",
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
],
},
),
DocumentationTopic(
id="files.assurance.process-and-release-readiness",
title="Assure a Files-backed process and release",
summary="Check a process against the implemented Files boundary, exercise permitted and denied paths, and retain evidence before approving a release or operational use.",
body=(
"A process owner must distinguish implemented controls from planned capabilities before relying on Files. A release is not ready until all package and manifest versions align and representative authorization, upload limits, conflict handling, download, deletion, connector, and recovery paths have been exercised. "
"Current limitations include no self-service restore or hard purge, no enforced retention or legal hold, and no dedicated canonical audit event for every ordinary Files mutation. Share grant, change, expiry, and revocation operations do emit dedicated audit records. Record remaining limitations in the process assessment instead of treating soft deletion or change-sequence entries as stronger evidence."
), ),
layer="configured", layer="configured",
documentation_types=("admin", "user"), documentation_types=("admin", "user"),
audience=("file_user", "file_admin", "tenant_admin", "operator"), audience=("process_owner", "release_manager", "file_admin", "operator", "security_auditor"),
order=42, order=53,
conditions=( conditions=(
DocumentationCondition( DocumentationCondition(
required_modules=("files",), required_modules=("files",),
any_scopes=( any_scopes=(
"files:file:read", "files:file:read",
"files:file:upload",
"files:file:admin", "files:file:admin",
"admin:settings:read", "admin:settings:read",
"system:settings:read", "system:settings:read",
"system:audit:read",
), ),
), ),
), ),
links=( links=(
DocumentationLink(label="Files", href="/files", kind="runtime"), DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Tenant connector administration", href="/admin?section=tenant-file-connectors", kind="runtime"), DocumentationLink(label="Files API", href="/api/v1/files", kind="api"),
DocumentationLink(label="Connector profiles API", href="/api/v1/files/connectors/profiles", kind="api"), DocumentationLink(label="Connector provider status", href="/api/v1/files/connectors/providers", kind="api"),
DocumentationLink(label="Connector providers API", href="/api/v1/files/connectors/providers", kind="api"), DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
DocumentationLink(label="Tenant connector policy API", href="/api/v1/files/connectors/policies/tenant", kind="api"),
), ),
related_modules=("campaigns",), related_modules=("campaigns", "audit", "ops"),
unlocks=("Campaigns can consume managed attachments while Files preserves frozen source provenance.",), unlocks=("Process owners and release managers can approve Files use against explicit controls, evidence, and known gaps.",),
configuration_keys=( configuration_keys=(
"FILE_STORAGE_BACKEND",
"FILE_STORAGE_LOCAL_ROOT",
"FILE_UPLOAD_MAX_BYTES",
"FILE_UPLOAD_ZIP_MAX_BYTES",
"MASTER_KEY_B64",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
"GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON",
"GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST",
), ),
metadata={ metadata={
"kind": "reference", "kind": "workflow",
"perspectives": { "route": "/files",
"user": "Choose a visible connection, browse it read-only, and import a selected file into managed storage before using it elsewhere.", "screen": "Files process and release assurance",
"admin": "Govern profile visibility, separate credential references, provider allow/deny policy, and permitted connector operations.", "help_contexts": ["files.list"],
"operator": "Control private-network access deployment-wide, provide credential and CA allowlists, and investigate fail-closed transport errors without weakening peer validation.", "prerequisites": [
}, "A named process owner has defined the intended users, data classification, retention expectations, and integrations.",
"security_invariants": [ "A candidate release is installed on a clean database and an upgrade copy with aligned Python, root package, WebUI package, and module-manifest versions.",
"Every network peer must be policy-validated and pinned at connection time.", "Representative permitted and denied accounts, bounded test files, and a coordinated database/blob/key recovery set are available.",
"Redirects and protocol referrals must be rejected or independently revalidated and pinned.",
"SDK transports that cannot provide those guarantees fail before client construction.",
"New API-managed external secret references fail closed until Files can prove ownership and provider-side deletion.",
"Deletion and destructive retirement scrub Files-owned encrypted connector material before completion and emit non-secret audit evidence.",
"Legacy non-owned external references are detached and audited, never sent to an arbitrary provider delete operation.",
], ],
"provenance_fields": [ "steps": [
"connector_id", "Compare the process requirements with the handbook's implemented/planned boundary and record every unsupported requirement or compensating control.",
"provider", "Confirm version alignment, apply migrations on clean and upgrade databases, and run the repository plus meta-repository security and static-analysis gates.",
"external_id", "Exercise allowed and denied personal, group, and share access with representative accounts.",
"external_path", "Exercise bounded upload and ZIP handling, every required conflict strategy, organization, download, and soft deletion.",
"revision", "Where connectors are configured, verify policy explanation and one pinned HTTP provider; verify live S3 and SMB access still fails closed.",
"metadata", "Restore a coordinated database/blob/key backup and compare representative downloaded bytes with their recorded SHA-256 checksums.",
"Record the tested versions, results, known limitations, evidence locations, residual risks, owner, and approval decision.",
],
"outcome": "The process or release has an explicit approval record tied to aligned versions, representative evidence, known limitations, and owned residual risks.",
"verification": "A reviewer can reproduce the recorded allow/deny, integrity, connector, and recovery checks and can trace each unmet requirement to a documented limitation or accepted compensating control.",
"related_topic_ids": [
"files.workflow.upload-managed-files",
"files.workflow.upload-and-unpack-zip",
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
"files.workflow.share-managed-files",
"files.workflow.delete-managed-files",
"files.workflow.import-managed-snapshot",
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
"files.reference.snapshot-provenance-and-capabilities",
], ],
"related_topic_ids": ["mail.profiles-and-policy"],
}, },
), ),
), ),
documentation_providers=(documentation_topics,),
migration_spec=MigrationSpec( migration_spec=MigrationSpec(
module_id="files", module_id="files",
metadata=Base.metadata, metadata=Base.metadata,

View File

@@ -0,0 +1,53 @@
"""file share lifecycle
Revision ID: b8c9d0e1f2a4
Revises: a7b8c9d0e1f3
Create Date: 2026-07-30 00:00:00.000000
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "b8c9d0e1f2a4"
down_revision = "a7b8c9d0e1f3"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("file_shares") as batch_op:
batch_op.add_column(
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True)
)
batch_op.add_column(
sa.Column("revoked_by_user_id", sa.String(length=36), nullable=True)
)
batch_op.create_foreign_key(
op.f("fk_file_shares_revoked_by_user_id_access_users"),
"access_users",
["revoked_by_user_id"],
["id"],
ondelete="SET NULL",
)
batch_op.create_index(
op.f("ix_file_shares_expires_at"), ["expires_at"], unique=False
)
batch_op.create_index(
op.f("ix_file_shares_revoked_by_user_id"),
["revoked_by_user_id"],
unique=False,
)
def downgrade() -> None:
with op.batch_alter_table("file_shares") as batch_op:
batch_op.drop_index(op.f("ix_file_shares_revoked_by_user_id"))
batch_op.drop_index(op.f("ix_file_shares_expires_at"))
batch_op.drop_constraint(
op.f("fk_file_shares_revoked_by_user_id_access_users"),
type_="foreignkey",
)
batch_op.drop_column("revoked_by_user_id")
batch_op.drop_column("expires_at")

View File

@@ -0,0 +1,168 @@
"""file integrity reconciliation
Revision ID: c9d0e1f2a3b5
Revises: b8c9d0e1f2a4
Create Date: 2026-07-30 00:00:00.000000
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "c9d0e1f2a3b5"
down_revision = "b8c9d0e1f2a4"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("file_blobs") as batch_op:
batch_op.add_column(
sa.Column(
"integrity_status",
sa.String(length=30),
nullable=False,
server_default="unchecked",
)
)
batch_op.add_column(
sa.Column("integrity_checked_at", sa.DateTime(timezone=True), nullable=True)
)
batch_op.add_column(
sa.Column("integrity_failure", sa.String(length=100), nullable=True)
)
batch_op.add_column(
sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
)
batch_op.create_index(
op.f("ix_file_blobs_integrity_status"),
["integrity_status"],
unique=False,
)
batch_op.create_index(
op.f("ix_file_blobs_integrity_checked_at"),
["integrity_checked_at"],
unique=False,
)
batch_op.create_index(
op.f("ix_file_blobs_quarantined_at"),
["quarantined_at"],
unique=False,
)
op.create_table(
"file_integrity_scans",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("storage_backend", sa.String(length=50), nullable=False),
sa.Column("storage_prefix", sa.String(length=1000), nullable=False),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("phase", sa.String(length=30), nullable=False),
sa.Column("verify_checksums", sa.Boolean(), nullable=False),
sa.Column("batch_size", sa.Integer(), nullable=False),
sa.Column("blob_cursor", sa.String(length=36), nullable=True),
sa.Column("object_cursor", sa.String(length=1000), nullable=True),
sa.Column("scanned_blob_count", sa.Integer(), nullable=False),
sa.Column("verified_blob_count", sa.Integer(), nullable=False),
sa.Column("quarantined_blob_count", sa.Integer(), nullable=False),
sa.Column("scanned_object_count", sa.Integer(), nullable=False),
sa.Column("orphan_object_count", sa.Integer(), nullable=False),
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["created_by_user_id"],
["access_users.id"],
name=op.f(
"fk_file_integrity_scans_created_by_user_id_access_users"
),
ondelete="SET NULL",
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_scans")),
)
for column in ("tenant_id", "status", "created_by_user_id"):
op.create_index(
op.f(f"ix_file_integrity_scans_{column}"),
"file_integrity_scans",
[column],
unique=False,
)
op.create_table(
"file_integrity_findings",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("scan_id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("kind", sa.String(length=40), nullable=False),
sa.Column("state", sa.String(length=30), nullable=False),
sa.Column("blob_id", sa.String(length=36), nullable=True),
sa.Column("storage_key", sa.String(length=1000), nullable=False),
sa.Column("expected_size_bytes", sa.Integer(), nullable=True),
sa.Column("observed_size_bytes", sa.Integer(), nullable=True),
sa.Column("expected_checksum_sha256", sa.String(length=64), nullable=True),
sa.Column("observed_checksum_sha256", sa.String(length=64), nullable=True),
sa.Column("resolved_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("resolved_by_user_id", sa.String(length=36), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["blob_id"],
["file_blobs.id"],
name=op.f("fk_file_integrity_findings_blob_id_file_blobs"),
ondelete="SET NULL",
),
sa.ForeignKeyConstraint(
["resolved_by_user_id"],
["access_users.id"],
name=op.f(
"fk_file_integrity_findings_resolved_by_user_id_access_users"
),
ondelete="SET NULL",
),
sa.ForeignKeyConstraint(
["scan_id"],
["file_integrity_scans.id"],
name=op.f(
"fk_file_integrity_findings_scan_id_file_integrity_scans"
),
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_findings")),
)
for column in (
"scan_id",
"tenant_id",
"kind",
"state",
"blob_id",
"resolved_by_user_id",
):
op.create_index(
op.f(f"ix_file_integrity_findings_{column}"),
"file_integrity_findings",
[column],
unique=False,
)
op.create_index(
"ix_file_integrity_findings_scan_state",
"file_integrity_findings",
["scan_id", "state"],
unique=False,
)
def downgrade() -> None:
op.drop_table("file_integrity_findings")
op.drop_table("file_integrity_scans")
with op.batch_alter_table("file_blobs") as batch_op:
batch_op.drop_index(op.f("ix_file_blobs_quarantined_at"))
batch_op.drop_index(op.f("ix_file_blobs_integrity_checked_at"))
batch_op.drop_index(op.f("ix_file_blobs_integrity_status"))
batch_op.drop_column("quarantined_at")
batch_op.drop_column("integrity_failure")
batch_op.drop_column("integrity_checked_at")
batch_op.drop_column("integrity_status")

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1 @@
"""Focused HTTP route modules for the Files API."""

View File

@@ -0,0 +1,134 @@
from __future__ import annotations
from io import BytesIO
from fastapi import APIRouter, Depends
from fastapi.responses import StreamingResponse
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_files.backend.schemas import (
BulkDeleteRequest,
BulkDeleteResponse,
FileAssetResponse,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import (
get_asset_for_user,
read_asset_bytes,
soft_delete_assets,
)
from govoplan_files.backend.route_support import (
_asset_response,
_attachment_disposition,
_audit_connector_event,
_http_error,
_is_admin,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.get("/{file_id}", response_model=FileAssetResponse)
def get_file(
file_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
try:
asset = get_asset_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
return _asset_response(session, asset, include_shares=True)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
@router.get("/{file_id}/download")
def download_file(
file_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
):
try:
asset = get_asset_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
data, version, blob = read_asset_bytes(session, asset)
_audit_connector_event(
session,
principal,
action="files.connector.accessed",
asset=asset,
version=version,
blob=blob,
operation="download",
commit=True,
)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
return StreamingResponse(
BytesIO(data),
media_type=blob.content_type or "application/octet-stream",
headers=headers,
)
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
def delete_file(
file_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
):
try:
asset = get_asset_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
require_write=True,
is_admin=_is_admin(principal),
)
count = soft_delete_assets(session, [asset])
session.commit()
return BulkDeleteResponse(deleted_count=count)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc, not_found=True) from exc
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
def bulk_delete_files(
payload: BulkDeleteRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
):
try:
assets = [
get_asset_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
require_write=True,
is_admin=_is_admin(principal),
)
for file_id in payload.file_ids
]
count = soft_delete_assets(session, assets)
session.commit()
return BulkDeleteResponse(deleted_count=count)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc) from exc

View File

@@ -0,0 +1,252 @@
from __future__ import annotations
import json
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_any_scope, require_scope
from govoplan_files.backend.schemas import (
FileConnectorBrowseItem,
FileConnectorBrowseResponse,
FileConnectorImportRequest,
FileConnectorSyncResponse,
FileUploadResponse,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.paths import UnsafeFilePathError
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_browse import (
ConnectorBrowseError,
ConnectorBrowseUnsupported,
browse_connector_profile,
normalize_connector_browse_path,
)
from govoplan_files.backend.storage.connector_imports import (
ConnectorImportError,
ConnectorImportUnsupported,
)
from govoplan_files.backend.storage.connector_deployment import (
connector_effective_endpoint_url,
)
from govoplan_files.backend.storage.connector_policy import (
ConnectorAccessRequest,
ConnectorPolicyDenied,
connector_policy_decision,
)
from govoplan_files.backend.storage.files import (
create_file_asset,
sync_file_asset_from_source,
)
from govoplan_files.backend.route_support import (
_asset_response,
_audit_connector_imports,
_audit_connector_sync,
_connector_browse_next_token,
_connector_policy_error,
_download_connector_payload,
_ensure_campaign_file_access,
_http_error,
_is_admin,
_visible_connector_profile,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.post(
"/connectors/profiles/{profile_id}/import", response_model=FileUploadResponse
)
def import_connector_file(
profile_id: str,
payload: FileConnectorImportRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
):
try:
if payload.campaign_id:
_ensure_campaign_file_access(session, principal, payload.campaign_id)
profile = _visible_connector_profile(
session, principal, profile_id, campaign_id=payload.campaign_id
)
_source_path, downloaded, metadata = _download_connector_payload(
profile, payload, operation="import"
)
target_owner = payload.owner_id or principal.user.id
stored = create_file_asset(
session,
tenant_id=principal.tenant_id,
owner_type=payload.owner_type,
owner_id=target_owner,
user_id=principal.user.id,
filename=downloaded.filename,
data=downloaded.data,
folder=payload.target_folder,
display_path=payload.target_path,
content_type=downloaded.content_type,
metadata=metadata,
campaign_id=payload.campaign_id,
conflict_strategy=payload.conflict_strategy,
is_admin=_is_admin(principal),
)
_audit_connector_imports(session, principal, [stored.asset])
session.commit()
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except ConnectorImportUnsupported as exc:
session.rollback()
raise HTTPException(
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
) from exc
except (
ConnectorImportError,
FileStorageError,
UnsafeFilePathError,
ValueError,
json.JSONDecodeError,
) as exc:
session.rollback()
raise _http_error(exc) from exc
return FileUploadResponse(
files=[_asset_response(session, stored.asset, include_shares=True)]
)
@router.post(
"/connectors/profiles/{profile_id}/sync", response_model=FileConnectorSyncResponse
)
def sync_connector_file(
profile_id: str,
payload: FileConnectorImportRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
):
try:
if payload.campaign_id:
_ensure_campaign_file_access(session, principal, payload.campaign_id)
profile = _visible_connector_profile(
session, principal, profile_id, campaign_id=payload.campaign_id
)
_source_path, downloaded, metadata = _download_connector_payload(
profile, payload, operation="sync"
)
target_owner = payload.owner_id or principal.user.id
stored, sync_action, previous_version_id = sync_file_asset_from_source(
session,
tenant_id=principal.tenant_id,
owner_type=payload.owner_type,
owner_id=target_owner,
user_id=principal.user.id,
filename=downloaded.filename,
data=downloaded.data,
folder=payload.target_folder,
display_path=payload.target_path,
content_type=downloaded.content_type,
metadata=metadata,
campaign_id=payload.campaign_id,
conflict_strategy=payload.conflict_strategy,
is_admin=_is_admin(principal),
)
_audit_connector_sync(
session,
principal,
stored.asset,
sync_action=sync_action,
previous_version_id=previous_version_id,
)
session.commit()
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except ConnectorImportUnsupported as exc:
session.rollback()
raise HTTPException(
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
) from exc
except (
ConnectorImportError,
FileStorageError,
UnsafeFilePathError,
ValueError,
json.JSONDecodeError,
) as exc:
session.rollback()
raise _http_error(exc) from exc
return FileConnectorSyncResponse(
file=_asset_response(session, stored.asset, include_shares=True),
action=sync_action,
previous_version_id=previous_version_id,
current_version_id=stored.version.id,
)
@router.get(
"/connectors/profiles/{profile_id}/browse",
response_model=FileConnectorBrowseResponse,
)
def browse_connector_profile_items(
profile_id: str,
path: str | None = None,
library_id: str | None = None,
continuation_token: str | None = None,
campaign_id: str | None = None,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:read",
"files:file:upload",
"files:file:download",
"files:file:admin",
"system:settings:read",
"admin:settings:read",
)
),
):
try:
profile = _visible_connector_profile(
session, principal, profile_id, campaign_id=campaign_id
)
browse_path = normalize_connector_browse_path(path)
decision = connector_policy_decision(
ConnectorAccessRequest(
connector_id=profile.id,
credential_id=profile.credential_profile_id,
provider=profile.provider,
external_path=browse_path,
external_url=connector_effective_endpoint_url(
provider=profile.provider,
endpoint_url=profile.endpoint_url,
metadata=profile.metadata,
),
operation="browse",
),
profile.policy_sources,
)
if not decision.allowed:
raise ConnectorPolicyDenied(decision)
items = browse_connector_profile(
profile,
path=browse_path,
library_id=library_id,
continuation_token=continuation_token,
)
except ConnectorPolicyDenied as exc:
raise _connector_policy_error(exc) from exc
except ConnectorBrowseUnsupported as exc:
raise HTTPException(
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
) from exc
except (ConnectorBrowseError, OSError, ValueError, json.JSONDecodeError) as exc:
raise _http_error(exc) from exc
return FileConnectorBrowseResponse(
profile_id=profile.id,
provider=profile.provider,
path=browse_path,
library_id=library_id,
next_continuation_token=_connector_browse_next_token(items),
has_more=any(bool(item.metadata.get("listing_truncated")) for item in items),
decision=decision.to_dict(),
items=[FileConnectorBrowseItem(**item.to_response()) for item in items],
)

View File

@@ -0,0 +1,261 @@
from __future__ import annotations
import json
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_any_scope
from govoplan_files.backend.change_tracking import (
FILES_CONNECTOR_PROFILES_COLLECTION,
)
from govoplan_files.backend.schemas import (
FileConnectorProfileResponse,
FileConnectorProfileUpdateRequest,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_credential_deletion import (
delete_connector_profile_row,
)
from govoplan_files.backend.storage.connector_deployment import (
connector_effective_endpoint_url,
)
from govoplan_files.backend.storage.connector_profile_store import (
connector_profile_from_row,
get_connector_profile_row,
update_connector_profile_row,
)
from govoplan_files.backend.storage.connector_policy import (
ConnectorPolicyDenied,
)
from govoplan_files.backend.route_support import (
FILES_CONNECTOR_PROFILE_RESOURCE,
_can_read_disabled_connector_profiles,
_connector_policy_error,
_credential_row_for_profile,
_ensure_connector_configuration_allowed,
_ensure_connector_local_policy_allowed,
_http_error,
_record_connector_settings_change,
_require_connector_profile_write,
_visible_connector_profile,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.get(
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
)
def get_connector_profile(
profile_id: str,
campaign_id: str | None = None,
include_disabled: bool = False,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:read",
"files:file:upload",
"files:file:download",
"files:file:admin",
"system:settings:read",
"admin:settings:read",
)
),
):
try:
profile = _visible_connector_profile(
session,
principal,
profile_id,
campaign_id=campaign_id,
include_disabled=include_disabled
and _can_read_disabled_connector_profiles(principal),
include_effective_policy=False,
)
except (OSError, ValueError, json.JSONDecodeError) as exc:
raise _http_error(exc) from exc
return FileConnectorProfileResponse(**profile.to_response())
@router.patch(
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
)
def update_connector_profile(
profile_id: str,
payload: FileConnectorProfileUpdateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
try:
row = get_connector_profile_row(
session,
tenant_id=principal.tenant_id,
profile_id=profile_id,
include_disabled=True,
)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
_require_connector_profile_write(principal, row.scope_type)
credentials = payload.credentials
try:
credential_profile_id = (
payload.credential_profile_id
if payload.credential_profile_id is not None
else row.credential_profile_id
)
provider = payload.provider if payload.provider is not None else row.provider
credential_row = _credential_row_for_profile(
session,
principal,
credential_profile_id=credential_profile_id,
provider=provider,
profile_id=row.id,
scope_type=row.scope_type,
scope_id=row.scope_id,
include_disabled=True,
)
_ensure_connector_configuration_allowed(
session,
principal,
connector_id=row.id,
credential_id=credential_profile_id,
provider=provider,
endpoint_url=connector_effective_endpoint_url(
provider=provider,
endpoint_url=payload.endpoint_url
if payload.endpoint_url is not None
else row.endpoint_url,
metadata=payload.metadata
if payload.metadata is not None
else row.metadata_,
),
base_path=payload.base_path
if payload.base_path is not None
else row.base_path,
scope_type=row.scope_type,
scope_id=row.scope_id,
operation="configure",
)
if payload.policy is not None:
_ensure_connector_local_policy_allowed(
session,
principal,
scope_type=row.scope_type,
scope_id=row.scope_id,
policy=payload.policy,
)
update_connector_profile_row(
session,
row,
user_id=principal.user.id,
label=payload.label,
provider=payload.provider,
endpoint_url=payload.endpoint_url,
base_path=payload.base_path,
enabled=payload.enabled,
credential_profile_id=payload.credential_profile_id,
credential_mode=payload.credential_mode,
username=credentials.username if credentials else None,
password=credentials.password if credentials else None,
token=credentials.token if credentials else None,
password_env=credentials.password_env if credentials else None,
token_env=credentials.token_env if credentials else None,
secret_ref=credentials.secret_ref if credentials else None,
clear_password=payload.clear_password,
clear_token=payload.clear_token,
capabilities=payload.capabilities,
policy=payload.policy,
metadata=payload.metadata,
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
resource_id=row.id,
operation="updated",
principal=principal,
tenant_id=row.tenant_id,
payload={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
},
)
session.commit()
session.refresh(row)
return FileConnectorProfileResponse(
**connector_profile_from_row(
row, credential_row=credential_row
).to_response()
)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.delete(
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
)
def deactivate_connector_profile(
profile_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
try:
row = get_connector_profile_row(
session,
tenant_id=principal.tenant_id,
profile_id=profile_id,
include_disabled=True,
)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
_require_connector_profile_write(principal, row.scope_type)
try:
changed = delete_connector_profile_row(
session,
row,
deletion_reason="api_delete",
user_id=principal.user.id,
api_key_id=principal.api_key.id if principal.api_key else None,
)
if changed:
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
resource_id=row.id,
operation="deleted",
principal=principal,
tenant_id=row.tenant_id,
payload={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
},
)
session.commit()
session.refresh(row)
return FileConnectorProfileResponse(
**connector_profile_from_row(row).to_response()
)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc) from exc
except Exception:
session.rollback()
raise

View File

@@ -0,0 +1,852 @@
from __future__ import annotations
import json
from typing import Literal
from fastapi import APIRouter, Depends, Query, status
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_any_scope
from govoplan_files.backend.change_tracking import (
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
FILES_CONNECTOR_POLICIES_COLLECTION,
FILES_CONNECTOR_PROFILES_COLLECTION,
)
from govoplan_files.backend.schemas import (
FileConnectorCredentialCreateRequest,
FileConnectorCredentialResponse,
FileConnectorCredentialsResponse,
FileConnectorCredentialUpdateRequest,
FileConnectorDiscoveryRequest,
FileConnectorDiscoveryResponse,
FileConnectorSettingsDeltaResponse,
FileConnectorPolicyEvaluateRequest,
FileConnectorPolicyEvaluateResponse,
FileConnectorPolicyResponse,
FileConnectorPolicyUpdateRequest,
FileConnectorProfileCreateRequest,
FileConnectorProfileResponse,
FileConnectorProfilesResponse,
FileConnectorProviderResponse,
FileConnectorProvidersResponse,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_credential_store import (
create_connector_credential_row,
get_connector_credential_row,
list_database_connector_credentials,
update_connector_credential_row,
)
from govoplan_files.backend.storage.connector_credential_deletion import (
delete_connector_credential_row,
)
from govoplan_files.backend.storage.connector_browse import (
ConnectorBrowseError,
ConnectorBrowseUnsupported,
browse_connector_profile,
)
from govoplan_files.backend.storage.connector_deployment import (
connector_effective_endpoint_url,
reject_api_controlled_deployment_references,
)
from govoplan_files.backend.storage.connector_profile_store import (
connector_profile_from_row,
create_connector_profile_row,
)
from govoplan_files.backend.storage.connector_providers import (
connector_provider_descriptors,
)
from govoplan_files.backend.storage.connector_policy import (
ConnectorAccessRequest,
ConnectorPolicyDenied,
connector_policy_decision,
connector_policy_sources_from_payload,
)
from govoplan_files.backend.storage.connector_policy_store import (
connector_policy_response,
set_connector_policy,
)
from govoplan_files.backend.route_support import (
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
FILES_CONNECTOR_POLICY_RESOURCE,
FILES_CONNECTOR_PROFILE_RESOURCE,
_audit_connector_discovery_attempt,
_can_read_disabled_connector_profiles,
_connector_credential_response,
_connector_policy_error,
_credential_row_for_profile,
_discovery_profile_from_payload,
_ensure_campaign_file_access,
_ensure_connector_configuration_allowed,
_ensure_connector_credential_configuration_allowed,
_ensure_connector_local_policy_allowed,
_file_connector_policy_resource_id,
_file_connector_settings_entries,
_http_error,
_record_connector_settings_change,
_require_connector_credential_write,
_require_connector_policy_read,
_require_connector_profile_write,
_same_endpoint,
_visible_connector_profiles,
_webdav_discovery_candidates,
)
from govoplan_files.backend.services.connector_settings_delta import (
_full_file_connector_settings_delta_response,
_incremental_file_connector_settings_delta_response,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.post(
"/connector-policy/evaluate", response_model=FileConnectorPolicyEvaluateResponse
)
def evaluate_connector_policy(
payload: FileConnectorPolicyEvaluateRequest,
principal: ApiPrincipal = Depends(
require_any_scope("files:file:read", "files:file:upload", "files:file:download")
),
):
del principal
sources = connector_policy_sources_from_payload(
[item.model_dump(mode="json") for item in payload.policy_sources]
)
request = ConnectorAccessRequest.from_provenance(
payload.source_provenance.model_dump(mode="json", exclude_none=True),
operation=payload.operation,
)
return FileConnectorPolicyEvaluateResponse(
decision=connector_policy_decision(request, sources).to_dict()
)
@router.get(
"/connectors/settings/delta", response_model=FileConnectorSettingsDeltaResponse
)
def connector_settings_delta(
scope_type: str = Query(default="tenant"),
scope_id: str | None = Query(default=None),
provider: str | None = None,
campaign_id: str | None = None,
include_disabled: bool = False,
include_inactive: bool = False,
owner_type: Literal["user", "group"] | None = None,
owner_id: str | None = None,
since: str | None = None,
limit: int = Query(default=100, ge=1, le=500),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:read", "admin:settings:read"
)
),
):
scope_type = scope_type.strip().casefold()
_require_connector_policy_read(principal, scope_type)
try:
if since is None:
return _full_file_connector_settings_delta_response(
session,
principal,
scope_type=scope_type,
scope_id=scope_id,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled,
include_inactive=include_inactive,
owner_type=owner_type,
owner_id=owner_id,
)
entries, has_more = _file_connector_settings_entries(
session, tenant_id=principal.tenant_id, since=since, limit=limit
)
if entries is None:
return _full_file_connector_settings_delta_response(
session,
principal,
scope_type=scope_type,
scope_id=scope_id,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled,
include_inactive=include_inactive,
owner_type=owner_type,
owner_id=owner_id,
)
return _incremental_file_connector_settings_delta_response(
session,
principal,
entries=entries,
has_more=has_more,
scope_type=scope_type,
scope_id=scope_id,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled,
include_inactive=include_inactive,
owner_type=owner_type,
owner_id=owner_id,
)
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
raise _http_error(exc) from exc
@router.get("/connectors/providers", response_model=FileConnectorProvidersResponse)
def list_connector_providers(
principal: ApiPrincipal = Depends(
require_any_scope("files:file:read", "files:file:upload", "files:file:admin")
),
):
del principal
return FileConnectorProvidersResponse(
providers=[
FileConnectorProviderResponse(**item.to_response())
for item in connector_provider_descriptors()
]
)
@router.post("/connectors/discover", response_model=FileConnectorDiscoveryResponse)
def discover_connector_endpoint(
payload: FileConnectorDiscoveryRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
try:
reject_api_controlled_deployment_references(
password_env=payload.credentials.password_env,
token_env=payload.credentials.token_env,
secret_ref=payload.credentials.secret_ref,
metadata=payload.metadata,
)
except ValueError as exc:
raise _http_error(exc) from exc
if payload.provider not in {"webdav", "nextcloud"}:
return FileConnectorDiscoveryResponse(
provider=payload.provider,
endpoint_url=None,
base_path=payload.base_path,
status="unsupported",
message=f"Discovery is not implemented for {payload.provider} connectors yet",
)
candidates: list[dict[str, str]] = []
for endpoint_url in _webdav_discovery_candidates(payload):
profile = _discovery_profile_from_payload(
payload, endpoint_url, principal=principal
)
try:
_ensure_connector_configuration_allowed(
session,
principal,
connector_id=None,
credential_id=None,
provider=profile.provider,
endpoint_url=endpoint_url,
base_path=profile.base_path,
scope_type="tenant",
scope_id=principal.tenant_id,
operation="discover",
)
_audit_connector_discovery_attempt(
session,
principal,
provider=profile.provider,
endpoint_url=endpoint_url,
base_path=profile.base_path,
)
browse_connector_profile(profile, path=payload.base_path or "")
except ConnectorPolicyDenied as exc:
raise _connector_policy_error(exc) from exc
except (
ConnectorBrowseError,
ConnectorBrowseUnsupported,
OSError,
ValueError,
json.JSONDecodeError,
) as exc:
message = str(exc)
if "credentials were rejected" in message.casefold():
if payload.require_valid_credentials:
candidates.append(
{
"endpoint_url": endpoint_url,
"status": "credentials_rejected",
"message": "The endpoint exists, but the credentials were rejected.",
}
)
return FileConnectorDiscoveryResponse(
provider=payload.provider,
endpoint_url=endpoint_url,
base_path=payload.base_path,
status="credentials_rejected",
message="The endpoint was found, but login failed with these credentials.",
candidates=candidates,
metadata={"discovered_by": "webdav-auth-challenge"},
)
candidates.append(
{
"endpoint_url": endpoint_url,
"status": "found",
"message": "The endpoint exists, but credentials are required or were rejected.",
}
)
return FileConnectorDiscoveryResponse(
provider=payload.provider,
endpoint_url=endpoint_url,
base_path=payload.base_path,
status="found",
message="The endpoint was found. Add working credentials before saving or testing the connection.",
candidates=candidates,
metadata={"discovered_by": "webdav-auth-challenge"},
)
candidates.append(
{"endpoint_url": endpoint_url, "status": "failed", "message": message}
)
continue
status_value = (
"usable" if _same_endpoint(endpoint_url, payload.endpoint_url) else "found"
)
message = (
"The supplied URL is directly usable."
if status_value == "usable"
else "A usable connector endpoint was discovered."
)
candidates.append(
{"endpoint_url": endpoint_url, "status": status_value, "message": message}
)
return FileConnectorDiscoveryResponse(
provider=payload.provider,
endpoint_url=endpoint_url,
base_path=payload.base_path,
status=status_value,
message=message,
candidates=candidates,
metadata={"discovered_by": "webdav-propfind"},
)
return FileConnectorDiscoveryResponse(
provider=payload.provider,
endpoint_url=None,
base_path=payload.base_path,
status="not_found",
message="No usable WebDAV endpoint was found for this server URL.",
candidates=candidates,
)
@router.get("/connectors/credentials", response_model=FileConnectorCredentialsResponse)
def list_connector_credentials(
provider: str | None = None,
include_disabled: bool = False,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:read", "admin:settings:read"
)
),
):
provider_norm = provider.strip().casefold() if provider else None
credentials = list_database_connector_credentials(
session,
tenant_id=principal.tenant_id,
include_disabled=include_disabled
and _can_read_disabled_connector_profiles(principal),
)
return FileConnectorCredentialsResponse(
credentials=[
FileConnectorCredentialResponse(**credential.to_response())
for credential in credentials
if provider_norm is None or credential.provider in {None, provider_norm}
]
)
@router.get(
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
)
def read_connector_policy(
scope_type: str,
scope_id: str | None = Query(default=None),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:read", "admin:settings:read"
)
),
):
_require_connector_policy_read(principal, scope_type)
try:
return FileConnectorPolicyResponse(
**connector_policy_response(
session,
tenant_id=principal.tenant_id,
scope_type=scope_type,
scope_id=scope_id,
)
)
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
raise _http_error(exc, not_found=True) from exc
@router.put(
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
)
def write_connector_policy(
scope_type: str,
payload: FileConnectorPolicyUpdateRequest,
scope_id: str | None = Query(default=None),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
_require_connector_profile_write(principal, scope_type)
try:
set_connector_policy(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
scope_type=scope_type,
scope_id=scope_id,
policy=payload.policy,
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_POLICIES_COLLECTION,
resource_type=FILES_CONNECTOR_POLICY_RESOURCE,
resource_id=_file_connector_policy_resource_id(scope_type, scope_id),
operation="updated",
principal=principal,
tenant_id=None
if scope_type.strip().casefold() == "system"
else principal.tenant_id,
payload={"scope_type": scope_type.strip().casefold(), "scope_id": scope_id},
)
session.commit()
return FileConnectorPolicyResponse(
**connector_policy_response(
session,
tenant_id=principal.tenant_id,
scope_type=scope_type,
scope_id=scope_id,
)
)
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.post(
"/connectors/credentials",
response_model=FileConnectorCredentialResponse,
status_code=status.HTTP_201_CREATED,
)
def create_connector_credential(
payload: FileConnectorCredentialCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
_require_connector_credential_write(principal, payload.scope_type)
credentials = payload.credentials
try:
_ensure_connector_credential_configuration_allowed(
session,
principal,
credential_id=payload.id,
provider=payload.provider,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
operation="configure_credentials",
)
_ensure_connector_local_policy_allowed(
session,
principal,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
policy=payload.policy,
)
row = create_connector_credential_row(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
credential_id=payload.id,
label=payload.label,
provider=payload.provider,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
enabled=payload.enabled,
credential_mode=payload.credential_mode,
username=credentials.username,
password=credentials.password,
token=credentials.token,
password_env=credentials.password_env,
token_env=credentials.token_env,
secret_ref=credentials.secret_ref,
policy=payload.policy,
metadata=payload.metadata,
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
resource_id=row.id,
operation="created",
principal=principal,
tenant_id=row.tenant_id,
payload={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
},
)
session.commit()
session.refresh(row)
return _connector_credential_response(row)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.get(
"/connectors/credentials/{credential_id}",
response_model=FileConnectorCredentialResponse,
)
def get_connector_credential(
credential_id: str,
include_disabled: bool = False,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:read", "admin:settings:read"
)
),
):
try:
row = get_connector_credential_row(
session,
tenant_id=principal.tenant_id,
credential_id=credential_id,
include_disabled=include_disabled
and _can_read_disabled_connector_profiles(principal),
)
return _connector_credential_response(row)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
@router.patch(
"/connectors/credentials/{credential_id}",
response_model=FileConnectorCredentialResponse,
)
def update_connector_credential(
credential_id: str,
payload: FileConnectorCredentialUpdateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
try:
row = get_connector_credential_row(
session,
tenant_id=principal.tenant_id,
credential_id=credential_id,
include_disabled=True,
)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
_require_connector_credential_write(principal, row.scope_type)
credentials = payload.credentials
try:
provider = payload.provider if payload.provider is not None else row.provider
_ensure_connector_credential_configuration_allowed(
session,
principal,
credential_id=row.id,
provider=provider,
scope_type=row.scope_type,
scope_id=row.scope_id,
operation="configure_credentials",
)
if payload.policy is not None:
_ensure_connector_local_policy_allowed(
session,
principal,
scope_type=row.scope_type,
scope_id=row.scope_id,
policy=payload.policy,
)
update_connector_credential_row(
session,
row,
user_id=principal.user.id,
label=payload.label,
provider=payload.provider,
enabled=payload.enabled,
credential_mode=payload.credential_mode,
username=credentials.username if credentials else None,
password=credentials.password if credentials else None,
token=credentials.token if credentials else None,
password_env=credentials.password_env if credentials else None,
token_env=credentials.token_env if credentials else None,
secret_ref=credentials.secret_ref if credentials else None,
clear_password=payload.clear_password,
clear_token=payload.clear_token,
policy=payload.policy,
metadata=payload.metadata,
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
resource_id=row.id,
operation="updated",
principal=principal,
tenant_id=row.tenant_id,
payload={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
},
)
session.commit()
session.refresh(row)
return _connector_credential_response(row)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.delete(
"/connectors/credentials/{credential_id}",
response_model=FileConnectorCredentialResponse,
)
def deactivate_connector_credential(
credential_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
try:
row = get_connector_credential_row(
session,
tenant_id=principal.tenant_id,
credential_id=credential_id,
include_disabled=True,
)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
_require_connector_credential_write(principal, row.scope_type)
try:
deletion = delete_connector_credential_row(
session,
row,
deletion_reason="api_delete",
user_id=principal.user.id,
api_key_id=principal.api_key.id if principal.api_key else None,
)
if deletion.changed:
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
resource_id=row.id,
operation="deleted",
principal=principal,
tenant_id=row.tenant_id,
payload={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
},
)
for profile in deletion.affected_profiles:
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
resource_id=profile.id,
operation="updated",
principal=principal,
tenant_id=profile.tenant_id,
payload={
"scope_type": profile.scope_type,
"scope_id": profile.scope_id,
"provider": profile.provider,
"reason": "credential_deleted",
},
)
session.commit()
session.refresh(row)
return _connector_credential_response(row)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc) from exc
except Exception:
session.rollback()
raise
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
def list_connector_profiles(
provider: str | None = None,
campaign_id: str | None = None,
include_disabled: bool = False,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:read",
"files:file:upload",
"files:file:download",
"files:file:admin",
"system:settings:read",
"admin:settings:read",
)
),
):
try:
if campaign_id:
_ensure_campaign_file_access(session, principal, campaign_id)
profiles = _visible_connector_profiles(
session,
principal,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled
and _can_read_disabled_connector_profiles(principal),
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
include_effective_policy=False,
)
except (OSError, ValueError, json.JSONDecodeError) as exc:
raise _http_error(exc) from exc
return FileConnectorProfilesResponse(
profiles=[
FileConnectorProfileResponse(**profile.to_response())
for profile in profiles
]
)
@router.post(
"/connectors/profiles",
response_model=FileConnectorProfileResponse,
status_code=status.HTTP_201_CREATED,
)
def create_connector_profile(
payload: FileConnectorProfileCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(
require_any_scope(
"files:file:admin", "system:settings:write", "admin:settings:write"
)
),
):
_require_connector_profile_write(principal, payload.scope_type)
credentials = payload.credentials
try:
credential_row = _credential_row_for_profile(
session,
principal,
credential_profile_id=payload.credential_profile_id,
provider=payload.provider,
profile_id=payload.id,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
)
_ensure_connector_configuration_allowed(
session,
principal,
connector_id=payload.id,
credential_id=payload.credential_profile_id,
provider=payload.provider,
endpoint_url=connector_effective_endpoint_url(
provider=payload.provider,
endpoint_url=payload.endpoint_url,
metadata=payload.metadata,
),
base_path=payload.base_path,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
operation="configure",
)
_ensure_connector_local_policy_allowed(
session,
principal,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
policy=payload.policy,
)
row = create_connector_profile_row(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
profile_id=payload.id,
label=payload.label,
provider=payload.provider,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
endpoint_url=payload.endpoint_url,
base_path=payload.base_path,
enabled=payload.enabled,
credential_profile_id=payload.credential_profile_id,
credential_mode=payload.credential_mode,
username=credentials.username,
password=credentials.password,
token=credentials.token,
password_env=credentials.password_env,
token_env=credentials.token_env,
secret_ref=credentials.secret_ref,
capabilities=payload.capabilities,
policy=payload.policy,
metadata=payload.metadata,
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
resource_id=row.id,
operation="created",
principal=principal,
tenant_id=row.tenant_id,
payload={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
},
)
session.commit()
session.refresh(row)
return FileConnectorProfileResponse(
**connector_profile_from_row(
row, credential_row=credential_row
).to_response()
)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc

View File

@@ -0,0 +1,151 @@
from __future__ import annotations
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_files.backend.schemas import (
FileFolderCreateRequest,
FileFolderDeleteRequest,
FileFolderDeleteResponse,
FileFolderResponse,
FileFoldersResponse,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.paths import UnsafeFilePathError
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.folders import (
create_folder,
list_folders_for_user,
list_folders_for_user_window,
soft_delete_folder,
)
from govoplan_files.backend.route_support import (
_folder_response,
_http_error,
_is_admin,
)
from govoplan_files.backend.services.list_queries import (
FOLDERS_LIST_CURSOR_SCOPE,
_cursor_page_size,
_files_delta_watermark,
_folder_cursor_values,
_folders_list_fingerprint,
_next_folder_list_cursor,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.get("/folders", response_model=FileFoldersResponse)
def list_file_folders(
owner_type: Literal["user", "group"],
owner_id: str,
page_size: int | None = Query(default=None, ge=1, le=1000),
cursor: str | None = None,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
try:
watermark = _files_delta_watermark(session, principal.tenant_id)
effective_page_size = _cursor_page_size(
FOLDERS_LIST_CURSOR_SCOPE, cursor, page_size
)
if effective_page_size is None:
folders = list_folders_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
is_admin=_is_admin(principal),
)
return FileFoldersResponse(
folders=[_folder_response(folder) for folder in folders],
watermark=watermark,
)
fingerprint = _folders_list_fingerprint(
principal,
owner_type=owner_type,
owner_id=owner_id,
page_size=effective_page_size,
)
after_path, after_id = _folder_cursor_values(cursor, fingerprint=fingerprint)
folders, has_more = list_folders_for_user_window(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
is_admin=_is_admin(principal),
page_size=effective_page_size,
after_path=after_path,
after_id=after_id,
)
return FileFoldersResponse(
folders=[_folder_response(folder) for folder in folders],
cursor=cursor,
next_cursor=_next_folder_list_cursor(
principal,
folders,
owner_type=owner_type,
owner_id=owner_id,
page_size=effective_page_size,
has_more=has_more,
),
watermark=watermark,
)
except FileStorageError as exc:
raise _http_error(exc) from exc
@router.post("/folders", response_model=FileFolderResponse)
def create_file_folder(
payload: FileFolderCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
):
try:
folder = create_folder(
session,
tenant_id=principal.tenant_id,
owner_type=payload.owner_type,
owner_id=payload.owner_id,
user_id=principal.user.id,
path=payload.path,
is_admin=_is_admin(principal),
)
session.commit()
return _folder_response(folder)
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
def delete_file_folder(
payload: FileFolderDeleteRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
):
try:
deleted_folders, deleted_files = soft_delete_folder(
session,
tenant_id=principal.tenant_id,
owner_type=payload.owner_type,
owner_id=payload.owner_id,
user_id=principal.user.id,
path=payload.path,
recursive=payload.recursive,
is_admin=_is_admin(principal),
)
session.commit()
return FileFolderDeleteResponse(
deleted_folders=deleted_folders, deleted_files=deleted_files
)
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
session.rollback()
raise _http_error(exc) from exc

View File

@@ -0,0 +1,337 @@
from __future__ import annotations
import hashlib
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_core.audit.logging import audit_from_principal
from govoplan_core.db.session import get_session
from govoplan_files.backend.db.models import (
FileIntegrityFinding,
FileIntegrityScan,
)
from govoplan_files.backend.schemas import (
FileIntegrityActionRequest,
FileIntegrityActionResponse,
FileIntegrityFindingResponse,
FileIntegrityFindingsResponse,
FileIntegrityScanCreateRequest,
FileIntegrityScanResponse,
FileIntegrityScansResponse,
)
from govoplan_files.backend.storage.backends import StorageBackendError
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.integrity import (
cleanup_orphan_finding,
create_integrity_scan,
mark_integrity_scan_failed,
recheck_integrity_finding,
run_integrity_scan_batch,
)
router = APIRouter(prefix="/files/integrity", tags=["files-integrity"])
@router.get("/scans", response_model=FileIntegrityScansResponse)
def list_integrity_scans(
limit: int = Query(default=50, ge=1, le=200),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
) -> FileIntegrityScansResponse:
rows = (
session.query(FileIntegrityScan)
.filter(FileIntegrityScan.tenant_id == principal.tenant_id)
.order_by(FileIntegrityScan.created_at.desc(), FileIntegrityScan.id.desc())
.limit(limit)
.all()
)
return FileIntegrityScansResponse(
scans=[_scan_response(row) for row in rows]
)
@router.post(
"/scans",
response_model=FileIntegrityScanResponse,
status_code=status.HTTP_201_CREATED,
)
def create_scan(
payload: FileIntegrityScanCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
) -> FileIntegrityScanResponse:
try:
scan = create_integrity_scan(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
verify_checksums=payload.verify_checksums,
batch_size=payload.batch_size,
)
audit_from_principal(
session,
principal,
action="files.integrity.scan_created",
object_type="file_integrity_scan",
object_id=scan.id,
details={
"storage_backend": scan.storage_backend,
"verify_checksums": scan.verify_checksums,
"batch_size": scan.batch_size,
},
)
session.commit()
return _scan_response(scan)
except (FileStorageError, StorageBackendError) as exc:
session.rollback()
raise _integrity_http_error(exc) from exc
@router.post("/scans/{scan_id}/run", response_model=FileIntegrityScanResponse)
def run_scan_batch(
scan_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
) -> FileIntegrityScanResponse:
scan = _scan_for_tenant(session, scan_id, principal.tenant_id)
previous_status = scan.status
try:
run_integrity_scan_batch(session, scan)
if scan.status == "completed" and previous_status != "completed":
audit_from_principal(
session,
principal,
action="files.integrity.scan_completed",
object_type="file_integrity_scan",
object_id=scan.id,
details={
"verified_blob_count": scan.verified_blob_count,
"quarantined_blob_count": scan.quarantined_blob_count,
"orphan_object_count": scan.orphan_object_count,
},
)
session.commit()
return _scan_response(scan)
except (FileStorageError, StorageBackendError) as exc:
session.rollback()
scan = _scan_for_tenant(session, scan_id, principal.tenant_id)
mark_integrity_scan_failed(scan, error=exc)
audit_from_principal(
session,
principal,
action="files.integrity.scan_failed",
object_type="file_integrity_scan",
object_id=scan.id,
details={"error_type": type(exc).__name__},
)
session.commit()
raise _integrity_http_error(exc) from exc
@router.get(
"/scans/{scan_id}/findings",
response_model=FileIntegrityFindingsResponse,
)
def list_integrity_findings(
scan_id: str,
state_filter: str | None = Query(default=None, alias="state"),
limit: int = Query(default=500, ge=1, le=1000),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
) -> FileIntegrityFindingsResponse:
scan = _scan_for_tenant(session, scan_id, principal.tenant_id)
query = session.query(FileIntegrityFinding).filter(
FileIntegrityFinding.scan_id == scan.id,
FileIntegrityFinding.tenant_id == principal.tenant_id,
)
if state_filter:
query = query.filter(FileIntegrityFinding.state == state_filter)
rows = (
query.order_by(
FileIntegrityFinding.created_at.asc(),
FileIntegrityFinding.id.asc(),
)
.limit(limit)
.all()
)
return FileIntegrityFindingsResponse(
findings=[_finding_response(row) for row in rows]
)
@router.post(
"/findings/{finding_id}/recheck",
response_model=FileIntegrityActionResponse,
)
def recheck_finding(
finding_id: str,
payload: FileIntegrityActionRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
) -> FileIntegrityActionResponse:
finding = _finding_for_tenant(session, finding_id, principal.tenant_id)
try:
result = recheck_integrity_finding(
session,
finding,
user_id=principal.user.id,
dry_run=payload.dry_run,
)
_audit_integrity_action(session, principal, result)
session.commit()
return _action_response(result)
except (FileStorageError, StorageBackendError) as exc:
session.rollback()
raise _integrity_http_error(exc) from exc
@router.post(
"/findings/{finding_id}/cleanup",
response_model=FileIntegrityActionResponse,
)
def cleanup_finding(
finding_id: str,
payload: FileIntegrityActionRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
) -> FileIntegrityActionResponse:
finding = _finding_for_tenant(session, finding_id, principal.tenant_id)
try:
result = cleanup_orphan_finding(
session,
finding,
user_id=principal.user.id,
dry_run=payload.dry_run,
)
_audit_integrity_action(session, principal, result)
session.commit()
return _action_response(result)
except (FileStorageError, StorageBackendError) as exc:
session.rollback()
raise _integrity_http_error(exc) from exc
def _scan_for_tenant(
session: Session,
scan_id: str,
tenant_id: str,
) -> FileIntegrityScan:
scan = session.get(FileIntegrityScan, scan_id)
if scan is None or scan.tenant_id != tenant_id:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Integrity scan not found",
)
return scan
def _finding_for_tenant(
session: Session,
finding_id: str,
tenant_id: str,
) -> FileIntegrityFinding:
finding = session.get(FileIntegrityFinding, finding_id)
if finding is None or finding.tenant_id != tenant_id:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Integrity finding not found",
)
return finding
def _audit_integrity_action(session, principal, result) -> None:
audit_from_principal(
session,
principal,
action=f"files.integrity.{result.action}",
object_type="file_integrity_finding",
object_id=result.finding.id,
details={
"scan_id": result.finding.scan_id,
"kind": result.finding.kind,
"dry_run": result.dry_run,
"changed": result.changed,
"storage_key_sha256": hashlib.sha256(
result.finding.storage_key.encode("utf-8")
).hexdigest(),
"inspection": result.inspection.kind
if result.inspection
else None,
},
)
def _scan_response(scan: FileIntegrityScan) -> FileIntegrityScanResponse:
return FileIntegrityScanResponse(
id=scan.id,
tenant_id=scan.tenant_id,
storage_backend=scan.storage_backend,
storage_prefix=scan.storage_prefix,
status=scan.status,
phase=scan.phase,
verify_checksums=scan.verify_checksums,
batch_size=scan.batch_size,
scanned_blob_count=scan.scanned_blob_count,
verified_blob_count=scan.verified_blob_count,
quarantined_blob_count=scan.quarantined_blob_count,
scanned_object_count=scan.scanned_object_count,
orphan_object_count=scan.orphan_object_count,
created_by_user_id=scan.created_by_user_id,
started_at=scan.started_at.isoformat() if scan.started_at else None,
completed_at=scan.completed_at.isoformat()
if scan.completed_at
else None,
last_error=scan.last_error,
created_at=scan.created_at.isoformat(),
updated_at=scan.updated_at.isoformat(),
)
def _finding_response(
finding: FileIntegrityFinding,
) -> FileIntegrityFindingResponse:
return FileIntegrityFindingResponse(
id=finding.id,
scan_id=finding.scan_id,
tenant_id=finding.tenant_id,
kind=finding.kind,
state=finding.state,
blob_id=finding.blob_id,
storage_key=finding.storage_key,
expected_size_bytes=finding.expected_size_bytes,
observed_size_bytes=finding.observed_size_bytes,
expected_checksum_sha256=finding.expected_checksum_sha256,
observed_checksum_sha256=finding.observed_checksum_sha256,
resolved_at=finding.resolved_at.isoformat()
if finding.resolved_at
else None,
resolved_by_user_id=finding.resolved_by_user_id,
created_at=finding.created_at.isoformat(),
updated_at=finding.updated_at.isoformat(),
)
def _action_response(result) -> FileIntegrityActionResponse:
return FileIntegrityActionResponse(
action=result.action,
changed=result.changed,
dry_run=result.dry_run,
finding=_finding_response(result.finding),
inspection_kind=result.inspection.kind if result.inspection else None,
inspection_valid=result.inspection.valid if result.inspection else None,
)
def _integrity_http_error(exc: Exception) -> HTTPException:
if isinstance(exc, FileStorageError):
return HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=str(exc),
)
return HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="The configured file storage backend could not complete the integrity operation",
)

View File

@@ -0,0 +1,167 @@
from __future__ import annotations
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_files.backend.schemas import (
FileDeltaResponse,
FileListResponse,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.files import (
count_assets_for_user,
list_assets_for_user,
list_assets_for_user_window,
)
from govoplan_files.backend.route_support import (
_asset_list_response,
_ensure_campaign_file_access,
_ensure_list_owner_access,
_is_admin,
)
from govoplan_files.backend.services.list_queries import (
FILES_LIST_CURSOR_SCOPE,
_cursor_page_size,
_file_cursor_values,
_files_delta_response,
_files_delta_watermark,
_files_list_fingerprint,
_full_file_delta_response,
_next_file_list_cursor,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.get("/delta", response_model=FileDeltaResponse)
def files_delta(
owner_type: Literal["user", "group"] | None = None,
owner_id: str | None = None,
campaign_id: str | None = None,
path_prefix: str | None = None,
since: str | None = None,
limit: int = Query(default=500, ge=1, le=1000),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
_ensure_list_owner_access(session, principal, owner_type, owner_id)
_ensure_campaign_file_access(session, principal, campaign_id)
if since is None:
return _full_file_delta_response(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
)
return _files_delta_response(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
since=since,
limit=limit,
)
@router.get("", response_model=FileListResponse)
def list_files(
owner_type: Literal["user", "group"] | None = None,
owner_id: str | None = None,
campaign_id: str | None = None,
path_prefix: str | None = None,
campaign_usage: Literal["linked", "unlinked"] | None = None,
audit_relevant: bool | None = None,
page_size: int | None = Query(default=None, ge=1, le=1000),
cursor: str | None = None,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
_ensure_list_owner_access(session, principal, owner_type, owner_id)
_ensure_campaign_file_access(session, principal, campaign_id)
watermark = _files_delta_watermark(session, principal.tenant_id)
total = count_assets_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
is_admin=_is_admin(principal),
)
effective_page_size = _cursor_page_size(FILES_LIST_CURSOR_SCOPE, cursor, page_size)
if effective_page_size is not None:
fingerprint = _files_list_fingerprint(
principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
page_size=effective_page_size,
)
after_display_path, after_updated_at, after_id = _file_cursor_values(
cursor, fingerprint=fingerprint
)
assets, has_more = list_assets_for_user_window(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
is_admin=_is_admin(principal),
page_size=effective_page_size,
after_display_path=after_display_path,
after_updated_at=after_updated_at,
after_id=after_id,
)
return FileListResponse(
files=_asset_list_response(session, assets, include_shares=True),
total=total,
cursor=cursor,
next_cursor=_next_file_list_cursor(
principal,
assets,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
page_size=effective_page_size,
has_more=has_more,
),
watermark=watermark,
)
assets = list_assets_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
is_admin=_is_admin(principal),
)
return FileListResponse(
files=_asset_list_response(session, assets, include_shares=True),
total=total,
watermark=watermark,
)

View File

@@ -0,0 +1,330 @@
from __future__ import annotations
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session
from govoplan_core.api.v1.schemas import (
ReferenceOptionListResponse,
ReferenceOptionResponse,
)
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_core.audit.logging import audit_from_principal
from govoplan_core.core.references import (
access_scope_reference_page,
access_scope_reference_provider_available,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.runtime import get_registry
from govoplan_files.backend.schemas import (
BulkFileShareRequest,
BulkFileShareResponse,
FileShareRequest,
FileShareResponse,
FileSharesResponse,
)
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import (
current_file_share_for_target,
get_asset_for_share_management,
list_file_shares,
revoke_file_share,
share_file,
share_files,
)
from govoplan_files.backend.route_support import (
_file_share_response,
_http_error,
_is_admin,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.get(
"/{file_id}/share-target-options",
response_model=ReferenceOptionListResponse,
)
def search_share_targets(
file_id: str,
target_type: str,
q: str = "",
selected: list[str] = Query(default=[]),
limit: int = Query(default=50, ge=1, le=200),
cursor: str | None = None,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
) -> ReferenceOptionListResponse:
if target_type not in {"user", "group"}:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="Share target type must be user or group",
)
try:
get_asset_for_share_management(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
registry = get_registry()
page = access_scope_reference_page(
registry,
principal,
scope_type=target_type,
reference_kind="membership" if target_type == "user" else "group",
query=q,
selected_values=selected,
limit=limit,
cursor=cursor,
administrative=True,
session=session,
)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail=str(exc),
) from exc
except FileStorageError as exc:
raise _http_error(exc) from exc
return ReferenceOptionListResponse(
options=[
ReferenceOptionResponse(**option.to_dict()) for option in page.options
],
provider_available=access_scope_reference_provider_available(registry),
next_cursor=page.next_cursor,
has_more=page.has_more,
)
@router.get("/{file_id}/shares", response_model=FileSharesResponse)
def list_shares(
file_id: str,
include_inactive: bool = False,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
):
try:
asset = get_asset_for_share_management(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
return FileSharesResponse(
shares=[
_file_share_response(share)
for share in list_file_shares(
session,
tenant_id=principal.tenant_id,
asset_id=asset.id,
include_inactive=include_inactive,
)
]
)
except FileStorageError as exc:
raise _http_error(exc) from exc
@router.post("/{file_id}/shares", response_model=FileShareResponse)
def create_share(
file_id: str,
payload: FileShareRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
):
try:
asset = get_asset_for_share_management(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
previous = current_file_share_for_target(
session,
tenant_id=principal.tenant_id,
asset_id=asset.id,
target_type=payload.target_type,
target_id=payload.target_id,
)
previous_permission = previous.permission if previous else None
previous_expiry = previous.expires_at if previous else None
share = share_file(
session,
tenant_id=principal.tenant_id,
asset=asset,
target_type=payload.target_type,
target_id=payload.target_id,
permission=payload.permission,
user_id=principal.user.id,
expires_at=payload.expires_at,
)
session.flush()
action = _share_audit_action(
existed=previous is not None,
previous_permission=previous_permission,
permission=share.permission,
previous_expiry=previous_expiry,
expiry=share.expires_at,
)
if action:
_audit_share_change(session, principal, share, action=action)
session.commit()
return _file_share_response(share)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc) from exc
@router.post("/bulk-shares", response_model=BulkFileShareResponse)
def create_bulk_shares(
payload: BulkFileShareRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
):
try:
file_ids = list(dict.fromkeys(payload.file_ids))
assets = [
get_asset_for_share_management(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
for file_id in file_ids
]
previous_by_asset = {
asset.id: current_file_share_for_target(
session,
tenant_id=principal.tenant_id,
asset_id=asset.id,
target_type=payload.target_type,
target_id=payload.target_id,
)
for asset in assets
}
previous_values = {
asset_id: (
share.permission if share else None,
share.expires_at if share else None,
)
for asset_id, share in previous_by_asset.items()
}
shares = share_files(
session,
tenant_id=principal.tenant_id,
assets=assets,
target_type=payload.target_type,
target_id=payload.target_id,
permission=payload.permission,
user_id=principal.user.id,
expires_at=payload.expires_at,
)
session.flush()
for share in shares:
previous_permission, previous_expiry = previous_values[share.file_asset_id]
action = _share_audit_action(
existed=previous_by_asset[share.file_asset_id] is not None,
previous_permission=previous_permission,
permission=share.permission,
previous_expiry=previous_expiry,
expiry=share.expires_at,
)
if action:
_audit_share_change(session, principal, share, action=action)
session.commit()
return BulkFileShareResponse(
shared_count=len(shares),
shares=[_file_share_response(share) for share in shares],
)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc) from exc
@router.delete("/{file_id}/shares/{share_id}", response_model=FileShareResponse)
def revoke_share(
file_id: str,
share_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
):
try:
asset = get_asset_for_share_management(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
share, changed = revoke_file_share(
session,
tenant_id=principal.tenant_id,
asset_id=asset.id,
share_id=share_id,
user_id=principal.user.id,
)
if changed:
_audit_share_change(
session, principal, share, action="files.share.revoked"
)
session.commit()
return _file_share_response(share)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc) from exc
def _share_audit_action(
*,
existed: bool,
previous_permission: str | None,
permission: str,
previous_expiry: datetime | None,
expiry: datetime | None,
) -> str | None:
if not existed:
return "files.share.granted"
permission_changed = previous_permission != permission
expiry_changed = _normalized_expiry(previous_expiry) != _normalized_expiry(expiry)
if permission_changed:
return "files.share.changed"
if expiry_changed:
return "files.share.expiry_changed"
return None
def _audit_share_change(
session: Session,
principal: ApiPrincipal,
share,
*,
action: str,
) -> None:
audit_from_principal(
session,
principal,
action=action,
object_type="file_share",
object_id=share.id,
details={
"file_asset_id": share.file_asset_id,
"target_type": share.target_type,
"target_id": share.target_id,
"permission": share.permission,
"expires_at": _normalized_expiry(share.expires_at),
},
)
def _normalized_expiry(value: datetime | None) -> str | None:
if value is None:
return None
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc).isoformat()

View File

@@ -0,0 +1,292 @@
from __future__ import annotations
import json
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_files.backend.change_tracking import (
FILES_CONNECTOR_SPACES_COLLECTION,
)
from govoplan_files.backend.schemas import (
FileConnectorSpaceCreateRequest,
FileConnectorSpaceResponse,
FileConnectorSpacesResponse,
FileConnectorSpaceUpdateRequest,
FileSpaceResponse,
FileSpacesResponse,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.access import group_refs_for_ids, user_group_ids
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_spaces import (
connector_space_owner_id,
create_connector_space,
get_connector_space_for_user,
list_connector_spaces_for_user,
soft_delete_connector_space,
update_connector_space,
)
from govoplan_files.backend.storage.connector_policy import (
ConnectorPolicyDenied,
)
from govoplan_files.backend.route_support import (
FILES_CONNECTOR_SPACE_RESOURCE,
_connector_policy_error,
_connector_space_file_space_response,
_connector_space_policy_decision,
_connector_space_response,
_http_error,
_is_admin,
_record_connector_settings_change,
_visible_connector_profile,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.get("/spaces", response_model=FileSpacesResponse)
def list_file_spaces(
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
spaces = [
FileSpaceResponse(
id=f"user:{principal.user.id}",
label="My files",
owner_type="user",
owner_id=principal.user.id,
description="Files owned by your user account.",
)
]
group_ids = user_group_ids(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
include_admin_groups=_is_admin(principal),
)
if group_ids:
groups = group_refs_for_ids(tenant_id=principal.tenant_id, group_ids=group_ids)
spaces.extend(
FileSpaceResponse(
id=f"group:{group.id}",
label=f"{group.name} files",
owner_type="group",
owner_id=group.id,
description="Files owned by this group.",
)
for group in groups
)
connector_spaces = list_connector_spaces_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
is_admin=_is_admin(principal),
)
spaces.extend(
_connector_space_file_space_response(space) for space in connector_spaces
)
return FileSpacesResponse(spaces=spaces)
@router.get("/connector-spaces", response_model=FileConnectorSpacesResponse)
def list_file_connector_spaces(
owner_type: Literal["user", "group"] | None = None,
owner_id: str | None = None,
include_inactive: bool = False,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
try:
spaces = list_connector_spaces_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
include_inactive=include_inactive and _is_admin(principal),
is_admin=_is_admin(principal),
)
return FileConnectorSpacesResponse(
spaces=[_connector_space_response(space) for space in spaces]
)
except FileStorageError as exc:
raise _http_error(exc) from exc
@router.post(
"/connector-spaces",
response_model=FileConnectorSpaceResponse,
status_code=status.HTTP_201_CREATED,
)
def create_file_connector_space(
payload: FileConnectorSpaceCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
):
if payload.owner_type == "group" and not payload.owner_id:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="owner_id is required for group connector spaces",
)
target_owner = payload.owner_id or principal.user.id
try:
profile = _visible_connector_profile(
session, principal, payload.connector_profile_id
)
decision = _connector_space_policy_decision(
profile,
library_id=payload.library_id,
remote_path=payload.remote_path,
operation="link",
)
if not decision.allowed:
raise ConnectorPolicyDenied(decision)
space = create_connector_space(
session,
tenant_id=principal.tenant_id,
owner_type=payload.owner_type,
owner_id=target_owner,
user_id=principal.user.id,
label=payload.label,
profile=profile,
library_id=payload.library_id,
remote_path=payload.remote_path,
sync_mode=payload.sync_mode,
metadata=payload.metadata,
is_admin=_is_admin(principal),
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_SPACES_COLLECTION,
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
resource_id=space.id,
operation="created",
principal=principal,
tenant_id=space.tenant_id,
payload={
"owner_type": space.owner_type,
"owner_id": connector_space_owner_id(space),
},
)
session.commit()
return _connector_space_response(space)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.patch("/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse)
def update_file_connector_space(
space_id: str,
payload: FileConnectorSpaceUpdateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
):
try:
space = get_connector_space_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
space_id=space_id,
include_inactive=_is_admin(principal),
is_admin=_is_admin(principal),
)
except FileStorageError as exc:
raise _http_error(exc, not_found=True) from exc
try:
if payload.library_id is not None or payload.remote_path is not None:
profile = _visible_connector_profile(
session, principal, space.connector_profile_id
)
decision = _connector_space_policy_decision(
profile,
library_id=payload.library_id
if payload.library_id is not None
else space.library_id,
remote_path=payload.remote_path
if payload.remote_path is not None
else space.remote_path,
operation="link",
)
if not decision.allowed:
raise ConnectorPolicyDenied(decision)
update_connector_space(
session,
space,
user_id=principal.user.id,
label=payload.label,
library_id=payload.library_id,
remote_path=payload.remote_path,
sync_mode=payload.sync_mode,
is_active=payload.is_active,
metadata=payload.metadata,
is_admin=_is_admin(principal),
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_SPACES_COLLECTION,
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
resource_id=space.id,
operation="updated",
principal=principal,
tenant_id=space.tenant_id,
payload={
"owner_type": space.owner_type,
"owner_id": connector_space_owner_id(space),
},
)
session.commit()
return _connector_space_response(space)
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.delete(
"/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse
)
def delete_file_connector_space(
space_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
):
try:
space = get_connector_space_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
space_id=space_id,
include_inactive=True,
is_admin=_is_admin(principal),
)
soft_delete_connector_space(
session, space, user_id=principal.user.id, is_admin=_is_admin(principal)
)
_record_connector_settings_change(
session,
collection=FILES_CONNECTOR_SPACES_COLLECTION,
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
resource_id=space.id,
operation="deleted",
principal=principal,
tenant_id=space.tenant_id,
payload={
"owner_type": space.owner_type,
"owner_id": connector_space_owner_id(space),
},
)
session.commit()
return _connector_space_response(space)
except FileStorageError as exc:
session.rollback()
raise _http_error(exc, not_found=True) from exc

View File

@@ -0,0 +1,213 @@
from __future__ import annotations
import tempfile
from fastapi import APIRouter, Depends
from fastapi.responses import FileResponse
from starlette.background import BackgroundTask
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_files.backend.schemas import (
ArchiveRequest,
PatternMatchResponse,
PatternResolveRequest,
PatternResolveResponse,
RenamePreviewItem,
RenameRequest,
RenameResponse,
TransferRequest,
TransferResponse,
_conflict_resolutions,
)
from govoplan_core.db.session import get_session
from govoplan_files.backend.storage.paths import (
UnsafeFilePathError,
filename_from_path,
normalize_logical_path,
)
from govoplan_files.backend.storage.archives import create_zip_file
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import (
get_asset_for_user,
list_assets_for_user,
)
from govoplan_files.backend.storage.search import resolve_patterns
from govoplan_files.backend.storage.transfers import (
rename_selection,
transfer_selection,
)
from govoplan_files.backend.route_support import (
_asset_response,
_attachment_disposition,
_audit_connector_access,
_cleanup_temp_file,
_ensure_campaign_file_access,
_ensure_list_owner_access,
_http_error,
_is_admin,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.post("/bulk-rename", response_model=RenameResponse)
def bulk_rename(
payload: RenameRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
):
try:
plan = rename_selection(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
file_ids=payload.file_ids,
folder_paths=payload.folder_paths,
owner_type=payload.owner_type,
owner_id=payload.owner_id,
mode=payload.mode,
new_name=payload.new_name,
find=payload.find,
replacement=payload.replacement,
prefix=payload.prefix,
suffix=payload.suffix,
recursive=payload.recursive,
dry_run=payload.dry_run,
is_admin=_is_admin(principal),
)
if not payload.dry_run:
session.commit()
return RenameResponse(
dry_run=payload.dry_run,
items=[
RenamePreviewItem(
kind=item.kind,
id=item.id,
file_id=item.id if item.kind == "file" else None,
folder_path=item.old_path if item.kind == "folder" else None,
old_path=item.old_path,
new_path=item.new_path,
)
for item in plan
],
)
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.post("/transfer", response_model=TransferResponse)
def transfer_files(
payload: TransferRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
):
try:
files, folders = transfer_selection(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
operation=payload.operation,
file_ids=payload.file_ids,
folder_paths=payload.folder_paths,
source_owner_type=payload.source_owner_type,
source_owner_id=payload.source_owner_id,
target_owner_type=payload.target_owner_type,
target_owner_id=payload.target_owner_id,
target_folder=payload.target_folder,
conflict_strategy=payload.conflict_strategy,
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
is_admin=_is_admin(principal),
)
session.commit()
return TransferResponse(
operation=payload.operation, files=files, folders=folders
)
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
session.rollback()
raise _http_error(exc) from exc
@router.post("/archive.zip")
def download_archive(
payload: ArchiveRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
):
try:
assets = [
get_asset_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
asset_id=file_id,
is_admin=_is_admin(principal),
)
for file_id in payload.file_ids
]
tmp = tempfile.NamedTemporaryFile(
prefix="govoplan-files-", suffix=".zip", delete=False
)
tmp_path = tmp.name
tmp.close()
try:
create_zip_file(session, assets, tmp_path)
except Exception:
_cleanup_temp_file(tmp_path)
raise
_audit_connector_access(session, principal, assets, operation="archive")
except FileStorageError as exc:
raise _http_error(exc) from exc
filename = filename_from_path(
normalize_logical_path(payload.filename, fallback_filename="files.zip")
)
headers = {"Content-Disposition": _attachment_disposition(filename)}
return FileResponse(
tmp_path,
media_type="application/zip",
headers=headers,
background=BackgroundTask(_cleanup_temp_file, tmp_path),
)
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
def resolve_file_patterns(
payload: PatternResolveRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
):
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
_ensure_campaign_file_access(session, principal, payload.campaign_id)
try:
assets = list_assets_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=payload.owner_type,
owner_id=payload.owner_id,
campaign_id=payload.campaign_id,
path_prefix=payload.path_prefix,
is_admin=_is_admin(principal),
)
resolved, unmatched = resolve_patterns(
assets,
payload.patterns,
base_path=payload.path_prefix,
case_sensitive=payload.case_sensitive,
)
return PatternResolveResponse(
patterns=[
PatternMatchResponse(
pattern=item.pattern,
matches=[_asset_response(session, asset) for asset in item.matches],
)
for item in resolved
],
unmatched=[_asset_response(session, asset) for asset in unmatched]
if payload.include_unmatched
else [],
)
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
raise _http_error(exc) from exc

View File

@@ -0,0 +1,207 @@
from __future__ import annotations
import json
from typing import Literal
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal, require_scope
from govoplan_files.backend.schemas import (
ConflictResolutionRequest,
FileUploadResponse,
_conflict_resolutions,
)
from govoplan_files.backend.db.models import FileAsset
from govoplan_core.db.session import get_session
from govoplan_files.backend.runtime import settings
from govoplan_files.backend.storage.paths import UnsafeFilePathError
from govoplan_files.backend.storage.archives import extract_zip_upload
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_policy import (
ConnectorPolicyDenied,
)
from govoplan_files.backend.storage.files import (
create_file_asset,
)
from govoplan_files.backend.route_support import (
_asset_response,
_audit_connector_imports,
_cleanup_temp_file,
_connector_policy_error,
_enforce_connector_policy,
_http_error,
_is_admin,
_read_limited_upload,
_source_metadata_from_form,
_spool_limited_upload_to_temp,
)
router = APIRouter(prefix="/files", tags=["files"])
@router.post("/upload", response_model=FileUploadResponse)
def upload_files(
files: list[UploadFile] = FastAPIFile(...),
owner_type: Literal["user", "group"] = Form(default="user"),
owner_id: str | None = Form(default=None),
path: str = Form(default=""),
campaign_id: str | None = Form(default=None),
unpack_zip: bool = Form(default=False),
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
default="reject"
),
conflict_resolutions_json: str | None = Form(default=None),
source_provenance_json: str | None = Form(default=None),
source_revision: str | None = Form(default=None),
connector_policy_json: str | None = Form(default=None),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
):
target_owner = owner_id or principal.user.id
uploaded_assets: list[FileAsset] = []
try:
raw_resolutions = (
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
)
upload_resolutions = _conflict_resolutions(
[ConflictResolutionRequest(**item) for item in raw_resolutions]
)
_enforce_connector_policy(
source_provenance_json, connector_policy_json, operation="import"
)
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
for upload in files:
filename = upload.filename or "file"
content_type = upload.content_type or None
upload_limit = (
settings.file_upload_zip_max_bytes
if unpack_zip and filename.lower().endswith(".zip")
else settings.file_upload_max_bytes
)
if unpack_zip and filename.lower().endswith(".zip"):
zip_path = _spool_limited_upload_to_temp(
upload, max_bytes=upload_limit, suffix=".zip"
)
try:
extracted = extract_zip_upload(
session,
tenant_id=principal.tenant_id,
owner_type=owner_type,
owner_id=target_owner,
user_id=principal.user.id,
zip_data=zip_path,
folder=path,
campaign_id=campaign_id,
conflict_strategy=conflict_strategy,
conflict_resolutions=upload_resolutions,
metadata=metadata,
is_admin=_is_admin(principal),
max_file_bytes=settings.file_upload_max_bytes,
max_total_bytes=settings.file_upload_zip_max_bytes,
)
finally:
_cleanup_temp_file(zip_path)
uploaded_assets.extend(item.asset for item in extracted)
continue
data = _read_limited_upload(upload, max_bytes=upload_limit)
stored = create_file_asset(
session,
tenant_id=principal.tenant_id,
owner_type=owner_type,
owner_id=target_owner,
user_id=principal.user.id,
filename=filename,
data=data,
folder=path,
content_type=content_type,
campaign_id=campaign_id,
conflict_strategy=conflict_strategy,
conflict_resolutions=upload_resolutions,
metadata=metadata,
is_admin=_is_admin(principal),
)
uploaded_assets.append(stored.asset)
_audit_connector_imports(session, principal, uploaded_assets)
session.commit()
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
session.rollback()
raise _http_error(exc) from exc
return FileUploadResponse(
files=[
_asset_response(session, asset, include_shares=True)
for asset in uploaded_assets
]
)
@router.post("/upload-zip", response_model=FileUploadResponse)
def upload_zip(
file: UploadFile = FastAPIFile(...),
owner_type: Literal["user", "group"] = Form(default="user"),
owner_id: str | None = Form(default=None),
path: str = Form(default=""),
campaign_id: str | None = Form(default=None),
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
default="reject"
),
conflict_resolutions_json: str | None = Form(default=None),
source_provenance_json: str | None = Form(default=None),
source_revision: str | None = Form(default=None),
connector_policy_json: str | None = Form(default=None),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
):
target_owner = owner_id or principal.user.id
zip_path: str | None = None
try:
raw_resolutions = (
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
)
upload_resolutions = _conflict_resolutions(
[ConflictResolutionRequest(**item) for item in raw_resolutions]
)
_enforce_connector_policy(
source_provenance_json, connector_policy_json, operation="import"
)
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
zip_path = _spool_limited_upload_to_temp(
file, max_bytes=settings.file_upload_zip_max_bytes, suffix=".zip"
)
extracted = extract_zip_upload(
session,
tenant_id=principal.tenant_id,
owner_type=owner_type,
owner_id=target_owner,
user_id=principal.user.id,
zip_data=zip_path,
folder=path,
campaign_id=campaign_id,
conflict_strategy=conflict_strategy,
conflict_resolutions=upload_resolutions,
metadata=metadata,
is_admin=_is_admin(principal),
max_file_bytes=settings.file_upload_max_bytes,
max_total_bytes=settings.file_upload_zip_max_bytes,
)
_audit_connector_imports(session, principal, [item.asset for item in extracted])
session.commit()
except ConnectorPolicyDenied as exc:
session.rollback()
raise _connector_policy_error(exc) from exc
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
session.rollback()
raise _http_error(exc) from exc
finally:
if zip_path:
_cleanup_temp_file(zip_path)
return FileUploadResponse(
files=[
_asset_response(session, item.asset, include_shares=True)
for item in extracted
]
)

View File

@@ -1,5 +1,6 @@
from __future__ import annotations from __future__ import annotations
from datetime import datetime
from typing import Any, Literal from typing import Any, Literal
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
@@ -73,11 +74,86 @@ class FileConnectorSpacesResponse(BaseModel):
class FileShareResponse(BaseModel): class FileShareResponse(BaseModel):
id: str id: str
file_asset_id: str
target_type: str target_type: str
target_id: str target_id: str
permission: str permission: str
created_by_user_id: str | None = None
created_at: str created_at: str
expires_at: str | None = None
revoked_at: str | None = None revoked_at: str | None = None
revoked_by_user_id: str | None = None
active: bool
class FileSharesResponse(BaseModel):
shares: list[FileShareResponse] = Field(default_factory=list)
class FileIntegrityScanCreateRequest(BaseModel):
verify_checksums: bool = True
batch_size: int = Field(default=100, ge=1, le=1000)
class FileIntegrityScanResponse(BaseModel):
id: str
tenant_id: str
storage_backend: str
storage_prefix: str
status: str
phase: str
verify_checksums: bool
batch_size: int
scanned_blob_count: int
verified_blob_count: int
quarantined_blob_count: int
scanned_object_count: int
orphan_object_count: int
created_by_user_id: str | None = None
started_at: str | None = None
completed_at: str | None = None
last_error: str | None = None
created_at: str
updated_at: str
class FileIntegrityScansResponse(BaseModel):
scans: list[FileIntegrityScanResponse] = Field(default_factory=list)
class FileIntegrityFindingResponse(BaseModel):
id: str
scan_id: str
tenant_id: str
kind: str
state: str
blob_id: str | None = None
storage_key: str
expected_size_bytes: int | None = None
observed_size_bytes: int | None = None
expected_checksum_sha256: str | None = None
observed_checksum_sha256: str | None = None
resolved_at: str | None = None
resolved_by_user_id: str | None = None
created_at: str
updated_at: str
class FileIntegrityFindingsResponse(BaseModel):
findings: list[FileIntegrityFindingResponse] = Field(default_factory=list)
class FileIntegrityActionRequest(BaseModel):
dry_run: bool = True
class FileIntegrityActionResponse(BaseModel):
action: str
changed: bool
dry_run: bool
finding: FileIntegrityFindingResponse
inspection_kind: str | None = None
inspection_valid: bool | None = None
class FileSourceProvenance(BaseModel): class FileSourceProvenance(BaseModel):
@@ -154,6 +230,7 @@ class FileFolderDeleteResponse(BaseModel):
class FileListResponse(BaseModel): class FileListResponse(BaseModel):
files: list[FileAssetResponse] files: list[FileAssetResponse]
total: int
cursor: str | None = None cursor: str | None = None
next_cursor: str | None = None next_cursor: str | None = None
watermark: str | None = None watermark: str | None = None
@@ -452,6 +529,7 @@ class FileShareRequest(BaseModel):
target_type: Literal["user", "group", "campaign", "tenant"] target_type: Literal["user", "group", "campaign", "tenant"]
target_id: str target_id: str
permission: Literal["read", "write", "manage"] = "read" permission: Literal["read", "write", "manage"] = "read"
expires_at: datetime | None = None
class BulkFileShareRequest(BaseModel): class BulkFileShareRequest(BaseModel):
@@ -459,6 +537,7 @@ class BulkFileShareRequest(BaseModel):
target_type: Literal["user", "group", "campaign", "tenant"] target_type: Literal["user", "group", "campaign", "tenant"]
target_id: str target_id: str
permission: Literal["read", "write", "manage"] = "read" permission: Literal["read", "write", "manage"] = "read"
expires_at: datetime | None = None
class BulkFileShareResponse(BaseModel): class BulkFileShareResponse(BaseModel):

View File

@@ -0,0 +1 @@
"""Query and response assembly services used by Files routes."""

View File

@@ -0,0 +1,292 @@
from __future__ import annotations
from typing import Literal
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal
from govoplan_core.api.v1.schemas import DeltaDeletedItem
from govoplan_core.core.change_sequence import (
ChangeSequenceEntry,
)
from govoplan_files.backend.change_tracking import (
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
FILES_CONNECTOR_POLICIES_COLLECTION,
FILES_CONNECTOR_PROFILES_COLLECTION,
FILES_CONNECTOR_SPACES_COLLECTION,
)
from govoplan_files.backend.schemas import (
FileConnectorCredentialResponse,
FileConnectorSettingsDeltaResponse,
FileConnectorPolicyResponse,
FileConnectorProfileResponse,
)
from govoplan_files.backend.db.models import FileConnectorSpace
from govoplan_files.backend.storage.connector_credential_store import (
ConnectorCredential,
)
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.connector_policy_store import (
connector_policy_response,
)
from govoplan_files.backend.route_support import (
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
FILES_CONNECTOR_PROFILE_RESOURCE,
FILES_CONNECTOR_SPACE_RESOURCE,
_can_read_disabled_connector_profiles,
_connector_deleted_item,
_connector_space_response,
_ensure_campaign_file_access,
_file_connector_settings_response_watermark,
_file_connector_settings_watermark,
_visible_connector_credentials,
_visible_connector_profiles,
_visible_connector_spaces,
)
def _full_file_connector_settings_delta_response(
session: Session,
principal: ApiPrincipal,
*,
scope_type: str,
scope_id: str | None,
provider: str | None,
campaign_id: str | None,
include_disabled: bool,
include_inactive: bool,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
) -> FileConnectorSettingsDeltaResponse:
if campaign_id:
_ensure_campaign_file_access(session, principal, campaign_id)
profiles = _visible_connector_profiles(
session,
principal,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled
and _can_read_disabled_connector_profiles(principal),
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
include_effective_policy=False,
)
credentials = _visible_connector_credentials(
session,
principal,
provider=provider,
include_disabled=include_disabled,
)
spaces = _visible_connector_spaces(
session,
principal,
owner_type=owner_type,
owner_id=owner_id,
include_inactive=include_inactive,
)
return FileConnectorSettingsDeltaResponse(
profiles=[
FileConnectorProfileResponse(**profile.to_response())
for profile in profiles
],
credentials=[
FileConnectorCredentialResponse(**credential.to_response())
for credential in credentials
],
spaces=[_connector_space_response(space) for space in spaces],
policy=FileConnectorPolicyResponse(
**connector_policy_response(
session,
tenant_id=principal.tenant_id,
scope_type=scope_type,
scope_id=scope_id,
)
),
changed_sections=["profiles", "credentials", "spaces", "policy"],
deleted=[],
watermark=_file_connector_settings_watermark(
session, tenant_id=principal.tenant_id
),
has_more=False,
full=True,
)
def _changed_file_connector_setting_ids(
entries: list[ChangeSequenceEntry],
) -> tuple[set[str], set[str], set[str], bool]:
changed_profile_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION
and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
}
changed_credential_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION
and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
}
changed_space_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION
and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
}
policy_changed = any(
entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries
)
return (
changed_profile_ids,
changed_credential_ids,
changed_space_ids,
policy_changed,
)
def _file_connector_settings_changed_sections(
*,
changed_profile_ids: set[str],
changed_credential_ids: set[str],
changed_space_ids: set[str],
policy_changed: bool,
profiles: list[ConnectorProfile],
) -> list[str]:
changed_sections = []
if changed_profile_ids or any(
profile.credential_profile_id
and profile.credential_profile_id in changed_credential_ids
for profile in profiles
):
changed_sections.append("profiles")
if changed_credential_ids:
changed_sections.append("credentials")
if changed_space_ids:
changed_sections.append("spaces")
if policy_changed:
changed_sections.append("policy")
return changed_sections
def _file_connector_settings_deleted_items(
entries: list[ChangeSequenceEntry],
*,
visible_profiles: dict[str, ConnectorProfile],
visible_credentials: dict[str, ConnectorCredential],
visible_spaces: dict[str, FileConnectorSpace],
) -> list[DeltaDeletedItem]:
return [
_connector_deleted_item(entry)
for entry in entries
if (
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
and entry.resource_id not in visible_profiles
)
or (
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
and entry.resource_id not in visible_credentials
)
or (
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
and entry.resource_id not in visible_spaces
)
]
def _incremental_file_connector_settings_delta_response(
session: Session,
principal: ApiPrincipal,
*,
entries: list[ChangeSequenceEntry],
has_more: bool,
scope_type: str,
scope_id: str | None,
provider: str | None,
campaign_id: str | None,
include_disabled: bool,
include_inactive: bool,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
) -> FileConnectorSettingsDeltaResponse:
changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed = (
_changed_file_connector_setting_ids(entries)
)
profiles = _visible_connector_profiles(
session,
principal,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled
and _can_read_disabled_connector_profiles(principal),
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
include_effective_policy=False,
)
visible_profiles = {
profile.id: profile
for profile in profiles
if profile.id in changed_profile_ids
or (
profile.credential_profile_id
and profile.credential_profile_id in changed_credential_ids
)
}
credentials = _visible_connector_credentials(
session,
principal,
provider=provider,
include_disabled=include_disabled,
)
visible_credentials = {
credential.id: credential
for credential in credentials
if credential.id in changed_credential_ids
}
spaces = _visible_connector_spaces(
session,
principal,
owner_type=owner_type,
owner_id=owner_id,
include_inactive=include_inactive,
)
visible_spaces = {
space.id: space for space in spaces if space.id in changed_space_ids
}
return FileConnectorSettingsDeltaResponse(
profiles=[
FileConnectorProfileResponse(**profile.to_response())
for profile in visible_profiles.values()
],
credentials=[
FileConnectorCredentialResponse(**credential.to_response())
for credential in visible_credentials.values()
],
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
policy=FileConnectorPolicyResponse(
**connector_policy_response(
session,
tenant_id=principal.tenant_id,
scope_type=scope_type,
scope_id=scope_id,
)
)
if policy_changed
else None,
changed_sections=_file_connector_settings_changed_sections(
changed_profile_ids=changed_profile_ids,
changed_credential_ids=changed_credential_ids,
changed_space_ids=changed_space_ids,
policy_changed=policy_changed,
profiles=profiles,
),
deleted=_file_connector_settings_deleted_items(
entries,
visible_profiles=visible_profiles,
visible_credentials=visible_credentials,
visible_spaces=visible_spaces,
),
watermark=_file_connector_settings_response_watermark(
session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more
),
has_more=has_more,
full=False,
)

View File

@@ -0,0 +1,620 @@
from __future__ import annotations
from datetime import datetime
from typing import Literal
from fastapi import HTTPException, status
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal
from govoplan_core.api.v1.schemas import DeltaDeletedItem
from govoplan_core.core.change_sequence import (
ChangeSequenceEntry,
decode_sequence_watermark,
encode_sequence_watermark,
max_sequence_id,
sequence_entries_since,
sequence_watermark_is_expired,
)
from govoplan_core.core.pagination import (
KeysetCursorError,
decode_keyset_cursor,
encode_keyset_cursor,
keyset_query_fingerprint,
)
from govoplan_files.backend.change_tracking import (
FILES_ASSETS_COLLECTION,
FILES_FOLDERS_COLLECTION,
FILES_MODULE_ID,
)
from govoplan_files.backend.schemas import (
FileDeltaResponse,
)
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
from govoplan_files.backend.storage.access import user_group_ids
from govoplan_files.backend.storage.files import (
list_assets_for_user,
)
from govoplan_files.backend.storage.folders import list_folders_for_user
from govoplan_files.backend.storage.share_state import effective_file_share_clause
from govoplan_files.backend.route_support import (
_asset_list_response,
_folder_response,
_is_admin,
)
_FILES_DELTA_COLLECTIONS = (FILES_ASSETS_COLLECTION, FILES_FOLDERS_COLLECTION)
FILES_LIST_CURSOR_SCOPE = "files.list.v1"
FOLDERS_LIST_CURSOR_SCOPE = "files.folders.list.v1"
DEFAULT_FILE_LIST_PAGE_SIZE = 500
def _cursor_http_error(exc: Exception) -> HTTPException:
return HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc))
def _cursor_page_size(
scope: str, cursor: str | None, explicit_page_size: int | None
) -> int | None:
if explicit_page_size is not None:
return explicit_page_size
if not cursor:
return DEFAULT_FILE_LIST_PAGE_SIZE
try:
values = decode_keyset_cursor(scope, cursor)
except KeysetCursorError as exc:
raise _cursor_http_error(exc) from exc
raw_page_size = values.get("page_size")
if not isinstance(raw_page_size, int) or raw_page_size < 1 or raw_page_size > 1000:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
)
return raw_page_size
def _files_list_fingerprint(
principal: ApiPrincipal,
*,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
campaign_usage: Literal["linked", "unlinked"] | None,
audit_relevant: bool | None,
page_size: int,
) -> str:
return keyset_query_fingerprint(
FILES_LIST_CURSOR_SCOPE,
{
"tenant_id": principal.tenant_id,
"actor": "admin" if _is_admin(principal) else principal.user.id,
"owner_type": owner_type,
"owner_id": owner_id,
"campaign_id": campaign_id,
"path_prefix": path_prefix or "",
"campaign_usage": campaign_usage or "",
"audit_relevant": audit_relevant,
"sort": "display_path.asc,updated_at.desc,id.asc",
"page_size": page_size,
},
)
def _folders_list_fingerprint(
principal: ApiPrincipal,
*,
owner_type: Literal["user", "group"],
owner_id: str,
page_size: int,
) -> str:
return keyset_query_fingerprint(
FOLDERS_LIST_CURSOR_SCOPE,
{
"tenant_id": principal.tenant_id,
"actor": "admin" if _is_admin(principal) else principal.user.id,
"owner_type": owner_type,
"owner_id": owner_id,
"sort": "path.asc,id.asc",
"page_size": page_size,
},
)
def _file_cursor_values(
cursor: str | None, *, fingerprint: str
) -> tuple[str | None, datetime | None, str | None]:
try:
values = decode_keyset_cursor(
FILES_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
)
except KeysetCursorError as exc:
raise _cursor_http_error(exc) from exc
if values is None:
return None, None, None
display_path = values.get("display_path")
updated_at = values.get("updated_at")
asset_id = values.get("id")
if (
not isinstance(display_path, str)
or not isinstance(updated_at, str)
or not isinstance(asset_id, str)
):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
)
try:
parsed_updated_at = datetime.fromisoformat(updated_at)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
) from exc
return display_path, parsed_updated_at, asset_id
def _folder_cursor_values(
cursor: str | None, *, fingerprint: str
) -> tuple[str | None, str | None]:
try:
values = decode_keyset_cursor(
FOLDERS_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
)
except KeysetCursorError as exc:
raise _cursor_http_error(exc) from exc
if values is None:
return None, None
folder_path = values.get("path")
folder_id = values.get("id")
if not isinstance(folder_path, str) or not isinstance(folder_id, str):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
)
return folder_path, folder_id
def _next_file_list_cursor(
principal: ApiPrincipal,
assets: list[FileAsset],
*,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
campaign_usage: Literal["linked", "unlinked"] | None,
audit_relevant: bool | None,
page_size: int,
has_more: bool,
) -> str | None:
if not has_more or not assets:
return None
last = assets[-1]
return encode_keyset_cursor(
FILES_LIST_CURSOR_SCOPE,
fingerprint=_files_list_fingerprint(
principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
page_size=page_size,
),
values={
"display_path": last.display_path,
"updated_at": last.updated_at,
"id": last.id,
"page_size": page_size,
},
)
def _next_folder_list_cursor(
principal: ApiPrincipal,
folders: list[FileFolder],
*,
owner_type: Literal["user", "group"],
owner_id: str,
page_size: int,
has_more: bool,
) -> str | None:
if not has_more or not folders:
return None
last = folders[-1]
return encode_keyset_cursor(
FOLDERS_LIST_CURSOR_SCOPE,
fingerprint=_folders_list_fingerprint(
principal, owner_type=owner_type, owner_id=owner_id, page_size=page_size
),
values={"path": last.path, "id": last.id, "page_size": page_size},
)
def _files_delta_watermark(session: Session, tenant_id: str) -> str:
return encode_sequence_watermark(
max_sequence_id(
session,
tenant_id=tenant_id,
module_id=FILES_MODULE_ID,
collections=_FILES_DELTA_COLLECTIONS,
)
)
def _full_file_delta_response(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
) -> FileDeltaResponse:
assets = list_assets_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
is_admin=_is_admin(principal),
)
folders = _visible_folders_for_delta(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=path_prefix,
)
return FileDeltaResponse(
files=_asset_list_response(session, assets, include_shares=True),
folders=[_folder_response(folder) for folder in folders],
deleted=[],
watermark=_files_delta_watermark(session, principal.tenant_id),
has_more=False,
full=True,
)
def _visible_folders_for_delta(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
path_prefix: str | None,
) -> list[FileFolder]:
if not owner_type or not owner_id:
return []
folders = list_folders_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
is_admin=_is_admin(principal),
)
if not path_prefix:
return folders
normalized = path_prefix.strip().strip("/")
if not normalized:
return folders
return [
folder
for folder in folders
if folder.path == normalized or folder.path.startswith(f"{normalized}/")
]
def _entry_path_matches(
entry_payload: dict[str, object], path_prefix: str | None
) -> bool:
if not path_prefix:
return True
normalized = path_prefix.strip().strip("/")
if not normalized:
return True
for key in ("path", "previous_path"):
value = entry_payload.get(key)
if isinstance(value, str) and (
value == normalized or value.startswith(f"{normalized}/")
):
return True
return False
def _entry_owner_matches(
entry_payload: dict[str, object],
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
) -> bool:
if not owner_type or not owner_id:
return True
return (
entry_payload.get("owner_type") == owner_type
and entry_payload.get("owner_id") == owner_id
)
def _entry_campaign_matches(session: Session, entry, campaign_id: str | None) -> bool:
if not campaign_id:
return True
payload = entry.payload or {}
if (
payload.get("share_target_type") == "campaign"
and payload.get("share_target_id") == campaign_id
):
return True
if entry.resource_type != "file":
return False
return (
session.query(FileShare)
.filter(
FileShare.tenant_id == entry.tenant_id,
FileShare.file_asset_id == entry.resource_id,
FileShare.target_type == "campaign",
FileShare.target_id == campaign_id,
effective_file_share_clause(),
)
.first()
is not None
)
def _principal_group_ids_for_delta(
session: Session, principal: ApiPrincipal
) -> set[str]:
cache = session.info.setdefault("files_delta_group_ids", {})
key = (principal.tenant_id, principal.user.id)
if key not in cache:
cache[key] = set(
user_group_ids(
session, tenant_id=principal.tenant_id, user_id=principal.user.id
)
)
return cache[key]
def _entry_subject_matches_principal(
session: Session, principal: ApiPrincipal, entry_payload: dict[str, object]
) -> bool:
if _is_admin(principal):
return True
owner_type = entry_payload.get("owner_type")
owner_id = entry_payload.get("owner_id")
if owner_type == "user" and owner_id == principal.user.id:
return True
if owner_type == "group" and isinstance(owner_id, str):
if owner_id in _principal_group_ids_for_delta(session, principal):
return True
share_target_type = entry_payload.get("share_target_type")
share_target_id = entry_payload.get("share_target_id")
if share_target_type == "user" and share_target_id == principal.user.id:
return True
if share_target_type == "tenant" and share_target_id == principal.tenant_id:
return True
if share_target_type == "group" and isinstance(share_target_id, str):
if share_target_id in _principal_group_ids_for_delta(session, principal):
return True
return False
def _entry_matches_delta_scope(
session: Session,
principal: ApiPrincipal,
entry,
*,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
) -> bool:
payload = entry.payload or {}
if (
not owner_type
and not campaign_id
and not _entry_subject_matches_principal(session, principal, payload)
):
return False
return (
_entry_owner_matches(payload, owner_type, owner_id)
and _entry_path_matches(payload, path_prefix)
and _entry_campaign_matches(session, entry, campaign_id)
)
def _decode_files_delta_watermark(since: str) -> int:
try:
return decode_sequence_watermark(since)
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)
) from exc
def _changed_delta_resource_ids(
entries: list[ChangeSequenceEntry],
) -> tuple[list[str], list[str]]:
file_ids = list(
dict.fromkeys(
entry.resource_id for entry in entries if entry.resource_type == "file"
)
)
folder_ids = list(
dict.fromkeys(
entry.resource_id for entry in entries if entry.resource_type == "folder"
)
)
return file_ids, folder_ids
def _visible_assets_for_delta(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
changed_file_ids: list[str],
) -> dict[str, FileAsset]:
return {
asset.id: asset
for asset in list_assets_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
is_admin=_is_admin(principal),
)
if asset.id in changed_file_ids
}
def _changed_visible_folders_for_delta(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
path_prefix: str | None,
changed_folder_ids: list[str],
) -> dict[str, FileFolder]:
return {
folder.id: folder
for folder in _visible_folders_for_delta(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=path_prefix,
)
if folder.id in changed_folder_ids
}
def _deleted_delta_items(
session: Session,
*,
principal: ApiPrincipal,
entries: list[ChangeSequenceEntry],
visible_assets: dict[str, FileAsset],
visible_folders: dict[str, FileFolder],
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
) -> list[DeltaDeletedItem]:
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
for entry in entries:
if entry.resource_type == "file" and entry.resource_id in visible_assets:
continue
if entry.resource_type == "folder" and entry.resource_id in visible_folders:
continue
if not _entry_matches_delta_scope(
session,
principal,
entry,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
):
continue
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
id=entry.resource_id,
resource_type=entry.resource_type,
revision=encode_sequence_watermark(entry.id),
deleted_at=entry.created_at if entry.operation == "deleted" else None,
)
return list(deleted.values())
def _files_delta_response(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
since: str,
limit: int,
) -> FileDeltaResponse:
since_sequence = _decode_files_delta_watermark(since)
if sequence_watermark_is_expired(
session,
since=since_sequence,
tenant_id=principal.tenant_id,
module_id=FILES_MODULE_ID,
collections=_FILES_DELTA_COLLECTIONS,
):
return _full_file_delta_response(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
)
entries_plus_one = sequence_entries_since(
session,
since=since_sequence,
tenant_id=principal.tenant_id,
module_id=FILES_MODULE_ID,
collections=_FILES_DELTA_COLLECTIONS,
limit=limit + 1,
)
has_more = len(entries_plus_one) > limit
entries = entries_plus_one[:limit]
changed_file_ids, changed_folder_ids = _changed_delta_resource_ids(entries)
visible_assets = _visible_assets_for_delta(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
changed_file_ids=changed_file_ids,
)
visible_folders = _changed_visible_folders_for_delta(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=path_prefix,
changed_folder_ids=changed_folder_ids,
)
deleted = _deleted_delta_items(
session,
principal=principal,
entries=entries,
visible_assets=visible_assets,
visible_folders=visible_folders,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
)
watermark = (
encode_sequence_watermark(entries[-1].id)
if has_more and entries
else _files_delta_watermark(session, principal.tenant_id)
)
return FileDeltaResponse(
files=_asset_list_response(
session, list(visible_assets.values()), include_shares=True
),
folders=[_folder_response(folder) for folder in visible_folders.values()],
deleted=deleted,
watermark=watermark,
has_more=has_more,
full=False,
)

View File

@@ -17,6 +17,7 @@ from govoplan_files.backend.storage.paths import filename_from_path, normalize_f
_ZIP_READ_CHUNK_SIZE = 1024 * 1024 _ZIP_READ_CHUNK_SIZE = 1024 * 1024
ZIP_UPLOAD_MAX_FILES = 1000
def _read_zip_member( def _read_zip_member(
@@ -76,7 +77,7 @@ def extract_zip_upload(
conflict_resolutions: Iterable[FileConflictResolution] | None = None, conflict_resolutions: Iterable[FileConflictResolution] | None = None,
metadata: dict[str, Any] | None = None, metadata: dict[str, Any] | None = None,
is_admin: bool = False, is_admin: bool = False,
max_files: int = 1000, max_files: int = ZIP_UPLOAD_MAX_FILES,
max_file_bytes: int = 50 * 1024 * 1024, max_file_bytes: int = 50 * 1024 * 1024,
max_total_bytes: int = 250 * 1024 * 1024, max_total_bytes: int = 250 * 1024 * 1024,
) -> list[UploadedStoredFile]: ) -> list[UploadedStoredFile]:

View File

@@ -17,6 +17,22 @@ class StorageBackendError(RuntimeError):
pass pass
class StorageObjectMissing(StorageBackendError):
pass
@dataclass(frozen=True, slots=True)
class StorageObjectInfo:
key: str
size_bytes: int
@dataclass(frozen=True, slots=True)
class StorageObjectPage:
objects: tuple[StorageObjectInfo, ...]
next_cursor: str | None = None
class StorageBackend(Protocol): class StorageBackend(Protocol):
name: str name: str
@@ -25,6 +41,14 @@ class StorageBackend(Protocol):
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]: ... def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]: ...
def delete(self, key: str) -> None: ... def delete(self, key: str) -> None: ...
def exists(self, key: str) -> bool: ... def exists(self, key: str) -> bool: ...
def stat(self, key: str) -> StorageObjectInfo: ...
def list_objects(
self,
*,
prefix: str,
after: str | None = None,
limit: int = 500,
) -> StorageObjectPage: ...
@dataclass(slots=True) @dataclass(slots=True)
@@ -55,7 +79,7 @@ class LocalFilesystemStorageBackend:
candidate = self._path_for_root(root, key) candidate = self._path_for_root(root, key)
if candidate.exists() and candidate.is_file(): if candidate.exists() and candidate.is_file():
return candidate return candidate
raise StorageBackendError("Stored object does not exist") raise StorageObjectMissing("Stored object does not exist")
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None: def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
path = self._path(key) path = self._path(key)
@@ -86,6 +110,33 @@ class LocalFilesystemStorageBackend:
return False return False
return True return True
def stat(self, key: str) -> StorageObjectInfo:
path = self._readable_path(key)
return StorageObjectInfo(key=key, size_bytes=path.stat().st_size)
def list_objects(
self,
*,
prefix: str,
after: str | None = None,
limit: int = 500,
) -> StorageObjectPage:
normalized_limit = max(1, min(int(limit), 5000))
candidates: list[StorageObjectInfo] = []
for path in _iter_local_files(self.root):
key = path.relative_to(self.root).as_posix()
if not key.startswith(prefix) or (after is not None and key <= after):
continue
candidates.append(StorageObjectInfo(key=key, size_bytes=path.stat().st_size))
if len(candidates) > normalized_limit:
break
has_more = len(candidates) > normalized_limit
page = tuple(candidates[:normalized_limit])
return StorageObjectPage(
objects=page,
next_cursor=page[-1].key if has_more and page else None,
)
@dataclass(slots=True) @dataclass(slots=True)
class S3StorageBackend: class S3StorageBackend:
@@ -175,6 +226,52 @@ class S3StorageBackend:
except Exception: except Exception:
return False return False
def stat(self, key: str) -> StorageObjectInfo:
try:
response = self.client.head_object(Bucket=self.bucket, Key=key)
except Exception as exc: # pragma: no cover - depends on S3 backend
if _s3_missing_error(exc):
raise StorageObjectMissing("Stored object does not exist") from exc
raise StorageBackendError(str(exc)) from exc
try:
size = int(response.get("ContentLength"))
except (AttributeError, TypeError, ValueError) as exc:
raise StorageBackendError("S3 object metadata did not include a valid size") from exc
return StorageObjectInfo(key=key, size_bytes=size)
def list_objects(
self,
*,
prefix: str,
after: str | None = None,
limit: int = 500,
) -> StorageObjectPage:
normalized_limit = max(1, min(int(limit), 1000))
kwargs: dict[str, object] = {
"Bucket": self.bucket,
"Prefix": prefix,
"MaxKeys": normalized_limit,
}
if after:
kwargs["StartAfter"] = after
try:
response = self.client.list_objects_v2(**kwargs)
except Exception as exc: # pragma: no cover - depends on S3 backend
raise StorageBackendError(str(exc)) from exc
objects = tuple(
StorageObjectInfo(
key=str(item["Key"]),
size_bytes=int(item.get("Size") or 0),
)
for item in response.get("Contents", ())
if isinstance(item, dict) and item.get("Key")
)
has_more = bool(response.get("IsTruncated"))
return StorageObjectPage(
objects=objects,
next_cursor=objects[-1].key if has_more and objects else None,
)
def _reject_declared_object_size(obj: object, *, max_bytes: int) -> None: def _reject_declared_object_size(obj: object, *, max_bytes: int) -> None:
if not isinstance(obj, dict): if not isinstance(obj, dict):
@@ -187,6 +284,25 @@ def _reject_declared_object_size(obj: object, *, max_bytes: int) -> None:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes") raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
def _iter_local_files(root: Path):
for entry in sorted(root.iterdir(), key=lambda item: item.name):
if entry.is_dir():
yield from _iter_local_files(entry)
elif entry.is_file():
yield entry
def _s3_missing_error(exc: Exception) -> bool:
response = getattr(exc, "response", None)
if not isinstance(response, dict):
return False
error = response.get("Error")
metadata = response.get("ResponseMetadata")
code = str(error.get("Code") if isinstance(error, dict) else "")
status_code = metadata.get("HTTPStatusCode") if isinstance(metadata, dict) else None
return code in {"404", "NoSuchKey", "NotFound"} or status_code == 404
def _fallback_roots() -> tuple[Path, ...]: def _fallback_roots() -> tuple[Path, ...]:
raw = getattr(settings, "file_storage_local_fallback_roots", "") or "" raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip()) return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())

View File

@@ -13,12 +13,17 @@ from typing import Any, Iterator
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
from govoplan_files.backend.storage.backends import StorageBackend, StorageBackendError, get_storage_backend from govoplan_files.backend.storage.backends import StorageBackend, get_storage_backend
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
from govoplan_files.backend.storage.access import ensure_owner_access from govoplan_files.backend.storage.access import ensure_owner_access
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
from govoplan_files.backend.storage.share_state import effective_file_share_clause
from govoplan_files.backend.storage.integrity import (
ensure_blob_is_readable,
read_verified_blob_bytes,
)
MANAGED_SOURCE_PREFIX = "managed:" MANAGED_SOURCE_PREFIX = "managed:"
@@ -165,7 +170,7 @@ def _campaign_linked_asset_ids(
FileShare.file_asset_id.in_(chunk), FileShare.file_asset_id.in_(chunk),
FileShare.target_type == "campaign", FileShare.target_type == "campaign",
FileShare.target_id == campaign_id, FileShare.target_id == campaign_id,
FileShare.revoked_at.is_(None), effective_file_share_clause(),
) )
.all() .all()
) )
@@ -301,11 +306,9 @@ def _materialize_managed_asset(
target = _safe_local_target(local_root, relative_path) target = _safe_local_target(local_root, relative_path)
target.parent.mkdir(parents=True, exist_ok=True) target.parent.mkdir(parents=True, exist_ok=True)
version, blob = version_blobs[asset.id] version, blob = version_blobs[asset.id]
ensure_blob_is_readable(blob)
if include_bytes: if include_bytes:
try: data = read_verified_blob_bytes(blob, backend=backend) if backend else b""
data = backend.get_bytes(blob.storage_key) if backend else b""
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
target.write_bytes(data) target.write_bytes(data)
else: else:
target.touch() target.touch()

View File

@@ -5,8 +5,17 @@ from dataclasses import dataclass
from typing import Any from typing import Any
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from sqlalchemy import inspect
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
from govoplan_core.security.credential_envelopes import (
CredentialAccessContext,
CredentialEnvelope,
CredentialEnvelopeError,
ResolvedCredentialEnvelope,
list_credential_envelopes,
resolve_credential_envelope,
)
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential from govoplan_files.backend.db.models import FileConnectorCredential
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
@@ -14,6 +23,8 @@ from govoplan_files.backend.storage.connector_deployment import reject_api_contr
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
CORE_CREDENTIAL_ENVELOPE_PREFIX = "credential-envelope:"
@dataclass(frozen=True, slots=True) @dataclass(frozen=True, slots=True)
class ConnectorCredential: class ConnectorCredential:
@@ -33,6 +44,7 @@ class ConnectorCredential:
policy_sources: tuple[ConnectorPolicySource, ...] = () policy_sources: tuple[ConnectorPolicySource, ...] = ()
metadata: Mapping[str, Any] | None = None metadata: Mapping[str, Any] | None = None
source_kind: str = "database" source_kind: str = "database"
has_secret: bool = False
@property @property
def source_path(self) -> str: def source_path(self) -> str:
@@ -58,7 +70,7 @@ class ConnectorCredential:
return True return True
# Environment references are deliberately unavailable to API-managed # Environment references are deliberately unavailable to API-managed
# credential rows. Legacy rows remain visible but fail closed. # credential rows. Legacy rows remain visible but fail closed.
return bool(self.secret_ref or self.password_value or self.token_value) return bool(self.has_secret or self.secret_ref or self.password_value or self.token_value)
def to_response(self) -> dict[str, Any]: def to_response(self) -> dict[str, Any]:
return { return {
@@ -85,7 +97,159 @@ def list_database_connector_credentials(
tenant_id: str, tenant_id: str,
include_disabled: bool = False, include_disabled: bool = False,
) -> list[ConnectorCredential]: ) -> list[ConnectorCredential]:
return [connector_credential_from_row(row) for row in list_connector_credential_rows(session, tenant_id=tenant_id, include_disabled=include_disabled)] local = [
connector_credential_from_row(row)
for row in list_connector_credential_rows(
session,
tenant_id=tenant_id,
include_disabled=include_disabled,
)
]
return [
*local,
*list_reusable_connector_credentials(
session,
tenant_id=tenant_id,
include_disabled=include_disabled,
),
]
def reusable_credential_reference(credential_id: str) -> str:
return f"{CORE_CREDENTIAL_ENVELOPE_PREFIX}{credential_id}"
def reusable_credential_id(credential_ref: str | None) -> str | None:
if not credential_ref or not credential_ref.startswith(CORE_CREDENTIAL_ENVELOPE_PREFIX):
return None
value = credential_ref.removeprefix(CORE_CREDENTIAL_ENVELOPE_PREFIX).strip()
return value or None
def file_credential_context(
*,
tenant_id: str,
profile_id: str | None = None,
scope_type: str = "tenant",
scope_id: str | None = None,
administrative: bool = False,
) -> CredentialAccessContext:
return CredentialAccessContext(
tenant_id=tenant_id,
user_id=scope_id if scope_type == "user" else None,
group_ids=frozenset({scope_id}) if scope_type == "group" and scope_id else frozenset(),
target_scope_type=scope_type,
target_scope_id=scope_id or tenant_id,
module_id="files",
server_ref=f"files:{profile_id}" if profile_id else None,
administrative=administrative,
)
def list_reusable_connector_credentials(
session: Session,
*,
tenant_id: str,
include_disabled: bool = False,
) -> list[ConnectorCredential]:
if not inspect(session.get_bind()).has_table(CredentialEnvelope.__tablename__):
return []
context = file_credential_context(tenant_id=tenant_id, administrative=True)
return [
connector_credential_from_envelope(row)
for row in list_credential_envelopes(
session,
context=context,
include_inactive=include_disabled,
)
]
def resolve_reusable_connector_credential(
session: Session,
*,
tenant_id: str,
credential_ref: str,
profile_id: str,
scope_type: str,
scope_id: str | None,
) -> ConnectorCredential:
credential_id = reusable_credential_id(credential_ref)
if credential_id is None:
raise FileStorageError("Reusable credential reference is invalid")
try:
resolved = resolve_credential_envelope(
session,
credential_id=credential_id,
context=file_credential_context(
tenant_id=tenant_id,
profile_id=profile_id,
scope_type=scope_type,
scope_id=scope_id,
),
)
except CredentialEnvelopeError as exc:
raise FileStorageError("Reusable credential is unavailable to this file connection") from exc
return connector_credential_from_resolved_envelope(
resolved,
scope_type=scope_type,
scope_id=scope_id,
)
def connector_credential_from_envelope(row: CredentialEnvelope) -> ConnectorCredential:
return ConnectorCredential(
id=reusable_credential_reference(row.id),
label=row.name,
scope_type=row.scope_type,
scope_id=row.scope_id,
enabled=row.is_active,
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_keys),
username=_clean_public_value(row.public_data, "username"),
source_kind="credential_envelope",
has_secret=bool(row.secret_data_encrypted),
metadata={
"credential_envelope_id": row.id,
"credential_kind": row.credential_kind,
"allowed_modules": list(row.allowed_modules or []),
"allowed_server_refs": list(row.allowed_server_refs or []),
"inherit_to_lower_scopes": bool(row.inherit_to_lower_scopes),
"revision": row.revision,
},
)
def connector_credential_from_resolved_envelope(
row: ResolvedCredentialEnvelope,
*,
scope_type: str,
scope_id: str | None,
) -> ConnectorCredential:
return ConnectorCredential(
id=reusable_credential_reference(row.id),
label=row.name,
scope_type=scope_type,
scope_id=scope_id,
enabled=True,
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_data),
username=_clean_public_value(row.public_data, "username"),
password_value=_first_secret(row.secret_data, "password", "secret"),
token_value=_first_secret(
row.secret_data,
"access_token",
"bearer_token",
"token",
"api_key",
),
source_kind="credential_envelope",
has_secret=bool(row.secret_data),
metadata={
"credential_envelope_id": row.id,
"credential_kind": row.credential_kind,
"inherit_to_lower_scopes": True,
"revision": row.revision,
},
)
def list_connector_credential_rows( def list_connector_credential_rows(
@@ -348,6 +512,27 @@ def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
} }
def _envelope_credential_mode(kind: str, secret_values: Mapping[str, Any] | list[str]) -> str:
keys = {str(key) for key in secret_values}
if kind in {"token", "oauth2", "api_key"} or keys.intersection(
{"access_token", "bearer_token", "token", "api_key"}
):
return "token"
return "basic"
def _clean_public_value(values: Mapping[str, Any] | None, key: str) -> str | None:
return _clean((values or {}).get(key))
def _first_secret(values: Mapping[str, Any], *keys: str) -> str | None:
for key in keys:
value = _clean(values.get(key))
if value is not None:
return value
return None
def _clean(value: object) -> str | None: def _clean(value: object) -> str | None:
if value is None: if value is None:
return None return None

View File

@@ -246,11 +246,11 @@ def _applied_fields(policy: Mapping[str, Any]) -> tuple[str, ...]:
def _matches_field(request: ConnectorAccessRequest, field: str, patterns: list[str]) -> bool: def _matches_field(request: ConnectorAccessRequest, field: str, patterns: list[str]) -> bool:
if field == "connectors": if field == "connectors":
return _matches_exact(request.connector_id, patterns) return _matches_reference(request.connector_id, patterns)
if field == "credentials": if field == "credentials":
return _matches_exact(request.credential_id, patterns) return _matches_reference(request.credential_id, patterns)
if field == "providers": if field == "providers":
return _matches_exact(request.provider, patterns) return _matches_reference(request.provider, patterns)
if field == "external_ids": if field == "external_ids":
return _matches_glob(request.external_id, patterns) return _matches_glob(request.external_id, patterns)
if field == "external_paths": if field == "external_paths":
@@ -260,11 +260,11 @@ def _matches_field(request: ConnectorAccessRequest, field: str, patterns: list[s
return False return False
def _matches_exact(value: str | None, patterns: list[str]) -> bool: def _matches_reference(value: str | None, patterns: list[str]) -> bool:
if value is None: if value is None:
return False return False
clean = value.casefold() clean = value.casefold()
return any(pattern == "*" or clean == pattern.casefold() for pattern in patterns) return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
def _matches_glob(value: str | None, patterns: list[str]) -> bool: def _matches_glob(value: str | None, patterns: list[str]) -> bool:

View File

@@ -1,6 +1,6 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Mapping from collections.abc import Callable, Mapping
from typing import Any from typing import Any
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -10,7 +10,13 @@ from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
from govoplan_files.backend.storage.connector_credential_store import connector_credential_from_row, credential_rows_by_id from govoplan_files.backend.storage.connector_credential_store import (
ConnectorCredential,
connector_credential_from_row,
credential_rows_by_id,
resolve_reusable_connector_credential,
reusable_credential_id,
)
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
@@ -20,7 +26,26 @@ def list_database_connector_profiles(
*, *,
tenant_id: str, tenant_id: str,
include_disabled: bool = False, include_disabled: bool = False,
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
) -> list[ConnectorProfile]: ) -> list[ConnectorProfile]:
profiles, _profile_ids = select_database_connector_profiles(
session,
tenant_id=tenant_id,
include_disabled=include_disabled,
row_visible=row_visible,
)
return profiles
def select_database_connector_profiles(
session: Session,
*,
tenant_id: str,
include_disabled: bool = False,
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
) -> tuple[list[ConnectorProfile], set[str]]:
"""Return visible profiles and every database id that shadows settings."""
query = session.query(FileConnectorProfile).filter( query = session.query(FileConnectorProfile).filter(
(FileConnectorProfile.scope_type == "system") (FileConnectorProfile.scope_type == "system")
| (FileConnectorProfile.tenant_id == tenant_id) | (FileConnectorProfile.tenant_id == tenant_id)
@@ -28,9 +53,33 @@ def list_database_connector_profiles(
if not include_disabled: if not include_disabled:
query = query.filter(FileConnectorProfile.enabled.is_(True)) query = query.filter(FileConnectorProfile.enabled.is_(True))
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all() rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
credential_ids = {_clean(row.credential_profile_id) for row in rows if _clean(row.credential_profile_id)} profile_ids = {row.id for row in rows}
if row_visible is not None:
rows = [row for row in rows if row_visible(row)]
credential_ids = {
_clean(row.credential_profile_id)
for row in rows
if _clean(row.credential_profile_id) and not reusable_credential_id(row.credential_profile_id)
}
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled) credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
return [connector_profile_from_row(row, credential_row=credentials.get(row.credential_profile_id or "")) for row in rows] return (
[
connector_profile_from_row(
row,
credential_row=(
_resolve_profile_reusable_credential(
session,
tenant_id=tenant_id,
row=row,
)
if reusable_credential_id(row.credential_profile_id)
else credentials.get(row.credential_profile_id or "")
),
)
for row in rows
],
profile_ids,
)
def list_connector_profile_rows( def list_connector_profile_rows(
@@ -48,7 +97,10 @@ def list_connector_profile_rows(
return query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all() return query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
def connector_profile_from_row(row: FileConnectorProfile, credential_row: FileConnectorCredential | None = None) -> ConnectorProfile: def connector_profile_from_row(
row: FileConnectorProfile,
credential_row: FileConnectorCredential | ConnectorCredential | None = None,
) -> ConnectorProfile:
policy_sources = [] policy_sources = []
if row.policy: if row.policy:
policy_sources = connector_policy_sources_from_payload({ policy_sources = connector_policy_sources_from_payload({
@@ -57,9 +109,18 @@ def connector_profile_from_row(row: FileConnectorProfile, credential_row: FileCo
"label": row.label, "label": row.label,
"policy": row.policy, "policy": row.policy,
}) })
credential = connector_credential_from_row(credential_row) if credential_row is not None else None credential = (
credential_row
if isinstance(credential_row, ConnectorCredential)
else connector_credential_from_row(credential_row)
if credential_row is not None
else None
)
if credential: if credential:
policy_sources.extend(credential.policy_sources) policy_sources.extend(credential.policy_sources)
metadata = dict(row.metadata_ or {})
if reusable_credential_id(row.credential_profile_id) and credential is None:
metadata["credential_unavailable"] = True
return ConnectorProfile( return ConnectorProfile(
id=row.id, id=row.id,
label=row.label, label=row.label,
@@ -80,11 +141,30 @@ def connector_profile_from_row(row: FileConnectorProfile, credential_row: FileCo
token_value=credential.token_value if credential else decrypt_secret(row.token_encrypted), token_value=credential.token_value if credential else decrypt_secret(row.token_encrypted),
capabilities=tuple(_string_list(row.capabilities)), capabilities=tuple(_string_list(row.capabilities)),
policy_sources=tuple(policy_sources), policy_sources=tuple(policy_sources),
metadata=dict(row.metadata_ or {}), metadata=metadata,
source_kind="database", source_kind="database",
) )
def _resolve_profile_reusable_credential(
session: Session,
*,
tenant_id: str,
row: FileConnectorProfile,
) -> ConnectorCredential | None:
try:
return resolve_reusable_connector_credential(
session,
tenant_id=tenant_id,
credential_ref=row.credential_profile_id or "",
profile_id=row.id,
scope_type=row.scope_type,
scope_id=row.scope_id,
)
except FileStorageError:
return None
def get_connector_profile_row( def get_connector_profile_row(
session: Session, session: Session,
*, *,
@@ -188,18 +268,80 @@ def update_connector_profile_row(
clear_password: bool = False, clear_password: bool = False,
clear_token: bool = False, clear_token: bool = False,
) -> FileConnectorProfile: ) -> FileConnectorProfile:
_validate_profile_update_references(
row,
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
metadata=metadata,
)
_update_profile_connection_fields(
row,
label=label,
provider=provider,
endpoint_url=endpoint_url,
base_path=base_path,
enabled=enabled,
credential_profile_id=credential_profile_id,
credential_mode=credential_mode,
)
_update_profile_credential_fields(
row,
username=username,
password=password,
token=token,
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
clear_password=clear_password,
clear_token=clear_token,
)
_update_profile_governance_fields(
row,
capabilities=capabilities,
policy=policy,
metadata=metadata,
)
row.updated_by_user_id = user_id
session.add(row)
session.flush()
return row
def _validate_profile_update_references(
row: FileConnectorProfile,
*,
password_env: str | None,
token_env: str | None,
secret_ref: str | None,
metadata: Mapping[str, Any] | None,
) -> None:
reject_api_controlled_deployment_references( reject_api_controlled_deployment_references(
password_env=password_env, password_env=password_env,
token_env=token_env, token_env=token_env,
secret_ref=secret_ref, secret_ref=secret_ref,
metadata=metadata, metadata=metadata,
) )
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref): if secret_ref is None or _clean(secret_ref) == _clean(row.secret_ref):
if _clean(row.secret_ref): return
raise FileStorageError( if _clean(row.secret_ref):
"An existing external secret reference cannot be replaced or cleared until Files can prove " raise FileStorageError(
"provider ownership and confirm provider-side deletion" "An existing external secret reference cannot be replaced or cleared until Files can prove "
) "provider ownership and confirm provider-side deletion"
)
def _update_profile_connection_fields(
row: FileConnectorProfile,
*,
label: str | None,
provider: str | None,
endpoint_url: str | None,
base_path: str | None,
enabled: bool | None,
credential_profile_id: str | None,
credential_mode: str | None,
) -> None:
if label is not None: if label is not None:
row.label = _normalize_label(label) row.label = _normalize_label(label)
if provider is not None: if provider is not None:
@@ -214,6 +356,20 @@ def update_connector_profile_row(
row.credential_profile_id = _clean(credential_profile_id) row.credential_profile_id = _clean(credential_profile_id)
if credential_mode is not None: if credential_mode is not None:
row.credential_mode = _normalize_credential_mode(credential_mode) row.credential_mode = _normalize_credential_mode(credential_mode)
def _update_profile_credential_fields(
row: FileConnectorProfile,
*,
username: str | None,
password: str | None,
token: str | None,
password_env: str | None,
token_env: str | None,
secret_ref: str | None,
clear_password: bool,
clear_token: bool,
) -> None:
if username is not None: if username is not None:
row.username = _clean(username) row.username = _clean(username)
if password is not None: if password is not None:
@@ -230,16 +386,21 @@ def update_connector_profile_row(
row.token_env = _clean(token_env) row.token_env = _clean(token_env)
if secret_ref is not None: if secret_ref is not None:
row.secret_ref = _clean(secret_ref) row.secret_ref = _clean(secret_ref)
def _update_profile_governance_fields(
row: FileConnectorProfile,
*,
capabilities: list[str] | None,
policy: Mapping[str, Any] | None,
metadata: Mapping[str, Any] | None,
) -> None:
if capabilities is not None: if capabilities is not None:
row.capabilities = _string_list(capabilities) row.capabilities = _string_list(capabilities)
if policy is not None: if policy is not None:
row.policy = dict(policy) row.policy = dict(policy)
if metadata is not None: if metadata is not None:
row.metadata_ = dict(metadata) row.metadata_ = dict(metadata)
row.updated_by_user_id = user_id
session.add(row)
session.flush()
return row
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]: def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:

View File

@@ -0,0 +1,218 @@
from __future__ import annotations
from collections.abc import Callable, Iterable
from dataclasses import replace
from sqlalchemy.orm import Session
from govoplan_files.backend.storage.connector_deployment import (
connector_effective_endpoint_url,
)
from govoplan_files.backend.storage.connector_profile_store import (
select_database_connector_profiles,
)
from govoplan_files.backend.storage.connector_profiles import (
ConnectorProfile,
connector_profiles_from_settings,
)
from govoplan_files.backend.storage.connector_policy import (
ConnectorAccessRequest,
connector_policy_decision,
)
from govoplan_files.backend.storage.connector_policy_store import (
effective_connector_policy_sources,
)
from govoplan_files.backend.storage.connector_providers import (
connector_provider_descriptors,
)
CampaignVisibility = Callable[[str], bool]
def visible_connector_profiles_for_actor(
session: Session,
*,
tenant_id: str,
user_id: str,
group_ids: Iterable[str],
settings: object | None,
provider: str | None = None,
campaign_id: str | None = None,
campaign_visible: CampaignVisibility | None = None,
include_disabled: bool = False,
include_admin_scopes: bool = False,
include_effective_policy: bool = True,
) -> list[ConnectorProfile]:
"""Return the same actor-filtered connector set used by API and docs surfaces."""
provider_norm = provider.strip().casefold() if provider else None
actor_group_ids = {str(group_id) for group_id in group_ids if str(group_id)}
database_profiles, database_profile_ids = select_database_connector_profiles(
session,
tenant_id=tenant_id,
include_disabled=include_disabled,
row_visible=lambda row: (
include_admin_scopes
and row.scope_type in {"user", "group", "campaign"}
) or _scope_visible_to_actor(
scope_type=row.scope_type,
scope_id=row.scope_id,
tenant_id=tenant_id,
user_id=user_id,
group_ids=actor_group_ids,
campaign_id=campaign_id,
campaign_visible=campaign_visible,
),
)
configured_profiles = connector_profiles_from_settings(settings)
profiles_by_id: dict[str, ConnectorProfile] = {}
for profile in database_profiles:
if profile.id not in profiles_by_id:
profiles_by_id[profile.id] = profile
for profile in configured_profiles:
if profile.id not in database_profile_ids and profile.id not in profiles_by_id:
profiles_by_id[profile.id] = profile
visible: list[ConnectorProfile] = []
for profile in profiles_by_id.values():
if not (profile.enabled or include_disabled):
continue
if provider_norm is not None and profile.provider != provider_norm:
continue
admin_scope_visible = include_admin_scopes and profile.scope_type in {
"user",
"group",
"campaign",
}
if not admin_scope_visible and not _profile_visible_to_actor(
profile,
tenant_id=tenant_id,
user_id=user_id,
group_ids=actor_group_ids,
campaign_id=campaign_id,
campaign_visible=campaign_visible,
):
continue
visible.append(
_with_effective_connector_policy(
session, tenant_id=tenant_id, profile=profile
)
if include_effective_policy
else profile
)
return visible
def connector_profile_usable_for_import(profile: ConnectorProfile) -> bool:
"""Whether a visible profile can safely offer the current live browse/import task."""
effective_endpoint_url = connector_effective_endpoint_url(
provider=profile.provider,
endpoint_url=profile.endpoint_url,
metadata=profile.metadata,
)
if (
not profile.enabled
or not effective_endpoint_url
or not profile.credentials_configured
or bool(profile.secret_ref)
):
return False
descriptor = next(
(
item
for item in connector_provider_descriptors()
if item.provider == profile.provider
),
None,
)
if descriptor is None:
return False
if not (
descriptor.implemented
and descriptor.installed
and descriptor.browse_supported
and descriptor.import_supported
):
return False
# These SDK paths intentionally fail closed until every connection peer and
# SDK-managed redirect/referral can be pinned and revalidated.
if profile.provider in {"s3", "smb"}:
return False
# Prove that the initial root browse performed by the current Files UI is
# policy-allowed. A later selected remote path/item is checked again.
return connector_policy_decision(
ConnectorAccessRequest(
connector_id=profile.id,
credential_id=profile.credential_profile_id,
provider=profile.provider,
external_path="",
external_url=effective_endpoint_url,
operation="import",
),
profile.policy_sources,
).allowed
def _profile_visible_to_actor(
profile: ConnectorProfile,
*,
tenant_id: str,
user_id: str,
group_ids: set[str],
campaign_id: str | None,
campaign_visible: CampaignVisibility | None,
) -> bool:
return _scope_visible_to_actor(
scope_type=profile.scope_type,
scope_id=profile.scope_id,
tenant_id=tenant_id,
user_id=user_id,
group_ids=group_ids,
campaign_id=campaign_id,
campaign_visible=campaign_visible,
)
def _scope_visible_to_actor(
*,
scope_type: str,
scope_id: str | None,
tenant_id: str,
user_id: str,
group_ids: set[str],
campaign_id: str | None,
campaign_visible: CampaignVisibility | None,
) -> bool:
if scope_type == "system":
return True
if scope_type == "tenant":
return scope_id == tenant_id
if scope_type == "user":
return scope_id == user_id
if scope_type == "group":
return bool(scope_id and scope_id in group_ids)
if scope_type != "campaign" or not scope_id:
return False
if campaign_id and scope_id != campaign_id:
return False
return bool(campaign_visible and campaign_visible(scope_id))
def _with_effective_connector_policy(
session: Session,
*,
tenant_id: str,
profile: ConnectorProfile,
) -> ConnectorProfile:
sources = effective_connector_policy_sources(
session,
tenant_id=tenant_id,
scope_type=profile.scope_type,
scope_id=profile.scope_id,
)
if not sources:
return profile
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))

View File

@@ -2,21 +2,31 @@ from __future__ import annotations
import hashlib import hashlib
import mimetypes import mimetypes
from datetime import datetime
from pathlib import PurePosixPath from pathlib import PurePosixPath
from typing import Any, Iterable from typing import Any, Iterable
from uuid import uuid4 from uuid import uuid4
from sqlalchemy import and_, or_ from sqlalchemy import and_, exists, func, or_
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from govoplan_core.core.campaigns import CAPABILITY_CAMPAIGNS_ACCESS, CampaignAccessProvider from govoplan_core.core.campaigns import CAPABILITY_CAMPAIGNS_ACCESS, CampaignAccessProvider
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileShare, FileVersion from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileShare, FileVersion
from govoplan_files.backend.runtime import get_registry, settings from govoplan_files.backend.runtime import get_registry, settings
from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend from govoplan_files.backend.storage.backends import (
StorageBackendError,
StorageObjectMissing,
get_storage_backend,
)
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
from govoplan_files.backend.storage.integrity import (
QUARANTINED_BLOB_STATUSES,
read_verified_blob_bytes,
)
from govoplan_files.backend.storage.share_state import effective_file_share_clause
def _campaign_access_provider() -> CampaignAccessProvider: def _campaign_access_provider() -> CampaignAccessProvider:
@@ -72,11 +82,25 @@ def _get_or_create_blob(
) )
if blob: if blob:
backend = get_storage_backend() backend = get_storage_backend()
if not backend.exists(blob.storage_key): repair_required = (
blob.integrity_status in QUARANTINED_BLOB_STATUSES
or blob.quarantined_at is not None
)
try:
backend.stat(blob.storage_key)
except StorageObjectMissing:
repair_required = True
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
if repair_required:
try: try:
backend.put_bytes(blob.storage_key, data, content_type=content_type) backend.put_bytes(blob.storage_key, data, content_type=content_type)
except StorageBackendError as exc: except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc raise FileStorageError(str(exc)) from exc
blob.integrity_status = "verified"
blob.integrity_checked_at = utcnow()
blob.integrity_failure = None
blob.quarantined_at = None
blob.ref_count += 1 blob.ref_count += 1
session.add(blob) session.add(blob)
return blob return blob
@@ -96,6 +120,8 @@ def _get_or_create_blob(
size_bytes=size, size_bytes=size,
content_type=content_type, content_type=content_type,
ref_count=1, ref_count=1,
integrity_status="verified",
integrity_checked_at=utcnow(),
) )
session.add(blob) session.add(blob)
session.flush() session.flush()
@@ -328,7 +354,7 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
.filter( .filter(
FileShare.tenant_id == tenant_id, FileShare.tenant_id == tenant_id,
FileShare.file_asset_id == asset.id, FileShare.file_asset_id == asset.id,
FileShare.revoked_at.is_(None), effective_file_share_clause(),
FileShare.permission.in_(permission_values), FileShare.permission.in_(permission_values),
or_( or_(
(FileShare.target_type == "user") & (FileShare.target_id == user_id), (FileShare.target_type == "user") & (FileShare.target_id == user_id),
@@ -343,6 +369,32 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
return asset return asset
def get_asset_for_share_management(
session: Session,
*,
tenant_id: str,
user_id: str,
asset_id: str,
is_admin: bool = False,
) -> FileAsset:
asset = session.get(FileAsset, asset_id)
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
raise FileStorageError("File not found")
if is_admin:
return asset
owns_asset = (
asset.owner_type == "user"
and asset.owner_user_id == user_id
) or (
asset.owner_type == "group"
and asset.owner_group_id
in user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
)
if not owns_asset:
raise FileStorageError("Only file owners and administrators can manage shares")
return asset
def list_assets_for_user( def list_assets_for_user(
session: Session, session: Session,
*, *,
@@ -352,6 +404,8 @@ def list_assets_for_user(
owner_id: str | None = None, owner_id: str | None = None,
campaign_id: str | None = None, campaign_id: str | None = None,
path_prefix: str | None = None, path_prefix: str | None = None,
campaign_usage: str | None = None,
audit_relevant: bool | None = None,
include_deleted: bool = False, include_deleted: bool = False,
is_admin: bool = False, is_admin: bool = False,
) -> list[FileAsset]: ) -> list[FileAsset]:
@@ -363,6 +417,8 @@ def list_assets_for_user(
owner_id=owner_id, owner_id=owner_id,
campaign_id=campaign_id, campaign_id=campaign_id,
path_prefix=path_prefix, path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
include_deleted=include_deleted, include_deleted=include_deleted,
is_admin=is_admin, is_admin=is_admin,
) )
@@ -378,6 +434,8 @@ def list_assets_for_user_window(
owner_id: str | None = None, owner_id: str | None = None,
campaign_id: str | None = None, campaign_id: str | None = None,
path_prefix: str | None = None, path_prefix: str | None = None,
campaign_usage: str | None = None,
audit_relevant: bool | None = None,
include_deleted: bool = False, include_deleted: bool = False,
is_admin: bool = False, is_admin: bool = False,
page_size: int, page_size: int,
@@ -393,6 +451,8 @@ def list_assets_for_user_window(
owner_id=owner_id, owner_id=owner_id,
campaign_id=campaign_id, campaign_id=campaign_id,
path_prefix=path_prefix, path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
include_deleted=include_deleted, include_deleted=include_deleted,
is_admin=is_admin, is_admin=is_admin,
) )
@@ -417,6 +477,8 @@ def _asset_visibility_query_for_user(
owner_id: str | None = None, owner_id: str | None = None,
campaign_id: str | None = None, campaign_id: str | None = None,
path_prefix: str | None = None, path_prefix: str | None = None,
campaign_usage: str | None = None,
audit_relevant: bool | None = None,
include_deleted: bool = False, include_deleted: bool = False,
is_admin: bool = False, is_admin: bool = False,
): ):
@@ -434,7 +496,7 @@ def _asset_visibility_query_for_user(
FileShare.tenant_id == tenant_id, FileShare.tenant_id == tenant_id,
FileShare.target_type == "campaign", FileShare.target_type == "campaign",
FileShare.target_id == campaign_id, FileShare.target_id == campaign_id,
FileShare.revoked_at.is_(None), effective_file_share_clause(),
) )
elif not is_admin and not owner_type: elif not is_admin and not owner_type:
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id) group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
@@ -442,16 +504,73 @@ def _asset_visibility_query_for_user(
or_( or_(
(FileAsset.owner_type == "user") & (FileAsset.owner_user_id == user_id), (FileAsset.owner_type == "user") & (FileAsset.owner_user_id == user_id),
(FileAsset.owner_type == "group") & (FileAsset.owner_group_id.in_(group_ids)), (FileAsset.owner_type == "group") & (FileAsset.owner_group_id.in_(group_ids)),
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "user") & (FileShare.target_id == user_id), effective_file_share_clause() & (FileShare.target_type == "user") & (FileShare.target_id == user_id),
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)), effective_file_share_clause() & (FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id), effective_file_share_clause() & (FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
) )
) )
if path_prefix: if path_prefix:
prefix = normalize_folder(path_prefix) prefix = normalize_folder(path_prefix)
if prefix: if prefix:
query = query.filter(FileAsset.display_path.like(f"{prefix}/%")) query = query.filter(FileAsset.display_path.like(f"{prefix}/%"))
return query campaign_share_exists = exists().where(
FileShare.tenant_id == tenant_id,
FileShare.file_asset_id == FileAsset.id,
FileShare.target_type == "campaign",
effective_file_share_clause(),
)
campaign_use_exists = exists().where(
CampaignAttachmentUse.tenant_id == tenant_id,
CampaignAttachmentUse.file_asset_id == FileAsset.id,
)
if campaign_usage == "linked":
query = query.filter(or_(campaign_share_exists, campaign_use_exists))
elif campaign_usage == "unlinked":
query = query.filter(~or_(campaign_share_exists, campaign_use_exists))
if audit_relevant is not None:
sent_use_exists = exists().where(
CampaignAttachmentUse.tenant_id == tenant_id,
CampaignAttachmentUse.file_asset_id == FileAsset.id,
CampaignAttachmentUse.use_stage == "sent",
)
query = query.filter(sent_use_exists if audit_relevant else ~sent_use_exists)
return query.distinct()
def count_assets_for_user(
session: Session,
*,
tenant_id: str,
user_id: str,
owner_type: str | None = None,
owner_id: str | None = None,
campaign_id: str | None = None,
path_prefix: str | None = None,
campaign_usage: str | None = None,
audit_relevant: bool | None = None,
include_deleted: bool = False,
is_admin: bool = False,
) -> int:
query = _asset_visibility_query_for_user(
session,
tenant_id=tenant_id,
user_id=user_id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
campaign_usage=campaign_usage,
audit_relevant=audit_relevant,
include_deleted=include_deleted,
is_admin=is_admin,
)
return int(
query.order_by(None)
.with_entities(func.count(func.distinct(FileAsset.id)))
.scalar()
or 0
)
def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVersion, FileBlob]: def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVersion, FileBlob]:
@@ -495,10 +614,7 @@ def current_versions_and_blobs(session: Session, assets: Iterable[FileAsset]) ->
def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVersion, FileBlob]: def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVersion, FileBlob]:
version, blob = current_version_and_blob(session, asset) version, blob = current_version_and_blob(session, asset)
backend = get_storage_backend() backend = get_storage_backend()
try: return read_verified_blob_bytes(blob, backend=backend), version, blob
return backend.get_bytes(blob.storage_key), version, blob
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
def share_file( def share_file(
@@ -510,6 +626,7 @@ def share_file(
target_id: str, target_id: str,
permission: str, permission: str,
user_id: str, user_id: str,
expires_at: datetime | None = None,
) -> FileShare: ) -> FileShare:
target_type = target_type.lower().strip() target_type = target_type.lower().strip()
permission = permission.lower().strip() permission = permission.lower().strip()
@@ -519,6 +636,7 @@ def share_file(
raise FileStorageError("Unsupported share target") raise FileStorageError("Unsupported share target")
if permission not in {"read", "write", "manage"}: if permission not in {"read", "write", "manage"}:
raise FileStorageError("Unsupported file permission") raise FileStorageError("Unsupported file permission")
_validate_share_expiry(expires_at)
if target_type in {"user", "group", "tenant"}: if target_type in {"user", "group", "tenant"}:
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id) ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
if target_type == "campaign": if target_type == "campaign":
@@ -536,6 +654,7 @@ def share_file(
) )
if existing: if existing:
existing.permission = permission existing.permission = permission
existing.expires_at = expires_at
session.add(existing) session.add(existing)
return existing return existing
share = FileShare( share = FileShare(
@@ -545,6 +664,7 @@ def share_file(
target_id=target_id, target_id=target_id,
permission=permission, permission=permission,
created_by_user_id=user_id, created_by_user_id=user_id,
expires_at=expires_at,
) )
session.add(share) session.add(share)
return share return share
@@ -559,6 +679,7 @@ def share_files(
target_id: str, target_id: str,
permission: str, permission: str,
user_id: str, user_id: str,
expires_at: datetime | None = None,
) -> list[FileShare]: ) -> list[FileShare]:
target_type = target_type.lower().strip() target_type = target_type.lower().strip()
permission = permission.lower().strip() permission = permission.lower().strip()
@@ -566,6 +687,7 @@ def share_files(
raise FileStorageError("Unsupported share target") raise FileStorageError("Unsupported share target")
if permission not in {"read", "write", "manage"}: if permission not in {"read", "write", "manage"}:
raise FileStorageError("Unsupported file permission") raise FileStorageError("Unsupported file permission")
_validate_share_expiry(expires_at)
if target_type in {"user", "group", "tenant"}: if target_type in {"user", "group", "tenant"}:
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id) ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
if target_type == "campaign": if target_type == "campaign":
@@ -593,6 +715,7 @@ def share_files(
existing = existing_by_asset.get(asset.id) existing = existing_by_asset.get(asset.id)
if existing: if existing:
existing.permission = permission existing.permission = permission
existing.expires_at = expires_at
session.add(existing) session.add(existing)
shares.append(existing) shares.append(existing)
continue continue
@@ -603,12 +726,88 @@ def share_files(
target_id=target_id, target_id=target_id,
permission=permission, permission=permission,
created_by_user_id=user_id, created_by_user_id=user_id,
expires_at=expires_at,
) )
session.add(share) session.add(share)
shares.append(share) shares.append(share)
return shares return shares
def list_file_shares(
session: Session,
*,
tenant_id: str,
asset_id: str,
include_inactive: bool = False,
) -> list[FileShare]:
query = session.query(FileShare).filter(
FileShare.tenant_id == tenant_id,
FileShare.file_asset_id == asset_id,
)
if not include_inactive:
query = query.filter(effective_file_share_clause())
return query.order_by(FileShare.created_at.desc(), FileShare.id.desc()).all()
def revoke_file_share(
session: Session,
*,
tenant_id: str,
asset_id: str,
share_id: str,
user_id: str,
) -> tuple[FileShare, bool]:
share = (
session.query(FileShare)
.filter(
FileShare.id == share_id,
FileShare.tenant_id == tenant_id,
FileShare.file_asset_id == asset_id,
)
.one_or_none()
)
if share is None:
raise FileStorageError("File share not found")
if share.revoked_at is not None:
return share, False
share.revoked_at = utcnow()
share.revoked_by_user_id = user_id
session.add(share)
return share, True
def current_file_share_for_target(
session: Session,
*,
tenant_id: str,
asset_id: str,
target_type: str,
target_id: str,
) -> FileShare | None:
return (
session.query(FileShare)
.filter(
FileShare.tenant_id == tenant_id,
FileShare.file_asset_id == asset_id,
FileShare.target_type == target_type.lower().strip(),
FileShare.target_id == target_id,
FileShare.revoked_at.is_(None),
)
.order_by(FileShare.created_at.desc(), FileShare.id.desc())
.first()
)
def _validate_share_expiry(expires_at: datetime | None) -> None:
if expires_at is None:
return
from govoplan_files.backend.storage.share_state import file_share_is_active
candidate = FileShare(expires_at=expires_at)
if not file_share_is_active(candidate):
raise FileStorageError("File share expiry must be in the future")
def soft_delete_assets(session: Session, assets: Iterable[FileAsset]) -> int: def soft_delete_assets(session: Session, assets: Iterable[FileAsset]) -> int:
count = 0 count = 0

View File

@@ -185,15 +185,23 @@ class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
self._connection_pool.close() self._connection_pool.close()
_CONNECTOR_HTTP_CLIENT = httpx.Client(
transport=_OutboundPolicyHTTPTransport(),
follow_redirects=False,
timeout=15.0,
)
@contextmanager @contextmanager
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]: def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
with httpx.Client( timeout = kwargs.pop("timeout", 15.0)
transport=_OutboundPolicyHTTPTransport(), with _CONNECTOR_HTTP_CLIENT.stream(
follow_redirects=False, method,
timeout=kwargs.pop("timeout", 15.0), url,
) as client: timeout=timeout,
with client.stream(method, url, **kwargs) as response: **kwargs,
yield response ) as response:
yield response
def request_connector_bytes( def request_connector_bytes(

View File

@@ -0,0 +1,512 @@
from __future__ import annotations
import hashlib
from dataclasses import dataclass
from sqlalchemy.orm import Session
from govoplan_files.backend.db.models import (
FileBlob,
FileIntegrityFinding,
FileIntegrityScan,
)
from govoplan_files.backend.storage.backends import (
StorageBackend,
StorageBackendError,
StorageObjectMissing,
get_storage_backend,
)
from govoplan_files.backend.storage.common import FileStorageError, utcnow
TERMINAL_SCAN_STATUSES = {"completed", "cancelled"}
QUARANTINED_BLOB_STATUSES = {
"missing",
"size_mismatch",
"checksum_mismatch",
}
@dataclass(frozen=True, slots=True)
class BlobInspection:
valid: bool
kind: str
observed_size_bytes: int | None = None
observed_checksum_sha256: str | None = None
@dataclass(frozen=True, slots=True)
class IntegrityActionResult:
action: str
changed: bool
dry_run: bool
finding: FileIntegrityFinding
inspection: BlobInspection | None = None
def storage_prefix_for_tenant(tenant_id: str) -> str:
return f"tenants/{tenant_id}/files/"
def create_integrity_scan(
session: Session,
*,
tenant_id: str,
user_id: str,
verify_checksums: bool = True,
batch_size: int = 100,
backend: StorageBackend | None = None,
) -> FileIntegrityScan:
active_backend = backend or get_storage_backend()
scan = FileIntegrityScan(
tenant_id=tenant_id,
storage_backend=active_backend.name,
storage_prefix=storage_prefix_for_tenant(tenant_id),
verify_checksums=verify_checksums,
batch_size=max(1, min(int(batch_size), 1000)),
created_by_user_id=user_id,
)
session.add(scan)
session.flush()
return scan
def run_integrity_scan_batch(
session: Session,
scan: FileIntegrityScan,
*,
backend: StorageBackend | None = None,
) -> FileIntegrityScan:
if scan.status in TERMINAL_SCAN_STATUSES:
return scan
active_backend = backend or get_storage_backend()
if active_backend.name != scan.storage_backend:
raise FileStorageError(
"The configured storage backend changed after this integrity scan started"
)
if scan.started_at is None:
scan.started_at = utcnow()
scan.status = "running"
scan.last_error = None
if scan.phase == "blobs":
_scan_blob_batch(session, scan, active_backend)
elif scan.phase == "objects":
_scan_object_batch(session, scan, active_backend)
else:
scan.phase = "completed"
scan.status = "completed"
scan.completed_at = utcnow()
session.add(scan)
return scan
def mark_integrity_scan_failed(
scan: FileIntegrityScan,
*,
error: Exception,
) -> None:
scan.status = "failed"
scan.last_error = type(error).__name__[:255]
def inspect_blob(
blob: FileBlob,
*,
backend: StorageBackend,
verify_checksum: bool = True,
) -> BlobInspection:
try:
info = backend.stat(blob.storage_key)
except StorageObjectMissing:
return BlobInspection(valid=False, kind="missing")
if info.size_bytes != blob.size_bytes:
return BlobInspection(
valid=False,
kind="size_mismatch",
observed_size_bytes=info.size_bytes,
)
if not verify_checksum:
return BlobInspection(
valid=True,
kind="verified",
observed_size_bytes=info.size_bytes,
)
digest = hashlib.sha256()
observed_size = 0
for chunk in backend.iter_bytes(blob.storage_key):
observed_size += len(chunk)
digest.update(chunk)
observed_checksum = digest.hexdigest()
if observed_size != blob.size_bytes:
return BlobInspection(
valid=False,
kind="size_mismatch",
observed_size_bytes=observed_size,
observed_checksum_sha256=observed_checksum,
)
if observed_checksum != blob.checksum_sha256:
return BlobInspection(
valid=False,
kind="checksum_mismatch",
observed_size_bytes=observed_size,
observed_checksum_sha256=observed_checksum,
)
return BlobInspection(
valid=True,
kind="verified",
observed_size_bytes=observed_size,
observed_checksum_sha256=observed_checksum,
)
def apply_blob_inspection(
session: Session,
blob: FileBlob,
inspection: BlobInspection,
*,
checked_at=None,
) -> None:
timestamp = checked_at or utcnow()
blob.integrity_checked_at = timestamp
if inspection.valid:
blob.integrity_status = "verified"
blob.integrity_failure = None
blob.quarantined_at = None
else:
blob.integrity_status = inspection.kind
blob.integrity_failure = inspection.kind
blob.quarantined_at = blob.quarantined_at or timestamp
session.add(blob)
def ensure_blob_is_readable(blob: FileBlob) -> None:
if blob.integrity_status in QUARANTINED_BLOB_STATUSES or blob.quarantined_at:
raise FileStorageError(
"Managed file content is quarantined because integrity verification failed"
)
def read_verified_blob_bytes(
blob: FileBlob,
*,
backend: StorageBackend,
) -> bytes:
ensure_blob_is_readable(blob)
try:
data = backend.get_bytes(blob.storage_key)
except StorageBackendError as exc:
raise FileStorageError("Managed file content is not available") from exc
if len(data) != blob.size_bytes:
raise FileStorageError(
"Managed file content failed its recorded size verification"
)
if hashlib.sha256(data).hexdigest() != blob.checksum_sha256:
raise FileStorageError(
"Managed file content failed its recorded checksum verification"
)
return data
def recheck_integrity_finding(
session: Session,
finding: FileIntegrityFinding,
*,
user_id: str,
dry_run: bool,
backend: StorageBackend | None = None,
) -> IntegrityActionResult:
if finding.kind == "orphan_object" or not finding.blob_id:
raise FileStorageError("Only blob integrity findings can be rechecked")
blob = session.get(FileBlob, finding.blob_id)
if blob is None or blob.tenant_id != finding.tenant_id:
raise FileStorageError("Integrity finding blob no longer exists")
active_backend = backend or get_storage_backend()
scan = session.get(FileIntegrityScan, finding.scan_id)
if scan is None or scan.storage_backend != active_backend.name:
raise FileStorageError(
"The configured storage backend does not match the integrity finding"
)
inspection = inspect_blob(
blob,
backend=active_backend,
verify_checksum=True,
)
changed = False
if not dry_run:
previous = (
blob.integrity_status,
blob.quarantined_at,
finding.state,
)
apply_blob_inspection(session, blob, inspection)
_update_finding_from_inspection(finding, inspection)
if inspection.valid:
finding.state = "resolved"
finding.resolved_at = utcnow()
finding.resolved_by_user_id = user_id
session.add(finding)
changed = previous != (
blob.integrity_status,
blob.quarantined_at,
finding.state,
)
return IntegrityActionResult(
action="recheck",
changed=changed,
dry_run=dry_run,
finding=finding,
inspection=inspection,
)
def cleanup_orphan_finding(
session: Session,
finding: FileIntegrityFinding,
*,
user_id: str,
dry_run: bool,
backend: StorageBackend | None = None,
) -> IntegrityActionResult:
if finding.kind != "orphan_object":
raise FileStorageError("Only orphan-object findings can be cleaned up")
scan = session.get(FileIntegrityScan, finding.scan_id)
if scan is None or scan.tenant_id != finding.tenant_id:
raise FileStorageError("Integrity scan not found")
if not finding.storage_key.startswith(scan.storage_prefix):
raise FileStorageError("Orphan cleanup is outside the scan storage scope")
referenced = (
session.query(FileBlob.id)
.filter(
FileBlob.tenant_id == finding.tenant_id,
FileBlob.storage_backend == scan.storage_backend,
FileBlob.storage_key == finding.storage_key,
)
.first()
)
if referenced:
if not dry_run and finding.state != "resolved":
finding.state = "resolved"
finding.resolved_at = utcnow()
finding.resolved_by_user_id = user_id
session.add(finding)
return IntegrityActionResult(
action="retained_referenced",
changed=True,
dry_run=False,
finding=finding,
)
return IntegrityActionResult(
action="retained_referenced",
changed=False,
dry_run=dry_run,
finding=finding,
)
if finding.state == "deleted":
return IntegrityActionResult(
action="already_deleted",
changed=False,
dry_run=dry_run,
finding=finding,
)
if dry_run:
return IntegrityActionResult(
action="would_delete",
changed=False,
dry_run=True,
finding=finding,
)
active_backend = backend or get_storage_backend()
if active_backend.name != scan.storage_backend:
raise FileStorageError(
"The configured storage backend does not match the integrity finding"
)
try:
active_backend.stat(finding.storage_key)
except StorageObjectMissing:
action = "already_absent"
else:
active_backend.delete(finding.storage_key)
action = "deleted"
finding.state = "deleted"
finding.resolved_at = utcnow()
finding.resolved_by_user_id = user_id
session.add(finding)
return IntegrityActionResult(
action=action,
changed=True,
dry_run=False,
finding=finding,
)
def _scan_blob_batch(
session: Session,
scan: FileIntegrityScan,
backend: StorageBackend,
) -> None:
query = session.query(FileBlob).filter(
FileBlob.tenant_id == scan.tenant_id,
FileBlob.storage_backend == scan.storage_backend,
)
if scan.blob_cursor:
query = query.filter(FileBlob.id > scan.blob_cursor)
blobs = query.order_by(FileBlob.id.asc()).limit(scan.batch_size).all()
for blob in blobs:
inspection = inspect_blob(
blob,
backend=backend,
verify_checksum=scan.verify_checksums,
)
apply_blob_inspection(session, blob, inspection)
scan.scanned_blob_count += 1
if inspection.valid:
scan.verified_blob_count += 1
_resolve_scan_blob_findings(session, scan, blob)
else:
scan.quarantined_blob_count += 1
_record_blob_finding(session, scan, blob, inspection)
scan.blob_cursor = blob.id
if len(blobs) < scan.batch_size:
scan.phase = "objects"
scan.object_cursor = None
def _scan_object_batch(
session: Session,
scan: FileIntegrityScan,
backend: StorageBackend,
) -> None:
page = backend.list_objects(
prefix=scan.storage_prefix,
after=scan.object_cursor,
limit=scan.batch_size,
)
keys = [item.key for item in page.objects]
referenced_keys = {
row[0]
for row in session.query(FileBlob.storage_key)
.filter(
FileBlob.tenant_id == scan.tenant_id,
FileBlob.storage_backend == scan.storage_backend,
FileBlob.storage_key.in_(keys),
)
.all()
} if keys else set()
for item in page.objects:
scan.scanned_object_count += 1
if item.key in referenced_keys:
continue
scan.orphan_object_count += 1
_record_orphan_finding(
session,
scan,
storage_key=item.key,
size_bytes=item.size_bytes,
)
scan.object_cursor = page.next_cursor
if page.next_cursor is None:
scan.phase = "completed"
scan.status = "completed"
scan.completed_at = utcnow()
def _record_blob_finding(
session: Session,
scan: FileIntegrityScan,
blob: FileBlob,
inspection: BlobInspection,
) -> FileIntegrityFinding:
finding = (
session.query(FileIntegrityFinding)
.filter(
FileIntegrityFinding.scan_id == scan.id,
FileIntegrityFinding.blob_id == blob.id,
FileIntegrityFinding.kind == inspection.kind,
)
.one_or_none()
)
if finding is None:
finding = FileIntegrityFinding(
scan_id=scan.id,
tenant_id=scan.tenant_id,
kind=inspection.kind,
blob_id=blob.id,
storage_key=blob.storage_key,
expected_size_bytes=blob.size_bytes,
expected_checksum_sha256=blob.checksum_sha256,
)
_update_finding_from_inspection(finding, inspection)
session.add(finding)
return finding
def _record_orphan_finding(
session: Session,
scan: FileIntegrityScan,
*,
storage_key: str,
size_bytes: int,
) -> FileIntegrityFinding:
finding = (
session.query(FileIntegrityFinding)
.filter(
FileIntegrityFinding.scan_id == scan.id,
FileIntegrityFinding.kind == "orphan_object",
FileIntegrityFinding.storage_key == storage_key,
)
.one_or_none()
)
if finding is None:
finding = FileIntegrityFinding(
scan_id=scan.id,
tenant_id=scan.tenant_id,
kind="orphan_object",
storage_key=storage_key,
observed_size_bytes=size_bytes,
)
session.add(finding)
return finding
def _resolve_scan_blob_findings(
session: Session,
scan: FileIntegrityScan,
blob: FileBlob,
) -> None:
for finding in (
session.query(FileIntegrityFinding)
.filter(
FileIntegrityFinding.scan_id == scan.id,
FileIntegrityFinding.blob_id == blob.id,
FileIntegrityFinding.state == "open",
)
.all()
):
finding.state = "resolved"
finding.resolved_at = utcnow()
session.add(finding)
def _update_finding_from_inspection(
finding: FileIntegrityFinding,
inspection: BlobInspection,
) -> None:
finding.observed_size_bytes = inspection.observed_size_bytes
finding.observed_checksum_sha256 = inspection.observed_checksum_sha256
__all__ = [
"BlobInspection",
"IntegrityActionResult",
"QUARANTINED_BLOB_STATUSES",
"apply_blob_inspection",
"cleanup_orphan_finding",
"create_integrity_scan",
"ensure_blob_is_readable",
"inspect_blob",
"mark_integrity_scan_failed",
"read_verified_blob_bytes",
"recheck_integrity_finding",
"run_integrity_scan_batch",
"storage_prefix_for_tenant",
]

View File

@@ -0,0 +1,34 @@
from __future__ import annotations
from datetime import datetime, timezone
from sqlalchemy import and_, or_
from govoplan_files.backend.db.models import FileShare
from govoplan_files.backend.storage.common import utcnow
def effective_file_share_clause(*, at: datetime | None = None):
checked_at = at or utcnow()
return and_(
FileShare.revoked_at.is_(None),
or_(FileShare.expires_at.is_(None), FileShare.expires_at > checked_at),
)
def file_share_is_active(share: FileShare, *, at: datetime | None = None) -> bool:
if share.revoked_at is not None:
return False
if share.expires_at is None:
return True
checked_at = _aware_utc(at or utcnow())
return _aware_utc(share.expires_at) > checked_at
def _aware_utc(value: datetime) -> datetime:
if value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc)
__all__ = ["effective_file_share_clause", "file_share_is_active"]

View File

@@ -1,15 +1,18 @@
from __future__ import annotations from __future__ import annotations
import unittest import unittest
from datetime import datetime, timedelta, timezone
from sqlalchemy import create_engine from sqlalchemy import create_engine, text
from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db.models import Account, Group, User from govoplan_access.backend.db.models import Account, Group, User
from govoplan_core.core.access import PrincipalRef from govoplan_core.core.access import PrincipalRef
from govoplan_core.core.change_sequence import ChangeSequenceEntry
from govoplan_core.db.base import Base from govoplan_core.db.base import Base
from govoplan_files.backend.capabilities import FilesAccessService, virtual_folder_resource_id from govoplan_files.backend.capabilities import FilesAccessService, virtual_folder_resource_id
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
from govoplan_files.backend.storage.files import count_assets_for_user, list_assets_for_user
TENANT_ID = "tenant-1" TENANT_ID = "tenant-1"
@@ -75,10 +78,123 @@ class FilesAccessProviderTests(unittest.TestCase):
self.assertTrue(any(item.kind == "owner" and item.id == GROUP_ID for item in virtual_items)) self.assertTrue(any(item.kind == "owner" and item.id == GROUP_ID for item in virtual_items))
self.assertEqual("files.not_found", missing_items[0].source) self.assertEqual("files.not_found", missing_items[0].source)
def test_file_property_filters_cover_campaign_and_audit_usage(self) -> None:
session = _session()
self.addCleanup(_close_session, session)
_seed_access_subjects(session)
assets = [
FileAsset(
id=f"file-{index}",
tenant_id=TENANT_ID,
owner_type="user",
owner_user_id=USER_ID,
display_path=f"{index}.pdf",
filename=f"{index}.pdf",
)
for index in range(1, 4)
]
session.add_all([
*assets,
FileShare(
id="share-campaign",
tenant_id=TENANT_ID,
file_asset_id=assets[0].id,
target_type="campaign",
target_id="campaign-1",
permission="read",
),
FileShare(
id="share-campaign-expired",
tenant_id=TENANT_ID,
file_asset_id=assets[2].id,
target_type="campaign",
target_id="campaign-1",
permission="read",
expires_at=datetime.now(timezone.utc) - timedelta(seconds=1),
),
])
session.commit()
session.execute(
text(
"INSERT INTO campaign_attachment_uses "
"(id, tenant_id, file_asset_id, use_stage) "
"VALUES (:id, :tenant_id, :file_asset_id, :use_stage)"
),
{
"id": "attachment-use-1",
"tenant_id": TENANT_ID,
"file_asset_id": assets[1].id,
"use_stage": "sent",
},
)
session.commit()
linked = list_assets_for_user(
session,
tenant_id=TENANT_ID,
user_id=USER_ID,
owner_type="user",
owner_id=USER_ID,
campaign_usage="linked",
)
unlinked = list_assets_for_user(
session,
tenant_id=TENANT_ID,
user_id=USER_ID,
owner_type="user",
owner_id=USER_ID,
campaign_usage="unlinked",
)
audit_relevant = list_assets_for_user(
session,
tenant_id=TENANT_ID,
user_id=USER_ID,
owner_type="user",
owner_id=USER_ID,
audit_relevant=True,
)
self.assertEqual([asset.id for asset in linked], ["file-1", "file-2"])
self.assertEqual([asset.id for asset in unlinked], ["file-3"])
self.assertEqual([asset.id for asset in audit_relevant], ["file-2"])
self.assertEqual(
count_assets_for_user(
session,
tenant_id=TENANT_ID,
user_id=USER_ID,
owner_type="user",
owner_id=USER_ID,
campaign_usage="unlinked",
),
1,
)
def _session(): def _session():
engine = create_engine("sqlite:///:memory:", future=True) engine = create_engine("sqlite:///:memory:", future=True)
Base.metadata.create_all(bind=engine, tables=[Account.__table__, User.__table__, Group.__table__, FileAsset.__table__, FileFolder.__table__, FileShare.__table__]) Base.metadata.create_all(
bind=engine,
tables=[
Account.__table__,
User.__table__,
Group.__table__,
ChangeSequenceEntry.__table__,
FileAsset.__table__,
FileFolder.__table__,
FileShare.__table__,
],
)
with engine.begin() as connection:
connection.execute(
text(
"CREATE TABLE campaign_attachment_uses ("
"id VARCHAR(36) PRIMARY KEY, "
"tenant_id VARCHAR(36) NOT NULL, "
"file_asset_id VARCHAR(36) NOT NULL, "
"use_stage VARCHAR(20) NOT NULL"
")"
)
)
return sessionmaker(bind=engine, future=True)() return sessionmaker(bind=engine, future=True)()

View File

@@ -12,7 +12,8 @@ from govoplan_core.core.change_sequence import ChangeSequenceEntry
from govoplan_core.db.base import Base from govoplan_core.db.base import Base
from govoplan_core.security.secrets import encrypt_secret from govoplan_core.security.secrets import encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
from govoplan_files.backend.router import deactivate_connector_credential, deactivate_connector_profile from govoplan_files.backend.routes.connector_profiles import deactivate_connector_profile
from govoplan_files.backend.routes.connector_settings import deactivate_connector_credential
class Principal: class Principal:
@@ -169,6 +170,10 @@ class ConnectorCredentialDeletionTests(unittest.TestCase):
"govoplan_files.backend.storage.connector_credential_deletion.audit_event" "govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit: ) as audit:
plan.destroy_data_executor(self.session, "files") plan.destroy_data_executor(self.session, "files")
# The retirement executor deliberately enlists secret scrubbing, audit
# writes, and DDL in the caller-owned transaction. Commit that unit of
# work before inspecting through a separate Engine connection.
self.session.commit()
self.assertEqual(2, audit.call_count) self.assertEqual(2, audit.call_count)
self.assertEqual( self.assertEqual(
@@ -193,6 +198,7 @@ class ConnectorCredentialDeletionTests(unittest.TestCase):
"govoplan_files.backend.storage.connector_credential_deletion.audit_event" "govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit: ) as audit:
plan.destroy_data_executor(self.session, "files") plan.destroy_data_executor(self.session, "files")
self.session.commit()
audit.assert_called_once() audit.assert_called_once()
details = audit.call_args.kwargs["details"] details = audit.call_args.kwargs["details"]

View File

@@ -10,7 +10,7 @@ from unittest.mock import MagicMock
from fastapi import HTTPException from fastapi import HTTPException
from govoplan_files.backend.router import discover_connector_endpoint from govoplan_files.backend.routes.connector_settings import discover_connector_endpoint
from govoplan_files.backend.schemas import FileConnectorDiscoveryRequest from govoplan_files.backend.schemas import FileConnectorDiscoveryRequest
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _profile_password, _s3_verify from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _profile_password, _s3_verify
from govoplan_files.backend.storage.connector_deployment import ( from govoplan_files.backend.storage.connector_deployment import (
@@ -182,7 +182,7 @@ class ConnectorDiscoveryBoundaryTests(unittest.TestCase):
credential_mode="basic", credential_mode="basic",
credentials={"username": "admin", "password_env": "MASTER_KEY_B64"}, credentials={"username": "admin", "password_env": "MASTER_KEY_B64"},
) )
with patch("govoplan_files.backend.router.browse_connector_profile") as browse, self.assertRaises( with patch("govoplan_files.backend.routes.connector_settings.browse_connector_profile") as browse, self.assertRaises(
HTTPException HTTPException
) as raised: ) as raised:
discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type] discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
@@ -212,10 +212,10 @@ class ConnectorDiscoveryBoundaryTests(unittest.TestCase):
events.append("io") events.append("io")
return [] return []
with patch("govoplan_files.backend.router._ensure_connector_configuration_allowed", side_effect=ensure), patch( with patch("govoplan_files.backend.routes.connector_settings._ensure_connector_configuration_allowed", side_effect=ensure), patch(
"govoplan_files.backend.router._audit_connector_discovery_attempt", "govoplan_files.backend.routes.connector_settings._audit_connector_discovery_attempt",
side_effect=audit, side_effect=audit,
), patch("govoplan_files.backend.router.browse_connector_profile", side_effect=browse): ), patch("govoplan_files.backend.routes.connector_settings.browse_connector_profile", side_effect=browse):
response = discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type] response = discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
self.assertEqual("usable", response.status) self.assertEqual("usable", response.status)

View File

@@ -0,0 +1,66 @@
from __future__ import annotations
import unittest
from govoplan_files.backend.storage.connector_policy import (
ConnectorAccessRequest,
ConnectorPolicySource,
connector_policy_decision,
)
class ConnectorPolicyPatternTests(unittest.TestCase):
def test_resource_reference_allow_patterns_match(self) -> None:
decision = connector_policy_decision(
ConnectorAccessRequest(
connector_id="dev-smb",
credential_id="credential-primary",
provider="smb",
),
(
ConnectorPolicySource(
scope_type="tenant",
scope_id="tenant-1",
label="Tenant",
policy={
"allow": {
"connectors": ["dev-*"],
"credentials": ["credential-*"],
"providers": ["s?b"],
}
},
),
),
)
self.assertTrue(decision.allowed)
def test_resource_reference_deny_patterns_take_precedence(self) -> None:
decision = connector_policy_decision(
ConnectorAccessRequest(
connector_id="dev-smb",
credential_id="credential-legacy",
provider="smb",
),
(
ConnectorPolicySource(
scope_type="tenant",
scope_id="tenant-1",
label="Tenant",
policy={
"allow": {"connectors": ["dev-*"]},
"deny": {"credentials": ["*-legacy"]},
},
),
),
)
self.assertFalse(decision.allowed)
self.assertEqual(
("connector_policy_denylist",),
decision.requirements,
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,228 @@
from __future__ import annotations
import json
import unittest
from unittest.mock import patch
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db import models as access_models # noqa: F401 - resolve Files user foreign keys
from govoplan_core.db.base import Base
from govoplan_core.security.credential_envelopes import CredentialEnvelope
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorProfile
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_profile_store import (
list_database_connector_profiles,
update_connector_profile_row,
)
class ConnectorProfileStoreUpdateTests(unittest.TestCase):
def setUp(self) -> None:
self.engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(
self.engine,
tables=[
FileConnectorProfile.__table__,
CredentialEnvelope.__table__,
],
)
self.session = sessionmaker(bind=self.engine)()
self.row = FileConnectorProfile(
id="profile-1",
tenant_id="tenant-1",
scope_type="tenant",
scope_id="tenant-1",
label="Original profile",
provider="webdav",
endpoint_url="https://dav.example.test/original",
base_path="original",
enabled=True,
credential_mode="basic",
username="original-user",
password_encrypted=encrypt_secret("original-password"),
token_encrypted=encrypt_secret("original-token"),
capabilities=["browse"],
policy={"allow": {"providers": ["webdav"]}},
metadata_={"original": True},
created_by_user_id=None,
updated_by_user_id=None,
)
self.session.add(self.row)
self.session.commit()
def tearDown(self) -> None:
self.session.close()
Base.metadata.drop_all(bind=self.engine)
self.engine.dispose()
def test_update_preserves_normalization_and_flush_only_transaction_boundary(self) -> None:
updated = update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
label=" Updated profile ",
provider="NEXTCLOUD",
endpoint_url=" https://cloud.example.test/dav ",
base_path=" shared/reports ",
enabled=False,
credential_profile_id=" credential-2 ",
credential_mode="TOKEN",
username=" updated-user ",
password="updated-password",
token="updated-token",
capabilities=["browse", " import ", ""],
policy={"deny": {"external_paths": ["private"]}},
metadata={"department": "reports"},
)
self.assertIs(self.row, updated)
self.assertEqual("Updated profile", updated.label)
self.assertEqual("nextcloud", updated.provider)
self.assertEqual("https://cloud.example.test/dav", updated.endpoint_url)
self.assertEqual("shared/reports", updated.base_path)
self.assertFalse(updated.enabled)
self.assertEqual("credential-2", updated.credential_profile_id)
self.assertEqual("token", updated.credential_mode)
self.assertEqual("updated-user", updated.username)
self.assertEqual("updated-password", decrypt_secret(updated.password_encrypted))
self.assertEqual("updated-token", decrypt_secret(updated.token_encrypted))
self.assertEqual(["browse", "import"], updated.capabilities)
self.assertEqual({"deny": {"external_paths": ["private"]}}, updated.policy)
self.assertEqual({"department": "reports"}, updated.metadata_)
self.assertEqual("user-2", updated.updated_by_user_id)
self.session.rollback()
persisted = self.session.get(FileConnectorProfile, self.row.id)
assert persisted is not None
self.assertEqual("Original profile", persisted.label)
self.assertEqual("original-password", decrypt_secret(persisted.password_encrypted))
self.assertTrue(persisted.enabled)
def test_secret_replacement_wins_over_clear_while_clear_removes_an_omitted_token(self) -> None:
update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
password="replacement-password",
clear_password=True,
clear_token=True,
)
self.assertEqual("replacement-password", decrypt_secret(self.row.password_encrypted))
self.assertIsNone(self.row.token_encrypted)
self.assertEqual("https://dav.example.test/original", self.row.endpoint_url)
self.assertEqual("original-user", self.row.username)
def test_legacy_external_secret_reference_cannot_be_cleared_or_partially_mutate_the_row(self) -> None:
self.row.secret_ref = "vault:tenant-1:files:profile"
self.session.commit()
with self.assertRaisesRegex(FileStorageError, "provider-side deletion"):
update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
label="Must not be applied",
secret_ref="",
)
self.assertEqual("Original profile", self.row.label)
self.assertEqual("vault:tenant-1:files:profile", self.row.secret_ref)
def test_flush_failure_remains_rollback_safe(self) -> None:
with patch.object(self.session, "flush", side_effect=RuntimeError("database unavailable")), self.assertRaisesRegex(
RuntimeError,
"database unavailable",
):
update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
label="Uncommitted profile",
password="uncommitted-password",
)
self.session.rollback()
persisted = self.session.get(FileConnectorProfile, self.row.id)
assert persisted is not None
self.assertEqual("Original profile", persisted.label)
self.assertEqual("original-password", decrypt_secret(persisted.password_encrypted))
def test_visibility_filter_runs_before_connector_secrets_are_decrypted(self) -> None:
invisible = FileConnectorProfile(
id="invisible-profile",
tenant_id="tenant-1",
scope_type="user",
scope_id="another-user",
label="Invisible profile",
provider="webdav",
endpoint_url="https://invisible.example.test",
enabled=True,
credential_mode="basic",
password_encrypted="not-a-valid-encrypted-secret",
capabilities=["browse"],
policy={},
metadata_={},
)
self.session.add(invisible)
self.session.commit()
profiles = list_database_connector_profiles(
self.session,
tenant_id="tenant-1",
row_visible=lambda row: row.id == self.row.id,
)
self.assertEqual([self.row.id], [profile.id for profile in profiles])
self.assertEqual("original-password", profiles[0].password_value)
def test_reusable_credential_resolves_and_revocation_keeps_profile_visible(self) -> None:
credential = CredentialEnvelope(
id="shared-credential",
tenant_id="tenant-1",
scope_type="tenant",
scope_id="tenant-1",
name="Shared WebDAV login",
credential_kind="username_password",
public_data={"username": "ada"},
secret_data_encrypted=encrypt_secret(json.dumps({"password": "secret"})),
secret_keys=["password"],
allowed_modules=["files"],
allowed_server_refs=[],
inherit_to_lower_scopes=True,
is_active=True,
revision="revision-1",
)
self.row.credential_profile_id = "credential-envelope:shared-credential"
self.row.username = None
self.row.password_encrypted = None
self.row.token_encrypted = None
self.session.add(credential)
self.session.commit()
profile = list_database_connector_profiles(
self.session,
tenant_id="tenant-1",
)[0]
self.assertEqual("ada", profile.username)
self.assertEqual("secret", profile.password_value)
self.assertTrue(profile.credentials_configured)
credential.is_active = False
self.session.commit()
profile = list_database_connector_profiles(
self.session,
tenant_id="tenant-1",
)[0]
self.assertIsNone(profile.password_value)
self.assertFalse(profile.credentials_configured)
self.assertTrue(profile.metadata["credential_unavailable"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,265 @@
from __future__ import annotations
import unittest
from dataclasses import replace
from unittest.mock import patch
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.connector_visibility import (
connector_profile_usable_for_import,
visible_connector_profiles_for_actor,
)
def _profile(
profile_id: str,
*,
scope_type: str = "system",
scope_id: str | None = None,
provider: str = "webdav",
enabled: bool = True,
) -> ConnectorProfile:
return ConnectorProfile(
id=profile_id,
label=profile_id,
provider=provider,
scope_type=scope_type,
scope_id=scope_id,
endpoint_url=f"https://{profile_id}.example.invalid",
enabled=enabled,
credential_mode="anonymous",
)
class ConnectorVisibilityTests(unittest.TestCase):
@patch(
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings"
)
@patch(
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
)
def test_profiles_are_filtered_to_actor_scopes_and_database_definition_wins(
self,
database_profiles,
configured_profiles,
_policy_sources,
) -> None:
profiles = [
_profile("system"),
_profile("tenant", scope_type="tenant", scope_id="tenant-1"),
_profile("other-tenant", scope_type="tenant", scope_id="tenant-2"),
_profile("user", scope_type="user", scope_id="user-1"),
_profile("other-user", scope_type="user", scope_id="user-2"),
_profile("group", scope_type="group", scope_id="group-1"),
_profile("campaign", scope_type="campaign", scope_id="campaign-1"),
_profile("disabled", enabled=False),
_profile("duplicate", provider="webdav"),
]
database_profiles.return_value = (profiles, {profile.id for profile in profiles})
configured_profiles.return_value = [_profile("duplicate", provider="nextcloud")]
visible = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids={"group-1"},
settings=object(),
campaign_visible=lambda campaign_id: campaign_id == "campaign-1",
)
self.assertEqual(
{"system", "tenant", "user", "group", "campaign", "duplicate"},
{profile.id for profile in visible},
)
duplicate = next(profile for profile in visible if profile.id == "duplicate")
self.assertEqual("webdav", duplicate.provider)
@patch(
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
)
def test_campaign_and_admin_visibility_remain_explicit(
self,
database_profiles,
_configured_profiles,
_policy_sources,
) -> None:
profiles = [
_profile("campaign-a", scope_type="campaign", scope_id="campaign-a"),
_profile("campaign-b", scope_type="campaign", scope_id="campaign-b"),
_profile("other-user", scope_type="user", scope_id="user-2"),
]
database_profiles.return_value = (profiles, {profile.id for profile in profiles})
campaign_only = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids=(),
settings=None,
campaign_id="campaign-a",
campaign_visible=lambda _campaign_id: True,
)
self.assertEqual(["campaign-a"], [profile.id for profile in campaign_only])
admin = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids=(),
settings=None,
campaign_visible=None,
include_admin_scopes=True,
)
self.assertEqual(
{"campaign-a", "campaign-b", "other-user"},
{profile.id for profile in admin},
)
@patch(
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
)
@patch(
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources"
)
def test_effective_policy_is_attached_without_mutating_profile(
self,
policy_sources,
database_profiles,
_configured_profiles,
) -> None:
profile = _profile("profile")
source = ConnectorPolicySource(
scope_type="system",
label="System",
policy={"allow": {"providers": ["webdav"]}},
)
database_profiles.return_value = ([profile], {profile.id})
policy_sources.return_value = [source]
visible = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids=(),
settings=None,
)
self.assertEqual((), profile.policy_sources)
self.assertEqual((source,), visible[0].policy_sources)
def test_import_usability_requires_safe_provider_credentials_endpoint_and_identity_policy(
self,
) -> None:
self.assertTrue(connector_profile_usable_for_import(_profile("webdav")))
self.assertFalse(
connector_profile_usable_for_import(_profile("s3", provider="s3"))
)
self.assertFalse(
connector_profile_usable_for_import(_profile("smb", provider="smb"))
)
self.assertFalse(
connector_profile_usable_for_import(
_profile("sharepoint", provider="sharepoint")
)
)
self.assertFalse(
connector_profile_usable_for_import(
ConnectorProfile(
id="no-endpoint",
label="No endpoint",
provider="webdav",
credential_mode="anonymous",
)
)
)
self.assertFalse(
connector_profile_usable_for_import(
ConnectorProfile(
id="no-credentials",
label="No credentials",
provider="webdav",
endpoint_url="https://files.example.invalid",
credential_mode="basic",
)
)
)
self.assertTrue(
connector_profile_usable_for_import(
ConnectorProfile(
id="metadata-endpoint",
label="Metadata endpoint",
provider="webdav",
credential_mode="anonymous",
metadata={"webdav_endpoint_url": "https://files.example.invalid"},
)
)
)
self.assertFalse(
connector_profile_usable_for_import(
ConnectorProfile(
id="unresolved-secret-reference",
label="Unresolved secret reference",
provider="webdav",
endpoint_url="https://files.example.invalid",
credential_mode="basic",
secret_ref="vault://files/webdav",
)
)
)
denied = replace(
_profile("denied"),
policy_sources=(
ConnectorPolicySource(
scope_type="system",
label="System",
policy={"deny": {"providers": ["webdav"]}},
),
),
)
self.assertFalse(connector_profile_usable_for_import(denied))
path_limited = replace(
_profile("path-limited"),
policy_sources=(
ConnectorPolicySource(
scope_type="system",
label="System",
policy={"allow": {"external_paths": ["/approved/*"]}},
),
),
)
self.assertFalse(connector_profile_usable_for_import(path_limited))
endpoint_allowed = replace(
_profile("endpoint-allowed"),
policy_sources=(
ConnectorPolicySource(
scope_type="system",
label="System",
policy={"allow": {"external_urls": ["https://*.example.invalid"]}},
),
),
)
self.assertTrue(connector_profile_usable_for_import(endpoint_allowed))
if __name__ == "__main__":
unittest.main()

241
tests/test_documentation.py Normal file
View File

@@ -0,0 +1,241 @@
from __future__ import annotations
import unittest
from types import SimpleNamespace
from unittest.mock import patch
from sqlalchemy.orm import Session
from govoplan_core.core.modules import DocumentationContext
from govoplan_files.backend.documentation import documentation_topics
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
class _Principal:
tenant_id = "tenant-1"
user = SimpleNamespace(id="user-1")
group_ids = frozenset({"group-1"})
def __init__(self, scopes: set[str]) -> None:
self.scopes = frozenset(scopes)
def has(self, scope: str) -> bool:
return scope in self.scopes
class FilesRuntimeDocumentationTests(unittest.TestCase):
def setUp(self) -> None:
self.session = Session()
def tearDown(self) -> None:
self.session.close()
def context(
self,
scopes: set[str],
*,
settings: object | None = None,
documentation_type: str = "user",
) -> DocumentationContext:
return DocumentationContext(
registry=object(),
principal=_Principal(scopes),
settings=settings,
session=self.session,
documentation_type=documentation_type, # type: ignore[arg-type]
)
def test_upload_tasks_state_exact_current_safe_limits(self) -> None:
settings = SimpleNamespace(
file_upload_max_bytes=7 * 1024 * 1024,
file_upload_zip_max_bytes=19 * 1024 * 1024,
)
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[],
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context(
{"files:file:read", "files:file:upload"}, settings=settings
)
)
}
upload = topics["files.workflow.upload-managed-files"]
self.assertIn("7 MiB (7,340,032 bytes)", upload.body)
self.assertEqual(["files.list"], upload.metadata["help_contexts"])
self.assertEqual(
{"files:file:read", "files:file:upload"},
set(upload.conditions[0].required_scopes),
)
archive = topics["files.workflow.upload-and-unpack-zip"]
self.assertIn("19 MiB (19,922,944 bytes)", archive.body)
self.assertIn("7 MiB (7,340,032 bytes)", archive.body)
self.assertIn("1,000", archive.body)
self.assertIn("Actual extracted bytes are counted", archive.body)
def test_connector_task_requires_authority_and_a_visible_usable_profile(
self,
) -> None:
usable = ConnectorProfile(
id="private-profile-id",
label="Private profile label",
provider="webdav",
scope_type="tenant",
scope_id="tenant-1",
endpoint_url="https://private.example.invalid/secret-root",
base_path="/secret-root",
credential_mode="anonymous",
)
context = self.context({"files:file:read", "files:file:upload"})
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[usable],
) as visible:
topics = {topic.id: topic for topic in documentation_topics(context)}
self.assertIn("files.workflow.import-managed-snapshot", topics)
self.assertNotIn("files.connector-import-unavailable", topics)
task = topics["files.workflow.import-managed-snapshot"]
self.assertEqual("configured", task.layer)
self.assertEqual(
["files.list", "files.connector-import"], task.metadata["help_contexts"]
)
self.assertIn("re-authorized", task.body)
self.assertNotIn("private-profile-id", repr(task))
self.assertNotIn("private.example.invalid", repr(task))
self.assertNotIn("secret-root", repr(task))
visible.assert_called_once()
self.assertEqual(("group-1",), tuple(visible.call_args.kwargs["group_ids"]))
without_authority = {
topic.id: topic
for topic in documentation_topics(self.context({"files:file:read"}))
}
self.assertNotIn("files.workflow.import-managed-snapshot", without_authority)
self.assertIn(
"both permission to view Files and permission to upload",
without_authority["files.connector-import-unavailable"].body,
)
def test_connector_limitation_is_precise_but_never_exposes_profile_internals(
self,
) -> None:
blocked = ConnectorProfile(
id="sensitive-profile-id",
label="Sensitive label",
provider="s3",
scope_type="tenant",
scope_id="tenant-1",
endpoint_url="https://internal.example.invalid/private",
base_path="/classified",
credential_mode="basic",
password_value="credential-secret",
)
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[blocked],
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context({"files:file:read", "files:file:upload"})
)
}
self.assertNotIn("files.workflow.import-managed-snapshot", topics)
limitation = topics["files.connector-import-unavailable"]
self.assertEqual("available", limitation.layer)
self.assertIn("pinning-safe provider", limitation.body)
payload = repr(limitation)
for secret in (
"sensitive-profile-id",
"Sensitive label",
"internal.example.invalid",
"classified",
"credential-secret",
):
self.assertNotIn(secret, payload)
def test_files_admin_connector_groups_do_not_widen_campaign_membership(self) -> None:
usable = ConnectorProfile(
id="group-profile",
label="Group profile",
provider="webdav",
scope_type="group",
scope_id="admin-visible-group",
endpoint_url="https://files.example.invalid",
credential_mode="anonymous",
)
def groups_for_actor(_session, *, include_admin_groups, **_kwargs):
return ["member-group", "admin-visible-group"] if include_admin_groups else ["member-group"]
with (
patch(
"govoplan_files.backend.documentation.user_group_ids",
side_effect=groups_for_actor,
),
patch(
"govoplan_files.backend.documentation._campaign_visibility",
return_value=None,
) as campaign_visibility,
patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[usable],
) as visible,
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context(
{
"files:file:read",
"files:file:upload",
"files:file:admin",
}
)
)
}
self.assertIn("files.workflow.import-managed-snapshot", topics)
self.assertEqual(
("member-group", "admin-visible-group"),
tuple(visible.call_args.kwargs["group_ids"]),
)
self.assertEqual(
("member-group",),
tuple(campaign_visibility.call_args.kwargs["group_ids"]),
)
def test_connector_evaluation_errors_fail_closed_without_error_details(
self,
) -> None:
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
side_effect=RuntimeError("private endpoint /root credential-id"),
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context({"files:file:read", "files:file:upload"})
)
}
self.assertNotIn("files.workflow.import-managed-snapshot", topics)
limitation = topics["files.connector-import-unavailable"]
self.assertIn("could not be safely evaluated", limitation.body)
self.assertNotIn("private endpoint", repr(limitation))
self.assertNotIn("credential-id", repr(limitation))
def test_runtime_provider_only_emits_user_documentation(self) -> None:
self.assertEqual(
(), documentation_topics(self.context(set(), documentation_type="admin"))
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,196 @@
from __future__ import annotations
import hashlib
import tempfile
import unittest
from pathlib import Path
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db.models import Account, Group, User
from govoplan_core.db.base import Base
from govoplan_files.backend.db.models import (
FileBlob,
FileIntegrityFinding,
FileIntegrityScan,
)
from govoplan_files.backend.storage.backends import LocalFilesystemStorageBackend
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.integrity import (
cleanup_orphan_finding,
create_integrity_scan,
read_verified_blob_bytes,
recheck_integrity_finding,
run_integrity_scan_batch,
)
TENANT_ID = "tenant-1"
USER_ID = "user-1"
class IntegrityReconciliationTests(unittest.TestCase):
def setUp(self) -> None:
self.temporary_directory = tempfile.TemporaryDirectory()
self.addCleanup(self.temporary_directory.cleanup)
self.backend = LocalFilesystemStorageBackend(
Path(self.temporary_directory.name)
)
self.engine = create_engine("sqlite:///:memory:", future=True)
Base.metadata.create_all(
bind=self.engine,
tables=[
Account.__table__,
User.__table__,
Group.__table__,
FileBlob.__table__,
FileIntegrityScan.__table__,
FileIntegrityFinding.__table__,
],
)
self.session = sessionmaker(bind=self.engine, future=True)()
self.addCleanup(self._close)
def _close(self) -> None:
self.session.close()
self.engine.dispose()
def test_scan_is_bounded_resumable_and_reconciles_both_orphan_directions(
self,
) -> None:
valid_data = b"valid"
restored_data = b"restore-me"
expected_corrupt_data = b"expected"
stored_corrupt_data = b"corrupt!"
valid = _blob("blob-1", "valid.bin", valid_data)
missing = _blob("blob-2", "missing.bin", restored_data)
corrupt = _blob("blob-3", "corrupt.bin", expected_corrupt_data)
self.session.add_all([valid, missing, corrupt])
self.session.commit()
self.backend.put_bytes(valid.storage_key, valid_data)
self.backend.put_bytes(corrupt.storage_key, stored_corrupt_data)
orphan_key = f"tenants/{TENANT_ID}/files/orphan.bin"
self.backend.put_bytes(orphan_key, b"orphan")
scan = create_integrity_scan(
self.session,
tenant_id=TENANT_ID,
user_id=USER_ID,
batch_size=1,
backend=self.backend,
)
self.session.commit()
invocations = 0
while scan.status != "completed":
run_integrity_scan_batch(
self.session,
scan,
backend=self.backend,
)
self.session.commit()
invocations += 1
scan = self.session.get(FileIntegrityScan, scan.id)
self.assertIsNotNone(scan)
self.session.expire_all()
if invocations > 12:
self.fail("Integrity scan did not complete")
self.assertGreater(invocations, 3)
self.assertEqual(3, scan.scanned_blob_count)
self.assertEqual(1, scan.verified_blob_count)
self.assertEqual(2, scan.quarantined_blob_count)
self.assertEqual(3, scan.scanned_object_count)
self.assertEqual(1, scan.orphan_object_count)
findings = (
self.session.query(FileIntegrityFinding)
.filter(FileIntegrityFinding.scan_id == scan.id)
.all()
)
self.assertEqual(
{"missing", "checksum_mismatch", "orphan_object"},
{finding.kind for finding in findings},
)
missing = self.session.get(FileBlob, missing.id)
corrupt = self.session.get(FileBlob, corrupt.id)
self.assertEqual("missing", missing.integrity_status)
self.assertEqual("checksum_mismatch", corrupt.integrity_status)
with self.assertRaisesRegex(FileStorageError, "quarantined"):
read_verified_blob_bytes(missing, backend=self.backend)
missing_finding = next(
finding for finding in findings if finding.kind == "missing"
)
self.backend.put_bytes(missing.storage_key, restored_data)
preview = recheck_integrity_finding(
self.session,
missing_finding,
user_id=USER_ID,
dry_run=True,
backend=self.backend,
)
self.assertTrue(preview.inspection.valid)
self.assertEqual("open", missing_finding.state)
repaired = recheck_integrity_finding(
self.session,
missing_finding,
user_id=USER_ID,
dry_run=False,
backend=self.backend,
)
self.session.commit()
self.assertTrue(repaired.changed)
self.assertEqual("resolved", missing_finding.state)
self.assertEqual(
restored_data,
read_verified_blob_bytes(missing, backend=self.backend),
)
orphan_finding = next(
finding for finding in findings if finding.kind == "orphan_object"
)
preview_cleanup = cleanup_orphan_finding(
self.session,
orphan_finding,
user_id=USER_ID,
dry_run=True,
backend=self.backend,
)
self.assertEqual("would_delete", preview_cleanup.action)
self.assertTrue(self.backend.exists(orphan_key))
cleanup = cleanup_orphan_finding(
self.session,
orphan_finding,
user_id=USER_ID,
dry_run=False,
backend=self.backend,
)
repeated = cleanup_orphan_finding(
self.session,
orphan_finding,
user_id=USER_ID,
dry_run=False,
backend=self.backend,
)
self.assertEqual("deleted", cleanup.action)
self.assertFalse(self.backend.exists(orphan_key))
self.assertEqual("already_deleted", repeated.action)
self.assertFalse(repeated.changed)
def _blob(blob_id: str, filename: str, expected_data: bytes) -> FileBlob:
return FileBlob(
id=blob_id,
tenant_id=TENANT_ID,
storage_backend="local",
storage_key=f"tenants/{TENANT_ID}/files/{filename}",
checksum_sha256=hashlib.sha256(expected_data).hexdigest(),
size_bytes=len(expected_data),
ref_count=1,
)
if __name__ == "__main__":
unittest.main()

View File

@@ -3,20 +3,213 @@ from __future__ import annotations
import unittest import unittest
STATIC_TOPIC_IDS = {
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
"files.workflow.share-managed-files",
"files.workflow.delete-managed-files",
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
"files.reference.snapshot-provenance-and-capabilities",
"files.assurance.process-and-release-readiness",
}
RUNTIME_TOPIC_IDS = {
"files.workflow.upload-managed-files",
"files.workflow.upload-and-unpack-zip",
"files.workflow.import-managed-snapshot",
"files.connector-import-unavailable",
}
HANDBOOK_HREF = "govoplan-files/docs/FILES_HANDBOOK.md"
class FilesManifestDocumentationTests(unittest.TestCase): class FilesManifestDocumentationTests(unittest.TestCase):
def test_connector_topic_covers_governance_pinning_and_provenance_perspectives(self) -> None: @classmethod
def setUpClass(cls) -> None:
from govoplan_files.backend.manifest import manifest from govoplan_files.backend.manifest import manifest
topic = next(item for item in manifest.documentation if item.id == "files.governed-connectors-and-provenance") cls.manifest = manifest
cls.topics = {topic.id: topic for topic in manifest.documentation}
def topic(self, topic_id: str):
return self.topics[topic_id]
def test_static_topics_have_role_scope_module_and_link_contracts(self) -> None:
self.assertEqual(STATIC_TOPIC_IDS, set(self.topics))
self.assertEqual(
{"admin", "user"},
{
item
for topic in self.topics.values()
for item in topic.documentation_types
},
)
for topic in self.topics.values():
with self.subTest(topic=topic.id):
self.assertTrue(topic.audience)
self.assertTrue(topic.conditions)
self.assertTrue(
any(
"files" in condition.required_modules
for condition in topic.conditions
)
)
self.assertTrue(
any(
condition.any_scopes or condition.required_scopes
for condition in topic.conditions
)
)
self.assertIn("runtime", {link.kind for link in topic.links})
self.assertIn(
HANDBOOK_HREF,
{link.href for link in topic.links if link.kind == "repository"},
)
self.assertIn(
"documentation_topics",
{provider.__name__ for provider in self.manifest.documentation_providers},
)
def test_user_tasks_are_independently_authorized_and_contextual(self) -> None:
expected_scopes = {
"files.workflow.organize-managed-files": {
"files:file:read",
"files:file:organize",
},
"files.workflow.find-and-download-files": {
"files:file:read",
"files:file:download",
},
"files.workflow.share-managed-files": {
"files:file:read",
"files:file:share",
},
"files.workflow.delete-managed-files": {
"files:file:read",
"files:file:delete",
},
}
for topic_id, scopes in expected_scopes.items():
with self.subTest(topic=topic_id):
topic = self.topic(topic_id)
self.assertEqual(("user",), topic.documentation_types)
self.assertEqual("workflow", topic.metadata["kind"])
self.assertEqual(scopes, set(topic.conditions[0].required_scopes))
self.assertEqual(["files.list"], topic.metadata["help_contexts"])
for key in ("prerequisites", "steps", "outcome", "verification"):
self.assertTrue(topic.metadata[key])
def test_share_and_delete_tasks_state_current_boundaries(self) -> None:
share = self.topic("files.workflow.share-managed-files")
self.assertEqual("available", share.layer)
self.assertIn("Expired and revoked grants stop authorizing", share.body)
self.assertIn("revocation is idempotent", share.body)
self.assertIn(
"/api/v1/files/{file_id}/shares", {link.href for link in share.links}
)
delete = self.topic("files.workflow.delete-managed-files")
self.assertIn("Soft-delete", delete.summary)
self.assertIn("no self-service restore or hard-purge", delete.body)
self.assertTrue(
any("not a hard purge" in item for item in delete.metadata["limitations"])
)
def test_admin_topic_covers_policy_redaction_and_atomic_credential_deletion(
self,
) -> None:
topic = self.topic("files.governed-connectors-and-provenance")
self.assertEqual(("admin",), topic.documentation_types)
self.assertEqual("reference", topic.metadata["kind"])
self.assertEqual("File connections", topic.metadata["screen"])
self.assertTrue(topic.metadata["section"])
self.assertIn("deny rules win", topic.body)
self.assertIn("redact secret values", topic.body)
self.assertIn("same transaction", topic.body)
self.assertTrue(
any(
"non-owned external references" in item
for item in topic.metadata["security_invariants"]
)
)
self.assertIn(
"/api/v1/files/connectors/policies/tenant",
{link.href for link in topic.links},
)
self.assertIn(
"/api/v1/files/connectors/credentials", {link.href for link in topic.links}
)
def test_operator_topic_covers_recovery_and_fail_closed_s3_smb(self) -> None:
topic = self.topic(
"files.reference.integrity-recovery-and-fail-closed-transports"
)
self.assertEqual(("admin",), topic.documentation_types)
self.assertEqual("reference", topic.metadata["kind"])
self.assertIn("S3", topic.body)
self.assertIn("SMB", topic.body)
self.assertIn("fail closed", topic.body)
self.assertIn("DFS referrals", topic.body)
self.assertIn("does not remove backend blob objects", topic.body)
self.assertIn("bounded resumable integrity scan", topic.body)
self.assertIn("quarantined", topic.body)
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
self.assertTrue(topic.metadata["verification"])
self.assertIn(
"/api/v1/files/integrity/scans",
{link.href for link in topic.links},
)
self.assertIn(
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", topic.configuration_keys
)
def test_integrator_topic_exposes_capability_and_provenance_boundary(self) -> None:
topic = self.topic("files.reference.snapshot-provenance-and-capabilities")
self.assertEqual(("admin", "user"), topic.documentation_types) self.assertEqual(("admin", "user"), topic.documentation_types)
self.assertEqual({"user", "admin", "operator"}, set(topic.metadata["perspectives"])) self.assertEqual("reference", topic.metadata["kind"])
self.assertIn("fails closed", topic.body) self.assertEqual(
self.assertIn("DFS referrals", topic.body) ["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
self.assertIn("immediately scrubs", topic.body) topic.metadata["provided_interfaces"],
self.assertTrue(any("non-owned external references" in item for item in topic.metadata["security_invariants"])) )
self.assertIn("revision", topic.metadata["provenance_fields"]) self.assertIn("revision", topic.metadata["provenance_fields"])
self.assertIn("/api/v1/files/connectors/profiles", {link.href for link in topic.links}) self.assertIn("exact asset, version, blob, checksum", topic.body)
self.assertIn("collaboration", topic.body)
def test_process_and_release_assurance_exposes_gates_evidence_and_limits(
self,
) -> None:
topic = self.topic("files.assurance.process-and-release-readiness")
self.assertEqual(("admin", "user"), topic.documentation_types)
self.assertEqual("workflow", topic.metadata["kind"])
self.assertEqual(["files.list"], topic.metadata["help_contexts"])
self.assertIn("process_owner", topic.audience)
self.assertIn("release_manager", topic.audience)
self.assertIn("versions align", topic.body)
self.assertIn("Share grant, change, expiry, and revocation", topic.body)
self.assertIn("no enforced retention or legal hold", topic.body)
for key in ("prerequisites", "steps", "outcome", "verification"):
self.assertTrue(topic.metadata[key])
self.assertTrue(
any("meta-repository security" in step for step in topic.metadata["steps"])
)
self.assertTrue(any("fails closed" in step for step in topic.metadata["steps"]))
self.assertTrue(any("SHA-256" in step for step in topic.metadata["steps"]))
def test_internal_related_topic_ids_resolve(self) -> None:
known_ids = STATIC_TOPIC_IDS | RUNTIME_TOPIC_IDS
for topic in self.topics.values():
for related_id in topic.metadata.get("related_topic_ids", []):
if related_id.startswith("files."):
self.assertIn(
related_id,
known_ids,
f"{topic.id} refers to missing topic {related_id}",
)
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -1,11 +1,59 @@
from __future__ import annotations from __future__ import annotations
import unittest import unittest
from collections import Counter
from inspect import signature
from govoplan_files.backend.router import router from govoplan_files.backend.router import router
from govoplan_files.backend.routes.assets import router as assets_router
from govoplan_files.backend.routes.connector_io import router as connector_io_router
from govoplan_files.backend.routes.connector_profiles import router as connector_profiles_router
from govoplan_files.backend.routes.connector_settings import router as connector_settings_router
from govoplan_files.backend.routes.folders import router as folders_router
from govoplan_files.backend.routes.integrity import router as integrity_router
from govoplan_files.backend.routes.listing import router as listing_router
from govoplan_files.backend.routes.shares import router as shares_router
from govoplan_files.backend.routes.spaces import router as spaces_router
from govoplan_files.backend.routes.transfers import router as transfers_router
from govoplan_files.backend.routes.uploads import router as uploads_router
class FilesRouterContractTests(unittest.TestCase): class FilesRouterContractTests(unittest.TestCase):
@staticmethod
def _operation_keys(candidate_router) -> list[tuple[str, str]]:
return [
(method, route.path)
for route in candidate_router.routes
for method in sorted(route.methods or ())
]
def test_composed_router_contains_every_workflow_operation_once(self) -> None:
workflow_routers = (
spaces_router,
folders_router,
integrity_router,
listing_router,
uploads_router,
connector_settings_router,
connector_io_router,
connector_profiles_router,
assets_router,
shares_router,
transfers_router,
)
expected = [
operation
for workflow_router in workflow_routers
for operation in self._operation_keys(workflow_router)
]
actual = self._operation_keys(router)
self.assertEqual(expected, actual)
self.assertEqual(50, len(actual))
self.assertFalse(
[operation for operation, count in Counter(actual).items() if count > 1]
)
def test_connector_routes_keep_existing_api_paths(self) -> None: def test_connector_routes_keep_existing_api_paths(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes} routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
@@ -31,6 +79,26 @@ class FilesRouterContractTests(unittest.TestCase):
self.assertIn((("POST",), "/files/bulk-rename"), routes) self.assertIn((("POST",), "/files/bulk-rename"), routes)
self.assertIn((("POST",), "/files/transfer"), routes) self.assertIn((("POST",), "/files/transfer"), routes)
def test_share_lifecycle_routes_are_exposed(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
self.assertIn((("GET",), "/files/{file_id}/shares"), routes)
self.assertIn((("POST",), "/files/{file_id}/shares"), routes)
self.assertIn((("DELETE",), "/files/{file_id}/shares/{share_id}"), routes)
self.assertIn((("GET",), "/files/{file_id}/share-target-options"), routes)
def test_file_listing_exposes_structured_property_filters(self) -> None:
route = next(
route
for route in listing_router.routes
if route.path == "/files" and "GET" in (route.methods or ())
)
parameters = signature(route.endpoint).parameters
self.assertIn("campaign_usage", parameters)
self.assertIn("audit_relevant", parameters)
self.assertIn("total", route.response_model.model_fields)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()

View File

@@ -0,0 +1,219 @@
from __future__ import annotations
import unittest
from datetime import timedelta
from unittest.mock import patch
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db.models import Account, Group, User
from govoplan_core.core.change_sequence import ChangeSequenceEntry
from govoplan_core.db.base import Base
from govoplan_files.backend.db.models import FileAsset, FileShare
from govoplan_files.backend.storage.common import FileStorageError, utcnow
from govoplan_files.backend.storage.files import (
get_asset_for_user,
list_file_shares,
revoke_file_share,
share_file,
)
TENANT_ID = "tenant-1"
OWNER_ID = "owner-1"
RECIPIENT_ID = "recipient-1"
GROUP_ID = "group-1"
class FileShareLifecycleTests(unittest.TestCase):
def setUp(self) -> None:
self.engine = create_engine("sqlite:///:memory:", future=True)
Base.metadata.create_all(
bind=self.engine,
tables=[
Account.__table__,
User.__table__,
Group.__table__,
ChangeSequenceEntry.__table__,
FileAsset.__table__,
FileShare.__table__,
],
)
self.session = sessionmaker(bind=self.engine, future=True)()
self.asset = FileAsset(
id="file-1",
tenant_id=TENANT_ID,
owner_type="user",
owner_user_id=OWNER_ID,
display_path="shared.pdf",
filename="shared.pdf",
)
self.session.add(self.asset)
self.session.commit()
def tearDown(self) -> None:
self.session.close()
self.engine.dispose()
@patch(
"govoplan_files.backend.storage.files.user_group_ids",
return_value=[],
)
def test_expired_share_stops_access_immediately(self, _groups) -> None:
self.session.add(
FileShare(
id="expired-share",
tenant_id=TENANT_ID,
file_asset_id=self.asset.id,
target_type="user",
target_id=RECIPIENT_ID,
permission="read",
expires_at=utcnow() - timedelta(seconds=1),
)
)
self.session.commit()
with self.assertRaisesRegex(FileStorageError, "No access"):
get_asset_for_user(
self.session,
tenant_id=TENANT_ID,
user_id=RECIPIENT_ID,
asset_id=self.asset.id,
)
self.assertEqual(
[],
list_file_shares(
self.session,
tenant_id=TENANT_ID,
asset_id=self.asset.id,
),
)
@patch(
"govoplan_files.backend.storage.files.user_group_ids",
return_value=[GROUP_ID],
)
def test_independent_active_grant_survives_other_expiry(self, _groups) -> None:
self.session.add_all(
[
FileShare(
id="expired-user-share",
tenant_id=TENANT_ID,
file_asset_id=self.asset.id,
target_type="user",
target_id=RECIPIENT_ID,
permission="read",
expires_at=utcnow() - timedelta(seconds=1),
),
FileShare(
id="active-group-share",
tenant_id=TENANT_ID,
file_asset_id=self.asset.id,
target_type="group",
target_id=GROUP_ID,
permission="read",
expires_at=utcnow() + timedelta(hours=1),
),
]
)
self.session.commit()
result = get_asset_for_user(
self.session,
tenant_id=TENANT_ID,
user_id=RECIPIENT_ID,
asset_id=self.asset.id,
)
self.assertEqual(self.asset.id, result.id)
self.assertEqual(
["active-group-share"],
[
share.id
for share in list_file_shares(
self.session,
tenant_id=TENANT_ID,
asset_id=self.asset.id,
)
],
)
@patch(
"govoplan_files.backend.storage.files.ensure_share_target_exists",
return_value=None,
)
def test_revoke_is_idempotent_and_future_expiry_is_persisted(
self, _target_exists
) -> None:
expiry = utcnow() + timedelta(days=1)
share = share_file(
self.session,
tenant_id=TENANT_ID,
asset=self.asset,
target_type="user",
target_id=RECIPIENT_ID,
permission="read",
user_id=OWNER_ID,
expires_at=expiry,
)
self.session.commit()
revoked, first_changed = revoke_file_share(
self.session,
tenant_id=TENANT_ID,
asset_id=self.asset.id,
share_id=share.id,
user_id=OWNER_ID,
)
self.session.commit()
repeated, second_changed = revoke_file_share(
self.session,
tenant_id=TENANT_ID,
asset_id=self.asset.id,
share_id=share.id,
user_id=OWNER_ID,
)
self.assertTrue(first_changed)
self.assertFalse(second_changed)
self.assertEqual(OWNER_ID, revoked.revoked_by_user_id)
self.assertEqual(revoked.revoked_at, repeated.revoked_at)
self.assertEqual([], list_file_shares(
self.session,
tenant_id=TENANT_ID,
asset_id=self.asset.id,
))
self.assertEqual(
[share.id],
[
item.id
for item in list_file_shares(
self.session,
tenant_id=TENANT_ID,
asset_id=self.asset.id,
include_inactive=True,
)
],
)
@patch(
"govoplan_files.backend.storage.files.ensure_share_target_exists",
return_value=None,
)
def test_past_expiry_is_rejected(self, _target_exists) -> None:
with self.assertRaisesRegex(FileStorageError, "future"):
share_file(
self.session,
tenant_id=TENANT_ID,
asset=self.asset,
target_type="user",
target_id=RECIPIENT_ID,
permission="read",
user_id=OWNER_ID,
expires_at=utcnow() - timedelta(seconds=1),
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,108 @@
from __future__ import annotations
import unittest
from sqlalchemy import create_engine, event
from sqlalchemy.orm import Session
from govoplan_access.backend.db.models import Account, Group, User
from govoplan_core.core.change_sequence import ChangeSequenceEntry
from govoplan_core.db.base import Base
from govoplan_files.backend.db.models import (
FileAsset,
FileConnectorCredential,
FileConnectorPolicy,
FileConnectorProfile,
FileConnectorSpace,
)
from govoplan_files.backend.manifest import _tenant_summary_batch
class FilesTenantSummaryBatchTests(unittest.TestCase):
def test_batch_summary_uses_one_grouped_query_per_owned_table(self) -> None:
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(
engine,
tables=[
Account.__table__,
User.__table__,
Group.__table__,
FileAsset.__table__,
FileConnectorCredential.__table__,
FileConnectorPolicy.__table__,
FileConnectorProfile.__table__,
FileConnectorSpace.__table__,
ChangeSequenceEntry.__table__,
],
)
try:
with Session(engine) as session:
session.add_all(
[
FileAsset(
id="file-1",
tenant_id="tenant-1",
owner_type="tenant",
display_path="/one.txt",
filename="one.txt",
),
FileConnectorCredential(
id="credential-1",
tenant_id="tenant-1",
label="Credential",
),
FileConnectorPolicy(
id="policy-1",
tenant_id="tenant-1",
scope_type="tenant",
),
FileConnectorProfile(
id="profile-1",
tenant_id="tenant-1",
label="Profile",
provider="webdav",
),
FileConnectorSpace(
id="space-1",
tenant_id="tenant-1",
owner_type="tenant",
label="Space",
connector_profile_id="profile-1",
provider="webdav",
),
]
)
session.commit()
query_count = 0
def count_query(*_args: object) -> None:
nonlocal query_count
query_count += 1
event.listen(engine, "before_cursor_execute", count_query)
try:
counts = _tenant_summary_batch(
session,
["tenant-1", "tenant-empty"],
)
finally:
event.remove(engine, "before_cursor_execute", count_query)
self.assertEqual(5, query_count)
self.assertEqual(
{
"files": 1,
"connector_credentials": 1,
"connector_policies": 1,
"connector_profiles": 1,
"connector_spaces": 1,
},
counts["tenant-1"],
)
self.assertEqual(0, counts["tenant-empty"]["files"])
finally:
engine.dispose()
if __name__ == "__main__":
unittest.main()

View File

@@ -14,7 +14,8 @@
"./styles/file-manager.css": "./src/styles/file-manager.css" "./styles/file-manager.css": "./src/styles/file-manager.css"
}, },
"scripts": { "scripts": {
"test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs" "test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs",
"test:file-property-filters": "node scripts/test-file-property-filters-structure.mjs"
}, },
"peerDependencies": { "peerDependencies": {
"@vitejs/plugin-react": "^4.3.4", "@vitejs/plugin-react": "^4.3.4",
@@ -22,7 +23,7 @@
"typescript": "^5.7.2", "typescript": "^5.7.2",
"react": "^19.0.0", "react": "^19.0.0",
"react-dom": "^19.0.0", "react-dom": "^19.0.0",
"react-router-dom": "^7.1.1", "react-router-dom": ">=7.18.2 <8",
"lucide-react": "^1.23.0", "lucide-react": "^1.23.0",
"@govoplan/core-webui": "^0.1.9" "@govoplan/core-webui": "^0.1.9"
}, },

View File

@@ -0,0 +1,36 @@
import { readFileSync } from "node:fs";
const page = readFileSync(new URL("../src/features/files/FilesPage.tsx", import.meta.url), "utf8").replace(/\s+/g, " ");
const api = readFileSync(new URL("../src/api/files.ts", import.meta.url), "utf8").replace(/\s+/g, " ");
function assertIncludes(source, fragment, message) {
if (!source.includes(fragment.replace(/\s+/g, " "))) throw new Error(message);
}
assertIncludes(
api,
"campaign_usage?: FileCampaignUsageFilter; audit_relevant?: boolean;",
"the Files API must expose structured campaign and audit filters"
);
assertIncludes(
api,
"do { const response = await listFiles",
"property filtering must traverse every server result page"
);
assertIncludes(
page,
"if (propertyFiltersActive && propertyFilterResults) return propertyFilterResults;",
"server-filtered rows must replace, not narrow, the currently loaded explorer window"
);
assertIncludes(
page,
'<option value="unlinked">Not linked or used</option>',
"the high-priority unlinked campaign-file filter must remain available"
);
assertIncludes(
page,
"setPropertyFilterTotal(response.total);",
"the UI must retain the exact server-side filtered count"
);
console.log("Files property filters cover the complete server result window.");

View File

@@ -1,4 +1,15 @@
import { ApiError, apiFetch, apiUrl, authHeaders, csrfToken, type ApiSettings, type DeltaDeletedItem, type FilesManagedFileLinkTarget } from "@govoplan/core-webui"; import {
ApiError,
apiFetch,
apiReferenceOptionProvider,
apiUrl,
authHeaders,
csrfToken,
type ApiSettings,
type DeltaDeletedItem,
type FilesManagedFileLinkTarget,
type ReferenceOptionProvider
} from "@govoplan/core-webui";
export { fetchResourceAccessExplanation } from "@govoplan/core-webui"; export { fetchResourceAccessExplanation } from "@govoplan/core-webui";
export type { export type {
AccessDecisionProvenanceItem, AccessDecisionProvenanceItem,
@@ -24,11 +35,16 @@ export type FileSpace = {
export type FileShare = { export type FileShare = {
id: string; id: string;
file_asset_id: string;
target_type: string; target_type: string;
target_id: string; target_id: string;
permission: string; permission: string;
created_by_user_id?: string | null;
created_at: string; created_at: string;
expires_at?: string | null;
revoked_at?: string | null; revoked_at?: string | null;
revoked_by_user_id?: string | null;
active: boolean;
}; };
export type FileSourceProvenance = { export type FileSourceProvenance = {
@@ -297,7 +313,8 @@ export type ManagedFile = {
shares?: FileShare[]; shares?: FileShare[];
}; };
export type FileListResponse = {files: ManagedFile[];cursor?: string | null;next_cursor?: string | null;watermark?: string | null;}; export type FileCampaignUsageFilter = "linked" | "unlinked";
export type FileListResponse = {files: ManagedFile[];total: number;cursor?: string | null;next_cursor?: string | null;watermark?: string | null;};
export type FileDeltaDeletedItem = {id: string;resource_type?: string | null;revision?: string | null;deleted_at?: string | null;}; export type FileDeltaDeletedItem = {id: string;resource_type?: string | null;revision?: string | null;deleted_at?: string | null;};
export type FileDeltaResponse = { export type FileDeltaResponse = {
files: ManagedFile[]; files: ManagedFile[];
@@ -383,15 +400,43 @@ payload: {owner_type: "user" | "group";owner_id: string;path: string;recursive?:
return apiFetch<FolderDeleteResponse>(settings, "/api/v1/files/folders/delete", { method: "POST", body: JSON.stringify({ recursive: true, ...payload }) }); return apiFetch<FolderDeleteResponse>(settings, "/api/v1/files/folders/delete", { method: "POST", body: JSON.stringify({ recursive: true, ...payload }) });
} }
export function listFiles(settings: ApiSettings, params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;page_size?: number;cursor?: string | null;} = {}): Promise<FileListResponse> { export function listFiles(settings: ApiSettings, params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;campaign_usage?: FileCampaignUsageFilter;audit_relevant?: boolean;page_size?: number;cursor?: string | null;} = {}): Promise<FileListResponse> {
const search = new URLSearchParams(); const search = new URLSearchParams();
for (const [key, value] of Object.entries(params)) { for (const [key, value] of Object.entries(params)) {
if (value) search.set(key, String(value)); if (value !== undefined && value !== null && value !== "") search.set(key, String(value));
} }
const suffix = search.toString() ? `?${search.toString()}` : ""; const suffix = search.toString() ? `?${search.toString()}` : "";
return apiFetch<FileListResponse>(settings, `/api/v1/files${suffix}`); return apiFetch<FileListResponse>(settings, `/api/v1/files${suffix}`);
} }
export async function listFilesByProperties(
settings: ApiSettings,
params: {
owner_type: "user" | "group";
owner_id: string;
path_prefix?: string;
campaign_usage?: FileCampaignUsageFilter;
audit_relevant?: boolean;
page_size?: number;
})
: Promise<{files: ManagedFile[];total: number;}> {
const pageSize = params.page_size ?? DEFAULT_MANAGED_FILE_WINDOW_SIZE;
let cursor: string | null | undefined = null;
let total = 0;
let files: ManagedFile[] = [];
do {
const response = await listFiles(settings, {
...params,
page_size: pageSize,
cursor
});
files = files.concat(response.files);
total = response.total;
cursor = response.next_cursor;
} while (cursor);
return { files, total };
}
export function listFilesDelta( export function listFilesDelta(
settings: ApiSettings, settings: ApiSettings,
params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;since?: string;limit?: number;} = {}) params: {owner_type?: string;owner_id?: string;campaign_id?: string;path_prefix?: string;since?: string;limit?: number;} = {})
@@ -564,6 +609,12 @@ export function bulkDeleteFiles(settings: ApiSettings, fileIds: string[]): Promi
} }
export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;}; export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;};
export type FileSharePayload = {
target_type: "user" | "group" | "tenant" | "campaign";
target_id: string;
permission: "read" | "write" | "manage";
expires_at?: string | null;
};
export function virtualFolderResourceId(options: {tenantId: string;ownerType: "user" | "group";ownerId: string;path: string;}): string { export function virtualFolderResourceId(options: {tenantId: string;ownerType: "user" | "group";ownerId: string;path: string;}): string {
return `virtual-folder:v1:${options.tenantId}:${options.ownerType}:${options.ownerId}:${base64Url(options.path.replace(/\\/g, "/").replace(/^\/+|\/+$/g, ""))}`; return `virtual-folder:v1:${options.tenantId}:${options.ownerType}:${options.ownerId}:${base64Url(options.path.replace(/\\/g, "/").replace(/^\/+|\/+$/g, ""))}`;
@@ -583,6 +634,54 @@ export function shareFilesWithTarget(settings: ApiSettings, fileIds: string[], t
}); });
} }
export function listFileShares(
settings: ApiSettings,
fileId: string,
options: { includeInactive?: boolean } = {}
): Promise<{ shares: FileShare[] }> {
const suffix = options.includeInactive ? "?include_inactive=true" : "";
return apiFetch<{ shares: FileShare[] }>(
settings,
`/api/v1/files/${encodeURIComponent(fileId)}/shares${suffix}`
);
}
export function createFileShare(
settings: ApiSettings,
fileId: string,
payload: FileSharePayload
): Promise<FileShare> {
return apiFetch<FileShare>(
settings,
`/api/v1/files/${encodeURIComponent(fileId)}/shares`,
{ method: "POST", body: JSON.stringify(payload) }
);
}
export function revokeFileShare(
settings: ApiSettings,
fileId: string,
shareId: string
): Promise<FileShare> {
return apiFetch<FileShare>(
settings,
`/api/v1/files/${encodeURIComponent(fileId)}/shares/${encodeURIComponent(shareId)}`,
{ method: "DELETE" }
);
}
export function fileShareTargetProvider(
settings: ApiSettings,
fileId: string,
targetType: "user" | "group"
): ReferenceOptionProvider {
return apiReferenceOptionProvider(
settings,
`/api/v1/files/${encodeURIComponent(fileId)}/share-target-options`,
{ target_type: targetType }
);
}
export function evaluateFileConnectorPolicy( export function evaluateFileConnectorPolicy(
settings: ApiSettings, settings: ApiSettings,
payload: {source_provenance: FileSourceProvenance;operation?: string;policy_sources?: FileConnectorPolicySource[];}) payload: {source_provenance: FileSourceProvenance;operation?: string;policy_sources?: FileConnectorPolicySource[];})

View File

@@ -14,16 +14,21 @@ import {
AdvancedOptionsPanel, AdvancedOptionsPanel,
LoadingFrame, LoadingFrame,
mergeDeltaRows, mergeDeltaRows,
ReferenceMultiSelect,
SegmentedControl, SegmentedControl,
staticReferenceOptionProvider,
StatusBadge, StatusBadge,
TableActionGroup, TableActionGroup,
ToggleSwitch, ToggleSwitch,
useDeltaWatermarks, useDeltaWatermarks,
usePlatformLanguage,
useUnsavedDraftGuard, useUnsavedDraftGuard,
wildcardReferenceOption,
type ApiSettings, type ApiSettings,
type ConnectionTreeColumn, type ConnectionTreeColumn,
type FileConnectorScope, type FileConnectorScope,
type FileConnectorTargetOption, type FileConnectorTargetOption,
type ReferenceOption,
i18nMessage } from i18nMessage } from
"@govoplan/core-webui"; "@govoplan/core-webui";
import { import {
@@ -186,6 +191,7 @@ export default function FileConnectorSettingsPanel({
const [credentialLoginResult, setCredentialLoginResult] = useState<{ok: boolean;message: string;} | null>(null); const [credentialLoginResult, setCredentialLoginResult] = useState<{ok: boolean;message: string;} | null>(null);
const [message, setMessage] = useState(""); const [message, setMessage] = useState("");
const [error, setError] = useState(""); const [error, setError] = useState("");
const { translateText } = usePlatformLanguage();
const { getDeltaWatermark, setDeltaWatermark, resetDeltaWatermark } = useDeltaWatermarks(); const { getDeltaWatermark, setDeltaWatermark, resetDeltaWatermark } = useDeltaWatermarks();
const requiresTarget = scopeType === "user" || scopeType === "group"; const requiresTarget = scopeType === "user" || scopeType === "group";
@@ -198,6 +204,10 @@ export default function FileConnectorSettingsPanel({
const profileDirty = Boolean(draft) && connectorProfileDraftKey(draft) !== savedDraftKey; const profileDirty = Boolean(draft) && connectorProfileDraftKey(draft) !== savedDraftKey;
const credentialDirty = Boolean(credentialDraft) && connectorCredentialDraftKey(credentialDraft) !== savedCredentialDraftKey; const credentialDirty = Boolean(credentialDraft) && connectorCredentialDraftKey(credentialDraft) !== savedCredentialDraftKey;
const policyDirty = scopeReady && centralPolicyDraftKey(policyDraft) !== savedPolicyDraftKey; const policyDirty = scopeReady && centralPolicyDraftKey(policyDraft) !== savedPolicyDraftKey;
const policyConflicts = useMemo(
() => policyReferenceConflicts(policyDraft),
[policyDraft]
);
useUnsavedDraftGuard({ useUnsavedDraftGuard({
dirty: profileDirty || credentialDirty || policyDirty, dirty: profileDirty || credentialDirty || policyDirty,
@@ -225,6 +235,79 @@ export default function FileConnectorSettingsPanel({
() => credentials.filter((credential) => connectorBelongsToEditableScope(credential, scopeType, activeScopeId)), () => credentials.filter((credential) => connectorBelongsToEditableScope(credential, scopeType, activeScopeId)),
[activeScopeId, credentials, scopeType] [activeScopeId, credentials, scopeType]
); );
const selectableCredentials = useMemo(
() => credentials.filter((credential) =>
connectorBelongsToEditableScope(credential, scopeType, activeScopeId) ||
credential.source_kind === "credential_envelope" &&
reusableCredentialAvailableAtScope(credential, scopeType, activeScopeId)
),
[activeScopeId, credentials, scopeType]
);
const connectorReferenceOptions = useMemo<ReferenceOption[]>(
() => profiles.map((profile) => ({
value: profile.id,
label: profile.label || profile.id,
description: [
profile.provider,
profile.source_path,
profile.enabled ? null : "Inactive"
].filter(Boolean).join(" · "),
kind: "file_connection",
availability: profile.enabled ? "available" : "inactive",
disabled: !profile.enabled,
sourceModule: "files",
provenance: {
scopeType: profile.scope_type,
scopeId: profile.scope_id,
sourcePath: profile.source_path
}
})),
[profiles]
);
const credentialReferenceOptions = useMemo<ReferenceOption[]>(
() => credentials.map((credential) => ({
value: credential.id,
label: credential.label || credential.id,
description: [
credential.provider || "shared",
credential.source_path,
credential.enabled ? null : "Inactive"
].filter(Boolean).join(" · "),
kind: "file_credential",
availability: credential.enabled ? "available" : "inactive",
disabled: !credential.enabled,
sourceModule: credential.source_kind === "credential_envelope" ? "core" : "files",
provenance: {
scopeType: credential.scope_type,
scopeId: credential.scope_id,
sourcePath: credential.source_path,
sourceKind: credential.source_kind
}
})),
[credentials]
);
const providerReferenceOptions = useMemo<ReferenceOption[]>(
() => PROVIDERS.map((provider) => ({
value: provider.id,
label: translateText(provider.label),
description: provider.id,
kind: "file_provider",
sourceModule: "files"
})),
[translateText]
);
const connectorReferenceProvider = useMemo(
() => staticReferenceOptionProvider(connectorReferenceOptions),
[connectorReferenceOptions]
);
const credentialReferenceProvider = useMemo(
() => staticReferenceOptionProvider(credentialReferenceOptions),
[credentialReferenceOptions]
);
const providerReferenceProvider = useMemo(
() => staticReferenceOptionProvider(providerReferenceOptions),
[providerReferenceOptions]
);
useEffect(() => { useEffect(() => {
resetDeltaWatermark(deltaKey); resetDeltaWatermark(deltaKey);
@@ -424,7 +507,7 @@ export default function FileConnectorSettingsPanel({
endpoint_url: cleanOrNull(draft.endpointUrl), endpoint_url: cleanOrNull(draft.endpointUrl),
base_path: cleanOrNull(draft.basePath), base_path: cleanOrNull(draft.basePath),
enabled: draft.enabled, enabled: draft.enabled,
credential_profile_id: cleanOrNull(credentialProfileIdForDraft(draft, scopedCredentials)), credential_profile_id: cleanOrNull(credentialProfileIdForDraft(draft, selectableCredentials)),
credential_mode: draft.credentialMode, credential_mode: draft.credentialMode,
capabilities, capabilities,
policy: policyFromDraft(draft), policy: policyFromDraft(draft),
@@ -694,7 +777,7 @@ export default function FileConnectorSettingsPanel({
const panelTitle = title ?? i18nMessage("i18n:govoplan-files.value_file_connections", { value0: scopeLabel(scopeType) }); const panelTitle = title ?? i18nMessage("i18n:govoplan-files.value_file_connections", { value0: scopeLabel(scopeType) });
const selectedCredentialIds = new Set(scopedProfiles.map((profile) => profile.credential_profile_id).filter((value): value is string => Boolean(value))); const selectedCredentialIds = new Set(scopedProfiles.map((profile) => profile.credential_profile_id).filter((value): value is string => Boolean(value)));
const firstCompatibleProfileByCredential = new Map<string, string>(); const firstCompatibleProfileByCredential = new Map<string, string>();
for (const credential of scopedCredentials) { for (const credential of selectableCredentials) {
const selectedProfile = scopedProfiles.find((profile) => profile.credential_profile_id === credential.id); const selectedProfile = scopedProfiles.find((profile) => profile.credential_profile_id === credential.id);
const fallbackProfile = selectedProfile ?? scopedProfiles.find((profile) => !selectedCredentialIds.has(credential.id) && credentialMatchesProfile(credential, profile)); const fallbackProfile = selectedProfile ?? scopedProfiles.find((profile) => !selectedCredentialIds.has(credential.id) && credentialMatchesProfile(credential, profile));
if (fallbackProfile) firstCompatibleProfileByCredential.set(credential.id, fallbackProfile.id); if (fallbackProfile) firstCompatibleProfileByCredential.set(credential.id, fallbackProfile.id);
@@ -717,8 +800,8 @@ export default function FileConnectorSettingsPanel({
const credentialTestUnsupported = Boolean(credentialTestProfile && credentialTestProfile.provider !== "webdav" && credentialTestProfile.provider !== "nextcloud"); const credentialTestUnsupported = Boolean(credentialTestProfile && credentialTestProfile.provider !== "webdav" && credentialTestProfile.provider !== "nextcloud");
const credentialTestDisabled = !credentialDraft || saving || testingCredentialLogin || !credentialTestProfile || credentialTestUnsupported; const credentialTestDisabled = !credentialDraft || saving || testingCredentialLogin || !credentialTestProfile || credentialTestUnsupported;
const credentialTestTooltip = credentialTestHelpText(credentialTestProfile, credentialTestUnsupported); const credentialTestTooltip = credentialTestHelpText(credentialTestProfile, credentialTestUnsupported);
const profileCredentialOptions = draft ? credentialOptionsForProfileDraft(scopedCredentials, draft) : []; const profileCredentialOptions = draft ? credentialOptionsForProfileDraft(selectableCredentials, draft) : [];
const profileCredentialSelectValue = draft ? credentialProfileIdForDraft(draft, scopedCredentials) : ""; const profileCredentialSelectValue = draft ? credentialProfileIdForDraft(draft, selectableCredentials) : "";
const connectorColumns: ConnectionTreeColumn<ConnectorTreeRow>[] = [ const connectorColumns: ConnectionTreeColumn<ConnectorTreeRow>[] = [
{ {
id: "connection", id: "connection",
@@ -755,7 +838,7 @@ export default function FileConnectorSettingsPanel({
function connectorChildren(row: ConnectorTreeRow): ConnectorTreeRow[] { function connectorChildren(row: ConnectorTreeRow): ConnectorTreeRow[] {
if (row.kind !== "profile") return []; if (row.kind !== "profile") return [];
return scopedCredentials. return selectableCredentials.
filter((credential) => firstCompatibleProfileByCredential.get(credential.id) === row.profile.id). filter((credential) => firstCompatibleProfileByCredential.get(credential.id) === row.profile.id).
map((credential) => ({ kind: "credential" as const, id: `credential:${row.profile.id}:${credential.id}`, credential, profile: row.profile })); map((credential) => ({ kind: "credential" as const, id: `credential:${row.profile.id}:${credential.id}`, credential, profile: row.profile }));
} }
@@ -857,22 +940,70 @@ export default function FileConnectorSettingsPanel({
<> <>
<div className="form-grid two"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8"> <FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8">
<textarea value={policyDraft.allowedConnectors} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedConnectors: event.target.value })} rows={3} placeholder={"tenant-webdav\nseafile-*"} /> <ReferenceMultiSelect
values={lines(policyDraft.allowedConnectors)}
onChange={(values) => patchPolicyDraft({ allowedConnectors: values.join("\n") })}
provider={connectorReferenceProvider}
createCustomOption={(value) => wildcardReferenceOption(value)}
aria-label="Allowed file connections"
placeholder="Add a connection or wildcard pattern"
disabled={saving || !canWrite}
/>
</FormField> </FormField>
<FormField label="i18n:govoplan-files.denied_connection_ids.a95218b4"> <FormField label="i18n:govoplan-files.denied_connection_ids.a95218b4">
<textarea value={policyDraft.deniedConnectors} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ deniedConnectors: event.target.value })} rows={3} placeholder={"legacy-*\nblocked-smb"} /> <ReferenceMultiSelect
values={lines(policyDraft.deniedConnectors)}
onChange={(values) => patchPolicyDraft({ deniedConnectors: values.join("\n") })}
provider={connectorReferenceProvider}
createCustomOption={(value) => wildcardReferenceOption(value)}
aria-label="Denied file connections"
placeholder="Add a connection or wildcard pattern"
disabled={saving || !canWrite}
/>
</FormField> </FormField>
<FormField label="i18n:govoplan-files.allowed_credential_ids.efcaba2d"> <FormField label="i18n:govoplan-files.allowed_credential_ids.efcaba2d">
<textarea value={policyDraft.allowedCredentials} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedCredentials: event.target.value })} rows={3} placeholder={"tenant-webdav-credentials\nshared-*"} /> <ReferenceMultiSelect
values={lines(policyDraft.allowedCredentials)}
onChange={(values) => patchPolicyDraft({ allowedCredentials: values.join("\n") })}
provider={credentialReferenceProvider}
createCustomOption={(value) => wildcardReferenceOption(value)}
aria-label="Allowed file credentials"
placeholder="Add a credential or wildcard pattern"
disabled={saving || !canWrite}
/>
</FormField> </FormField>
<FormField label="i18n:govoplan-files.denied_credential_ids.b6838bc2"> <FormField label="i18n:govoplan-files.denied_credential_ids.b6838bc2">
<textarea value={policyDraft.deniedCredentials} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ deniedCredentials: event.target.value })} rows={3} placeholder={"personal-*\nblocked-*"} /> <ReferenceMultiSelect
values={lines(policyDraft.deniedCredentials)}
onChange={(values) => patchPolicyDraft({ deniedCredentials: values.join("\n") })}
provider={credentialReferenceProvider}
createCustomOption={(value) => wildcardReferenceOption(value)}
aria-label="Denied file credentials"
placeholder="Add a credential or wildcard pattern"
disabled={saving || !canWrite}
/>
</FormField> </FormField>
<FormField label="i18n:govoplan-files.allowed_providers.82a9c23e"> <FormField label="i18n:govoplan-files.allowed_providers.82a9c23e">
<textarea value={policyDraft.allowedProviders} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedProviders: event.target.value })} rows={3} placeholder={"webdav\nnextcloud\nsmb"} /> <ReferenceMultiSelect
values={lines(policyDraft.allowedProviders)}
onChange={(values) => patchPolicyDraft({ allowedProviders: values.join("\n") })}
provider={providerReferenceProvider}
createCustomOption={(value) => wildcardReferenceOption(value)}
aria-label="Allowed file providers"
placeholder="Add a provider or wildcard pattern"
disabled={saving || !canWrite}
/>
</FormField> </FormField>
<FormField label="i18n:govoplan-files.denied_providers.149b29f4"> <FormField label="i18n:govoplan-files.denied_providers.149b29f4">
<textarea value={policyDraft.deniedProviders} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ deniedProviders: event.target.value })} rows={3} placeholder={"generic\nnfs"} /> <ReferenceMultiSelect
values={lines(policyDraft.deniedProviders)}
onChange={(values) => patchPolicyDraft({ deniedProviders: values.join("\n") })}
provider={providerReferenceProvider}
createCustomOption={(value) => wildcardReferenceOption(value)}
aria-label="Denied file providers"
placeholder="Add a provider or wildcard pattern"
disabled={saving || !canWrite}
/>
</FormField> </FormField>
<FormField label="i18n:govoplan-files.allowed_paths.c953b4be"> <FormField label="i18n:govoplan-files.allowed_paths.c953b4be">
<textarea value={policyDraft.allowedPaths} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedPaths: event.target.value })} rows={3} placeholder={"GovOPlaN\nreports/*"} /> <textarea value={policyDraft.allowedPaths} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedPaths: event.target.value })} rows={3} placeholder={"GovOPlaN\nreports/*"} />
@@ -887,6 +1018,20 @@ export default function FileConnectorSettingsPanel({
<textarea value={policyDraft.deniedUrls} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ deniedUrls: event.target.value })} rows={3} placeholder={"http://*\n*://legacy.example.test/*"} /> <textarea value={policyDraft.deniedUrls} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ deniedUrls: event.target.value })} rows={3} placeholder={"http://*\n*://legacy.example.test/*"} />
</FormField> </FormField>
</div> </div>
{policyConflicts.length ? (
<DismissibleAlert
tone="warning"
resetKey={policyConflicts.join("\u0000")}
>
Allow and deny rules contain the same values for{" "}
{policyConflicts.join("; ")}. Deny rules take precedence.
</DismissibleAlert>
) : (
<p className="muted small-note">
When an allow and deny rule both match, the deny rule takes
precedence.
</p>
)}
<div className="file-connector-settings-section file-connector-policy-locks"> <div className="file-connector-settings-section file-connector-policy-locks">
<ToggleSwitch label="i18n:govoplan-files.lower_connection_limits.4f9838bc" checked={policyDraft.allowLowerConnectionLimits} disabled={saving || !canWrite} onChange={(allowLowerConnectionLimits) => patchPolicyDraft({ allowLowerConnectionLimits })} /> <ToggleSwitch label="i18n:govoplan-files.lower_connection_limits.4f9838bc" checked={policyDraft.allowLowerConnectionLimits} disabled={saving || !canWrite} onChange={(allowLowerConnectionLimits) => patchPolicyDraft({ allowLowerConnectionLimits })} />
<ToggleSwitch label="i18n:govoplan-files.lower_credential_limits.ec2b72bc" checked={policyDraft.allowLowerCredentialLimits} disabled={saving || !canWrite} onChange={(allowLowerCredentialLimits) => patchPolicyDraft({ allowLowerCredentialLimits })} /> <ToggleSwitch label="i18n:govoplan-files.lower_credential_limits.ec2b72bc" checked={policyDraft.allowLowerCredentialLimits} disabled={saving || !canWrite} onChange={(allowLowerCredentialLimits) => patchPolicyDraft({ allowLowerCredentialLimits })} />
@@ -1432,10 +1577,39 @@ function connectorCredentialDraftKey(draft: ConnectorCredentialDraft): string {
function credentialOptionsForProfileDraft(credentials: FileConnectorCredential[], draft: ConnectorProfileDraft): FileConnectorCredential[] { function credentialOptionsForProfileDraft(credentials: FileConnectorCredential[], draft: ConnectorProfileDraft): FileConnectorCredential[] {
return credentials.filter((credential) => return credentials.filter((credential) =>
credential.id === draft.credentialProfileId || credential.id === draft.credentialProfileId ||
credential.enabled && (!credential.provider || credential.provider === draft.provider) credential.enabled &&
(!credential.provider || credential.provider === draft.provider) &&
reusableCredentialAllowsProfile(credential, draft.id)
); );
} }
function reusableCredentialAvailableAtScope(
credential: FileConnectorCredential,
scopeType: ConnectorScope,
scopeId: string | null
): boolean {
if (connectorBelongsToEditableScope(credential, scopeType, scopeId)) return true;
if (credential.source_kind !== "credential_envelope") return false;
const inherited = credential.metadata?.inherit_to_lower_scopes === true;
if (!inherited) return false;
if (credential.scope_type === "system") return scopeType !== "system";
if (credential.scope_type === "tenant") {
return scopeType === "group" || scopeType === "user";
}
return false;
}
function reusableCredentialAllowsProfile(
credential: FileConnectorCredential,
profileId: string
): boolean {
if (credential.source_kind !== "credential_envelope") return true;
const refs = Array.isArray(credential.metadata?.allowed_server_refs)
? credential.metadata.allowed_server_refs.filter((value): value is string => typeof value === "string")
: [];
return refs.length === 0 || refs.includes(`files:${profileId}`);
}
function credentialProfileIdForDraft(draft: ConnectorProfileDraft, credentials: FileConnectorCredential[]): string { function credentialProfileIdForDraft(draft: ConnectorProfileDraft, credentials: FileConnectorCredential[]): string {
const options = credentialOptionsForProfileDraft(credentials, draft); const options = credentialOptionsForProfileDraft(credentials, draft);
if (draft.credentialProfileId && options.some((credential) => credential.id === draft.credentialProfileId)) { if (draft.credentialProfileId && options.some((credential) => credential.id === draft.credentialProfileId)) {
@@ -1656,6 +1830,26 @@ function lines(value: string): string[] {
return value.split(/\r?\n/).map((line) => line.trim()).filter(Boolean); return value.split(/\r?\n/).map((line) => line.trim()).filter(Boolean);
} }
function policyReferenceConflicts(
draft: CentralConnectorPolicyDraft
): string[] {
return [
["connections", draft.allowedConnectors, draft.deniedConnectors],
["credentials", draft.allowedCredentials, draft.deniedCredentials],
["providers", draft.allowedProviders, draft.deniedProviders],
["paths", draft.allowedPaths, draft.deniedPaths],
["endpoint URLs", draft.allowedUrls, draft.deniedUrls]
].flatMap(([label, allowed, denied]) => {
const deniedValues = new Set(
lines(denied).map((value) => value.toLocaleLowerCase())
);
const overlaps = lines(allowed).filter((value) =>
deniedValues.has(value.toLocaleLowerCase())
);
return overlaps.length ? [`${label}: ${overlaps.join(", ")}`] : [];
});
}
function policyRuleList(policy: Record<string, unknown>, kind: "allow" | "deny", field: "connectors" | "providers" | "credentials" | "external_paths" | "external_urls"): string[] { function policyRuleList(policy: Record<string, unknown>, kind: "allow" | "deny", field: "connectors" | "providers" | "credentials" | "external_paths" | "external_urls"): string[] {
const rule = recordValue(policy[kind]) ?? const rule = recordValue(policy[kind]) ??
recordValue(policy[kind === "allow" ? "allowlist" : "denylist"]) ?? recordValue(policy[kind === "allow" ? "allowlist" : "denylist"]) ??

View File

@@ -0,0 +1,98 @@
import { useCallback } from "react";
import { Folder, Network } from "lucide-react";
import { Link } from "react-router-dom";
import {
DashboardWidgetList,
DismissibleAlert,
LoadingFrame,
StatusBadge,
useDashboardWidgetData,
type ApiSettings,
type DashboardWidgetConfiguration
} from "@govoplan/core-webui";
import {
listFileSpaces,
type FileSpace
} from "../../api/files";
export default function FileSpacesWidget({
settings,
refreshKey,
configuration
}: {
settings: ApiSettings;
refreshKey: number;
configuration: DashboardWidgetConfiguration;
}) {
const maxItems = numberSetting(configuration.maxItems, 6, 1, 16);
const includeConnectors = configuration.includeConnectors !== false;
const load = useCallback(async () => {
const response = await listFileSpaces(settings);
return response.spaces
.filter(
(space) =>
includeConnectors || (space.space_type ?? "managed") === "managed"
)
.sort((left, right) => left.label.localeCompare(right.label))
.slice(0, maxItems);
}, [includeConnectors, maxItems, settings]);
const { data: spaces, loading, error } = useDashboardWidgetData(
load,
refreshKey
);
return (
<LoadingFrame loading={loading} label="Loading file spaces">
{error && (
<DismissibleAlert tone="warning" resetKey={error}>
{error}
</DismissibleAlert>
)}
<DashboardWidgetList
emptyText="No file spaces are available."
items={(spaces ?? []).map((space) => ({
id: space.id,
title: space.label,
detail: spaceDescription(space),
meta: space.owner_type === "group" ? "Group" : "Personal",
leading:
space.space_type === "connector" ? (
<Network size={17} aria-hidden="true" />
) : (
<Folder size={17} aria-hidden="true" />
),
trailing: space.read_only ? (
<StatusBadge status="locked" label="Read only" />
) : undefined,
to: "/files"
}))}
/>
<div className="dashboard-contribution-footer">
<Link className="btn btn-secondary" to="/files">
Open files
</Link>
</div>
</LoadingFrame>
);
}
function spaceDescription(space: FileSpace): string {
if (space.space_type !== "connector") {
return space.description || "Managed storage";
}
return [space.provider, space.library_id, space.remote_path]
.filter(Boolean)
.join(" · ") || "Connected storage";
}
function numberSetting(
value: unknown,
fallback: number,
minimum: number,
maximum: number
): number {
const numeric = typeof value === "number" ? value : Number(value);
return Number.isFinite(numeric)
? Math.max(minimum, Math.min(maximum, Math.floor(numeric)))
: fallback;
}

View File

@@ -1,5 +1,5 @@
import { useEffect, useMemo, useRef, useState, type DragEvent as ReactDragEvent, type KeyboardEvent as ReactKeyboardEvent, type MouseEvent as ReactMouseEvent } from "react"; import { useEffect, useMemo, useRef, useState, type DragEvent as ReactDragEvent, type KeyboardEvent as ReactKeyboardEvent, type MouseEvent as ReactMouseEvent } from "react";
import { ArrowUp, ChevronRight, Copy, Download, File, Folder, Home, KeyRound, Link2, MoveRight, Plus, RefreshCw, Search, Trash2, UploadCloud } from "lucide-react"; import { ArrowUp, ChevronRight, Copy, Download, File, Folder, Home, KeyRound, Link2, ListFilter, MoveRight, Plus, RefreshCw, Search, Share2, Trash2, UploadCloud } from "lucide-react";
import { import {
Button, Button,
ConfirmDialog, ConfirmDialog,
@@ -25,6 +25,7 @@ import {
downloadFilesAsZip, downloadFilesAsZip,
fetchResourceAccessExplanation, fetchResourceAccessExplanation,
listFilesDelta, listFilesDelta,
listFilesByProperties,
listFileConnectorProfiles, listFileConnectorProfiles,
listFileSpaces, listFileSpaces,
listManagedFileSnapshot, listManagedFileSnapshot,
@@ -38,6 +39,7 @@ import {
type FileConnectorBrowseItem, type FileConnectorBrowseItem,
type FileConnectorProfile, type FileConnectorProfile,
type FileDeltaResponse, type FileDeltaResponse,
type FileCampaignUsageFilter,
type FileFolder, type FileFolder,
type FileSpace, type FileSpace,
type ManagedFile, type ManagedFile,
@@ -46,6 +48,7 @@ import {
"../../api/files"; "../../api/files";
import { EMPTY_FILES, EMPTY_FOLDERS, EMPTY_SPACES, INTERNAL_DRAG_TYPE } from "./constants"; import { EMPTY_FILES, EMPTY_FOLDERS, EMPTY_SPACES, INTERNAL_DRAG_TYPE } from "./constants";
import { FileConflictDialog, FileContextMenu, FileDialog, FolderTree, RenamePreviewList, TransferFolderSelector } from "./components/FileManagerComponents"; import { FileConflictDialog, FileContextMenu, FileDialog, FolderTree, RenamePreviewList, TransferFolderSelector } from "./components/FileManagerComponents";
import FileShareDialog from "./components/FileShareDialog";
import type { import type {
ContextMenuState, ContextMenuState,
DialogKind, DialogKind,
@@ -88,6 +91,7 @@ import { useFileDialogs } from "./hooks/useFileDialogs";
import { useFileDragDropState } from "./hooks/useFileDragDropState"; import { useFileDragDropState } from "./hooks/useFileDragDropState";
type UploadPhase = "idle" | "uploading" | "unpacking" | "finalizing"; type UploadPhase = "idle" | "uploading" | "unpacking" | "finalizing";
type AuditRelevantFilter = "" | "true" | "false";
type FileAccessExplanationTarget = { type FileAccessExplanationTarget = {
resourceType: "file" | "folder"; resourceType: "file" | "folder";
resourceId: string; resourceId: string;
@@ -103,6 +107,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const canUpload = hasScope(auth, "files:upload"); const canUpload = hasScope(auth, "files:upload");
const canOrganize = hasScope(auth, "files:organize"); const canOrganize = hasScope(auth, "files:organize");
const canDelete = hasScope(auth, "files:delete"); const canDelete = hasScope(auth, "files:delete");
const canShare = hasScope(auth, "files:file:share");
const [spaces, setSpaces] = useState<FileSpace[]>(EMPTY_SPACES); const [spaces, setSpaces] = useState<FileSpace[]>(EMPTY_SPACES);
const [activeSpaceId, setActiveSpaceId] = useState(""); const [activeSpaceId, setActiveSpaceId] = useState("");
const activeSpace = spaces.find((space) => space.id === activeSpaceId) ?? spaces[0] ?? null; const activeSpace = spaces.find((space) => space.id === activeSpaceId) ?? spaces[0] ?? null;
@@ -118,6 +123,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const [searchCaseSensitive, setSearchCaseSensitive] = useState(false); const [searchCaseSensitive, setSearchCaseSensitive] = useState(false);
const [searchActive, setSearchActive] = useState(false); const [searchActive, setSearchActive] = useState(false);
const [searchResults, setSearchResults] = useState<ManagedFile[] | null>(null); const [searchResults, setSearchResults] = useState<ManagedFile[] | null>(null);
const [campaignUsageFilter, setCampaignUsageFilter] = useState<"" | FileCampaignUsageFilter>("");
const [auditRelevantFilter, setAuditRelevantFilter] = useState<AuditRelevantFilter>("");
const [propertyFiltersActive, setPropertyFiltersActive] = useState(false);
const [propertyFilterResults, setPropertyFilterResults] = useState<ManagedFile[] | null>(null);
const [propertyFilterTotal, setPropertyFilterTotal] = useState<number | null>(null);
const [sortColumn, setSortColumn] = useState<SortColumn>("name"); const [sortColumn, setSortColumn] = useState<SortColumn>("name");
const [sortDirection, setSortDirection] = useState<SortDirection>("asc"); const [sortDirection, setSortDirection] = useState<SortDirection>("asc");
const [unmatchedCount, setUnmatchedCount] = useState<number | null>(null); const [unmatchedCount, setUnmatchedCount] = useState<number | null>(null);
@@ -146,6 +156,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const [accessExplanationTarget, setAccessExplanationTarget] = useState<FileAccessExplanationTarget | null>(null); const [accessExplanationTarget, setAccessExplanationTarget] = useState<FileAccessExplanationTarget | null>(null);
const [resourceAccessExplanation, setResourceAccessExplanation] = useState<ResourceAccessExplanationResponse | null>(null); const [resourceAccessExplanation, setResourceAccessExplanation] = useState<ResourceAccessExplanationResponse | null>(null);
const [resourceAccessLoading, setResourceAccessLoading] = useState(false); const [resourceAccessLoading, setResourceAccessLoading] = useState(false);
const [shareDialogFile, setShareDialogFile] = useState<ManagedFile | null>(null);
const { setDragActive, internalDrag, setInternalDrag, dropTargetKey, setDropTargetKey, clearDropState: clearDragDropState } = useFileDragDropState(); const { setDragActive, internalDrag, setInternalDrag, dropTargetKey, setDropTargetKey, clearDropState: clearDragDropState } = useFileDragDropState();
const { const {
dialog, dialog,
@@ -176,11 +187,19 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const [renameSuffix, setRenameSuffix] = useState(""); const [renameSuffix, setRenameSuffix] = useState("");
const [renameRecursive, setRenameRecursive] = useState(false); const [renameRecursive, setRenameRecursive] = useState(false);
const [renamePreview, setRenamePreview] = useState<RenameResponse | null>(null); const [renamePreview, setRenamePreview] = useState<RenameResponse | null>(null);
const visibleFiles = searchActive && searchResults ? searchResults : files; const visibleFiles = useMemo(() => {
if (searchActive && searchResults && propertyFiltersActive && propertyFilterResults) {
const ids = new Set(propertyFilterResults.map((file) => file.id));
return searchResults.filter((file) => ids.has(file.id));
}
if (propertyFiltersActive && propertyFilterResults) return propertyFilterResults;
if (searchActive && searchResults) return searchResults;
return files;
}, [files, propertyFilterResults, propertyFiltersActive, searchActive, searchResults]);
const explorerEntries = useMemo(() => { const explorerEntries = useMemo(() => {
const entries = buildExplorerEntries(visibleFiles, folders, currentFolder, searchActive); const entries = buildExplorerEntries(visibleFiles, folders, currentFolder, searchActive || propertyFiltersActive);
return sortExplorerEntries(entries, sortColumn, sortDirection); return sortExplorerEntries(entries, sortColumn, sortDirection);
}, [visibleFiles, folders, currentFolder, searchActive, sortColumn, sortDirection]); }, [visibleFiles, folders, currentFolder, searchActive, propertyFiltersActive, sortColumn, sortDirection]);
const fileListViewportRef = useRef<HTMLDivElement | null>(null); const fileListViewportRef = useRef<HTMLDivElement | null>(null);
const fileListMeasureRowRef = useRef<HTMLDivElement | null>(null); const fileListMeasureRowRef = useRef<HTMLDivElement | null>(null);
const managedSpaceLoadsInFlightRef = useRef<Set<string>>(new Set()); const managedSpaceLoadsInFlightRef = useRef<Set<string>>(new Set());
@@ -229,6 +248,14 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
onOpenFolder: openFolder onOpenFolder: openFolder
}); });
const selectedFiles = useMemo(() => files.filter((file) => selectedFileIds.has(file.id)), [files, selectedFileIds]); const selectedFiles = useMemo(() => files.filter((file) => selectedFileIds.has(file.id)), [files, selectedFileIds]);
const shareManageableFile = useMemo(() => {
if (!canShare || selectedFiles.length !== 1 || selectedFolderPaths.size > 0) return null;
const file = selectedFiles[0];
const ownsFile = file.owner_type === "user"
? file.owner_id === auth.user.id
: auth.groups.some((group) => group.id === file.owner_id);
return ownsFile || hasScope(auth, "files:file:admin") ? file : null;
}, [auth, canShare, selectedFiles, selectedFolderPaths]);
const selectedDownloadFileIds = useMemo(() => fileIdsForSelection(files, selectedFileIds, selectedFolderPaths), [files, selectedFileIds, selectedFolderPaths]); const selectedDownloadFileIds = useMemo(() => fileIdsForSelection(files, selectedFileIds, selectedFolderPaths), [files, selectedFileIds, selectedFolderPaths]);
const accessExplainableTarget = useMemo( const accessExplainableTarget = useMemo(
() => accessTargetForSelection(selectedFileIds, selectedFolderPaths, activeSpaceId), () => accessTargetForSelection(selectedFileIds, selectedFolderPaths, activeSpaceId),
@@ -285,7 +312,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
if (!viewport) return; if (!viewport) return;
viewport.scrollTop = 0; viewport.scrollTop = 0;
setFileListScrollTop(0); setFileListScrollTop(0);
}, [activeSpaceId, currentFolder, searchActive, searchResults, sortColumn, sortDirection]); }, [activeSpaceId, currentFolder, propertyFiltersActive, propertyFilterResults, searchActive, searchResults, sortColumn, sortDirection]);
async function loadSpaces() { async function loadSpaces() {
setBusy(true); setBusy(true);
@@ -373,6 +400,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setSearchActive(false); setSearchActive(false);
setSearchPattern(""); setSearchPattern("");
setSearchResults(null); setSearchResults(null);
setCampaignUsageFilter("");
setAuditRelevantFilter("");
setPropertyFiltersActive(false);
setPropertyFilterResults(null);
setPropertyFilterTotal(null);
} }
} catch (err) { } catch (err) {
setError(err instanceof Error ? err.message : String(err)); setError(err instanceof Error ? err.message : String(err));
@@ -410,6 +442,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setSearchPattern(""); setSearchPattern("");
setSearchActive(false); setSearchActive(false);
setSearchResults(null); setSearchResults(null);
setCampaignUsageFilter("");
setAuditRelevantFilter("");
setPropertyFiltersActive(false);
setPropertyFilterResults(null);
setPropertyFilterTotal(null);
setSelectedFileIds(new Set()); setSelectedFileIds(new Set());
setSelectedFolderPaths(new Set()); setSelectedFolderPaths(new Set());
setConnectorSpaceSelectedItem(null); setConnectorSpaceSelectedItem(null);
@@ -431,6 +468,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setSearchActive(false); setSearchActive(false);
setSearchPattern(""); setSearchPattern("");
setSearchResults(null); setSearchResults(null);
setCampaignUsageFilter("");
setAuditRelevantFilter("");
setPropertyFiltersActive(false);
setPropertyFilterResults(null);
setPropertyFilterTotal(null);
} }
setUnmatchedCount(null); setUnmatchedCount(null);
setRenamePreview(null); setRenamePreview(null);
@@ -1001,6 +1043,50 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setSelectedFolderPaths(new Set()); setSelectedFolderPaths(new Set());
} }
async function runPropertyFilters(
campaignUsage: "" | FileCampaignUsageFilter,
auditRelevant: AuditRelevantFilter
) {
if (!activeSpace || activeSpaceIsConnector) return;
if (!campaignUsage && !auditRelevant) {
clearPropertyFilters();
return;
}
setBusy(true);
setError("");
setMessage("");
try {
const response = await listFilesByProperties(settings, {
owner_type: activeSpace.owner_type,
owner_id: activeSpace.owner_id,
path_prefix: currentFolder,
campaign_usage: campaignUsage || undefined,
audit_relevant: auditRelevant ? auditRelevant === "true" : undefined
});
setCampaignUsageFilter(campaignUsage);
setAuditRelevantFilter(auditRelevant);
setPropertyFilterResults(response.files);
setPropertyFilterTotal(response.total);
setPropertyFiltersActive(true);
setSelectedFileIds(new Set());
setSelectedFolderPaths(new Set());
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
} finally {
setBusy(false);
}
}
function clearPropertyFilters() {
setCampaignUsageFilter("");
setAuditRelevantFilter("");
setPropertyFiltersActive(false);
setPropertyFilterResults(null);
setPropertyFilterTotal(null);
setSelectedFileIds(new Set());
setSelectedFolderPaths(new Set());
}
function deleteSelected( function deleteSelected(
fileIds: Set<string> = selectedFileIds, fileIds: Set<string> = selectedFileIds,
folderPaths: Set<string> = selectedFolderPaths, folderPaths: Set<string> = selectedFolderPaths,
@@ -1821,6 +1907,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
<Link2 size={16} aria-hidden="true" /> i18n:govoplan-files.add_space.e4d674d4 <Link2 size={16} aria-hidden="true" /> i18n:govoplan-files.add_space.e4d674d4
</Button> </Button>
<Button onClick={() => void downloadSelection()} disabled={busy || activeSpaceIsConnector || !canDownload || selectedDownloadFileIds.length === 0}><Download size={16} aria-hidden="true" /> {downloadLabel}</Button> <Button onClick={() => void downloadSelection()} disabled={busy || activeSpaceIsConnector || !canDownload || selectedDownloadFileIds.length === 0}><Download size={16} aria-hidden="true" /> {downloadLabel}</Button>
<Button onClick={() => shareManageableFile && setShareDialogFile(shareManageableFile)} disabled={busy || activeSpaceIsConnector || !shareManageableFile}><Share2 size={16} aria-hidden="true" /> Manage shares</Button>
<Button onClick={() => openDialog("create-folder", toolbarTarget())} disabled={busy || !activeSpace || activeSpaceIsConnector || !canOrganize}><Plus size={16} aria-hidden="true" /> i18n:govoplan-files.create_folder.97bafaba</Button> <Button onClick={() => openDialog("create-folder", toolbarTarget())} disabled={busy || !activeSpace || activeSpaceIsConnector || !canOrganize}><Plus size={16} aria-hidden="true" /> i18n:govoplan-files.create_folder.97bafaba</Button>
<Button onClick={() => openTransferDialog("move")} disabled={busy || activeSpaceIsConnector || !canOrganize || !hasSelection}><MoveRight size={16} aria-hidden="true" /> i18n:govoplan-files.move.76cdb950</Button> <Button onClick={() => openTransferDialog("move")} disabled={busy || activeSpaceIsConnector || !canOrganize || !hasSelection}><MoveRight size={16} aria-hidden="true" /> i18n:govoplan-files.move.76cdb950</Button>
<Button onClick={() => openTransferDialog("copy")} disabled={busy || activeSpaceIsConnector || !canOrganize || !hasSelection}><Copy size={16} aria-hidden="true" /> i18n:govoplan-files.copy.af74f7c5</Button> <Button onClick={() => openTransferDialog("copy")} disabled={busy || activeSpaceIsConnector || !canOrganize || !hasSelection}><Copy size={16} aria-hidden="true" /> i18n:govoplan-files.copy.af74f7c5</Button>
@@ -2067,6 +2154,15 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
disabled={!activeSpace || activeSpaceIsConnector} disabled={!activeSpace || activeSpaceIsConnector}
onSearch={(patternValue, caseSensitiveValue) => void runPatternSearch(patternValue, caseSensitiveValue)} onSearch={(patternValue, caseSensitiveValue) => void runPatternSearch(patternValue, caseSensitiveValue)}
onClear={() => void clearSearch()} /> onClear={() => void clearSearch()} />
<FilePropertyFilterRow
campaignUsage={campaignUsageFilter}
auditRelevant={auditRelevantFilter}
busy={busy}
active={propertyFiltersActive}
disabled={!activeSpace || activeSpaceIsConnector}
total={propertyFilterTotal}
onApply={(campaignUsage, auditRelevant) => void runPropertyFilters(campaignUsage, auditRelevant)}
onClear={clearPropertyFilters} />
<div className="file-list-meta"> <div className="file-list-meta">
<span>{activeSpaceIsConnector ? connectorSpaceSelectedItem?.kind === "file" ? connectorSpaceSelectedItem.name : "i18n:govoplan-files.remote_connector_space.d8956863" : selectedSummary}</span> <span>{activeSpaceIsConnector ? connectorSpaceSelectedItem?.kind === "file" ? connectorSpaceSelectedItem.name : "i18n:govoplan-files.remote_connector_space.d8956863" : selectedSummary}</span>
@@ -2223,6 +2319,17 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
</FileDialog> </FileDialog>
} }
{shareDialogFile &&
<FileShareDialog
settings={settings}
file={shareDialogFile}
tenantId={auth.active_tenant?.id || auth.tenant.id}
onClose={() => setShareDialogFile(null)}
onChanged={() => {
if (activeSpace) void loadSpaceContents(activeSpace, { silent: true });
}} />
}
{dialog === "upload" && {dialog === "upload" &&
<FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}> <FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}>
<FileDropZone <FileDropZone
@@ -2509,6 +2616,72 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} }
function FilePropertyFilterRow({
campaignUsage,
auditRelevant,
busy,
active,
disabled,
total,
onApply,
onClear
}: {
campaignUsage: "" | FileCampaignUsageFilter;
auditRelevant: AuditRelevantFilter;
busy: boolean;
active: boolean;
disabled: boolean;
total: number | null;
onApply: (campaignUsage: "" | FileCampaignUsageFilter, auditRelevant: AuditRelevantFilter) => void;
onClear: () => void;
}) {
const [campaignUsageDraft, setCampaignUsageDraft] = useState(campaignUsage);
const [auditRelevantDraft, setAuditRelevantDraft] = useState(auditRelevant);
useEffect(() => {
setCampaignUsageDraft(campaignUsage);
}, [campaignUsage]);
useEffect(() => {
setAuditRelevantDraft(auditRelevant);
}, [auditRelevant]);
return (
<div className="file-property-filter-row">
<ListFilter size={17} aria-hidden="true" />
<label className="file-property-filter-field">
<span>Campaign use</span>
<select
value={campaignUsageDraft}
disabled={busy || disabled}
onChange={(event) => setCampaignUsageDraft(event.target.value as "" | FileCampaignUsageFilter)}>
<option value="">Any</option>
<option value="linked">Linked or used</option>
<option value="unlinked">Not linked or used</option>
</select>
</label>
<label className="file-property-filter-field">
<span>Audit evidence</span>
<select
value={auditRelevantDraft}
disabled={busy || disabled}
onChange={(event) => setAuditRelevantDraft(event.target.value as AuditRelevantFilter)}>
<option value="">Any</option>
<option value="true">Audit relevant</option>
<option value="false">Not audit relevant</option>
</select>
</label>
<Button
onClick={() => onApply(campaignUsageDraft, auditRelevantDraft)}
disabled={busy || disabled || !campaignUsageDraft && !auditRelevantDraft}>
Apply filters
</Button>
{active && <Button onClick={onClear} disabled={busy}>Clear filters</Button>}
{active && total !== null && <span className="file-property-filter-count">{total} matching file{total === 1 ? "" : "s"}</span>}
</div>
);
}
function FileSearchRow({ function FileSearchRow({
value, value,
caseSensitive, caseSensitive,
@@ -2559,7 +2732,15 @@ function FileSearchRow({
} }
function activeCampaignShares(file: ManagedFile) { function activeCampaignShares(file: ManagedFile) {
return (file.shares ?? []).filter((share) => share.target_type === "campaign" && !share.revoked_at); return (file.shares ?? []).filter((share) =>
share.target_type === "campaign" && fileShareIsActive(share)
);
}
function fileShareIsActive(share: NonNullable<ManagedFile["shares"]>[number]): boolean {
if (typeof share.active === "boolean") return share.active;
return !share.revoked_at
&& (!share.expires_at || new Date(share.expires_at).getTime() > Date.now());
} }
function activeCampaignShareCount(file: ManagedFile): number { function activeCampaignShareCount(file: ManagedFile): number {

View File

@@ -0,0 +1,300 @@
import { useEffect, useMemo, useState } from "react";
import { Trash2 } from "lucide-react";
import {
Button,
ConfirmDialog,
DataGrid,
DismissibleAlert,
FormField,
ReferenceSelect,
StatusBadge,
TableActionGroup,
type ApiSettings,
type DataGridColumn
} from "@govoplan/core-webui";
import {
createFileShare,
fileShareTargetProvider,
listFileShares,
revokeFileShare,
type FileShare,
type ManagedFile
} from "../../../api/files";
import { FileDialog } from "./FileManagerComponents";
import { formatDate } from "../utils/fileManagerUtils";
type ShareTargetType = "user" | "group" | "tenant";
type SharePermission = "read" | "write" | "manage";
export default function FileShareDialog({
settings,
file,
tenantId,
onClose,
onChanged
}: {
settings: ApiSettings;
file: ManagedFile;
tenantId: string;
onClose: () => void;
onChanged: () => void;
}) {
const [shares, setShares] = useState<FileShare[]>([]);
const [targetType, setTargetType] = useState<ShareTargetType>("user");
const [targetId, setTargetId] = useState("");
const [permission, setPermission] = useState<SharePermission>("read");
const [expiresAt, setExpiresAt] = useState("");
const [busy, setBusy] = useState(false);
const [error, setError] = useState("");
const [revokeTarget, setRevokeTarget] = useState<FileShare | null>(null);
const targetProvider = useMemo(
() => targetType === "tenant"
? null
: fileShareTargetProvider(settings, file.id, targetType),
[file.id, settings, targetType]
);
useEffect(() => {
void refresh();
}, [file.id]);
async function refresh() {
setBusy(true);
setError("");
try {
const response = await listFileShares(settings, file.id, {
includeInactive: true
});
setShares(response.shares);
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
} finally {
setBusy(false);
}
}
async function grant() {
const effectiveTargetId = targetType === "tenant" ? tenantId : targetId;
if (!effectiveTargetId) return;
setBusy(true);
setError("");
try {
await createFileShare(settings, file.id, {
target_type: targetType,
target_id: effectiveTargetId,
permission,
expires_at: expiresAt ? new Date(expiresAt).toISOString() : null
});
setTargetId("");
setExpiresAt("");
await refresh();
onChanged();
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
setBusy(false);
}
}
async function revoke() {
if (!revokeTarget) return;
setBusy(true);
setError("");
try {
await revokeFileShare(settings, file.id, revokeTarget.id);
setRevokeTarget(null);
await refresh();
onChanged();
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
setBusy(false);
}
}
const columns = useMemo<DataGridColumn<FileShare>[]>(() => [
{
id: "target",
header: "Target",
minWidth: 220,
resizable: true,
sortable: true,
filterable: true,
value: (share) => `${share.target_type}: ${share.target_id}`
},
{
id: "permission",
header: "Permission",
width: 115,
sortable: true,
filterable: true,
value: (share) => share.permission
},
{
id: "created",
header: "Created",
minWidth: 160,
resizable: true,
sortable: true,
value: (share) => formatDate(share.created_at)
},
{
id: "creator",
header: "Created by",
minWidth: 150,
resizable: true,
filterable: true,
value: (share) => share.created_by_user_id || "System"
},
{
id: "expiry",
header: "Expires",
minWidth: 160,
resizable: true,
sortable: true,
value: (share) => share.expires_at ? formatDate(share.expires_at) : "No expiry"
},
{
id: "status",
header: "Status",
width: 115,
sortable: true,
value: (share) => shareStatus(share),
render: (share) => (
<StatusBadge
status={share.active ? "active" : "inactive"}
label={shareStatus(share)}
/>
)
},
{
id: "actions",
header: "Actions",
width: 72,
sticky: "end",
align: "right",
render: (share) => (
<TableActionGroup
minimumSlots={1}
actions={[
{
id: "revoke",
label: "Revoke share",
icon: <Trash2 size={16} aria-hidden="true" />,
variant: "danger",
disabled: !share.active || busy,
disabledReason: !share.active ? "This share is already inactive." : undefined,
onClick: () => setRevokeTarget(share)
}
]}
/>
)
}
], [busy]);
return (
<>
<FileDialog title={`Manage shares - ${file.filename}`} onClose={() => !busy && onClose()}>
<div className="file-share-dialog-content">
{error ? (
<DismissibleAlert tone="danger" resetKey={error}>
{error}
</DismissibleAlert>
) : null}
<div className="file-share-form">
<FormField label="Target type">
<select
value={targetType}
disabled={busy}
onChange={(event) => {
setTargetType(event.target.value as ShareTargetType);
setTargetId("");
}}
>
<option value="user">User</option>
<option value="group">Group</option>
<option value="tenant">Entire tenant</option>
</select>
</FormField>
<FormField label="Target">
{targetType === "tenant" ? (
<input value={tenantId} disabled readOnly />
) : targetProvider ? (
<ReferenceSelect
value={targetId}
onChange={setTargetId}
provider={targetProvider}
disabled={busy}
placeholder={`Select a ${targetType}`}
aria-label={`Share target ${targetType}`}
required
/>
) : null}
</FormField>
<FormField label="Permission">
<select
value={permission}
disabled={busy}
onChange={(event) => setPermission(event.target.value as SharePermission)}
>
<option value="read">Read</option>
<option value="write">Write</option>
<option value="manage">Manage</option>
</select>
</FormField>
<FormField label="Expires (optional)">
<input
type="datetime-local"
value={expiresAt}
min={minimumLocalDateTime()}
disabled={busy}
onChange={(event) => setExpiresAt(event.target.value)}
/>
</FormField>
<Button
variant="primary"
disabled={busy || (targetType !== "tenant" && !targetId)}
onClick={() => void grant()}
>
Grant access
</Button>
</div>
<DataGrid
id={`file-${file.id}-shares`}
rows={shares}
columns={columns}
getRowKey={(share) => share.id}
emptyText={busy ? "Loading shares..." : "This file has no direct shares."}
initialFit="container"
/>
<div className="button-row compact-actions align-end">
<Button onClick={onClose} disabled={busy}>Close</Button>
</div>
</div>
</FileDialog>
<ConfirmDialog
open={Boolean(revokeTarget)}
title="Revoke file share"
message="Revoke this direct grant? Other independent grants remain in effect."
confirmLabel="Revoke"
cancelLabel="Cancel"
tone="danger"
busy={busy}
onConfirm={() => void revoke()}
onCancel={() => setRevokeTarget(null)}
/>
</>
);
}
function shareStatus(share: FileShare): string {
if (share.revoked_at) return "Revoked";
if (share.expires_at && new Date(share.expires_at).getTime() <= Date.now()) {
return "Expired";
}
return "Active";
}
function minimumLocalDateTime(): string {
const now = new Date(Date.now() + 60_000);
const local = new Date(now.getTime() - now.getTimezoneOffset() * 60_000);
return local.toISOString().slice(0, 16);
}

View File

@@ -1024,7 +1024,14 @@ renderPatternPreview?: FilesManagedFileChooserProps["renderPatternPreview"])
function isLinkedToTarget(file: ManagedFile, target?: FilesManagedFileLinkTarget): boolean { function isLinkedToTarget(file: ManagedFile, target?: FilesManagedFileLinkTarget): boolean {
if (!target) return false; if (!target) return false;
return Boolean(file.shares?.some((share) => share.target_type === target.type && share.target_id === target.id && !share.revoked_at)); return Boolean(file.shares?.some((share) =>
share.target_type === target.type
&& share.target_id === target.id
&& (typeof share.active === "boolean"
? share.active
: !share.revoked_at
&& (!share.expires_at || new Date(share.expires_at).getTime() > Date.now()))
));
} }
function readRememberedState(key: string): RememberedChooserState { function readRememberedState(key: string): RememberedChooserState {

View File

@@ -1,8 +1,16 @@
import { createElement, lazy } from "react"; import { createElement, lazy } from "react";
import { hasScope, type AdminSectionsUiCapability, type FilesConnectorsUiCapability, type FilesFileExplorerUiCapability, type PlatformWebModule } from "@govoplan/core-webui"; import {
hasScope,
type AdminSectionsUiCapability,
type DashboardWidgetsUiCapability,
type FilesConnectorsUiCapability,
type FilesFileExplorerUiCapability,
type PlatformWebModule
} from "@govoplan/core-webui";
import { FolderTree } from "./features/files/components/FileManagerComponents"; import { FolderTree } from "./features/files/components/FileManagerComponents";
import FileConnectorSettingsPanel from "./features/files/FileConnectorSettingsPanel"; import FileConnectorSettingsPanel from "./features/files/FileConnectorSettingsPanel";
import ManagedFileChooser from "./features/files/components/ManagedFileChooser"; import ManagedFileChooser from "./features/files/components/ManagedFileChooser";
import FileSpacesWidget from "./features/files/FileSpacesWidget";
import { listFileSpaces, listFiles, listFilesDelta, listFolders, resolveFilePatterns, shareFilesWithTarget } from "./api/files"; import { listFileSpaces, listFiles, listFilesDelta, listFolders, resolveFilePatterns, shareFilesWithTarget } from "./api/files";
import { generatedTranslations } from "./i18n/generatedTranslations"; import { generatedTranslations } from "./i18n/generatedTranslations";
import "./styles/file-manager.css"; import "./styles/file-manager.css";
@@ -14,10 +22,55 @@ const translations = {
en: generatedTranslations.en, en: generatedTranslations.en,
de: generatedTranslations.de de: generatedTranslations.de
}; };
const fileDashboardWidgets: DashboardWidgetsUiCapability = {
widgets: [
{
id: "files.spaces",
surfaceId: "files.widget.spaces",
title: "File spaces",
description: "Managed and connected file spaces available to you.",
moduleId: "files",
category: "Content",
order: 35,
defaultVisible: false,
defaultSize: "medium",
supportedSizes: ["medium", "wide"],
anyOf: fileRead,
refreshIntervalMs: 60_000,
defaultConfiguration: {
maxItems: 6,
includeConnectors: true
},
configurationFields: [
{
id: "maxItems",
label: "Maximum spaces",
kind: "number",
min: 1,
max: 16,
step: 1,
required: true
},
{
id: "includeConnectors",
label: "Include connected storage",
kind: "boolean"
}
],
render: ({ settings, refreshKey, configuration }) =>
createElement(FileSpacesWidget, {
settings,
refreshKey,
configuration
})
}
]
};
const fileConnectorAdminSections: AdminSectionsUiCapability = { const fileConnectorAdminSections: AdminSectionsUiCapability = {
sections: [ sections: [
{ {
id: "system-file-connectors", id: "system-file-connectors",
surfaceId: "files.admin.system-connectors",
label: "i18n:govoplan-files.file_connections.1e362326", label: "i18n:govoplan-files.file_connections.1e362326",
group: "SYSTEM", group: "SYSTEM",
order: 75, order: 75,
@@ -30,6 +83,7 @@ const fileConnectorAdminSections: AdminSectionsUiCapability = {
}, },
{ {
id: "tenant-file-connectors", id: "tenant-file-connectors",
surfaceId: "files.admin.tenant-connectors",
label: "i18n:govoplan-files.file_connections.1e362326", label: "i18n:govoplan-files.file_connections.1e362326",
group: "TENANT", group: "TENANT",
order: 65, order: 65,
@@ -49,6 +103,14 @@ export const filesModule: PlatformWebModule = {
version: "1.0.0", version: "1.0.0",
dependencies: ["access"], dependencies: ["access"],
translations, translations,
viewSurfaces: [
{ id: "files.admin.system-connectors", moduleId: "files", kind: "section", label: "System file connections", order: 75 },
{ id: "files.admin.tenant-connectors", moduleId: "files", kind: "section", label: "Tenant file connections", order: 65 },
{ id: "files.admin.group-connectors", moduleId: "files", kind: "section", label: "Group file connections", order: 65 },
{ id: "files.admin.user-connectors", moduleId: "files", kind: "section", label: "User file connections", order: 65 },
{ id: "files.settings.connectors", moduleId: "files", kind: "section", label: "Personal file connections", order: 20 },
{ id: "files.widget.spaces", moduleId: "files", kind: "section", label: "File spaces widget", order: 35 }
],
navItems: [{ to: "/files", label: "i18n:govoplan-files.files.6ce6c512", iconName: "folder", anyOf: fileRead, order: 40 }], navItems: [{ to: "/files", label: "i18n:govoplan-files.files.6ce6c512", iconName: "folder", anyOf: fileRead, order: 40 }],
routes: [ routes: [
{ path: "/files", anyOf: fileRead, order: 40, render: ({ settings, auth }) => createElement(FilesPage, { settings, auth }) }], { path: "/files", anyOf: fileRead, order: 40, render: ({ settings, auth }) => createElement(FilesPage, { settings, auth }) }],
@@ -67,7 +129,8 @@ export const filesModule: PlatformWebModule = {
"files.connectors": { "files.connectors": {
FileConnectorScopeManager: FileConnectorSettingsPanel FileConnectorScopeManager: FileConnectorSettingsPanel
} satisfies FilesConnectorsUiCapability, } satisfies FilesConnectorsUiCapability,
"admin.sections": fileConnectorAdminSections "admin.sections": fileConnectorAdminSections,
"dashboard.widgets": fileDashboardWidgets
} }
}; };

View File

@@ -11,6 +11,31 @@
padding: 4px 0 10px 14px; padding: 4px 0 10px 14px;
} }
.file-property-filter-row {
display: flex;
align-items: end;
gap: 12px;
min-width: 0;
padding: 0 0 10px 14px;
}
.file-property-filter-field {
display: grid;
gap: 4px;
min-width: 180px;
}
.file-property-filter-field > span,
.file-property-filter-count {
color: var(--muted);
font-size: var(--font-size-sm);
}
.file-property-filter-count {
align-self: center;
margin-left: auto;
}
.file-list-drop-target.is-active, .file-list-drop-target.is-active,
.file-list-drop-target.is-drop-target { .file-list-drop-target.is-drop-target {
background: var(--line); background: var(--line);
@@ -263,6 +288,35 @@
width: min(820px, 100%); width: min(820px, 100%);
} }
.file-dialog:has(.file-share-dialog-content) {
width: min(1080px, 100%);
}
.file-share-dialog-content {
display: grid;
min-width: 0;
gap: 16px;
}
.file-share-form {
display: grid;
grid-template-columns: minmax(130px, .7fr) minmax(220px, 1.4fr) minmax(120px, .7fr) minmax(190px, 1fr) auto;
gap: 12px;
align-items: end;
}
@media (max-width: 900px) {
.file-share-form {
grid-template-columns: minmax(0, 1fr) minmax(0, 1fr);
}
}
@media (max-width: 620px) {
.file-share-form {
grid-template-columns: minmax(0, 1fr);
}
}
.connector-sync-grid { .connector-sync-grid {
display: grid; display: grid;
grid-template-columns: minmax(0, .9fr) minmax(0, 1.1fr); grid-template-columns: minmax(0, .9fr) minmax(0, 1.1fr);
@@ -565,6 +619,15 @@
.file-search-row { .file-search-row {
grid-template-columns: 1fr; grid-template-columns: 1fr;
} }
.file-property-filter-row {
align-items: stretch;
flex-direction: column;
}
.file-property-filter-count {
margin-left: 0;
}
} }
.file-conflict-summary { .file-conflict-summary {