Compare commits
14 Commits
cffe161f29
...
v0.1.9
| Author | SHA1 | Date | |
|---|---|---|---|
| 2b34f6e305 | |||
| 58af1a20a7 | |||
| 4722161592 | |||
| 1444ba80a0 | |||
| 02ef83ecee | |||
| 1069f85796 | |||
| 1401c78c8a | |||
| b6109245a7 | |||
| f5d40b23c2 | |||
| 94ea629635 | |||
| 6c8a8c655d | |||
| 062ad5ddfb | |||
| 9adfa91e74 | |||
| 65d8ed80b5 |
18
README.md
18
README.md
@@ -81,8 +81,14 @@ Profiles can be supplied as JSON through
|
|||||||
deployment-owned credential references such as `password_env`, `token_env`, or
|
deployment-owned credential references such as `password_env`, `token_env`, or
|
||||||
`secret_ref`. Environment references require an exact name in the deployment-wide
|
`secret_ref`. Environment references require an exact name in the deployment-wide
|
||||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
|
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
|
||||||
process environment variables and use encrypted stored credentials or scoped
|
process environment variables and may use only Files-owned encrypted password or
|
||||||
secret-provider references instead.
|
token values. API-created `secret_ref` values fail closed until Files has an
|
||||||
|
ownership contract that can confirm provider-side deletion. Legacy external
|
||||||
|
references are treated as non-owned: deleting a profile or credential detaches
|
||||||
|
and audits the reference but never passes it to an arbitrary secret provider.
|
||||||
|
Profile and credential deletion immediately clears encrypted values, credential
|
||||||
|
identities, deployment references, and private metadata in the same transaction
|
||||||
|
as the non-secret audit record; an audit failure rolls the deletion back.
|
||||||
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
|
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
|
||||||
current principal (system, tenant, user, group, or accessible campaign scope) and
|
current principal (system, tenant, user, group, or accessible campaign scope) and
|
||||||
redacts secret values and environment variable names. Use the returned
|
redacts secret values and environment variable names. Use the returned
|
||||||
@@ -119,12 +125,20 @@ GovOPlaN pinned HTTP transport and may manage redirects itself, so neither an
|
|||||||
explicit endpoint nor SDK endpoint discovery is allowed until both peer pinning
|
explicit endpoint nor SDK endpoint discovery is allowed until both peer pinning
|
||||||
and redirect revalidation are enforced.
|
and redirect revalidation are enforced.
|
||||||
|
|
||||||
|
Destructive Files-module retirement applies the same credential lifecycle before
|
||||||
|
dropping tables. Every remaining Files-owned encrypted connector secret is
|
||||||
|
scrubbed and audited first, while legacy non-owned external references are
|
||||||
|
detached and identified as such in the audit record. Retirement does not claim
|
||||||
|
or attempt provider-side deletion for references Files cannot prove it owns.
|
||||||
|
|
||||||
Local connector development assets live in `dev/connectors/`. The compose stack
|
Local connector development assets live in `dev/connectors/`. The compose stack
|
||||||
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
|
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
|
||||||
manual interoperability testing.
|
manual interoperability testing.
|
||||||
|
|
||||||
Connector and collaboration ownership boundaries are documented in
|
Connector and collaboration ownership boundaries are documented in
|
||||||
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
|
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
|
||||||
|
The role-adaptive user, administration, integration, and operator guide is the
|
||||||
|
[Files handbook](docs/FILES_HANDBOOK.md).
|
||||||
|
|
||||||
ZIP uploads are processed without buffering the whole archive in memory. The API
|
ZIP uploads are processed without buffering the whole archive in memory. The API
|
||||||
spools incoming ZIP request bodies to a bounded temporary file, then extracts
|
spools incoming ZIP request bodies to a bounded temporary file, then extracts
|
||||||
|
|||||||
782
docs/FILES_HANDBOOK.md
Normal file
782
docs/FILES_HANDBOOK.md
Normal file
@@ -0,0 +1,782 @@
|
|||||||
|
# GovOPlaN Files Handbook
|
||||||
|
|
||||||
|
This handbook describes the Files module as implemented in version `0.1.9`.
|
||||||
|
It is the operational source of truth for users, process owners, administrators,
|
||||||
|
operators, auditors, and module integrators. Statements about future behavior
|
||||||
|
are marked **planned**; an unmarked statement describes the current code.
|
||||||
|
|
||||||
|
Files is a governed snapshot store. It owns managed file content, versions,
|
||||||
|
logical folders, shares, source provenance, and the evidence that another
|
||||||
|
GovOPlaN module used a particular file version. It can browse selected external
|
||||||
|
stores read-only and import a frozen copy. It is not a general remote filesystem,
|
||||||
|
a document collaboration engine, or a records-management system.
|
||||||
|
|
||||||
|
## Choose a reading path
|
||||||
|
|
||||||
|
| If you need to... | Start with... |
|
||||||
|
| --- | --- |
|
||||||
|
| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) |
|
||||||
|
| Design a governed process that uses files | [Process perspective](#process-perspective) |
|
||||||
|
| Decide who can use a file or connector | [Ownership and access](#ownership-and-access) and [Administration and policy](#administration-and-policy) |
|
||||||
|
| Operate storage, connectors, backups, or recovery | [Operator runbook](#operator-runbook) |
|
||||||
|
| Integrate another GovOPlaN module | [Capabilities and integration](#capabilities-and-integration) |
|
||||||
|
| Review evidence, deletion, or security behavior | [Security, provenance, audit, deletion, and retention](#security-provenance-audit-deletion-and-retention) |
|
||||||
|
| Verify a release or scenario | [Acceptance scenarios](#acceptance-scenarios) |
|
||||||
|
| Check whether an idea exists today | [Implemented and planned boundary](#implemented-and-planned-boundary) |
|
||||||
|
|
||||||
|
## The service contract
|
||||||
|
|
||||||
|
A managed file is a tenant-scoped logical asset with exactly one user or group
|
||||||
|
owner, a normalized path, and a current version. The current version points to a
|
||||||
|
blob record containing the storage location, SHA-256 checksum, byte size, and
|
||||||
|
content type. Current service paths append a version when connector sync finds
|
||||||
|
changed content; they do not mutate the previous version record.
|
||||||
|
|
||||||
|
Content with the same tenant, SHA-256 checksum, and size can reuse one blob.
|
||||||
|
This is storage deduplication and integrity evidence, not proof of authorship or
|
||||||
|
source authenticity.
|
||||||
|
|
||||||
|
The main domain objects are:
|
||||||
|
|
||||||
|
| Object | Meaning | Lifecycle today |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted |
|
||||||
|
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained |
|
||||||
|
| File blob | Stored bytes plus checksum, size, backend, key, reference count, and optional retention timestamp | Reused within a tenant; no automated garbage collection |
|
||||||
|
| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted |
|
||||||
|
| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet |
|
||||||
|
| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion |
|
||||||
|
| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion |
|
||||||
|
| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use |
|
||||||
|
| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted |
|
||||||
|
| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence |
|
||||||
|
|
||||||
|
## User tasks
|
||||||
|
|
||||||
|
The Files page is available at `/files`. Actions appear only when the current
|
||||||
|
principal has the required permission and resource access.
|
||||||
|
|
||||||
|
The configured Help Center projects these sections as independently authorized
|
||||||
|
tasks, so upload, ZIP import, organization, download, sharing, and deletion do
|
||||||
|
not disappear merely because an unrelated permission is absent. It states the
|
||||||
|
deployment's actual upload limits. External import appears as a task only when
|
||||||
|
the actor has the required permissions and at least one actor-visible profile
|
||||||
|
is eligible for the current fail-closed browse/import path; endpoint, path, and
|
||||||
|
item policy are still enforced when the operation runs.
|
||||||
|
|
||||||
|
### Choose a space
|
||||||
|
|
||||||
|
Every file user sees **My files**. Group spaces are added for active groups of
|
||||||
|
which the user is a member. The space list also exposes every tenant group to a
|
||||||
|
Files administrator; governed API operations can administer other tenant-owned
|
||||||
|
Files resources when their owner is specified. Linked connector spaces appear
|
||||||
|
beside managed spaces when they are active and visible to the user.
|
||||||
|
|
||||||
|
A connector space is intentionally read-only. It is a view of an approved
|
||||||
|
remote location and a starting point for importing or synchronizing selected
|
||||||
|
files into managed storage; it is not a mounted write-through filesystem.
|
||||||
|
|
||||||
|
### Upload files
|
||||||
|
|
||||||
|
The managed-space UI supports upload and drag-and-drop. A caller chooses the
|
||||||
|
user or group owner and destination folder. The normal per-file deployment
|
||||||
|
limit defaults to 50 MiB through `FILE_UPLOAD_MAX_BYTES`.
|
||||||
|
|
||||||
|
When a target path already exists, the operation must use one of these conflict
|
||||||
|
strategies:
|
||||||
|
|
||||||
|
- `reject` stops instead of silently replacing content;
|
||||||
|
- `rename` selects the next available `copy` name;
|
||||||
|
- `overwrite` soft-deletes the conflicting asset and creates the new asset;
|
||||||
|
- an item-specific conflict resolution can also `skip` that item.
|
||||||
|
|
||||||
|
An ordinary upload does not append a new version to an existing asset. The
|
||||||
|
`overwrite` strategy retires the old asset at that path. Version-preserving
|
||||||
|
updates currently belong to connector sync.
|
||||||
|
|
||||||
|
### Upload and unpack ZIP files
|
||||||
|
|
||||||
|
The UI can unpack a ZIP upload. The request body is spooled to a bounded
|
||||||
|
temporary file rather than buffered wholly in memory. The defaults are:
|
||||||
|
|
||||||
|
- 250 MiB compressed request and total extracted data;
|
||||||
|
- 50 MiB per extracted member;
|
||||||
|
- 1,000 non-directory members;
|
||||||
|
- encrypted archives are rejected;
|
||||||
|
- member paths are normalized and `..` traversal is rejected;
|
||||||
|
- the actual bytes read are counted, not only the ZIP header declarations.
|
||||||
|
|
||||||
|
The byte limits use `FILE_UPLOAD_ZIP_MAX_BYTES` and
|
||||||
|
`FILE_UPLOAD_MAX_BYTES`. The 1,000-member limit is currently fixed in the
|
||||||
|
extraction service.
|
||||||
|
|
||||||
|
### Organize files and folders
|
||||||
|
|
||||||
|
With `files:file:organize`, a user can:
|
||||||
|
|
||||||
|
- create logical folders;
|
||||||
|
- rename one selection directly;
|
||||||
|
- preview and apply bulk prefix, suffix, or replacement renames;
|
||||||
|
- move or copy files and folder trees between accessible user/group spaces;
|
||||||
|
- resolve target conflicts by rejecting, renaming, overwriting, or skipping;
|
||||||
|
- use drag-and-drop for move operations in the file explorer.
|
||||||
|
|
||||||
|
Copies create new assets and versions while reusing the immutable blob bytes.
|
||||||
|
Moves keep the asset identity and change its owner/path. All source and target
|
||||||
|
owners are validated; knowing an identifier does not bypass space membership.
|
||||||
|
|
||||||
|
Recursive folder deletion is the default. It soft-deletes the selected folder,
|
||||||
|
its child folders, and files below it. A non-recursive delete fails when the
|
||||||
|
folder is not empty.
|
||||||
|
|
||||||
|
### Find and download files
|
||||||
|
|
||||||
|
Files can list by owner and path, use cursor pagination, and consume incremental
|
||||||
|
changes through a watermark. The UI supports path/name pattern search and
|
||||||
|
sorting. The pattern API can resolve campaign-style wildcard selections and can
|
||||||
|
return unmatched files.
|
||||||
|
|
||||||
|
A user with download permission and resource access can download one current
|
||||||
|
version or create a ZIP archive from a selection. Downloads use an attachment
|
||||||
|
content disposition. The archive is generated in a temporary file and removed
|
||||||
|
after the response completes.
|
||||||
|
|
||||||
|
There is no dedicated content-preview service in the Files API today. File
|
||||||
|
responses expose content type, size, checksum, and current version metadata.
|
||||||
|
|
||||||
|
### Share files
|
||||||
|
|
||||||
|
The API can grant or update a share for a user, group, the tenant, or a campaign.
|
||||||
|
The supported permissions are `read`, `write`, and `manage`. Ownership remains
|
||||||
|
unchanged. A write operation accepts a `write` or `manage` share; read/download
|
||||||
|
accepts any of the three.
|
||||||
|
|
||||||
|
The current Files page shows campaign linkage but does not offer a general
|
||||||
|
user/group share editor. The API also has no share-revocation route yet. Treat
|
||||||
|
share revocation and a complete share-management UI as planned work.
|
||||||
|
|
||||||
|
### Browse and import an external file
|
||||||
|
|
||||||
|
With a visible connector profile, a user can browse the allowed remote path,
|
||||||
|
select one file, and import it into an accessible managed space. The imported
|
||||||
|
asset records the connector, provider, remote identity/path/URL, selected remote
|
||||||
|
metadata, and source revision when supplied by the provider.
|
||||||
|
|
||||||
|
Manual sync looks for an existing managed asset with the same source identity
|
||||||
|
inside the chosen owner space:
|
||||||
|
|
||||||
|
- no match creates a managed asset;
|
||||||
|
- identical checksum and size updates provenance and returns `unchanged`;
|
||||||
|
- changed bytes append a version and return `updated`.
|
||||||
|
|
||||||
|
Browse, import, and sync never write, rename, or delete the remote source.
|
||||||
|
|
||||||
|
## Process perspective
|
||||||
|
|
||||||
|
### Managed ingestion
|
||||||
|
|
||||||
|
The managed-file flow is:
|
||||||
|
|
||||||
|
1. Core authenticates the principal and evaluates the operation permission.
|
||||||
|
2. Files validates the tenant, owner, group membership, or applicable share.
|
||||||
|
3. Files normalizes the logical path and resolves conflicts explicitly.
|
||||||
|
4. Upload or connector response limits are enforced before content is retained.
|
||||||
|
5. Files calculates SHA-256 and stores or reuses a tenant blob.
|
||||||
|
6. Files creates the asset/version records and optional campaign share.
|
||||||
|
7. The database transaction commits and emits change-sequence entries.
|
||||||
|
8. Connector-originated operations also emit their connector audit evidence.
|
||||||
|
|
||||||
|
Blob storage is not part of the database transaction. An object may therefore
|
||||||
|
be left without committed metadata after a process or database failure. No
|
||||||
|
automatic orphan reconciliation exists yet; operators must account for this in
|
||||||
|
integrity checks and retention plans.
|
||||||
|
|
||||||
|
### Governed connector import
|
||||||
|
|
||||||
|
The connector flow separates four concerns:
|
||||||
|
|
||||||
|
1. An administrator defines reusable credential material.
|
||||||
|
2. An administrator defines a scoped endpoint profile that may reference that
|
||||||
|
credential.
|
||||||
|
3. System, tenant, and leaf policy sources narrow the allowed profile,
|
||||||
|
credential, provider, URL, and remote path.
|
||||||
|
4. A user optionally links an allowed remote root as a user/group connector
|
||||||
|
space, then browses and imports selected content.
|
||||||
|
|
||||||
|
Before each network operation, Files checks profile visibility, connector
|
||||||
|
policy, endpoint safety, transport support, and response size. A successful
|
||||||
|
import becomes an independent managed snapshot. Later source changes have no
|
||||||
|
effect until an explicit sync.
|
||||||
|
|
||||||
|
### Campaign attachment evidence
|
||||||
|
|
||||||
|
When Campaign uses Files, the integration follows a freeze-before-send model:
|
||||||
|
|
||||||
|
1. The campaign refers to managed user/group sources and attachment patterns.
|
||||||
|
2. Files verifies access and resolves matching managed assets.
|
||||||
|
3. A prepared campaign snapshot records the exact asset, version, blob,
|
||||||
|
checksum, size, relative path, and source provenance.
|
||||||
|
4. Files materializes those bytes for the campaign build without exposing its
|
||||||
|
database models to Campaign.
|
||||||
|
5. Campaign job/entry use is recorded and later marked as sent.
|
||||||
|
|
||||||
|
Changing the current file after preparation does not change the version already
|
||||||
|
recorded as campaign evidence. A file response is marked `audit_relevant` once
|
||||||
|
the asset has a sent campaign attachment-use record.
|
||||||
|
|
||||||
|
### Process ownership
|
||||||
|
|
||||||
|
| Concern | Owner |
|
||||||
|
| --- | --- |
|
||||||
|
| Authentication, tenants, RBAC evaluation, audit service, change sequence, settings, and module lifecycle | GovOPlaN Core |
|
||||||
|
| Managed assets, blobs, versions, folders, shares, connector baseline, provenance, and campaign attachment evidence | Files |
|
||||||
|
| Campaign definition, recipient data, message build/send state, and delivery policy | Campaign |
|
||||||
|
| Remote ACLs, remote source content, and upstream revision semantics | The external provider |
|
||||||
|
| Storage durability, egress policy, master key, secret environment, backup, and recovery | Deployment operator |
|
||||||
|
| Collaborative editing, comments, review, locks, and semantic document workflows | A future Documents/workflow/provider module |
|
||||||
|
|
||||||
|
## Ownership and access
|
||||||
|
|
||||||
|
Files applies both permission checks and resource checks. A broad operation
|
||||||
|
permission alone does not make another user's file visible.
|
||||||
|
|
||||||
|
### Resource access
|
||||||
|
|
||||||
|
- A user owns their personal space.
|
||||||
|
- A group member can use the group's file space.
|
||||||
|
- A Files administrator can access all Files resources in the active tenant.
|
||||||
|
- A file share can grant read or write access to a user, group, or the tenant.
|
||||||
|
- Campaign shares are resolved only in a verified campaign context and do not
|
||||||
|
become ordinary user shares.
|
||||||
|
- Folders are owned by a user or group; they are not independently shared.
|
||||||
|
- Soft-deleted resources are excluded from normal access and listing.
|
||||||
|
- Tenant identifiers are checked on every managed object lookup.
|
||||||
|
|
||||||
|
The `files.access` capability can explain why a principal has access: resource,
|
||||||
|
owner, administrator scope, or active share. It also explains virtual folders
|
||||||
|
that exist through child assets even when there is no explicit folder row.
|
||||||
|
|
||||||
|
Deleting an organization/access group is vetoed while it owns Files assets,
|
||||||
|
folders, connector spaces, or is the target of file shares. Reassign or remove
|
||||||
|
those relationships first.
|
||||||
|
|
||||||
|
### Operation permissions
|
||||||
|
|
||||||
|
| Permission | Allows |
|
||||||
|
| --- | --- |
|
||||||
|
| `files:file:read` | List and inspect accessible files, folders, spaces, and visible connectors |
|
||||||
|
| `files:file:download` | Download an accessible current version or ZIP archive |
|
||||||
|
| `files:file:upload` | Upload managed assets and import/sync selected connector files |
|
||||||
|
| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces |
|
||||||
|
| `files:file:share` | Create or update file shares |
|
||||||
|
| `files:file:delete` | Soft-delete accessible writable files and folders |
|
||||||
|
| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant |
|
||||||
|
|
||||||
|
The `file_manager` role template grants all normal file operations except
|
||||||
|
`files:file:admin`. The `file_viewer` template grants read and download.
|
||||||
|
System and tenant settings permissions can also authorize the corresponding
|
||||||
|
connector administration endpoints.
|
||||||
|
|
||||||
|
## Administration and policy
|
||||||
|
|
||||||
|
### Profiles, credentials, policies, and spaces
|
||||||
|
|
||||||
|
Keep these definitions separate:
|
||||||
|
|
||||||
|
- a **credential** holds reusable authentication material and may be restricted
|
||||||
|
to a provider;
|
||||||
|
- a **profile** holds the endpoint, scope, base path, credential reference,
|
||||||
|
local policy, and descriptive operation capabilities;
|
||||||
|
- a **policy** restricts what a scope may configure or use;
|
||||||
|
- a **connector space** links one approved profile/library/path to one user or
|
||||||
|
group and always uses manual, read-only synchronization today.
|
||||||
|
|
||||||
|
Profile capability values such as `browse`, `import`, and `sync` are stored and
|
||||||
|
returned, but they are descriptive today. Provider implementation and policy
|
||||||
|
checks enforce actual availability; do not use the capability list as the sole
|
||||||
|
security control.
|
||||||
|
|
||||||
|
Profiles, credentials, and policies support `system`, `tenant`, `user`,
|
||||||
|
`group`, and `campaign` scopes. A normal user sees system and active-tenant
|
||||||
|
profiles plus leaf profiles that match the user, one of their groups, or an
|
||||||
|
accessible campaign. Disabled definitions are visible only through authorized
|
||||||
|
administrative reads.
|
||||||
|
|
||||||
|
### Policy evaluation
|
||||||
|
|
||||||
|
For a leaf scope, the effective source chain is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
system -> tenant -> user | group | campaign
|
||||||
|
```
|
||||||
|
|
||||||
|
Policy fields are:
|
||||||
|
|
||||||
|
- connector/profile IDs;
|
||||||
|
- credential IDs;
|
||||||
|
- providers;
|
||||||
|
- external IDs;
|
||||||
|
- external path prefixes or glob patterns;
|
||||||
|
- external URL glob patterns.
|
||||||
|
|
||||||
|
Rules use `allow` and `deny` objects. Legacy synonyms `allowlist`, `whitelist`,
|
||||||
|
`denylist`, and `blacklist` are normalized. A matching deny at any source wins.
|
||||||
|
Every allow field defined by a source must match, so lower sources can narrow an
|
||||||
|
inherited set. The effective-policy response includes the contributing source
|
||||||
|
path and applied fields for explanation.
|
||||||
|
|
||||||
|
A parent may set `allow_lower_level_limits` for individual fields such as
|
||||||
|
`allow.providers` or `deny.external_paths`. An explicit `false` prevents a lower
|
||||||
|
scope from configuring that field. Absence does not lock the field.
|
||||||
|
|
||||||
|
Example tenant policy:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"allow": {
|
||||||
|
"providers": ["webdav", "nextcloud"],
|
||||||
|
"external_urls": ["https://files.example.edu/*"],
|
||||||
|
"external_paths": ["departments/finance"]
|
||||||
|
},
|
||||||
|
"deny": {
|
||||||
|
"external_paths": ["departments/finance/private"]
|
||||||
|
},
|
||||||
|
"allow_lower_level_limits": {
|
||||||
|
"allow.providers": true,
|
||||||
|
"deny.external_paths": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `POST /api/v1/files/connector-policy/evaluate` for an explainable preflight.
|
||||||
|
The normal profile browse/import/sync routes perform their own policy checks;
|
||||||
|
preflight does not replace enforcement.
|
||||||
|
|
||||||
|
### Credential rules
|
||||||
|
|
||||||
|
Database-created passwords and tokens use Core's Fernet encryption and require
|
||||||
|
the deployment `MASTER_KEY_B64` outside development/test/local environments.
|
||||||
|
Secret values, environment variable names, and local CA paths are never returned
|
||||||
|
in connector profile responses.
|
||||||
|
|
||||||
|
API-managed profiles and credentials cannot select process environment
|
||||||
|
variables, create an external `secret_ref`, or conceal secret-like values in
|
||||||
|
nested metadata. Deployment-owned JSON/file profiles may use `password_env` or
|
||||||
|
`token_env` only when the exact environment variable name appears in
|
||||||
|
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`.
|
||||||
|
|
||||||
|
Custom CA bundles must be absolute existing files in
|
||||||
|
`GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST`. TLS verification can be disabled only
|
||||||
|
in a development/test runtime.
|
||||||
|
|
||||||
|
### Provider status
|
||||||
|
|
||||||
|
| Provider | Current status |
|
||||||
|
| --- | --- |
|
||||||
|
| Seafile | Read-only native API browse/download-link import and manual sync implemented using the pinned HTTP transport; WebDAV opt-in supported |
|
||||||
|
| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport |
|
||||||
|
| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport |
|
||||||
|
| SMB | Browse/import code and descriptor implemented, but every live connection fails closed until initial connections and DFS referrals can be policy-validated and pinned |
|
||||||
|
| S3 connector | Browse/import code and descriptor implemented, but every live SDK connection fails closed until botocore connections, redirects, and endpoint discovery can be validated and pinned |
|
||||||
|
| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned |
|
||||||
|
| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature |
|
||||||
|
|
||||||
|
Provider descriptors are available from
|
||||||
|
`GET /api/v1/files/connectors/providers`. Use their `implemented`, `installed`,
|
||||||
|
and support fields for display, but expect a fail-closed transport error where
|
||||||
|
the table above says live access is disabled.
|
||||||
|
|
||||||
|
## Operator runbook
|
||||||
|
|
||||||
|
### Storage configuration
|
||||||
|
|
||||||
|
| Setting | Default | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `FILE_STORAGE_BACKEND` | `local` | Selects `local` or `s3` managed blob storage |
|
||||||
|
| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root |
|
||||||
|
| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root |
|
||||||
|
| `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket |
|
||||||
|
| `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and ZIP-member maximum |
|
||||||
|
| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | ZIP request and extracted-total maximum |
|
||||||
|
| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets |
|
||||||
|
|
||||||
|
The local backend is the operational baseline. It resolves every storage key
|
||||||
|
under the configured root and rejects escape attempts. Fallback roots support a
|
||||||
|
controlled storage-root migration: new writes go to the primary root while
|
||||||
|
reads can still find older objects.
|
||||||
|
|
||||||
|
The S3 managed-storage adapter currently fails closed before creating a boto3
|
||||||
|
client because the SDK cannot yet guarantee connection-time DNS/IP pinning and
|
||||||
|
redirect revalidation. Do not select `FILE_STORAGE_BACKEND=s3` for a live Files
|
||||||
|
deployment until that boundary is implemented and the status above changes.
|
||||||
|
|
||||||
|
Multiple API replicas require the same durable blob namespace. Separate local
|
||||||
|
container filesystems will produce incomplete reads. Until a pinned shared
|
||||||
|
object-storage transport is available, use one durable shared mount or constrain
|
||||||
|
Files traffic to a deployment topology that preserves one consistent local
|
||||||
|
root.
|
||||||
|
|
||||||
|
### Connector egress
|
||||||
|
|
||||||
|
Connector access to private networks is a deployment-wide decision:
|
||||||
|
|
||||||
|
```text
|
||||||
|
GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true|false
|
||||||
|
```
|
||||||
|
|
||||||
|
Production-like configuration validation requires an explicit value. Public-only
|
||||||
|
mode rejects any hostname whose DNS answers include a non-public address.
|
||||||
|
Private-enabled mode still rejects link-local, multicast, unspecified, and
|
||||||
|
limited-broadcast addresses.
|
||||||
|
|
||||||
|
The built-in HTTP transport:
|
||||||
|
|
||||||
|
- resolves and validates every connection attempt;
|
||||||
|
- connects the socket to the exact approved address while retaining the
|
||||||
|
original hostname for HTTP Host, TLS SNI, and certificate verification;
|
||||||
|
- does not inherit proxy settings;
|
||||||
|
- refuses redirects instead of following a new peer implicitly;
|
||||||
|
- bounds structured responses to 16 MiB and file transfers to 512 MiB by
|
||||||
|
default.
|
||||||
|
|
||||||
|
Override the connector response limits with
|
||||||
|
`GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` and
|
||||||
|
`GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES`. The smaller applicable limit wins
|
||||||
|
when an import is also subject to `FILE_UPLOAD_MAX_BYTES`.
|
||||||
|
|
||||||
|
Never work around a pinning failure by adding a raw IP, disabling TLS, or
|
||||||
|
enabling private networks. SMB may redirect through DFS, and S3 SDKs may perform
|
||||||
|
their own redirects or endpoint discovery; both remain disabled even for an IP
|
||||||
|
literal until every connection peer can be governed.
|
||||||
|
|
||||||
|
### Backup and restore
|
||||||
|
|
||||||
|
The database and blob namespace are one logical backup set. A usable backup must
|
||||||
|
include:
|
||||||
|
|
||||||
|
- Files database rows, including asset/version/blob relationships, shares,
|
||||||
|
connector settings, and campaign attachment-use evidence;
|
||||||
|
- every object below `FILE_STORAGE_LOCAL_ROOT` and any still-used fallback root;
|
||||||
|
- the exact `MASTER_KEY_B64` needed to decrypt retained connector credentials;
|
||||||
|
- deployment-owned connector profile files, referenced CA bundles, and secret
|
||||||
|
environment configuration where those definitions are in use.
|
||||||
|
|
||||||
|
There is no Files backup/restore API and no automated blob integrity or orphan
|
||||||
|
reconciliation job. Use a write quiesce or coordinated snapshots so database
|
||||||
|
references and objects represent the same recovery point.
|
||||||
|
|
||||||
|
After restore:
|
||||||
|
|
||||||
|
1. Verify the active tenant and module migration state.
|
||||||
|
2. Verify the storage backend and roots before allowing writes.
|
||||||
|
3. Verify that the original master key is available before testing connectors.
|
||||||
|
4. Download representative files and compare bytes with their recorded SHA-256.
|
||||||
|
5. Test one authorized and one unauthorized owner/share path.
|
||||||
|
6. Test a permitted pinned HTTP connector, if connectors are configured.
|
||||||
|
7. Review audit and change-sequence continuity around the recovery point.
|
||||||
|
|
||||||
|
An inconsistent restore should fail closed for missing objects or undecryptable
|
||||||
|
credentials. Do not repair it by deleting evidence rows without an approved,
|
||||||
|
audited data-recovery decision.
|
||||||
|
|
||||||
|
### Disable, uninstall, and retire
|
||||||
|
|
||||||
|
Disabling a module preserves its persistent data. Ordinary uninstall is guarded
|
||||||
|
while Files tables contain persistent rows.
|
||||||
|
|
||||||
|
Destructive retirement is separate and irreversible at the application level.
|
||||||
|
The installer records a database snapshot, then Files scrubs and audits
|
||||||
|
remaining encrypted connector material before its database tables are dropped.
|
||||||
|
Legacy external secret references are detached and identified as non-owned;
|
||||||
|
Files never calls a provider delete operation for them.
|
||||||
|
|
||||||
|
The retirement executor drops database tables but does not delete corresponding
|
||||||
|
objects from the configured blob backend. Operators must include those orphaned
|
||||||
|
objects in the approved retention/destruction plan. Validate the installer
|
||||||
|
snapshot and independent blob backup before retirement.
|
||||||
|
|
||||||
|
### Operational signals
|
||||||
|
|
||||||
|
Use these symptoms as routing hints:
|
||||||
|
|
||||||
|
| Symptom | Likely boundary |
|
||||||
|
| --- | --- |
|
||||||
|
| `Stored object does not exist` | Database/blob restore mismatch, wrong root, or missing shared storage |
|
||||||
|
| `Stored secret cannot be decrypted` | Wrong or rotated `MASTER_KEY_B64` |
|
||||||
|
| Private/non-public endpoint blocked | Deployment-wide egress policy is public-only or DNS returned a forbidden answer |
|
||||||
|
| SDK cannot pin redirects/referrals | Expected fail-closed S3/SMB boundary, not a transient connector outage |
|
||||||
|
| Connector response exceeds limit | Remote payload exceeds connector or upload limit |
|
||||||
|
| Profile is not visible | Scope, disabled state, campaign access, or policy mismatch |
|
||||||
|
| Group removal is vetoed | The group still owns a file/folder/connector space or is a share target |
|
||||||
|
|
||||||
|
## Capabilities and integration
|
||||||
|
|
||||||
|
Other modules must integrate through Core contracts, Files capabilities, or the
|
||||||
|
public HTTP API. They must not import Files ORM models or storage helpers.
|
||||||
|
|
||||||
|
### Provided capabilities
|
||||||
|
|
||||||
|
| Capability | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `files.access` (`0.1.6`) | Explain resource access provenance for managed files, explicit folders, and virtual folders |
|
||||||
|
| `files.campaign_attachments` (`0.1.6`) | Resolve managed attachment matches, prepare frozen campaign snapshots, annotate built messages, share assets with a campaign, and record/mark exact attachment use |
|
||||||
|
|
||||||
|
Files requires Core principal resolution and permission evaluation. Campaign is
|
||||||
|
an optional dependency; when installed, Files consumes the optional
|
||||||
|
`campaigns.access` interface to verify campaign existence and access. Missing
|
||||||
|
optional Campaign support fails explicitly rather than bypassing the check.
|
||||||
|
|
||||||
|
### API families
|
||||||
|
|
||||||
|
All routes below are under `/api/v1/files`.
|
||||||
|
|
||||||
|
| Area | Routes |
|
||||||
|
| --- | --- |
|
||||||
|
| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` |
|
||||||
|
| Upload and folders | `POST /upload`, `POST /upload-zip`, `POST /folders`, `POST /folders/delete` |
|
||||||
|
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
|
||||||
|
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
|
||||||
|
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
|
||||||
|
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` |
|
||||||
|
| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` |
|
||||||
|
| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` |
|
||||||
|
| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` |
|
||||||
|
| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` |
|
||||||
|
| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` |
|
||||||
|
| Incremental connector settings | `GET /connectors/settings/delta` |
|
||||||
|
|
||||||
|
Consumers should use cursor/watermark contracts instead of assuming an
|
||||||
|
unbounded complete list. The default full-list page size is 500 and public page
|
||||||
|
sizes are capped at 1,000.
|
||||||
|
|
||||||
|
### Integration invariants
|
||||||
|
|
||||||
|
An integrating module should:
|
||||||
|
|
||||||
|
- ask Files/Core for access rather than trusting a submitted file ID;
|
||||||
|
- retain an exact version/blob/checksum at every governed evidence point;
|
||||||
|
- import external content before using it in a campaign, report, workflow, or
|
||||||
|
generated document;
|
||||||
|
- preserve Files provenance when producing a derived managed snapshot;
|
||||||
|
- treat source provenance as captured context, not cryptographic attestation;
|
||||||
|
- avoid writing remote providers through the baseline connector layer;
|
||||||
|
- define a separate capability when it needs behavior beyond managed snapshots;
|
||||||
|
- tolerate Files being absent when the integration is declared optional.
|
||||||
|
|
||||||
|
Future Postbox, Templates/Reports, BI, Documents, DMS, and workflow modules
|
||||||
|
should keep their own domain state and use Files for governed input/output
|
||||||
|
snapshots. Long-running provider sync, OAuth, remote mutation, and
|
||||||
|
provider-specific health belong in connector modules rather than expanding the
|
||||||
|
Files baseline indiscriminately.
|
||||||
|
|
||||||
|
## Security, provenance, audit, deletion, and retention
|
||||||
|
|
||||||
|
### Security controls implemented
|
||||||
|
|
||||||
|
- Core authentication, tenant scoping, CSRF/API handling, and RBAC protect the
|
||||||
|
public routes.
|
||||||
|
- Owner and share checks protect individual resources after operation-scope
|
||||||
|
checks.
|
||||||
|
- Logical paths reject traversal and storage keys cannot escape the local root.
|
||||||
|
- Upload, ZIP extraction, connector response, and S3 stream code use bounded
|
||||||
|
reads; encrypted ZIP archives are rejected.
|
||||||
|
- Connector HTTP sockets use connection-time DNS/IP validation and pinning,
|
||||||
|
redirects are refused, and unsafe SDK transports fail before client creation.
|
||||||
|
- Database-managed connector passwords/tokens are encrypted; responses redact
|
||||||
|
secrets and deployment references.
|
||||||
|
- API metadata is recursively checked for secret-like values.
|
||||||
|
- Downloads use sanitized attachment filenames.
|
||||||
|
- SHA-256 and byte size are recorded for every blob/version.
|
||||||
|
|
||||||
|
The module does **not** currently provide malware scanning, content disarm and
|
||||||
|
reconstruction, a file-type allowlist, per-user quota, at-rest encryption for
|
||||||
|
local blob bytes, or a dedicated preview sandbox. Deployments that require
|
||||||
|
these controls must supply them outside Files until explicit module contracts
|
||||||
|
exist.
|
||||||
|
|
||||||
|
### Provenance
|
||||||
|
|
||||||
|
Connector-originated assets can retain:
|
||||||
|
|
||||||
|
- source type;
|
||||||
|
- connector/profile ID and provider;
|
||||||
|
- external ID, path, and URL;
|
||||||
|
- revision and revision label;
|
||||||
|
- observation/import timestamps when supplied;
|
||||||
|
- selected provider metadata.
|
||||||
|
|
||||||
|
The normalized provenance and source revision are returned in file responses,
|
||||||
|
carried into campaign attachment matches, and included in connector audit
|
||||||
|
events. External metadata is provider/user input and is not a digital signature.
|
||||||
|
|
||||||
|
### Audit and change evidence
|
||||||
|
|
||||||
|
Files records canonical audit events for:
|
||||||
|
|
||||||
|
- connector discovery attempts, before the attempted external I/O;
|
||||||
|
- connector imports and manual syncs;
|
||||||
|
- download/archive access to connector-originated managed files;
|
||||||
|
- immediate connector profile and credential deletion/scrubbing;
|
||||||
|
- credential scrubbing during destructive module retirement.
|
||||||
|
|
||||||
|
Connector audit details include the managed asset/version/blob, checksum, size,
|
||||||
|
operation, source revision, and provenance where applicable. Deletion audit
|
||||||
|
details name secret/reference kinds but never the secret values.
|
||||||
|
|
||||||
|
Assets, folders, shares, profiles, credentials, policies, and connector spaces
|
||||||
|
also feed Core's incremental change sequence for UI synchronization. A change
|
||||||
|
entry is not equivalent to a canonical audit event. Ordinary local upload,
|
||||||
|
rename, share, and soft-delete operations do not yet all emit dedicated Files
|
||||||
|
audit events.
|
||||||
|
|
||||||
|
Campaign attachment-use records provide separate domain evidence for an exact
|
||||||
|
file version used in campaign preparation and delivery.
|
||||||
|
|
||||||
|
### Deletion semantics
|
||||||
|
|
||||||
|
The word "delete" has different meanings by object type:
|
||||||
|
|
||||||
|
| Object | Current delete behavior |
|
||||||
|
| --- | --- |
|
||||||
|
| File asset | Sets `deleted_at`; content, versions, blob references, and campaign evidence remain |
|
||||||
|
| Folder | Sets `deleted_at`; recursive deletion also soft-deletes descendants |
|
||||||
|
| Connector space | Sets `deleted_at` and disables the space |
|
||||||
|
| Connector credential | Immediately disables the tombstone and clears username, encrypted password/token, environment references, legacy external reference, mode, and private metadata; dependent profiles are disabled and detached |
|
||||||
|
| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata |
|
||||||
|
| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed |
|
||||||
|
| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility |
|
||||||
|
|
||||||
|
Credential/profile scrubbing and its audit event use the same database
|
||||||
|
transaction. If audit creation fails, the deletion rolls back. Repeating a
|
||||||
|
delete against an already scrubbed tombstone does not recreate secret evidence.
|
||||||
|
|
||||||
|
### Retention boundary
|
||||||
|
|
||||||
|
File versions and blobs are effectively retained indefinitely today. Although a
|
||||||
|
blob has `ref_count` and `retained_until` fields, no complete retention-policy,
|
||||||
|
legal-hold, hard-purge, or garbage-collection service enforces them. There is
|
||||||
|
also no supported user restore endpoint for soft-deleted assets/folders and no
|
||||||
|
automated orphan-object reconciliation.
|
||||||
|
|
||||||
|
Do not promise erasure, timed retention, legal hold, or self-service recovery
|
||||||
|
from the current soft-delete behavior. Those require an explicit, auditable
|
||||||
|
retention/purge design that preserves campaign and other evidence references.
|
||||||
|
|
||||||
|
## Acceptance scenarios
|
||||||
|
|
||||||
|
These scenarios describe expected behavior at the current boundary.
|
||||||
|
|
||||||
|
### Personal and group ownership
|
||||||
|
|
||||||
|
Given Alice has `files:file:upload` and belongs to Finance, when she uploads to
|
||||||
|
her personal or Finance space, then the asset has the selected owner and tenant.
|
||||||
|
Given Bob is outside Finance and has no share or Files admin permission, the
|
||||||
|
same asset ID must not make the asset readable to Bob.
|
||||||
|
|
||||||
|
### Safe ZIP import
|
||||||
|
|
||||||
|
Given a ZIP member contains `../../secret.txt`, is encrypted, exceeds 50 MiB,
|
||||||
|
or pushes actual extracted bytes above 250 MiB, unpacking must fail without a
|
||||||
|
committed managed asset. A valid archive must preserve normalized relative paths
|
||||||
|
below the chosen logical folder.
|
||||||
|
|
||||||
|
### Explicit conflicts
|
||||||
|
|
||||||
|
Given a target path already exists, `reject` must leave it unchanged, `rename`
|
||||||
|
must choose a non-conflicting path, and `overwrite` must soft-delete the old
|
||||||
|
asset before creating the replacement. A per-item `skip` must not create that
|
||||||
|
item.
|
||||||
|
|
||||||
|
### Policy-denied connector
|
||||||
|
|
||||||
|
Given the tenant permits WebDAV only below `departments/finance` and denies its
|
||||||
|
`private` child, a browse/import/sync request below the denied child must return
|
||||||
|
an explainable policy denial. Directly invoking the import endpoint must not
|
||||||
|
bypass the same rule.
|
||||||
|
|
||||||
|
### Pinned connector transport
|
||||||
|
|
||||||
|
Given public-only mode and DNS returns any private address, the connection must
|
||||||
|
be rejected before a socket opens. Given private mode, the HTTP connection must
|
||||||
|
still use the validated address and reject redirects. SMB and S3 must fail
|
||||||
|
before their SDK clients connect until all SDK-managed peers can be pinned.
|
||||||
|
|
||||||
|
### Imported evidence and sync
|
||||||
|
|
||||||
|
Given a permitted WebDAV file is imported, its managed response and audit event
|
||||||
|
must carry source identity, revision when available, current version, checksum,
|
||||||
|
and size. Re-syncing identical bytes must return `unchanged`; changed bytes must
|
||||||
|
create a higher version while preserving the previous version.
|
||||||
|
|
||||||
|
### Campaign freeze
|
||||||
|
|
||||||
|
Given a campaign snapshot selected version V1, when the managed asset later
|
||||||
|
advances to V2, the prepared/sent attachment-use evidence must still identify
|
||||||
|
V1 and its original blob/checksum.
|
||||||
|
|
||||||
|
### Immediate credential deletion
|
||||||
|
|
||||||
|
Given a database credential contains an encrypted password and is referenced by
|
||||||
|
two profiles, deleting it must scrub the credential, disable/detach both
|
||||||
|
profiles, record non-secret audit evidence, and publish connector-setting
|
||||||
|
changes in one transaction. If audit creation fails, no part of the deletion
|
||||||
|
may commit.
|
||||||
|
|
||||||
|
### Recovery
|
||||||
|
|
||||||
|
Given a coordinated database/blob backup and the original master key, restoring
|
||||||
|
it must allow representative downloads whose bytes match recorded SHA-256
|
||||||
|
values. A missing blob or wrong key must surface an error rather than silently
|
||||||
|
returning different content or credentials.
|
||||||
|
|
||||||
|
## Implemented and planned boundary
|
||||||
|
|
||||||
|
| Area | Implemented now | Planned or explicitly outside the current boundary |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Managed storage | Local durable-root backend, fallback read roots, tenant blob deduplication, checksums | Operational S3 after pinned SDK transport ([#34](https://git.add-ideas.de/add-ideas/govoplan-files/issues/34)); automated integrity/orphan reconciliation ([#36](https://git.add-ideas.de/add-ideas/govoplan-files/issues/36)) |
|
||||||
|
| Upload | Bounded direct upload, drag-and-drop UI, ZIP spool/extract limits, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload |
|
||||||
|
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
|
||||||
|
| Sharing | User/group/tenant/campaign grant or permission update through API; campaign linkage display | Share revocation/expiry and complete general share-management UI ([#37](https://git.add-ideas.de/add-ideas/govoplan-files/issues/37)) |
|
||||||
|
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/add-ideas/govoplan-files/issues/38)) |
|
||||||
|
| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected |
|
||||||
|
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers |
|
||||||
|
| SMB and S3 connectors | Provider descriptors and browse/import logic | Live access only after SDK connections plus DFS referrals/redirects are validated and pinned ([SMB #35](https://git.add-ideas.de/add-ideas/govoplan-files/issues/35), [S3 #34](https://git.add-ideas.de/add-ideas/govoplan-files/issues/34)) |
|
||||||
|
| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors |
|
||||||
|
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs |
|
||||||
|
| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate |
|
||||||
|
| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation |
|
||||||
|
| Preview | File metadata and attachment download | Dedicated safe content-preview service |
|
||||||
|
| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign |
|
||||||
|
| Collaboration | Governed input/output snapshots | Co-editing, comments, review, presence, locks, and semantic document versions belong to Documents/workflow/provider modules |
|
||||||
|
|
||||||
|
## Release and change checklist
|
||||||
|
|
||||||
|
Before releasing Files:
|
||||||
|
|
||||||
|
1. Keep `pyproject.toml`, root `package.json`, WebUI `package.json`, and the
|
||||||
|
module manifest version aligned. Version alignment is a release gate.
|
||||||
|
2. Run the Files Python tests and security/static-analysis gates from the
|
||||||
|
GovOPlaN meta repository.
|
||||||
|
3. Build/type-check the WebUI through the Core host application.
|
||||||
|
4. Verify database migrations on an upgrade copy and a clean database.
|
||||||
|
5. Exercise an allowed and denied owner/share path.
|
||||||
|
6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion.
|
||||||
|
7. Exercise connector policy explanation and one pinned HTTP provider where
|
||||||
|
configured; verify S3/SMB still fail closed.
|
||||||
|
8. Verify credential deletion scrubs dependents and produces audit evidence.
|
||||||
|
9. Verify a campaign attachment snapshot still identifies its exact version and
|
||||||
|
checksum after the current file changes.
|
||||||
|
10. Update the implemented/planned table whenever a boundary changes.
|
||||||
|
|
||||||
|
## Related documents
|
||||||
|
|
||||||
|
- [Repository overview](../README.md)
|
||||||
|
- [Connector ownership boundary](CONNECTOR_BOUNDARY.md)
|
||||||
|
- [Connector spaces design and implementation history](CONNECTOR_SPACES.md)
|
||||||
|
- [Document collaboration boundary](DOCUMENT_COLLABORATION_BOUNDARY.md)
|
||||||
|
|
||||||
|
Where an older planning section conflicts with current code or this handbook's
|
||||||
|
implemented/planned table, verify the code and update both documents in the same
|
||||||
|
reviewable documentation slice.
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@govoplan/files-webui",
|
"name": "@govoplan/files-webui",
|
||||||
"version": "0.1.8",
|
"version": "0.1.9",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "webui/src/index.ts",
|
"main": "webui/src/index.ts",
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "govoplan-files"
|
name = "govoplan-files"
|
||||||
version = "0.1.8"
|
version = "0.1.9"
|
||||||
description = "GovOPlaN files module with backend and WebUI integration."
|
description = "GovOPlaN files module with backend and WebUI integration."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
|
|||||||
461
src/govoplan_files/backend/documentation.py
Normal file
461
src/govoplan_files/backend/documentation.py
Normal file
@@ -0,0 +1,461 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.campaigns import (
|
||||||
|
CAPABILITY_CAMPAIGNS_ACCESS,
|
||||||
|
CampaignAccessProvider,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.modules import (
|
||||||
|
DocumentationCondition,
|
||||||
|
DocumentationContext,
|
||||||
|
DocumentationLink,
|
||||||
|
DocumentationTopic,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.archives import ZIP_UPLOAD_MAX_FILES
|
||||||
|
from govoplan_files.backend.storage.access import user_group_ids
|
||||||
|
from govoplan_files.backend.storage.connector_visibility import (
|
||||||
|
connector_profile_usable_for_import,
|
||||||
|
visible_connector_profiles_for_actor,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
|
||||||
|
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
|
||||||
|
_FILES_READ_SCOPE = "files:file:read"
|
||||||
|
_FILES_UPLOAD_SCOPE = "files:file:upload"
|
||||||
|
|
||||||
|
|
||||||
|
def documentation_topics(
|
||||||
|
context: DocumentationContext,
|
||||||
|
) -> tuple[DocumentationTopic, ...]:
|
||||||
|
if context.documentation_type != "user":
|
||||||
|
return ()
|
||||||
|
|
||||||
|
upload_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_upload_max_bytes",
|
||||||
|
default=_DEFAULT_UPLOAD_MAX_BYTES,
|
||||||
|
)
|
||||||
|
zip_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_upload_zip_max_bytes",
|
||||||
|
default=_DEFAULT_ZIP_MAX_BYTES,
|
||||||
|
)
|
||||||
|
topics: list[DocumentationTopic] = []
|
||||||
|
if upload_limit is not None:
|
||||||
|
topics.append(_upload_topic(upload_limit))
|
||||||
|
if upload_limit is not None and zip_limit is not None:
|
||||||
|
topics.append(_zip_topic(upload_limit, zip_limit))
|
||||||
|
topics.append(_connector_import_topic(context))
|
||||||
|
return tuple(topics)
|
||||||
|
|
||||||
|
|
||||||
|
def _upload_topic(max_bytes: int) -> DocumentationTopic:
|
||||||
|
limit = _format_byte_limit(max_bytes)
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.workflow.upload-managed-files",
|
||||||
|
title="Upload managed files",
|
||||||
|
summary=f"Upload files to a personal or accessible group space; each uploaded file may contain at most {limit}.",
|
||||||
|
body=(
|
||||||
|
f"The current deployment accepts at most {limit} for each ordinary upload. "
|
||||||
|
"A name conflict is never resolved silently: reject stops the upload, rename chooses a copy name, and overwrite retires the old asset before creating a new one."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=39,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(
|
||||||
|
label="Files handbook",
|
||||||
|
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||||
|
kind="repository",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
related_modules=("campaigns",),
|
||||||
|
unlocks=(
|
||||||
|
"Users and connected processes can place governed content in managed storage.",
|
||||||
|
),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and upload managed files.",
|
||||||
|
"The destination personal or group space grants this account write access; upload permission alone does not grant access to every space.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and choose My files or an accessible group space.",
|
||||||
|
"Open the intended destination folder and choose Upload, or drag files into the file list.",
|
||||||
|
"For every name conflict, explicitly reject, rename, overwrite, or skip the affected item.",
|
||||||
|
"Wait for the upload to finish, then open the resulting file details.",
|
||||||
|
],
|
||||||
|
"outcome": "Each accepted file is stored as a governed managed asset in the selected space.",
|
||||||
|
"verification": "Confirm the owner, logical path, size, checksum, and current version in Files.",
|
||||||
|
"constraints": [
|
||||||
|
{
|
||||||
|
"id": "ordinary-upload-size",
|
||||||
|
"label": "Maximum size per file",
|
||||||
|
"description": f"The current safe upload limit is {limit} per file.",
|
||||||
|
"values": [limit],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.upload-and-unpack-zip",
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _zip_topic(max_file_bytes: int, max_zip_bytes: int) -> DocumentationTopic:
|
||||||
|
member_limit = _format_byte_limit(max_file_bytes)
|
||||||
|
archive_limit = _format_byte_limit(max_zip_bytes)
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.workflow.upload-and-unpack-zip",
|
||||||
|
title="Upload and unpack a ZIP archive",
|
||||||
|
summary=(
|
||||||
|
f"Safely unpack up to {ZIP_UPLOAD_MAX_FILES:,} files from a ZIP whose request and extracted total are each limited to {archive_limit}."
|
||||||
|
),
|
||||||
|
body=(
|
||||||
|
f"The current deployment limits the ZIP request and actual extracted total to {archive_limit}, each member to {member_limit}, "
|
||||||
|
f"and the archive to {ZIP_UPLOAD_MAX_FILES:,} non-directory members. Encrypted archives and unsafe member paths are rejected. "
|
||||||
|
"Actual extracted bytes are counted instead of trusting ZIP headers."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=40,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(
|
||||||
|
label="Files handbook",
|
||||||
|
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||||
|
kind="repository",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
unlocks=(
|
||||||
|
"A bounded archive can become governed managed files without trusting archive paths or size declarations.",
|
||||||
|
),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and upload managed files.",
|
||||||
|
"The archive is not encrypted and fits the current configured limits.",
|
||||||
|
"The destination personal or group space grants this account write access.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and choose the managed destination space and folder.",
|
||||||
|
"Enable Unpack ZIP uploads, then choose or drag the ZIP archive.",
|
||||||
|
"Resolve every destination conflict explicitly.",
|
||||||
|
"Wait for extraction and finalization to finish before leaving the page.",
|
||||||
|
],
|
||||||
|
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
|
||||||
|
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
|
||||||
|
"constraints": [
|
||||||
|
{
|
||||||
|
"id": "zip-request-and-total",
|
||||||
|
"label": "Maximum ZIP request and extracted total",
|
||||||
|
"description": f"Both the compressed request and the actual extracted total are limited to {archive_limit}.",
|
||||||
|
"values": [archive_limit],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "zip-member-size",
|
||||||
|
"label": "Maximum extracted member size",
|
||||||
|
"description": f"Each extracted file is limited to {member_limit}.",
|
||||||
|
"values": [member_limit],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "zip-member-count",
|
||||||
|
"label": "Maximum file count",
|
||||||
|
"description": f"A ZIP may contain at most {ZIP_UPLOAD_MAX_FILES:,} non-directory members.",
|
||||||
|
"values": [f"{ZIP_UPLOAD_MAX_FILES:,} files"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.upload-managed-files",
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _connector_import_topic(context: DocumentationContext) -> DocumentationTopic:
|
||||||
|
principal = context.principal
|
||||||
|
if not _has_all_scopes(principal, (_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE)):
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"Connector import is not available to this account because both permission to view Files and permission to upload managed files are required."
|
||||||
|
)
|
||||||
|
|
||||||
|
tenant_id = _safe_text_attribute(principal, "tenant_id")
|
||||||
|
user_id = _safe_text_attribute(getattr(principal, "user", None), "id")
|
||||||
|
session = context.session
|
||||||
|
if not tenant_id or not user_id or not isinstance(session, Session):
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
member_group_ids = _actor_group_ids(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
include_admin_groups=False,
|
||||||
|
)
|
||||||
|
connector_group_ids = (
|
||||||
|
_actor_group_ids(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
include_admin_groups=True,
|
||||||
|
)
|
||||||
|
if _has_all_scopes(principal, ("files:file:admin",))
|
||||||
|
else member_group_ids
|
||||||
|
)
|
||||||
|
profiles = visible_connector_profiles_for_actor(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=connector_group_ids,
|
||||||
|
settings=context.settings,
|
||||||
|
campaign_visible=_campaign_visibility(
|
||||||
|
context,
|
||||||
|
session=session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=member_group_ids,
|
||||||
|
),
|
||||||
|
include_effective_policy=True,
|
||||||
|
)
|
||||||
|
usable_profiles = tuple(
|
||||||
|
profile
|
||||||
|
for profile in profiles
|
||||||
|
if connector_profile_usable_for_import(profile)
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||||
|
)
|
||||||
|
if not usable_profiles:
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"No connection visible to this account is currently eligible to offer an enabled, credential-ready, policy-allowed browse/import path through a pinning-safe provider. Ask a Files administrator to configure or authorize one."
|
||||||
|
)
|
||||||
|
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.workflow.import-managed-snapshot",
|
||||||
|
title="Import an external file as a governed snapshot",
|
||||||
|
summary="Browse an authorized connection read-only and import one selected file into managed storage as a frozen, traceable snapshot.",
|
||||||
|
body=(
|
||||||
|
"At least one connection visible to this account is currently eligible to offer the governed browse/import path. "
|
||||||
|
"The selected remote path and item are re-authorized when used, and browse, import, and sync never mutate the remote source. "
|
||||||
|
"The managed snapshot stays unchanged until an explicit manual sync."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "campaign_manager", "report_author"),
|
||||||
|
order=41,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(
|
||||||
|
label="Files handbook",
|
||||||
|
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||||
|
kind="repository",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
related_modules=("campaigns",),
|
||||||
|
unlocks=(
|
||||||
|
"Campaigns, reports, and workflows can consume a managed snapshot with stable source evidence.",
|
||||||
|
),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list", "files.connector-import"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and upload managed files.",
|
||||||
|
"At least one enabled, credential-ready, policy-allowed connection using a pinning-safe provider is visible to this account.",
|
||||||
|
"The selected remote path and item must pass their operation-time policy checks.",
|
||||||
|
"The managed destination space grants this account write access.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and choose a managed destination space.",
|
||||||
|
"Choose Sync from connection, select an available connection, and browse to the permitted remote file.",
|
||||||
|
"Import the selected file and resolve any destination conflict explicitly.",
|
||||||
|
"Review the managed file's source and current-version details before using it in another task.",
|
||||||
|
],
|
||||||
|
"current_configuration": [
|
||||||
|
"At least one actor-visible connection is eligible to offer a safe browse/import operation; the selected endpoint, path, and item are still checked when used.",
|
||||||
|
"Every selected remote path and item is re-authorized at operation time.",
|
||||||
|
],
|
||||||
|
"outcome": "The external content is a tenant-managed snapshot with a checksum, exact version, and recorded source context.",
|
||||||
|
"verification": "Reopen the managed file and confirm its recorded source context, source revision when available, checksum, and current version.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.governed-connectors-and-provenance",
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _connector_import_limitation(message: str) -> DocumentationTopic:
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.connector-import-unavailable",
|
||||||
|
title="External file import is not currently available",
|
||||||
|
summary=message,
|
||||||
|
body=(
|
||||||
|
f"{message} Files never exposes connection endpoints, storage paths, credential references, or raw connector policies in user documentation."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("user",),
|
||||||
|
order=41,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
any_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(DocumentationLink(label="Files", href="/files", kind="runtime"),),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list", "files.connector-import"],
|
||||||
|
"limitations": [message],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _configured_positive_int(
|
||||||
|
settings: object | None, name: str, *, default: int
|
||||||
|
) -> int | None:
|
||||||
|
raw_value = getattr(settings, name, default) if settings is not None else default
|
||||||
|
try:
|
||||||
|
value = int(raw_value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
return value if value > 0 else None
|
||||||
|
|
||||||
|
|
||||||
|
def _format_byte_limit(value: int) -> str:
|
||||||
|
units = ((1024**3, "GiB"), (1024**2, "MiB"), (1024, "KiB"))
|
||||||
|
for divisor, label in units:
|
||||||
|
if value % divisor == 0:
|
||||||
|
return f"{value // divisor:,} {label} ({value:,} bytes)"
|
||||||
|
return f"{value:,} bytes"
|
||||||
|
|
||||||
|
|
||||||
|
def _has_all_scopes(principal: object | None, scopes: tuple[str, ...]) -> bool:
|
||||||
|
checker = getattr(principal, "has", None)
|
||||||
|
if callable(checker):
|
||||||
|
try:
|
||||||
|
return all(bool(checker(scope)) for scope in scopes)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
granted = {str(scope) for scope in getattr(principal, "scopes", ())}
|
||||||
|
return all(scope in granted for scope in scopes)
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_text_attribute(value: object | None, name: str) -> str:
|
||||||
|
try:
|
||||||
|
result = getattr(value, name, "")
|
||||||
|
except Exception:
|
||||||
|
return ""
|
||||||
|
return str(result or "")
|
||||||
|
|
||||||
|
|
||||||
|
def _principal_group_ids(principal: object) -> tuple[str, ...]:
|
||||||
|
try:
|
||||||
|
values = getattr(principal, "group_ids", ())
|
||||||
|
except Exception:
|
||||||
|
return ()
|
||||||
|
return tuple(str(value) for value in values if str(value))
|
||||||
|
|
||||||
|
|
||||||
|
def _actor_group_ids(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: object,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
include_admin_groups: bool,
|
||||||
|
) -> tuple[str, ...]:
|
||||||
|
try:
|
||||||
|
values = user_group_ids(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
include_admin_groups=include_admin_groups,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return _principal_group_ids(principal)
|
||||||
|
return tuple(str(value) for value in values if str(value))
|
||||||
|
|
||||||
|
|
||||||
|
def _campaign_visibility(
|
||||||
|
context: DocumentationContext,
|
||||||
|
*,
|
||||||
|
session: Session,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: tuple[str, ...],
|
||||||
|
):
|
||||||
|
principal = context.principal
|
||||||
|
registry = context.registry
|
||||||
|
if not _has_all_scopes(principal, ("campaigns:campaign:read",)):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
if not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||||
|
return None
|
||||||
|
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
if not isinstance(capability, CampaignAccessProvider):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def campaign_visible(campaign_id: str) -> bool:
|
||||||
|
try:
|
||||||
|
return capability.campaign_exists(
|
||||||
|
session, tenant_id=tenant_id, campaign_id=campaign_id
|
||||||
|
) and capability.can_read_campaign(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=group_ids,
|
||||||
|
tenant_admin=_has_all_scopes(principal, ("tenant:*",)),
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
return campaign_visible
|
||||||
@@ -1,7 +1,10 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import replace
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
from sqlalchemy import inspect
|
||||||
|
|
||||||
from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER
|
from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER
|
||||||
from govoplan_core.core.files import CAPABILITY_FILES_ACCESS
|
from govoplan_core.core.files import CAPABILITY_FILES_ACCESS
|
||||||
from govoplan_core.core.module_guards import drop_table_retirement_provider, persistent_table_uninstall_guard
|
from govoplan_core.core.module_guards import drop_table_retirement_provider, persistent_table_uninstall_guard
|
||||||
@@ -22,10 +25,60 @@ from govoplan_core.core.modules import (
|
|||||||
from govoplan_core.db.base import Base
|
from govoplan_core.db.base import Base
|
||||||
from govoplan_files.backend.change_tracking import register_files_change_tracking
|
from govoplan_files.backend.change_tracking import register_files_change_tracking
|
||||||
from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata
|
from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata
|
||||||
|
from govoplan_files.backend.documentation import documentation_topics
|
||||||
|
|
||||||
register_files_change_tracking()
|
register_files_change_tracking()
|
||||||
|
|
||||||
|
|
||||||
|
_files_table_retirement_provider = drop_table_retirement_provider(
|
||||||
|
file_models.FileBlob,
|
||||||
|
file_models.FileFolder,
|
||||||
|
file_models.FileAsset,
|
||||||
|
file_models.FileVersion,
|
||||||
|
file_models.FileShare,
|
||||||
|
file_models.FileConnectorCredential,
|
||||||
|
file_models.FileConnectorPolicy,
|
||||||
|
file_models.FileConnectorProfile,
|
||||||
|
file_models.FileConnectorSpace,
|
||||||
|
file_models.CampaignAttachmentUse,
|
||||||
|
label="Files",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _files_retirement_provider(session: object | None, module_id: str):
|
||||||
|
plan = _files_table_retirement_provider(session, module_id)
|
||||||
|
base_executor = plan.destroy_data_executor
|
||||||
|
if base_executor is None:
|
||||||
|
return plan
|
||||||
|
|
||||||
|
def executor(execute_session: object, execute_module_id: str) -> None:
|
||||||
|
if not hasattr(execute_session, "get_bind") or not hasattr(execute_session, "query"):
|
||||||
|
raise RuntimeError("No database session is available for Files credential retirement.")
|
||||||
|
live_inspector = inspect(execute_session.get_bind())
|
||||||
|
if any(
|
||||||
|
live_inspector.has_table(table_name)
|
||||||
|
for table_name in (
|
||||||
|
file_models.FileConnectorCredential.__tablename__,
|
||||||
|
file_models.FileConnectorProfile.__tablename__,
|
||||||
|
)
|
||||||
|
):
|
||||||
|
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||||
|
delete_connector_credentials_for_retirement,
|
||||||
|
)
|
||||||
|
|
||||||
|
delete_connector_credentials_for_retirement(execute_session)
|
||||||
|
base_executor(execute_session, execute_module_id)
|
||||||
|
|
||||||
|
return replace(
|
||||||
|
plan,
|
||||||
|
destroy_data_warnings=(
|
||||||
|
*plan.destroy_data_warnings,
|
||||||
|
"Files-owned encrypted connector credentials are scrubbed and audited immediately before tables are dropped; legacy non-owned external references are detached without claiming provider-side deletion.",
|
||||||
|
),
|
||||||
|
destroy_data_executor=executor,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
|
def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
|
||||||
module_id, resource, action = scope.split(":", 2)
|
module_id, resource, action = scope.split(":", 2)
|
||||||
return PermissionDefinition(
|
return PermissionDefinition(
|
||||||
@@ -45,7 +98,7 @@ PERMISSIONS = (
|
|||||||
_permission("files:file:download", "Download files", "Download managed files and generated archives."),
|
_permission("files:file:download", "Download files", "Download managed files and generated archives."),
|
||||||
_permission("files:file:upload", "Upload files", "Upload new managed file versions."),
|
_permission("files:file:upload", "Upload files", "Upload new managed file versions."),
|
||||||
_permission("files:file:organize", "Organize files", "Create folders, rename, move or copy managed files."),
|
_permission("files:file:organize", "Organize files", "Create folders, rename, move or copy managed files."),
|
||||||
_permission("files:file:share", "Share files", "Grant or revoke managed file shares."),
|
_permission("files:file:share", "Share files", "Grant or update managed file shares; revocation is not available yet."),
|
||||||
_permission("files:file:delete", "Delete files", "Delete or hide managed files and folders where policy allows it."),
|
_permission("files:file:delete", "Delete files", "Delete or hide managed files and folders where policy allows it."),
|
||||||
_permission("files:file:admin", "Administer file spaces", "Administer all file spaces in the tenant."),
|
_permission("files:file:admin", "Administer file spaces", "Administer all file spaces in the tenant."),
|
||||||
)
|
)
|
||||||
@@ -115,7 +168,7 @@ def _files_router(context: ModuleContext):
|
|||||||
manifest = ModuleManifest(
|
manifest = ModuleManifest(
|
||||||
id="files",
|
id="files",
|
||||||
name="Files",
|
name="Files",
|
||||||
version="0.1.8",
|
version="0.1.9",
|
||||||
required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR),
|
required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR),
|
||||||
optional_dependencies=("campaigns",),
|
optional_dependencies=("campaigns",),
|
||||||
provides_interfaces=(
|
provides_interfaces=(
|
||||||
@@ -143,89 +196,432 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
documentation=(
|
documentation=(
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="files.governed-connectors-and-provenance",
|
id="files.workflow.organize-managed-files",
|
||||||
title="Governed file connectors and source provenance",
|
title="Organize managed files and folders",
|
||||||
summary="Connector profiles control who may browse an external source, which destinations are permitted, and how imported files retain verifiable source context.",
|
summary="Create folders and rename, move, or copy accessible managed content with explicit conflict handling.",
|
||||||
body=(
|
body=(
|
||||||
"Users select only connector profiles visible in their current scope and import remote content into managed Files storage before another module uses it. "
|
"Organization stays inside governed personal or group spaces. Moves preserve the asset identity, while copies create new assets and versions that reuse immutable blob bytes. "
|
||||||
"Administrators define profiles, separate credential references, and ordered system/tenant/owner policies; deny rules win and profile responses never expose secrets. "
|
"Every target conflict must be rejected, renamed, overwritten, or skipped explicitly."
|
||||||
"Operators control private-network access deployment-wide and must keep every remote connection pinned to a policy-validated DNS/IP answer. "
|
),
|
||||||
"The built-in HTTP transport pins each connection and refuses redirects. Live S3 and SMB SDK access fails closed until S3 redirects and SMB DFS referrals can be revalidated and pinned. "
|
layer="configured",
|
||||||
"Successful imports store the connector id, provider, remote path and identity, source revision, and selected metadata as provenance on the managed file and its audit events."
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=42,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=("files:file:read", "files:file:organize"),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(label="Move or copy API", href="/api/v1/files/transfer", kind="api"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
unlocks=("Managed content can be placed at stable logical paths without bypassing space access.",),
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and organize managed files.",
|
||||||
|
"You have write or owner access to every source item and destination space used by the operation; the global organize permission alone does not grant resource access.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and select the personal or group space to organize.",
|
||||||
|
"Create the required destination folders or select the files and folders to rename, move, or copy.",
|
||||||
|
"Choose the destination and resolve each target conflict explicitly.",
|
||||||
|
"Apply the operation and reopen the destination folder.",
|
||||||
|
],
|
||||||
|
"outcome": "The selected content has the intended governed owner and logical path.",
|
||||||
|
"verification": "Confirm each resulting path and owner; for a move, also confirm the old path is gone, and for a copy, confirm the source remains.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.upload-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
"files.workflow.delete-managed-files",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.workflow.find-and-download-files",
|
||||||
|
title="Find and download managed files",
|
||||||
|
summary="Search accessible managed content and download a current file version or a ZIP archive of a selection.",
|
||||||
|
body=(
|
||||||
|
"Files can be sorted and searched by logical path or name pattern. A download always uses the accessible current version; a multi-file selection can be generated as a temporary ZIP archive. "
|
||||||
|
"There is no dedicated content-preview service yet."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=43,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=("files:file:read", "files:file:download"),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
unlocks=("Authorized users can retrieve governed current versions without gaining organization or sharing authority.",),
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": ["You may view and download managed files in the relevant space."],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and select the relevant personal or group space.",
|
||||||
|
"Navigate folders, sort the list, or use a path/name pattern to find the intended content.",
|
||||||
|
"Review the displayed owner, path, size, checksum, and version details.",
|
||||||
|
"Download one current file version, or select several files and choose Download ZIP.",
|
||||||
|
],
|
||||||
|
"outcome": "The authorized current file bytes or generated archive are downloaded to the local device.",
|
||||||
|
"verification": "Confirm the downloaded names and, where integrity matters, compare the file bytes with the displayed checksum.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.workflow.share-managed-files",
|
||||||
|
title="Grant or update access to managed files",
|
||||||
|
summary="Grant or update read, write, or manage access through a supporting workflow or API client without changing ownership.",
|
||||||
|
body=(
|
||||||
|
"The Files service can grant or update a share for a user, group, tenant, or campaign. The Files page does not yet provide a general share editor, and the API has no share-revocation route. "
|
||||||
|
"Do not use this task when access must later be revoked until that explicit capability is implemented."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_manager", "process_participant"),
|
||||||
|
order=44,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=("files:file:read", "files:file:share"),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(label="Grant or update a share", href="/api/v1/files/{file_id}/shares", kind="api"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
related_modules=("campaigns",),
|
||||||
|
unlocks=("A supporting process can grant governed file access without changing file ownership.",),
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and share the managed file and have write/manage access to that specific asset; the global share permission alone does not grant resource access.",
|
||||||
|
"A supporting workflow or API client is available; the Files page has no general share editor yet.",
|
||||||
|
"The process does not require share revocation, because no revocation route exists yet.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and verify the file owner, logical path, current version, and checksum.",
|
||||||
|
"Through the supporting workflow, identify the intended user, group, tenant, or campaign and choose read, write, or manage access.",
|
||||||
|
"Grant or update the share without changing file ownership.",
|
||||||
|
"Have the intended recipient verify access and an unrelated account verify denial.",
|
||||||
|
],
|
||||||
|
"limitations": [
|
||||||
|
"The Files page has no general share editor.",
|
||||||
|
"Shares can be granted or updated, but not revoked through the current API.",
|
||||||
|
],
|
||||||
|
"outcome": "The requested access is granted or updated while the managed asset keeps its owner.",
|
||||||
|
"verification": "Test one intended and one denied access path. Do not claim that the share can later be revoked.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
"files.assurance.process-and-release-readiness",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.workflow.delete-managed-files",
|
||||||
|
title="Delete managed files and folders",
|
||||||
|
summary="Soft-delete accessible managed files or a folder tree where current policy allows it.",
|
||||||
|
body=(
|
||||||
|
"Deletion hides the selected managed assets rather than hard-purging their stored evidence. Folder deletion is recursive by default and includes child folders and files; a non-recursive request fails for a non-empty folder. "
|
||||||
|
"There is no self-service restore or hard-purge workflow today."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=45,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=("files:file:read", "files:file:delete"),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
unlocks=("Authorized users can remove obsolete content from active file views while preserving the current soft-delete boundary.",),
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and delete the selected managed content and have write or owner access to every affected asset or folder.",
|
||||||
|
"You have reviewed the complete folder tree when deleting recursively.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and select the files or folder to delete.",
|
||||||
|
"Review the selection and, for a folder, all content below it.",
|
||||||
|
"Confirm the delete action.",
|
||||||
|
"Refresh or reopen the space and verify that the selected paths are no longer active.",
|
||||||
|
],
|
||||||
|
"limitations": [
|
||||||
|
"Deletion is soft deletion, not a hard purge.",
|
||||||
|
"There is no self-service restore or hard-purge workflow.",
|
||||||
|
],
|
||||||
|
"outcome": "The selected content is hidden from active Files views under the current soft-delete model.",
|
||||||
|
"verification": "Confirm the deleted paths no longer appear in the active space; do not treat the action as physical erasure.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.assurance.process-and-release-readiness",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.governed-connectors-and-provenance",
|
||||||
|
title="Govern file connections and credential deletion",
|
||||||
|
summary="Keep endpoint profiles, reusable credentials, and inherited connector policy separate, and understand what DELETE removes immediately.",
|
||||||
|
body=(
|
||||||
|
"System, tenant, and one user/group/campaign leaf form the effective policy chain: deny rules win and every configured allow rule must match. "
|
||||||
|
"Responses redact secret values and deployment references. Deleting a database-managed credential or profile immediately scrubs Files-owned encrypted material and private metadata in the same transaction as a non-secret audit event; dependent profiles are disabled, while legacy non-owned references are only detached and audited."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("admin",),
|
||||||
|
audience=("file_admin", "tenant_admin", "system_admin", "security_auditor"),
|
||||||
|
order=50,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
any_scopes=(
|
||||||
|
"files:file:admin",
|
||||||
|
"admin:settings:read",
|
||||||
|
"admin:settings:write",
|
||||||
|
"system:settings:read",
|
||||||
|
"system:settings:write",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Tenant connector administration", href="/admin?section=tenant-file-connectors", kind="runtime"),
|
||||||
|
DocumentationLink(label="System connector administration", href="/admin?section=system-file-connectors", kind="runtime"),
|
||||||
|
DocumentationLink(label="Tenant connector policy API", href="/api/v1/files/connectors/policies/tenant", kind="api"),
|
||||||
|
DocumentationLink(label="Connector credentials API", href="/api/v1/files/connectors/credentials", kind="api"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
related_modules=("access", "audit", "mail"),
|
||||||
|
unlocks=("Scoped, explainable external-file access without exposing credentials to consuming modules.",),
|
||||||
|
configuration_keys=(
|
||||||
|
"GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON",
|
||||||
|
"GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE",
|
||||||
|
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST",
|
||||||
|
"GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST",
|
||||||
|
"MASTER_KEY_B64",
|
||||||
|
),
|
||||||
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
|
"route": "/admin?section=tenant-file-connectors",
|
||||||
|
"screen": "File connections",
|
||||||
|
"section": "Profiles, credentials, and effective connector policy",
|
||||||
|
"security_invariants": [
|
||||||
|
"New API-managed external secret references fail closed until Files can prove ownership and provider-side deletion.",
|
||||||
|
"Deletion and destructive retirement scrub Files-owned encrypted connector material before completion and emit non-secret audit evidence.",
|
||||||
|
"Legacy non-owned external references are detached and audited, never sent to an arbitrary provider delete operation.",
|
||||||
|
],
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.import-managed-snapshot",
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
"mail.profiles-and-policy",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
title="Operate Files integrity, recovery, and connector transport safety",
|
||||||
|
summary="Back up database evidence, blob bytes, and the encryption key as one recovery unit, and keep unsupported SDK transports fail-closed.",
|
||||||
|
body=(
|
||||||
|
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the original master key, then verify representative checksums and access paths. "
|
||||||
|
"Live S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Destructive module retirement drops database tables but does not remove backend blob objects."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("admin",),
|
||||||
|
audience=("operator", "system_admin", "security_auditor"),
|
||||||
|
order=51,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
any_scopes=(
|
||||||
|
"files:file:admin",
|
||||||
|
"admin:settings:read",
|
||||||
|
"system:settings:read",
|
||||||
|
"system:audit:read",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="System file connections", href="/admin?section=system-file-connectors", kind="runtime"),
|
||||||
|
DocumentationLink(label="Connector provider status", href="/api/v1/files/connectors/providers", kind="api"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
related_modules=("audit", "ops"),
|
||||||
|
unlocks=("Recoverable managed-file evidence without weakening outbound peer validation.",),
|
||||||
|
configuration_keys=(
|
||||||
|
"FILE_STORAGE_BACKEND",
|
||||||
|
"FILE_STORAGE_LOCAL_ROOT",
|
||||||
|
"FILE_STORAGE_LOCAL_FALLBACK_ROOTS",
|
||||||
|
"FILE_UPLOAD_MAX_BYTES",
|
||||||
|
"FILE_UPLOAD_ZIP_MAX_BYTES",
|
||||||
|
"MASTER_KEY_B64",
|
||||||
|
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
|
||||||
|
"GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES",
|
||||||
|
"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES",
|
||||||
|
),
|
||||||
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
|
"route": "/admin?section=system-file-connectors",
|
||||||
|
"screen": "System file connections and deployment operations",
|
||||||
|
"section": "Storage integrity, backup/recovery, and fail-closed transports",
|
||||||
|
"recovery_unit": ["Files database rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"],
|
||||||
|
"verification": "After restore, download representative files, compare their bytes with recorded SHA-256 values, verify authorized and denied access, and test one permitted pinned HTTP connector.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.governed-connectors-and-provenance",
|
||||||
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.reference.snapshot-provenance-and-capabilities",
|
||||||
|
title="Integrate through managed snapshots and Files capabilities",
|
||||||
|
summary="Other modules consume stable Files capabilities or HTTP contracts and retain exact version evidence instead of importing Files internals.",
|
||||||
|
body=(
|
||||||
|
"Use files.access to explain resource access and files.campaign_attachments to freeze campaign inputs at an exact asset, version, blob, checksum, and source revision. "
|
||||||
|
"Import external content before a governed use, preserve provenance on derived snapshots, and keep collaboration, provider sync, OAuth, remote mutation, and domain workflow state in their owning modules."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=("module_integrator", "file_admin", "campaign_admin", "process_designer"),
|
||||||
|
order=52,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
any_scopes=("files:file:read", "files:file:upload", "files:file:admin"),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(label="Files API", href="/api/v1/files", kind="api"),
|
||||||
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
|
),
|
||||||
|
related_modules=("campaigns", "docs"),
|
||||||
|
unlocks=("Campaign, report, template, workflow, and document modules can exchange governed input/output snapshots.",),
|
||||||
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files and module integration",
|
||||||
|
"section": "Managed snapshot provenance and capability boundaries",
|
||||||
|
"provided_interfaces": ["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
|
||||||
|
"provenance_fields": ["connector_id", "provider", "external_id", "external_path", "revision", "metadata"],
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.import-managed-snapshot",
|
||||||
|
"files.governed-connectors-and-provenance",
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="files.assurance.process-and-release-readiness",
|
||||||
|
title="Assure a Files-backed process and release",
|
||||||
|
summary="Check a process against the implemented Files boundary, exercise permitted and denied paths, and retain evidence before approving a release or operational use.",
|
||||||
|
body=(
|
||||||
|
"A process owner must distinguish implemented controls from planned capabilities before relying on Files. A release is not ready until all package and manifest versions align and representative authorization, upload limits, conflict handling, download, deletion, connector, and recovery paths have been exercised. "
|
||||||
|
"Current limitations include no general share-management UI or share revocation, no self-service restore or hard purge, no enforced retention or legal hold, and no dedicated canonical audit event for every ordinary Files mutation. Record these limitations in the process assessment instead of treating soft deletion or change-sequence entries as stronger evidence."
|
||||||
),
|
),
|
||||||
layer="configured",
|
layer="configured",
|
||||||
documentation_types=("admin", "user"),
|
documentation_types=("admin", "user"),
|
||||||
audience=("file_user", "file_admin", "tenant_admin", "operator"),
|
audience=("process_owner", "release_manager", "file_admin", "operator", "security_auditor"),
|
||||||
order=42,
|
order=53,
|
||||||
conditions=(
|
conditions=(
|
||||||
DocumentationCondition(
|
DocumentationCondition(
|
||||||
required_modules=("files",),
|
required_modules=("files",),
|
||||||
any_scopes=(
|
any_scopes=(
|
||||||
"files:file:read",
|
"files:file:read",
|
||||||
"files:file:upload",
|
|
||||||
"files:file:admin",
|
"files:file:admin",
|
||||||
"admin:settings:read",
|
"admin:settings:read",
|
||||||
"system:settings:read",
|
"system:settings:read",
|
||||||
|
"system:audit:read",
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
links=(
|
links=(
|
||||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
DocumentationLink(label="Tenant connector administration", href="/admin?section=tenant-file-connectors", kind="runtime"),
|
DocumentationLink(label="Files API", href="/api/v1/files", kind="api"),
|
||||||
DocumentationLink(label="Connector profiles API", href="/api/v1/files/connectors/profiles", kind="api"),
|
DocumentationLink(label="Connector provider status", href="/api/v1/files/connectors/providers", kind="api"),
|
||||||
DocumentationLink(label="Connector providers API", href="/api/v1/files/connectors/providers", kind="api"),
|
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
|
||||||
DocumentationLink(label="Tenant connector policy API", href="/api/v1/files/connectors/policies/tenant", kind="api"),
|
|
||||||
),
|
),
|
||||||
related_modules=("campaigns",),
|
related_modules=("campaigns", "audit", "ops"),
|
||||||
unlocks=("Campaigns can consume managed attachments while Files preserves frozen source provenance.",),
|
unlocks=("Process owners and release managers can approve Files use against explicit controls, evidence, and known gaps.",),
|
||||||
configuration_keys=(
|
configuration_keys=(
|
||||||
|
"FILE_STORAGE_BACKEND",
|
||||||
|
"FILE_STORAGE_LOCAL_ROOT",
|
||||||
|
"FILE_UPLOAD_MAX_BYTES",
|
||||||
|
"FILE_UPLOAD_ZIP_MAX_BYTES",
|
||||||
|
"MASTER_KEY_B64",
|
||||||
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
|
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
|
||||||
"GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON",
|
|
||||||
"GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE",
|
|
||||||
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST",
|
|
||||||
),
|
),
|
||||||
metadata={
|
metadata={
|
||||||
"kind": "reference",
|
"kind": "workflow",
|
||||||
"perspectives": {
|
"route": "/files",
|
||||||
"user": "Choose a visible connection, browse it read-only, and import a selected file into managed storage before using it elsewhere.",
|
"screen": "Files process and release assurance",
|
||||||
"admin": "Govern profile visibility, separate credential references, provider allow/deny policy, and permitted connector operations.",
|
"help_contexts": ["files.list"],
|
||||||
"operator": "Control private-network access deployment-wide, provide credential and CA allowlists, and investigate fail-closed transport errors without weakening peer validation.",
|
"prerequisites": [
|
||||||
},
|
"A named process owner has defined the intended users, data classification, retention expectations, and integrations.",
|
||||||
"security_invariants": [
|
"A candidate release is installed on a clean database and an upgrade copy with aligned Python, root package, WebUI package, and module-manifest versions.",
|
||||||
"Every network peer must be policy-validated and pinned at connection time.",
|
"Representative permitted and denied accounts, bounded test files, and a coordinated database/blob/key recovery set are available.",
|
||||||
"Redirects and protocol referrals must be rejected or independently revalidated and pinned.",
|
|
||||||
"SDK transports that cannot provide those guarantees fail before client construction.",
|
|
||||||
],
|
],
|
||||||
"provenance_fields": [
|
"steps": [
|
||||||
"connector_id",
|
"Compare the process requirements with the handbook's implemented/planned boundary and record every unsupported requirement or compensating control.",
|
||||||
"provider",
|
"Confirm version alignment, apply migrations on clean and upgrade databases, and run the repository plus meta-repository security and static-analysis gates.",
|
||||||
"external_id",
|
"Exercise allowed and denied personal, group, and share access with representative accounts.",
|
||||||
"external_path",
|
"Exercise bounded upload and ZIP handling, every required conflict strategy, organization, download, and soft deletion.",
|
||||||
"revision",
|
"Where connectors are configured, verify policy explanation and one pinned HTTP provider; verify live S3 and SMB access still fails closed.",
|
||||||
"metadata",
|
"Restore a coordinated database/blob/key backup and compare representative downloaded bytes with their recorded SHA-256 checksums.",
|
||||||
|
"Record the tested versions, results, known limitations, evidence locations, residual risks, owner, and approval decision.",
|
||||||
|
],
|
||||||
|
"outcome": "The process or release has an explicit approval record tied to aligned versions, representative evidence, known limitations, and owned residual risks.",
|
||||||
|
"verification": "A reviewer can reproduce the recorded allow/deny, integrity, connector, and recovery checks and can trace each unmet requirement to a documented limitation or accepted compensating control.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.upload-managed-files",
|
||||||
|
"files.workflow.upload-and-unpack-zip",
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
"files.workflow.share-managed-files",
|
||||||
|
"files.workflow.delete-managed-files",
|
||||||
|
"files.workflow.import-managed-snapshot",
|
||||||
|
"files.governed-connectors-and-provenance",
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
],
|
],
|
||||||
"related_topic_ids": ["mail.profiles-and-policy"],
|
|
||||||
},
|
},
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
documentation_providers=(documentation_topics,),
|
||||||
migration_spec=MigrationSpec(
|
migration_spec=MigrationSpec(
|
||||||
module_id="files",
|
module_id="files",
|
||||||
metadata=Base.metadata,
|
metadata=Base.metadata,
|
||||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||||
retirement_supported=True,
|
retirement_supported=True,
|
||||||
retirement_provider=drop_table_retirement_provider(
|
retirement_provider=_files_retirement_provider,
|
||||||
file_models.FileBlob,
|
|
||||||
file_models.FileFolder,
|
|
||||||
file_models.FileAsset,
|
|
||||||
file_models.FileVersion,
|
|
||||||
file_models.FileShare,
|
|
||||||
file_models.FileConnectorCredential,
|
|
||||||
file_models.FileConnectorPolicy,
|
|
||||||
file_models.FileConnectorProfile,
|
|
||||||
file_models.FileConnectorSpace,
|
|
||||||
file_models.CampaignAttachmentUse,
|
|
||||||
label="Files",
|
|
||||||
),
|
|
||||||
retirement_notes="Destructive retirement drops files-owned database tables after the installer captures a database snapshot.",
|
retirement_notes="Destructive retirement drops files-owned database tables after the installer captures a database snapshot.",
|
||||||
),
|
),
|
||||||
uninstall_guard_providers=(
|
uninstall_guard_providers=(
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ from __future__ import annotations
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import tempfile
|
import tempfile
|
||||||
from dataclasses import replace
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
@@ -108,11 +107,14 @@ from govoplan_files.backend.storage.connector_credential_store import (
|
|||||||
ConnectorCredential,
|
ConnectorCredential,
|
||||||
connector_credential_from_row,
|
connector_credential_from_row,
|
||||||
create_connector_credential_row,
|
create_connector_credential_row,
|
||||||
deactivate_connector_credential_row,
|
|
||||||
get_connector_credential_row,
|
get_connector_credential_row,
|
||||||
list_database_connector_credentials,
|
list_database_connector_credentials,
|
||||||
update_connector_credential_row,
|
update_connector_credential_row,
|
||||||
)
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||||
|
delete_connector_credential_row,
|
||||||
|
delete_connector_profile_row,
|
||||||
|
)
|
||||||
from govoplan_files.backend.storage.connector_browse import (
|
from govoplan_files.backend.storage.connector_browse import (
|
||||||
ConnectorBrowseError,
|
ConnectorBrowseError,
|
||||||
ConnectorBrowseUnsupported,
|
ConnectorBrowseUnsupported,
|
||||||
@@ -127,13 +129,12 @@ from govoplan_files.backend.storage.connector_deployment import (
|
|||||||
from govoplan_files.backend.storage.connector_profile_store import (
|
from govoplan_files.backend.storage.connector_profile_store import (
|
||||||
connector_profile_from_row,
|
connector_profile_from_row,
|
||||||
create_connector_profile_row,
|
create_connector_profile_row,
|
||||||
deactivate_connector_profile_row,
|
|
||||||
get_connector_profile_row,
|
get_connector_profile_row,
|
||||||
list_database_connector_profiles,
|
|
||||||
update_connector_profile_row,
|
update_connector_profile_row,
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, connector_profiles_from_settings
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
from govoplan_files.backend.storage.connector_providers import connector_provider_descriptors
|
from govoplan_files.backend.storage.connector_providers import connector_provider_descriptors
|
||||||
|
from govoplan_files.backend.storage.connector_visibility import visible_connector_profiles_for_actor
|
||||||
from govoplan_files.backend.storage.connector_spaces import (
|
from govoplan_files.backend.storage.connector_spaces import (
|
||||||
connector_space_owner_id,
|
connector_space_owner_id,
|
||||||
create_connector_space,
|
create_connector_space,
|
||||||
@@ -453,35 +454,6 @@ def _enforce_connector_policy(source_provenance_json: str | None, connector_poli
|
|||||||
ensure_connector_policy_allows(request, sources)
|
ensure_connector_policy_allows(request, sources)
|
||||||
|
|
||||||
|
|
||||||
def _connector_profile_visible(
|
|
||||||
session: Session,
|
|
||||||
principal: ApiPrincipal,
|
|
||||||
profile: ConnectorProfile,
|
|
||||||
*,
|
|
||||||
campaign_id: str | None,
|
|
||||||
group_ids: set[str],
|
|
||||||
) -> bool:
|
|
||||||
if profile.scope_type == "system":
|
|
||||||
return True
|
|
||||||
if profile.scope_type == "tenant":
|
|
||||||
return profile.scope_id == principal.tenant_id
|
|
||||||
if profile.scope_type == "user":
|
|
||||||
return profile.scope_id == principal.user.id
|
|
||||||
if profile.scope_type == "group":
|
|
||||||
return bool(profile.scope_id and profile.scope_id in group_ids)
|
|
||||||
if profile.scope_type == "campaign":
|
|
||||||
if not profile.scope_id:
|
|
||||||
return False
|
|
||||||
if campaign_id and profile.scope_id != campaign_id:
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
_ensure_campaign_file_access(session, principal, profile.scope_id)
|
|
||||||
except HTTPException:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _visible_connector_profiles(
|
def _visible_connector_profiles(
|
||||||
session: Session,
|
session: Session,
|
||||||
principal: ApiPrincipal,
|
principal: ApiPrincipal,
|
||||||
@@ -492,30 +464,33 @@ def _visible_connector_profiles(
|
|||||||
include_admin_scopes: bool = False,
|
include_admin_scopes: bool = False,
|
||||||
include_effective_policy: bool = True,
|
include_effective_policy: bool = True,
|
||||||
) -> list[ConnectorProfile]:
|
) -> list[ConnectorProfile]:
|
||||||
provider_norm = provider.strip().casefold() if provider else None
|
group_ids = user_group_ids(
|
||||||
group_ids = set(user_group_ids(session, tenant_id=principal.tenant_id, user_id=principal.user.id, include_admin_groups=_is_admin(principal)))
|
|
||||||
database_profiles = list_database_connector_profiles(
|
|
||||||
session,
|
session,
|
||||||
tenant_id=principal.tenant_id,
|
tenant_id=principal.tenant_id,
|
||||||
include_disabled=include_disabled,
|
user_id=principal.user.id,
|
||||||
|
include_admin_groups=_is_admin(principal),
|
||||||
|
)
|
||||||
|
|
||||||
|
def campaign_visible(profile_campaign_id: str) -> bool:
|
||||||
|
try:
|
||||||
|
_ensure_campaign_file_access(session, principal, profile_campaign_id)
|
||||||
|
except HTTPException:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
return visible_connector_profiles_for_actor(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
group_ids=group_ids,
|
||||||
|
settings=settings,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
include_admin_scopes=include_admin_scopes,
|
||||||
|
include_effective_policy=include_effective_policy,
|
||||||
)
|
)
|
||||||
env_profiles = connector_profiles_from_settings(settings)
|
|
||||||
profiles_by_id: dict[str, ConnectorProfile] = {}
|
|
||||||
for profile in [*database_profiles, *env_profiles]:
|
|
||||||
if profile.id not in profiles_by_id:
|
|
||||||
profiles_by_id[profile.id] = profile
|
|
||||||
profiles = list(profiles_by_id.values())
|
|
||||||
visible: list[ConnectorProfile] = []
|
|
||||||
for profile in profiles:
|
|
||||||
if not (profile.enabled or include_disabled):
|
|
||||||
continue
|
|
||||||
if provider_norm is not None and profile.provider != provider_norm:
|
|
||||||
continue
|
|
||||||
admin_scope_visible = include_admin_scopes and profile.scope_type in {"user", "group", "campaign"}
|
|
||||||
if not admin_scope_visible and not _connector_profile_visible(session, principal, profile, campaign_id=campaign_id, group_ids=group_ids):
|
|
||||||
continue
|
|
||||||
visible.append(_with_effective_connector_policy(session, principal, profile) if include_effective_policy else profile)
|
|
||||||
return visible
|
|
||||||
|
|
||||||
|
|
||||||
def _webdav_discovery_candidates(payload: FileConnectorDiscoveryRequest) -> list[str]:
|
def _webdav_discovery_candidates(payload: FileConnectorDiscoveryRequest) -> list[str]:
|
||||||
@@ -639,18 +614,6 @@ def _visible_connector_profile(
|
|||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Connector profile not found")
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Connector profile not found")
|
||||||
|
|
||||||
|
|
||||||
def _with_effective_connector_policy(session: Session, principal: ApiPrincipal, profile: ConnectorProfile) -> ConnectorProfile:
|
|
||||||
sources = effective_connector_policy_sources(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
scope_type=profile.scope_type,
|
|
||||||
scope_id=profile.scope_id,
|
|
||||||
)
|
|
||||||
if not sources:
|
|
||||||
return profile
|
|
||||||
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))
|
|
||||||
|
|
||||||
|
|
||||||
def _require_connector_profile_write(principal: ApiPrincipal, scope_type: str) -> None:
|
def _require_connector_profile_write(principal: ApiPrincipal, scope_type: str) -> None:
|
||||||
clean_scope = scope_type.strip().casefold()
|
clean_scope = scope_type.strip().casefold()
|
||||||
if clean_scope == "system":
|
if clean_scope == "system":
|
||||||
@@ -2530,6 +2493,7 @@ def discover_connector_endpoint(
|
|||||||
reject_api_controlled_deployment_references(
|
reject_api_controlled_deployment_references(
|
||||||
password_env=payload.credentials.password_env,
|
password_env=payload.credentials.password_env,
|
||||||
token_env=payload.credentials.token_env,
|
token_env=payload.credentials.token_env,
|
||||||
|
secret_ref=payload.credentials.secret_ref,
|
||||||
metadata=payload.metadata,
|
metadata=payload.metadata,
|
||||||
)
|
)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
@@ -2856,7 +2820,14 @@ def deactivate_connector_credential(
|
|||||||
raise _http_error(exc, not_found=True) from exc
|
raise _http_error(exc, not_found=True) from exc
|
||||||
_require_connector_credential_write(principal, row.scope_type)
|
_require_connector_credential_write(principal, row.scope_type)
|
||||||
try:
|
try:
|
||||||
deactivate_connector_credential_row(session, row, user_id=principal.user.id)
|
deletion = delete_connector_credential_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason="api_delete",
|
||||||
|
user_id=principal.user.id,
|
||||||
|
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||||
|
)
|
||||||
|
if deletion.changed:
|
||||||
_record_connector_settings_change(
|
_record_connector_settings_change(
|
||||||
session,
|
session,
|
||||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||||
@@ -2867,12 +2838,31 @@ def deactivate_connector_credential(
|
|||||||
tenant_id=row.tenant_id,
|
tenant_id=row.tenant_id,
|
||||||
payload={"scope_type": row.scope_type, "scope_id": row.scope_id, "provider": row.provider},
|
payload={"scope_type": row.scope_type, "scope_id": row.scope_id, "provider": row.provider},
|
||||||
)
|
)
|
||||||
|
for profile in deletion.affected_profiles:
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
resource_id=profile.id,
|
||||||
|
operation="updated",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=profile.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": profile.scope_type,
|
||||||
|
"scope_id": profile.scope_id,
|
||||||
|
"provider": profile.provider,
|
||||||
|
"reason": "credential_deleted",
|
||||||
|
},
|
||||||
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
session.refresh(row)
|
session.refresh(row)
|
||||||
return _connector_credential_response(row)
|
return _connector_credential_response(row)
|
||||||
except FileStorageError as exc:
|
except FileStorageError as exc:
|
||||||
session.rollback()
|
session.rollback()
|
||||||
raise _http_error(exc) from exc
|
raise _http_error(exc) from exc
|
||||||
|
except Exception:
|
||||||
|
session.rollback()
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
|
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
|
||||||
@@ -3248,7 +3238,14 @@ def deactivate_connector_profile(
|
|||||||
raise _http_error(exc, not_found=True) from exc
|
raise _http_error(exc, not_found=True) from exc
|
||||||
_require_connector_profile_write(principal, row.scope_type)
|
_require_connector_profile_write(principal, row.scope_type)
|
||||||
try:
|
try:
|
||||||
deactivate_connector_profile_row(session, row, user_id=principal.user.id)
|
changed = delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason="api_delete",
|
||||||
|
user_id=principal.user.id,
|
||||||
|
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||||
|
)
|
||||||
|
if changed:
|
||||||
_record_connector_settings_change(
|
_record_connector_settings_change(
|
||||||
session,
|
session,
|
||||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
@@ -3265,6 +3262,9 @@ def deactivate_connector_profile(
|
|||||||
except FileStorageError as exc:
|
except FileStorageError as exc:
|
||||||
session.rollback()
|
session.rollback()
|
||||||
raise _http_error(exc) from exc
|
raise _http_error(exc) from exc
|
||||||
|
except Exception:
|
||||||
|
session.rollback()
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from govoplan_files.backend.storage.paths import filename_from_path, normalize_f
|
|||||||
|
|
||||||
|
|
||||||
_ZIP_READ_CHUNK_SIZE = 1024 * 1024
|
_ZIP_READ_CHUNK_SIZE = 1024 * 1024
|
||||||
|
ZIP_UPLOAD_MAX_FILES = 1000
|
||||||
|
|
||||||
|
|
||||||
def _read_zip_member(
|
def _read_zip_member(
|
||||||
@@ -76,7 +77,7 @@ def extract_zip_upload(
|
|||||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||||
metadata: dict[str, Any] | None = None,
|
metadata: dict[str, Any] | None = None,
|
||||||
is_admin: bool = False,
|
is_admin: bool = False,
|
||||||
max_files: int = 1000,
|
max_files: int = ZIP_UPLOAD_MAX_FILES,
|
||||||
max_file_bytes: int = 50 * 1024 * 1024,
|
max_file_bytes: int = 50 * 1024 * 1024,
|
||||||
max_total_bytes: int = 250 * 1024 * 1024,
|
max_total_bytes: int = 250 * 1024 * 1024,
|
||||||
) -> list[UploadedStoredFile]:
|
) -> list[UploadedStoredFile]:
|
||||||
|
|||||||
@@ -0,0 +1,337 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from sqlalchemy import inspect
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.audit.logging import audit_event
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorCredentialDeletionResult:
|
||||||
|
changed: bool
|
||||||
|
affected_profiles: tuple[FileConnectorProfile, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
def delete_connector_credential_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorCredential,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
user_id: str | None = None,
|
||||||
|
api_key_id: str | None = None,
|
||||||
|
) -> ConnectorCredentialDeletionResult:
|
||||||
|
"""Scrub a credential tombstone and disable every profile that used it.
|
||||||
|
|
||||||
|
``secret_ref`` predates a Files-owned secret-provider contract. It may be
|
||||||
|
shared or deployment-owned, so it is detached locally and explicitly
|
||||||
|
audited without ever being passed to a provider delete operation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
dependent_profiles = (
|
||||||
|
tuple(
|
||||||
|
session.query(FileConnectorProfile)
|
||||||
|
.filter(FileConnectorProfile.credential_profile_id == row.id)
|
||||||
|
.order_by(FileConnectorProfile.id.asc())
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
if inspect(session.get_bind()).has_table(FileConnectorProfile.__tablename__)
|
||||||
|
else ()
|
||||||
|
)
|
||||||
|
affected_profiles: list[FileConnectorProfile] = []
|
||||||
|
for profile in dependent_profiles:
|
||||||
|
if _delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
profile,
|
||||||
|
deletion_reason="credential_deleted",
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
):
|
||||||
|
affected_profiles.append(profile)
|
||||||
|
|
||||||
|
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||||
|
removed_reference_kinds = _credential_reference_kinds(row)
|
||||||
|
removed_metadata = bool(row.metadata_)
|
||||||
|
changed = _credential_row_requires_deletion(row)
|
||||||
|
if changed:
|
||||||
|
_scrub_credential_row(row, user_id=user_id)
|
||||||
|
session.add(row)
|
||||||
|
_audit_credential_deletion(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason=deletion_reason,
|
||||||
|
deleted_secret_kinds=deleted_secret_kinds,
|
||||||
|
removed_reference_kinds=removed_reference_kinds,
|
||||||
|
removed_metadata=removed_metadata,
|
||||||
|
affected_profile_count=len(affected_profiles),
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
return ConnectorCredentialDeletionResult(
|
||||||
|
changed=changed or bool(affected_profiles),
|
||||||
|
affected_profiles=tuple(affected_profiles),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def delete_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
user_id: str | None = None,
|
||||||
|
api_key_id: str | None = None,
|
||||||
|
) -> bool:
|
||||||
|
changed = _delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason=deletion_reason,
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
return changed
|
||||||
|
|
||||||
|
|
||||||
|
def delete_connector_credentials_for_retirement(session: Session) -> int:
|
||||||
|
"""Scrub and audit stored connector material before destructive retirement.
|
||||||
|
|
||||||
|
Legacy external references are detached and audited as non-owned. Files
|
||||||
|
never sends those arbitrary references to a provider delete operation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
inspector = inspect(session.get_bind())
|
||||||
|
profiles = (
|
||||||
|
session.query(FileConnectorProfile).order_by(FileConnectorProfile.id.asc()).all()
|
||||||
|
if inspector.has_table(FileConnectorProfile.__tablename__)
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
credentials = (
|
||||||
|
session.query(FileConnectorCredential).order_by(FileConnectorCredential.id.asc()).all()
|
||||||
|
if inspector.has_table(FileConnectorCredential.__tablename__)
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
deleted = 0
|
||||||
|
for profile in profiles:
|
||||||
|
if not _profile_row_has_credential_material(profile):
|
||||||
|
continue
|
||||||
|
if _delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
profile,
|
||||||
|
deletion_reason="module_data_retired",
|
||||||
|
):
|
||||||
|
deleted += 1
|
||||||
|
for credential in credentials:
|
||||||
|
if not _credential_row_has_material(credential):
|
||||||
|
continue
|
||||||
|
result = delete_connector_credential_row(
|
||||||
|
session,
|
||||||
|
credential,
|
||||||
|
deletion_reason="module_data_retired",
|
||||||
|
)
|
||||||
|
if result.changed:
|
||||||
|
deleted += 1
|
||||||
|
session.flush()
|
||||||
|
return deleted
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
user_id: str | None = None,
|
||||||
|
api_key_id: str | None = None,
|
||||||
|
) -> bool:
|
||||||
|
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||||
|
removed_reference_kinds = _profile_reference_kinds(row)
|
||||||
|
removed_metadata = bool(row.metadata_)
|
||||||
|
changed = _profile_row_requires_deletion(row)
|
||||||
|
if not changed:
|
||||||
|
return False
|
||||||
|
_scrub_profile_row(row, user_id=user_id)
|
||||||
|
session.add(row)
|
||||||
|
_audit_profile_deletion(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason=deletion_reason,
|
||||||
|
deleted_secret_kinds=deleted_secret_kinds,
|
||||||
|
removed_reference_kinds=removed_reference_kinds,
|
||||||
|
removed_metadata=removed_metadata,
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _scrub_credential_row(row: FileConnectorCredential, *, user_id: str | None) -> None:
|
||||||
|
row.enabled = False
|
||||||
|
row.credential_mode = "none"
|
||||||
|
row.username = None
|
||||||
|
row.password_encrypted = None
|
||||||
|
row.token_encrypted = None
|
||||||
|
row.password_env = None
|
||||||
|
row.token_env = None
|
||||||
|
row.secret_ref = None
|
||||||
|
row.metadata_ = {}
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
|
||||||
|
|
||||||
|
def _scrub_profile_row(row: FileConnectorProfile, *, user_id: str | None) -> None:
|
||||||
|
row.enabled = False
|
||||||
|
row.credential_profile_id = None
|
||||||
|
row.credential_mode = "none"
|
||||||
|
row.username = None
|
||||||
|
row.password_encrypted = None
|
||||||
|
row.token_encrypted = None
|
||||||
|
row.password_env = None
|
||||||
|
row.token_env = None
|
||||||
|
row.secret_ref = None
|
||||||
|
row.metadata_ = {}
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_row_requires_deletion(row: FileConnectorCredential) -> bool:
|
||||||
|
return bool(row.enabled or _credential_row_has_material(row))
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_row_requires_deletion(row: FileConnectorProfile) -> bool:
|
||||||
|
return bool(row.enabled or _profile_row_has_credential_material(row))
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_row_has_material(row: FileConnectorCredential) -> bool:
|
||||||
|
return bool(
|
||||||
|
row.username
|
||||||
|
or row.password_encrypted
|
||||||
|
or row.token_encrypted
|
||||||
|
or row.password_env
|
||||||
|
or row.token_env
|
||||||
|
or row.secret_ref
|
||||||
|
or row.metadata_
|
||||||
|
or row.credential_mode not in {"", "none", "anonymous"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_row_has_credential_material(row: FileConnectorProfile) -> bool:
|
||||||
|
return bool(
|
||||||
|
row.credential_profile_id
|
||||||
|
or row.username
|
||||||
|
or row.password_encrypted
|
||||||
|
or row.token_encrypted
|
||||||
|
or row.password_env
|
||||||
|
or row.token_env
|
||||||
|
or row.secret_ref
|
||||||
|
or row.metadata_
|
||||||
|
or row.credential_mode not in {"", "none", "anonymous"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _encrypted_secret_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||||
|
kinds: list[str] = []
|
||||||
|
if row.password_encrypted:
|
||||||
|
kinds.append("password")
|
||||||
|
if row.token_encrypted:
|
||||||
|
kinds.append("token")
|
||||||
|
return kinds
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_reference_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||||
|
kinds: list[str] = []
|
||||||
|
if row.password_env:
|
||||||
|
kinds.append("password_env")
|
||||||
|
if row.token_env:
|
||||||
|
kinds.append("token_env")
|
||||||
|
if row.secret_ref:
|
||||||
|
kinds.append("unowned_external_secret_ref")
|
||||||
|
return kinds
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_reference_kinds(row: FileConnectorProfile) -> list[str]:
|
||||||
|
kinds = _credential_reference_kinds(row)
|
||||||
|
if row.credential_profile_id:
|
||||||
|
kinds.append("credential_profile")
|
||||||
|
return kinds
|
||||||
|
|
||||||
|
|
||||||
|
def _storage_backend(deleted_secret_kinds: list[str], removed_reference_kinds: list[str]) -> str:
|
||||||
|
if "unowned_external_secret_ref" in removed_reference_kinds:
|
||||||
|
return "unowned_external_reference_detached"
|
||||||
|
if deleted_secret_kinds:
|
||||||
|
return "encrypted_database"
|
||||||
|
if removed_reference_kinds:
|
||||||
|
return "reference_only"
|
||||||
|
return "none"
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_credential_deletion(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorCredential,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
deleted_secret_kinds: list[str],
|
||||||
|
removed_reference_kinds: list[str],
|
||||||
|
removed_metadata: bool,
|
||||||
|
affected_profile_count: int,
|
||||||
|
user_id: str | None,
|
||||||
|
api_key_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
audit_event(
|
||||||
|
session,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
scope=_audit_scope(row.scope_type),
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
action="files.connector_credential_deleted",
|
||||||
|
object_type="file_connector_credential",
|
||||||
|
object_id=row.id,
|
||||||
|
details={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||||
|
"deleted_secret_kinds": deleted_secret_kinds,
|
||||||
|
"removed_reference_kinds": removed_reference_kinds,
|
||||||
|
"removed_metadata": removed_metadata,
|
||||||
|
"affected_profile_count": affected_profile_count,
|
||||||
|
"deletion_reason": deletion_reason,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_profile_deletion(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
deleted_secret_kinds: list[str],
|
||||||
|
removed_reference_kinds: list[str],
|
||||||
|
removed_metadata: bool,
|
||||||
|
user_id: str | None,
|
||||||
|
api_key_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
audit_event(
|
||||||
|
session,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
scope=_audit_scope(row.scope_type),
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
action="files.connector_profile_deleted",
|
||||||
|
object_type="file_connector_profile",
|
||||||
|
object_id=row.id,
|
||||||
|
details={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||||
|
"deleted_secret_kinds": deleted_secret_kinds,
|
||||||
|
"removed_reference_kinds": removed_reference_kinds,
|
||||||
|
"removed_metadata": removed_metadata,
|
||||||
|
"deletion_reason": deletion_reason,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_scope(scope_type: str) -> str:
|
||||||
|
return "system" if scope_type == "system" else "tenant"
|
||||||
@@ -52,6 +52,8 @@ class ConnectorCredential:
|
|||||||
|
|
||||||
@property
|
@property
|
||||||
def credentials_configured(self) -> bool:
|
def credentials_configured(self) -> bool:
|
||||||
|
if not self.enabled:
|
||||||
|
return False
|
||||||
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
|
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
|
||||||
return True
|
return True
|
||||||
# Environment references are deliberately unavailable to API-managed
|
# Environment references are deliberately unavailable to API-managed
|
||||||
@@ -188,6 +190,7 @@ def create_connector_credential_row(
|
|||||||
reject_api_controlled_deployment_references(
|
reject_api_controlled_deployment_references(
|
||||||
password_env=password_env,
|
password_env=password_env,
|
||||||
token_env=token_env,
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
metadata=metadata,
|
metadata=metadata,
|
||||||
)
|
)
|
||||||
clean_id = _normalize_id(credential_id)
|
clean_id = _normalize_id(credential_id)
|
||||||
@@ -242,8 +245,15 @@ def update_connector_credential_row(
|
|||||||
reject_api_controlled_deployment_references(
|
reject_api_controlled_deployment_references(
|
||||||
password_env=password_env,
|
password_env=password_env,
|
||||||
token_env=token_env,
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
metadata=metadata,
|
metadata=metadata,
|
||||||
)
|
)
|
||||||
|
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
|
||||||
|
if _clean(row.secret_ref):
|
||||||
|
raise FileStorageError(
|
||||||
|
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||||
|
"provider ownership and confirm provider-side deletion"
|
||||||
|
)
|
||||||
if label is not None:
|
if label is not None:
|
||||||
row.label = _normalize_label(label)
|
row.label = _normalize_label(label)
|
||||||
if provider is not None:
|
if provider is not None:
|
||||||
@@ -278,14 +288,6 @@ def update_connector_credential_row(
|
|||||||
return row
|
return row
|
||||||
|
|
||||||
|
|
||||||
def deactivate_connector_credential_row(session: Session, row: FileConnectorCredential, *, user_id: str | None) -> FileConnectorCredential:
|
|
||||||
row.enabled = False
|
|
||||||
row.updated_by_user_id = user_id
|
|
||||||
session.add(row)
|
|
||||||
session.flush()
|
|
||||||
return row
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||||
clean_scope_id = _clean(scope_id)
|
clean_scope_id = _clean(scope_id)
|
||||||
|
|||||||
@@ -17,6 +17,22 @@ _SECRET_ENV_METADATA_KEYS = frozenset(
|
|||||||
"session_token_env",
|
"session_token_env",
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
_SECRET_VALUE_METADATA_KEYS = frozenset(
|
||||||
|
{
|
||||||
|
"password",
|
||||||
|
"token",
|
||||||
|
"access_token",
|
||||||
|
"auth_token",
|
||||||
|
"bearer_token",
|
||||||
|
"refresh_token",
|
||||||
|
"api_key",
|
||||||
|
"access_key",
|
||||||
|
"access_key_id",
|
||||||
|
"secret_key",
|
||||||
|
"secret_access_key",
|
||||||
|
"session_token",
|
||||||
|
}
|
||||||
|
)
|
||||||
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
|
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
|
||||||
|
|
||||||
|
|
||||||
@@ -65,6 +81,7 @@ def reject_api_controlled_deployment_references(
|
|||||||
*,
|
*,
|
||||||
password_env: str | None = None,
|
password_env: str | None = None,
|
||||||
token_env: str | None = None,
|
token_env: str | None = None,
|
||||||
|
secret_ref: str | None = None,
|
||||||
metadata: Mapping[str, Any] | None = None,
|
metadata: Mapping[str, Any] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Reject process-secret selectors controlled through tenant-facing APIs."""
|
"""Reject process-secret selectors controlled through tenant-facing APIs."""
|
||||||
@@ -73,14 +90,39 @@ def reject_api_controlled_deployment_references(
|
|||||||
raise ConnectorDeploymentConfigurationError(
|
raise ConnectorDeploymentConfigurationError(
|
||||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||||
)
|
)
|
||||||
for key, value in (metadata or {}).items():
|
if _clean(secret_ref):
|
||||||
if _clean(value) and (str(key).strip().casefold() in _SECRET_ENV_METADATA_KEYS or str(key).strip().casefold().endswith("_env")):
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"External secret references cannot be managed through the Files API until Files can prove "
|
||||||
|
"provider ownership and confirm provider-side deletion"
|
||||||
|
)
|
||||||
|
for key, value in _metadata_entries(metadata or {}):
|
||||||
|
clean_key = str(key).strip().casefold()
|
||||||
|
if _clean(value) and (clean_key in _SECRET_ENV_METADATA_KEYS or clean_key.endswith("_env")):
|
||||||
raise ConnectorDeploymentConfigurationError(
|
raise ConnectorDeploymentConfigurationError(
|
||||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||||
)
|
)
|
||||||
|
if _clean(value) and (
|
||||||
|
clean_key in _SECRET_VALUE_METADATA_KEYS
|
||||||
|
or clean_key.endswith(("_password", "_secret", "_api_key"))
|
||||||
|
):
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"Connector credential values must use the dedicated encrypted credential fields, not metadata"
|
||||||
|
)
|
||||||
validate_connector_tls_metadata(metadata)
|
validate_connector_tls_metadata(metadata)
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata_entries(value: object) -> list[tuple[str, object]]:
|
||||||
|
entries: list[tuple[str, object]] = []
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
for key, item in value.items():
|
||||||
|
entries.append((str(key), item))
|
||||||
|
entries.extend(_metadata_entries(item))
|
||||||
|
elif isinstance(value, (list, tuple)):
|
||||||
|
for item in value:
|
||||||
|
entries.extend(_metadata_entries(item))
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
def connector_ca_bundle_path(value: str) -> str:
|
def connector_ca_bundle_path(value: str) -> str:
|
||||||
clean_value = _clean(value)
|
clean_value = _clean(value)
|
||||||
if not clean_value:
|
if not clean_value:
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from collections.abc import Mapping
|
from collections.abc import Callable, Mapping
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
@@ -20,7 +20,26 @@ def list_database_connector_profiles(
|
|||||||
*,
|
*,
|
||||||
tenant_id: str,
|
tenant_id: str,
|
||||||
include_disabled: bool = False,
|
include_disabled: bool = False,
|
||||||
|
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||||
) -> list[ConnectorProfile]:
|
) -> list[ConnectorProfile]:
|
||||||
|
profiles, _profile_ids = select_database_connector_profiles(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
row_visible=row_visible,
|
||||||
|
)
|
||||||
|
return profiles
|
||||||
|
|
||||||
|
|
||||||
|
def select_database_connector_profiles(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||||
|
) -> tuple[list[ConnectorProfile], set[str]]:
|
||||||
|
"""Return visible profiles and every database id that shadows settings."""
|
||||||
|
|
||||||
query = session.query(FileConnectorProfile).filter(
|
query = session.query(FileConnectorProfile).filter(
|
||||||
(FileConnectorProfile.scope_type == "system")
|
(FileConnectorProfile.scope_type == "system")
|
||||||
| (FileConnectorProfile.tenant_id == tenant_id)
|
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||||
@@ -28,9 +47,15 @@ def list_database_connector_profiles(
|
|||||||
if not include_disabled:
|
if not include_disabled:
|
||||||
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||||
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||||
|
profile_ids = {row.id for row in rows}
|
||||||
|
if row_visible is not None:
|
||||||
|
rows = [row for row in rows if row_visible(row)]
|
||||||
credential_ids = {_clean(row.credential_profile_id) for row in rows if _clean(row.credential_profile_id)}
|
credential_ids = {_clean(row.credential_profile_id) for row in rows if _clean(row.credential_profile_id)}
|
||||||
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
|
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
|
||||||
return [connector_profile_from_row(row, credential_row=credentials.get(row.credential_profile_id or "")) for row in rows]
|
return (
|
||||||
|
[connector_profile_from_row(row, credential_row=credentials.get(row.credential_profile_id or "")) for row in rows],
|
||||||
|
profile_ids,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def list_connector_profile_rows(
|
def list_connector_profile_rows(
|
||||||
@@ -128,6 +153,7 @@ def create_connector_profile_row(
|
|||||||
reject_api_controlled_deployment_references(
|
reject_api_controlled_deployment_references(
|
||||||
password_env=password_env,
|
password_env=password_env,
|
||||||
token_env=token_env,
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
metadata=metadata,
|
metadata=metadata,
|
||||||
)
|
)
|
||||||
clean_id = _normalize_profile_id(profile_id)
|
clean_id = _normalize_profile_id(profile_id)
|
||||||
@@ -187,11 +213,80 @@ def update_connector_profile_row(
|
|||||||
clear_password: bool = False,
|
clear_password: bool = False,
|
||||||
clear_token: bool = False,
|
clear_token: bool = False,
|
||||||
) -> FileConnectorProfile:
|
) -> FileConnectorProfile:
|
||||||
|
_validate_profile_update_references(
|
||||||
|
row,
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
_update_profile_connection_fields(
|
||||||
|
row,
|
||||||
|
label=label,
|
||||||
|
provider=provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=base_path,
|
||||||
|
enabled=enabled,
|
||||||
|
credential_profile_id=credential_profile_id,
|
||||||
|
credential_mode=credential_mode,
|
||||||
|
)
|
||||||
|
_update_profile_credential_fields(
|
||||||
|
row,
|
||||||
|
username=username,
|
||||||
|
password=password,
|
||||||
|
token=token,
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
clear_password=clear_password,
|
||||||
|
clear_token=clear_token,
|
||||||
|
)
|
||||||
|
_update_profile_governance_fields(
|
||||||
|
row,
|
||||||
|
capabilities=capabilities,
|
||||||
|
policy=policy,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
session.add(row)
|
||||||
|
session.flush()
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_profile_update_references(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
password_env: str | None,
|
||||||
|
token_env: str | None,
|
||||||
|
secret_ref: str | None,
|
||||||
|
metadata: Mapping[str, Any] | None,
|
||||||
|
) -> None:
|
||||||
reject_api_controlled_deployment_references(
|
reject_api_controlled_deployment_references(
|
||||||
password_env=password_env,
|
password_env=password_env,
|
||||||
token_env=token_env,
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
metadata=metadata,
|
metadata=metadata,
|
||||||
)
|
)
|
||||||
|
if secret_ref is None or _clean(secret_ref) == _clean(row.secret_ref):
|
||||||
|
return
|
||||||
|
if _clean(row.secret_ref):
|
||||||
|
raise FileStorageError(
|
||||||
|
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||||
|
"provider ownership and confirm provider-side deletion"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_profile_connection_fields(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
label: str | None,
|
||||||
|
provider: str | None,
|
||||||
|
endpoint_url: str | None,
|
||||||
|
base_path: str | None,
|
||||||
|
enabled: bool | None,
|
||||||
|
credential_profile_id: str | None,
|
||||||
|
credential_mode: str | None,
|
||||||
|
) -> None:
|
||||||
if label is not None:
|
if label is not None:
|
||||||
row.label = _normalize_label(label)
|
row.label = _normalize_label(label)
|
||||||
if provider is not None:
|
if provider is not None:
|
||||||
@@ -206,6 +301,20 @@ def update_connector_profile_row(
|
|||||||
row.credential_profile_id = _clean(credential_profile_id)
|
row.credential_profile_id = _clean(credential_profile_id)
|
||||||
if credential_mode is not None:
|
if credential_mode is not None:
|
||||||
row.credential_mode = _normalize_credential_mode(credential_mode)
|
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_profile_credential_fields(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
username: str | None,
|
||||||
|
password: str | None,
|
||||||
|
token: str | None,
|
||||||
|
password_env: str | None,
|
||||||
|
token_env: str | None,
|
||||||
|
secret_ref: str | None,
|
||||||
|
clear_password: bool,
|
||||||
|
clear_token: bool,
|
||||||
|
) -> None:
|
||||||
if username is not None:
|
if username is not None:
|
||||||
row.username = _clean(username)
|
row.username = _clean(username)
|
||||||
if password is not None:
|
if password is not None:
|
||||||
@@ -222,24 +331,21 @@ def update_connector_profile_row(
|
|||||||
row.token_env = _clean(token_env)
|
row.token_env = _clean(token_env)
|
||||||
if secret_ref is not None:
|
if secret_ref is not None:
|
||||||
row.secret_ref = _clean(secret_ref)
|
row.secret_ref = _clean(secret_ref)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_profile_governance_fields(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
capabilities: list[str] | None,
|
||||||
|
policy: Mapping[str, Any] | None,
|
||||||
|
metadata: Mapping[str, Any] | None,
|
||||||
|
) -> None:
|
||||||
if capabilities is not None:
|
if capabilities is not None:
|
||||||
row.capabilities = _string_list(capabilities)
|
row.capabilities = _string_list(capabilities)
|
||||||
if policy is not None:
|
if policy is not None:
|
||||||
row.policy = dict(policy)
|
row.policy = dict(policy)
|
||||||
if metadata is not None:
|
if metadata is not None:
|
||||||
row.metadata_ = dict(metadata)
|
row.metadata_ = dict(metadata)
|
||||||
row.updated_by_user_id = user_id
|
|
||||||
session.add(row)
|
|
||||||
session.flush()
|
|
||||||
return row
|
|
||||||
|
|
||||||
|
|
||||||
def deactivate_connector_profile_row(session: Session, row: FileConnectorProfile, *, user_id: str | None) -> FileConnectorProfile:
|
|
||||||
row.enabled = False
|
|
||||||
row.updated_by_user_id = user_id
|
|
||||||
session.add(row)
|
|
||||||
session.flush()
|
|
||||||
return row
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||||
|
|||||||
@@ -80,6 +80,8 @@ class ConnectorProfile:
|
|||||||
|
|
||||||
@property
|
@property
|
||||||
def credentials_configured(self) -> bool:
|
def credentials_configured(self) -> bool:
|
||||||
|
if not self.enabled:
|
||||||
|
return False
|
||||||
mode = self.credential_mode.casefold()
|
mode = self.credential_mode.casefold()
|
||||||
if mode in {"", "none", "anonymous"}:
|
if mode in {"", "none", "anonymous"}:
|
||||||
return True
|
return True
|
||||||
|
|||||||
218
src/govoplan_files/backend/storage/connector_visibility.py
Normal file
218
src/govoplan_files/backend/storage/connector_visibility.py
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable, Iterable
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
connector_effective_endpoint_url,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profile_store import (
|
||||||
|
select_database_connector_profiles,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import (
|
||||||
|
ConnectorProfile,
|
||||||
|
connector_profiles_from_settings,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorAccessRequest,
|
||||||
|
connector_policy_decision,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy_store import (
|
||||||
|
effective_connector_policy_sources,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_providers import (
|
||||||
|
connector_provider_descriptors,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
CampaignVisibility = Callable[[str], bool]
|
||||||
|
|
||||||
|
|
||||||
|
def visible_connector_profiles_for_actor(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: Iterable[str],
|
||||||
|
settings: object | None,
|
||||||
|
provider: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
campaign_visible: CampaignVisibility | None = None,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
include_admin_scopes: bool = False,
|
||||||
|
include_effective_policy: bool = True,
|
||||||
|
) -> list[ConnectorProfile]:
|
||||||
|
"""Return the same actor-filtered connector set used by API and docs surfaces."""
|
||||||
|
|
||||||
|
provider_norm = provider.strip().casefold() if provider else None
|
||||||
|
actor_group_ids = {str(group_id) for group_id in group_ids if str(group_id)}
|
||||||
|
database_profiles, database_profile_ids = select_database_connector_profiles(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
row_visible=lambda row: (
|
||||||
|
include_admin_scopes
|
||||||
|
and row.scope_type in {"user", "group", "campaign"}
|
||||||
|
) or _scope_visible_to_actor(
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=actor_group_ids,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
configured_profiles = connector_profiles_from_settings(settings)
|
||||||
|
profiles_by_id: dict[str, ConnectorProfile] = {}
|
||||||
|
for profile in database_profiles:
|
||||||
|
if profile.id not in profiles_by_id:
|
||||||
|
profiles_by_id[profile.id] = profile
|
||||||
|
for profile in configured_profiles:
|
||||||
|
if profile.id not in database_profile_ids and profile.id not in profiles_by_id:
|
||||||
|
profiles_by_id[profile.id] = profile
|
||||||
|
|
||||||
|
visible: list[ConnectorProfile] = []
|
||||||
|
for profile in profiles_by_id.values():
|
||||||
|
if not (profile.enabled or include_disabled):
|
||||||
|
continue
|
||||||
|
if provider_norm is not None and profile.provider != provider_norm:
|
||||||
|
continue
|
||||||
|
admin_scope_visible = include_admin_scopes and profile.scope_type in {
|
||||||
|
"user",
|
||||||
|
"group",
|
||||||
|
"campaign",
|
||||||
|
}
|
||||||
|
if not admin_scope_visible and not _profile_visible_to_actor(
|
||||||
|
profile,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=actor_group_ids,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
visible.append(
|
||||||
|
_with_effective_connector_policy(
|
||||||
|
session, tenant_id=tenant_id, profile=profile
|
||||||
|
)
|
||||||
|
if include_effective_policy
|
||||||
|
else profile
|
||||||
|
)
|
||||||
|
return visible
|
||||||
|
|
||||||
|
|
||||||
|
def connector_profile_usable_for_import(profile: ConnectorProfile) -> bool:
|
||||||
|
"""Whether a visible profile can safely offer the current live browse/import task."""
|
||||||
|
|
||||||
|
effective_endpoint_url = connector_effective_endpoint_url(
|
||||||
|
provider=profile.provider,
|
||||||
|
endpoint_url=profile.endpoint_url,
|
||||||
|
metadata=profile.metadata,
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
not profile.enabled
|
||||||
|
or not effective_endpoint_url
|
||||||
|
or not profile.credentials_configured
|
||||||
|
or bool(profile.secret_ref)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
descriptor = next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in connector_provider_descriptors()
|
||||||
|
if item.provider == profile.provider
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if descriptor is None:
|
||||||
|
return False
|
||||||
|
if not (
|
||||||
|
descriptor.implemented
|
||||||
|
and descriptor.installed
|
||||||
|
and descriptor.browse_supported
|
||||||
|
and descriptor.import_supported
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
# These SDK paths intentionally fail closed until every connection peer and
|
||||||
|
# SDK-managed redirect/referral can be pinned and revalidated.
|
||||||
|
if profile.provider in {"s3", "smb"}:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Prove that the initial root browse performed by the current Files UI is
|
||||||
|
# policy-allowed. A later selected remote path/item is checked again.
|
||||||
|
return connector_policy_decision(
|
||||||
|
ConnectorAccessRequest(
|
||||||
|
connector_id=profile.id,
|
||||||
|
credential_id=profile.credential_profile_id,
|
||||||
|
provider=profile.provider,
|
||||||
|
external_path="",
|
||||||
|
external_url=effective_endpoint_url,
|
||||||
|
operation="import",
|
||||||
|
),
|
||||||
|
profile.policy_sources,
|
||||||
|
).allowed
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_visible_to_actor(
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: set[str],
|
||||||
|
campaign_id: str | None,
|
||||||
|
campaign_visible: CampaignVisibility | None,
|
||||||
|
) -> bool:
|
||||||
|
return _scope_visible_to_actor(
|
||||||
|
scope_type=profile.scope_type,
|
||||||
|
scope_id=profile.scope_id,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=group_ids,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_visible_to_actor(
|
||||||
|
*,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: set[str],
|
||||||
|
campaign_id: str | None,
|
||||||
|
campaign_visible: CampaignVisibility | None,
|
||||||
|
) -> bool:
|
||||||
|
if scope_type == "system":
|
||||||
|
return True
|
||||||
|
if scope_type == "tenant":
|
||||||
|
return scope_id == tenant_id
|
||||||
|
if scope_type == "user":
|
||||||
|
return scope_id == user_id
|
||||||
|
if scope_type == "group":
|
||||||
|
return bool(scope_id and scope_id in group_ids)
|
||||||
|
if scope_type != "campaign" or not scope_id:
|
||||||
|
return False
|
||||||
|
if campaign_id and scope_id != campaign_id:
|
||||||
|
return False
|
||||||
|
return bool(campaign_visible and campaign_visible(scope_id))
|
||||||
|
|
||||||
|
|
||||||
|
def _with_effective_connector_policy(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
) -> ConnectorProfile:
|
||||||
|
sources = effective_connector_policy_sources(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
scope_type=profile.scope_type,
|
||||||
|
scope_id=profile.scope_id,
|
||||||
|
)
|
||||||
|
if not sources:
|
||||||
|
return profile
|
||||||
|
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))
|
||||||
277
tests/test_connector_credential_deletion.py
Normal file
277
tests/test_connector_credential_deletion.py
Normal file
@@ -0,0 +1,277 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine, inspect
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_access.backend.db import models as access_models # noqa: F401 - resolve Files user foreign keys
|
||||||
|
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_core.security.secrets import encrypt_secret
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||||
|
from govoplan_files.backend.router import deactivate_connector_credential, deactivate_connector_profile
|
||||||
|
|
||||||
|
|
||||||
|
class Principal:
|
||||||
|
tenant_id = "tenant-1"
|
||||||
|
user = SimpleNamespace(id="user-1")
|
||||||
|
api_key = None
|
||||||
|
|
||||||
|
def has(self, scope: str) -> bool:
|
||||||
|
return scope == "files:file:admin"
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorCredentialDeletionTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.engine = create_engine("sqlite:///:memory:")
|
||||||
|
Base.metadata.create_all(
|
||||||
|
self.engine,
|
||||||
|
tables=[
|
||||||
|
FileConnectorCredential.__table__,
|
||||||
|
FileConnectorProfile.__table__,
|
||||||
|
ChangeSequenceEntry.__table__,
|
||||||
|
],
|
||||||
|
)
|
||||||
|
self.session = sessionmaker(bind=self.engine)()
|
||||||
|
self.principal = Principal()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
Base.metadata.drop_all(bind=self.engine)
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_delete_credential_scrubs_material_audits_and_disables_dependents(self) -> None:
|
||||||
|
credential = self._credential()
|
||||||
|
profile = self._profile(credential_profile_id=credential.id)
|
||||||
|
self.session.add_all([credential, profile])
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||||
|
) as audit:
|
||||||
|
response = deactivate_connector_credential(
|
||||||
|
credential.id,
|
||||||
|
session=self.session,
|
||||||
|
principal=self.principal, # type: ignore[arg-type]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(response.enabled)
|
||||||
|
self.assertFalse(response.credentials_configured)
|
||||||
|
self.assertIsNone(response.credential_secret_source)
|
||||||
|
self.session.refresh(credential)
|
||||||
|
self.session.refresh(profile)
|
||||||
|
self._assert_credential_scrubbed(credential)
|
||||||
|
self._assert_profile_scrubbed(profile)
|
||||||
|
|
||||||
|
calls = {call.kwargs["action"]: call.kwargs for call in audit.call_args_list}
|
||||||
|
self.assertEqual(
|
||||||
|
{"files.connector_profile_deleted", "files.connector_credential_deleted"},
|
||||||
|
set(calls),
|
||||||
|
)
|
||||||
|
credential_audit = calls["files.connector_credential_deleted"]
|
||||||
|
self.assertEqual("encrypted_database", credential_audit["details"]["storage_backend"])
|
||||||
|
self.assertEqual(["password", "token"], credential_audit["details"]["deleted_secret_kinds"])
|
||||||
|
self.assertEqual(1, credential_audit["details"]["affected_profile_count"])
|
||||||
|
self.assertNotIn("do-not-audit", repr(audit.call_args_list))
|
||||||
|
|
||||||
|
changes = self.session.query(ChangeSequenceEntry).order_by(ChangeSequenceEntry.id.asc()).all()
|
||||||
|
self.assertEqual([credential.id, profile.id], [change.resource_id for change in changes])
|
||||||
|
self.assertEqual(["deleted", "updated"], [change.operation for change in changes])
|
||||||
|
|
||||||
|
def test_delete_profile_scrubs_inline_credentials_and_audits(self) -> None:
|
||||||
|
profile = self._profile(credential_profile_id="shared-credential")
|
||||||
|
self.session.add(profile)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||||
|
) as audit:
|
||||||
|
response = deactivate_connector_profile(
|
||||||
|
profile.id,
|
||||||
|
session=self.session,
|
||||||
|
principal=self.principal, # type: ignore[arg-type]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(response.enabled)
|
||||||
|
self.assertFalse(response.credentials_configured)
|
||||||
|
self.assertIsNone(response.credential_profile_id)
|
||||||
|
self.session.refresh(profile)
|
||||||
|
self._assert_profile_scrubbed(profile)
|
||||||
|
audit.assert_called_once()
|
||||||
|
call = audit.call_args.kwargs
|
||||||
|
self.assertEqual("files.connector_profile_deleted", call["action"])
|
||||||
|
self.assertEqual("api_delete", call["details"]["deletion_reason"])
|
||||||
|
self.assertIn("credential_profile", call["details"]["removed_reference_kinds"])
|
||||||
|
self.assertNotIn("do-not-audit", repr(call))
|
||||||
|
|
||||||
|
def test_delete_rolls_back_when_audit_fails(self) -> None:
|
||||||
|
credential = self._credential()
|
||||||
|
self.session.add(credential)
|
||||||
|
self.session.commit()
|
||||||
|
original_password = credential.password_encrypted
|
||||||
|
original_token = credential.token_encrypted
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.connector_credential_deletion.audit_event",
|
||||||
|
side_effect=RuntimeError("audit unavailable"),
|
||||||
|
), self.assertRaisesRegex(RuntimeError, "audit unavailable"):
|
||||||
|
deactivate_connector_credential(
|
||||||
|
credential.id,
|
||||||
|
session=self.session,
|
||||||
|
principal=self.principal, # type: ignore[arg-type]
|
||||||
|
)
|
||||||
|
|
||||||
|
persisted = self.session.get(FileConnectorCredential, credential.id)
|
||||||
|
assert persisted is not None
|
||||||
|
self.assertTrue(persisted.enabled)
|
||||||
|
self.assertEqual(original_password, persisted.password_encrypted)
|
||||||
|
self.assertEqual(original_token, persisted.token_encrypted)
|
||||||
|
self.assertEqual(0, self.session.query(ChangeSequenceEntry).count())
|
||||||
|
|
||||||
|
def test_delete_detaches_unowned_legacy_secret_reference_without_claiming_provider_deletion(self) -> None:
|
||||||
|
credential = self._credential(secret_ref="vault:tenant-1:files:credential")
|
||||||
|
self.session.add(credential)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||||
|
) as audit:
|
||||||
|
response = deactivate_connector_credential(
|
||||||
|
credential.id,
|
||||||
|
session=self.session,
|
||||||
|
principal=self.principal, # type: ignore[arg-type]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(response.enabled)
|
||||||
|
persisted = self.session.get(FileConnectorCredential, credential.id)
|
||||||
|
assert persisted is not None
|
||||||
|
self._assert_credential_scrubbed(persisted)
|
||||||
|
audit.assert_called_once()
|
||||||
|
details = audit.call_args.kwargs["details"]
|
||||||
|
self.assertEqual("unowned_external_reference_detached", details["storage_backend"])
|
||||||
|
self.assertIn("unowned_external_secret_ref", details["removed_reference_kinds"])
|
||||||
|
self.assertNotIn("vault:tenant-1:files:credential", repr(audit.call_args))
|
||||||
|
|
||||||
|
def test_retirement_scrubs_and_audits_before_tables_are_dropped(self) -> None:
|
||||||
|
from govoplan_files.backend.manifest import manifest
|
||||||
|
|
||||||
|
self.session.add_all([self._credential(), self._profile()])
|
||||||
|
self.session.commit()
|
||||||
|
retirement_provider = manifest.migration_spec.retirement_provider
|
||||||
|
assert retirement_provider is not None
|
||||||
|
plan = retirement_provider(self.session, "files")
|
||||||
|
assert plan.destroy_data_executor is not None
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||||
|
) as audit:
|
||||||
|
plan.destroy_data_executor(self.session, "files")
|
||||||
|
# The retirement executor deliberately enlists secret scrubbing, audit
|
||||||
|
# writes, and DDL in the caller-owned transaction. Commit that unit of
|
||||||
|
# work before inspecting through a separate Engine connection.
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
self.assertEqual(2, audit.call_count)
|
||||||
|
self.assertEqual(
|
||||||
|
{"module_data_retired"},
|
||||||
|
{call.kwargs["details"]["deletion_reason"] for call in audit.call_args_list},
|
||||||
|
)
|
||||||
|
self.assertNotIn("do-not-audit", repr(audit.call_args_list))
|
||||||
|
self.assertFalse(inspect(self.engine).has_table(FileConnectorCredential.__tablename__))
|
||||||
|
self.assertFalse(inspect(self.engine).has_table(FileConnectorProfile.__tablename__))
|
||||||
|
|
||||||
|
def test_retirement_detaches_unowned_external_reference_before_drop(self) -> None:
|
||||||
|
from govoplan_files.backend.manifest import manifest
|
||||||
|
|
||||||
|
self.session.add(self._credential(secret_ref="vault:tenant-1:files:credential"))
|
||||||
|
self.session.commit()
|
||||||
|
retirement_provider = manifest.migration_spec.retirement_provider
|
||||||
|
assert retirement_provider is not None
|
||||||
|
plan = retirement_provider(self.session, "files")
|
||||||
|
assert plan.destroy_data_executor is not None
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||||
|
) as audit:
|
||||||
|
plan.destroy_data_executor(self.session, "files")
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
audit.assert_called_once()
|
||||||
|
details = audit.call_args.kwargs["details"]
|
||||||
|
self.assertIn("unowned_external_secret_ref", details["removed_reference_kinds"])
|
||||||
|
self.assertNotIn("vault:tenant-1:files:credential", repr(audit.call_args))
|
||||||
|
self.assertFalse(inspect(self.engine).has_table(FileConnectorCredential.__tablename__))
|
||||||
|
self.assertFalse(inspect(self.engine).has_table(FileConnectorProfile.__tablename__))
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _credential(*, secret_ref: str | None = None) -> FileConnectorCredential:
|
||||||
|
return FileConnectorCredential(
|
||||||
|
id="credential-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
scope_type="tenant",
|
||||||
|
scope_id="tenant-1",
|
||||||
|
label="Credential",
|
||||||
|
provider="webdav",
|
||||||
|
enabled=True,
|
||||||
|
credential_mode="basic",
|
||||||
|
username="connector-user",
|
||||||
|
password_encrypted=encrypt_secret("do-not-audit-password"),
|
||||||
|
token_encrypted=encrypt_secret("do-not-audit-token"),
|
||||||
|
password_env="DEPLOYMENT_PASSWORD",
|
||||||
|
token_env="DEPLOYMENT_TOKEN",
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
policy={},
|
||||||
|
metadata_={"private_hint": "do-not-audit-metadata"},
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _profile(*, credential_profile_id: str | None = None) -> FileConnectorProfile:
|
||||||
|
return FileConnectorProfile(
|
||||||
|
id="profile-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
scope_type="tenant",
|
||||||
|
scope_id="tenant-1",
|
||||||
|
label="Profile",
|
||||||
|
provider="webdav",
|
||||||
|
endpoint_url="https://dav.example.test",
|
||||||
|
enabled=True,
|
||||||
|
credential_profile_id=credential_profile_id,
|
||||||
|
credential_mode="basic",
|
||||||
|
username="connector-user",
|
||||||
|
password_encrypted=encrypt_secret("do-not-audit-profile-password"),
|
||||||
|
token_encrypted=encrypt_secret("do-not-audit-profile-token"),
|
||||||
|
password_env="DEPLOYMENT_PROFILE_PASSWORD",
|
||||||
|
token_env="DEPLOYMENT_PROFILE_TOKEN",
|
||||||
|
policy={},
|
||||||
|
metadata_={"private_hint": "do-not-audit-profile-metadata"},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _assert_credential_scrubbed(self, row: FileConnectorCredential) -> None:
|
||||||
|
self.assertFalse(row.enabled)
|
||||||
|
self.assertEqual("none", row.credential_mode)
|
||||||
|
self.assertIsNone(row.username)
|
||||||
|
self.assertIsNone(row.password_encrypted)
|
||||||
|
self.assertIsNone(row.token_encrypted)
|
||||||
|
self.assertIsNone(row.password_env)
|
||||||
|
self.assertIsNone(row.token_env)
|
||||||
|
self.assertIsNone(row.secret_ref)
|
||||||
|
self.assertEqual({}, row.metadata_)
|
||||||
|
|
||||||
|
def _assert_profile_scrubbed(self, row: FileConnectorProfile) -> None:
|
||||||
|
self.assertFalse(row.enabled)
|
||||||
|
self.assertIsNone(row.credential_profile_id)
|
||||||
|
self.assertEqual("none", row.credential_mode)
|
||||||
|
self.assertIsNone(row.username)
|
||||||
|
self.assertIsNone(row.password_encrypted)
|
||||||
|
self.assertIsNone(row.token_encrypted)
|
||||||
|
self.assertIsNone(row.password_env)
|
||||||
|
self.assertIsNone(row.token_env)
|
||||||
|
self.assertIsNone(row.secret_ref)
|
||||||
|
self.assertEqual({}, row.metadata_)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -81,6 +81,37 @@ class ConnectorDeploymentBoundaryTests(unittest.TestCase):
|
|||||||
)
|
)
|
||||||
session.get.assert_not_called()
|
session.get.assert_not_called()
|
||||||
|
|
||||||
|
def test_api_profiles_cannot_create_unowned_external_secret_references(self) -> None:
|
||||||
|
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "provider ownership"):
|
||||||
|
reject_api_controlled_deployment_references(secret_ref="vault:tenant-1:files")
|
||||||
|
|
||||||
|
session = MagicMock()
|
||||||
|
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "provider ownership"):
|
||||||
|
create_connector_profile_row(
|
||||||
|
session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
user_id="user-1",
|
||||||
|
profile_id="unsafe-secret-ref",
|
||||||
|
label="Unsafe secret reference",
|
||||||
|
provider="webdav",
|
||||||
|
scope_type="tenant",
|
||||||
|
credential_mode="secret_ref",
|
||||||
|
secret_ref="vault:tenant-1:files",
|
||||||
|
)
|
||||||
|
session.get.assert_not_called()
|
||||||
|
|
||||||
|
def test_api_profiles_cannot_hide_plaintext_credentials_in_metadata(self) -> None:
|
||||||
|
for metadata in (
|
||||||
|
{"secret_access_key": "plaintext-secret"},
|
||||||
|
{"credentials": {"password": "plaintext-secret"}},
|
||||||
|
{"provider": {"refresh_token": "plaintext-secret"}},
|
||||||
|
):
|
||||||
|
with self.subTest(metadata=metadata), self.assertRaisesRegex(
|
||||||
|
ConnectorDeploymentConfigurationError,
|
||||||
|
"dedicated encrypted credential fields",
|
||||||
|
):
|
||||||
|
reject_api_controlled_deployment_references(metadata=metadata)
|
||||||
|
|
||||||
def test_profile_responses_hide_environment_and_local_ca_references(self) -> None:
|
def test_profile_responses_hide_environment_and_local_ca_references(self) -> None:
|
||||||
profile = ConnectorProfile(
|
profile = ConnectorProfile(
|
||||||
id="deployment-s3",
|
id="deployment-s3",
|
||||||
|
|||||||
176
tests/test_connector_profile_store.py
Normal file
176
tests/test_connector_profile_store.py
Normal file
@@ -0,0 +1,176 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_access.backend.db import models as access_models # noqa: F401 - resolve Files user foreign keys
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorProfile
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_profile_store import (
|
||||||
|
list_database_connector_profiles,
|
||||||
|
update_connector_profile_row,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorProfileStoreUpdateTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.engine = create_engine("sqlite:///:memory:")
|
||||||
|
Base.metadata.create_all(self.engine, tables=[FileConnectorProfile.__table__])
|
||||||
|
self.session = sessionmaker(bind=self.engine)()
|
||||||
|
self.row = FileConnectorProfile(
|
||||||
|
id="profile-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
scope_type="tenant",
|
||||||
|
scope_id="tenant-1",
|
||||||
|
label="Original profile",
|
||||||
|
provider="webdav",
|
||||||
|
endpoint_url="https://dav.example.test/original",
|
||||||
|
base_path="original",
|
||||||
|
enabled=True,
|
||||||
|
credential_mode="basic",
|
||||||
|
username="original-user",
|
||||||
|
password_encrypted=encrypt_secret("original-password"),
|
||||||
|
token_encrypted=encrypt_secret("original-token"),
|
||||||
|
capabilities=["browse"],
|
||||||
|
policy={"allow": {"providers": ["webdav"]}},
|
||||||
|
metadata_={"original": True},
|
||||||
|
created_by_user_id=None,
|
||||||
|
updated_by_user_id=None,
|
||||||
|
)
|
||||||
|
self.session.add(self.row)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
Base.metadata.drop_all(bind=self.engine)
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_update_preserves_normalization_and_flush_only_transaction_boundary(self) -> None:
|
||||||
|
updated = update_connector_profile_row(
|
||||||
|
self.session,
|
||||||
|
self.row,
|
||||||
|
user_id="user-2",
|
||||||
|
label=" Updated profile ",
|
||||||
|
provider="NEXTCLOUD",
|
||||||
|
endpoint_url=" https://cloud.example.test/dav ",
|
||||||
|
base_path=" shared/reports ",
|
||||||
|
enabled=False,
|
||||||
|
credential_profile_id=" credential-2 ",
|
||||||
|
credential_mode="TOKEN",
|
||||||
|
username=" updated-user ",
|
||||||
|
password="updated-password",
|
||||||
|
token="updated-token",
|
||||||
|
capabilities=["browse", " import ", ""],
|
||||||
|
policy={"deny": {"external_paths": ["private"]}},
|
||||||
|
metadata={"department": "reports"},
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertIs(self.row, updated)
|
||||||
|
self.assertEqual("Updated profile", updated.label)
|
||||||
|
self.assertEqual("nextcloud", updated.provider)
|
||||||
|
self.assertEqual("https://cloud.example.test/dav", updated.endpoint_url)
|
||||||
|
self.assertEqual("shared/reports", updated.base_path)
|
||||||
|
self.assertFalse(updated.enabled)
|
||||||
|
self.assertEqual("credential-2", updated.credential_profile_id)
|
||||||
|
self.assertEqual("token", updated.credential_mode)
|
||||||
|
self.assertEqual("updated-user", updated.username)
|
||||||
|
self.assertEqual("updated-password", decrypt_secret(updated.password_encrypted))
|
||||||
|
self.assertEqual("updated-token", decrypt_secret(updated.token_encrypted))
|
||||||
|
self.assertEqual(["browse", "import"], updated.capabilities)
|
||||||
|
self.assertEqual({"deny": {"external_paths": ["private"]}}, updated.policy)
|
||||||
|
self.assertEqual({"department": "reports"}, updated.metadata_)
|
||||||
|
self.assertEqual("user-2", updated.updated_by_user_id)
|
||||||
|
|
||||||
|
self.session.rollback()
|
||||||
|
persisted = self.session.get(FileConnectorProfile, self.row.id)
|
||||||
|
assert persisted is not None
|
||||||
|
self.assertEqual("Original profile", persisted.label)
|
||||||
|
self.assertEqual("original-password", decrypt_secret(persisted.password_encrypted))
|
||||||
|
self.assertTrue(persisted.enabled)
|
||||||
|
|
||||||
|
def test_secret_replacement_wins_over_clear_while_clear_removes_an_omitted_token(self) -> None:
|
||||||
|
update_connector_profile_row(
|
||||||
|
self.session,
|
||||||
|
self.row,
|
||||||
|
user_id="user-2",
|
||||||
|
password="replacement-password",
|
||||||
|
clear_password=True,
|
||||||
|
clear_token=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual("replacement-password", decrypt_secret(self.row.password_encrypted))
|
||||||
|
self.assertIsNone(self.row.token_encrypted)
|
||||||
|
self.assertEqual("https://dav.example.test/original", self.row.endpoint_url)
|
||||||
|
self.assertEqual("original-user", self.row.username)
|
||||||
|
|
||||||
|
def test_legacy_external_secret_reference_cannot_be_cleared_or_partially_mutate_the_row(self) -> None:
|
||||||
|
self.row.secret_ref = "vault:tenant-1:files:profile"
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
with self.assertRaisesRegex(FileStorageError, "provider-side deletion"):
|
||||||
|
update_connector_profile_row(
|
||||||
|
self.session,
|
||||||
|
self.row,
|
||||||
|
user_id="user-2",
|
||||||
|
label="Must not be applied",
|
||||||
|
secret_ref="",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual("Original profile", self.row.label)
|
||||||
|
self.assertEqual("vault:tenant-1:files:profile", self.row.secret_ref)
|
||||||
|
|
||||||
|
def test_flush_failure_remains_rollback_safe(self) -> None:
|
||||||
|
with patch.object(self.session, "flush", side_effect=RuntimeError("database unavailable")), self.assertRaisesRegex(
|
||||||
|
RuntimeError,
|
||||||
|
"database unavailable",
|
||||||
|
):
|
||||||
|
update_connector_profile_row(
|
||||||
|
self.session,
|
||||||
|
self.row,
|
||||||
|
user_id="user-2",
|
||||||
|
label="Uncommitted profile",
|
||||||
|
password="uncommitted-password",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.session.rollback()
|
||||||
|
persisted = self.session.get(FileConnectorProfile, self.row.id)
|
||||||
|
assert persisted is not None
|
||||||
|
self.assertEqual("Original profile", persisted.label)
|
||||||
|
self.assertEqual("original-password", decrypt_secret(persisted.password_encrypted))
|
||||||
|
|
||||||
|
def test_visibility_filter_runs_before_connector_secrets_are_decrypted(self) -> None:
|
||||||
|
invisible = FileConnectorProfile(
|
||||||
|
id="invisible-profile",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
scope_type="user",
|
||||||
|
scope_id="another-user",
|
||||||
|
label="Invisible profile",
|
||||||
|
provider="webdav",
|
||||||
|
endpoint_url="https://invisible.example.test",
|
||||||
|
enabled=True,
|
||||||
|
credential_mode="basic",
|
||||||
|
password_encrypted="not-a-valid-encrypted-secret",
|
||||||
|
capabilities=["browse"],
|
||||||
|
policy={},
|
||||||
|
metadata_={},
|
||||||
|
)
|
||||||
|
self.session.add(invisible)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
profiles = list_database_connector_profiles(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
row_visible=lambda row: row.id == self.row.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual([self.row.id], [profile.id for profile in profiles])
|
||||||
|
self.assertEqual("original-password", profiles[0].password_value)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
265
tests/test_connector_visibility.py
Normal file
265
tests/test_connector_visibility.py
Normal file
@@ -0,0 +1,265 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from dataclasses import replace
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
from govoplan_files.backend.storage.connector_visibility import (
|
||||||
|
connector_profile_usable_for_import,
|
||||||
|
visible_connector_profiles_for_actor,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _profile(
|
||||||
|
profile_id: str,
|
||||||
|
*,
|
||||||
|
scope_type: str = "system",
|
||||||
|
scope_id: str | None = None,
|
||||||
|
provider: str = "webdav",
|
||||||
|
enabled: bool = True,
|
||||||
|
) -> ConnectorProfile:
|
||||||
|
return ConnectorProfile(
|
||||||
|
id=profile_id,
|
||||||
|
label=profile_id,
|
||||||
|
provider=provider,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
endpoint_url=f"https://{profile_id}.example.invalid",
|
||||||
|
enabled=enabled,
|
||||||
|
credential_mode="anonymous",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorVisibilityTests(unittest.TestCase):
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources",
|
||||||
|
return_value=[],
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings"
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
|
||||||
|
)
|
||||||
|
def test_profiles_are_filtered_to_actor_scopes_and_database_definition_wins(
|
||||||
|
self,
|
||||||
|
database_profiles,
|
||||||
|
configured_profiles,
|
||||||
|
_policy_sources,
|
||||||
|
) -> None:
|
||||||
|
profiles = [
|
||||||
|
_profile("system"),
|
||||||
|
_profile("tenant", scope_type="tenant", scope_id="tenant-1"),
|
||||||
|
_profile("other-tenant", scope_type="tenant", scope_id="tenant-2"),
|
||||||
|
_profile("user", scope_type="user", scope_id="user-1"),
|
||||||
|
_profile("other-user", scope_type="user", scope_id="user-2"),
|
||||||
|
_profile("group", scope_type="group", scope_id="group-1"),
|
||||||
|
_profile("campaign", scope_type="campaign", scope_id="campaign-1"),
|
||||||
|
_profile("disabled", enabled=False),
|
||||||
|
_profile("duplicate", provider="webdav"),
|
||||||
|
]
|
||||||
|
database_profiles.return_value = (profiles, {profile.id for profile in profiles})
|
||||||
|
configured_profiles.return_value = [_profile("duplicate", provider="nextcloud")]
|
||||||
|
|
||||||
|
visible = visible_connector_profiles_for_actor(
|
||||||
|
object(), # type: ignore[arg-type]
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
user_id="user-1",
|
||||||
|
group_ids={"group-1"},
|
||||||
|
settings=object(),
|
||||||
|
campaign_visible=lambda campaign_id: campaign_id == "campaign-1",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(
|
||||||
|
{"system", "tenant", "user", "group", "campaign", "duplicate"},
|
||||||
|
{profile.id for profile in visible},
|
||||||
|
)
|
||||||
|
duplicate = next(profile for profile in visible if profile.id == "duplicate")
|
||||||
|
self.assertEqual("webdav", duplicate.provider)
|
||||||
|
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources",
|
||||||
|
return_value=[],
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings",
|
||||||
|
return_value=[],
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
|
||||||
|
)
|
||||||
|
def test_campaign_and_admin_visibility_remain_explicit(
|
||||||
|
self,
|
||||||
|
database_profiles,
|
||||||
|
_configured_profiles,
|
||||||
|
_policy_sources,
|
||||||
|
) -> None:
|
||||||
|
profiles = [
|
||||||
|
_profile("campaign-a", scope_type="campaign", scope_id="campaign-a"),
|
||||||
|
_profile("campaign-b", scope_type="campaign", scope_id="campaign-b"),
|
||||||
|
_profile("other-user", scope_type="user", scope_id="user-2"),
|
||||||
|
]
|
||||||
|
database_profiles.return_value = (profiles, {profile.id for profile in profiles})
|
||||||
|
|
||||||
|
campaign_only = visible_connector_profiles_for_actor(
|
||||||
|
object(), # type: ignore[arg-type]
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
user_id="user-1",
|
||||||
|
group_ids=(),
|
||||||
|
settings=None,
|
||||||
|
campaign_id="campaign-a",
|
||||||
|
campaign_visible=lambda _campaign_id: True,
|
||||||
|
)
|
||||||
|
self.assertEqual(["campaign-a"], [profile.id for profile in campaign_only])
|
||||||
|
|
||||||
|
admin = visible_connector_profiles_for_actor(
|
||||||
|
object(), # type: ignore[arg-type]
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
user_id="user-1",
|
||||||
|
group_ids=(),
|
||||||
|
settings=None,
|
||||||
|
campaign_visible=None,
|
||||||
|
include_admin_scopes=True,
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
{"campaign-a", "campaign-b", "other-user"},
|
||||||
|
{profile.id for profile in admin},
|
||||||
|
)
|
||||||
|
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings",
|
||||||
|
return_value=[],
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources"
|
||||||
|
)
|
||||||
|
def test_effective_policy_is_attached_without_mutating_profile(
|
||||||
|
self,
|
||||||
|
policy_sources,
|
||||||
|
database_profiles,
|
||||||
|
_configured_profiles,
|
||||||
|
) -> None:
|
||||||
|
profile = _profile("profile")
|
||||||
|
source = ConnectorPolicySource(
|
||||||
|
scope_type="system",
|
||||||
|
label="System",
|
||||||
|
policy={"allow": {"providers": ["webdav"]}},
|
||||||
|
)
|
||||||
|
database_profiles.return_value = ([profile], {profile.id})
|
||||||
|
policy_sources.return_value = [source]
|
||||||
|
|
||||||
|
visible = visible_connector_profiles_for_actor(
|
||||||
|
object(), # type: ignore[arg-type]
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
user_id="user-1",
|
||||||
|
group_ids=(),
|
||||||
|
settings=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual((), profile.policy_sources)
|
||||||
|
self.assertEqual((source,), visible[0].policy_sources)
|
||||||
|
|
||||||
|
def test_import_usability_requires_safe_provider_credentials_endpoint_and_identity_policy(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
self.assertTrue(connector_profile_usable_for_import(_profile("webdav")))
|
||||||
|
self.assertFalse(
|
||||||
|
connector_profile_usable_for_import(_profile("s3", provider="s3"))
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
connector_profile_usable_for_import(_profile("smb", provider="smb"))
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
connector_profile_usable_for_import(
|
||||||
|
_profile("sharepoint", provider="sharepoint")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
connector_profile_usable_for_import(
|
||||||
|
ConnectorProfile(
|
||||||
|
id="no-endpoint",
|
||||||
|
label="No endpoint",
|
||||||
|
provider="webdav",
|
||||||
|
credential_mode="anonymous",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
connector_profile_usable_for_import(
|
||||||
|
ConnectorProfile(
|
||||||
|
id="no-credentials",
|
||||||
|
label="No credentials",
|
||||||
|
provider="webdav",
|
||||||
|
endpoint_url="https://files.example.invalid",
|
||||||
|
credential_mode="basic",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertTrue(
|
||||||
|
connector_profile_usable_for_import(
|
||||||
|
ConnectorProfile(
|
||||||
|
id="metadata-endpoint",
|
||||||
|
label="Metadata endpoint",
|
||||||
|
provider="webdav",
|
||||||
|
credential_mode="anonymous",
|
||||||
|
metadata={"webdav_endpoint_url": "https://files.example.invalid"},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
connector_profile_usable_for_import(
|
||||||
|
ConnectorProfile(
|
||||||
|
id="unresolved-secret-reference",
|
||||||
|
label="Unresolved secret reference",
|
||||||
|
provider="webdav",
|
||||||
|
endpoint_url="https://files.example.invalid",
|
||||||
|
credential_mode="basic",
|
||||||
|
secret_ref="vault://files/webdav",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
denied = replace(
|
||||||
|
_profile("denied"),
|
||||||
|
policy_sources=(
|
||||||
|
ConnectorPolicySource(
|
||||||
|
scope_type="system",
|
||||||
|
label="System",
|
||||||
|
policy={"deny": {"providers": ["webdav"]}},
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertFalse(connector_profile_usable_for_import(denied))
|
||||||
|
|
||||||
|
path_limited = replace(
|
||||||
|
_profile("path-limited"),
|
||||||
|
policy_sources=(
|
||||||
|
ConnectorPolicySource(
|
||||||
|
scope_type="system",
|
||||||
|
label="System",
|
||||||
|
policy={"allow": {"external_paths": ["/approved/*"]}},
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertFalse(connector_profile_usable_for_import(path_limited))
|
||||||
|
|
||||||
|
endpoint_allowed = replace(
|
||||||
|
_profile("endpoint-allowed"),
|
||||||
|
policy_sources=(
|
||||||
|
ConnectorPolicySource(
|
||||||
|
scope_type="system",
|
||||||
|
label="System",
|
||||||
|
policy={"allow": {"external_urls": ["https://*.example.invalid"]}},
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertTrue(connector_profile_usable_for_import(endpoint_allowed))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
241
tests/test_documentation.py
Normal file
241
tests/test_documentation.py
Normal file
@@ -0,0 +1,241 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.modules import DocumentationContext
|
||||||
|
from govoplan_files.backend.documentation import documentation_topics
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
|
||||||
|
|
||||||
|
class _Principal:
|
||||||
|
tenant_id = "tenant-1"
|
||||||
|
user = SimpleNamespace(id="user-1")
|
||||||
|
group_ids = frozenset({"group-1"})
|
||||||
|
|
||||||
|
def __init__(self, scopes: set[str]) -> None:
|
||||||
|
self.scopes = frozenset(scopes)
|
||||||
|
|
||||||
|
def has(self, scope: str) -> bool:
|
||||||
|
return scope in self.scopes
|
||||||
|
|
||||||
|
|
||||||
|
class FilesRuntimeDocumentationTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.session = Session()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
|
||||||
|
def context(
|
||||||
|
self,
|
||||||
|
scopes: set[str],
|
||||||
|
*,
|
||||||
|
settings: object | None = None,
|
||||||
|
documentation_type: str = "user",
|
||||||
|
) -> DocumentationContext:
|
||||||
|
return DocumentationContext(
|
||||||
|
registry=object(),
|
||||||
|
principal=_Principal(scopes),
|
||||||
|
settings=settings,
|
||||||
|
session=self.session,
|
||||||
|
documentation_type=documentation_type, # type: ignore[arg-type]
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_upload_tasks_state_exact_current_safe_limits(self) -> None:
|
||||||
|
settings = SimpleNamespace(
|
||||||
|
file_upload_max_bytes=7 * 1024 * 1024,
|
||||||
|
file_upload_zip_max_bytes=19 * 1024 * 1024,
|
||||||
|
)
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
|
||||||
|
return_value=[],
|
||||||
|
):
|
||||||
|
topics = {
|
||||||
|
topic.id: topic
|
||||||
|
for topic in documentation_topics(
|
||||||
|
self.context(
|
||||||
|
{"files:file:read", "files:file:upload"}, settings=settings
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
upload = topics["files.workflow.upload-managed-files"]
|
||||||
|
self.assertIn("7 MiB (7,340,032 bytes)", upload.body)
|
||||||
|
self.assertEqual(["files.list"], upload.metadata["help_contexts"])
|
||||||
|
self.assertEqual(
|
||||||
|
{"files:file:read", "files:file:upload"},
|
||||||
|
set(upload.conditions[0].required_scopes),
|
||||||
|
)
|
||||||
|
|
||||||
|
archive = topics["files.workflow.upload-and-unpack-zip"]
|
||||||
|
self.assertIn("19 MiB (19,922,944 bytes)", archive.body)
|
||||||
|
self.assertIn("7 MiB (7,340,032 bytes)", archive.body)
|
||||||
|
self.assertIn("1,000", archive.body)
|
||||||
|
self.assertIn("Actual extracted bytes are counted", archive.body)
|
||||||
|
|
||||||
|
def test_connector_task_requires_authority_and_a_visible_usable_profile(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
usable = ConnectorProfile(
|
||||||
|
id="private-profile-id",
|
||||||
|
label="Private profile label",
|
||||||
|
provider="webdav",
|
||||||
|
scope_type="tenant",
|
||||||
|
scope_id="tenant-1",
|
||||||
|
endpoint_url="https://private.example.invalid/secret-root",
|
||||||
|
base_path="/secret-root",
|
||||||
|
credential_mode="anonymous",
|
||||||
|
)
|
||||||
|
context = self.context({"files:file:read", "files:file:upload"})
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
|
||||||
|
return_value=[usable],
|
||||||
|
) as visible:
|
||||||
|
topics = {topic.id: topic for topic in documentation_topics(context)}
|
||||||
|
|
||||||
|
self.assertIn("files.workflow.import-managed-snapshot", topics)
|
||||||
|
self.assertNotIn("files.connector-import-unavailable", topics)
|
||||||
|
task = topics["files.workflow.import-managed-snapshot"]
|
||||||
|
self.assertEqual("configured", task.layer)
|
||||||
|
self.assertEqual(
|
||||||
|
["files.list", "files.connector-import"], task.metadata["help_contexts"]
|
||||||
|
)
|
||||||
|
self.assertIn("re-authorized", task.body)
|
||||||
|
self.assertNotIn("private-profile-id", repr(task))
|
||||||
|
self.assertNotIn("private.example.invalid", repr(task))
|
||||||
|
self.assertNotIn("secret-root", repr(task))
|
||||||
|
visible.assert_called_once()
|
||||||
|
self.assertEqual(("group-1",), tuple(visible.call_args.kwargs["group_ids"]))
|
||||||
|
|
||||||
|
without_authority = {
|
||||||
|
topic.id: topic
|
||||||
|
for topic in documentation_topics(self.context({"files:file:read"}))
|
||||||
|
}
|
||||||
|
self.assertNotIn("files.workflow.import-managed-snapshot", without_authority)
|
||||||
|
self.assertIn(
|
||||||
|
"both permission to view Files and permission to upload",
|
||||||
|
without_authority["files.connector-import-unavailable"].body,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_connector_limitation_is_precise_but_never_exposes_profile_internals(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
blocked = ConnectorProfile(
|
||||||
|
id="sensitive-profile-id",
|
||||||
|
label="Sensitive label",
|
||||||
|
provider="s3",
|
||||||
|
scope_type="tenant",
|
||||||
|
scope_id="tenant-1",
|
||||||
|
endpoint_url="https://internal.example.invalid/private",
|
||||||
|
base_path="/classified",
|
||||||
|
credential_mode="basic",
|
||||||
|
password_value="credential-secret",
|
||||||
|
)
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
|
||||||
|
return_value=[blocked],
|
||||||
|
):
|
||||||
|
topics = {
|
||||||
|
topic.id: topic
|
||||||
|
for topic in documentation_topics(
|
||||||
|
self.context({"files:file:read", "files:file:upload"})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
self.assertNotIn("files.workflow.import-managed-snapshot", topics)
|
||||||
|
limitation = topics["files.connector-import-unavailable"]
|
||||||
|
self.assertEqual("available", limitation.layer)
|
||||||
|
self.assertIn("pinning-safe provider", limitation.body)
|
||||||
|
payload = repr(limitation)
|
||||||
|
for secret in (
|
||||||
|
"sensitive-profile-id",
|
||||||
|
"Sensitive label",
|
||||||
|
"internal.example.invalid",
|
||||||
|
"classified",
|
||||||
|
"credential-secret",
|
||||||
|
):
|
||||||
|
self.assertNotIn(secret, payload)
|
||||||
|
|
||||||
|
def test_files_admin_connector_groups_do_not_widen_campaign_membership(self) -> None:
|
||||||
|
usable = ConnectorProfile(
|
||||||
|
id="group-profile",
|
||||||
|
label="Group profile",
|
||||||
|
provider="webdav",
|
||||||
|
scope_type="group",
|
||||||
|
scope_id="admin-visible-group",
|
||||||
|
endpoint_url="https://files.example.invalid",
|
||||||
|
credential_mode="anonymous",
|
||||||
|
)
|
||||||
|
|
||||||
|
def groups_for_actor(_session, *, include_admin_groups, **_kwargs):
|
||||||
|
return ["member-group", "admin-visible-group"] if include_admin_groups else ["member-group"]
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch(
|
||||||
|
"govoplan_files.backend.documentation.user_group_ids",
|
||||||
|
side_effect=groups_for_actor,
|
||||||
|
),
|
||||||
|
patch(
|
||||||
|
"govoplan_files.backend.documentation._campaign_visibility",
|
||||||
|
return_value=None,
|
||||||
|
) as campaign_visibility,
|
||||||
|
patch(
|
||||||
|
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
|
||||||
|
return_value=[usable],
|
||||||
|
) as visible,
|
||||||
|
):
|
||||||
|
topics = {
|
||||||
|
topic.id: topic
|
||||||
|
for topic in documentation_topics(
|
||||||
|
self.context(
|
||||||
|
{
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:upload",
|
||||||
|
"files:file:admin",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
self.assertIn("files.workflow.import-managed-snapshot", topics)
|
||||||
|
self.assertEqual(
|
||||||
|
("member-group", "admin-visible-group"),
|
||||||
|
tuple(visible.call_args.kwargs["group_ids"]),
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
("member-group",),
|
||||||
|
tuple(campaign_visibility.call_args.kwargs["group_ids"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_connector_evaluation_errors_fail_closed_without_error_details(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
with patch(
|
||||||
|
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
|
||||||
|
side_effect=RuntimeError("private endpoint /root credential-id"),
|
||||||
|
):
|
||||||
|
topics = {
|
||||||
|
topic.id: topic
|
||||||
|
for topic in documentation_topics(
|
||||||
|
self.context({"files:file:read", "files:file:upload"})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
self.assertNotIn("files.workflow.import-managed-snapshot", topics)
|
||||||
|
limitation = topics["files.connector-import-unavailable"]
|
||||||
|
self.assertIn("could not be safely evaluated", limitation.body)
|
||||||
|
self.assertNotIn("private endpoint", repr(limitation))
|
||||||
|
self.assertNotIn("credential-id", repr(limitation))
|
||||||
|
|
||||||
|
def test_runtime_provider_only_emits_user_documentation(self) -> None:
|
||||||
|
self.assertEqual(
|
||||||
|
(), documentation_topics(self.context(set(), documentation_type="admin"))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -3,18 +3,207 @@ from __future__ import annotations
|
|||||||
import unittest
|
import unittest
|
||||||
|
|
||||||
|
|
||||||
|
STATIC_TOPIC_IDS = {
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
"files.workflow.share-managed-files",
|
||||||
|
"files.workflow.delete-managed-files",
|
||||||
|
"files.governed-connectors-and-provenance",
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
|
"files.assurance.process-and-release-readiness",
|
||||||
|
}
|
||||||
|
RUNTIME_TOPIC_IDS = {
|
||||||
|
"files.workflow.upload-managed-files",
|
||||||
|
"files.workflow.upload-and-unpack-zip",
|
||||||
|
"files.workflow.import-managed-snapshot",
|
||||||
|
"files.connector-import-unavailable",
|
||||||
|
}
|
||||||
|
HANDBOOK_HREF = "govoplan-files/docs/FILES_HANDBOOK.md"
|
||||||
|
|
||||||
|
|
||||||
class FilesManifestDocumentationTests(unittest.TestCase):
|
class FilesManifestDocumentationTests(unittest.TestCase):
|
||||||
def test_connector_topic_covers_governance_pinning_and_provenance_perspectives(self) -> None:
|
@classmethod
|
||||||
|
def setUpClass(cls) -> None:
|
||||||
from govoplan_files.backend.manifest import manifest
|
from govoplan_files.backend.manifest import manifest
|
||||||
|
|
||||||
topic = next(item for item in manifest.documentation if item.id == "files.governed-connectors-and-provenance")
|
cls.manifest = manifest
|
||||||
|
cls.topics = {topic.id: topic for topic in manifest.documentation}
|
||||||
|
|
||||||
|
def topic(self, topic_id: str):
|
||||||
|
return self.topics[topic_id]
|
||||||
|
|
||||||
|
def test_static_topics_have_role_scope_module_and_link_contracts(self) -> None:
|
||||||
|
self.assertEqual(STATIC_TOPIC_IDS, set(self.topics))
|
||||||
|
self.assertEqual(
|
||||||
|
{"admin", "user"},
|
||||||
|
{
|
||||||
|
item
|
||||||
|
for topic in self.topics.values()
|
||||||
|
for item in topic.documentation_types
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
for topic in self.topics.values():
|
||||||
|
with self.subTest(topic=topic.id):
|
||||||
|
self.assertTrue(topic.audience)
|
||||||
|
self.assertTrue(topic.conditions)
|
||||||
|
self.assertTrue(
|
||||||
|
any(
|
||||||
|
"files" in condition.required_modules
|
||||||
|
for condition in topic.conditions
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertTrue(
|
||||||
|
any(
|
||||||
|
condition.any_scopes or condition.required_scopes
|
||||||
|
for condition in topic.conditions
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertIn("runtime", {link.kind for link in topic.links})
|
||||||
|
self.assertIn(
|
||||||
|
HANDBOOK_HREF,
|
||||||
|
{link.href for link in topic.links if link.kind == "repository"},
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertIn(
|
||||||
|
"documentation_topics",
|
||||||
|
{provider.__name__ for provider in self.manifest.documentation_providers},
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_user_tasks_are_independently_authorized_and_contextual(self) -> None:
|
||||||
|
expected_scopes = {
|
||||||
|
"files.workflow.organize-managed-files": {
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:organize",
|
||||||
|
},
|
||||||
|
"files.workflow.find-and-download-files": {
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:download",
|
||||||
|
},
|
||||||
|
"files.workflow.share-managed-files": {
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:share",
|
||||||
|
},
|
||||||
|
"files.workflow.delete-managed-files": {
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:delete",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for topic_id, scopes in expected_scopes.items():
|
||||||
|
with self.subTest(topic=topic_id):
|
||||||
|
topic = self.topic(topic_id)
|
||||||
|
self.assertEqual(("user",), topic.documentation_types)
|
||||||
|
self.assertEqual("workflow", topic.metadata["kind"])
|
||||||
|
self.assertEqual(scopes, set(topic.conditions[0].required_scopes))
|
||||||
|
self.assertEqual(["files.list"], topic.metadata["help_contexts"])
|
||||||
|
for key in ("prerequisites", "steps", "outcome", "verification"):
|
||||||
|
self.assertTrue(topic.metadata[key])
|
||||||
|
|
||||||
|
def test_share_and_delete_tasks_state_current_boundaries(self) -> None:
|
||||||
|
share = self.topic("files.workflow.share-managed-files")
|
||||||
|
self.assertEqual("available", share.layer)
|
||||||
|
self.assertIn("does not yet provide a general share editor", share.body)
|
||||||
|
self.assertIn("no share-revocation route", share.body)
|
||||||
|
self.assertIn(
|
||||||
|
"/api/v1/files/{file_id}/shares", {link.href for link in share.links}
|
||||||
|
)
|
||||||
|
|
||||||
|
delete = self.topic("files.workflow.delete-managed-files")
|
||||||
|
self.assertIn("Soft-delete", delete.summary)
|
||||||
|
self.assertIn("no self-service restore or hard-purge", delete.body)
|
||||||
|
self.assertTrue(
|
||||||
|
any("not a hard purge" in item for item in delete.metadata["limitations"])
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_admin_topic_covers_policy_redaction_and_atomic_credential_deletion(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
topic = self.topic("files.governed-connectors-and-provenance")
|
||||||
|
|
||||||
|
self.assertEqual(("admin",), topic.documentation_types)
|
||||||
|
self.assertEqual("reference", topic.metadata["kind"])
|
||||||
|
self.assertEqual("File connections", topic.metadata["screen"])
|
||||||
|
self.assertTrue(topic.metadata["section"])
|
||||||
|
self.assertIn("deny rules win", topic.body)
|
||||||
|
self.assertIn("redact secret values", topic.body)
|
||||||
|
self.assertIn("same transaction", topic.body)
|
||||||
|
self.assertTrue(
|
||||||
|
any(
|
||||||
|
"non-owned external references" in item
|
||||||
|
for item in topic.metadata["security_invariants"]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"/api/v1/files/connectors/policies/tenant",
|
||||||
|
{link.href for link in topic.links},
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"/api/v1/files/connectors/credentials", {link.href for link in topic.links}
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_operator_topic_covers_recovery_and_fail_closed_s3_smb(self) -> None:
|
||||||
|
topic = self.topic(
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports"
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(("admin",), topic.documentation_types)
|
||||||
|
self.assertEqual("reference", topic.metadata["kind"])
|
||||||
|
self.assertIn("S3", topic.body)
|
||||||
|
self.assertIn("SMB", topic.body)
|
||||||
|
self.assertIn("fail closed", topic.body)
|
||||||
|
self.assertIn("DFS referrals", topic.body)
|
||||||
|
self.assertIn("does not remove backend blob objects", topic.body)
|
||||||
|
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
|
||||||
|
self.assertTrue(topic.metadata["verification"])
|
||||||
|
self.assertIn(
|
||||||
|
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", topic.configuration_keys
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_integrator_topic_exposes_capability_and_provenance_boundary(self) -> None:
|
||||||
|
topic = self.topic("files.reference.snapshot-provenance-and-capabilities")
|
||||||
|
|
||||||
self.assertEqual(("admin", "user"), topic.documentation_types)
|
self.assertEqual(("admin", "user"), topic.documentation_types)
|
||||||
self.assertEqual({"user", "admin", "operator"}, set(topic.metadata["perspectives"]))
|
self.assertEqual("reference", topic.metadata["kind"])
|
||||||
self.assertIn("fails closed", topic.body)
|
self.assertEqual(
|
||||||
self.assertIn("DFS referrals", topic.body)
|
["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
|
||||||
|
topic.metadata["provided_interfaces"],
|
||||||
|
)
|
||||||
self.assertIn("revision", topic.metadata["provenance_fields"])
|
self.assertIn("revision", topic.metadata["provenance_fields"])
|
||||||
self.assertIn("/api/v1/files/connectors/profiles", {link.href for link in topic.links})
|
self.assertIn("exact asset, version, blob, checksum", topic.body)
|
||||||
|
self.assertIn("collaboration", topic.body)
|
||||||
|
|
||||||
|
def test_process_and_release_assurance_exposes_gates_evidence_and_limits(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
topic = self.topic("files.assurance.process-and-release-readiness")
|
||||||
|
|
||||||
|
self.assertEqual(("admin", "user"), topic.documentation_types)
|
||||||
|
self.assertEqual("workflow", topic.metadata["kind"])
|
||||||
|
self.assertEqual(["files.list"], topic.metadata["help_contexts"])
|
||||||
|
self.assertIn("process_owner", topic.audience)
|
||||||
|
self.assertIn("release_manager", topic.audience)
|
||||||
|
self.assertIn("versions align", topic.body)
|
||||||
|
self.assertIn("no general share-management UI or share revocation", topic.body)
|
||||||
|
self.assertIn("no enforced retention or legal hold", topic.body)
|
||||||
|
for key in ("prerequisites", "steps", "outcome", "verification"):
|
||||||
|
self.assertTrue(topic.metadata[key])
|
||||||
|
self.assertTrue(
|
||||||
|
any("meta-repository security" in step for step in topic.metadata["steps"])
|
||||||
|
)
|
||||||
|
self.assertTrue(any("fails closed" in step for step in topic.metadata["steps"]))
|
||||||
|
self.assertTrue(any("SHA-256" in step for step in topic.metadata["steps"]))
|
||||||
|
|
||||||
|
def test_internal_related_topic_ids_resolve(self) -> None:
|
||||||
|
known_ids = STATIC_TOPIC_IDS | RUNTIME_TOPIC_IDS
|
||||||
|
for topic in self.topics.values():
|
||||||
|
for related_id in topic.metadata.get("related_topic_ids", []):
|
||||||
|
if related_id.startswith("files."):
|
||||||
|
self.assertIn(
|
||||||
|
related_id,
|
||||||
|
known_ids,
|
||||||
|
f"{topic.id} refers to missing topic {related_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@govoplan/files-webui",
|
"name": "@govoplan/files-webui",
|
||||||
"version": "0.1.8",
|
"version": "0.1.9",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
|
|||||||
Reference in New Issue
Block a user