GovOPlaN/govoplan-files
2
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: govern file lifecycle and connector writes

Browse Source
This commit is contained in:
zemion
2026-08-20 22:06:57 +02:00
parent 95aef18955
commit 6c3cf1c55e
27 changed files with 2721 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FILES_HANDBOOK.md
+115 -42
View File
@@ -1,6 +1,6 @@
# GovOPlaN Files Handbook
This handbook describes the Files module as implemented in version `0.1.9`.
This handbook describes the Files module as implemented in version `0.1.18`.
It is the operational source of truth for users, process owners, administrators,
operators, auditors, and module integrators. Statements about future behavior
are marked **planned**; an unmarked statement describes the current code.
@@ -8,7 +8,8 @@ are marked **planned**; an unmarked statement describes the current code.
Files is a governed snapshot store. It owns managed file content, versions,
logical folders, shares, source provenance, and the evidence that another
GovOPlaN module used a particular file version. It can browse selected external
stores read-only and import a frozen copy. It is not a general remote filesystem,
stores and import a frozen copy. Explicit, conditional S3 write-back is available
only through an administrator-enabled two-way connector space. Files is not a general remote filesystem,
a document collaboration engine, or a records-management system.
## Choose a reading path
@@ -50,15 +51,15 @@ The main domain objects are:
| Object | Meaning | Lifecycle today |
| --- | --- | --- |
| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted |
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained |
| File blob | Stored plaintext semantics plus stored-byte integrity, backend key, optional Encryption envelope, reference count, and retention timestamp | Reused only within a tenant and matching protection boundary; no automated garbage collection |
| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted |
| File asset | The user-facing file identity, owner, logical path, description, metadata, current version, retention deadline, legal hold, and lifecycle revision | Created, organized, shared, soft-deleted, restored, or governed-purged |
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained until its asset is eligible for purge |
| File blob | Stored plaintext semantics plus stored-byte integrity, backend key, optional Encryption envelope, reference count, and retention timestamp | Reused only within a tenant and matching protection boundary; collected only after a fenced reference recheck |
| Folder | An explicit logical path in a user or group space | Created, moved/renamed, soft-deleted, and restored when its original path is free |
| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet |
| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion |
| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion |
| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use |
| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted |
| Connector space | A manually synchronized remote folder/library linked to a user or group space; read-only by default, with explicit S3 two-way mode available | Created, updated, disabled, soft-deleted, and restored |
| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence |
| Form evidence upload grant | A one-use, hash-only bearer grant tied to an exact Form instance/revision, purpose, custodian, size, and media-type policy | Issued for at most 15 minutes, consumed by one managed upload, then retained as evidence provenance |
@@ -105,9 +106,12 @@ Files administrator; governed API operations can administer other tenant-owned
Files resources when their owner is specified. Linked connector spaces appear
beside managed spaces when they are active and visible to the user.
A connector space is intentionally read-only. It is a view of an approved
remote location and a starting point for importing or synchronizing selected
files into managed storage; it is not a mounted write-through filesystem.
A connector space is read-only by default. It is a view of an approved remote
location and a starting point for importing or synchronizing selected files
into managed storage. An administrator may opt an S3 space into two-way mode
only after enabling the profile's `write` capability. Even then, Files exposes
explicit, conditional file write-back—not a mounted filesystem. Automatic
remote delete, rename, move, and ACL propagation stay disabled.
### Upload files
@@ -170,6 +174,40 @@ Recursive folder deletion is the default. It soft-deletes the selected folder,
its child folders, and files below it. A non-recursive delete fails when the
folder is not empty.
### Restore, retain, hold, and purge
Soft deletion is reversible for callers with `files:file:restore`. Restoring a
file preserves its asset identity, complete version history, blob references,
and source provenance. Recursive folder restore reactivates the folder tree and
its deleted files. Connector-space restore reactivates only the local link.
Each restore fails if an active resource already occupies the original path or
label.
Retention and legal hold are independent lifecycle controls. A caller with
`files:file:retention` supplies the current lifecycle revision, a reason, an
optional retained-until time, and legal-hold state. Stale revisions fail rather
than overwriting a concurrent decision.
Hard purge is deliberately separate from ordinary delete:
1. A caller with `files:file:purge` previews 1–100 soft-deleted assets.
2. The preview reports retention, legal-hold, active-share, Campaign-evidence,
and Form-evidence blockers and returns a SHA-256 over current lifecycle and
blob-reference state.
3. Execution requires that exact hash, a stable idempotency key, an approval
reference, and the literal `PURGE` confirmation.
4. The recovery-ledger operation and tenant purge fence are durable before the
irreversible database transaction. A stale preview or blocker rejects the
operation without erasure.
5. Purge removes eligible asset, version, and inactive-share rows and
recalculates blob reference counts. It does not delete bytes inline.
6. A separately authorized bounded blob-GC call takes the same per-blob lease
used by uploads, rechecks all `FileVersion` references, deletes the exact
object, verifies absence, and only then deletes `FileBlob` metadata.
Automatic time-based purge scheduling is not implemented. Operators initiate
preview, execute, and garbage collection under their local retention process.
### Find and download files
Files can list by owner and path, use cursor pagination, and consume incremental
@@ -210,7 +248,14 @@ inside the chosen owner space:
- identical checksum and size updates provenance and returns `unchanged`;
- changed bytes append a version and return `updated`.
Browse, import, and sync never write, rename, or delete the remote source.
Browse, import, and inbound sync never mutate the remote source. An S3 space in
explicit two-way mode can write a selected managed file to one remote object
path through `POST /api/v1/files/connector-spaces/{space_id}/write-back`.
Creating a path uses `If-None-Match`; overwriting requires the currently
observed ETag or version and uses a conditional request. The provider object is
then re-read and must contain both the expected content digest and the recovery
operation marker. Uncertain or mismatching outcomes remain fenced and visible
in Ops. Remote deletion, rename, move, and permission propagation are disabled.
Connector administration separates endpoint profiles, reusable credentials,
and inherited policy. Ordinary setup uses typed fields and provider discovery;
@@ -320,6 +365,10 @@ those relationships first.
| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces |
| `files:file:share` | Create or update file shares |
| `files:file:delete` | Soft-delete accessible writable files and folders |
| `files:file:restore` | Restore owned or administered soft-deleted files, folders, and connector-space links |
| `files:file:retention` | Set retention deadlines and legal holds with optimistic revision checks |
| `files:file:purge` | Preview and execute irreversible metadata purge and collect unreferenced blobs |
| `files:connector:write` | Write an accessible managed file to an explicitly writable connector space |
| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant |
The `file_manager` role template grants all normal file operations except
@@ -339,12 +388,14 @@ Keep these definitions separate:
local policy, and descriptive operation capabilities;
- a **policy** restricts what a scope may configure or use;
- a **connector space** links one approved profile/library/path to one user or
group and always uses manual, read-only synchronization today.
group, uses manual synchronization, and is read-only unless explicitly
configured for the supported two-way S3 write boundary.
Profile capability values such as `browse`, `import`, and `sync` are stored and
returned, but they are descriptive today. Provider implementation and policy
checks enforce actual availability; do not use the capability list as the sole
security control.
Profile capability values such as `browse`, `import`, `sync`, and `write` are
stored and returned. `write` is additionally enforced for S3 write-back, but
provider implementation, connector-space mode, operation permission, resource
access, and inherited policy all remain mandatory; never use the capability
list as the sole security control.
Profiles, credentials, and policies support `system`, `tenant`, `user`,
`group`, and `campaign` scopes. A normal user sees system and active-tenant
@@ -429,7 +480,7 @@ in a development/test runtime.
| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport |
| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport |
| SMB | Read-only browse/import/manual sync implemented through a pinned smbprotocol transport for initial peers, reconnects, aliases, and DFS referral targets |
| S3 connector | Read-only bucket/prefix browse, import, and manual sync implemented through pinned botocore pools covering retries, redirects, endpoint discovery, and provider aliases |
| S3 connector | Bucket/prefix browse, import, manual inbound sync, and explicit conditional write-back implemented through pinned botocore pools covering retries, redirects, endpoint discovery, and provider aliases; automatic remote delete/rename/move/ACL propagation disabled |
| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned |
| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature |
@@ -567,9 +618,9 @@ with a dry-run preview and requires separate destructive confirmation. The
confirmation reuses the finding revision from that preview, rechecks that no
database reference exists, remains scoped to the scanned tenant prefix, and is
idempotent. Both applied and dry-run actions emit audit evidence. A shared
reference blocks deletion. Files currently has no legal-hold or hard-purge
model, so retention-controlled objects must not be treated as cleanup
candidates until those controls are implemented.
reference blocks orphan cleanup. Managed-asset purge separately enforces asset
retention, legal hold, active shares, and retained Campaign/Form evidence before
releasing a blob for fenced garbage collection.
### Recovery ledger for object effects
@@ -614,6 +665,22 @@ and the durable finding state are verified afterward. If the caller transaction
rolls back after deletion, Files may forward-complete only that existing
finding after rechecking that the key is still unreferenced.
Hard purge uses an irreversible Core recovery plan with an approval reference
and a tenant-wide purge lease. The preview hash binds target IDs, deletion
state, lifecycle revisions, holds, deadlines, blockers, and blob IDs. Asset and
version removal plus audit evidence commit together. Released objects are not
deleted by that transaction. Blob GC later takes `files:blob:<tenant>:<blob>`,
the same distributed resource fence as upload/repair, locks and rechecks the
blob, and verifies object absence before removing metadata. Provider or database
ambiguity remains recovery-required or outcome-unknown in Ops.
Explicit S3 connector writes use forward recovery because the provider cannot
join the database transaction. The durable request contains only tenant,
profile, opaque target digest, content SHA-256/size, and expected revision. A
conditional create or overwrite is followed by a metadata probe for the exact
content and operation markers. Never retry an unresolved target with another
request key; reconcile the owning Files operation from provider evidence first.
Archive preview and confirmation use bounded process-local temporary staging.
Staging is not authoritative and is removed on every handled exit; extracted
members enter the same per-blob recovery boundary as direct uploads. A hard
@@ -621,11 +688,9 @@ process loss may leave a temporary OS file for normal host temporary-file
cleanup, but cannot make that staging path a managed Files object.
Use the Ops recovery-operation view to inspect `files` operations. Do not retry
a busy or unresolved blob blindly: first verify the FileBlob row, object hash,
integrity state, and any Encryption envelope named by the blob. Hard purge,
legal hold, and two-way remote connector mutation are not implemented yet, so
they cannot claim recovery-ledger adoption; their owning work remains tracked
separately.
a busy or unresolved blob or connector path blindly: first verify the FileBlob
row and object hash, or the remote request/content markers and revision, plus
any Encryption envelope named by a managed blob.
After restore:
@@ -715,17 +780,19 @@ All routes below are under `/api/v1/files`.
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` |
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}`, `POST /connector-spaces/{space_id}/restore`, `POST /connector-spaces/{space_id}/write-back` |
| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` |
| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` |
| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` |
| Restore and lifecycle | `POST /assets/{file_id}/restore`, `POST /folders/restore`, `PATCH /{file_id}/lifecycle` |
| Governed erasure | `POST /purge/preview`, `POST /purge/execute`, `POST /purge/blobs` |
| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` |
| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` |
| Incremental connector settings | `GET /connectors/settings/delta` |
| Form evidence | `POST /form-evidence/upload` with a short-lived `X-Form-Evidence-Token` issued by Forms Runtime |
The Files workspace exposes **Remove space** only for read-only connector
spaces and only to actors with file-organization authority over the owning user
The Files workspace exposes **Remove space** only for connector spaces and only
to actors with file-organization authority over the owning user
or group space. Confirmation explains the exact boundary: removal soft-deletes
the local connector-space definition and makes that virtual view disappear. It
does not mutate or delete remote provider content, previously imported managed
@@ -847,6 +914,9 @@ The word "delete" has different meanings by object type:
| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata |
| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed |
| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility |
| Governed hard purge | Removes only preview-matched, soft-deleted asset/version rows without active retention, legal hold, active shares, Campaign evidence, or Form evidence |
| Blob garbage collection | Deletes an exact managed object only after a fresh zero-reference check under the shared blob fence; metadata follows verified object absence |
| Connector write-back | Explicit conditional S3 create/overwrite only; automatic provider delete, rename, move, and ACL propagation remain disabled |
Credential/profile scrubbing and its audit event use the same database
transaction. If audit creation fails, the deletion rolls back. Repeating a
@@ -854,16 +924,19 @@ delete against an already scrubbed tombstone does not recreate secret evidence.
### Retention boundary
File versions and blobs are effectively retained indefinitely today. Although a
blob has `ref_count` and `retained_until` fields, no complete retention-policy,
legal-hold, hard-purge, or garbage-collection service enforces them. There is
also no supported user restore endpoint for soft-deleted assets/folders.
Integrity reconciliation is operator-triggered and is not a retention or
automatic garbage-collection policy.
Each asset has an enforceable retained-until value, legal-hold flag, reason, and
optimistic lifecycle revision. These controls block hard purge; they do not
automatically schedule it. Restore, purge preview/execute, and blob GC are
explicit authorized operations. Blob `retained_until` remains an additional
storage-level safeguard and must also have expired before automated collection
is introduced. Campaign and Form evidence are hard blockers rather than
silently cascaded references.
Do not promise erasure, timed retention, legal hold, or self-service recovery
from the current soft-delete behavior. Those require an explicit, auditable
retention/purge design that preserves campaign and other evidence references.
Do not equate soft deletion with erasure. Erasure is complete only after the
approved purge removes asset/version metadata, bounded GC verifies that no
version references the blob and removes its bytes/metadata, and any owning
Encryption retention/key-custody consequence has been handled under that
module's policy.
## Acceptance scenarios
@@ -942,15 +1015,15 @@ returning different content or credentials.
| --- | --- | --- |
| Managed storage | Core local/S3 backend, exact managed-Garage or explicitly trusted HTTPS external S3, state-profile validation, fallback local read roots, tenant blob deduplication, checksums, bounded resumable integrity scans, quarantine, dry-run-first orphan cleanup, and Core-ledger verification/forward recovery | Scheduled scan execution and deployment-specific S3 HA/backup automation |
| Upload | Bounded direct upload, drag-and-drop UI, archive preview/selective extraction, password-protected ZIP support, explicit conflicts, opaque new object keys, and rollback compensation | Malware scanning, quotas, type policy, resumable/chunked upload |
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution, and API restoration preserving versions/provenance | General file-history UI and user-driven append-version UI |
| Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles |
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/38)) |
| Deletion/retention | Soft-delete and restore assets/folders/spaces; optimistic retention and legal-hold controls; preview-bound, approval-referenced hard purge; reference-checked blob GC; immediate audited connector-secret scrubbing | Automatic time-based purge scheduling and richer lifecycle administration UI |
| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected |
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers |
| SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, and redirect/retry/referral transport-contract tests | Live topology smoke evidence, provider-specific OAuth, remote writes, and background indexing remain separate deployment or connector-module concerns |
| SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, redirect/retry/referral transport-contract tests, and explicit conditional S3 write-back with Core-ledger recovery | Live topology smoke evidence, provider-specific OAuth, additional provider writes, and background indexing remain separate deployment or connector-module concerns |
| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors |
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs |
| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate |
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete/restore, read-only default, and opt-in S3 two-way mode | Background sync and automatic remote delete/rename/move/ACL propagation |
| Profile capabilities | Stored, displayed, and enforced for explicit connector write-back | Broader provider-specific capability negotiation |
| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation |
| Preview | File metadata and attachment download | Dedicated safe content-preview service |
| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign |