diff --git a/README.md b/README.md index 02b2271..42d61a2 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,15 @@ for example for managed attachment selection and campaign file sharing. Files does not import campaign internals; campaign share/existence checks use the core `campaigns.access` capability registered by the campaign module. +Files also publishes the optional `privacy.dsar.files` capability. The Core +data-subject workflow can use it to collect bounded, tenant-scoped file, +version, share, folder, evidence, and non-secret configuration metadata for a +direct membership subject. The provider may revoke a subject-targeted share or +detach mutable actor references idempotently, but it never exports credential +material or raw bytes and never bypasses Files retention, legal hold, evidence, +purge approval, audit, or recovery controls. See the handbook's data-subject +request coverage section for the review and erasure boundary. + Platform RBAC and governance rules are documented in `govoplan-core/docs/`. Managed files can carry source provenance for connector and import workflows. diff --git a/docs/FILES_HANDBOOK.md b/docs/FILES_HANDBOOK.md index 4bb3175..4b01819 100644 --- a/docs/FILES_HANDBOOK.md +++ b/docs/FILES_HANDBOOK.md @@ -208,6 +208,39 @@ Hard purge is deliberately separate from ordinary delete: Automatic time-based purge scheduling is not implemented. Operators initiate preview, execute, and garbage collection under their local retention process. +### Data-subject request coverage + +Files registers `privacy.dsar.files` when the module is active. The provider +requires a corroborated tenant membership identifier (or a namespaced Files +user reference), searches only that tenant, and fails explicitly if its bounded +result limit would be exceeded. It reports managed assets, exact versions, +folders, user-targeted shares, Form and Campaign evidence, connector +configuration actor references, and integrity-operation evidence. + +The export includes only governed metadata. It never embeds file bytes, blob +storage keys, passwords, tokens, environment-variable names, secret-provider +references, or encrypted connector values. A reviewer follows the authorized +version download route when the file itself must be inspected. + +The erasure plan deliberately separates four outcomes: + +- an active share aimed at the subject can be revoked idempotently; +- mutable creator/updater references can be detached after tenant, subject, and + current-value revalidation; +- legal hold, active retention, submitted Form evidence, Campaign delivery + evidence, connector configuration history, and integrity evidence are + retained with a reason; and +- unstructured file content, ownership, filenames, and paths require manual + review. + +DSAR execution never invokes physical byte deletion. If the privacy decision +authorizes erasure, the operator must use the separate Files soft-delete, purge +preview, approval, execution, and blob-GC sequence. This preserves its distinct +authority, evidence blockers, audit trail, distributed fencing, and recovery +ledger semantics. Email, account, or identity selectors alone are insufficient +because Files does not import the Access directory; the Access search supplies +the corroborated membership reference for Files coverage. + ### Find and download files Files can list by owner and path, use cursor pagination, and consume incremental @@ -1018,6 +1051,7 @@ returning different content or credentials. | Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution, and API restoration preserving versions/provenance | General file-history UI and user-driven append-version UI | | Sharing | User/group/tenant/campaign grants, expiry, idempotent revocation, searchable share-management UI, and campaign linkage display | Richer policy-driven share lifecycles | | Deletion/retention | Soft-delete and restore assets/folders/spaces; optimistic retention and legal-hold controls; preview-bound, approval-referenced hard purge; reference-checked blob GC; immediate audited connector-secret scrubbing | Automatic time-based purge scheduling and richer lifecycle administration UI | +| Privacy requests | Tenant-scoped bounded DSAR metadata search; retained/manual/revoke/detach planning; idempotent share revocation and mutable actor-reference detachment; explicit separation from byte purge | Content-specific automated redaction and policy-specific approval remain manual or belong to the owning process | | Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected | | HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers | | SMB and S3 connectors | Provider descriptors, browse/import/manual sync, pinned SDK transports, redirect/retry/referral transport-contract tests, and explicit conditional S3 write-back with Core-ledger recovery | Live topology smoke evidence, provider-specific OAuth, additional provider writes, and background indexing remain separate deployment or connector-module concerns | @@ -1053,7 +1087,10 @@ Before releasing Files: and applied orphan cleanup; inspect their `files` operations in Ops. 11. Exercise a Form evidence grant, token replay, wrong-submission reference, expired token, unsupported media type, and quarantined-file rejection. -12. Update the implemented/planned table whenever a boundary changes. +12. Exercise a Files DSAR search/plan, repeat an approved reversible action, and + confirm retained file content can only be erased through the separate purge + authority and recovery path. +13. Update the implemented/planned table whenever a boundary changes. ## Related documents diff --git a/src/govoplan_files/backend/dsar_provider.py b/src/govoplan_files/backend/dsar_provider.py new file mode 100644 index 0000000..1a99520 --- /dev/null +++ b/src/govoplan_files/backend/dsar_provider.py @@ -0,0 +1,879 @@ +from __future__ import annotations + +from collections.abc import Sequence +from datetime import datetime, timezone + +from sqlalchemy import inspect, or_ +from sqlalchemy.orm import Session + +from govoplan_core.core.dsar import ( + DsarErasureActionRef, + DsarExecutionResultRef, + DsarRecordRef, + DsarSubjectRef, + dsar_capability_name, +) +from govoplan_files.backend.db.models import ( + CampaignAttachmentUse, + FileAsset, + FileConnectorCredential, + FileConnectorPolicy, + FileConnectorProfile, + FileConnectorSpace, + FileFolder, + FileFormEvidenceGrant, + FileIntegrityFinding, + FileIntegrityScan, + FileShare, + FileVersion, +) + + +FILES_DSAR_CAPABILITY = dsar_capability_name("files") +_MAX_RECORDS = 5_000 + + +class FilesDsarProvider: + provider_id = "files" + module_id = "files" + + def search_subject( + self, + session: object, + *, + tenant_id: str, + subject: DsarSubjectRef, + ) -> Sequence[DsarRecordRef]: + db = _session(session) + subject_user_id = _subject_user_id(subject) + if subject_user_id is None: + return () + + records: list[DsarRecordRef] = [] + form_evidence_available = _has_table(db, FileFormEvidenceGrant) + campaign_evidence_available = _has_table(db, CampaignAttachmentUse) + retention_reasons: dict[str, str | None] = {} + + def retention_reason(asset: FileAsset) -> str | None: + if asset.id not in retention_reasons: + retention_reasons[asset.id] = _asset_retention_reason( + db, + asset, + form_evidence_available=form_evidence_available, + campaign_evidence_available=campaign_evidence_available, + ) + return retention_reasons[asset.id] + + def append(record: DsarRecordRef) -> None: + if len(records) >= _MAX_RECORDS: + raise ValueError( + "Files DSAR match limit exceeded; narrow the subject selectors." + ) + records.append(record) + + assets = _bounded_rows( + db.query(FileAsset) + .filter( + FileAsset.tenant_id == tenant_id, + or_( + FileAsset.owner_user_id == subject_user_id, + FileAsset.created_by_user_id == subject_user_id, + ), + ) + .order_by(FileAsset.id) + ) + for asset in assets: + match_fields = _matching_fields( + asset, + subject_user_id, + ("owner_user_id", "created_by_user_id"), + ) + asset_retention_reason = retention_reason(asset) + append( + _record( + "file_asset", + asset.id, + "managed_file", + asset.filename, + { + "match_fields": match_fields, + "owner_type": asset.owner_type, + "display_path": asset.display_path, + "filename": asset.filename, + "description": asset.description, + "deleted_at": _iso(asset.deleted_at), + "retained_until": _iso(asset.retained_until), + "legal_hold": asset.legal_hold, + "lifecycle_reason": asset.lifecycle_reason, + "lifecycle_revision": asset.lifecycle_revision, + }, + observed_at=asset.updated_at, + immutable=asset_retention_reason is not None, + retention_reason=asset_retention_reason, + source_path=f"/files?file={asset.id}", + ) + ) + + version_rows = _bounded_rows( + db.query(FileVersion, FileAsset) + .join(FileAsset, FileAsset.id == FileVersion.file_asset_id) + .filter( + FileVersion.tenant_id == tenant_id, + FileAsset.tenant_id == tenant_id, + or_( + FileAsset.owner_user_id == subject_user_id, + FileVersion.created_by_user_id == subject_user_id, + ), + ) + .order_by(FileVersion.id) + ) + for version, asset in version_rows: + asset_retention_reason = retention_reason(asset) + append( + _record( + "file_version", + version.id, + "managed_file_version", + version.filename_at_upload, + { + "match_fields": ( + ["asset.owner_user_id"] + if asset.owner_user_id == subject_user_id + else [] + ) + + ( + ["created_by_user_id"] + if version.created_by_user_id == subject_user_id + else [] + ), + "file_asset_id": version.file_asset_id, + "version_number": version.version_number, + "filename_at_upload": version.filename_at_upload, + "display_path_at_upload": version.display_path_at_upload, + "content_type": version.content_type, + "size_bytes": version.size_bytes, + "checksum_sha256": version.checksum_sha256, + "created_at": _iso(version.created_at), + }, + observed_at=version.updated_at, + immutable=asset_retention_reason is not None, + retention_reason=asset_retention_reason, + source_path=( + f"/api/v1/files/{version.file_asset_id}/versions/" + f"{version.id}/download" + ), + ) + ) + + for folder in _bounded_rows( + db.query(FileFolder) + .filter( + FileFolder.tenant_id == tenant_id, + or_( + FileFolder.owner_user_id == subject_user_id, + FileFolder.created_by_user_id == subject_user_id, + ), + ) + .order_by(FileFolder.id) + ): + append( + _record( + "file_folder", + folder.id, + "managed_folder", + folder.path, + { + "match_fields": _matching_fields( + folder, + subject_user_id, + ("owner_user_id", "created_by_user_id"), + ), + "owner_type": folder.owner_type, + "path": folder.path, + "deleted_at": _iso(folder.deleted_at), + }, + observed_at=folder.updated_at, + source_path="/files", + ) + ) + + for share in _bounded_rows( + db.query(FileShare) + .filter( + FileShare.tenant_id == tenant_id, + or_( + (FileShare.target_type == "user") + & (FileShare.target_id == subject_user_id), + FileShare.created_by_user_id == subject_user_id, + FileShare.revoked_by_user_id == subject_user_id, + ), + ) + .order_by(FileShare.id) + ): + target_matches = ( + share.target_type == "user" and share.target_id == subject_user_id + ) + match_fields = _matching_fields( + share, + subject_user_id, + ("created_by_user_id", "revoked_by_user_id"), + ) + if target_matches: + match_fields.insert(0, "target_id") + append( + _record( + "file_share", + share.id, + "file_access_evidence", + f"Share for file {share.file_asset_id}", + { + "match_fields": match_fields, + "file_asset_id": share.file_asset_id, + "target_type": share.target_type, + "target_is_subject": target_matches, + "permission": share.permission, + "expires_at": _iso(share.expires_at), + "revoked_at": _iso(share.revoked_at), + }, + observed_at=share.updated_at, + immutable=True, + retention_reason=( + "File-sharing history is institutional access evidence." + ), + source_path=f"/files?file={share.file_asset_id}", + ) + ) + + if form_evidence_available: + evidence_rows = _bounded_rows( + db.query(FileFormEvidenceGrant) + .outerjoin( + FileAsset, + FileAsset.id == FileFormEvidenceGrant.file_asset_id, + ) + .filter( + FileFormEvidenceGrant.tenant_id == tenant_id, + or_( + FileFormEvidenceGrant.custodian_user_id == subject_user_id, + FileAsset.owner_user_id == subject_user_id, + ), + ) + .order_by(FileFormEvidenceGrant.id) + ) + for grant in evidence_rows: + append( + _record( + "file_form_evidence", + grant.id, + "form_evidence", + f"Form evidence {grant.form_definition_id}", + { + "match_fields": ( + ["custodian_user_id"] + if grant.custodian_user_id == subject_user_id + else ["asset.owner_user_id"] + ), + "form_instance_id": grant.form_instance_id, + "form_definition_id": grant.form_definition_id, + "form_definition_revision": grant.form_definition_revision, + "evidence_kind": grant.evidence_kind, + "purpose": grant.purpose, + "status": grant.status, + "expires_at": _iso(grant.expires_at), + "file_asset_id": grant.file_asset_id, + "file_version_id": grant.file_version_id, + }, + observed_at=grant.updated_at, + immutable=True, + retention_reason=( + "Submitted Form attachment evidence follows the owning " + "process retention and cannot be erased through Files alone." + ), + ) + ) + + if campaign_evidence_available: + campaign_rows = _bounded_rows( + db.query(CampaignAttachmentUse) + .join(FileAsset, FileAsset.id == CampaignAttachmentUse.file_asset_id) + .filter( + CampaignAttachmentUse.tenant_id == tenant_id, + FileAsset.owner_user_id == subject_user_id, + ) + .order_by(CampaignAttachmentUse.id) + ) + for use in campaign_rows: + append( + _record( + "campaign_attachment_use", + use.id, + "delivery_evidence", + use.filename_used, + { + "match_fields": ["asset.owner_user_id"], + "campaign_id": use.campaign_id, + "campaign_version_id": use.campaign_version_id, + "campaign_job_id": use.campaign_job_id, + "file_asset_id": use.file_asset_id, + "file_version_id": use.file_version_id, + "filename_used": use.filename_used, + "checksum_sha256": use.checksum_sha256, + "size_bytes": use.size_bytes, + "use_stage": use.use_stage, + "used_at": _iso(use.used_at), + }, + observed_at=use.updated_at, + immutable=True, + retention_reason=( + "Campaign attachment use is immutable delivery evidence." + ), + ) + ) + + self._append_configuration_references( + db, + append=append, + tenant_id=tenant_id, + subject_user_id=subject_user_id, + ) + self._append_integrity_references( + db, + append=append, + tenant_id=tenant_id, + subject_user_id=subject_user_id, + ) + return tuple(records) + + def plan_erasure( + self, + session: object, + *, + tenant_id: str, + subject: DsarSubjectRef, + records: Sequence[DsarRecordRef], + ) -> Sequence[DsarErasureActionRef]: + del session + subject_user_id = _subject_user_id(subject) + if subject_user_id is None: + return () + actions: list[DsarErasureActionRef] = [] + for record in records: + if record.provider_id != self.provider_id or record.module_id != self.module_id: + raise ValueError("Files DSAR received a foreign provider record.") + match_fields = { + str(value) for value in record.data.get("match_fields", ()) + } + if record.immutable_evidence: + actions.append( + _action( + f"files:retain:{record.resource_type}:{record.resource_id}", + "retain", + record, + f"Retain {record.title}", + record.retention_reason + or "Institutional evidence must be retained.", + executable=False, + ) + ) + elif record.resource_type in { + "file_asset", + "file_version", + "file_folder", + }: + actions.append( + _action( + f"files:review:{record.resource_type}:{record.resource_id}", + "manual_review", + record, + f"Review {record.title}", + ( + "Managed file content, names, paths, ownership, and shared " + "references require a case decision. Approved byte erasure " + "must use the separate Files purge workflow." + ), + executable=False, + ) + ) + if ( + record.resource_type == "file_share" + and "target_id" in match_fields + and record.data.get("revoked_at") is None + ): + actions.append( + _action( + f"files:revoke:file_share:{record.resource_id}", + "revoke", + record, + "Revoke active file share", + "The active user-targeted share can be revoked without deleting file evidence.", + executable=True, + metadata={"subject_user_id": subject_user_id}, + ) + ) + for field_name in sorted( + match_fields.intersection(_detachable_fields(record.resource_type)) + ): + actions.append( + _action( + ( + f"files:detach:{record.resource_type}:" + f"{field_name}:{record.resource_id}" + ), + "detach", + record, + f"Detach {field_name.replace('_', ' ')}", + ( + "Remove the mutable subject reference while preserving the " + "governed resource and DSAR evidence." + ), + executable=True, + metadata={ + "field": field_name, + "subject_user_id": subject_user_id, + }, + ) + ) + action_ids = [action.action_id for action in actions] + if len(action_ids) != len(set(action_ids)): + raise ValueError("Files DSAR produced duplicate action ids.") + return tuple(actions) + + def execute_erasure( + self, + session: object, + *, + tenant_id: str, + subject: DsarSubjectRef, + actions: Sequence[DsarErasureActionRef], + request_id: str, + ) -> Sequence[DsarExecutionResultRef]: + db = _session(session) + subject_user_id = _subject_user_id(subject) + if subject_user_id is None: + return tuple( + _blocked(action, "Files requires a direct membership subject reference.") + for action in actions + ) + results: list[DsarExecutionResultRef] = [] + for action in actions: + if ( + action.provider_id != self.provider_id + or action.module_id != self.module_id + or action.metadata.get("subject_user_id") != subject_user_id + ): + results.append(_blocked(action, "The Files DSAR action is stale or invalid.")) + continue + if action.action_id.startswith("files:revoke:file_share:"): + results.append( + _revoke_share( + db, + tenant_id=tenant_id, + subject_user_id=subject_user_id, + action=action, + request_id=request_id, + ) + ) + elif action.action_id.startswith("files:detach:"): + results.append( + _detach_reference( + db, + tenant_id=tenant_id, + subject_user_id=subject_user_id, + action=action, + request_id=request_id, + ) + ) + else: + results.append(_blocked(action, "Files does not execute this action kind.")) + db.flush() + return tuple(results) + + def _append_configuration_references( + self, + db: Session, + *, + append: object, + tenant_id: str, + subject_user_id: str, + ) -> None: + configurations = ( + (FileConnectorProfile, "connector_profile", ("created_by_user_id", "updated_by_user_id")), + (FileConnectorCredential, "connector_credential", ("created_by_user_id", "updated_by_user_id")), + (FileConnectorPolicy, "connector_policy", ("created_by_user_id", "updated_by_user_id")), + (FileConnectorSpace, "connector_space", ("owner_user_id", "created_by_user_id")), + ) + for model, resource_type, fields in configurations: + if not _has_table(db, model): + continue + conditions = [getattr(model, field) == subject_user_id for field in fields] + query = db.query(model).filter(or_(*conditions)) + if hasattr(model, "tenant_id"): + query = query.filter(model.tenant_id == tenant_id) + for row in _bounded_rows(query.order_by(model.id)): + match_fields = _matching_fields(row, subject_user_id, fields) + data: dict[str, object] = { + "match_fields": match_fields, + "label": getattr(row, "label", None), + "provider": getattr(row, "provider", None), + } + if isinstance(row, FileConnectorCredential): + data["credential_mode"] = row.credential_mode + elif isinstance(row, FileConnectorSpace): + data.update( + { + "remote_path": row.remote_path, + "sync_mode": row.sync_mode, + "read_only": row.read_only, + "deleted_at": _iso(row.deleted_at), + } + ) + append( # type: ignore[operator] + _record( + resource_type, + row.id, + "connector_configuration_evidence", + getattr(row, "label", None) or resource_type.replace("_", " "), + data, + observed_at=row.updated_at, + immutable=True, + retention_reason=( + "Connector configuration history is institutional evidence; " + "credential secrets are excluded from the DSAR export." + ), + ) + ) + + def _append_integrity_references( + self, + db: Session, + *, + append: object, + tenant_id: str, + subject_user_id: str, + ) -> None: + if _has_table(db, FileIntegrityScan): + for scan in _bounded_rows( + db.query(FileIntegrityScan) + .filter( + FileIntegrityScan.tenant_id == tenant_id, + FileIntegrityScan.created_by_user_id == subject_user_id, + ) + .order_by(FileIntegrityScan.id) + ): + append( # type: ignore[operator] + _record( + "file_integrity_scan", + scan.id, + "storage_integrity_evidence", + f"Integrity scan {scan.id}", + { + "match_fields": ["created_by_user_id"], + "storage_backend": scan.storage_backend, + "status": scan.status, + "started_at": _iso(scan.started_at), + "completed_at": _iso(scan.completed_at), + }, + observed_at=scan.updated_at, + immutable=True, + retention_reason="Storage integrity scans are operator evidence.", + ) + ) + if _has_table(db, FileIntegrityFinding): + for finding in _bounded_rows( + db.query(FileIntegrityFinding) + .filter( + FileIntegrityFinding.tenant_id == tenant_id, + FileIntegrityFinding.resolved_by_user_id == subject_user_id, + ) + .order_by(FileIntegrityFinding.id) + ): + append( # type: ignore[operator] + _record( + "file_integrity_finding", + finding.id, + "storage_integrity_evidence", + f"Integrity finding {finding.kind}", + { + "match_fields": ["resolved_by_user_id"], + "kind": finding.kind, + "state": finding.state, + "resolved_at": _iso(finding.resolved_at), + }, + observed_at=finding.updated_at, + immutable=True, + retention_reason="Integrity resolution is operator evidence.", + ) + ) + + +def _asset_retention_reason( + session: Session, + asset: FileAsset, + *, + form_evidence_available: bool, + campaign_evidence_available: bool, +) -> str | None: + reasons: list[str] = [] + if asset.legal_hold: + reasons.append("The file is under legal hold.") + retained_until = _aware(asset.retained_until) + if retained_until is not None and retained_until > datetime.now(timezone.utc): + reasons.append(f"The file is retained until {retained_until.isoformat()}.") + if form_evidence_available: + has_form_evidence = ( + session.query(FileFormEvidenceGrant.id) + .filter( + FileFormEvidenceGrant.tenant_id == asset.tenant_id, + FileFormEvidenceGrant.file_asset_id == asset.id, + ) + .first() + is not None + ) + if has_form_evidence: + reasons.append("The file is referenced by submitted Form evidence.") + if campaign_evidence_available: + has_campaign_evidence = ( + session.query(CampaignAttachmentUse.id) + .filter( + CampaignAttachmentUse.tenant_id == asset.tenant_id, + CampaignAttachmentUse.file_asset_id == asset.id, + ) + .first() + is not None + ) + if has_campaign_evidence: + reasons.append("The file is referenced by Campaign delivery evidence.") + return " ".join(reasons) or None + + +_DETACHABLE_MODELS: dict[str, tuple[type[object], frozenset[str]]] = { + "file_asset": (FileAsset, frozenset({"created_by_user_id"})), + "file_version": (FileVersion, frozenset({"created_by_user_id"})), + "file_folder": (FileFolder, frozenset({"created_by_user_id"})), + "file_share": ( + FileShare, + frozenset({"created_by_user_id", "revoked_by_user_id"}), + ), + "connector_profile": ( + FileConnectorProfile, + frozenset({"created_by_user_id", "updated_by_user_id"}), + ), + "connector_credential": ( + FileConnectorCredential, + frozenset({"created_by_user_id", "updated_by_user_id"}), + ), + "connector_policy": ( + FileConnectorPolicy, + frozenset({"created_by_user_id", "updated_by_user_id"}), + ), + "connector_space": (FileConnectorSpace, frozenset({"created_by_user_id"})), +} + + +def _detachable_fields(resource_type: str) -> frozenset[str]: + entry = _DETACHABLE_MODELS.get(resource_type) + return entry[1] if entry is not None else frozenset() + + +def _revoke_share( + session: Session, + *, + tenant_id: str, + subject_user_id: str, + action: DsarErasureActionRef, + request_id: str, +) -> DsarExecutionResultRef: + row = ( + session.query(FileShare) + .filter(FileShare.id == action.resource_id) + .with_for_update() + .one_or_none() + ) + if ( + row is None + or row.tenant_id != tenant_id + or row.target_type != "user" + or row.target_id != subject_user_id + ): + return _blocked(action, "The subject-targeted file share is no longer available.") + if row.revoked_at is not None: + return _result( + action, + "unchanged", + "The file share was already revoked.", + {"request_id": request_id, "revoked_at": _iso(row.revoked_at)}, + ) + row.revoked_at = datetime.now(timezone.utc) + row.revoked_by_user_id = None + return _result( + action, + "executed", + "The subject-targeted file share was revoked.", + {"request_id": request_id, "revoked_at": _iso(row.revoked_at)}, + ) + + +def _detach_reference( + session: Session, + *, + tenant_id: str, + subject_user_id: str, + action: DsarErasureActionRef, + request_id: str, +) -> DsarExecutionResultRef: + entry = _DETACHABLE_MODELS.get(action.resource_type) + field_name = str(action.metadata.get("field") or "") + if entry is None or field_name not in entry[1]: + return _blocked(action, "The requested Files subject reference is not detachable.") + model = entry[0] + row = ( + session.query(model) + .filter(getattr(model, "id") == action.resource_id) + .with_for_update() + .one_or_none() + ) + if row is None or getattr(row, "tenant_id", None) != tenant_id: + return _blocked(action, "The Files resource is no longer available.") + current = getattr(row, field_name) + if current is None: + return _result( + action, + "unchanged", + "The subject reference was already detached.", + {"request_id": request_id, "field": field_name}, + ) + if current != subject_user_id: + return _blocked(action, "The Files subject reference changed after planning.") + setattr(row, field_name, None) + return _result( + action, + "executed", + "The mutable subject reference was detached.", + {"request_id": request_id, "field": field_name}, + ) + + +def _subject_user_id(subject: DsarSubjectRef) -> str | None: + candidates: list[str] = [] + if subject.membership_id: + candidates.append(subject.membership_id) + for key, value in subject.external_references.items(): + if key in { + "files.user", + "files.membership", + "access.membership", + "membership_id", + }: + candidates.append(value) + normalized = {value.strip() for value in candidates if value.strip()} + if len(normalized) != 1: + return None + return normalized.pop() + + +def _matching_fields( + row: object, + subject_user_id: str, + fields: Sequence[str], +) -> list[str]: + return [field for field in fields if getattr(row, field) == subject_user_id] + + +def _record( + resource_type: str, + resource_id: str, + category: str, + title: str, + data: dict[str, object], + *, + observed_at: datetime | None = None, + immutable: bool = False, + retention_reason: str | None = None, + source_path: str | None = None, +) -> DsarRecordRef: + return DsarRecordRef( + provider_id="files", + module_id="files", + resource_type=resource_type, + resource_id=resource_id, + category=category, + title=title, + data=data, + observed_at=observed_at, + immutable_evidence=immutable, + retention_reason=retention_reason, + source_path=source_path, + ) + + +def _action( + action_id: str, + kind: str, + record: DsarRecordRef, + title: str, + rationale: str, + *, + executable: bool, + metadata: dict[str, object] | None = None, +) -> DsarErasureActionRef: + return DsarErasureActionRef( + action_id=action_id, + provider_id="files", + module_id="files", + kind=kind, # type: ignore[arg-type] + resource_type=record.resource_type, + resource_id=record.resource_id, + title=title, + rationale=rationale, + executable=executable, + metadata=metadata or {}, + ) + + +def _result( + action: DsarErasureActionRef, + status: str, + summary: str, + evidence: dict[str, object] | None = None, +) -> DsarExecutionResultRef: + return DsarExecutionResultRef( + action_id=action.action_id, + status=status, # type: ignore[arg-type] + summary=summary, + evidence=evidence or {}, + ) + + +def _blocked(action: DsarErasureActionRef, summary: str) -> DsarExecutionResultRef: + return _result(action, "blocked", summary) + + +def _session(value: object) -> Session: + if not isinstance(value, Session): + raise TypeError("Files DSAR provider requires a SQLAlchemy session.") + return value + + +def _bounded_rows(query: object) -> list[object]: + rows = query.limit(_MAX_RECORDS + 1).all() # type: ignore[attr-defined] + if len(rows) > _MAX_RECORDS: + raise ValueError("Files DSAR match limit exceeded; narrow the subject selectors.") + return rows + + +def _has_table(session: Session, model: type[object]) -> bool: + return inspect(session.connection()).has_table(model.__tablename__) + + +def _aware(value: datetime | None) -> datetime | None: + if value is None or value.tzinfo is not None: + return value + return value.replace(tzinfo=timezone.utc) + + +def _iso(value: datetime | None) -> str | None: + aware = _aware(value) + return aware.isoformat() if aware else None + + +__all__ = ["FILES_DSAR_CAPABILITY", "FilesDsarProvider"] diff --git a/src/govoplan_files/backend/manifest.py b/src/govoplan_files/backend/manifest.py index 2e710c0..00056a9 100644 --- a/src/govoplan_files/backend/manifest.py +++ b/src/govoplan_files/backend/manifest.py @@ -54,6 +54,7 @@ from govoplan_files.backend.configuration_provider import ( ) from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata from govoplan_files.backend.documentation import documentation_topics +from govoplan_files.backend.dsar_provider import FILES_DSAR_CAPABILITY from govoplan_files.backend.form_evidence import ( CAPABILITY_FORM_EVIDENCE_FILES, create_files_form_evidence_provider, @@ -440,6 +441,13 @@ REMOTE_STORAGE_PROVIDER = ExternalProviderDeclaration( ) +def _dsar_provider(context: ModuleContext) -> object: + del context + from govoplan_files.backend.dsar_provider import FilesDsarProvider + + return FilesDsarProvider() + + manifest = ModuleManifest( id="files", name="Files", @@ -456,6 +464,7 @@ manifest = ModuleManifest( ModuleInterfaceProvider(name=CAPABILITY_FILES_POSTBOX_REFERENCES, version="1.0.0"), ModuleInterfaceProvider(name=CAPABILITY_RECORD_SOURCE_FILES, version="1.0.0"), ModuleInterfaceProvider(name=CAPABILITY_FORM_EVIDENCE_FILES, version="1.0.0"), + ModuleInterfaceProvider(name=FILES_DSAR_CAPABILITY, version="0.1.0"), ), requires_interfaces=( ModuleInterfaceRequirement( @@ -1180,6 +1189,69 @@ manifest = ModuleManifest( ], }, ), + DocumentationTopic( + id="files.privacy.data-subject-requests", + title="Review Files data in a data-subject request", + summary="Collect safe Files metadata and keep retention, evidence, and byte-erasure decisions explicit.", + body=( + "The Files DSAR provider searches only the effective tenant and requires a direct membership or namespaced Files user reference. " + "It exports bounded file, version, folder, sharing, evidence, connector-configuration, and integrity metadata without raw file bytes, storage locations, tokens, passwords, secret references, or encrypted credential values. " + "Plans may revoke an active share aimed at the subject or detach a mutable actor reference. Legal hold, active retention, Form evidence, Campaign delivery evidence, configuration history, and integrity evidence remain retained with a reason. File content, ownership, names, and paths require manual review. Approved physical erasure must use the separately authorized Files purge and blob-garbage-collection workflow so DSAR execution cannot bypass evidence blockers, approval, audit, or recovery controls." + ), + layer="configured", + documentation_types=("admin",), + audience=("privacy_officer", "file_admin", "records_manager", "operator"), + order=47, + conditions=( + DocumentationCondition( + required_modules=("files", "access"), + any_scopes=( + "access:privacy:read", + "access:privacy:manage", + "access:privacy:erase", + ), + ), + ), + links=( + DocumentationLink( + label="Data-subject requests", + href="/admin?section=tenant-data-subject-requests", + kind="runtime", + ), + DocumentationLink( + label="Files handbook", + href="govoplan-files/docs/FILES_HANDBOOK.md", + kind="repository", + ), + ), + related_modules=("access", "audit", "campaigns", "forms-runtime", "ops"), + metadata={ + "kind": "workflow", + "route": "/admin?section=tenant-data-subject-requests", + "screen": "Data-subject requests", + "help_contexts": ["admin.privacy.data-subject-requests"], + "prerequisites": [ + "The request has been authorized and contains a direct tenant membership or Files subject reference.", + "The privacy reviewer can distinguish access export from erasure authority and Files purge authority.", + ], + "steps": [ + "Run the provider search and confirm Files reports complete coverage rather than a failed or absent provider.", + "Review file/version metadata, evidence retention reasons, and the source path for manual content review.", + "Generate the erasure plan and execute only the approved reversible share-revocation or subject-reference actions.", + "For approved byte erasure, resolve every lifecycle blocker and use Files purge preview, execution, and blob garbage collection separately.", + ], + "limitations": [ + "Email, account, or identity selectors alone cannot be resolved by Files because Files does not own the Access directory; supply the corroborated membership reference.", + "The provider does not embed raw file content in the JSON export and never performs physical blob deletion as a DSAR side effect.", + ], + "outcome": "Files-owned subject references are reviewed or removed without silently destroying retained content or evidence.", + "verification": "Confirm every Files record has a retain, review, revoke, or detach disposition and inspect any separate purge through its audit and recovery evidence.", + "related_topic_ids": [ + "files.workflow.restore-retain-and-purge", + "files.reference.integrity-recovery-and-fail-closed-transports", + ], + }, + ), DocumentationTopic( id="files.governed-connectors-and-provenance", title="Govern file connections and credential deletion", @@ -1635,6 +1707,7 @@ manifest = ModuleManifest( ).campaign_capability(context), CAPABILITY_RECORD_SOURCE_FILES: create_files_record_source, CAPABILITY_FORM_EVIDENCE_FILES: create_files_form_evidence_provider, + FILES_DSAR_CAPABILITY: _dsar_provider, }, capability_documentation={ CAPABILITY_RECORD_SOURCE_FILES: CapabilityDocumentation( @@ -1647,6 +1720,11 @@ manifest = ModuleManifest( summary="Issues one-time managed attachment grants and verifies exact Form evidence versions.", contract_version="1.0.0", ), + FILES_DSAR_CAPABILITY: CapabilityDocumentation( + label="Files data-subject request provider", + summary="Finds safe Files metadata and classifies reversible references, manual file review, and retained evidence.", + contract_version="0.1.0", + ), }, operational_check_providers=( OperationalCheckProviderRegistration( diff --git a/tests/test_dsar_provider.py b/tests/test_dsar_provider.py new file mode 100644 index 0000000..effe4b0 --- /dev/null +++ b/tests/test_dsar_provider.py @@ -0,0 +1,468 @@ +from __future__ import annotations + +import unittest +from datetime import datetime, timedelta, timezone + +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from govoplan_access.backend.db.models import Account, Group, User +from govoplan_core.core.dsar import DsarProvider, DsarSubjectRef +from govoplan_core.core.change_sequence import ChangeSequenceEntry +from govoplan_core.db.base import Base +from govoplan_core.privacy.dsar_workflow import ( + DataSubjectRequest, + create_data_subject_request, + execute_data_subject_erasure, + plan_data_subject_erasure, + search_data_subject_request, +) +from govoplan_files.backend.db.models import ( + FileAsset, + FileBlob, + FileConnectorCredential, + FileConnectorPolicy, + FileConnectorProfile, + FileConnectorSpace, + FileFolder, + FileFormEvidenceGrant, + FileIntegrityFinding, + FileIntegrityScan, + FileShare, + FileVersion, +) +from govoplan_files.backend.dsar_provider import ( + FILES_DSAR_CAPABILITY, + FilesDsarProvider, +) +from govoplan_files.backend.manifest import manifest + + +class _Registry: + def __init__(self, provider: FilesDsarProvider, *, files_active: bool = True) -> None: + self.provider = provider + self.files_active = files_active + + def capability_names(self): + return (FILES_DSAR_CAPABILITY,) + + def capability_owner(self, name): + self._assert_capability(name) + return "files" + + def tenant_entitlement_resolver(self): + files_active = self.files_active + + class _Resolver: + @staticmethod + def resolve(session, tenant_id): + del session, tenant_id + return type( + "State", + (), + {"effective_modules": ("files",) if files_active else ()}, + )() + + return _Resolver() + + def require_tenant_capability(self, name, session, **kwargs): + del session, kwargs + self._assert_capability(name) + return self.provider + + def manifests(self): + return (type("Manifest", (), {"id": "files"})(),) + + @staticmethod + def _assert_capability(name: str) -> None: + if name != FILES_DSAR_CAPABILITY: + raise KeyError(name) + + +class FilesDsarProviderTests(unittest.TestCase): + def setUp(self) -> None: + self.engine = create_engine("sqlite:///:memory:", future=True) + Base.metadata.create_all( + bind=self.engine, + tables=[ + Account.__table__, + User.__table__, + Group.__table__, + ChangeSequenceEntry.__table__, + DataSubjectRequest.__table__, + FileBlob.__table__, + FileAsset.__table__, + FileVersion.__table__, + FileFolder.__table__, + FileFormEvidenceGrant.__table__, + FileShare.__table__, + FileConnectorCredential.__table__, + FileConnectorPolicy.__table__, + FileConnectorProfile.__table__, + FileConnectorSpace.__table__, + FileIntegrityScan.__table__, + FileIntegrityFinding.__table__, + ], + ) + self.session = sessionmaker(bind=self.engine, future=True)() + self.account = Account( + id="account-1", + email="subject@example.test", + normalized_email="subject@example.test", + display_name="Subject", + password_hash="not-exported", + ) + self.user = User( + id="membership-1", + tenant_id="tenant-1", + account_id=self.account.id, + email="subject@example.test", + display_name="Subject", + ) + self.blob = FileBlob( + id="blob-1", + tenant_id="tenant-1", + storage_backend="local", + storage_key="private/storage/key-do-not-export", + checksum_sha256="a" * 64, + size_bytes=12, + ref_count=1, + ) + self.asset = FileAsset( + id="asset-1", + tenant_id="tenant-1", + owner_type="user", + owner_user_id=self.user.id, + created_by_user_id=self.user.id, + current_version_id="version-1", + display_path="subjects/private.txt", + filename="private.txt", + description="Subject-provided document", + retained_until=datetime.now(timezone.utc) + timedelta(days=30), + lifecycle_reason="Pending proceeding", + metadata_={"password": "metadata-secret-do-not-export"}, + ) + self.version = FileVersion( + id="version-1", + tenant_id="tenant-1", + file_asset_id=self.asset.id, + blob_id=self.blob.id, + version_number=1, + filename_at_upload=self.asset.filename, + display_path_at_upload=self.asset.display_path, + content_type="text/plain", + size_bytes=12, + checksum_sha256="a" * 64, + created_by_user_id=self.user.id, + ) + self.folder = FileFolder( + id="folder-1", + tenant_id="tenant-1", + owner_type="user", + owner_user_id=self.user.id, + path="subjects", + created_by_user_id=self.user.id, + ) + self.share = FileShare( + id="share-1", + tenant_id="tenant-1", + file_asset_id=self.asset.id, + target_type="user", + target_id=self.user.id, + permission="read", + created_by_user_id=self.user.id, + ) + self.form_evidence = FileFormEvidenceGrant( + id="evidence-1", + tenant_id="tenant-1", + form_instance_id="form-instance-1", + form_definition_id="application", + form_definition_revision="7", + token_sha256="b" * 64, + idempotency_key="evidence-key-1", + request_sha256="c" * 64, + custodian_user_id=self.user.id, + evidence_kind="attachment", + purpose="Submitted application evidence", + status="uploaded", + expires_at=datetime.now(timezone.utc) + timedelta(days=1), + max_size_bytes=1024, + allowed_content_types=["text/plain"], + file_asset_id=self.asset.id, + file_version_id=self.version.id, + metadata_={}, + ) + self.credential = FileConnectorCredential( + id="credential-1", + tenant_id="tenant-1", + scope_type="tenant", + scope_id="tenant-1", + label="Subject-created credential", + provider="s3", + credential_mode="database", + username="subject-user", + password_encrypted="encrypted-password-do-not-export", + token_encrypted="encrypted-token-do-not-export", + password_env="PASSWORD_ENV_DO_NOT_EXPORT", + secret_ref="vault://do-not-export", + created_by_user_id=self.user.id, + updated_by_user_id=self.user.id, + ) + self.integrity_scan = FileIntegrityScan( + id="scan-1", + tenant_id="tenant-1", + storage_backend="local", + storage_prefix="private-prefix-do-not-export", + created_by_user_id=self.user.id, + ) + tenant_two_asset = FileAsset( + id="asset-tenant-2", + tenant_id="tenant-2", + owner_type="user", + owner_user_id=self.user.id, + display_path="other-tenant.txt", + filename="other-tenant.txt", + ) + self.session.add_all( + [ + self.account, + self.user, + self.blob, + self.asset, + self.version, + self.folder, + self.share, + self.form_evidence, + self.credential, + self.integrity_scan, + tenant_two_asset, + ] + ) + self.session.commit() + self.provider = FilesDsarProvider() + self.subject = DsarSubjectRef(membership_id=self.user.id) + + def tearDown(self) -> None: + self.session.close() + self.engine.dispose() + + def test_manifest_publishes_protocol_conforming_provider(self) -> None: + provided_names = {item.name for item in manifest.provides_interfaces} + self.assertIn(FILES_DSAR_CAPABILITY, provided_names) + provider = manifest.capability_factories[FILES_DSAR_CAPABILITY](None) + self.assertIsInstance(provider, DsarProvider) + + def test_search_is_tenant_scoped_and_excludes_bytes_and_secrets(self) -> None: + records = self.provider.search_subject( + self.session, + tenant_id="tenant-1", + subject=self.subject, + ) + resource_types = {record.resource_type for record in records} + self.assertTrue( + { + "file_asset", + "file_version", + "file_folder", + "file_share", + "file_form_evidence", + "connector_credential", + "file_integrity_scan", + }.issubset(resource_types) + ) + serialized = repr([record.to_dict() for record in records]) + self.assertNotIn("asset-tenant-2", serialized) + self.assertNotIn("private/storage/key-do-not-export", serialized) + self.assertNotIn("metadata-secret-do-not-export", serialized) + self.assertNotIn("encrypted-password-do-not-export", serialized) + self.assertNotIn("encrypted-token-do-not-export", serialized) + self.assertNotIn("subject-user", serialized) + self.assertNotIn("PASSWORD_ENV_DO_NOT_EXPORT", serialized) + self.assertNotIn("vault://do-not-export", serialized) + self.assertNotIn("private-prefix-do-not-export", serialized) + + def test_conflicting_direct_subject_references_fail_closed(self) -> None: + records = self.provider.search_subject( + self.session, + tenant_id="tenant-1", + subject=DsarSubjectRef( + membership_id=self.user.id, + external_references={"files.user": "another-membership"}, + ), + ) + + self.assertEqual((), records) + + def test_plan_classifies_retention_and_manual_review(self) -> None: + records = self.provider.search_subject( + self.session, + tenant_id="tenant-1", + subject=self.subject, + ) + actions = self.provider.plan_erasure( + self.session, + tenant_id="tenant-1", + subject=self.subject, + records=records, + ) + + kinds = {action.kind for action in actions} + self.assertTrue({"retain", "manual_review", "revoke", "detach"}.issubset(kinds)) + asset_retention = next( + action + for action in actions + if action.action_id == "files:retain:file_asset:asset-1" + ) + self.assertIn("retained until", asset_retention.rationale) + self.assertTrue( + any(action.action_id == "files:revoke:file_share:share-1" for action in actions) + ) + self.assertFalse( + any(action.kind == "delete" and action.executable for action in actions) + ) + + def test_execution_is_revalidated_and_idempotent(self) -> None: + records = self.provider.search_subject( + self.session, + tenant_id="tenant-1", + subject=self.subject, + ) + actions = self.provider.plan_erasure( + self.session, + tenant_id="tenant-1", + subject=self.subject, + records=records, + ) + executable = tuple(action for action in actions if action.executable) + + first = self.provider.execute_erasure( + self.session, + tenant_id="tenant-1", + subject=self.subject, + actions=executable, + request_id="dsar-1", + ) + self.assertEqual({"executed"}, {result.status for result in first}) + self.assertIsNotNone(self.share.revoked_at) + self.assertIsNone(self.asset.created_by_user_id) + self.assertIsNone(self.version.created_by_user_id) + self.assertIsNone(self.folder.created_by_user_id) + self.assertIsNone(self.credential.created_by_user_id) + self.assertIsNone(self.credential.updated_by_user_id) + + repeated = self.provider.execute_erasure( + self.session, + tenant_id="tenant-1", + subject=self.subject, + actions=executable, + request_id="dsar-1", + ) + self.assertEqual({"unchanged"}, {result.status for result in repeated}) + + def test_execution_blocks_when_reference_changed_after_planning(self) -> None: + records = self.provider.search_subject( + self.session, + tenant_id="tenant-1", + subject=self.subject, + ) + actions = self.provider.plan_erasure( + self.session, + tenant_id="tenant-1", + subject=self.subject, + records=records, + ) + detach = next( + action + for action in actions + if action.action_id + == "files:detach:file_version:created_by_user_id:version-1" + ) + self.version.created_by_user_id = "replacement-user" + self.session.flush() + + result = self.provider.execute_erasure( + self.session, + tenant_id="tenant-1", + subject=self.subject, + actions=(detach,), + request_id="dsar-2", + ) + + self.assertEqual("blocked", result[0].status) + self.assertEqual("replacement-user", self.version.created_by_user_id) + + def test_core_workflow_discovers_active_provider_and_skips_it_when_disabled( + self, + ) -> None: + request = create_data_subject_request( + self.session, + tenant_id="tenant-1", + reference="DSAR-FILES-1", + request_kind="access_and_erasure", + subject=self.subject, + purpose="Respond to an authorized privacy request.", + legal_basis="Article 15 and 17 GDPR", + due_at=None, + requested_by_account_id="privacy-officer", + ) + self.session.commit() + registry = _Registry(self.provider) + + search_data_subject_request( + self.session, + registry=registry, + row=request, + expected_revision=1, + ) + self.assertEqual( + "searched", request.status, request.search_result["provider_runs"] + ) + self.assertEqual(["files"], request.coverage["covered_modules"]) + self.assertEqual([], request.coverage["modules_without_provider"]) + plan_data_subject_erasure( + self.session, + registry=registry, + row=request, + expected_revision=2, + ) + executable_ids = [ + action["action_id"] + for action in request.erasure_plan["actions"] + if action["executable"] + ] + execute_data_subject_erasure( + self.session, + registry=registry, + row=request, + expected_revision=3, + action_ids=executable_ids, + ) + self.assertEqual("completed", request.status) + + disabled = create_data_subject_request( + self.session, + tenant_id="tenant-1", + reference="DSAR-FILES-DISABLED", + request_kind="access", + subject=self.subject, + purpose="Verify disabled-module coverage.", + legal_basis="Article 15 GDPR", + due_at=None, + requested_by_account_id="privacy-officer", + ) + search_data_subject_request( + self.session, + registry=_Registry(self.provider, files_active=False), + row=disabled, + expected_revision=1, + ) + + self.assertEqual(0, disabled.search_result["record_count"]) + self.assertEqual( + [FILES_DSAR_CAPABILITY], + disabled.coverage["inactive_provider_capabilities"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_manifest_documentation.py b/tests/test_manifest_documentation.py index d9edc7f..0bf2c11 100644 --- a/tests/test_manifest_documentation.py +++ b/tests/test_manifest_documentation.py @@ -12,6 +12,7 @@ STATIC_TOPIC_IDS = { "files.workflow.share-managed-files", "files.workflow.delete-managed-files", "files.workflow.restore-retain-and-purge", + "files.privacy.data-subject-requests", "files.governed-connectors-and-provenance", "files.reference.integrity-recovery-and-fail-closed-transports", "files.reference.shared-storage-profile", @@ -130,6 +131,14 @@ class FilesManifestDocumentationTests(unittest.TestCase): self.assertIn("preview hash", lifecycle.body) self.assertIn("recovery ledger", lifecycle.body) + privacy = self.topic("files.privacy.data-subject-requests") + self.assertEqual(("admin",), privacy.documentation_types) + self.assertIn("without raw file bytes", privacy.body) + self.assertIn("separately authorized Files purge", privacy.body) + self.assertTrue( + any("membership" in item for item in privacy.metadata["limitations"]) + ) + def test_admin_topic_covers_policy_redaction_and_atomic_credential_deletion( self, ) -> None: