Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6176e9f40e | ||
|
|
11b9b7c4c6 | ||
|
|
ab08c73777 | ||
|
|
74c26090d7 | ||
|
|
62047e7a20 | ||
|
|
378f4d6ac5 | ||
|
|
a8c3192c3a | ||
|
|
ce4eaefbfd | ||
|
|
6c3cf1c55e | ||
|
|
95aef18955 | ||
|
|
a9c7a7a40c | ||
|
|
e1361427b8 | ||
|
|
ad55d47645 | ||
|
|
21122e058e | ||
|
|
25140fb0b6 | ||
|
|
e9561a87e1 | ||
|
|
9d7d3f96ec | ||
|
|
44ba0ec9c1 | ||
|
|
1e511e4c15 | ||
|
|
cb37079465 | ||
|
|
04e2bae190 | ||
|
|
aa9f90c71e | ||
|
|
60c3d14f08 | ||
|
|
3a3360650f | ||
|
|
c9f8e9262e | ||
|
|
edee29af51 | ||
|
|
0293e49def | ||
|
|
aaa364b8d4 | ||
|
|
4bb17c0f4c | ||
|
|
92e649477f | ||
|
|
0c68e904cf | ||
|
|
8ec31b16ee | ||
|
|
7e6be4b017 | ||
|
|
04882f1628 | ||
|
|
d8ae506ff8 | ||
|
|
6baf2a421b | ||
|
|
b993d8e31a | ||
|
|
359c4e9570 | ||
|
|
233ce40983 | ||
|
|
f084a0f3a9 | ||
|
|
b752dea610 | ||
|
|
159a012833 | ||
|
|
835eacfc5d | ||
|
|
85606d5580 | ||
|
|
632cc6cf7d | ||
|
|
86a905a3a7 | ||
|
|
5b868272b9 | ||
|
|
93eea839c8 | ||
|
|
b667e7ff0e | ||
|
|
0fd8f6e02a | ||
|
|
2b34f6e305 | ||
|
|
58af1a20a7 | ||
|
|
4722161592 | ||
|
|
1444ba80a0 | ||
|
|
02ef83ecee | ||
|
|
1069f85796 | ||
|
|
1401c78c8a | ||
|
|
b6109245a7 | ||
|
|
f5d40b23c2 | ||
|
|
94ea629635 | ||
|
|
6c8a8c655d | ||
|
|
062ad5ddfb | ||
|
|
9adfa91e74 | ||
|
|
65d8ed80b5 | ||
|
|
cffe161f29 | ||
|
|
5248e7de4a | ||
|
|
d5d0df792b | ||
|
|
15ade8df75 | ||
|
|
f3c485ef61 | ||
|
|
0e36b20a14 | ||
|
|
06e6e7191b | ||
|
|
3449cbc8a5 | ||
|
|
92950af6f4 | ||
|
|
8826cf2890 | ||
|
|
f2dfb6c90e | ||
|
|
3bc1d3489e | ||
|
|
d1051293b2 | ||
|
|
8c5f149f07 | ||
|
|
f3210234d3 | ||
|
|
b8b395e8b5 | ||
|
|
8bf7296bf5 | ||
|
|
620fdda2fc | ||
|
|
58c0441763 | ||
|
|
ba7fb2def7 | ||
|
|
4e7c77135c | ||
|
|
13fd7fc3bd | ||
|
|
18e6c3eb9b | ||
|
|
a5c24fe73e | ||
|
|
ee5b06b1e7 |
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: "Bug"
|
||||
about: "Report a reproducible defect, regression, or incorrect behavior"
|
||||
title: "[Bug] "
|
||||
labels:
|
||||
- type/bug
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
- Repository:
|
||||
- Area/module:
|
||||
- Affected version or commit:
|
||||
|
||||
## Behavior
|
||||
|
||||
Expected:
|
||||
|
||||
Actual:
|
||||
|
||||
## Reproduction
|
||||
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
|
||||
## Evidence
|
||||
|
||||
Logs, screenshots, traces, or failing test output:
|
||||
|
||||
## Verification Target
|
||||
|
||||
Command or workflow that should pass when fixed:
|
||||
@@ -0,0 +1 @@
|
||||
blank_issues_enabled: false
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: "Docs / workflow"
|
||||
about: "Request documentation, process, or developer workflow changes"
|
||||
title: "[Docs] "
|
||||
labels:
|
||||
- type/docs
|
||||
- status/triage
|
||||
- module/files
|
||||
- area/docs
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
- Repository:
|
||||
- Document or workflow:
|
||||
|
||||
## Current State
|
||||
|
||||
What is missing, unclear, duplicated, or stale?
|
||||
|
||||
## Desired State
|
||||
|
||||
What should the docs or workflow make clear?
|
||||
|
||||
## Verification Target
|
||||
|
||||
How should this be checked?
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: "Feature"
|
||||
about: "Propose new user-visible behavior or platform capability"
|
||||
title: "[Feature] "
|
||||
labels:
|
||||
- type/feature
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
What user, operator, or developer problem should this solve?
|
||||
|
||||
## Proposed Capability
|
||||
|
||||
What should exist when this is done?
|
||||
|
||||
## Ownership
|
||||
|
||||
- Owning repository:
|
||||
- Related module repositories:
|
||||
- Extension point or integration boundary:
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ]
|
||||
- [ ]
|
||||
|
||||
## Verification Target
|
||||
|
||||
Command, scenario, or UI flow that should prove completion:
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: "Task"
|
||||
about: "Track implementation, maintenance, or migration work"
|
||||
title: "[Task] "
|
||||
labels:
|
||||
- type/task
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
What needs to be completed?
|
||||
|
||||
## Scope
|
||||
|
||||
- Owning repository:
|
||||
- In-scope:
|
||||
- Out-of-scope:
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ]
|
||||
- [ ]
|
||||
|
||||
## Verification Target
|
||||
|
||||
Command or manual check:
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: "Tech debt"
|
||||
about: "Track cleanup, refactoring, risk reduction, or deferred engineering work"
|
||||
title: "[Debt] "
|
||||
labels:
|
||||
- type/debt
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Current Cost
|
||||
|
||||
What does this make harder, riskier, slower, or more fragile?
|
||||
|
||||
## Desired Shape
|
||||
|
||||
What should the code, tests, or architecture look like afterwards?
|
||||
|
||||
## Constraints
|
||||
|
||||
What behavior, compatibility, or module boundary must be preserved?
|
||||
|
||||
## Verification Target
|
||||
|
||||
Focused checks that should pass:
|
||||
@@ -0,0 +1,15 @@
|
||||
## Issue
|
||||
|
||||
Closes #
|
||||
|
||||
## Summary
|
||||
|
||||
-
|
||||
|
||||
## Verification
|
||||
|
||||
-
|
||||
|
||||
## Notes
|
||||
|
||||
Follow-up issues:
|
||||
@@ -0,0 +1,270 @@
|
||||
name: Module Package Release
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_tag:
|
||||
description: Existing protected version tag to publish
|
||||
required: true
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
publish-packages:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
GITEA_REPOSITORY: ${{ gitea.repository }}
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
|
||||
with:
|
||||
node-version: "22"
|
||||
- name: Select and validate protected release tag
|
||||
shell: bash
|
||||
env:
|
||||
REQUESTED_TAG: ${{ inputs.release_tag }}
|
||||
TRIGGER_TAG: ${{ gitea.ref_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tag="${REQUESTED_TAG:-$TRIGGER_TAG}"
|
||||
case "$tag" in
|
||||
v[0-9]*.[0-9]*.[0-9]*) ;;
|
||||
*) echo "Release tag must start with a SemVer-shaped vX.Y.Z value" >&2; exit 1 ;;
|
||||
esac
|
||||
git fetch --force origin "refs/tags/$tag:refs/tags/$tag" refs/heads/main:refs/remotes/origin/main
|
||||
tag_commit="$(git rev-list -n 1 "$tag")"
|
||||
git merge-base --is-ancestor "$tag_commit" refs/remotes/origin/main || {
|
||||
echo "Release tag is not contained in main" >&2
|
||||
exit 1
|
||||
}
|
||||
git checkout --detach "$tag"
|
||||
printf 'RELEASE_TAG=%s\n' "$tag" >> "$GITEA_ENV"
|
||||
printf 'SOURCE_DATE_EPOCH=%s\n' "$(git show -s --format=%ct HEAD)" >> "$GITEA_ENV"
|
||||
- name: Validate package versions
|
||||
run: |
|
||||
python - <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
import os
|
||||
import re
|
||||
import tomllib
|
||||
|
||||
tag = os.environ["RELEASE_TAG"]
|
||||
expected = tag.removeprefix("v")
|
||||
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
|
||||
if project.get("version") != expected:
|
||||
raise SystemExit(f"pyproject version {project.get('version')!r} does not match {tag}")
|
||||
if re.fullmatch(r"govoplan-[a-z0-9-]+", str(project.get("name", ""))) is None:
|
||||
raise SystemExit("Python distribution name must use the govoplan-* namespace")
|
||||
webui = Path("webui/package.json")
|
||||
if webui.is_file():
|
||||
package = json.loads(webui.read_text(encoding="utf-8"))
|
||||
if package.get("version") != expected:
|
||||
raise SystemExit(f"WebUI version {package.get('version')!r} does not match {tag}")
|
||||
if re.fullmatch(r"@govoplan/[a-z0-9-]+-webui", str(package.get("name", ""))) is None:
|
||||
raise SystemExit("WebUI package name must use the @govoplan/*-webui namespace")
|
||||
release = Path("webui/package.release.json")
|
||||
if release.is_file():
|
||||
release_package = json.loads(release.read_text(encoding="utf-8"))
|
||||
if (
|
||||
release_package.get("name") != package.get("name")
|
||||
or release_package.get("version") != expected
|
||||
):
|
||||
raise SystemExit("WebUI release package identity does not match package.json and the release tag")
|
||||
PY
|
||||
- name: Build immutable package artifacts
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m pip install --disable-pip-version-check build==1.5.0 twine==7.0.0
|
||||
rm -rf dist .package-webui
|
||||
python -m build --wheel --outdir dist
|
||||
python -m twine check dist/*.whl
|
||||
if [[ -f webui/package.json ]]; then
|
||||
mkdir .package-webui
|
||||
cp -a webui/. .package-webui/
|
||||
rm -rf .package-webui/node_modules .package-webui/dist
|
||||
if [[ -f .package-webui/package.release.json ]]; then
|
||||
cp .package-webui/package.release.json .package-webui/package.json
|
||||
fi
|
||||
node <<'NODE'
|
||||
const fs = require("node:fs");
|
||||
const path = ".package-webui/package.json";
|
||||
const packageJson = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||
const groups = ["dependencies", "optionalDependencies", "peerDependencies"];
|
||||
for (const group of groups) {
|
||||
for (const [name, specifier] of Object.entries(packageJson[group] || {})) {
|
||||
if (!name.startsWith("@govoplan/")) continue;
|
||||
if (typeof specifier !== "string") {
|
||||
throw new Error(`${group}.${name} must use a string version`);
|
||||
}
|
||||
const packageSlug = name.slice("@govoplan/".length);
|
||||
if (!packageSlug.endsWith("-webui")) {
|
||||
throw new Error(`${group}.${name} is outside the WebUI package namespace`);
|
||||
}
|
||||
const repository = `govoplan-${packageSlug.slice(0, -"-webui".length)}`;
|
||||
const escapedRepository = repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const gitTag = specifier.match(
|
||||
new RegExp(
|
||||
`^git\\+(?:ssh://git@|https://)git\\.add-ideas\\.de/(?:GovOPlaN|add-ideas)/${escapedRepository}\\.git#v([0-9]+\\.[0-9]+\\.[0-9]+)$`,
|
||||
),
|
||||
);
|
||||
if (gitTag) {
|
||||
packageJson[group][name] = gitTag[1];
|
||||
continue;
|
||||
}
|
||||
if (specifier.startsWith("file:") || specifier.startsWith("git+")) {
|
||||
throw new Error(
|
||||
`${group}.${name} must resolve to an exact registry version for publication`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
delete packageJson.private;
|
||||
fs.writeFileSync(path, `${JSON.stringify(packageJson, null, 2)}\n`);
|
||||
NODE
|
||||
npm pkg delete private --prefix .package-webui
|
||||
(cd .package-webui && npm pack --ignore-scripts --pack-destination ../dist)
|
||||
fi
|
||||
python - <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
artifacts = []
|
||||
for path in sorted(Path("dist").iterdir()):
|
||||
if path.suffix not in {".whl", ".tgz"}:
|
||||
continue
|
||||
digest = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
artifacts.append({"filename": path.name, "sha256": digest, "size": path.stat().st_size})
|
||||
payload = {
|
||||
"schema_version": "1",
|
||||
"repository": os.environ["GITEA_REPOSITORY"],
|
||||
"tag": os.environ["RELEASE_TAG"],
|
||||
"commit": subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip(),
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
Path("dist/package-artifacts.json").write_text(
|
||||
json.dumps(payload, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
PY
|
||||
- name: Retain package hash evidence
|
||||
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
|
||||
with:
|
||||
name: module-packages-${{ gitea.ref_name }}
|
||||
path: dist/package-artifacts.json
|
||||
- name: Check immutable registry state
|
||||
shell: bash
|
||||
env:
|
||||
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test -n "$PACKAGE_TOKEN"
|
||||
python - <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tomllib
|
||||
from urllib.error import HTTPError
|
||||
from urllib.parse import quote
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
api_root = "https://git.add-ideas.de/api/v1/packages/GovOPlaN"
|
||||
token = os.environ["PACKAGE_TOKEN"]
|
||||
|
||||
def should_publish(kind, name, version, path):
|
||||
package_url = "/".join(
|
||||
(api_root, kind, quote(name, safe=""), quote(version, safe=""), "files")
|
||||
)
|
||||
request = Request(
|
||||
package_url,
|
||||
headers={"Accept": "application/json", "Authorization": f"token {token}"},
|
||||
)
|
||||
try:
|
||||
with urlopen(request, timeout=30) as response:
|
||||
files = json.load(response)
|
||||
except HTTPError as exc:
|
||||
if exc.code == 404:
|
||||
print(f"{kind} package {name}=={version} is not published yet")
|
||||
return True
|
||||
raise
|
||||
if not isinstance(files, list) or len(files) != 1:
|
||||
raise SystemExit(
|
||||
f"immutable {kind} package {name}=={version} has an unexpected file set"
|
||||
)
|
||||
expected_sha256 = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
if files[0].get("sha256") != expected_sha256:
|
||||
raise SystemExit(
|
||||
f"immutable {kind} package {name}=={version} already exists with a different SHA-256"
|
||||
)
|
||||
print(f"verified existing {kind} package {name}=={version} ({expected_sha256})")
|
||||
return False
|
||||
|
||||
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
|
||||
wheels = tuple(Path("dist").glob("*.whl"))
|
||||
if len(wheels) != 1:
|
||||
raise SystemExit("release build must contain exactly one wheel")
|
||||
publish_pypi = should_publish(
|
||||
"pypi", str(project["name"]), str(project["version"]), wheels[0]
|
||||
)
|
||||
|
||||
tarballs = tuple(Path("dist").glob("*.tgz"))
|
||||
if len(tarballs) > 1:
|
||||
raise SystemExit("release build must contain at most one npm package")
|
||||
publish_npm = False
|
||||
if tarballs:
|
||||
webui = json.loads(
|
||||
Path(".package-webui/package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
publish_npm = should_publish(
|
||||
"npm", str(webui["name"]), str(webui["version"]), tarballs[0]
|
||||
)
|
||||
|
||||
with Path(os.environ["GITEA_ENV"]).open("a", encoding="utf-8") as env_file:
|
||||
env_file.write(f"PUBLISH_PYPI={int(publish_pypi)}\n")
|
||||
env_file.write(f"PUBLISH_NPM={int(publish_npm)}\n")
|
||||
PY
|
||||
- name: Publish wheel and WebUI package
|
||||
shell: bash
|
||||
env:
|
||||
PACKAGE_USERNAME: ${{ secrets.GOVOPLAN_PACKAGE_USERNAME }}
|
||||
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test -n "$PACKAGE_USERNAME"
|
||||
test -n "$PACKAGE_TOKEN"
|
||||
if [[ "$PUBLISH_PYPI" == 1 ]]; then
|
||||
TWINE_USERNAME="$PACKAGE_USERNAME" TWINE_PASSWORD="$PACKAGE_TOKEN" \
|
||||
python -m twine upload --non-interactive \
|
||||
--repository-url https://git.add-ideas.de/api/packages/GovOPlaN/pypi \
|
||||
dist/*.whl
|
||||
else
|
||||
echo "Exact wheel is already present; skipping immutable retry."
|
||||
fi
|
||||
shopt -s nullglob
|
||||
webui_packages=(dist/*.tgz)
|
||||
if (( ${#webui_packages[@]} )) && [[ "$PUBLISH_NPM" == 1 ]]; then
|
||||
npmrc="$(mktemp)"
|
||||
trap 'rm -f "$npmrc"' EXIT
|
||||
chmod 600 "$npmrc"
|
||||
printf '%s\n' \
|
||||
'@govoplan:registry=https://git.add-ideas.de/api/packages/GovOPlaN/npm/' \
|
||||
"//git.add-ideas.de/api/packages/GovOPlaN/npm/:_authToken=$PACKAGE_TOKEN" \
|
||||
> "$npmrc"
|
||||
NPM_CONFIG_USERCONFIG="$npmrc" npm publish "./${webui_packages[0]}" \
|
||||
--ignore-scripts --access public \
|
||||
--registry https://git.add-ideas.de/api/packages/GovOPlaN/npm/
|
||||
elif (( ${#webui_packages[@]} )); then
|
||||
echo "Exact WebUI package is already present; skipping immutable retry."
|
||||
fi
|
||||
+18
@@ -327,3 +327,21 @@ dist
|
||||
# Built Visual Studio Code Extensions
|
||||
*.vsix
|
||||
|
||||
# GovOPlaN shared ignore rules from govoplan-core
|
||||
# Local WebUI test/build scratch directories
|
||||
.component-test-build/
|
||||
.module-test-build/
|
||||
.policy-test-build/
|
||||
.template-preview-test-build/
|
||||
.import-test-build/
|
||||
webui/.component-test-build/
|
||||
webui/.module-test-build/
|
||||
webui/.policy-test-build/
|
||||
webui/.template-preview-test-build/
|
||||
webui/.import-test-build/
|
||||
*.db
|
||||
# GovOPlaN local runtime state
|
||||
runtime/
|
||||
# GovOPlaN WebUI test output
|
||||
webui/.module-test-build/
|
||||
webui/.component-test-build/
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# GovOPlaN Files Codex Guide
|
||||
|
||||
## Documentation Contract
|
||||
|
||||
- Treat documentation as part of every behavior change. Update this module's manifest-driven `DocumentationTopic` contributions for affected user and administrator behavior.
|
||||
- Keep feature content here; `govoplan-docs` projects it without importing Files internals.
|
||||
- Maintain a static user/admin baseline and run `/mnt/DATA/git/govoplan/tools/checks/check-manifest-shapes.py` after behavior or manifest changes.
|
||||
|
||||
## Scope
|
||||
|
||||
This repository owns the `files` module: managed file storage APIs, file metadata, shares, uploads/downloads, folder and pattern helpers, backend module manifest, and `@govoplan/files-webui`.
|
||||
@@ -18,8 +24,8 @@ cd /mnt/DATA/git/govoplan-core
|
||||
For combined checks, run:
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-core
|
||||
./scripts/check-focused.sh
|
||||
cd /mnt/DATA/git/govoplan
|
||||
tools/checks/check-focused.sh
|
||||
```
|
||||
|
||||
For WebUI permutation checks that include files:
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# govoplan-files
|
||||
|
||||
<!-- govoplan-repository-type:start -->
|
||||
**Repository type:** module (domain).
|
||||
<!-- govoplan-repository-type:end -->
|
||||
|
||||
GovOPlaN Files is the managed file module. It bundles backend storage APIs and the Files WebUI package so file features can be installed as one module.
|
||||
|
||||
## Ownership
|
||||
@@ -46,10 +50,166 @@ Frontend package:
|
||||
@govoplan/files-webui
|
||||
```
|
||||
|
||||
The campaign module can integrate with files when both modules are installed, for example for managed attachment selection and campaign file sharing.
|
||||
The campaign module can integrate with files when both modules are installed,
|
||||
for example for managed attachment selection and campaign file sharing. Files
|
||||
does not import campaign internals; campaign share/existence checks use the core
|
||||
`campaigns.access` capability registered by the campaign module.
|
||||
|
||||
Files also publishes the optional `privacy.dsar.files` capability. The Core
|
||||
data-subject workflow can use it to collect bounded, tenant-scoped file,
|
||||
version, share, folder, evidence, and non-secret configuration metadata for a
|
||||
direct membership subject. The provider may revoke a subject-targeted share or
|
||||
detach mutable actor references idempotently, but it never exports credential
|
||||
material or raw bytes and never bypasses Files retention, legal hold, evidence,
|
||||
purge approval, audit, or recovery controls. See the handbook's data-subject
|
||||
request coverage section for the review and erasure boundary.
|
||||
|
||||
Platform RBAC and governance rules are documented in `govoplan-core/docs/`.
|
||||
|
||||
Managed files can carry source provenance for connector and import workflows.
|
||||
Upload callers may provide `source_provenance_json` and `source_revision`; the
|
||||
module stores those values under file metadata, returns normalized
|
||||
`source_provenance` and `source_revision` fields in file responses, and carries
|
||||
them into managed campaign attachment matches for frozen execution evidence.
|
||||
|
||||
Connector policy preflight is available through
|
||||
`POST /api/v1/files/connector-policy/evaluate`, and provenance-bearing uploads
|
||||
can pass `connector_policy_json` to enforce the same policy before file content
|
||||
is read. Policy payloads contain ordered `sources`; each source has
|
||||
`scope_type`, optional `scope_id`, optional `label`, and a `policy` object. The
|
||||
policy object supports `allow`/`allowlist`/`whitelist` and
|
||||
`deny`/`denylist`/`blacklist` rules for `connectors`, `providers`,
|
||||
`external_ids`, `external_paths`, and `external_urls`. Deny rules win across the
|
||||
hierarchy; allow rules narrow access at each source that defines them.
|
||||
|
||||
Connector endpoint settings are exposed through governed connector profiles.
|
||||
Profiles can be supplied as JSON through
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`,
|
||||
`endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and
|
||||
deployment-owned credential references such as `password_env`, `token_env`, or
|
||||
`secret_ref`. Environment references require an exact name in the deployment-wide
|
||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
|
||||
process environment variables and may use only Files-owned encrypted password or
|
||||
token values. API-created `secret_ref` values fail closed until Files has an
|
||||
ownership contract that can confirm provider-side deletion. Legacy external
|
||||
references are treated as non-owned: deleting a profile or credential detaches
|
||||
and audits the reference but never passes it to an arbitrary secret provider.
|
||||
Profile and credential deletion immediately clears encrypted values, credential
|
||||
identities, deployment references, and private metadata in the same transaction
|
||||
as the non-secret audit record; an audit failure rolls the deletion back.
|
||||
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
|
||||
current principal (system, tenant, user, group, or accessible campaign scope) and
|
||||
redacts secret values and environment variable names. Use the returned
|
||||
`policy_sources` with connector policy preflight before importing files.
|
||||
`GET /api/v1/files/connectors/providers` exposes provider descriptors for
|
||||
Seafile, Nextcloud, WebDAV, SMB, NFS, and local filesystem connectors. The
|
||||
descriptor declares implementation status, optional dependencies, permission
|
||||
mapping, sync/index strategy, conflict handling, preview behavior, and audit
|
||||
events so provider coverage remains visible without forcing every optional
|
||||
protocol dependency to be installed.
|
||||
`GET /api/v1/files/connectors/profiles/{profile_id}/browse` provides read-only
|
||||
connector browsing. Browse entries use a shared `library`/`folder`/`file` shape,
|
||||
run profile policy with the `browse` operation, and support Seafile,
|
||||
WebDAV/Nextcloud, and SMB when the optional `smb` extra is installed. Seafile
|
||||
profiles browse libraries and directories via Seafile's read-only API; profiles
|
||||
can still opt into WebDAV browsing by setting `metadata.webdav_endpoint_url` or
|
||||
`metadata.browse_protocol` to `webdav`. S3-compatible profiles support bucket
|
||||
and prefix browsing when the optional `s3` extra is installed.
|
||||
`POST /api/v1/files/connectors/profiles/{profile_id}/import` imports a Seafile
|
||||
or WebDAV/Nextcloud/SMB file into managed storage through the same governance,
|
||||
conflict handling, source provenance, and connector audit path as direct
|
||||
uploads. The Seafile provider uses account-token auth and the native file
|
||||
download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET`
|
||||
requests against the configured WebDAV endpoint. SMB profiles use
|
||||
`smb://server[:port]/share[/path]` endpoints and deployment-owned or encrypted
|
||||
stored credentials through `smbprotocol`. Files installs a pinned transport for
|
||||
every initial session, reconnect, alias, and DFS referral target. The deployment
|
||||
private-network policy is re-evaluated immediately before each socket opens.
|
||||
|
||||
S3 connector browse/import binds botocore HTTP and HTTPS pools to the same pinned
|
||||
socket policy. Retries, redirects, endpoint discovery, bucket aliases, and new
|
||||
connections are therefore revalidated while TLS keeps the configured hostname
|
||||
for SNI and certificate checks. Connector clients do not use outbound proxies or
|
||||
ambient AWS credential discovery; configure credentials on the governed profile,
|
||||
or explicitly use an anonymous profile for public objects. Keep connector spaces
|
||||
read-only unless a remote write is explicitly needed.
|
||||
An S3 profile with the `write` capability can back a two-way connector space.
|
||||
`POST /api/v1/files/connector-spaces/{space_id}/write-back` then conditionally
|
||||
creates or replaces one remote object from a managed file, records durable
|
||||
recovery intent before the provider effect, and verifies request/content markers
|
||||
afterward. Automatic remote delete, rename, move, and ACL propagation remain
|
||||
disabled; other connector providers remain read-only.
|
||||
An incompatible SDK upgrade fails closed before a usable client/session is
|
||||
returned.
|
||||
|
||||
Durable platform storage has a separate deployment boundary: installer-owned
|
||||
Garage is accepted only at its exact generated endpoint, while an
|
||||
operator-selected external backend requires a clean HTTPS origin and
|
||||
`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`. That flag is deployment configuration,
|
||||
cannot be supplied through a Files connector profile, and does not replace
|
||||
operator responsibility for DNS, certificates, egress, bucket policy,
|
||||
versioning, and recovery.
|
||||
|
||||
The actual local/S3 backend implementation is owned by Core so Files, Campaign,
|
||||
and workers resolve the same object namespace. Files owns file metadata and key
|
||||
layout. Node-local storage is supported only for `local` or one-host
|
||||
`host-shared` profiles; multi-host `shared` deployments require S3.
|
||||
|
||||
Destructive Files-module retirement applies the same credential lifecycle before
|
||||
dropping tables. Every remaining Files-owned encrypted connector secret is
|
||||
scrubbed and audited first, while legacy non-owned external references are
|
||||
detached and identified as such in the audit record. Retirement does not claim
|
||||
or attempt provider-side deletion for references Files cannot prove it owns.
|
||||
|
||||
Local connector development assets live in `dev/connectors/`. The compose stack
|
||||
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
|
||||
manual interoperability testing.
|
||||
|
||||
Connector and collaboration ownership boundaries are documented in
|
||||
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
|
||||
The role-adaptive user, administration, integration, and operator guide is the
|
||||
[Files handbook](docs/FILES_HANDBOOK.md).
|
||||
The Files route, connector surfaces, consequence classes, and pattern-language
|
||||
verification are recorded in
|
||||
[Files interface pattern migration](docs/INTERFACE_PATTERN_MIGRATION.md).
|
||||
|
||||
Archive imports use a two-phase preview and confirmation flow for ZIP, TAR,
|
||||
TAR.GZ, TAR.BZ2, and TAR.XZ. Requests are spooled to bounded temporary files;
|
||||
the server validates paths, entry count, expanded size, and expansion ratio,
|
||||
then returns a 30-minute tenant/user-bound preview token. Confirmation reuploads
|
||||
the original archive and stores only the selected members. Password-protected
|
||||
ZIP passwords remain request-only and are never included in the preview token.
|
||||
|
||||
Managed blob writes and applied orphan cleanup use Core's durable recovery
|
||||
ledger. On PostgreSQL, intent, request digests, recovery mode, and a distributed
|
||||
lease are committed before physical storage effects; the Files session commit
|
||||
verifies database and streamed object evidence, while rollback compensates only
|
||||
a newly reserved unreferenced key. Development SQLite records blob intent in
|
||||
the caller transaction to avoid its second-writer deadlock, then verifies on
|
||||
commit or reconstructs compensation evidence after handled rollback. Because a
|
||||
hard loss before that commit can leave an unrecorded object, SQLite requires a
|
||||
complete integrity scan after a crash and is not a production recovery profile.
|
||||
New object keys are opaque and do not retain the uploaded filename. Uncertain
|
||||
or mismatched effects remain visible through Ops.
|
||||
|
||||
Operators with `files:file:admin` can run bounded, resumable integrity scans in
|
||||
Administration. Scan batches and finding actions carry monotonic revisions;
|
||||
stale resume, recheck, or cleanup requests fail before touching object storage.
|
||||
Orphan cleanup always requires a dry-run preview followed by separate
|
||||
confirmation and records recovery-ledger evidence.
|
||||
|
||||
Bulk rename and transfer APIs are owner-scoped: callers must provide the active
|
||||
user or group file space with `owner_type` and `owner_id`. The storage layer
|
||||
keeps a named legacy file-only helper for historical callers that lack owner
|
||||
context, and regression tests cover its write-access checks.
|
||||
|
||||
Optional producer modules can persist generated output through the provider-
|
||||
neutral `files.artifact_store` capability. Files remains responsible for upload
|
||||
authorization, ownership, path normalization, versioning, and blob storage;
|
||||
producers receive stable file/version references without importing Files
|
||||
internals. See [Generated Artifact Store](docs/GENERATED_ARTIFACT_STORE.md).
|
||||
|
||||
## Release packaging
|
||||
|
||||
The repository root includes a `package.json` for git-based WebUI installs. It exports the package `@govoplan/files-webui` from `webui/src` so release builds can depend on tagged git refs instead of local `file:` paths.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
NEXTCLOUD_HOST_PORT=9081
|
||||
NEXTCLOUD_DB_ROOT_PASSWORD=govoplan-nextcloud-root
|
||||
NEXTCLOUD_DB_PASSWORD=govoplan-nextcloud
|
||||
NEXTCLOUD_ADMIN_USER=admin
|
||||
NEXTCLOUD_ADMIN_PASSWORD=govoplan-nextcloud-admin
|
||||
|
||||
SEAFILE_HOST_PORT=9082
|
||||
SEAFILE_MYSQL_ROOT_PASSWORD=govoplan-seafile-root
|
||||
SEAFILE_ADMIN_EMAIL=admin@example.local
|
||||
SEAFILE_ADMIN_PASSWORD=govoplan-seafile-admin
|
||||
|
||||
WEBDAV_HOST_PORT=9083
|
||||
WEBDAV_USER=govoplan
|
||||
WEBDAV_PASSWORD=govoplan-webdav
|
||||
|
||||
SMB_HOST_PORT=1445
|
||||
SMB_SHARE_NAME=files
|
||||
SMB_USER=govoplan
|
||||
SMB_PASSWORD=govoplan-smb
|
||||
|
||||
MINIO_API_HOST_PORT=9000
|
||||
MINIO_CONSOLE_HOST_PORT=9001
|
||||
MINIO_ROOT_USER=govoplan
|
||||
MINIO_ROOT_PASSWORD=govoplan-minio
|
||||
MINIO_BUCKET=govoplan
|
||||
@@ -0,0 +1,180 @@
|
||||
# Connector Dev Stack
|
||||
|
||||
This directory keeps local Docker Compose assets for GovOPlaN file connector
|
||||
development. The stack is intentionally separate from production deployment and
|
||||
binds services to localhost high ports.
|
||||
|
||||
## Start
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
cp .env.example .env
|
||||
mkdir -p data/smb data/webdav
|
||||
docker compose up -d nextcloud nextcloud-db webdav smb minio
|
||||
docker compose up -d seafile-db seafile-memcached seafile
|
||||
```
|
||||
|
||||
If the SMB service was already running before a compose change, recreate it so
|
||||
the image and share configuration are applied:
|
||||
|
||||
```bash
|
||||
docker compose rm -sf smb
|
||||
docker compose up -d --build smb
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
|
||||
- Nextcloud: `http://127.0.0.1:9081`, WebDAV root
|
||||
`http://127.0.0.1:9081/remote.php/dav/files/admin/`
|
||||
- Seafile: `http://127.0.0.1:9082`, WebDAV root
|
||||
`http://127.0.0.1:9082/seafdav/`
|
||||
- WebDAV: `http://127.0.0.1:9083`
|
||||
- SMB: `smb://127.0.0.1:1445/files`
|
||||
- MinIO/S3 API: `http://127.0.0.1:9000`, console
|
||||
`http://127.0.0.1:9001`
|
||||
|
||||
The local fixture data under `data/` is ignored by git.
|
||||
|
||||
## GovOPlaN Profile Config
|
||||
|
||||
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. This deployment-owned configuration
|
||||
may reference environment variables only when their exact names are listed in
|
||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; profile API responses expose only the
|
||||
credential source and configured state.
|
||||
|
||||
Example local profile file:
|
||||
|
||||
```json
|
||||
{
|
||||
"profiles": [
|
||||
{
|
||||
"id": "dev-seafile",
|
||||
"label": "Dev Seafile",
|
||||
"provider": "seafile",
|
||||
"endpoint_url": "http://127.0.0.1:9082",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "admin@example.local",
|
||||
"password_env": "SEAFILE_ADMIN_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"metadata": { "webdav_endpoint_url": "http://127.0.0.1:9082/seafdav/" },
|
||||
"policy": { "allow": { "providers": ["seafile"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-nextcloud",
|
||||
"label": "Dev Nextcloud",
|
||||
"provider": "nextcloud",
|
||||
"endpoint_url": "http://127.0.0.1:9081/remote.php/dav/files/admin/",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "admin",
|
||||
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["nextcloud", "webdav"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-webdav",
|
||||
"label": "Dev WebDAV",
|
||||
"provider": "webdav",
|
||||
"endpoint_url": "http://127.0.0.1:9083",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "WEBDAV_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["webdav"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-smb",
|
||||
"label": "Dev SMB",
|
||||
"provider": "smb",
|
||||
"endpoint_url": "smb://127.0.0.1:1445/files",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "SMB_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["smb"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"label": "Dev MinIO",
|
||||
"provider": "s3",
|
||||
"endpoint_url": "http://127.0.0.1:9000",
|
||||
"base_path": "",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "MINIO_ROOT_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"metadata": {
|
||||
"bucket": "govoplan",
|
||||
"region": "us-east-1",
|
||||
"path_style": true,
|
||||
"verify_tls": false
|
||||
},
|
||||
"policy": { "allow": { "providers": ["s3"] } }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Start GovOPlaN with:
|
||||
|
||||
```bash
|
||||
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
|
||||
export GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST=SEAFILE_ADMIN_PASSWORD,NEXTCLOUD_ADMIN_PASSWORD,WEBDAV_PASSWORD,SMB_PASSWORD,MINIO_ROOT_PASSWORD
|
||||
export GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true
|
||||
```
|
||||
|
||||
## Smoke Test
|
||||
|
||||
Run the connector helper smoke checks from an environment where
|
||||
`govoplan-files` and `httpx` are importable:
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py
|
||||
```
|
||||
|
||||
The script reads `.env` from this directory when present and seeds tiny WebDAV,
|
||||
Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud use the pinned HTTP
|
||||
transport. SMB uses the Files-owned `smbprotocol` connection cache so initial
|
||||
connections, reconnects, aliases, and DFS referral targets are pinned. S3 uses
|
||||
pinned botocore pools for retries, redirects, discovery, and aliases. Every
|
||||
socket destination is revalidated when it opens, including when private-network
|
||||
access is enabled for these local fixtures.
|
||||
|
||||
SMB and S3 are optional by default: a missing provider dependency or unavailable
|
||||
target is reported as `SKIP`. Use `--require-smb` and `--require-s3` to turn such
|
||||
provider failures into a non-zero smoke result during deployment validation. The
|
||||
bundled Samba service is a deterministic single-share target, not a DFS
|
||||
topology. Referral and reconnect behavior is covered by the transport contract
|
||||
tests; deployments using DFS must additionally run the required SMB smoke check
|
||||
against their actual referral topology and record the resolved peers.
|
||||
|
||||
SMB smoke checks need the optional Python dependency in the environment running
|
||||
the script:
|
||||
|
||||
```bash
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
|
||||
```
|
||||
|
||||
S3 smoke checks need the optional boto3 dependency in the environment running
|
||||
the script:
|
||||
|
||||
```bash
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[s3]
|
||||
```
|
||||
|
||||
The SMB service is built from `dev/connectors/smb/` so the development share is
|
||||
deterministic: one `files` share backed by `data/smb`, with the credentials from
|
||||
`.env`.
|
||||
|
||||
The SMB image runs as the non-root `govoplan` user with UID/GID `1000` and
|
||||
listens on unprivileged container port `1445`. The default host endpoint remains
|
||||
`smb://127.0.0.1:1445/files`. `SMB_USER` must stay `govoplan` unless the image is
|
||||
rebuilt with a matching user; `SMB_PORT` must be `1024` or higher. `SMB_PASSWORD`
|
||||
is applied when the image is built, so rebuild the `smb` service after changing
|
||||
it in `.env`.
|
||||
@@ -0,0 +1,2 @@
|
||||
*
|
||||
!.gitignore
|
||||
@@ -0,0 +1,120 @@
|
||||
name: govoplan-files-connectors
|
||||
|
||||
services:
|
||||
nextcloud-db:
|
||||
image: mariadb:10.11
|
||||
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${NEXTCLOUD_DB_ROOT_PASSWORD:-govoplan-nextcloud-root}
|
||||
MYSQL_DATABASE: nextcloud
|
||||
MYSQL_USER: nextcloud
|
||||
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
|
||||
volumes:
|
||||
- nextcloud-db:/var/lib/mysql
|
||||
|
||||
nextcloud:
|
||||
image: nextcloud:apache
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- nextcloud-db
|
||||
ports:
|
||||
- "127.0.0.1:${NEXTCLOUD_HOST_PORT:-9081}:80"
|
||||
environment:
|
||||
MYSQL_HOST: nextcloud-db
|
||||
MYSQL_DATABASE: nextcloud
|
||||
MYSQL_USER: nextcloud
|
||||
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
|
||||
NEXTCLOUD_ADMIN_USER: ${NEXTCLOUD_ADMIN_USER:-admin}
|
||||
NEXTCLOUD_ADMIN_PASSWORD: ${NEXTCLOUD_ADMIN_PASSWORD:-govoplan-nextcloud-admin}
|
||||
NEXTCLOUD_TRUSTED_DOMAINS: "localhost 127.0.0.1"
|
||||
volumes:
|
||||
- nextcloud:/var/www/html
|
||||
|
||||
seafile-db:
|
||||
image: mariadb:10.11
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
|
||||
MARIADB_AUTO_UPGRADE: "1"
|
||||
MYSQL_LOG_CONSOLE: "true"
|
||||
volumes:
|
||||
- seafile-db:/var/lib/mysql
|
||||
|
||||
seafile-memcached:
|
||||
image: memcached:1.6
|
||||
restart: unless-stopped
|
||||
command: memcached -m 256
|
||||
|
||||
seafile:
|
||||
image: seafileltd/seafile-mc:11.0-latest
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- seafile-db
|
||||
- seafile-memcached
|
||||
ports:
|
||||
- "127.0.0.1:${SEAFILE_HOST_PORT:-9082}:80"
|
||||
environment:
|
||||
DB_HOST: seafile-db
|
||||
DB_ROOT_PASSWD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
|
||||
TIME_ZONE: Etc/UTC
|
||||
SEAFILE_ADMIN_EMAIL: ${SEAFILE_ADMIN_EMAIL:-admin@example.local}
|
||||
SEAFILE_ADMIN_PASSWORD: ${SEAFILE_ADMIN_PASSWORD:-govoplan-seafile-admin}
|
||||
SEAFILE_SERVER_LETSENCRYPT: "false"
|
||||
SEAFILE_SERVER_HOSTNAME: "127.0.0.1:${SEAFILE_HOST_PORT:-9082}"
|
||||
volumes:
|
||||
- seafile-data:/shared
|
||||
|
||||
webdav:
|
||||
image: rclone/rclone:latest
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- serve
|
||||
- webdav
|
||||
- /data
|
||||
- --addr
|
||||
- :8080
|
||||
- --user
|
||||
- ${WEBDAV_USER:-govoplan}
|
||||
- --pass
|
||||
- ${WEBDAV_PASSWORD:-govoplan-webdav}
|
||||
ports:
|
||||
- "127.0.0.1:${WEBDAV_HOST_PORT:-9083}:8080"
|
||||
volumes:
|
||||
- ./data/webdav:/data
|
||||
|
||||
smb:
|
||||
build:
|
||||
context: ./smb
|
||||
args:
|
||||
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
|
||||
image: govoplan-files-samba-dev:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
|
||||
SMB_USER: ${SMB_USER:-govoplan}
|
||||
SMB_PORT: ${SMB_PORT:-1445}
|
||||
ports:
|
||||
- "127.0.0.1:${SMB_HOST_PORT:-1445}:${SMB_PORT:-1445}"
|
||||
volumes:
|
||||
- ./data/smb:/storage
|
||||
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-govoplan}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-govoplan-minio}
|
||||
ports:
|
||||
- "127.0.0.1:${MINIO_API_HOST_PORT:-9000}:9000"
|
||||
- "127.0.0.1:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"
|
||||
volumes:
|
||||
- minio:/data
|
||||
|
||||
volumes:
|
||||
nextcloud-db:
|
||||
nextcloud:
|
||||
seafile-db:
|
||||
seafile-data:
|
||||
minio:
|
||||
@@ -0,0 +1,29 @@
|
||||
FROM alpine:3.20
|
||||
|
||||
ARG SMB_PASSWORD=govoplan-smb
|
||||
|
||||
RUN apk add --no-cache samba-server samba-common-tools \
|
||||
&& addgroup -S -g 1000 govoplan \
|
||||
&& adduser -S -D -H -h /nonexistent -s /sbin/nologin -u 1000 -G govoplan govoplan \
|
||||
&& mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba /var/log/samba/cores \
|
||||
&& printf '%s\n' \
|
||||
'[global]' \
|
||||
' passdb backend = tdbsam' \
|
||||
' private dir = /var/lib/samba/private' \
|
||||
' lock directory = /run/samba' \
|
||||
' state directory = /var/lib/samba' \
|
||||
' cache directory = /var/cache/samba' \
|
||||
' pid directory = /run/samba' \
|
||||
> /etc/samba/smb.conf \
|
||||
&& printf '%s\n%s\n' "$SMB_PASSWORD" "$SMB_PASSWORD" | smbpasswd -s -a govoplan \
|
||||
&& smbpasswd -e govoplan \
|
||||
&& chown -R govoplan:govoplan /storage /var/lib/samba /var/cache/samba /run/samba /etc/samba /var/log/samba
|
||||
|
||||
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
|
||||
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
|
||||
|
||||
EXPOSE 1445
|
||||
|
||||
USER govoplan:govoplan
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
share_name="${SMB_SHARE_NAME:-files}"
|
||||
user_name="${SMB_USER:-govoplan}"
|
||||
smb_port="${SMB_PORT:-1445}"
|
||||
runtime_user="$(id -un)"
|
||||
runtime_group="$(id -gn)"
|
||||
|
||||
case "$share_name" in
|
||||
"" | *[!A-Za-z0-9_.-]*)
|
||||
printf 'SMB_SHARE_NAME must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
case "$user_name" in
|
||||
"" | *[!A-Za-z0-9_.-]*)
|
||||
printf 'SMB_USER must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
case "$smb_port" in
|
||||
"" | *[!0-9]*)
|
||||
printf 'SMB_PORT must be numeric.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if [ "$smb_port" -lt 1024 ]; then
|
||||
printf 'SMB_PORT must be >= 1024 because the dev Samba container runs as a non-root user.\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$user_name" != "$runtime_user" ]; then
|
||||
printf 'SMB_USER=%s is not supported by the non-root dev image; use %s or rebuild the image with a matching user.\n' "$user_name" "$runtime_user" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba
|
||||
|
||||
cat > /etc/samba/smb.conf <<EOF
|
||||
[global]
|
||||
server role = standalone server
|
||||
workgroup = WORKGROUP
|
||||
security = user
|
||||
map to guest = Never
|
||||
guest account = $runtime_user
|
||||
server min protocol = SMB2
|
||||
server signing = mandatory
|
||||
smb ports = $smb_port
|
||||
load printers = no
|
||||
printing = bsd
|
||||
disable spoolss = yes
|
||||
log level = 1
|
||||
passdb backend = tdbsam
|
||||
private dir = /var/lib/samba/private
|
||||
lock directory = /run/samba
|
||||
state directory = /var/lib/samba
|
||||
cache directory = /var/cache/samba
|
||||
pid directory = /run/samba
|
||||
|
||||
[$share_name]
|
||||
path = /storage
|
||||
browseable = yes
|
||||
read only = no
|
||||
guest ok = no
|
||||
valid users = $user_name
|
||||
force user = $runtime_user
|
||||
force group = $runtime_group
|
||||
create mask = 0664
|
||||
directory mask = 0775
|
||||
EOF
|
||||
|
||||
if ! pdbedit -L -u "$user_name" >/dev/null 2>&1; then
|
||||
printf 'Samba user %s is missing from passdb. Rebuild the smb image so the non-root passdb is seeded.\n' "$user_name" >&2
|
||||
printf 'Run: docker compose build --no-cache smb && docker compose up -d smb\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec smbd --foreground --no-process-group --debug-stdout -p "$smb_port"
|
||||
@@ -0,0 +1,282 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import socket
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import browse_connector_profile
|
||||
from govoplan_files.backend.storage.connector_imports import read_connector_file
|
||||
from govoplan_files.backend.storage.connector_profiles import connector_profiles_from_payload
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
|
||||
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
|
||||
parser.add_argument("--require-s3", action="store_true", help="Fail if the S3 connector cannot browse/import.")
|
||||
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
|
||||
args = parser.parse_args()
|
||||
|
||||
_load_dotenv()
|
||||
_default_env()
|
||||
preflight_failures = _preflight_services(require_smb=args.require_smb, require_s3=args.require_s3)
|
||||
if preflight_failures:
|
||||
for failure in preflight_failures:
|
||||
print(f"FAIL {failure}")
|
||||
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb minio")
|
||||
return 1
|
||||
_seed_webdav_fixture()
|
||||
_seed_smb_fixture()
|
||||
try:
|
||||
_seed_nextcloud_fixture()
|
||||
except httpx.HTTPError as exc:
|
||||
print(f"FAIL dev-nextcloud seed failed: {exc}")
|
||||
return 1
|
||||
try:
|
||||
_seed_s3_fixture()
|
||||
except Exception as exc:
|
||||
if args.require_s3:
|
||||
print(f"FAIL dev-s3 seed failed: {exc}")
|
||||
return 1
|
||||
print(f"SKIP dev-s3 seed failed: {exc}")
|
||||
|
||||
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
|
||||
if args.debug_smb:
|
||||
_debug_smb(profiles["dev-smb"])
|
||||
failures: list[str] = []
|
||||
for profile_id, folder, file_path, expected in (
|
||||
("dev-webdav", "GovOPlaN", "GovOPlaN/webdav-live.txt", "webdav live fixture"),
|
||||
("dev-nextcloud", "GovOPlaN", "GovOPlaN/nextcloud-live.txt", "nextcloud live fixture"),
|
||||
):
|
||||
try:
|
||||
_exercise_profile(profiles[profile_id], folder=folder, file_path=file_path, expected=expected)
|
||||
except Exception as exc:
|
||||
failures.append(f"{profile_id}: {exc}")
|
||||
|
||||
try:
|
||||
_exercise_profile(profiles["dev-smb"], folder="GovOPlaN", file_path="GovOPlaN/smb-live.txt", expected="smb live fixture")
|
||||
except Exception as exc:
|
||||
message = f"dev-smb: {exc}"
|
||||
if args.require_smb:
|
||||
failures.append(message)
|
||||
else:
|
||||
print(f"SKIP {message}")
|
||||
|
||||
try:
|
||||
_exercise_profile(profiles["dev-s3"], folder="GovOPlaN", file_path="GovOPlaN/s3-live.txt", expected="s3 live fixture")
|
||||
except Exception as exc:
|
||||
message = f"dev-s3: {exc}"
|
||||
if args.require_s3:
|
||||
failures.append(message)
|
||||
else:
|
||||
print(f"SKIP {message}")
|
||||
|
||||
if failures:
|
||||
for failure in failures:
|
||||
print(f"FAIL {failure}")
|
||||
return 1
|
||||
print("OK connector dev stack smoke checks passed")
|
||||
return 0
|
||||
|
||||
|
||||
def _default_env() -> None:
|
||||
defaults = {
|
||||
"NEXTCLOUD_ADMIN_USER": "admin",
|
||||
"NEXTCLOUD_ADMIN_PASSWORD": "govoplan-nextcloud-admin",
|
||||
"WEBDAV_USER": "govoplan",
|
||||
"WEBDAV_PASSWORD": "govoplan-webdav",
|
||||
"SMB_SHARE_NAME": "files",
|
||||
"SMB_USER": "govoplan",
|
||||
"SMB_PASSWORD": "govoplan-smb",
|
||||
"MINIO_ROOT_USER": "govoplan",
|
||||
"MINIO_ROOT_PASSWORD": "govoplan-minio",
|
||||
"MINIO_BUCKET": "govoplan",
|
||||
}
|
||||
for key, value in defaults.items():
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
|
||||
def _load_dotenv() -> None:
|
||||
path = ROOT / ".env"
|
||||
if not path.exists():
|
||||
return
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
text = line.strip()
|
||||
if not text or text.startswith("#") or "=" not in text:
|
||||
continue
|
||||
key, value = text.split("=", 1)
|
||||
key = key.strip()
|
||||
if not key or key in os.environ:
|
||||
continue
|
||||
os.environ[key] = value.strip().strip("\"'")
|
||||
|
||||
|
||||
def _preflight_services(*, require_smb: bool, require_s3: bool) -> list[str]:
|
||||
failures: list[str] = []
|
||||
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
|
||||
try:
|
||||
response = httpx.get(nextcloud_url, timeout=3.0)
|
||||
if response.status_code != 200:
|
||||
failures.append(f"dev-nextcloud expected HTTP 200 at {nextcloud_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-nextcloud is not reachable at {nextcloud_url}: {exc}")
|
||||
|
||||
webdav_url = f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/"
|
||||
try:
|
||||
response = httpx.get(webdav_url, timeout=3.0)
|
||||
if response.status_code not in {200, 401}:
|
||||
failures.append(f"dev-webdav expected HTTP 200/401 at {webdav_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-webdav is not reachable at {webdav_url}: {exc}")
|
||||
|
||||
if require_smb:
|
||||
smb_port = int(os.getenv("SMB_HOST_PORT", "1445"))
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", smb_port), timeout=3.0):
|
||||
pass
|
||||
except OSError as exc:
|
||||
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
|
||||
if require_s3:
|
||||
minio_url = f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}/minio/health/live"
|
||||
try:
|
||||
response = httpx.get(minio_url, timeout=3.0)
|
||||
if response.status_code != 200:
|
||||
failures.append(f"dev-s3 expected HTTP 200 at {minio_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-s3 is not reachable at {minio_url}: {exc}")
|
||||
return failures
|
||||
|
||||
|
||||
def _profile_payloads() -> list[dict[str, object]]:
|
||||
return [
|
||||
{
|
||||
"id": "dev-webdav",
|
||||
"provider": "webdav",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("WEBDAV_USER", "govoplan"),
|
||||
"password_env": "WEBDAV_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-nextcloud",
|
||||
"provider": "nextcloud",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("NEXTCLOUD_ADMIN_USER", "admin"),
|
||||
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-smb",
|
||||
"provider": "smb",
|
||||
"endpoint_url": f"smb://127.0.0.1:{os.getenv('SMB_HOST_PORT', '1445')}/{os.getenv('SMB_SHARE_NAME', 'files')}",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("SMB_USER", "govoplan"),
|
||||
"password_env": "SMB_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"provider": "s3",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||
"password_env": "MINIO_ROOT_PASSWORD",
|
||||
"metadata": {
|
||||
"bucket": os.getenv("MINIO_BUCKET", "govoplan"),
|
||||
"region": "us-east-1",
|
||||
"path_style": True,
|
||||
"verify_tls": False,
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _exercise_profile(profile, *, folder: str, file_path: str, expected: str) -> None:
|
||||
items = browse_connector_profile(profile, path=folder)
|
||||
paths = {item.path for item in items}
|
||||
if file_path not in paths:
|
||||
raise RuntimeError(f"{file_path!r} not found; saw {sorted(paths)!r}")
|
||||
downloaded = read_connector_file(profile, library_id="", path=file_path, max_bytes=1024 * 1024)
|
||||
text = downloaded.data.decode("utf-8").strip()
|
||||
if text != expected:
|
||||
raise RuntimeError(f"{file_path!r} content mismatch: {text!r}")
|
||||
print(f"OK {profile.id} browse/import {file_path}")
|
||||
|
||||
|
||||
def _debug_smb(profile) -> None:
|
||||
try:
|
||||
import smbclient
|
||||
except ImportError as exc:
|
||||
print(f"DEBUG smbclient import failed: {exc}")
|
||||
return
|
||||
from govoplan_files.backend.storage.connector_browse import _smb_client_kwargs, _smb_location, _smb_unc_path
|
||||
|
||||
location = _smb_location(profile)
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
print(f"DEBUG SMB endpoint=//{location.server}:{location.port}/{location.share} root_path={location.root_path!r}")
|
||||
print(f"DEBUG SMB user={kwargs.get('username')!r} require_signing={kwargs.get('require_signing')!r} auth_protocol={kwargs.get('auth_protocol')!r}")
|
||||
try:
|
||||
smbclient.reset_connection_cache()
|
||||
except Exception as exc:
|
||||
print(f"DEBUG SMB reset cache failed: {exc}")
|
||||
for path in (_smb_unc_path(location, ""), _smb_unc_path(location, "GovOPlaN")):
|
||||
try:
|
||||
print(f"DEBUG SMB list {path}: {smbclient.listdir(path, **kwargs)}")
|
||||
except Exception as exc:
|
||||
print(f"DEBUG SMB list {path} failed: {type(exc).__name__}: {exc}")
|
||||
|
||||
|
||||
def _seed_webdav_fixture() -> None:
|
||||
path = ROOT / "data" / "webdav" / "GovOPlaN"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
(path / "webdav-live.txt").write_text("webdav live fixture\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _seed_smb_fixture() -> None:
|
||||
path = ROOT / "data" / "smb" / "GovOPlaN"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
(path / "smb-live.txt").write_text("smb live fixture\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _seed_nextcloud_fixture() -> None:
|
||||
base_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/GovOPlaN"
|
||||
auth = (os.getenv("NEXTCLOUD_ADMIN_USER", "admin"), os.getenv("NEXTCLOUD_ADMIN_PASSWORD", "govoplan-nextcloud-admin"))
|
||||
response = httpx.request("MKCOL", base_url, auth=auth, timeout=15.0)
|
||||
if response.status_code not in {201, 405}:
|
||||
raise RuntimeError(f"Nextcloud MKCOL failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||
response = httpx.put(f"{base_url}/nextcloud-live.txt", content=b"nextcloud live fixture\n", auth=auth, timeout=15.0)
|
||||
if response.status_code not in {200, 201, 204}:
|
||||
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||
|
||||
|
||||
def _seed_s3_fixture() -> None:
|
||||
try:
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("boto3 is not installed; install govoplan-files[s3]") from exc
|
||||
bucket = os.getenv("MINIO_BUCKET", "govoplan")
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||
aws_access_key_id=os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||
aws_secret_access_key=os.getenv("MINIO_ROOT_PASSWORD", "govoplan-minio"),
|
||||
region_name="us-east-1",
|
||||
config=Config(s3={"addressing_style": "path"}),
|
||||
)
|
||||
try:
|
||||
client.create_bucket(Bucket=bucket)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code")
|
||||
if code not in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
|
||||
raise
|
||||
client.put_object(Bucket=bucket, Key="GovOPlaN/s3-live.txt", Body=b"s3 live fixture\n", ContentType="text/plain")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,78 @@
|
||||
# File Connector Boundary
|
||||
|
||||
GovOPlaN Files owns the managed-file boundary: frozen blobs, versions, shares,
|
||||
campaign attachment evidence, provenance, connector policy, connector profiles,
|
||||
and read-only browse/import APIs that turn external files into managed
|
||||
GovOPlaN files.
|
||||
|
||||
The built-in connector layer is intentionally on-demand. It does not run remote
|
||||
indexing, background sync, remote mutation, or upstream permission management.
|
||||
Connectors may browse an external source, import one selected file, freeze it as
|
||||
a managed file, record provenance, and audit the access.
|
||||
|
||||
## Built-In Baseline
|
||||
|
||||
The files module may keep small baseline providers when they are needed for
|
||||
normal product workflows and can share the same governance model:
|
||||
|
||||
- Seafile through the native read/download API
|
||||
- Nextcloud through WebDAV
|
||||
- generic WebDAV
|
||||
- SMB through the optional `smb` extra
|
||||
- S3-compatible stores through the optional `s3` extra
|
||||
|
||||
These providers are surfaced through connector descriptors at
|
||||
`GET /api/v1/files/connectors/providers`. Provider descriptors declare whether
|
||||
the provider is implemented, whether its optional dependency is installed, and
|
||||
which browse/import behaviors are available.
|
||||
|
||||
## Separate Connector Module Candidates
|
||||
|
||||
A separate connector module becomes appropriate when integration needs exceed
|
||||
on-demand browse/import:
|
||||
|
||||
- background indexing or synchronization
|
||||
- bidirectional remote mutation
|
||||
- long-running transfer workers
|
||||
- provider-specific credential lifecycle or OAuth flows
|
||||
- DMS/eAkte metadata models, registers, retention, or filing plans
|
||||
- S3-compatible object store administration
|
||||
- NFS host-mount lifecycle or sidecar coordination
|
||||
- provider-specific WebUI administration beyond profile fields
|
||||
|
||||
In that model, `govoplan-files` should keep the managed-file import contract,
|
||||
policy checks, provenance model, and audit events. A connector module should own
|
||||
provider-specific discovery, synchronization, credentials, health checks, and
|
||||
any remote write behavior.
|
||||
|
||||
## Provider Responsibilities
|
||||
|
||||
Every provider must:
|
||||
|
||||
- enforce GovOPlaN profile visibility and connector policy before browse/import
|
||||
- keep credentials as encrypted values or scoped secret references, never API
|
||||
response values; process-environment references are allowed only in
|
||||
deployment-owned profiles with an exact deployment allowlist
|
||||
- import external files into managed storage before they are used in campaigns
|
||||
or workflows
|
||||
- preserve source provenance and revision metadata
|
||||
- emit connector audit events for imported or accessed files
|
||||
- treat remote ACLs as upstream checks, not as a replacement for GovOPlaN policy
|
||||
- use a transport that pins every connection to a policy-validated DNS/IP answer
|
||||
and revalidates redirects; SDK transports without that guarantee fail closed
|
||||
|
||||
SMB initial connections, reconnects, aliases, and DFS referral targets are
|
||||
created through a Files-owned pinned `smbprotocol` transport. S3 HTTP/HTTPS pools
|
||||
use the equivalent botocore adapter for every connection selected by retries,
|
||||
redirects, endpoint discovery, and virtual-host addressing. Both adapters apply
|
||||
the deployment-wide private-network policy at socket creation. S3 keeps the
|
||||
configured hostname for TLS SNI and certificate verification, but does not use
|
||||
outbound proxies or ambient AWS credential discovery. An incompatible optional
|
||||
SDK release fails closed before a usable session or client is returned.
|
||||
|
||||
## Non-Goals For Files
|
||||
|
||||
Files does not own collaborative editing, comments, document review workflows,
|
||||
remote lock orchestration, DMS records management, or external system data
|
||||
models. Those belong to future document, workflow, DMS, or connector modules and
|
||||
should integrate through capabilities and managed-file imports.
|
||||
@@ -0,0 +1,155 @@
|
||||
# Connector Spaces
|
||||
|
||||
GovOPlaN Files should treat external file shares as two separate product
|
||||
objects:
|
||||
|
||||
1. A governed connection profile defines a reusable remote endpoint such as a
|
||||
WebDAV server, Nextcloud account, Seafile server, or SMB share. Profiles are
|
||||
administered in settings at system or tenant scope, with the same
|
||||
inheritance and limit semantics used by mail-server profiles.
|
||||
2. A governed credential profile defines reusable authentication material for
|
||||
one provider or for any compatible provider. Credentials are administered
|
||||
separately from connection profiles and can carry their own allow/deny
|
||||
policy.
|
||||
3. A linked file space binds one concrete remote folder or library path from an
|
||||
allowed profile to a user or group. Linked spaces appear beside "My files"
|
||||
and group file spaces in the files module.
|
||||
|
||||
This keeps secrets, endpoint governance, and tenant limits in settings, while
|
||||
keeping user/group workspaces and concrete folder choices in the files module.
|
||||
|
||||
## Model
|
||||
|
||||
Connection profile:
|
||||
|
||||
- provider: `seafile`, `nextcloud`, `webdav`, `smb`, or a future provider
|
||||
- endpoint URL and optional base path
|
||||
- scope: `system` or `tenant` for administered profiles; user/group profiles
|
||||
may be allowed later only when policy explicitly permits them
|
||||
- optional credential profile id
|
||||
- capabilities: browse, sync, import, optional write when a provider supports it
|
||||
- profile-local policy for allow/deny rules
|
||||
|
||||
Credential profile:
|
||||
|
||||
- provider: a specific provider or any provider
|
||||
- scope: `system` or `tenant` for administered credentials
|
||||
- credential mode: anonymous, scoped secret reference, or encrypted stored
|
||||
password/token; environment references are reserved for deployment-owned JSON
|
||||
profiles and exact deployment allowlisting
|
||||
- username and redacted secret configuration
|
||||
- credential-local policy for allow/deny rules
|
||||
|
||||
Connector policy:
|
||||
|
||||
- system policy is the baseline
|
||||
- tenant policy inherits system policy and may narrow it unless system allows
|
||||
lower-level relaxation
|
||||
- future user/group policy may narrow tenant policy for self-service links
|
||||
- policies can allow or block providers, profile ids, endpoint URLs, external
|
||||
path prefixes, credential ids, credential inheritance, and local linked-space
|
||||
creation
|
||||
|
||||
Linked connector space:
|
||||
|
||||
- owner type: user or group
|
||||
- display name
|
||||
- connector profile id
|
||||
- remote library id or share id
|
||||
- remote root path
|
||||
- sync mode: manual initially; background sync is future work
|
||||
- read-only flag from provider/policy
|
||||
- active/deleted state
|
||||
|
||||
The linked space is addressable as a normal file space in the Files UI. It can
|
||||
browse the remote folder, synchronize one selected file, or run a bounded manual
|
||||
folder synchronization into managed storage. Background sync jobs remain a
|
||||
separate scheduling concern.
|
||||
|
||||
## Policy Semantics
|
||||
|
||||
Use mail-profile terminology because administrators already see it there:
|
||||
|
||||
- must use: lower scopes are restricted to selected profile ids or providers
|
||||
- can use: lower scopes may choose from inherited allowed profiles
|
||||
- shall not use anything outside: endpoint URLs and path prefixes are enforced
|
||||
by deny/allow rules before browse, import, or sync
|
||||
- may create local links: controls whether users/groups can add linked spaces
|
||||
from inherited profiles
|
||||
|
||||
Connector policy should provide an explainable effective policy response with
|
||||
source path entries: system, tenant, user, group, campaign when applicable.
|
||||
|
||||
## Current State
|
||||
|
||||
Implemented:
|
||||
|
||||
- provider descriptors for Seafile, Nextcloud, WebDAV, and SMB
|
||||
- database-backed connector profiles with system and tenant scope
|
||||
- database-backed connector credentials with system and tenant scope
|
||||
- encrypted stored password/token support for database credentials
|
||||
- JSON/environment-defined connector profiles with system, tenant, user, group,
|
||||
and campaign visibility
|
||||
- connector allow/deny policy enforcement before browse/import/sync, including
|
||||
provider, connection profile, credential profile, and path checks
|
||||
- read-only browse endpoints
|
||||
- import and sync into managed files with provenance and revision metadata
|
||||
- audit events for connector import, sync, and access
|
||||
- settings/admin UI sections for system and tenant file connections and
|
||||
credentials
|
||||
- linked connector-space rows owned by users or groups
|
||||
- file-space API responses that include linked connector spaces
|
||||
- Files UI entry point to create linked connector spaces from a browsed remote
|
||||
profile/folder
|
||||
- Files UI sync dialog for choosing a profile, browsing a remote folder, and
|
||||
syncing a selected file into a managed destination folder
|
||||
- Files UI connector-space view with provider/read-only/manual-sync state
|
||||
- bounded folder-level manual sync with recursive traversal, pagination,
|
||||
per-item review outcomes, source-provenance matching, and one summary audit
|
||||
event
|
||||
- Docker dev stack smoke checks for WebDAV, Nextcloud, and SMB
|
||||
|
||||
Missing:
|
||||
|
||||
- explicit connector profile policy rows equivalent to mail-profile policies
|
||||
- effective connector policy/explain UI equivalent to mail-profile policies
|
||||
- edit/manage actions for existing linked connector spaces
|
||||
- optional background sync or remote-write semantics
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
1. Persist governed connector profiles and policies.
|
||||
- `file_connector_profiles` exists for system and tenant profiles.
|
||||
- `file_connector_credentials` exists for system and tenant credentials.
|
||||
- Environment JSON profiles remain bootstrap/compatibility profiles.
|
||||
- Profile and credential CRUD endpoints exist for database-backed records.
|
||||
- Remaining: `file_connector_policies` plus effective policy/explain output.
|
||||
|
||||
2. Add linked connector spaces.
|
||||
- `file_connector_spaces` exists with owner user/group, profile id, library
|
||||
id, remote path, display label, sync mode, and active state.
|
||||
- Connector policy is enforced before creating or using a link.
|
||||
- Linked connector spaces are returned from `/api/v1/files/spaces`.
|
||||
|
||||
3. Surface linked spaces in the Files UI.
|
||||
- Linked spaces appear beside managed user/group spaces.
|
||||
- The add-space dialog browses an allowed profile and links the current
|
||||
remote folder/library root.
|
||||
- The connector browser/sync flow is reused when a linked space is open.
|
||||
- Remaining: edit/manage actions for existing linked spaces.
|
||||
|
||||
4. Add sync orchestration.
|
||||
- Manual selected-file and folder-level synchronization are available.
|
||||
- Folder sync preserves remote relative paths, bounds file count and depth,
|
||||
and reports created, updated, unchanged, skipped, conflict, policy-denied,
|
||||
and failed items without rolling back successful siblings.
|
||||
- The default conflict strategy is `skip`, so unrelated managed files are
|
||||
left untouched. `rename`, `reject`, and `overwrite` remain explicit choices.
|
||||
- Add optional scheduled/background sync separately; it must reuse the same
|
||||
policy, provenance, result, and audit contracts.
|
||||
|
||||
5. Add provider-specific expansion.
|
||||
- OAuth/secret-store credentials.
|
||||
- Remote write and delete only where policy and provider support it.
|
||||
- DMS/eAkte metadata integrations in a separate connector or documents
|
||||
module.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Document Collaboration Boundary
|
||||
|
||||
GovOPlaN Files is the governed managed-file module. It stores uploaded/imported
|
||||
blobs, versions, shares, provenance, ZIP imports/exports, connector imports, and
|
||||
campaign attachment evidence. It should stay reliable, auditable, and simple.
|
||||
|
||||
Collaborative document behavior should be owned by a future documents module or
|
||||
by provider-specific connector modules when the behavior is delegated to systems
|
||||
such as Nextcloud, Seafile, Collabora, OnlyOffice, OpenDesk, or DMS/eAkte
|
||||
platforms.
|
||||
|
||||
## Files Owns
|
||||
|
||||
- managed blobs and immutable version evidence
|
||||
- owner-scoped user/group file spaces
|
||||
- shares and access checks for managed files
|
||||
- connector browse/import into managed storage
|
||||
- source provenance, source revision, and audit events
|
||||
- freeze-before-send campaign attachment behavior
|
||||
- previews generated from managed file versions
|
||||
|
||||
## Documents Or Connectors Should Own
|
||||
|
||||
- co-editing sessions and editor launch URLs
|
||||
- comments, suggestions, document tasks, and review state
|
||||
- check-in/check-out, remote locks, and lock timeouts
|
||||
- semantic document versions beyond blob versions
|
||||
- template merge flows and generated-document lifecycle
|
||||
- external DMS metadata, filing plans, record categories, and retention classes
|
||||
- provider-specific sync state and conflict resolution
|
||||
- collaborative presence and notification behavior
|
||||
|
||||
## Integration Shape
|
||||
|
||||
The documents layer should integrate with files through stable contracts:
|
||||
|
||||
- create or import a managed file version for evidence points
|
||||
- attach source provenance when a file came from an external editor or DMS
|
||||
- ask files for read/download access rather than importing files internals
|
||||
- emit workflow/audit events when a collaborative document reaches a governed
|
||||
state such as reviewed, approved, frozen, sent, or archived
|
||||
- use connector providers for provider-specific browse/import/write behavior
|
||||
|
||||
Files may expose links to a document or connector module, but it should not own
|
||||
the collaboration state machine. A collaborative document can produce many
|
||||
working states; GovOPlaN Files should persist the governed snapshots that other
|
||||
modules can safely use as evidence.
|
||||
|
||||
## Practical Rule
|
||||
|
||||
If a feature is about storing, sharing, importing, downloading, or freezing a
|
||||
file, it belongs in `govoplan-files`. If a feature is about people jointly
|
||||
editing, reviewing, locking, commenting on, or routing a document through a
|
||||
semantic process, it belongs in a documents/workflow/DMS integration module.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
||||
# Generated Artifact Store
|
||||
|
||||
Files implements Core's optional `files.artifact_store` capability for modules
|
||||
that generate deterministic output without owning file storage.
|
||||
|
||||
The producer supplies bytes, filename, content type, destination folder,
|
||||
optional idempotency key, and bounded non-secret provenance. Files applies the
|
||||
actor's `files:file:upload` permission, tenant/user ownership, path rules,
|
||||
versioning, configured blob backend, and conflict behavior. An idempotency key
|
||||
is represented as source provenance so an unchanged retry does not create an
|
||||
unrelated file version.
|
||||
|
||||
The shared Files session owns finalization. Before a new managed object is
|
||||
written, Files commits a lease-fenced Core recovery operation containing only
|
||||
identifiers and digests. The caller's eventual session commit independently
|
||||
verifies both `FileBlob` metadata and stored bytes; rollback compensates only an
|
||||
unreferenced key. Producers must therefore complete the supplied transaction
|
||||
normally and must not bypass or replace Files session lifecycle handling.
|
||||
|
||||
The response contains only file/version identifiers, display path, media type,
|
||||
size, digest, and storage provenance. Producers must not put credentials,
|
||||
tokens, or rendered plaintext into metadata. Storing an artifact proves Files
|
||||
accepted it; it does not prove printing, mailing, or any other external effect.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Files Interface Pattern Migration
|
||||
|
||||
This inventory records the Files-owned part of the GovOPlaN interface pattern
|
||||
language. Core owns the shell and shared components; Files owns the composition
|
||||
and consequences described here.
|
||||
|
||||
## Surface inventory
|
||||
|
||||
| Surface | Primary task | Archetype | Consequence | Pattern evidence |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `/files` space and folder panes | Browse managed and connected content without losing location | Directory/explorer | Low for navigation; medium for exposing filenames and provenance | Full-height two-pane workspace, bounded panes, stable selection and contextual Help Center link |
|
||||
| `/files` toolbar and property filters | Find and act on the current selection | Explorer actions and local filtering | Medium for upload, move, copy, share and synchronization; high for delete | Actions remain beside the affected list, disabled controls explain permission/state/selection blockers, destructive work uses `ConfirmDialog` |
|
||||
| Upload/archive, transfer, rename and connector-import dialogs | Supply and review one bounded change | Adaptive create/edit or guided import | Medium to high because files, paths and external bytes change | Shared `Dialog`, `FileDropZone`, validation, conflict review, unsaved inputs and explicit confirmation |
|
||||
| File share dialog | Inspect and change access | Review/decision | High because another actor gains access | Shared dialog, access explanation, stable row actions and destructive confirmation |
|
||||
| System/tenant/group/user connector surfaces | Compare connections, credentials and effective policy | Administration/configuration | High because endpoints, secrets and inherited policy control external access | Shared `ConnectionTree`, adaptive forms, `ActionBlockerHint`, policy provenance and contextual admin help |
|
||||
| Connection and credential editors | Create or edit one governed endpoint or secret | Adaptive create/edit | High because a saved change may enable remote access | Relevant fields only, typed credential controls, discovery/test, advanced compatibility section, unsaved-change guard and disabled-save reasons |
|
||||
| Connector policy card | Narrow inherited connector access | Effective-policy editor | High because deny/allow changes affect lower scopes | Typed reference selectors, deny precedence warning, effective source evidence and permission blocker |
|
||||
| `files.widget.spaces` | See available managed/connected spaces and open Files | Dashboard widget | Low; names and provider state may still be sensitive | Shared loading/alert/status components, bounded item count, permission-filtered contribution |
|
||||
| Files chooser capability used by another module | Select a managed snapshot without importing Files internals | Directory chooser | Medium because the exact selected version becomes another module's input | Shared dialog/confirmation, capability boundary and exact file/version evidence |
|
||||
|
||||
## State and consequence contract
|
||||
|
||||
- Loading, errors, success, empty results, access explanations and confirmation
|
||||
use Core components. Files does not reproduce the application shell.
|
||||
- A connector that comes from deployment settings remains visible but read-only;
|
||||
its action explains that bootstrap configuration and a restart are required.
|
||||
- Missing permission, target, selection, endpoint, or compatible provider is an
|
||||
explained disabled state. It is not represented only by color or absence.
|
||||
- Provider metadata JSON is an expert compatibility escape hatch inside the
|
||||
collapsed shared advanced-options component. Ordinary connector setup uses
|
||||
typed provider, endpoint, credential, capability and policy controls.
|
||||
- Endpoint discovery and credential tests are explicit and report their result;
|
||||
they do not save the draft. Save remains the only committing action.
|
||||
- Connector/profile disable and managed-file delete remain confirmed actions and
|
||||
state their immediate effect. Soft deletion must not be described as purge.
|
||||
- External connector data, paths and credential references are rendered only in
|
||||
already-authorized administration or explorer contexts. Secret values are
|
||||
never returned for rendering.
|
||||
|
||||
## Accessibility and responsive evidence
|
||||
|
||||
Shared `Dialog` owns focus entry, Escape handling and focus return. Form and
|
||||
toolbar DOM order is the keyboard order; disabled-action tooltips are themselves
|
||||
focusable and expose the reason. The connector form uses semantic sections and
|
||||
labels, status is textual as well as colored, and result alerts are announced by
|
||||
the shared alert component. The explorer collapses to one column below 1050 px;
|
||||
connector forms and action rows collapse below 760 px while preserving source
|
||||
order. Long provider choices scroll inside the segmented control rather than
|
||||
expanding the page.
|
||||
|
||||
The focused structural test guards these contracts, optional-module boundaries,
|
||||
confirmation, contextual help, advanced-only JSON and responsive rules. Core's
|
||||
TypeScript build, structural localization audit, module-permutation suite and
|
||||
full-product bundle check provide the integration gates.
|
||||
|
||||
+8
-8
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.1",
|
||||
"version": "0.1.24",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "webui/src/index.ts",
|
||||
@@ -19,14 +19,14 @@
|
||||
"LICENSE"
|
||||
],
|
||||
"peerDependencies": {
|
||||
"@govoplan/core-webui": "^0.1.1",
|
||||
"lucide-react": "^0.555.0",
|
||||
"react": "^19.0.0",
|
||||
"react-dom": "^19.0.0",
|
||||
"react-router-dom": "^7.1.1",
|
||||
"@vitejs/plugin-react": "^4.3.4",
|
||||
"@vitejs/plugin-react": "^5.2.0",
|
||||
"vite": "^7.3.6",
|
||||
"typescript": "^5.7.2",
|
||||
"vite": "^6.0.6"
|
||||
"react": ">=19.2.7 <20",
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.18"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
+13
-2
@@ -4,14 +4,25 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-files"
|
||||
version = "0.1.3"
|
||||
version = "0.1.24"
|
||||
description = "GovOPlaN files module with backend and WebUI integration."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = { file = "LICENSE" }
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.3",
|
||||
"govoplan-core>=0.1.42",
|
||||
"defusedxml>=0.7,<1",
|
||||
"pyzipper>=0.3.6,<1",
|
||||
"python-multipart>=0.0.31,<1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
s3 = [
|
||||
"boto3>=1.34,<2",
|
||||
]
|
||||
smb = [
|
||||
"smbprotocol>=1.13",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
|
||||
@@ -1,17 +1,68 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.auth import ApiPrincipal, has_scope
|
||||
from govoplan_core.core.access import AccessDecisionProvenance, PrincipalRef
|
||||
from govoplan_core.core.files import FileAccessProvider
|
||||
from govoplan_core.core.files import (
|
||||
ManagedTabularFile,
|
||||
ManagedTabularFileAccessError,
|
||||
ManagedTabularFileContent,
|
||||
ManagedTabularFileNotFoundError,
|
||||
ManagedTabularFileProvider,
|
||||
ManagedTabularFileUnavailableError,
|
||||
ManagedTabularFileValidationError,
|
||||
PostboxFileReferenceRef,
|
||||
PostboxFileReferenceRequest,
|
||||
PostboxFileReferenceProvider,
|
||||
ManagedArtifactRef,
|
||||
ManagedArtifactStore,
|
||||
ManagedArtifactWriteRequest,
|
||||
)
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_core.security.module_permissions import scopes_grant_compatible
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileFolder, FileShare, FileVersion
|
||||
from govoplan_files.backend.runtime import configure_runtime
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.campaign_attachments import (
|
||||
annotate_built_messages_with_managed_files,
|
||||
managed_match_payloads,
|
||||
prepared_campaign_snapshot,
|
||||
public_attachment_summary_payload,
|
||||
share_assets_with_campaign,
|
||||
)
|
||||
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
list_recent_assets_for_user,
|
||||
read_asset_version_bytes,
|
||||
sync_file_asset_from_source,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
VIRTUAL_FOLDER_RESOURCE_PREFIX = "virtual-folder:v1"
|
||||
|
||||
WRITE_ACTIONS = {
|
||||
"files:file:upload",
|
||||
"files:file:organize",
|
||||
"files:file:share",
|
||||
"files:file:delete",
|
||||
"files:upload",
|
||||
"files:organize",
|
||||
"files:share",
|
||||
"files:delete",
|
||||
}
|
||||
|
||||
|
||||
class FilesCampaignCapability:
|
||||
@@ -21,6 +72,7 @@ class FilesCampaignCapability:
|
||||
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
|
||||
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
|
||||
current_version_and_blob = staticmethod(current_version_and_blob)
|
||||
share_assets_with_campaign = staticmethod(share_assets_with_campaign)
|
||||
|
||||
@staticmethod
|
||||
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:
|
||||
@@ -30,5 +82,679 @@ class FilesCampaignCapability:
|
||||
|
||||
|
||||
def campaign_capability(context: ModuleContext) -> FilesCampaignCapability:
|
||||
configure_runtime(settings=context.settings)
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesCampaignCapability()
|
||||
|
||||
|
||||
class FilesArtifactStore(ManagedArtifactStore):
|
||||
def store_artifact(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: ManagedArtifactWriteRequest,
|
||||
) -> ManagedArtifactRef:
|
||||
if not hasattr(session, "query") or not hasattr(session, "flush"):
|
||||
raise TypeError("Files artifact storage requires a SQLAlchemy session.")
|
||||
if not hasattr(principal, "has") or not principal.has("files:file:upload"):
|
||||
raise PermissionError("Managed artifact storage requires files:file:upload.")
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(getattr(user, "id", "") or "")
|
||||
tenant_id = str(getattr(principal, "tenant_id", "") or "")
|
||||
if not user_id or not tenant_id:
|
||||
raise PermissionError("Managed artifact storage requires a tenant user principal.")
|
||||
metadata = dict(request.metadata)
|
||||
if request.idempotency_key:
|
||||
metadata["source_provenance"] = {
|
||||
"source_type": "generated_artifact",
|
||||
"connector_id": "files.artifact_store",
|
||||
"provider": str(metadata.get("producer_module") or "platform"),
|
||||
"external_id": request.idempotency_key,
|
||||
"revision": str(metadata.get("output_sha256") or "") or None,
|
||||
}
|
||||
stored, _action, _previous_version_id = sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=user_id,
|
||||
user_id=user_id,
|
||||
filename=request.filename,
|
||||
data=request.payload,
|
||||
metadata=metadata,
|
||||
folder=request.folder,
|
||||
content_type=request.content_type,
|
||||
conflict_strategy="rename",
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
else:
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=user_id,
|
||||
user_id=user_id,
|
||||
filename=request.filename,
|
||||
data=request.payload,
|
||||
folder=request.folder,
|
||||
content_type=request.content_type,
|
||||
description=request.description,
|
||||
metadata=metadata,
|
||||
conflict_strategy="rename",
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
return ManagedArtifactRef(
|
||||
file_asset_id=stored.asset.id,
|
||||
file_version_id=stored.version.id,
|
||||
filename=stored.version.filename_at_upload,
|
||||
display_path=stored.asset.display_path,
|
||||
content_type=stored.version.content_type or request.content_type,
|
||||
size_bytes=stored.version.size_bytes,
|
||||
sha256=stored.version.checksum_sha256,
|
||||
provenance={
|
||||
"module": "files",
|
||||
"owner_type": stored.asset.owner_type,
|
||||
"managed": True,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def artifact_store_capability(context: ModuleContext) -> FilesArtifactStore:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesArtifactStore()
|
||||
|
||||
|
||||
class FilesPostboxReferenceService(PostboxFileReferenceProvider):
|
||||
"""Resolve Postbox evidence without turning message access into file access."""
|
||||
|
||||
def resolve_postbox_references(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
requests: tuple[PostboxFileReferenceRequest, ...],
|
||||
) -> tuple[PostboxFileReferenceRef, ...]:
|
||||
if not hasattr(session, "get"):
|
||||
raise TypeError("Postbox file resolution requires a SQLAlchemy session.")
|
||||
principal_tenant = str(getattr(principal, "tenant_id", "") or "")
|
||||
can_download = bool(
|
||||
hasattr(principal, "has") and principal.has("files:file:download")
|
||||
)
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(getattr(user, "id", "") or "")
|
||||
is_admin = bool(
|
||||
hasattr(principal, "has") and principal.has("files:file:admin")
|
||||
)
|
||||
results: list[PostboxFileReferenceRef] = []
|
||||
for request in requests:
|
||||
if principal_tenant != tenant_id:
|
||||
results.append(_unavailable_postbox_reference(request, "tenant_mismatch"))
|
||||
continue
|
||||
if not can_download or not user_id:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(
|
||||
request,
|
||||
"download_permission_missing",
|
||||
)
|
||||
)
|
||||
continue
|
||||
reference_type = request.reference_type.strip().casefold()
|
||||
version: FileVersion | None = None
|
||||
if reference_type in {"file", "file_asset", "files:file"}:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "exact_version_required")
|
||||
)
|
||||
continue
|
||||
elif reference_type in {"file_version", "files:file_version"}:
|
||||
version = session.get(FileVersion, request.reference_id) # type: ignore[attr-defined]
|
||||
asset = (
|
||||
session.get(FileAsset, version.file_asset_id) # type: ignore[attr-defined]
|
||||
if version is not None
|
||||
else None
|
||||
)
|
||||
else:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(
|
||||
request,
|
||||
"unsupported_reference_type",
|
||||
)
|
||||
)
|
||||
continue
|
||||
if (
|
||||
asset is None
|
||||
or version is None
|
||||
or asset.tenant_id != tenant_id
|
||||
or version.tenant_id != tenant_id
|
||||
or asset.deleted_at is not None
|
||||
):
|
||||
results.append(_unavailable_postbox_reference(request, "file_not_found"))
|
||||
continue
|
||||
try:
|
||||
get_asset_for_user(
|
||||
session, # type: ignore[arg-type]
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=asset.id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
except FileStorageError:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "file_access_denied")
|
||||
)
|
||||
continue
|
||||
blob = session.get(FileBlob, version.blob_id) # type: ignore[attr-defined]
|
||||
if blob is None or blob.tenant_id != tenant_id:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "file_payload_missing")
|
||||
)
|
||||
continue
|
||||
results.append(
|
||||
PostboxFileReferenceRef(
|
||||
reference_type=request.reference_type,
|
||||
reference_id=request.reference_id,
|
||||
available=True,
|
||||
reason_code="available",
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=version.id,
|
||||
filename=version.filename_at_upload or asset.filename,
|
||||
content_type=version.content_type or blob.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
sha256=version.checksum_sha256,
|
||||
download_path=(
|
||||
f"/api/v1/files/{asset.id}/versions/{version.id}/download"
|
||||
),
|
||||
provenance={
|
||||
"module": "files",
|
||||
"postbox_id": request.postbox_id,
|
||||
"message_id": request.message_id,
|
||||
"display_path": asset.display_path,
|
||||
"exact_version": True,
|
||||
},
|
||||
)
|
||||
)
|
||||
return tuple(results)
|
||||
|
||||
|
||||
def _unavailable_postbox_reference(
|
||||
request: PostboxFileReferenceRequest,
|
||||
reason_code: str,
|
||||
) -> PostboxFileReferenceRef:
|
||||
return PostboxFileReferenceRef(
|
||||
reference_type=request.reference_type,
|
||||
reference_id=request.reference_id,
|
||||
available=False,
|
||||
reason_code=reason_code,
|
||||
)
|
||||
|
||||
|
||||
def postbox_reference_capability(
|
||||
context: ModuleContext,
|
||||
) -> FilesPostboxReferenceService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesPostboxReferenceService()
|
||||
|
||||
|
||||
class FilesAccessService(FileAccessProvider):
|
||||
def explain_resource_provenance(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
action: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
normalized_type = resource_type.lower().strip()
|
||||
if normalized_type in {"file", "file_asset", "files:file"}:
|
||||
return self._explain_file(session, principal, resource_id=resource_id, action=action)
|
||||
if normalized_type in {"folder", "file_folder", "files:folder"}:
|
||||
return self._explain_folder(session, principal, resource_id=resource_id, action=action)
|
||||
return ()
|
||||
|
||||
def _explain_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_id: str,
|
||||
action: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
asset = session.get(FileAsset, resource_id) # type: ignore[attr-defined]
|
||||
if asset is None or (principal.tenant_id and asset.tenant_id != principal.tenant_id):
|
||||
return (_missing_resource("file", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
items = [_file_resource(asset)]
|
||||
items.extend(_owner_provenance(asset.owner_type, _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id), principal, source="files.owner"))
|
||||
items.extend(_admin_provenance(principal, "files:file:admin", source="files.admin_scope"))
|
||||
permission_values = {"write", "manage"} if _requires_write_share(action) else {"read", "write", "manage"}
|
||||
shares = (
|
||||
session.query(FileShare) # type: ignore[attr-defined]
|
||||
.filter(
|
||||
FileShare.tenant_id == asset.tenant_id,
|
||||
FileShare.file_asset_id == asset.id,
|
||||
effective_file_share_clause(),
|
||||
FileShare.permission.in_(sorted(permission_values)),
|
||||
or_(
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == principal.membership_id),
|
||||
(FileShare.target_type == "group") & (FileShare.target_id.in_(sorted(principal.group_ids))),
|
||||
(FileShare.target_type == "tenant") & (FileShare.target_id == asset.tenant_id),
|
||||
),
|
||||
)
|
||||
.order_by(FileShare.target_type.asc(), FileShare.target_id.asc())
|
||||
.all()
|
||||
)
|
||||
for share in shares:
|
||||
items.append(_share_provenance(share, source="files.share"))
|
||||
return tuple(items)
|
||||
|
||||
def _explain_folder(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_id: str,
|
||||
action: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
del action
|
||||
folder = session.get(FileFolder, resource_id) # type: ignore[attr-defined]
|
||||
if folder is None:
|
||||
return self._explain_virtual_folder(session, principal, resource_id=resource_id)
|
||||
if principal.tenant_id and folder.tenant_id != principal.tenant_id:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
return tuple([
|
||||
AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=folder.id,
|
||||
label=folder.path,
|
||||
tenant_id=folder.tenant_id,
|
||||
source="files.folder",
|
||||
details={
|
||||
"resource_type": "folder",
|
||||
"path": folder.path,
|
||||
"owner_type": folder.owner_type,
|
||||
"deleted": folder.deleted_at is not None,
|
||||
},
|
||||
),
|
||||
*_owner_provenance(folder.owner_type, _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id), principal, source="files.owner"),
|
||||
*_admin_provenance(principal, "files:file:admin", source="files.admin_scope"),
|
||||
])
|
||||
|
||||
def _explain_virtual_folder(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_id: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
folder_ref = parse_virtual_folder_resource_id(resource_id)
|
||||
if folder_ref is None:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
tenant_id, owner_type, owner_id, path = folder_ref
|
||||
if principal.tenant_id and tenant_id != principal.tenant_id:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
child_prefix = f"{path}/"
|
||||
owner_filter = FileAsset.owner_user_id == owner_id if owner_type == "user" else FileAsset.owner_group_id == owner_id
|
||||
child = (
|
||||
session.query(FileAsset.id) # type: ignore[attr-defined]
|
||||
.filter(
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
FileAsset.owner_type == owner_type,
|
||||
owner_filter,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
FileAsset.display_path.like(f"{child_prefix}%"),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if child is None:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
return tuple([
|
||||
AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=resource_id,
|
||||
label=path,
|
||||
tenant_id=tenant_id,
|
||||
source="files.virtual_folder",
|
||||
details={
|
||||
"resource_type": "folder",
|
||||
"path": path,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"virtual": True,
|
||||
"deleted": False,
|
||||
},
|
||||
),
|
||||
*_owner_provenance(owner_type, owner_id, principal, source="files.owner"),
|
||||
*_admin_provenance(principal, "files:file:admin", source="files.admin_scope"),
|
||||
])
|
||||
|
||||
|
||||
def access_capability(context: ModuleContext) -> FilesAccessService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesAccessService()
|
||||
|
||||
|
||||
class FilesManagedTabularFileService(ManagedTabularFileProvider):
|
||||
"""Expose authorized immutable CSV/XLSX versions to optional consumers."""
|
||||
|
||||
def list_tabular_files(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
query: str = "",
|
||||
limit: int = 100,
|
||||
) -> tuple[ManagedTabularFile, ...]:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:read",
|
||||
)
|
||||
normalized_query = str(query or "").strip().casefold()
|
||||
requested_limit = max(1, min(int(limit), 100))
|
||||
assets = list_recent_assets_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
limit=min(500, requested_limit * 5),
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
results: list[ManagedTabularFile] = []
|
||||
for asset in assets:
|
||||
if normalized_query and normalized_query not in (
|
||||
f"{asset.filename} {asset.display_path} {asset.description or ''}"
|
||||
).casefold():
|
||||
continue
|
||||
try:
|
||||
version, blob = current_version_and_blob(db, asset)
|
||||
except FileStorageError:
|
||||
continue
|
||||
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||
continue
|
||||
results.append(_managed_tabular_file(asset, version, blob))
|
||||
if len(results) >= requested_limit:
|
||||
break
|
||||
return tuple(results)
|
||||
|
||||
def get_tabular_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str | None = None,
|
||||
) -> ManagedTabularFile | None:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:read",
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_asset_id,
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
except FileStorageError:
|
||||
return None
|
||||
if file_version_id:
|
||||
version = db.get(FileVersion, file_version_id)
|
||||
if (
|
||||
version is None
|
||||
or version.file_asset_id != asset.id
|
||||
or version.tenant_id != api_principal.tenant_id
|
||||
):
|
||||
return None
|
||||
blob = db.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != api_principal.tenant_id:
|
||||
return None
|
||||
else:
|
||||
try:
|
||||
version, blob = current_version_and_blob(db, asset)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileUnavailableError(
|
||||
"Managed file version metadata is unavailable."
|
||||
) from exc
|
||||
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||
return None
|
||||
return _managed_tabular_file(asset, version, blob)
|
||||
|
||||
def read_tabular_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str,
|
||||
max_bytes: int,
|
||||
) -> ManagedTabularFileContent:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:download",
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_asset_id,
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileNotFoundError(
|
||||
"Managed tabular file not found."
|
||||
) from exc
|
||||
metadata = self.get_tabular_file(
|
||||
db,
|
||||
api_principal,
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=file_version_id,
|
||||
)
|
||||
if metadata is None:
|
||||
raise ManagedTabularFileNotFoundError(
|
||||
"Managed tabular file version not found."
|
||||
)
|
||||
effective_max = max(1, int(max_bytes))
|
||||
if metadata.size_bytes > effective_max:
|
||||
raise ManagedTabularFileValidationError(
|
||||
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||
)
|
||||
try:
|
||||
payload, version, blob = read_asset_version_bytes(
|
||||
db,
|
||||
asset,
|
||||
file_version_id,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileUnavailableError(
|
||||
"Managed tabular file content is unavailable or failed integrity verification."
|
||||
) from exc
|
||||
if len(payload) > effective_max:
|
||||
raise ManagedTabularFileValidationError(
|
||||
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||
)
|
||||
result = _managed_tabular_file(asset, version, blob)
|
||||
audit_from_principal(
|
||||
db,
|
||||
api_principal,
|
||||
action="files.tabular_content.read",
|
||||
object_type="file_version",
|
||||
object_id=version.id,
|
||||
details={
|
||||
"file_asset_id": asset.id,
|
||||
"size_bytes": version.size_bytes,
|
||||
"checksum_sha256": version.checksum_sha256,
|
||||
"consumer": "tabular_content",
|
||||
},
|
||||
)
|
||||
return ManagedTabularFileContent(file=result, payload=payload)
|
||||
|
||||
|
||||
def managed_tabular_file_capability(
|
||||
context: ModuleContext,
|
||||
) -> FilesManagedTabularFileService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesManagedTabularFileService()
|
||||
|
||||
|
||||
def _tabular_context(
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
required_scope: str,
|
||||
) -> tuple[Session, ApiPrincipal, str]:
|
||||
if not isinstance(session, Session):
|
||||
raise TypeError("Managed tabular file access requires a SQLAlchemy session.")
|
||||
if not isinstance(principal, ApiPrincipal):
|
||||
raise ManagedTabularFileAccessError(
|
||||
"Managed tabular file access requires a tenant API principal."
|
||||
)
|
||||
if not (
|
||||
has_scope(principal, required_scope)
|
||||
or has_scope(principal, "files:file:admin")
|
||||
):
|
||||
raise ManagedTabularFileAccessError(
|
||||
f"Managed tabular file access requires {required_scope}."
|
||||
)
|
||||
user_id = str(getattr(principal.user, "id", "") or "").strip()
|
||||
if not principal.tenant_id or not user_id:
|
||||
raise ManagedTabularFileAccessError(
|
||||
"Managed tabular file access requires a tenant user principal."
|
||||
)
|
||||
return session, principal, user_id
|
||||
|
||||
|
||||
def _managed_tabular_file(
|
||||
asset: FileAsset,
|
||||
version: FileVersion,
|
||||
blob: FileBlob,
|
||||
) -> ManagedTabularFile:
|
||||
return ManagedTabularFile(
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=version.id,
|
||||
filename=version.filename_at_upload or asset.filename,
|
||||
display_path=version.display_path_at_upload or asset.display_path,
|
||||
content_type=version.content_type or blob.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
sha256=version.checksum_sha256,
|
||||
updated_at=version.created_at,
|
||||
current_version=asset.current_version_id == version.id,
|
||||
)
|
||||
|
||||
|
||||
def _is_tabular_file(filename: str, content_type: str | None) -> bool:
|
||||
normalized_name = str(filename or "").strip().casefold()
|
||||
normalized_type = str(content_type or "").split(";", 1)[0].strip().casefold()
|
||||
return normalized_name.endswith((".csv", ".xlsx")) or normalized_type in {
|
||||
"text/csv",
|
||||
"application/csv",
|
||||
"application/vnd.ms-excel",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
}
|
||||
|
||||
|
||||
def virtual_folder_resource_id(*, tenant_id: str, owner_type: str, owner_id: str, path: str) -> str:
|
||||
normalized_path = normalize_folder(path)
|
||||
encoded_path = base64.urlsafe_b64encode(normalized_path.encode("utf-8")).decode("ascii").rstrip("=")
|
||||
return f"{VIRTUAL_FOLDER_RESOURCE_PREFIX}:{tenant_id}:{owner_type}:{owner_id}:{encoded_path}"
|
||||
|
||||
|
||||
def parse_virtual_folder_resource_id(resource_id: str) -> tuple[str, str, str, str] | None:
|
||||
parts = resource_id.split(":", 5)
|
||||
if len(parts) != 6 or f"{parts[0]}:{parts[1]}" != VIRTUAL_FOLDER_RESOURCE_PREFIX:
|
||||
return None
|
||||
_, _, tenant_id, owner_type, owner_id, encoded_path = parts
|
||||
if owner_type not in {"user", "group"} or not tenant_id or not owner_id or not encoded_path:
|
||||
return None
|
||||
padding = "=" * (-len(encoded_path) % 4)
|
||||
try:
|
||||
decoded_path = base64.urlsafe_b64decode(f"{encoded_path}{padding}").decode("utf-8")
|
||||
path = normalize_folder(decoded_path)
|
||||
except (binascii.Error, UnicodeDecodeError, ValueError):
|
||||
return None
|
||||
if not path:
|
||||
return None
|
||||
return tenant_id, owner_type, owner_id, path
|
||||
|
||||
|
||||
def _file_resource(asset: FileAsset) -> AccessDecisionProvenance:
|
||||
return AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=asset.id,
|
||||
label=asset.filename,
|
||||
tenant_id=asset.tenant_id,
|
||||
source="files.file",
|
||||
details={
|
||||
"resource_type": "file",
|
||||
"display_path": asset.display_path,
|
||||
"owner_type": asset.owner_type,
|
||||
"deleted": asset.deleted_at is not None,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _missing_resource(resource_type: str, resource_id: str, tenant_id: str | None, *, source: str) -> AccessDecisionProvenance:
|
||||
return AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=resource_id,
|
||||
tenant_id=tenant_id,
|
||||
source=source,
|
||||
details={"resource_type": resource_type, "found": False},
|
||||
)
|
||||
|
||||
|
||||
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||
return owner_user_id if owner_type == "user" else owner_group_id
|
||||
|
||||
|
||||
def _owner_provenance(owner_type: str, owner_id: str | None, principal: PrincipalRef, *, source: str) -> tuple[AccessDecisionProvenance, ...]:
|
||||
if owner_id is None:
|
||||
return ()
|
||||
matches_user = owner_type == "user" and owner_id == principal.membership_id
|
||||
matches_group = owner_type == "group" and owner_id in principal.group_ids
|
||||
if not (matches_user or matches_group):
|
||||
return ()
|
||||
return (
|
||||
AccessDecisionProvenance(
|
||||
kind="owner",
|
||||
id=owner_id,
|
||||
tenant_id=principal.tenant_id,
|
||||
source=source,
|
||||
details={"owner_type": owner_type},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _admin_provenance(principal: PrincipalRef, required_scope: str, *, source: str) -> tuple[AccessDecisionProvenance, ...]:
|
||||
if not scopes_grant_compatible(principal.scopes, required_scope):
|
||||
return ()
|
||||
return (
|
||||
AccessDecisionProvenance(
|
||||
kind="policy",
|
||||
id=required_scope,
|
||||
label=required_scope,
|
||||
tenant_id=principal.tenant_id,
|
||||
source=source,
|
||||
details={"grant": "tenant_admin"},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _share_provenance(share: FileShare, *, source: str) -> AccessDecisionProvenance:
|
||||
return AccessDecisionProvenance(
|
||||
kind="share",
|
||||
id=share.id,
|
||||
label=share.permission,
|
||||
tenant_id=share.tenant_id,
|
||||
source=source,
|
||||
details={
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _requires_write_share(action: str) -> bool:
|
||||
return action in WRITE_ACTIONS
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import event
|
||||
from sqlalchemy.orm import Session as OrmSession
|
||||
|
||||
from govoplan_core.core.change_sequence import record_change
|
||||
from govoplan_core.core.sqlalchemy_change_tracking import (
|
||||
ensure_object_id,
|
||||
has_attr_changes,
|
||||
object_state,
|
||||
operation_for_soft_deletable,
|
||||
previous_value,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid
|
||||
|
||||
FILES_MODULE_ID = "files"
|
||||
FILES_ASSETS_COLLECTION = "files.assets"
|
||||
FILES_FOLDERS_COLLECTION = "files.folders"
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION = "files.connector_profiles"
|
||||
FILES_CONNECTOR_CREDENTIALS_COLLECTION = "files.connector_credentials"
|
||||
FILES_CONNECTOR_POLICIES_COLLECTION = "files.connector_policies"
|
||||
FILES_CONNECTOR_SPACES_COLLECTION = "files.connector_spaces"
|
||||
|
||||
_REGISTERED = False
|
||||
|
||||
|
||||
def register_files_change_tracking() -> None:
|
||||
global _REGISTERED
|
||||
if _REGISTERED:
|
||||
return
|
||||
event.listen(OrmSession, "before_flush", _record_files_changes)
|
||||
_REGISTERED = True
|
||||
|
||||
|
||||
def _record_files_changes(session: OrmSession, _flush_context: object, _instances: object) -> None:
|
||||
for obj in tuple(session.new) + tuple(session.dirty):
|
||||
if isinstance(obj, FileAsset):
|
||||
_record_asset_change(session, obj)
|
||||
elif isinstance(obj, FileFolder):
|
||||
_record_folder_change(session, obj)
|
||||
elif isinstance(obj, FileShare):
|
||||
_record_share_visibility_change(session, obj)
|
||||
|
||||
|
||||
def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
operation = operation_for_soft_deletable(
|
||||
asset,
|
||||
changed_attrs=(
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
"current_version_id",
|
||||
"display_path",
|
||||
"filename",
|
||||
"description",
|
||||
"deleted_at",
|
||||
"retained_until",
|
||||
"legal_hold",
|
||||
"lifecycle_revision",
|
||||
"lifecycle_reason",
|
||||
"metadata_",
|
||||
),
|
||||
)
|
||||
if operation is None:
|
||||
return
|
||||
resource_id = _ensure_id(asset)
|
||||
record_change(
|
||||
session,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collection=FILES_ASSETS_COLLECTION,
|
||||
resource_type="file",
|
||||
resource_id=resource_id,
|
||||
operation=operation,
|
||||
tenant_id=asset.tenant_id,
|
||||
actor_type="user" if asset.created_by_user_id else None,
|
||||
actor_id=asset.created_by_user_id,
|
||||
payload={
|
||||
"owner_type": asset.owner_type,
|
||||
"owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id),
|
||||
"path": asset.display_path,
|
||||
"previous_path": previous_value(asset, "display_path"),
|
||||
"filename": asset.filename,
|
||||
"deleted_at": _isoformat(asset.deleted_at),
|
||||
"retained_until": _isoformat(asset.retained_until),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
|
||||
operation = operation_for_soft_deletable(
|
||||
folder,
|
||||
changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"),
|
||||
)
|
||||
if operation is None:
|
||||
return
|
||||
resource_id = _ensure_id(folder)
|
||||
record_change(
|
||||
session,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collection=FILES_FOLDERS_COLLECTION,
|
||||
resource_type="folder",
|
||||
resource_id=resource_id,
|
||||
operation=operation,
|
||||
tenant_id=folder.tenant_id,
|
||||
actor_type="user" if folder.created_by_user_id else None,
|
||||
actor_id=folder.created_by_user_id,
|
||||
payload={
|
||||
"owner_type": folder.owner_type,
|
||||
"owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id),
|
||||
"path": folder.path,
|
||||
"previous_path": previous_value(folder, "path"),
|
||||
"deleted_at": _isoformat(folder.deleted_at),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None:
|
||||
state = object_state(share)
|
||||
if not share.file_asset_id:
|
||||
return
|
||||
if not state.pending and not has_attr_changes(
|
||||
state,
|
||||
(
|
||||
"file_asset_id",
|
||||
"target_type",
|
||||
"target_id",
|
||||
"permission",
|
||||
"expires_at",
|
||||
"revoked_at",
|
||||
"revoked_by_user_id",
|
||||
),
|
||||
):
|
||||
return
|
||||
_ensure_id(share)
|
||||
record_change(
|
||||
session,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collection=FILES_ASSETS_COLLECTION,
|
||||
resource_type="file",
|
||||
resource_id=share.file_asset_id,
|
||||
operation="updated",
|
||||
tenant_id=share.tenant_id,
|
||||
actor_type="user" if share.created_by_user_id else None,
|
||||
actor_id=share.created_by_user_id,
|
||||
payload={
|
||||
"share_id": share.id,
|
||||
"share_target_type": share.target_type,
|
||||
"share_target_id": share.target_id,
|
||||
"share_permission": share.permission,
|
||||
"share_expires_at": _isoformat(share.expires_at),
|
||||
"share_revoked_at": _isoformat(share.revoked_at),
|
||||
"share_revoked_by_user_id": share.revoked_by_user_id,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _ensure_id(obj: object) -> str:
|
||||
return ensure_object_id(obj, new_uuid)
|
||||
|
||||
|
||||
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||
if owner_type == "user":
|
||||
return owner_user_id
|
||||
if owner_type == "group":
|
||||
return owner_group_id
|
||||
return None
|
||||
|
||||
|
||||
def _isoformat(value: datetime | None) -> str | None:
|
||||
return value.isoformat() if value else None
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FILES_ASSETS_COLLECTION",
|
||||
"FILES_CONNECTOR_CREDENTIALS_COLLECTION",
|
||||
"FILES_CONNECTOR_POLICIES_COLLECTION",
|
||||
"FILES_CONNECTOR_PROFILES_COLLECTION",
|
||||
"FILES_CONNECTOR_SPACES_COLLECTION",
|
||||
"FILES_FOLDERS_COLLECTION",
|
||||
"FILES_MODULE_ID",
|
||||
"register_files_change_tracking",
|
||||
]
|
||||
@@ -0,0 +1,668 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from govoplan_core.core.configuration_packages import (
|
||||
ConfigurationApplyResult,
|
||||
ConfigurationDiagnostic,
|
||||
ConfigurationExportResult,
|
||||
ConfigurationExportSelection,
|
||||
ConfigurationPackageFragment,
|
||||
ConfigurationPlanItem,
|
||||
ConfigurationPreflightContext,
|
||||
ConfigurationPreflightResult,
|
||||
ConfigurationProvider,
|
||||
ConfigurationProviderDescription,
|
||||
)
|
||||
from govoplan_core.core.infrastructure_capabilities import (
|
||||
InfrastructureCapability,
|
||||
InfrastructureCapabilityReceipt,
|
||||
InfrastructureDependency,
|
||||
InfrastructureDependencyProvider,
|
||||
)
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import FileBlob
|
||||
from govoplan_files.backend.runtime import settings as runtime_settings
|
||||
|
||||
|
||||
FILES_CONFIGURATION_CAPABILITY = "files.configuration"
|
||||
FILES_INFRASTRUCTURE_DEPENDENCY_CAPABILITY = (
|
||||
"infrastructure.dependency_inventory.files"
|
||||
)
|
||||
MANAGED_STORAGE_FRAGMENT = "managed_storage"
|
||||
_PAYLOAD_KEYS = frozenset(
|
||||
{"capability_id", "expected_backend", "expected_source"}
|
||||
)
|
||||
|
||||
|
||||
class FilesConfigurationProvider(
|
||||
ConfigurationProvider,
|
||||
InfrastructureDependencyProvider,
|
||||
):
|
||||
module_id = "files"
|
||||
capability_ids = ("files.storage",)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
settings: object | None = None,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> None:
|
||||
self._settings = runtime_settings if settings is None else settings
|
||||
self._environment = os.environ if environment is None else environment
|
||||
|
||||
def describe(self) -> ConfigurationProviderDescription:
|
||||
return ConfigurationProviderDescription(
|
||||
module_id=self.module_id,
|
||||
fragment_types=(MANAGED_STORAGE_FRAGMENT,),
|
||||
schema_refs={
|
||||
MANAGED_STORAGE_FRAGMENT: "govoplan/files/configuration/managed-storage.v1"
|
||||
},
|
||||
exported_scopes=("system",),
|
||||
)
|
||||
|
||||
def preflight(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationPreflightResult:
|
||||
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||
return ConfigurationPreflightResult(
|
||||
diagnostics=(_unsupported(fragment),),
|
||||
plan=(_blocked_plan(fragment, "Fragment type is unsupported."),),
|
||||
)
|
||||
diagnostics, binding_ref = self._binding_diagnostics(fragment, context)
|
||||
blocked = any(item.severity == "blocker" for item in diagnostics)
|
||||
return ConfigurationPreflightResult(
|
||||
diagnostics=tuple(diagnostics),
|
||||
plan=(
|
||||
ConfigurationPlanItem(
|
||||
action="blocked" if blocked else "skip",
|
||||
module_id="files",
|
||||
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||
fragment_id=fragment.fragment_id or binding_ref,
|
||||
summary=(
|
||||
"Managed storage does not match the deployment receipt."
|
||||
if blocked
|
||||
else "Deployment-owned managed storage already matches the receipt; no module state is rewritten."
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
def apply(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
supplied_data: Mapping[str, Any],
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationApplyResult:
|
||||
del supplied_data
|
||||
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||
return ConfigurationApplyResult(diagnostics=(_unsupported(fragment),))
|
||||
diagnostics, _binding_ref = self._binding_diagnostics(fragment, context)
|
||||
return ConfigurationApplyResult(
|
||||
diagnostics=tuple(
|
||||
item for item in diagnostics if item.severity == "blocker"
|
||||
)
|
||||
)
|
||||
|
||||
def export(
|
||||
self,
|
||||
selection: ConfigurationExportSelection,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationExportResult:
|
||||
del selection
|
||||
receipt = context.infrastructure_receipt
|
||||
if context.infrastructure_receipt_error:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
_receipt_error(
|
||||
context.infrastructure_receipt_error,
|
||||
object_ref="files.storage",
|
||||
),
|
||||
)
|
||||
)
|
||||
if receipt is None:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_missing",
|
||||
message="Files managed-storage export requires the deployment capability receipt.",
|
||||
module_id="files",
|
||||
object_ref="files.storage",
|
||||
resolution="Mount the installer-generated receipt before exporting deployment configuration.",
|
||||
),
|
||||
)
|
||||
)
|
||||
capability = receipt.capability("files.storage")
|
||||
if capability is None:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_missing",
|
||||
message="The deployment receipt does not declare managed file storage.",
|
||||
module_id="files",
|
||||
object_ref="files.storage",
|
||||
),
|
||||
)
|
||||
)
|
||||
return ConfigurationExportResult(
|
||||
fragments=(
|
||||
ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||
fragment_id=f"{receipt.installation_id}:files.storage",
|
||||
payload={
|
||||
"capability_id": "files.storage",
|
||||
"expected_backend": _expected_backend(capability.source),
|
||||
"expected_source": capability.source,
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def health(
|
||||
self,
|
||||
import_result: ConfigurationApplyResult,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> tuple[ConfigurationDiagnostic, ...]:
|
||||
del context
|
||||
return tuple(
|
||||
item for item in import_result.diagnostics if item.severity == "blocker"
|
||||
)
|
||||
|
||||
def infrastructure_dependencies(self) -> tuple[InfrastructureDependency, ...]:
|
||||
active_backend = _normalized_backend(
|
||||
getattr(self._settings, "file_storage_backend", "local")
|
||||
)
|
||||
dependencies = [
|
||||
InfrastructureDependency(
|
||||
capability_id="files.storage",
|
||||
module_id="files",
|
||||
dependency_type="runtime_storage_binding",
|
||||
dependency_ref=f"files-storage:{active_backend}",
|
||||
state="runtime_binding",
|
||||
scope="system",
|
||||
summary=(
|
||||
"The active Files runtime is bound to this deployment storage backend."
|
||||
),
|
||||
metrics={},
|
||||
required_action=(
|
||||
"Provision and verify the replacement backend before rebinding the Files runtime."
|
||||
),
|
||||
)
|
||||
]
|
||||
with get_database().session() as session:
|
||||
rows = session.execute(
|
||||
select(
|
||||
FileBlob.storage_backend,
|
||||
func.count(FileBlob.id),
|
||||
func.coalesce(func.sum(FileBlob.size_bytes), 0),
|
||||
)
|
||||
.group_by(FileBlob.storage_backend)
|
||||
.order_by(FileBlob.storage_backend)
|
||||
)
|
||||
for backend, blob_count, size_bytes in rows:
|
||||
normalized_backend = _normalized_backend(str(backend or "local"))
|
||||
dependencies.append(
|
||||
InfrastructureDependency(
|
||||
capability_id="files.storage",
|
||||
module_id="files",
|
||||
dependency_type="stored_blob_set",
|
||||
dependency_ref=f"file-blobs:{normalized_backend}",
|
||||
state="data_present",
|
||||
scope="all-tenants",
|
||||
summary=(
|
||||
"Persisted Files blob metadata references content in this storage backend."
|
||||
),
|
||||
metrics={
|
||||
"blob_count": int(blob_count or 0),
|
||||
"content_bytes": int(size_bytes or 0),
|
||||
},
|
||||
required_action=(
|
||||
"Copy and checksum-verify every referenced blob, switch the runtime binding, and retain rollback evidence before removing or replacing storage."
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(dependencies)
|
||||
|
||||
def _binding_diagnostics(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> tuple[list[ConfigurationDiagnostic], str]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
payload = _files_fragment_payload(fragment, diagnostics)
|
||||
if payload is None:
|
||||
return diagnostics, fragment.fragment_id or "files.storage"
|
||||
receipt_state = _files_receipt_capability(
|
||||
fragment,
|
||||
context,
|
||||
payload,
|
||||
diagnostics,
|
||||
)
|
||||
if receipt_state is None:
|
||||
return diagnostics, fragment.fragment_id or "files.storage"
|
||||
_receipt, capability, binding_ref = receipt_state
|
||||
expected_backend = _expected_backend(capability.source)
|
||||
diagnostics.extend(
|
||||
_package_expectation_diagnostics(
|
||||
payload,
|
||||
capability,
|
||||
expected_backend,
|
||||
binding_ref,
|
||||
)
|
||||
)
|
||||
active_backend = _normalized_backend(
|
||||
getattr(self._settings, "file_storage_backend", "local")
|
||||
)
|
||||
if active_backend != expected_backend:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_runtime_mismatch",
|
||||
message=(
|
||||
f"Files runtime uses {active_backend!r}, while the deployment receipt declares {expected_backend!r}."
|
||||
),
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Reconcile deployment environment and receipt before starting or importing Files configuration.",
|
||||
)
|
||||
)
|
||||
return diagnostics, binding_ref
|
||||
diagnostics.extend(
|
||||
self._runtime_storage_diagnostics(
|
||||
expected_backend,
|
||||
capability,
|
||||
binding_ref,
|
||||
)
|
||||
)
|
||||
if "files" not in capability.dependent_modules:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="info",
|
||||
code="infrastructure_consumer_not_declared",
|
||||
message="Files is active but was not selected as a receipt consumer when the deployment plan was generated.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Regenerate the deployment plan so removal-impact inventory includes Files.",
|
||||
)
|
||||
)
|
||||
return diagnostics, binding_ref
|
||||
|
||||
def _runtime_storage_diagnostics(
|
||||
self,
|
||||
expected_backend: str,
|
||||
capability: InfrastructureCapability,
|
||||
binding_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
if expected_backend == "s3":
|
||||
return self._s3_diagnostics(
|
||||
capability.endpoint,
|
||||
capability.secret_refs,
|
||||
capability.source,
|
||||
binding_ref,
|
||||
)
|
||||
root = Path(
|
||||
str(getattr(self._settings, "file_storage_local_root", "") or "")
|
||||
)
|
||||
if root.is_absolute():
|
||||
return []
|
||||
return [
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_local_root_not_durable",
|
||||
message="Receipt-bound local file storage requires an absolute deployment-managed path.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Set FILE_STORAGE_LOCAL_ROOT to the mounted persistent-volume path.",
|
||||
)
|
||||
]
|
||||
|
||||
def _s3_diagnostics(
|
||||
self,
|
||||
endpoint: Mapping[str, object],
|
||||
secret_refs: tuple[str, ...],
|
||||
source: str,
|
||||
object_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
endpoint_url = str(
|
||||
getattr(self._settings, "file_storage_s3_endpoint_url", "")
|
||||
or getattr(self._settings, "s3_endpoint_url", "")
|
||||
or ""
|
||||
).strip()
|
||||
runtime_endpoint = _redacted_endpoint(
|
||||
endpoint_url,
|
||||
default_port=int(endpoint.get("port") or 443),
|
||||
)
|
||||
receipt_endpoint = {
|
||||
key: endpoint.get(key) for key in ("scheme", "host", "port") if endpoint.get(key) is not None
|
||||
}
|
||||
if receipt_endpoint and runtime_endpoint != receipt_endpoint:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_endpoint_mismatch",
|
||||
message="Files S3 endpoint does not match the sanitized deployment receipt endpoint.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Reconcile FILE_STORAGE_S3_ENDPOINT_URL and the deployment plan before apply.",
|
||||
)
|
||||
)
|
||||
bucket = str(
|
||||
getattr(self._settings, "file_storage_s3_bucket", "")
|
||||
or getattr(self._settings, "s3_bucket", "")
|
||||
or ""
|
||||
).strip()
|
||||
if not bucket:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_bucket_missing",
|
||||
message="Files S3 storage has no configured bucket.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Set FILE_STORAGE_S3_BUCKET in the deployment environment.",
|
||||
)
|
||||
)
|
||||
for reference in secret_refs:
|
||||
variable = reference.removeprefix("env:")
|
||||
if not variable.startswith("FILE_STORAGE_"):
|
||||
continue
|
||||
if not str(self._environment.get(variable, "")).strip():
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_secret_reference_unresolved",
|
||||
message=f"Required storage secret reference {reference} is not available to the runtime.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Provide the referenced environment secret without copying its value into the package.",
|
||||
)
|
||||
)
|
||||
deployment_managed = bool(
|
||||
getattr(
|
||||
self._settings,
|
||||
"file_storage_s3_deployment_managed",
|
||||
False,
|
||||
)
|
||||
)
|
||||
endpoint_trusted = bool(
|
||||
getattr(self._settings, "file_storage_s3_endpoint_trusted", False)
|
||||
)
|
||||
if source == "installer-managed-garage" and not deployment_managed:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_management_boundary_mismatch",
|
||||
message="Garage is installer-managed in the receipt but not marked deployment-managed in Files runtime.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Set FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true for the installer-managed Garage endpoint.",
|
||||
)
|
||||
)
|
||||
if source == "operator-supplied-s3" and not endpoint_trusted:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_trust_boundary_missing",
|
||||
message="External S3 storage must be explicitly marked as a trusted deployment endpoint.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Review the endpoint and set FILE_STORAGE_S3_ENDPOINT_TRUSTED=true in deployment configuration.",
|
||||
)
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
|
||||
def _files_fragment_payload(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
diagnostics: list[ConfigurationDiagnostic],
|
||||
) -> Mapping[str, Any] | None:
|
||||
payload = fragment.payload
|
||||
if not isinstance(payload, Mapping):
|
||||
diagnostics.append(
|
||||
_invalid(fragment, "Managed-storage payload must be an object.")
|
||||
)
|
||||
return None
|
||||
unknown = sorted(set(payload) - _PAYLOAD_KEYS)
|
||||
if not unknown:
|
||||
return payload
|
||||
secret_like = any(
|
||||
marker in key.casefold()
|
||||
for key in unknown
|
||||
for marker in ("password", "secret", "token", "credential", "access_key")
|
||||
)
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code=(
|
||||
"files_configuration_secret_forbidden"
|
||||
if secret_like
|
||||
else "files_configuration_payload_invalid"
|
||||
),
|
||||
message=(
|
||||
"Managed-storage fragments accept receipt references and non-secret expectations only."
|
||||
if secret_like
|
||||
else f"Managed-storage payload contains unsupported fields: {', '.join(unknown)}."
|
||||
),
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or MANAGED_STORAGE_FRAGMENT,
|
||||
resolution="Keep storage credentials in deployment environment references, never in a configuration package.",
|
||||
)
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def _files_receipt_capability(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
payload: Mapping[str, Any],
|
||||
diagnostics: list[ConfigurationDiagnostic],
|
||||
) -> tuple[
|
||||
InfrastructureCapabilityReceipt,
|
||||
InfrastructureCapability,
|
||||
str,
|
||||
] | None:
|
||||
if context.infrastructure_receipt_error:
|
||||
diagnostics.append(
|
||||
_receipt_error(
|
||||
context.infrastructure_receipt_error,
|
||||
object_ref=fragment.fragment_id or "files.storage",
|
||||
)
|
||||
)
|
||||
return None
|
||||
receipt = context.infrastructure_receipt
|
||||
if receipt is None:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_missing",
|
||||
message="Files managed-storage configuration requires the deployment capability receipt.",
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or "files.storage",
|
||||
resolution="Mount the installer-generated receipt and rerun preflight.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
binding_ref = fragment.fragment_id or f"{receipt.installation_id}:files.storage"
|
||||
capability_id = _text(payload.get("capability_id")) or "files.storage"
|
||||
if capability_id != "files.storage":
|
||||
diagnostics.append(
|
||||
_invalid(
|
||||
fragment,
|
||||
"Files managed-storage fragments must reference capability 'files.storage'.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
capability = receipt.capability(capability_id)
|
||||
if capability is None:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_missing",
|
||||
message="The deployment receipt does not declare managed file storage.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Regenerate the receipt from a deployment profile that declares managed storage.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
state_diagnostic = _storage_capability_state_diagnostic(capability, binding_ref)
|
||||
if state_diagnostic is not None:
|
||||
diagnostics.append(state_diagnostic)
|
||||
return None
|
||||
return receipt, capability, binding_ref
|
||||
|
||||
|
||||
def _storage_capability_state_diagnostic(
|
||||
capability: InfrastructureCapability,
|
||||
binding_ref: str,
|
||||
) -> ConfigurationDiagnostic | None:
|
||||
if capability.state == "unavailable":
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_unavailable",
|
||||
message="The deployment receipt states that managed file storage is unavailable.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Select local, managed Garage, or external S3 storage in the deployment profile.",
|
||||
)
|
||||
if capability.state == "available_unconfigured":
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_unconfigured",
|
||||
message="Managed file storage is available but has not been bound by the deployment runtime.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Complete the deployment-owned storage configuration before importing Files configuration.",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _package_expectation_diagnostics(
|
||||
payload: Mapping[str, Any],
|
||||
capability: InfrastructureCapability,
|
||||
expected_backend: str,
|
||||
binding_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
package_backend = (
|
||||
_text(payload.get("expected_backend")) or expected_backend
|
||||
).casefold()
|
||||
if package_backend != expected_backend:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_backend_mismatch",
|
||||
message="The package storage backend expectation conflicts with the deployment receipt.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Use the receipt backend or regenerate the deployment package after review.",
|
||||
)
|
||||
)
|
||||
package_source = _text(payload.get("expected_source"))
|
||||
if package_source and package_source != capability.source:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_source_mismatch",
|
||||
message="The package storage source expectation conflicts with the deployment receipt.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Review the provider replacement and regenerate the package from the active receipt.",
|
||||
)
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
|
||||
def _expected_backend(source: str) -> str:
|
||||
return "local" if source == "host-local" else "s3"
|
||||
|
||||
|
||||
def _normalized_backend(value: object) -> str:
|
||||
clean = str(value or "local").strip().casefold()
|
||||
if clean in {"local", "filesystem", "fs"}:
|
||||
return "local"
|
||||
if clean in {"s3", "garage"}:
|
||||
return "s3"
|
||||
return clean
|
||||
|
||||
|
||||
def _redacted_endpoint(value: str, *, default_port: int) -> dict[str, object]:
|
||||
try:
|
||||
parsed = urlsplit(value)
|
||||
host = parsed.hostname
|
||||
if not parsed.scheme or not host:
|
||||
return {"reference": "unresolved"}
|
||||
port = parsed.port or default_port
|
||||
except ValueError:
|
||||
return {"reference": "unresolved"}
|
||||
return {"scheme": parsed.scheme, "host": host, "port": port}
|
||||
|
||||
|
||||
def _text(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _receipt_error(error: str, *, object_ref: str) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_invalid",
|
||||
message=f"The deployment capability receipt is invalid: {error}",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Repair or regenerate the deployment receipt before importing Files configuration.",
|
||||
)
|
||||
|
||||
|
||||
def _invalid(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
message: str,
|
||||
) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_configuration_payload_invalid",
|
||||
message=message,
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||
resolution="Review the Files configuration-package fragment and rerun preflight.",
|
||||
)
|
||||
|
||||
|
||||
def _unsupported(fragment: ConfigurationPackageFragment) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="fragment_type_unsupported",
|
||||
message=f"Files configuration does not support fragment type {fragment.fragment_type!r}.",
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||
)
|
||||
|
||||
|
||||
def _blocked_plan(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
summary: str,
|
||||
) -> ConfigurationPlanItem:
|
||||
return ConfigurationPlanItem(
|
||||
action="blocked",
|
||||
module_id="files",
|
||||
fragment_type=fragment.fragment_type,
|
||||
fragment_id=fragment.fragment_id,
|
||||
summary=summary,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["FILES_CONFIGURATION_CAPABILITY", "FilesConfigurationProvider"]
|
||||
@@ -1 +1,25 @@
|
||||
from govoplan_files.backend.db.models import *
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"CampaignAttachmentUse",
|
||||
"FileAsset",
|
||||
"FileBlob",
|
||||
"FileConnectorCredential",
|
||||
"FileConnectorPolicy",
|
||||
"FileConnectorProfile",
|
||||
"FileConnectorSpace",
|
||||
"FileFolder",
|
||||
"FileShare",
|
||||
"FileVersion",
|
||||
]
|
||||
|
||||
@@ -4,7 +4,18 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, Index, Integer, JSON, String, Text, UniqueConstraint, text
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
JSON,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from govoplan_core.db.base import Base, TimestampMixin
|
||||
@@ -16,18 +27,123 @@ def new_uuid() -> str:
|
||||
|
||||
class FileBlob(Base, TimestampMixin):
|
||||
__tablename__ = "file_blobs"
|
||||
__table_args__ = (UniqueConstraint("tenant_id", "checksum_sha256", "size_bytes", name="uq_file_blobs_tenant_checksum_size"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"checksum_sha256",
|
||||
"size_bytes",
|
||||
"protection_discriminator",
|
||||
name="uq_file_blobs_tenant_checksum_size_protection",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||
storage_bucket: Mapped[str | None] = mapped_column(String(255))
|
||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
protection_discriminator: Mapped[str] = mapped_column(
|
||||
String(320), default="plaintext", nullable=False, index=True
|
||||
)
|
||||
encryption_envelope_id: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True, index=True
|
||||
)
|
||||
storage_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
storage_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
integrity_status: Mapped[str] = mapped_column(
|
||||
String(30), default="unchecked", nullable=False, index=True
|
||||
)
|
||||
integrity_checked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
integrity_failure: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
quarantined_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileIntegrityScan(Base, TimestampMixin):
|
||||
__tablename__ = "file_integrity_scans"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||
storage_prefix: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30), default="pending", nullable=False, index=True
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
phase: Mapped[str] = mapped_column(String(30), default="blobs", nullable=False)
|
||||
verify_checksums: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False
|
||||
)
|
||||
batch_size: Mapped[int] = mapped_column(Integer, default=100, nullable=False)
|
||||
blob_cursor: Mapped[str | None] = mapped_column(String(36), nullable=True)
|
||||
object_cursor: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
scanned_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
verified_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
quarantined_blob_count: Mapped[int] = mapped_column(
|
||||
Integer, default=0, nullable=False
|
||||
)
|
||||
scanned_object_count: Mapped[int] = mapped_column(
|
||||
Integer, default=0, nullable=False
|
||||
)
|
||||
orphan_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
started_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
completed_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
last_error: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
|
||||
class FileIntegrityFinding(Base, TimestampMixin):
|
||||
__tablename__ = "file_integrity_findings"
|
||||
__table_args__ = (
|
||||
Index("ix_file_integrity_findings_scan_state", "scan_id", "state"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
scan_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_integrity_scans.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
kind: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||
state: Mapped[str] = mapped_column(
|
||||
String(30), default="open", nullable=False, index=True
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
blob_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
expected_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
observed_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
expected_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
observed_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
resolved_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileFolder(Base, TimestampMixin):
|
||||
@@ -35,14 +151,18 @@ class FileFolder(Base, TimestampMixin):
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"uq_file_folders_active_user_path",
|
||||
"tenant_id", "owner_user_id", "path",
|
||||
"tenant_id",
|
||||
"owner_user_id",
|
||||
"path",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
),
|
||||
Index(
|
||||
"uq_file_folders_active_group_path",
|
||||
"tenant_id", "owner_group_id", "path",
|
||||
"tenant_id",
|
||||
"owner_group_id",
|
||||
"path",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
@@ -50,91 +170,414 @@ class FileFolder(Base, TimestampMixin):
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class FileAsset(Base, TimestampMixin):
|
||||
__tablename__ = "file_assets"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
current_version_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
current_version_id: Mapped[str | None] = mapped_column(
|
||||
String(36), nullable=True, index=True
|
||||
)
|
||||
display_path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||
filename: Mapped[str] = mapped_column(String(500), nullable=False, index=True)
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
retained_until: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
legal_hold: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, nullable=False, index=True
|
||||
)
|
||||
lifecycle_revision: Mapped[int] = mapped_column(
|
||||
Integer, default=1, nullable=False
|
||||
)
|
||||
lifecycle_reason: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class FileVersion(Base, TimestampMixin):
|
||||
__tablename__ = "file_versions"
|
||||
__table_args__ = (UniqueConstraint("file_asset_id", "version_number", name="uq_file_versions_asset_number"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"file_asset_id", "version_number", name="uq_file_versions_asset_number"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
blob_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
version_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
filename_at_upload: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
display_path_at_upload: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileFormEvidenceGrant(Base, TimestampMixin):
|
||||
__tablename__ = "file_form_evidence_grants"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_file_form_evidence_grants_idempotency",
|
||||
),
|
||||
Index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
"tenant_id",
|
||||
"form_instance_id",
|
||||
"form_definition_id",
|
||||
"form_definition_revision",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
form_instance_id: Mapped[str] = mapped_column(
|
||||
String(36), nullable=False, index=True
|
||||
)
|
||||
form_definition_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
form_definition_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
token_sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
request_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
custodian_user_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="RESTRICT"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
evidence_kind: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
purpose: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30), default="issued", nullable=False, index=True
|
||||
)
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
max_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
allowed_content_types: Mapped[list[str]] = mapped_column(
|
||||
JSON, default=list, nullable=False
|
||||
)
|
||||
file_asset_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||
)
|
||||
file_version_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||
)
|
||||
uploaded_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any]] = mapped_column(
|
||||
"metadata", JSON, default=dict, nullable=False
|
||||
)
|
||||
|
||||
|
||||
class FileShare(Base, TimestampMixin):
|
||||
__tablename__ = "file_shares"
|
||||
__table_args__ = (UniqueConstraint("file_asset_id", "target_type", "target_id", "revoked_at", name="uq_file_shares_active_target"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"file_asset_id",
|
||||
"target_type",
|
||||
"target_id",
|
||||
"revoked_at",
|
||||
name="uq_file_shares_active_target",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
target_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
revoked_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorProfile(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_profiles"
|
||||
__table_args__ = (
|
||||
Index("ix_file_connector_profiles_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(
|
||||
String(20), default="tenant", nullable=False, index=True
|
||||
)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||
endpoint_url: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
base_path: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
credential_profile_id: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True, index=True
|
||||
)
|
||||
credential_mode: Mapped[str] = mapped_column(
|
||||
String(30), default="none", nullable=False
|
||||
)
|
||||
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
password_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
capabilities: Mapped[list[str] | None] = mapped_column(JSON, nullable=True)
|
||||
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorCredential(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_credentials"
|
||||
__table_args__ = (
|
||||
Index("ix_file_connector_credentials_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(
|
||||
String(20), default="tenant", nullable=False, index=True
|
||||
)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
provider: Mapped[str | None] = mapped_column(String(50), nullable=True, index=True)
|
||||
enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
credential_mode: Mapped[str] = mapped_column(
|
||||
String(30), default="none", nullable=False
|
||||
)
|
||||
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
password_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorPolicy(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_policies"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
name="uq_file_connector_policies_scope",
|
||||
),
|
||||
Index("ix_file_connector_policies_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
policy: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorSpace(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_spaces"
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"ix_file_connector_spaces_owner",
|
||||
"tenant_id",
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
),
|
||||
Index(
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"tenant_id",
|
||||
"owner_user_id",
|
||||
"label",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
),
|
||||
Index(
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"tenant_id",
|
||||
"owner_group_id",
|
||||
"label",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
connector_profile_id: Mapped[str] = mapped_column(
|
||||
String(255), nullable=False, index=True
|
||||
)
|
||||
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||
library_id: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
remote_path: Mapped[str] = mapped_column(String(1000), default="", nullable=False)
|
||||
sync_mode: Mapped[str] = mapped_column(String(30), default="manual", nullable=False)
|
||||
read_only: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
is_active: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class CampaignAttachmentUse(Base, TimestampMixin):
|
||||
__tablename__ = "campaign_attachment_uses"
|
||||
__table_args__ = (UniqueConstraint("campaign_job_id", "file_version_id", "filename_used", "use_stage", name="uq_campaign_attachment_uses_job_file_stage"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"campaign_job_id",
|
||||
"file_version_id",
|
||||
"filename_used",
|
||||
"use_stage",
|
||||
name="uq_campaign_attachment_uses_job_file_stage",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_id: Mapped[str] = mapped_column(ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_version_id: Mapped[str] = mapped_column(ForeignKey("campaign_versions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_job_id: Mapped[str | None] = mapped_column(ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
campaign_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
campaign_version_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("campaign_versions.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
campaign_job_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
entry_index: Mapped[int | None] = mapped_column(Integer)
|
||||
entry_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_version_id: Mapped[str] = mapped_column(ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
file_version_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
file_blob_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
filename_used: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
use_stage: Mapped[str] = mapped_column(String(20), default="built", nullable=False, index=True)
|
||||
used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
use_stage: Mapped[str] = mapped_column(
|
||||
String(20), default="built", nullable=False, index=True
|
||||
)
|
||||
used_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CampaignAttachmentUse",
|
||||
"FileAsset",
|
||||
"FileBlob",
|
||||
"FileConnectorCredential",
|
||||
"FileConnectorPolicy",
|
||||
"FileConnectorProfile",
|
||||
"FileConnectorSpace",
|
||||
"FileFolder",
|
||||
"FileFormEvidenceGrant",
|
||||
"FileShare",
|
||||
"FileVersion",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,517 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.campaigns import (
|
||||
CAPABILITY_CAMPAIGNS_ACCESS,
|
||||
CampaignAccessProvider,
|
||||
)
|
||||
from govoplan_core.core.modules import (
|
||||
DocumentationCondition,
|
||||
DocumentationContext,
|
||||
DocumentationLink,
|
||||
DocumentationTopic,
|
||||
)
|
||||
from govoplan_files.backend.storage.archives import ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||
from govoplan_files.backend.storage.access import user_group_ids
|
||||
from govoplan_files.backend.storage.connector_visibility import (
|
||||
connector_profile_usable_for_import,
|
||||
visible_connector_profiles_for_actor,
|
||||
)
|
||||
|
||||
|
||||
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
|
||||
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
|
||||
_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES = 2 * 1024 * 1024 * 1024
|
||||
_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO = 100
|
||||
_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS = 30 * 60
|
||||
_FILES_READ_SCOPE = "files:file:read"
|
||||
_FILES_UPLOAD_SCOPE = "files:file:upload"
|
||||
|
||||
|
||||
def documentation_topics(
|
||||
context: DocumentationContext,
|
||||
) -> tuple[DocumentationTopic, ...]:
|
||||
if context.documentation_type != "user":
|
||||
return ()
|
||||
|
||||
upload_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_upload_max_bytes",
|
||||
default=_DEFAULT_UPLOAD_MAX_BYTES,
|
||||
)
|
||||
zip_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_upload_zip_max_bytes",
|
||||
default=_DEFAULT_ZIP_MAX_BYTES,
|
||||
)
|
||||
archive_expanded_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_max_expanded_bytes",
|
||||
default=_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES,
|
||||
)
|
||||
archive_entry_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_max_entries",
|
||||
default=ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
)
|
||||
archive_ratio_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_max_expansion_ratio",
|
||||
default=_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO,
|
||||
)
|
||||
archive_preview_ttl = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_preview_ttl_seconds",
|
||||
default=_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS,
|
||||
)
|
||||
topics: list[DocumentationTopic] = []
|
||||
if upload_limit is not None:
|
||||
topics.append(_upload_topic(upload_limit))
|
||||
if (
|
||||
upload_limit is not None
|
||||
and zip_limit is not None
|
||||
and archive_expanded_limit is not None
|
||||
and archive_entry_limit is not None
|
||||
and archive_ratio_limit is not None
|
||||
and archive_preview_ttl is not None
|
||||
):
|
||||
topics.append(
|
||||
_archive_topic(
|
||||
upload_limit,
|
||||
zip_limit,
|
||||
archive_expanded_limit,
|
||||
archive_entry_limit,
|
||||
archive_ratio_limit,
|
||||
archive_preview_ttl,
|
||||
)
|
||||
)
|
||||
topics.append(_connector_import_topic(context))
|
||||
return tuple(topics)
|
||||
|
||||
|
||||
def _upload_topic(max_bytes: int) -> DocumentationTopic:
|
||||
limit = _format_byte_limit(max_bytes)
|
||||
return DocumentationTopic(
|
||||
id="files.workflow.upload-managed-files",
|
||||
title="Upload managed files",
|
||||
summary=f"Upload files to a personal or accessible group space; each uploaded file may contain at most {limit}.",
|
||||
body=(
|
||||
f"The current deployment accepts at most {limit} for each ordinary upload. "
|
||||
"A name conflict is never resolved silently: reject stops the upload, rename chooses a copy name, and overwrite retires the old asset before creating a new one."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "file_manager", "process_participant"),
|
||||
order=39,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(
|
||||
label="Files handbook",
|
||||
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||
kind="repository",
|
||||
),
|
||||
),
|
||||
related_modules=("campaigns",),
|
||||
unlocks=(
|
||||
"Users and connected processes can place governed content in managed storage.",
|
||||
),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/files",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list"],
|
||||
"prerequisites": [
|
||||
"You may view and upload managed files.",
|
||||
"The destination personal or group space grants this account write access; upload permission alone does not grant access to every space.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and choose My files or an accessible group space.",
|
||||
"Open the intended destination folder and choose Upload, or drag files into the file list.",
|
||||
"For every name conflict, explicitly reject, rename, overwrite, or skip the affected item.",
|
||||
"Wait for the upload to finish, then open the resulting file details.",
|
||||
],
|
||||
"outcome": "Each accepted file is stored as a governed managed asset in the selected space.",
|
||||
"verification": "Confirm the owner, logical path, size, checksum, and current version in Files.",
|
||||
"constraints": [
|
||||
{
|
||||
"id": "ordinary-upload-size",
|
||||
"label": "Maximum size per file",
|
||||
"description": f"The current safe upload limit is {limit} per file.",
|
||||
"values": [limit],
|
||||
},
|
||||
],
|
||||
"related_topic_ids": [
|
||||
"files.workflow.upload-and-unpack-zip",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.find-and-download-files",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _archive_topic(
|
||||
max_file_bytes: int,
|
||||
max_archive_request_bytes: int,
|
||||
max_expanded_bytes: int,
|
||||
max_entries: int,
|
||||
max_expansion_ratio: int,
|
||||
preview_ttl_seconds: int,
|
||||
) -> DocumentationTopic:
|
||||
member_limit = _format_byte_limit(max_file_bytes)
|
||||
request_limit = _format_byte_limit(max_archive_request_bytes)
|
||||
expanded_limit = _format_byte_limit(max_expanded_bytes)
|
||||
preview_minutes = max(1, preview_ttl_seconds // 60)
|
||||
return DocumentationTopic(
|
||||
id="files.workflow.upload-and-unpack-zip",
|
||||
title="Preview and unpack an archive",
|
||||
summary=(
|
||||
f"Review and selectively unpack up to {max_entries:,} entries from ZIP or TAR archives before any managed file is created."
|
||||
),
|
||||
body=(
|
||||
f"The deployment accepts ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ requests up to {request_limit}, limits actual expanded data to {expanded_limit}, "
|
||||
f"each member to {member_limit}, the archive to {max_entries:,} entries, and expansion to {max_expansion_ratio}:1. "
|
||||
f"The server-issued preview expires after {preview_minutes} minutes. Password-protected ZIP archives are supported; passwords remain request-only. "
|
||||
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "file_manager", "process_participant"),
|
||||
order=40,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(
|
||||
label="Files handbook",
|
||||
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||
kind="repository",
|
||||
),
|
||||
),
|
||||
unlocks=(
|
||||
"A bounded archive can become governed managed files without trusting archive paths or size declarations.",
|
||||
),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/files",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list"],
|
||||
"prerequisites": [
|
||||
"You may view and upload managed files.",
|
||||
"The archive fits the configured request, expansion, size, and entry limits.",
|
||||
"The destination personal or group space grants this account write access.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and choose the managed destination space and folder.",
|
||||
"Enable Preview and unpack archive, then choose or drag one supported archive.",
|
||||
"Review the discovered entries, supply a ZIP password when required, and select the files or folders to import.",
|
||||
"Resolve every destination conflict explicitly.",
|
||||
"Confirm the selection, then wait for extraction and finalization to finish before leaving the page.",
|
||||
],
|
||||
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
|
||||
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
|
||||
"constraints": [
|
||||
{
|
||||
"id": "archive-request-and-total",
|
||||
"label": "Maximum archive request and expanded total",
|
||||
"description": f"The compressed request is limited to {request_limit}; actual expanded data is limited to {expanded_limit}.",
|
||||
"values": [request_limit, expanded_limit],
|
||||
},
|
||||
{
|
||||
"id": "archive-member-size",
|
||||
"label": "Maximum extracted member size",
|
||||
"description": f"Each extracted file is limited to {member_limit}.",
|
||||
"values": [member_limit],
|
||||
},
|
||||
{
|
||||
"id": "archive-entry-count",
|
||||
"label": "Maximum entry count",
|
||||
"description": f"An archive may contain at most {max_entries:,} declared entries.",
|
||||
"values": [f"{max_entries:,} entries"],
|
||||
},
|
||||
{
|
||||
"id": "archive-expansion-ratio",
|
||||
"label": "Maximum expansion ratio",
|
||||
"description": f"Declared and actual output may not exceed {max_expansion_ratio} times the compressed request size.",
|
||||
"values": [f"{max_expansion_ratio}:1"],
|
||||
},
|
||||
],
|
||||
"related_topic_ids": [
|
||||
"files.workflow.upload-managed-files",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.find-and-download-files",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _connector_import_topic(context: DocumentationContext) -> DocumentationTopic:
|
||||
principal = context.principal
|
||||
if not _has_all_scopes(principal, (_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE)):
|
||||
return _connector_import_limitation(
|
||||
"Connector import is not available to this account because both permission to view Files and permission to upload managed files are required."
|
||||
)
|
||||
|
||||
tenant_id = _safe_text_attribute(principal, "tenant_id")
|
||||
user_id = _safe_text_attribute(getattr(principal, "user", None), "id")
|
||||
session = context.session
|
||||
if not tenant_id or not user_id or not isinstance(session, Session):
|
||||
return _connector_import_limitation(
|
||||
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||
)
|
||||
|
||||
try:
|
||||
member_group_ids = _actor_group_ids(
|
||||
session,
|
||||
principal=principal,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
include_admin_groups=False,
|
||||
)
|
||||
connector_group_ids = (
|
||||
_actor_group_ids(
|
||||
session,
|
||||
principal=principal,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
include_admin_groups=True,
|
||||
)
|
||||
if _has_all_scopes(principal, ("files:file:admin",))
|
||||
else member_group_ids
|
||||
)
|
||||
profiles = visible_connector_profiles_for_actor(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=connector_group_ids,
|
||||
settings=context.settings,
|
||||
campaign_visible=_campaign_visibility(
|
||||
context,
|
||||
session=session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=member_group_ids,
|
||||
),
|
||||
include_effective_policy=True,
|
||||
)
|
||||
usable_profiles = tuple(
|
||||
profile
|
||||
for profile in profiles
|
||||
if connector_profile_usable_for_import(profile)
|
||||
)
|
||||
except Exception:
|
||||
return _connector_import_limitation(
|
||||
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||
)
|
||||
if not usable_profiles:
|
||||
return _connector_import_limitation(
|
||||
"No connection visible to this account is currently eligible to offer an enabled, credential-ready, policy-allowed browse/import path through a pinning-safe provider. Ask a Files administrator to configure or authorize one."
|
||||
)
|
||||
|
||||
return DocumentationTopic(
|
||||
id="files.workflow.import-managed-snapshot",
|
||||
title="Import an external file as a governed snapshot",
|
||||
summary="Browse an authorized connection read-only and import one selected file into managed storage as a frozen, traceable snapshot.",
|
||||
body=(
|
||||
"At least one connection visible to this account is currently eligible to offer the governed browse/import path. "
|
||||
"The selected remote path and item are re-authorized when used, and browse, import, and sync never mutate the remote source. "
|
||||
"The managed snapshot stays unchanged until an explicit manual sync."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "campaign_manager", "report_author"),
|
||||
order=41,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(
|
||||
label="Files handbook",
|
||||
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||
kind="repository",
|
||||
),
|
||||
),
|
||||
related_modules=("campaigns",),
|
||||
unlocks=(
|
||||
"Campaigns, reports, and workflows can consume a managed snapshot with stable source evidence.",
|
||||
),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/files",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list", "files.connector-import"],
|
||||
"prerequisites": [
|
||||
"You may view and upload managed files.",
|
||||
"At least one enabled, credential-ready, policy-allowed connection using a pinning-safe provider is visible to this account.",
|
||||
"The selected remote path and item must pass their operation-time policy checks.",
|
||||
"The managed destination space grants this account write access.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and choose a managed destination space.",
|
||||
"Choose Sync from connection, select an available connection, and browse to the permitted remote file.",
|
||||
"Import the selected file and resolve any destination conflict explicitly.",
|
||||
"Review the managed file's source and current-version details before using it in another task.",
|
||||
],
|
||||
"current_configuration": [
|
||||
"At least one actor-visible connection is eligible to offer a safe browse/import operation; the selected endpoint, path, and item are still checked when used.",
|
||||
"Every selected remote path and item is re-authorized at operation time.",
|
||||
],
|
||||
"outcome": "The external content is a tenant-managed snapshot with a checksum, exact version, and recorded source context.",
|
||||
"verification": "Reopen the managed file and confirm its recorded source context, source revision when available, checksum, and current version.",
|
||||
"related_topic_ids": [
|
||||
"files.governed-connectors-and-provenance",
|
||||
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||
"files.reference.snapshot-provenance-and-capabilities",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _connector_import_limitation(message: str) -> DocumentationTopic:
|
||||
return DocumentationTopic(
|
||||
id="files.connector-import-unavailable",
|
||||
title="External file import is not currently available",
|
||||
summary=message,
|
||||
body=(
|
||||
f"{message} Files never exposes connection endpoints, storage paths, credential references, or raw connector policies in user documentation."
|
||||
),
|
||||
layer="available",
|
||||
documentation_types=("user",),
|
||||
order=41,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
any_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(DocumentationLink(label="Files", href="/files", kind="runtime"),),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "reference",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list", "files.connector-import"],
|
||||
"limitations": [message],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _configured_positive_int(
|
||||
settings: object | None, name: str, *, default: int
|
||||
) -> int | None:
|
||||
raw_value = getattr(settings, name, default) if settings is not None else default
|
||||
try:
|
||||
value = int(raw_value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return value if value > 0 else None
|
||||
|
||||
|
||||
def _format_byte_limit(value: int) -> str:
|
||||
units = ((1024**3, "GiB"), (1024**2, "MiB"), (1024, "KiB"))
|
||||
for divisor, label in units:
|
||||
if value % divisor == 0:
|
||||
return f"{value // divisor:,} {label} ({value:,} bytes)"
|
||||
return f"{value:,} bytes"
|
||||
|
||||
|
||||
def _has_all_scopes(principal: object | None, scopes: tuple[str, ...]) -> bool:
|
||||
checker = getattr(principal, "has", None)
|
||||
if callable(checker):
|
||||
try:
|
||||
return all(bool(checker(scope)) for scope in scopes)
|
||||
except Exception:
|
||||
return False
|
||||
granted = {str(scope) for scope in getattr(principal, "scopes", ())}
|
||||
return all(scope in granted for scope in scopes)
|
||||
|
||||
|
||||
def _safe_text_attribute(value: object | None, name: str) -> str:
|
||||
try:
|
||||
result = getattr(value, name, "")
|
||||
except Exception:
|
||||
return ""
|
||||
return str(result or "")
|
||||
|
||||
|
||||
def _principal_group_ids(principal: object) -> tuple[str, ...]:
|
||||
try:
|
||||
values = getattr(principal, "group_ids", ())
|
||||
except Exception:
|
||||
return ()
|
||||
return tuple(str(value) for value in values if str(value))
|
||||
|
||||
|
||||
def _actor_group_ids(
|
||||
session: Session,
|
||||
*,
|
||||
principal: object,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
include_admin_groups: bool,
|
||||
) -> tuple[str, ...]:
|
||||
try:
|
||||
values = user_group_ids(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
include_admin_groups=include_admin_groups,
|
||||
)
|
||||
except Exception:
|
||||
return _principal_group_ids(principal)
|
||||
return tuple(str(value) for value in values if str(value))
|
||||
|
||||
|
||||
def _campaign_visibility(
|
||||
context: DocumentationContext,
|
||||
*,
|
||||
session: Session,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: tuple[str, ...],
|
||||
):
|
||||
principal = context.principal
|
||||
registry = context.registry
|
||||
if not _has_all_scopes(principal, ("campaigns:campaign:read",)):
|
||||
return None
|
||||
try:
|
||||
if not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||
return None
|
||||
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(capability, CampaignAccessProvider):
|
||||
return None
|
||||
|
||||
def campaign_visible(campaign_id: str) -> bool:
|
||||
try:
|
||||
return capability.campaign_exists(
|
||||
session, tenant_id=tenant_id, campaign_id=campaign_id
|
||||
) and capability.can_read_campaign(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
user_id=user_id,
|
||||
group_ids=group_ids,
|
||||
tenant_admin=_has_all_scopes(principal, ("tenant:*",)),
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
return campaign_visible
|
||||
@@ -0,0 +1,879 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import inspect, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import (
|
||||
DsarErasureActionRef,
|
||||
DsarExecutionResultRef,
|
||||
DsarRecordRef,
|
||||
DsarSubjectRef,
|
||||
dsar_capability_name,
|
||||
)
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
|
||||
|
||||
FILES_DSAR_CAPABILITY = dsar_capability_name("files")
|
||||
_MAX_RECORDS = 5_000
|
||||
|
||||
|
||||
class FilesDsarProvider:
|
||||
provider_id = "files"
|
||||
module_id = "files"
|
||||
|
||||
def search_subject(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
) -> Sequence[DsarRecordRef]:
|
||||
db = _session(session)
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return ()
|
||||
|
||||
records: list[DsarRecordRef] = []
|
||||
form_evidence_available = _has_table(db, FileFormEvidenceGrant)
|
||||
campaign_evidence_available = _has_table(db, CampaignAttachmentUse)
|
||||
retention_reasons: dict[str, str | None] = {}
|
||||
|
||||
def retention_reason(asset: FileAsset) -> str | None:
|
||||
if asset.id not in retention_reasons:
|
||||
retention_reasons[asset.id] = _asset_retention_reason(
|
||||
db,
|
||||
asset,
|
||||
form_evidence_available=form_evidence_available,
|
||||
campaign_evidence_available=campaign_evidence_available,
|
||||
)
|
||||
return retention_reasons[asset.id]
|
||||
|
||||
def append(record: DsarRecordRef) -> None:
|
||||
if len(records) >= _MAX_RECORDS:
|
||||
raise ValueError(
|
||||
"Files DSAR match limit exceeded; narrow the subject selectors."
|
||||
)
|
||||
records.append(record)
|
||||
|
||||
assets = _bounded_rows(
|
||||
db.query(FileAsset)
|
||||
.filter(
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
FileAsset.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileAsset.id)
|
||||
)
|
||||
for asset in assets:
|
||||
match_fields = _matching_fields(
|
||||
asset,
|
||||
subject_user_id,
|
||||
("owner_user_id", "created_by_user_id"),
|
||||
)
|
||||
asset_retention_reason = retention_reason(asset)
|
||||
append(
|
||||
_record(
|
||||
"file_asset",
|
||||
asset.id,
|
||||
"managed_file",
|
||||
asset.filename,
|
||||
{
|
||||
"match_fields": match_fields,
|
||||
"owner_type": asset.owner_type,
|
||||
"display_path": asset.display_path,
|
||||
"filename": asset.filename,
|
||||
"description": asset.description,
|
||||
"deleted_at": _iso(asset.deleted_at),
|
||||
"retained_until": _iso(asset.retained_until),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_reason": asset.lifecycle_reason,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
},
|
||||
observed_at=asset.updated_at,
|
||||
immutable=asset_retention_reason is not None,
|
||||
retention_reason=asset_retention_reason,
|
||||
source_path=f"/files?file={asset.id}",
|
||||
)
|
||||
)
|
||||
|
||||
version_rows = _bounded_rows(
|
||||
db.query(FileVersion, FileAsset)
|
||||
.join(FileAsset, FileAsset.id == FileVersion.file_asset_id)
|
||||
.filter(
|
||||
FileVersion.tenant_id == tenant_id,
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
FileVersion.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileVersion.id)
|
||||
)
|
||||
for version, asset in version_rows:
|
||||
asset_retention_reason = retention_reason(asset)
|
||||
append(
|
||||
_record(
|
||||
"file_version",
|
||||
version.id,
|
||||
"managed_file_version",
|
||||
version.filename_at_upload,
|
||||
{
|
||||
"match_fields": (
|
||||
["asset.owner_user_id"]
|
||||
if asset.owner_user_id == subject_user_id
|
||||
else []
|
||||
)
|
||||
+ (
|
||||
["created_by_user_id"]
|
||||
if version.created_by_user_id == subject_user_id
|
||||
else []
|
||||
),
|
||||
"file_asset_id": version.file_asset_id,
|
||||
"version_number": version.version_number,
|
||||
"filename_at_upload": version.filename_at_upload,
|
||||
"display_path_at_upload": version.display_path_at_upload,
|
||||
"content_type": version.content_type,
|
||||
"size_bytes": version.size_bytes,
|
||||
"checksum_sha256": version.checksum_sha256,
|
||||
"created_at": _iso(version.created_at),
|
||||
},
|
||||
observed_at=version.updated_at,
|
||||
immutable=asset_retention_reason is not None,
|
||||
retention_reason=asset_retention_reason,
|
||||
source_path=(
|
||||
f"/api/v1/files/{version.file_asset_id}/versions/"
|
||||
f"{version.id}/download"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for folder in _bounded_rows(
|
||||
db.query(FileFolder)
|
||||
.filter(
|
||||
FileFolder.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileFolder.owner_user_id == subject_user_id,
|
||||
FileFolder.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileFolder.id)
|
||||
):
|
||||
append(
|
||||
_record(
|
||||
"file_folder",
|
||||
folder.id,
|
||||
"managed_folder",
|
||||
folder.path,
|
||||
{
|
||||
"match_fields": _matching_fields(
|
||||
folder,
|
||||
subject_user_id,
|
||||
("owner_user_id", "created_by_user_id"),
|
||||
),
|
||||
"owner_type": folder.owner_type,
|
||||
"path": folder.path,
|
||||
"deleted_at": _iso(folder.deleted_at),
|
||||
},
|
||||
observed_at=folder.updated_at,
|
||||
source_path="/files",
|
||||
)
|
||||
)
|
||||
|
||||
for share in _bounded_rows(
|
||||
db.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
or_(
|
||||
(FileShare.target_type == "user")
|
||||
& (FileShare.target_id == subject_user_id),
|
||||
FileShare.created_by_user_id == subject_user_id,
|
||||
FileShare.revoked_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileShare.id)
|
||||
):
|
||||
target_matches = (
|
||||
share.target_type == "user" and share.target_id == subject_user_id
|
||||
)
|
||||
match_fields = _matching_fields(
|
||||
share,
|
||||
subject_user_id,
|
||||
("created_by_user_id", "revoked_by_user_id"),
|
||||
)
|
||||
if target_matches:
|
||||
match_fields.insert(0, "target_id")
|
||||
append(
|
||||
_record(
|
||||
"file_share",
|
||||
share.id,
|
||||
"file_access_evidence",
|
||||
f"Share for file {share.file_asset_id}",
|
||||
{
|
||||
"match_fields": match_fields,
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_is_subject": target_matches,
|
||||
"permission": share.permission,
|
||||
"expires_at": _iso(share.expires_at),
|
||||
"revoked_at": _iso(share.revoked_at),
|
||||
},
|
||||
observed_at=share.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"File-sharing history is institutional access evidence."
|
||||
),
|
||||
source_path=f"/files?file={share.file_asset_id}",
|
||||
)
|
||||
)
|
||||
|
||||
if form_evidence_available:
|
||||
evidence_rows = _bounded_rows(
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.outerjoin(
|
||||
FileAsset,
|
||||
FileAsset.id == FileFormEvidenceGrant.file_asset_id,
|
||||
)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileFormEvidenceGrant.custodian_user_id == subject_user_id,
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileFormEvidenceGrant.id)
|
||||
)
|
||||
for grant in evidence_rows:
|
||||
append(
|
||||
_record(
|
||||
"file_form_evidence",
|
||||
grant.id,
|
||||
"form_evidence",
|
||||
f"Form evidence {grant.form_definition_id}",
|
||||
{
|
||||
"match_fields": (
|
||||
["custodian_user_id"]
|
||||
if grant.custodian_user_id == subject_user_id
|
||||
else ["asset.owner_user_id"]
|
||||
),
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"form_definition_id": grant.form_definition_id,
|
||||
"form_definition_revision": grant.form_definition_revision,
|
||||
"evidence_kind": grant.evidence_kind,
|
||||
"purpose": grant.purpose,
|
||||
"status": grant.status,
|
||||
"expires_at": _iso(grant.expires_at),
|
||||
"file_asset_id": grant.file_asset_id,
|
||||
"file_version_id": grant.file_version_id,
|
||||
},
|
||||
observed_at=grant.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Submitted Form attachment evidence follows the owning "
|
||||
"process retention and cannot be erased through Files alone."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
if campaign_evidence_available:
|
||||
campaign_rows = _bounded_rows(
|
||||
db.query(CampaignAttachmentUse)
|
||||
.join(FileAsset, FileAsset.id == CampaignAttachmentUse.file_asset_id)
|
||||
.filter(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(CampaignAttachmentUse.id)
|
||||
)
|
||||
for use in campaign_rows:
|
||||
append(
|
||||
_record(
|
||||
"campaign_attachment_use",
|
||||
use.id,
|
||||
"delivery_evidence",
|
||||
use.filename_used,
|
||||
{
|
||||
"match_fields": ["asset.owner_user_id"],
|
||||
"campaign_id": use.campaign_id,
|
||||
"campaign_version_id": use.campaign_version_id,
|
||||
"campaign_job_id": use.campaign_job_id,
|
||||
"file_asset_id": use.file_asset_id,
|
||||
"file_version_id": use.file_version_id,
|
||||
"filename_used": use.filename_used,
|
||||
"checksum_sha256": use.checksum_sha256,
|
||||
"size_bytes": use.size_bytes,
|
||||
"use_stage": use.use_stage,
|
||||
"used_at": _iso(use.used_at),
|
||||
},
|
||||
observed_at=use.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Campaign attachment use is immutable delivery evidence."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
self._append_configuration_references(
|
||||
db,
|
||||
append=append,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
)
|
||||
self._append_integrity_references(
|
||||
db,
|
||||
append=append,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
)
|
||||
return tuple(records)
|
||||
|
||||
def plan_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
records: Sequence[DsarRecordRef],
|
||||
) -> Sequence[DsarErasureActionRef]:
|
||||
del session
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return ()
|
||||
actions: list[DsarErasureActionRef] = []
|
||||
for record in records:
|
||||
if record.provider_id != self.provider_id or record.module_id != self.module_id:
|
||||
raise ValueError("Files DSAR received a foreign provider record.")
|
||||
match_fields = {
|
||||
str(value) for value in record.data.get("match_fields", ())
|
||||
}
|
||||
if record.immutable_evidence:
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:retain:{record.resource_type}:{record.resource_id}",
|
||||
"retain",
|
||||
record,
|
||||
f"Retain {record.title}",
|
||||
record.retention_reason
|
||||
or "Institutional evidence must be retained.",
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
elif record.resource_type in {
|
||||
"file_asset",
|
||||
"file_version",
|
||||
"file_folder",
|
||||
}:
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:review:{record.resource_type}:{record.resource_id}",
|
||||
"manual_review",
|
||||
record,
|
||||
f"Review {record.title}",
|
||||
(
|
||||
"Managed file content, names, paths, ownership, and shared "
|
||||
"references require a case decision. Approved byte erasure "
|
||||
"must use the separate Files purge workflow."
|
||||
),
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
if (
|
||||
record.resource_type == "file_share"
|
||||
and "target_id" in match_fields
|
||||
and record.data.get("revoked_at") is None
|
||||
):
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:revoke:file_share:{record.resource_id}",
|
||||
"revoke",
|
||||
record,
|
||||
"Revoke active file share",
|
||||
"The active user-targeted share can be revoked without deleting file evidence.",
|
||||
executable=True,
|
||||
metadata={"subject_user_id": subject_user_id},
|
||||
)
|
||||
)
|
||||
for field_name in sorted(
|
||||
match_fields.intersection(_detachable_fields(record.resource_type))
|
||||
):
|
||||
actions.append(
|
||||
_action(
|
||||
(
|
||||
f"files:detach:{record.resource_type}:"
|
||||
f"{field_name}:{record.resource_id}"
|
||||
),
|
||||
"detach",
|
||||
record,
|
||||
f"Detach {field_name.replace('_', ' ')}",
|
||||
(
|
||||
"Remove the mutable subject reference while preserving the "
|
||||
"governed resource and DSAR evidence."
|
||||
),
|
||||
executable=True,
|
||||
metadata={
|
||||
"field": field_name,
|
||||
"subject_user_id": subject_user_id,
|
||||
},
|
||||
)
|
||||
)
|
||||
action_ids = [action.action_id for action in actions]
|
||||
if len(action_ids) != len(set(action_ids)):
|
||||
raise ValueError("Files DSAR produced duplicate action ids.")
|
||||
return tuple(actions)
|
||||
|
||||
def execute_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
actions: Sequence[DsarErasureActionRef],
|
||||
request_id: str,
|
||||
) -> Sequence[DsarExecutionResultRef]:
|
||||
db = _session(session)
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return tuple(
|
||||
_blocked(action, "Files requires a direct membership subject reference.")
|
||||
for action in actions
|
||||
)
|
||||
results: list[DsarExecutionResultRef] = []
|
||||
for action in actions:
|
||||
if (
|
||||
action.provider_id != self.provider_id
|
||||
or action.module_id != self.module_id
|
||||
or action.metadata.get("subject_user_id") != subject_user_id
|
||||
):
|
||||
results.append(_blocked(action, "The Files DSAR action is stale or invalid."))
|
||||
continue
|
||||
if action.action_id.startswith("files:revoke:file_share:"):
|
||||
results.append(
|
||||
_revoke_share(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
action=action,
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
elif action.action_id.startswith("files:detach:"):
|
||||
results.append(
|
||||
_detach_reference(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
action=action,
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
else:
|
||||
results.append(_blocked(action, "Files does not execute this action kind."))
|
||||
db.flush()
|
||||
return tuple(results)
|
||||
|
||||
def _append_configuration_references(
|
||||
self,
|
||||
db: Session,
|
||||
*,
|
||||
append: object,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
) -> None:
|
||||
configurations = (
|
||||
(FileConnectorProfile, "connector_profile", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorCredential, "connector_credential", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorPolicy, "connector_policy", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorSpace, "connector_space", ("owner_user_id", "created_by_user_id")),
|
||||
)
|
||||
for model, resource_type, fields in configurations:
|
||||
if not _has_table(db, model):
|
||||
continue
|
||||
conditions = [getattr(model, field) == subject_user_id for field in fields]
|
||||
query = db.query(model).filter(or_(*conditions))
|
||||
if hasattr(model, "tenant_id"):
|
||||
query = query.filter(model.tenant_id == tenant_id)
|
||||
for row in _bounded_rows(query.order_by(model.id)):
|
||||
match_fields = _matching_fields(row, subject_user_id, fields)
|
||||
data: dict[str, object] = {
|
||||
"match_fields": match_fields,
|
||||
"label": getattr(row, "label", None),
|
||||
"provider": getattr(row, "provider", None),
|
||||
}
|
||||
if isinstance(row, FileConnectorCredential):
|
||||
data["credential_mode"] = row.credential_mode
|
||||
elif isinstance(row, FileConnectorSpace):
|
||||
data.update(
|
||||
{
|
||||
"remote_path": row.remote_path,
|
||||
"sync_mode": row.sync_mode,
|
||||
"read_only": row.read_only,
|
||||
"deleted_at": _iso(row.deleted_at),
|
||||
}
|
||||
)
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
resource_type,
|
||||
row.id,
|
||||
"connector_configuration_evidence",
|
||||
getattr(row, "label", None) or resource_type.replace("_", " "),
|
||||
data,
|
||||
observed_at=row.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Connector configuration history is institutional evidence; "
|
||||
"credential secrets are excluded from the DSAR export."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def _append_integrity_references(
|
||||
self,
|
||||
db: Session,
|
||||
*,
|
||||
append: object,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
) -> None:
|
||||
if _has_table(db, FileIntegrityScan):
|
||||
for scan in _bounded_rows(
|
||||
db.query(FileIntegrityScan)
|
||||
.filter(
|
||||
FileIntegrityScan.tenant_id == tenant_id,
|
||||
FileIntegrityScan.created_by_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(FileIntegrityScan.id)
|
||||
):
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
"file_integrity_scan",
|
||||
scan.id,
|
||||
"storage_integrity_evidence",
|
||||
f"Integrity scan {scan.id}",
|
||||
{
|
||||
"match_fields": ["created_by_user_id"],
|
||||
"storage_backend": scan.storage_backend,
|
||||
"status": scan.status,
|
||||
"started_at": _iso(scan.started_at),
|
||||
"completed_at": _iso(scan.completed_at),
|
||||
},
|
||||
observed_at=scan.updated_at,
|
||||
immutable=True,
|
||||
retention_reason="Storage integrity scans are operator evidence.",
|
||||
)
|
||||
)
|
||||
if _has_table(db, FileIntegrityFinding):
|
||||
for finding in _bounded_rows(
|
||||
db.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.tenant_id == tenant_id,
|
||||
FileIntegrityFinding.resolved_by_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(FileIntegrityFinding.id)
|
||||
):
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
"file_integrity_finding",
|
||||
finding.id,
|
||||
"storage_integrity_evidence",
|
||||
f"Integrity finding {finding.kind}",
|
||||
{
|
||||
"match_fields": ["resolved_by_user_id"],
|
||||
"kind": finding.kind,
|
||||
"state": finding.state,
|
||||
"resolved_at": _iso(finding.resolved_at),
|
||||
},
|
||||
observed_at=finding.updated_at,
|
||||
immutable=True,
|
||||
retention_reason="Integrity resolution is operator evidence.",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _asset_retention_reason(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
form_evidence_available: bool,
|
||||
campaign_evidence_available: bool,
|
||||
) -> str | None:
|
||||
reasons: list[str] = []
|
||||
if asset.legal_hold:
|
||||
reasons.append("The file is under legal hold.")
|
||||
retained_until = _aware(asset.retained_until)
|
||||
if retained_until is not None and retained_until > datetime.now(timezone.utc):
|
||||
reasons.append(f"The file is retained until {retained_until.isoformat()}.")
|
||||
if form_evidence_available:
|
||||
has_form_evidence = (
|
||||
session.query(FileFormEvidenceGrant.id)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == asset.tenant_id,
|
||||
FileFormEvidenceGrant.file_asset_id == asset.id,
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
if has_form_evidence:
|
||||
reasons.append("The file is referenced by submitted Form evidence.")
|
||||
if campaign_evidence_available:
|
||||
has_campaign_evidence = (
|
||||
session.query(CampaignAttachmentUse.id)
|
||||
.filter(
|
||||
CampaignAttachmentUse.tenant_id == asset.tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == asset.id,
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
if has_campaign_evidence:
|
||||
reasons.append("The file is referenced by Campaign delivery evidence.")
|
||||
return " ".join(reasons) or None
|
||||
|
||||
|
||||
_DETACHABLE_MODELS: dict[str, tuple[type[object], frozenset[str]]] = {
|
||||
"file_asset": (FileAsset, frozenset({"created_by_user_id"})),
|
||||
"file_version": (FileVersion, frozenset({"created_by_user_id"})),
|
||||
"file_folder": (FileFolder, frozenset({"created_by_user_id"})),
|
||||
"file_share": (
|
||||
FileShare,
|
||||
frozenset({"created_by_user_id", "revoked_by_user_id"}),
|
||||
),
|
||||
"connector_profile": (
|
||||
FileConnectorProfile,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_credential": (
|
||||
FileConnectorCredential,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_policy": (
|
||||
FileConnectorPolicy,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_space": (FileConnectorSpace, frozenset({"created_by_user_id"})),
|
||||
}
|
||||
|
||||
|
||||
def _detachable_fields(resource_type: str) -> frozenset[str]:
|
||||
entry = _DETACHABLE_MODELS.get(resource_type)
|
||||
return entry[1] if entry is not None else frozenset()
|
||||
|
||||
|
||||
def _revoke_share(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
action: DsarErasureActionRef,
|
||||
request_id: str,
|
||||
) -> DsarExecutionResultRef:
|
||||
row = (
|
||||
session.query(FileShare)
|
||||
.filter(FileShare.id == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if (
|
||||
row is None
|
||||
or row.tenant_id != tenant_id
|
||||
or row.target_type != "user"
|
||||
or row.target_id != subject_user_id
|
||||
):
|
||||
return _blocked(action, "The subject-targeted file share is no longer available.")
|
||||
if row.revoked_at is not None:
|
||||
return _result(
|
||||
action,
|
||||
"unchanged",
|
||||
"The file share was already revoked.",
|
||||
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||
)
|
||||
row.revoked_at = datetime.now(timezone.utc)
|
||||
row.revoked_by_user_id = None
|
||||
return _result(
|
||||
action,
|
||||
"executed",
|
||||
"The subject-targeted file share was revoked.",
|
||||
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||
)
|
||||
|
||||
|
||||
def _detach_reference(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
action: DsarErasureActionRef,
|
||||
request_id: str,
|
||||
) -> DsarExecutionResultRef:
|
||||
entry = _DETACHABLE_MODELS.get(action.resource_type)
|
||||
field_name = str(action.metadata.get("field") or "")
|
||||
if entry is None or field_name not in entry[1]:
|
||||
return _blocked(action, "The requested Files subject reference is not detachable.")
|
||||
model = entry[0]
|
||||
row = (
|
||||
session.query(model)
|
||||
.filter(getattr(model, "id") == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if row is None or getattr(row, "tenant_id", None) != tenant_id:
|
||||
return _blocked(action, "The Files resource is no longer available.")
|
||||
current = getattr(row, field_name)
|
||||
if current is None:
|
||||
return _result(
|
||||
action,
|
||||
"unchanged",
|
||||
"The subject reference was already detached.",
|
||||
{"request_id": request_id, "field": field_name},
|
||||
)
|
||||
if current != subject_user_id:
|
||||
return _blocked(action, "The Files subject reference changed after planning.")
|
||||
setattr(row, field_name, None)
|
||||
return _result(
|
||||
action,
|
||||
"executed",
|
||||
"The mutable subject reference was detached.",
|
||||
{"request_id": request_id, "field": field_name},
|
||||
)
|
||||
|
||||
|
||||
def _subject_user_id(subject: DsarSubjectRef) -> str | None:
|
||||
candidates: list[str] = []
|
||||
if subject.membership_id:
|
||||
candidates.append(subject.membership_id)
|
||||
for key, value in subject.external_references.items():
|
||||
if key in {
|
||||
"files.user",
|
||||
"files.membership",
|
||||
"access.membership",
|
||||
"membership_id",
|
||||
}:
|
||||
candidates.append(value)
|
||||
normalized = {value.strip() for value in candidates if value.strip()}
|
||||
if len(normalized) != 1:
|
||||
return None
|
||||
return normalized.pop()
|
||||
|
||||
|
||||
def _matching_fields(
|
||||
row: object,
|
||||
subject_user_id: str,
|
||||
fields: Sequence[str],
|
||||
) -> list[str]:
|
||||
return [field for field in fields if getattr(row, field) == subject_user_id]
|
||||
|
||||
|
||||
def _record(
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
category: str,
|
||||
title: str,
|
||||
data: dict[str, object],
|
||||
*,
|
||||
observed_at: datetime | None = None,
|
||||
immutable: bool = False,
|
||||
retention_reason: str | None = None,
|
||||
source_path: str | None = None,
|
||||
) -> DsarRecordRef:
|
||||
return DsarRecordRef(
|
||||
provider_id="files",
|
||||
module_id="files",
|
||||
resource_type=resource_type,
|
||||
resource_id=resource_id,
|
||||
category=category,
|
||||
title=title,
|
||||
data=data,
|
||||
observed_at=observed_at,
|
||||
immutable_evidence=immutable,
|
||||
retention_reason=retention_reason,
|
||||
source_path=source_path,
|
||||
)
|
||||
|
||||
|
||||
def _action(
|
||||
action_id: str,
|
||||
kind: str,
|
||||
record: DsarRecordRef,
|
||||
title: str,
|
||||
rationale: str,
|
||||
*,
|
||||
executable: bool,
|
||||
metadata: dict[str, object] | None = None,
|
||||
) -> DsarErasureActionRef:
|
||||
return DsarErasureActionRef(
|
||||
action_id=action_id,
|
||||
provider_id="files",
|
||||
module_id="files",
|
||||
kind=kind, # type: ignore[arg-type]
|
||||
resource_type=record.resource_type,
|
||||
resource_id=record.resource_id,
|
||||
title=title,
|
||||
rationale=rationale,
|
||||
executable=executable,
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
|
||||
def _result(
|
||||
action: DsarErasureActionRef,
|
||||
status: str,
|
||||
summary: str,
|
||||
evidence: dict[str, object] | None = None,
|
||||
) -> DsarExecutionResultRef:
|
||||
return DsarExecutionResultRef(
|
||||
action_id=action.action_id,
|
||||
status=status, # type: ignore[arg-type]
|
||||
summary=summary,
|
||||
evidence=evidence or {},
|
||||
)
|
||||
|
||||
|
||||
def _blocked(action: DsarErasureActionRef, summary: str) -> DsarExecutionResultRef:
|
||||
return _result(action, "blocked", summary)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files DSAR provider requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
def _bounded_rows(query: object) -> list[object]:
|
||||
rows = query.limit(_MAX_RECORDS + 1).all() # type: ignore[attr-defined]
|
||||
if len(rows) > _MAX_RECORDS:
|
||||
raise ValueError("Files DSAR match limit exceeded; narrow the subject selectors.")
|
||||
return rows
|
||||
|
||||
|
||||
def _has_table(session: Session, model: type[object]) -> bool:
|
||||
return inspect(session.connection()).has_table(model.__tablename__)
|
||||
|
||||
|
||||
def _aware(value: datetime | None) -> datetime | None:
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
aware = _aware(value)
|
||||
return aware.isoformat() if aware else None
|
||||
|
||||
|
||||
__all__ = ["FILES_DSAR_CAPABILITY", "FilesDsarProvider"]
|
||||
@@ -0,0 +1,449 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.access import CAPABILITY_ACCESS_DIRECTORY, AccessDirectory
|
||||
from govoplan_core.core.form_evidence import (
|
||||
FormEvidenceContractError,
|
||||
FormEvidenceGrant,
|
||||
FormEvidenceGrantRequest,
|
||||
FormEvidenceInspection,
|
||||
FormEvidenceInspectionRequest,
|
||||
FormEvidenceState,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileFormEvidenceGrant,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.runtime import configure_runtime
|
||||
|
||||
|
||||
PROVIDER_ID = "files"
|
||||
CAPABILITY_FORM_EVIDENCE_FILES = "forms_runtime.evidence.files"
|
||||
MAX_GRANT_TTL = timedelta(minutes=15)
|
||||
|
||||
|
||||
class FilesFormEvidenceProvider:
|
||||
provider_id = PROVIDER_ID
|
||||
|
||||
def __init__(self, registry: object | None, settings: object) -> None:
|
||||
self._registry = registry
|
||||
self._settings = settings
|
||||
|
||||
def supported_kinds(self) -> Sequence[str]:
|
||||
return ("document",)
|
||||
|
||||
def create_upload_grant(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: FormEvidenceGrantRequest,
|
||||
) -> FormEvidenceGrant:
|
||||
db = _session(session)
|
||||
_assert_tenant(principal, request.tenant_id)
|
||||
if request.evidence_kind != "document":
|
||||
raise FormEvidenceContractError(
|
||||
"Files can accept only document evidence for Forms Runtime."
|
||||
)
|
||||
custodian_user_id = _custodian_user_id(request.custodian_ref)
|
||||
self._assert_active_custodian(
|
||||
tenant_id=request.tenant_id,
|
||||
user_id=custodian_user_id,
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
if request.expires_at <= now:
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence upload grant expiry must be in the future."
|
||||
)
|
||||
allowed_content_types = _content_types(request.allowed_content_types)
|
||||
configured_max = int(getattr(self._settings, "file_upload_max_bytes"))
|
||||
max_size_bytes = min(request.max_size_bytes or configured_max, configured_max)
|
||||
request_sha256 = _request_sha256(
|
||||
request,
|
||||
custodian_user_id=custodian_user_id,
|
||||
max_size_bytes=max_size_bytes,
|
||||
allowed_content_types=allowed_content_types,
|
||||
)
|
||||
existing = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.idempotency_key == request.idempotency_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is not None:
|
||||
if not secrets.compare_digest(existing.request_sha256, request_sha256):
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence grant idempotency conflict."
|
||||
)
|
||||
if (
|
||||
existing.status in {"expired", "revoked"}
|
||||
or _aware(existing.expires_at) <= now
|
||||
):
|
||||
raise FormEvidenceContractError(
|
||||
"The existing Form evidence upload grant is no longer usable; "
|
||||
"request a new grant with a new idempotency key."
|
||||
)
|
||||
return _grant_response(existing, upload_token=None, replayed=True)
|
||||
|
||||
metadata = _bounded_metadata(request.metadata)
|
||||
remaining_attachments = metadata.get("remaining_attachments")
|
||||
if isinstance(remaining_attachments, int):
|
||||
existing_attachment_ids = set(metadata.get("existing_attachment_ids", ()))
|
||||
active_grants = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||
FileFormEvidenceGrant.form_definition_id
|
||||
== request.definition_ref.object_id,
|
||||
FileFormEvidenceGrant.form_definition_revision
|
||||
== str(request.definition_ref.version),
|
||||
FileFormEvidenceGrant.status.in_(("issued", "uploaded")),
|
||||
FileFormEvidenceGrant.expires_at > now,
|
||||
)
|
||||
.with_for_update()
|
||||
.all()
|
||||
)
|
||||
outstanding = sum(
|
||||
1
|
||||
for item in active_grants
|
||||
if not item.file_asset_id
|
||||
or item.file_asset_id not in existing_attachment_ids
|
||||
)
|
||||
if outstanding >= remaining_attachments:
|
||||
raise FormEvidenceContractError(
|
||||
"This Form already has the maximum number of active attachment uploads."
|
||||
)
|
||||
|
||||
upload_token = secrets.token_urlsafe(32)
|
||||
grant = FileFormEvidenceGrant(
|
||||
tenant_id=request.tenant_id,
|
||||
form_instance_id=request.instance_id,
|
||||
form_definition_id=request.definition_ref.object_id,
|
||||
form_definition_revision=str(request.definition_ref.version),
|
||||
token_sha256=_token_sha256(upload_token),
|
||||
idempotency_key=request.idempotency_key,
|
||||
request_sha256=request_sha256,
|
||||
custodian_user_id=custodian_user_id,
|
||||
evidence_kind=request.evidence_kind,
|
||||
purpose=request.purpose,
|
||||
status="issued",
|
||||
expires_at=min(request.expires_at, now + MAX_GRANT_TTL),
|
||||
max_size_bytes=max_size_bytes,
|
||||
allowed_content_types=list(allowed_content_types),
|
||||
metadata_=metadata,
|
||||
)
|
||||
db.add(grant)
|
||||
db.flush()
|
||||
return _grant_response(grant, upload_token=upload_token, replayed=False)
|
||||
|
||||
def inspect_evidence(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: FormEvidenceInspectionRequest,
|
||||
) -> FormEvidenceInspection:
|
||||
db = _session(session)
|
||||
_assert_tenant(principal, request.tenant_id)
|
||||
now = datetime.now(UTC)
|
||||
if request.evidence.owner_module != PROVIDER_ID:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The evidence owner does not match the Files provider.",
|
||||
)
|
||||
if request.evidence.kind != "document" or not request.evidence.version:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="Files evidence requires an exact document version.",
|
||||
)
|
||||
grant = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||
FileFormEvidenceGrant.form_definition_id
|
||||
== request.definition_ref.object_id,
|
||||
FileFormEvidenceGrant.form_definition_revision
|
||||
== str(request.definition_ref.version),
|
||||
FileFormEvidenceGrant.file_asset_id == request.evidence.evidence_id,
|
||||
FileFormEvidenceGrant.file_version_id == request.evidence.version,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if grant is None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The document was not captured for this exact Form submission.",
|
||||
)
|
||||
if grant.status == "revoked":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="revoked",
|
||||
observed_at=now,
|
||||
reason="The Form evidence grant was revoked.",
|
||||
)
|
||||
if grant.status != "uploaded":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="pending",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The Form evidence upload has not completed.",
|
||||
)
|
||||
asset = db.get(FileAsset, grant.file_asset_id)
|
||||
version = db.get(FileVersion, grant.file_version_id)
|
||||
blob = db.get(FileBlob, version.blob_id) if version is not None else None
|
||||
if asset is None or version is None or blob is None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="unavailable",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The managed document cannot currently be reconstructed.",
|
||||
)
|
||||
if (
|
||||
asset.tenant_id != request.tenant_id
|
||||
or version.tenant_id != request.tenant_id
|
||||
or blob.tenant_id != request.tenant_id
|
||||
or version.file_asset_id != asset.id
|
||||
):
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document crosses an evidence ownership boundary.",
|
||||
)
|
||||
if asset.deleted_at is not None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="revoked",
|
||||
observed_at=now,
|
||||
reason="The managed document is no longer active.",
|
||||
)
|
||||
if blob.quarantined_at is not None or blob.integrity_status == "quarantined":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document failed its integrity gate.",
|
||||
)
|
||||
if blob.integrity_status != "verified":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="pending",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The managed document is awaiting integrity verification.",
|
||||
)
|
||||
if (
|
||||
not request.evidence.checksum
|
||||
or not secrets.compare_digest(
|
||||
request.evidence.checksum,
|
||||
version.checksum_sha256,
|
||||
)
|
||||
or not secrets.compare_digest(
|
||||
version.checksum_sha256,
|
||||
blob.checksum_sha256,
|
||||
)
|
||||
):
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document checksum does not match the evidence.",
|
||||
)
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="accepted",
|
||||
observed_at=now,
|
||||
metadata={
|
||||
"content_type": version.content_type,
|
||||
"size_bytes": version.size_bytes,
|
||||
"integrity_status": blob.integrity_status,
|
||||
"grant_id": grant.id,
|
||||
},
|
||||
)
|
||||
|
||||
def _assert_active_custodian(self, *, tenant_id: str, user_id: str) -> None:
|
||||
registry = self._registry
|
||||
if registry is None or not hasattr(registry, "has_capability"):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory is unavailable for Form evidence custody."
|
||||
)
|
||||
if not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory is unavailable for Form evidence custody."
|
||||
)
|
||||
directory = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||
if not isinstance(directory, AccessDirectory):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory capability is invalid."
|
||||
)
|
||||
user = directory.get_user(user_id)
|
||||
if user is None or user.tenant_id != tenant_id or user.status != "active":
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence requires an active same-tenant custodian."
|
||||
)
|
||||
|
||||
|
||||
def create_files_form_evidence_provider(
|
||||
context: ModuleContext,
|
||||
) -> FilesFormEvidenceProvider:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesFormEvidenceProvider(context.registry, context.settings)
|
||||
|
||||
|
||||
def _grant_response(
|
||||
grant: FileFormEvidenceGrant,
|
||||
*,
|
||||
upload_token: str | None,
|
||||
replayed: bool,
|
||||
) -> FormEvidenceGrant:
|
||||
return FormEvidenceGrant(
|
||||
provider_id=PROVIDER_ID,
|
||||
grant_id=grant.id,
|
||||
upload_token=upload_token,
|
||||
upload_url="/api/v1/files/form-evidence/upload",
|
||||
expires_at=_aware(grant.expires_at),
|
||||
max_size_bytes=grant.max_size_bytes,
|
||||
allowed_content_types=tuple(grant.allowed_content_types),
|
||||
replayed=replayed,
|
||||
)
|
||||
|
||||
|
||||
def _inspection(
|
||||
reference: EvidenceReference,
|
||||
*,
|
||||
state: FormEvidenceState,
|
||||
observed_at: datetime,
|
||||
retryable: bool = False,
|
||||
reason: str | None = None,
|
||||
metadata: Mapping[str, object] | None = None,
|
||||
) -> FormEvidenceInspection:
|
||||
return FormEvidenceInspection(
|
||||
provider_id=PROVIDER_ID,
|
||||
reference=reference,
|
||||
state=state,
|
||||
observed_at=observed_at,
|
||||
retryable=retryable,
|
||||
reason=reason,
|
||||
metadata=dict(metadata or {}),
|
||||
)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files Form evidence requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
def _assert_tenant(principal: object, tenant_id: str) -> None:
|
||||
principal_tenant = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not principal_tenant or principal_tenant != tenant_id:
|
||||
raise PermissionError("Form evidence cannot cross tenants.")
|
||||
|
||||
|
||||
def _custodian_user_id(value: str | None) -> str:
|
||||
clean = str(value or "").strip()
|
||||
if not clean.startswith("user:") or len(clean) <= len("user:"):
|
||||
raise FormEvidenceContractError(
|
||||
"Files Form evidence requires a user custodian."
|
||||
)
|
||||
return clean.removeprefix("user:")
|
||||
|
||||
|
||||
def _content_types(values: Sequence[str]) -> tuple[str, ...]:
|
||||
cleaned = tuple(
|
||||
dict.fromkeys(
|
||||
value.split(";", 1)[0].strip().casefold()
|
||||
for value in values
|
||||
if value.strip()
|
||||
)
|
||||
)
|
||||
if len(cleaned) > 50 or any(len(value) > 255 for value in cleaned):
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence content-type restrictions are too large."
|
||||
)
|
||||
return cleaned
|
||||
|
||||
|
||||
def _bounded_metadata(value: Mapping[str, object]) -> dict[str, object]:
|
||||
remaining = value.get("remaining_attachments")
|
||||
result: dict[str, object] = {}
|
||||
if isinstance(remaining, int) and remaining >= 0:
|
||||
result["remaining_attachments"] = remaining
|
||||
raw_ids = value.get("existing_attachment_ids")
|
||||
if isinstance(raw_ids, (list, tuple)):
|
||||
clean_ids = tuple(
|
||||
dict.fromkeys(
|
||||
item.strip()
|
||||
for item in raw_ids
|
||||
if isinstance(item, str) and item.strip() and len(item.strip()) <= 255
|
||||
)
|
||||
)
|
||||
if len(clean_ids) <= 1000:
|
||||
result["existing_attachment_ids"] = clean_ids
|
||||
return result
|
||||
|
||||
|
||||
def _request_sha256(
|
||||
request: FormEvidenceGrantRequest,
|
||||
*,
|
||||
custodian_user_id: str,
|
||||
max_size_bytes: int,
|
||||
allowed_content_types: Sequence[str],
|
||||
) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
{
|
||||
"tenant_id": request.tenant_id,
|
||||
"instance_id": request.instance_id,
|
||||
"definition_ref": request.definition_ref.to_dict(),
|
||||
"evidence_kind": request.evidence_kind,
|
||||
"purpose": request.purpose,
|
||||
"expires_at": request.expires_at.isoformat(),
|
||||
"custodian_user_id": custodian_user_id,
|
||||
"max_size_bytes": max_size_bytes,
|
||||
"allowed_content_types": list(allowed_content_types),
|
||||
"metadata": _bounded_metadata(request.metadata),
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _token_sha256(token: str) -> str:
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CAPABILITY_FORM_EVIDENCE_FILES",
|
||||
"FilesFormEvidenceProvider",
|
||||
"PROVIDER_ID",
|
||||
"create_files_form_evidence_provider",
|
||||
]
|
||||
@@ -0,0 +1,132 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
from typing import Iterable
|
||||
|
||||
from govoplan_core.core.modules import DocumentationTopic
|
||||
|
||||
|
||||
_TRANSLATIONS = {
|
||||
"files.tabular-content": {
|
||||
"title": "Verwaltete CSV- und XLSX-Versionen als gesteuerte Datenquellen verwenden",
|
||||
"summary": "Exakte autorisierte Dateiversionen für Connectors bereitstellen, ohne Files-Kontrollen zu umgehen.",
|
||||
"body": (
|
||||
"Files listet nur CSV- und XLSX-Dateien auf, die für die aktuelle Person im Mandanten sichtbar sind. Das Öffnen erfordert zusätzlich die Download-Berechtigung und eine exakte unveränderliche Versionsreferenz. Größenobergrenzen werden vor dem Speicherzugriff geprüft; Prüfsumme, Quarantäne, Verschlüsselungshülle, Löschung, Eigentum und Freigaben bleiben in Files maßgeblich. Connectors erhält Metadaten und verifizierte Bytes über die Core-Fähigkeit und importiert weder Files-Modelle noch Speicherschlüssel oder Verschlüsselungsinterna. Eine neuere aktuelle Version ersetzt eine gebundene Quellversion nie still; Connectors meldet die Änderung zur ausdrücklichen Prüfung."
|
||||
),
|
||||
},
|
||||
"files.records.exact-version-source": {
|
||||
"title": "Eine exakte Dateiversion in einer eAkte ablegen",
|
||||
"summary": "Records nach aktueller Zugriffs- und Integritätsprüfung eine unveränderliche Files-Versionsreferenz übergeben.",
|
||||
"body": (
|
||||
"Wenn Records aktiv ist, stellt Files exakte Identitäten verwalteter FileVersion-Objekte über den anbieterneutralen Aktenquellenvertrag bereit. Die Ablage prüft aktiven Mandanten, aktuelle Files-Berechtigung, Eigentums- oder Freigabezugriff, unveränderliche Versionsidentität und Integrität des verwalteten Blobs. Records erhält Dateiname, Pfad, Version, Digest, Medientyp, Größe, Schutzstatus und Aufruflink; Files bleibt Eigentümer der Bytes."
|
||||
),
|
||||
},
|
||||
"files.search.managed-content": {
|
||||
"title": "Verwaltete Dateien und Ordner durchsuchen",
|
||||
"summary": "Dateinamen, logische Pfade und Beschreibungen für die berechtigungsbewusste Plattformsuche bereitstellen.",
|
||||
"body": (
|
||||
"Wenn Search installiert ist, trägt Files verwaltete Dateien und Ordner zu dessen abgeleitetem Index bei. Jedes Ergebnis ist auf den Mandanten begrenzt und prüft vor der Ausgabe aktuelles Eigentum, Gruppenmitgliedschaft, direkte Freigaben, Ablauf, Widerruf, Löschung und Files-Berechtigungen erneut. Bestätigte Datei- und Freigabeänderungen werden über die Plattform-Event-Outbox übertragen; eine Administration kann den abgeleiteten Index neu aufbauen, ohne führende Files-Daten zu verändern."
|
||||
),
|
||||
},
|
||||
"files.forms-runtime.managed-evidence": {
|
||||
"title": "Formularanhänge als verwaltete Nachweise erfassen",
|
||||
"summary": "Kurzlebige Upload-Freigaben ausstellen und verwaltete Dateiversionen an genau eine Formulareinreichung binden.",
|
||||
"body": (
|
||||
"Fordert Forms Runtime einen Dokumentnachweis an, stellt Files einer aktiven, mandantengleichen verwahrenden Person eine zweckgebundene Freigabe aus. Das Bearer-Token wird einmal angezeigt, nur als SHA-256-Digest gespeichert, läuft spätestens nach 15 Minuten ab und kann genau eine verwaltete Dateiversion erzeugen. Bei der endgültigen Einreichung werden exaktes Formular, Freigabe, Asset, Version, Prüfsumme, Löschstatus und Files-Integritätsgrenze erneut geprüft. Öffentliche Erfassung erhält dadurch keine allgemeinen Files-Berechtigungen; ein Dokument für eine Einreichung kann nicht still für eine andere wiederverwendet werden."
|
||||
),
|
||||
},
|
||||
"files.workflow.organize-managed-files": {
|
||||
"title": "Verwaltete Dateien und Ordner organisieren",
|
||||
"summary": "Ordner anlegen und zugängliche Inhalte mit ausdrücklicher Konfliktbehandlung umbenennen, verschieben oder kopieren.",
|
||||
"body": (
|
||||
"Die Organisation bleibt in gesteuerten persönlichen oder Gruppenbereichen. Verschieben erhält die Asset-Identität; Kopieren erzeugt neue Assets und Versionen, die unveränderliche Blob-Bytes wiederverwenden. Jeder Zielkonflikt muss ausdrücklich abgelehnt, durch Umbenennen gelöst, überschrieben oder übersprungen werden."
|
||||
),
|
||||
},
|
||||
"files.workflow.find-and-download-files": {
|
||||
"title": "Verwaltete Dateien finden und herunterladen",
|
||||
"summary": "Zugängliche Inhalte suchen und eine aktuelle Dateiversion oder ein ZIP einer Auswahl herunterladen.",
|
||||
"body": (
|
||||
"Files kann nach logischem Pfad oder Namensmuster sortiert und durchsucht werden. Ein Download verwendet immer die zugängliche aktuelle Version; aus einer Mehrfachauswahl kann ein temporäres ZIP-Archiv erzeugt werden. Einen eigenen Dienst zur Inhaltsvorschau gibt es derzeit nicht."
|
||||
),
|
||||
},
|
||||
"files.workflow.share-managed-files": {
|
||||
"title": "Zugriff auf verwaltete Dateien steuern",
|
||||
"summary": "Direkte Lese-, Schreib- und Verwaltungsfreigaben auflisten, vergeben, ändern, befristen oder widerrufen, ohne das Eigentum zu ändern.",
|
||||
"body": (
|
||||
"Dateieigentümer und Files-Administrationen verwalten direkte Freigaben für Personen, Gruppen, den Mandanten und Campaign. Abgelaufene oder widerrufene Freigaben berechtigen sofort nicht mehr; unabhängige aktive Freigaben bleiben wirksam. Der Freigabedialog zeigt aktive und historische Einträge, und ein Widerruf ist idempotent. Die Zugriffserklärung prüft standardmäßig die angemeldete Person. Erlaubt Policy ausgewählte Fremdprüfungen, kann derselbe zentrale Dialog eine andere aktive Person im Mandanten bewerten; diese Diagnose wird im Audit protokolliert und gewährt keinen Zugriff."
|
||||
),
|
||||
},
|
||||
"files.workflow.delete-managed-files": {
|
||||
"title": "Verwaltete Dateien und Ordner löschen",
|
||||
"summary": "Zugängliche Dateien oder einen Ordnerbaum nach aktueller Richtlinie weich löschen.",
|
||||
"body": (
|
||||
"Löschen blendet ausgewählte verwaltete Assets aus, statt gespeicherte Nachweise physisch zu vernichten. Ordner werden standardmäßig einschließlich ihrer Unterordner und Dateien rekursiv gelöscht; eine nicht rekursive Anforderung scheitert bei einem nicht leeren Ordner. Eine autorisierte Wiederherstellung aktiviert dasselbe Asset mit Versionen, Provenienz, Ordnerbaum oder Connector-Bereich erneut, sofern Pfad oder Bezeichnung frei ist. Die unwiderrufliche Vernichtung ist ein eigener Administrationsablauf mit Aufbewahrungs- und Legal-Hold-Prüfungen."
|
||||
),
|
||||
},
|
||||
"files.workflow.restore-retain-and-purge": {
|
||||
"title": "Verwaltete Dateien wiederherstellen, aufbewahren und vernichten",
|
||||
"summary": "Weich gelöschte Ressourcen wiederherstellen und irreversible Löschung mit Aufbewahrung, Legal Hold, Vorschau, Freigabe und Recovery-Nachweis steuern.",
|
||||
"body": (
|
||||
"Wiederherstellen erhält ursprüngliche Asset-Identität, Versionen, Blob-Referenzen und Connector-Provenienz. Eine Aufbewahrungsadministration kann mit Begründung und Revisionsprüfung ein Aufbewahrungsdatum oder Legal Hold setzen. Eine physische Vernichtung benötigt eigene Berechtigung, aktuellen unveränderlichen Vorschau-Hash, Idempotenzschlüssel, die wörtliche Bestätigung PURGE und eine Freigabereferenz. Aktive Aufbewahrung, Legal Hold, Freigaben sowie Campaign- und Form-Nachweise blockieren sie. Zuerst werden berechtigte Datenbankeinträge entfernt und Blobs freigegeben; eine getrennte begrenzte Garbage Collection prüft unter derselben verteilten Blob-Sperre wie Uploads alle FileVersion-Referenzen erneut, bevor Bytes und Metadaten gelöscht werden. Jede irreversible Aktion erscheint im Core-Recovery-Ledger und Audit-Nachweis."
|
||||
),
|
||||
},
|
||||
"files.privacy.data-subject-requests": {
|
||||
"title": "Files-Daten in einer Datenschutzanfrage prüfen",
|
||||
"summary": "Sichere Files-Metadaten ermitteln und Entscheidungen über Aufbewahrung, Nachweise und Byte-Löschung ausdrücklich behandeln.",
|
||||
"body": (
|
||||
"Der Files-DSAR-Anbieter sucht nur im wirksamen Mandanten und verlangt eine direkte Mitgliedschaft oder namensraumbezogene Files-Benutzerreferenz. Er exportiert begrenzte Datei-, Versions-, Ordner-, Freigabe-, Nachweis-, Connector-Konfigurations- und Integritätsmetadaten ohne Dateibytes, Speicherorte, Tokens, Passwörter, Geheimnisreferenzen oder verschlüsselte Zugangsdaten. Pläne dürfen eine aktive Freigabe für die betroffene Person widerrufen oder eine veränderbare Akteursreferenz lösen. Legal Hold, aktive Aufbewahrung, Form- und Campaign-Nachweise, Konfigurationshistorie und Integritätsnachweise bleiben mit Begründung erhalten. Dateiinhalte, Eigentum, Namen und Pfade benötigen manuelle Prüfung. Freigegebene physische Löschung muss den getrennt autorisierten Files-Vernichtungs- und Blob-Garbage-Collection-Ablauf nutzen, damit DSAR keine Nachweisblocker, Freigabe, Audit- oder Recovery-Kontrollen umgeht."
|
||||
),
|
||||
},
|
||||
"files.reference.integrity-recovery-and-fail-closed-transports": {
|
||||
"title": "Files-Integrität, Recovery und ausfallsichere Connector-Transporte betreiben",
|
||||
"summary": "Datenbanknachweise, Blob-Chiffretexte und Encryption-Verwahrung als eine Recovery-Einheit sichern und jeden SDK-verwalteten Gegenpunkt binden.",
|
||||
"body": (
|
||||
"Dauerhafter lokaler Speicher ist die betriebliche Basis. Files wird aus einem abgestimmten Datenbank-/Blob-Snapshot mit passenden Encryption-Tabellen und ursprünglichem Deployment-Hauptschlüssel wiederhergestellt; anschließend ist der begrenzte fortsetzbare Integritätsscan auszuführen und geschützter sowie ungeschützter Zugriff stichprobenartig zu prüfen. Scan- und Befundaktionen benötigen die angezeigte Revision. Geschützte Scans prüfen erst den Chiffretext, dann nach Entschlüsselung den semantischen Klartextnachweis. Unter PostgreSQL sichern lease-gebundene Core-Recovery-Absichten Objektauswirkungen ab; Abweichungen werden quarantänisiert und bleiben in Ops sichtbar. SQLite verwendet für Entwicklung eine prozesslokale Sperre und ist kein Produktions-Recovery-Profil; nach hartem Prozessverlust ist ein Integritätsscan erforderlich. Fehlende oder abweichende Blobs werden quarantänisiert, verwaiste Objekte vor einer ausdrücklich autorisierten Bereinigung zunächst nur gemeldet. Physische Vernichtung und Blob-Garbage-Collection prüfen ihre irreversiblen Wirkungen getrennt. S3-Schreibvorgänge nutzen bedingte Effekte und digestbasierte Vorwärts-Recovery; S3- und SMB-Transporte binden Wiederholungen, Umleitungen, Aliasse und erkannte Endpunkte und wenden die Richtlinie für private Netze vor jeder Verbindung erneut an. Fehlt eine verifizierbare Transportnaht, wird geschlossen abgebrochen. Destruktive Modulstilllegung entfernt Datenbanktabellen, nicht jedoch Blob-Objekte im Backend."
|
||||
),
|
||||
},
|
||||
"files.reference.snapshot-provenance-and-capabilities": {
|
||||
"title": "Snapshot-Provenienz und Files-Fähigkeiten integrieren",
|
||||
"summary": "Stabile Files-Fähigkeiten oder HTTP-Verträge verwenden und exakte Versionsnachweise bewahren, statt Files-Interna zu importieren.",
|
||||
"body": (
|
||||
"files.access erklärt Ressourcenzugriff; files.campaign_attachments friert Campaign-Eingaben auf exaktes Asset, Version, Blob, Prüfsumme und Quellrevision ein. Externe Inhalte müssen vor einer gesteuerten Nutzung importiert und ihre Provenienz auf abgeleiteten Snapshots erhalten werden. Zusammenarbeit, Provider-Synchronisierung, OAuth, externe Änderungen und fachlicher Workflow-Zustand bleiben in den jeweils verantwortlichen Modulen."
|
||||
),
|
||||
},
|
||||
"files.assurance.process-and-release-readiness": {
|
||||
"title": "Files-gestützte Prozesse und Releases absichern",
|
||||
"summary": "Einen Prozess gegen die implementierte Files-Grenze prüfen, erlaubte und abgelehnte Pfade erproben und vor Freigabe Nachweise bewahren.",
|
||||
"body": (
|
||||
"Prozessverantwortliche müssen implementierte Kontrollen von geplanten Fähigkeiten unterscheiden. Ein Release ist erst bereit, wenn Paket- und Manifestversionen übereinstimmen und repräsentative Pfade für Autorisierung, Upload-Grenzen, Konflikte, Download, Löschung, Connectoren und Recovery erprobt wurden. Wiederherstellung, Aufbewahrung, Legal Hold, gesteuerte Vernichtung, Blob-Garbage-Collection, Freigabelebenszyklus und ausdrückliches S3-Zurückschreiben erzeugen eigene Policy-, Recovery- oder Audit-Nachweise. Gewöhnliche Organisationsänderungen stützen sich weiterhin vor allem auf die Files-Änderungssequenz statt auf ein eigenes kanonisches Audit-Ereignis. Verbleibende Grenzen sind in der Prozessbewertung zu dokumentieren und Nachweisarten dürfen nicht gleichgesetzt werden."
|
||||
),
|
||||
},
|
||||
"files.reference.shared-storage-profile": {
|
||||
"title": "Files mit einem gemeinsamen Speicherprofil betreiben",
|
||||
"summary": "Lokalen, hostweit gemeinsamen oder S3-basierten Speicher passend zur Laufzeittopologie wählen.",
|
||||
"body": (
|
||||
"Core stellt das gemeinsame lokale/S3-Objektspeicher-Backend bereit; Files verantwortet Dateimetadaten und Objektschlüssel. Lokaler Speicher eignet sich für einen Laufzeitprozess, ein gemeinsames Host-Volume für Replikate auf demselben Host. Unabhängige Hosts benötigen einen ausdrücklich vertrauenswürdigen HTTPS-S3-kompatiblen Endpunkt. PostgreSQL, Objekte und Hauptschlüssel müssen auf denselben abgestimmten Recovery-Zeitpunkt wiederhergestellt werden."
|
||||
),
|
||||
},
|
||||
"files.reference.generated-artifact-store": {
|
||||
"title": "Erzeugte Modulausgaben speichern",
|
||||
"summary": "Optionalen Erzeugermodulen erlauben, Ausgaben über die Autoritätsgrenze von Files dauerhaft abzulegen.",
|
||||
"body": (
|
||||
"Die Fähigkeit files.artifact_store nimmt erzeugte Bytes mit begrenzter, nicht geheimer Provenienz entgegen, wendet Upload-Autorisierung, Eigentums-, Pfad-, Versions- und Blob-Speicherregeln von Files an und liefert anbieterneutrale Datei-/Versionsreferenzen. Idempotenz beruht auf der Quellprovenienz. Die Annahme eines Artefakts beweist weder Druck, Versand noch eine andere externe Wirkung."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def localize_documentation_topics(
|
||||
topics: Iterable[DocumentationTopic],
|
||||
) -> tuple[DocumentationTopic, ...]:
|
||||
localized: list[DocumentationTopic] = []
|
||||
for topic in topics:
|
||||
german = _TRANSLATIONS.get(topic.id)
|
||||
if german is None:
|
||||
localized.append(topic)
|
||||
continue
|
||||
translations = {
|
||||
locale: dict(value) for locale, value in topic.translations.items()
|
||||
}
|
||||
translations["de"] = {**translations.get("de", {}), **german}
|
||||
localized.append(replace(topic, translations=translations))
|
||||
return tuple(localized)
|
||||
@@ -0,0 +1,458 @@
|
||||
"""German translations for public structured documentation metadata."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'files.assurance.process-and-release-readiness': {'outcome': 'Der Prozess oder die '
|
||||
'Veröffentlichung hat eine explizite '
|
||||
'Genehmigungsaufzeichnung, die an '
|
||||
'abgestimmte Versionen, '
|
||||
'repräsentative Nachweise, bekannte '
|
||||
'Einschränkungen und eigene '
|
||||
'Restrisiken gebunden ist.',
|
||||
'prerequisites': ['Ein benannter '
|
||||
'Prozessbesitzer hat die '
|
||||
'beabsichtigten Benutzer, '
|
||||
'Datenklassifizierung, '
|
||||
'Aufbewahrungserwartungen und '
|
||||
'Integrationen definiert.',
|
||||
'Ein Kandidaten-Release wird '
|
||||
'in einer sauberen Datenbank '
|
||||
'und einer Upgrade-Kopie mit '
|
||||
'ausgerichteten Python-, '
|
||||
'Root-Paket-, WebUI-Paket- '
|
||||
'und Modulmanifestversionen '
|
||||
'installiert.',
|
||||
'Repräsentative zugelassene '
|
||||
'und verweigerte Konten, '
|
||||
'begrenzte Testdateien und '
|
||||
'ein koordiniertes '
|
||||
'Datenbank-/Blob-/Key-Wiederherstellungsset '
|
||||
'sind verfügbar.'],
|
||||
'steps': ['Vergleichen Sie die '
|
||||
'Prozessanforderungen mit der '
|
||||
'implementierten / geplanten Grenze '
|
||||
'des Handbuchs und notieren Sie jede '
|
||||
'nicht unterstützte Anforderung oder '
|
||||
'Kompensationskontrolle.',
|
||||
'Bestätigen Sie die '
|
||||
'Versionsausrichtung, wenden Sie '
|
||||
'Migrationen auf bereinigte und '
|
||||
'aktualisierte Datenbanken an und '
|
||||
'führen Sie das Repository sowie '
|
||||
'Meta-Repository-Sicherheits- und '
|
||||
'Statikanalyse-Gates aus.',
|
||||
'Übung erlaubt und verweigert '
|
||||
'persönlichen, Gruppen und teilen Sie '
|
||||
'den Zugriff mit repräsentativen '
|
||||
'Konten.',
|
||||
'Übung begrenzte Archivvorschau und '
|
||||
'Bestätigung, jede erforderliche '
|
||||
'Konfliktstrategie, Organisation, '
|
||||
'Download und weiche Löschung.',
|
||||
'Wenn Konnektoren konfiguriert sind, '
|
||||
'überprüfen Sie die '
|
||||
'Richtlinienerklärung und einen '
|
||||
'angehängten HTTP-Provider; '
|
||||
'überprüfen Sie die '
|
||||
'Installateur-eigene Garage, wenn Sie '
|
||||
'ausgewählt sind, und üben Sie dann '
|
||||
'konfigurierte S3-Retries/Aliases und '
|
||||
'SMB-Reconnect/Referral-Ziele unter '
|
||||
'der Bereitstellungsrichtlinie für '
|
||||
'das private Netzwerk aus, um zu '
|
||||
'bestätigen, dass eine inkompatible '
|
||||
'SDK-Transportnaht nicht geschlossen '
|
||||
'ist.',
|
||||
'Stellen Sie ein koordiniertes '
|
||||
'Datenbank-/Blob-/Key-Backup wieder '
|
||||
'her und vergleichen Sie '
|
||||
'repräsentative heruntergeladene '
|
||||
'Bytes mit den aufgezeichneten '
|
||||
'SHA-256-Prüfsummen.',
|
||||
'Notieren Sie die getesteten '
|
||||
'Versionen, Ergebnisse, bekannte '
|
||||
'Einschränkungen, Evidenzorte, '
|
||||
'Restrisiken, Eigentümer und '
|
||||
'Genehmigungsentscheidung.'],
|
||||
'verification': 'Ein Überprüfer kann die '
|
||||
'aufgezeichneten '
|
||||
'Berechtigungs-/Verweigerungs-, '
|
||||
'Integritäts-, Konnektor- und '
|
||||
'Wiederherstellungsüberprüfungen '
|
||||
'reproduzieren und jede nicht '
|
||||
'erfüllte Anforderung auf eine '
|
||||
'dokumentierte Einschränkung '
|
||||
'oder akzeptierte '
|
||||
'Kompensationskontrolle '
|
||||
'zurückführen.'},
|
||||
'files.forms-runtime.managed-evidence': {'limitations': ['Das erste Signaturprofil ist eine '
|
||||
'authentifizierte Bestätigung; '
|
||||
'fortgeschrittene und qualifizierte '
|
||||
'elektronische Signaturen erfordern '
|
||||
'einen separaten Anbieter.'],
|
||||
'outcome': 'Der Entwurf verweist auf eine '
|
||||
'unveränderliche verwaltete Dateiversion mit '
|
||||
'genauen Prüfsummennachweisen.',
|
||||
'prerequisites': ['Formulare Laufzeit und Dateien sind '
|
||||
'aktiviert, und das Aufnahmeprofil hat '
|
||||
'eine aktive Benutzerdepotbank für den '
|
||||
'gleichen Mandant.'],
|
||||
'steps': ['Fordern Sie eine Beweisbewilligung für '
|
||||
'Dateien von der editierbaren Formularinstanz '
|
||||
'an, laden Sie sie über die zurückgegebene '
|
||||
'einmalige Trägerroute hoch und fügen Sie dann '
|
||||
'die zurückgegebene Beweisreferenz zum Entwurf '
|
||||
'hinzu.'],
|
||||
'verification': 'Reichen Sie das Formular ein und '
|
||||
'vergewissern Sie sich, dass geänderte, '
|
||||
'gelöschte, unter Quarantäne gestellte, '
|
||||
'mieterübergreifende und '
|
||||
'gebührenübergreifende Referenzen '
|
||||
'abgelehnt werden.'},
|
||||
'files.governed-connectors-and-provenance': {'limitations': ['Die manuelle '
|
||||
'Ordner-Synchronisierung plant keine '
|
||||
'zukünftigen Runs oder mutiert, '
|
||||
'löscht, umbenennt, verschiebt oder '
|
||||
'ändert ACLs für entfernte Inhalte.',
|
||||
'Ein erfolgreicher Teildurchlauf ist '
|
||||
'keine vollständige Abdeckung, wenn '
|
||||
'die Antwort verkürzt wird oder '
|
||||
'Überprüfungsergebnisse enthält.'],
|
||||
'outcome': 'Zulässige Konnektordateien werden zu '
|
||||
'verwalteten Assets oder Versionen, '
|
||||
'während jede Ausnahme explizit und '
|
||||
'überprüfbar bleibt.',
|
||||
'prerequisites': ['Ein sichtbarer aktiver '
|
||||
'Verbindungsraum verwendet '
|
||||
'manuelle Synchronisierung und '
|
||||
'Links zu einem zugänglichen '
|
||||
'verwalteten Benutzer- oder '
|
||||
'Gruppenraum.',
|
||||
'Der Betreiber hat '
|
||||
'file:file:upload und die '
|
||||
'effektive Konnektorrichtlinie '
|
||||
'erlaubt die angeforderten Root- '
|
||||
'und Source-Pfade.'],
|
||||
'steps': ['Öffnen Sie den Ordner des verknüpften '
|
||||
'Verbinders, wählen Sie den Ordner Sync '
|
||||
'und überprüfen Sie das verwaltete Ziel.',
|
||||
'Halten Sie den sicheren Sprung '
|
||||
'standardmäßig oder wählen Sie explizit '
|
||||
'umbenennen, ablehnen oder überschreiben '
|
||||
'für nicht verwandte Zielkollisionen.',
|
||||
'Legen Sie die Auswahl der begrenzten '
|
||||
'Dateianzahl und des Unterordners fest, '
|
||||
'führen Sie die Synchronisierung aus und '
|
||||
'überprüfen Sie jedes eingegebene '
|
||||
'Elementergebnis und jede '
|
||||
'Abkürzungswarnung.',
|
||||
'Setzen Sie einen verkürzten Lauf aus '
|
||||
'einem schmaleren entfernten Ordner fort; '
|
||||
'konfigurieren Sie die Planung separat, '
|
||||
'wenn eine Hintergrundoperation '
|
||||
'erforderlich ist.'],
|
||||
'verification': 'Vergleichen Sie die '
|
||||
'Antwortzusammenfassung und die '
|
||||
'Ergebnisse pro Element mit dem '
|
||||
'Auditereignis '
|
||||
'files.connector.folder synced und '
|
||||
'prüfen Sie dann die Herkunft der '
|
||||
'erstellten, aktualisierten und '
|
||||
'unveränderten repräsentativen '
|
||||
'Assets.'},
|
||||
'files.privacy.data-subject-requests': {'limitations': ['E-Mail-, Konto- oder '
|
||||
'Identitätsselektoren allein können nicht '
|
||||
'von Dateien aufgelöst werden, da Dateien '
|
||||
'nicht im Besitz des '
|
||||
'Access-Verzeichnisses sind; geben Sie '
|
||||
'die bestätigte Mitgliedschaftsreferenz '
|
||||
'an.',
|
||||
'Der Anbieter bettet keine rohen '
|
||||
'Dateiinhalte in den JSON-Export ein und '
|
||||
'führt niemals eine physische '
|
||||
'Blob-Löschung als DSAR-Nebenwirkung '
|
||||
'durch.'],
|
||||
'outcome': 'Dateieigene Betreffreferenzen werden '
|
||||
'überprüft oder entfernt, ohne den '
|
||||
'zurückgehaltenen Inhalt oder Nachweise '
|
||||
'stillschweigend zu zerstören.',
|
||||
'prerequisites': ['Die Anfrage wurde autorisiert und '
|
||||
'enthält eine direkte '
|
||||
'Mandantenmitgliedschaft oder eine '
|
||||
'Betreffreferenz Dateien.',
|
||||
'Der Datenschutzprüfer kann den '
|
||||
'Zugriffsexport von der Löschbehörde '
|
||||
'und der Löschbehörde für Dateien '
|
||||
'unterscheiden.'],
|
||||
'steps': ['Führen Sie die Anbietersuche aus und '
|
||||
'bestätigen Sie die vollständige Abdeckung der '
|
||||
'Dateiberichte anstelle eines ausgefallenen '
|
||||
'oder abwesenden Anbieters.',
|
||||
'Überprüfen Sie Datei-/Versionsmetadaten, '
|
||||
'Gründe für die Aufbewahrung von Nachweisen und '
|
||||
'den Quellpfad für die manuelle '
|
||||
'Inhaltsüberprüfung.',
|
||||
'Generieren Sie den Löschplan und führen Sie '
|
||||
'nur die genehmigten reversiblen '
|
||||
'Aktienrücknahme- oder Subjektreferenzaktionen '
|
||||
'aus.',
|
||||
'Für die genehmigte Byte-Löschung lösen Sie '
|
||||
'jeden Lifecycle-Blocker auf und verwenden Sie '
|
||||
'Files Purge Preview, Execution und Blob '
|
||||
'Garbage Collection separat.'],
|
||||
'verification': 'Bestätigen Sie, dass jeder '
|
||||
'Dateidatensatz eine Aufbewahrungs-, '
|
||||
'Überprüfungs-, Widerrufs- oder '
|
||||
'Loslösungsdisposition hat und prüfen Sie '
|
||||
'jede separate Bereinigung durch seine '
|
||||
'Audit- und Wiederherstellungsnachweise.'},
|
||||
'files.records.exact-version-source': {'outcome': 'Records behält eine exakt geregelte Referenz, '
|
||||
'während Files Byte Authority bleibt.',
|
||||
'prerequisites': ['Die genaue Dateiversion existiert und '
|
||||
'übergibt das aktuelle Files Integrity '
|
||||
'Gate.',
|
||||
'Sie haben derzeit die Datei Lesezugriff '
|
||||
'und Records Einreichung Behörde.'],
|
||||
'steps': ['Wählen Sie die genaue verwaltete Dateiversion '
|
||||
'aus, die zum institutionellen Datensatz gehört.',
|
||||
'Wählen Sie den Zieldatensatz und geben Sie den '
|
||||
'Zugangszweck und den Anmeldegrund an.',
|
||||
'Bestätigen Sie die Einreichung; Dateien '
|
||||
'überprüft den aktuellen Zugriff erneut und löst '
|
||||
'die genaue Version auf.',
|
||||
'Öffnen Sie die Datensatzchronologie und '
|
||||
'überprüfen Sie die Versionsidentität und den '
|
||||
'SHA-256 Digest.']},
|
||||
'files.reference.integrity-recovery-and-fail-closed-transports': {'verification': 'Führen Sie '
|
||||
'nach der '
|
||||
'Wiederherstellung '
|
||||
'einen '
|
||||
'Checksummen-fähigen '
|
||||
'Integritätsscan '
|
||||
'aus, lösen Sie '
|
||||
'jedes fehlende '
|
||||
'/ korrupte '
|
||||
'Finden, '
|
||||
'genehmigen '
|
||||
'oder behalten '
|
||||
'Sie jedes '
|
||||
'gemeldete '
|
||||
'Waisenkind, '
|
||||
'inspizieren '
|
||||
'Sie die '
|
||||
'Wiederherstellungsvorgänge '
|
||||
'von Dateien in '
|
||||
'Ops, '
|
||||
'überprüfen Sie '
|
||||
'den '
|
||||
'autorisierten '
|
||||
'und '
|
||||
'verweigerten '
|
||||
'Zugriff und '
|
||||
'testen Sie '
|
||||
'konfigurierte '
|
||||
'HTTP-, S3- und '
|
||||
'SMB-Verbindungen '
|
||||
'mit ihrer '
|
||||
'aufgezeichneten '
|
||||
'Zieltopologie.'},
|
||||
'files.reference.shared-storage-profile': {'limitations': ['Installer-verwaltete Garage ist '
|
||||
'Single-Node, es sei denn, ein '
|
||||
'externer Multi-Node-Cluster wird '
|
||||
'separat betrieben.',
|
||||
'Die Anwendung erstellt oder '
|
||||
'verifiziert keine '
|
||||
'PostgreSQL/Objekt/Schlüssel-Backups.'],
|
||||
'verification': 'Führen Sie die '
|
||||
'Dateispeicher-Round-Trip-Prüfung und '
|
||||
'eine koordinierte '
|
||||
'Wiederherstellungsübung gegen die '
|
||||
'genaue Bereitstellungstopologie aus.'},
|
||||
'files.tabular-content': {'consequence_classes': {'fail_closed': 'Nicht verfügbare, '
|
||||
'überdimensionierte, unter '
|
||||
'Quarantäne gestellte oder nicht '
|
||||
'autorisierte Inhalte werden '
|
||||
'nicht analysiert oder in der '
|
||||
'Vorschau angezeigt.',
|
||||
'pin_exact_version': 'Eine Quelle behält ihre '
|
||||
'überprüfte '
|
||||
'unveränderliche '
|
||||
'Dateiversion, bis sie '
|
||||
'explizit aktualisiert '
|
||||
'wird.',
|
||||
'preserve_file_controls': 'Zugriff auf Dateien, '
|
||||
'Integrität, '
|
||||
'Verschlüsselung, '
|
||||
'Aufbewahrung und '
|
||||
'rechtliche Kontrolle '
|
||||
'bleiben '
|
||||
'autoritativ.'}},
|
||||
'files.workflow.delete-managed-files': {'limitations': ['Löschung ist eine weiche Löschung, keine '
|
||||
'harte Säuberung.',
|
||||
'Die Wiederherstellung schlägt fehl, wenn '
|
||||
'eine aktive Datei, ein Ordner oder ein '
|
||||
'Verbindungsraum bereits den Pfad oder '
|
||||
'das Label verwendet.'],
|
||||
'outcome': 'Der ausgewählte Inhalt wird aus aktiven '
|
||||
'Dateiansichten unter dem aktuellen '
|
||||
'Soft-Löschmodell ausgeblendet.',
|
||||
'prerequisites': ['Sie können die ausgewählten '
|
||||
'verwalteten Inhalte anzeigen und '
|
||||
'löschen und haben Schreib- oder '
|
||||
'Eigentümerzugriff auf jedes betroffene '
|
||||
'Asset oder jeden Ordner.',
|
||||
'Sie haben den gesamten Ordnerbaum beim '
|
||||
'rekursiven Löschen überprüft.'],
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie die Dateien '
|
||||
'oder den Ordner zum Löschen aus.',
|
||||
'Überprüfen Sie die Auswahl und, für einen '
|
||||
'Ordner, alle Inhalte darunter.',
|
||||
'Bestätigen Sie die Löschaktion.',
|
||||
'Erfrischen oder öffnen Sie den Raum und '
|
||||
'überprüfen Sie, ob die ausgewählten Pfade '
|
||||
'nicht mehr aktiv sind.'],
|
||||
'verification': 'Bestätigen Sie, dass die gelöschten '
|
||||
'Pfade nicht mehr im aktiven Raum '
|
||||
'erscheinen; behandeln Sie die Aktion '
|
||||
'nicht als physisches Löschen.'},
|
||||
'files.workflow.find-and-download-files': {'outcome': 'Die autorisierten aktuellen Dateibytes '
|
||||
'oder das generierte Archiv werden auf das '
|
||||
'lokale Gerät heruntergeladen.',
|
||||
'prerequisites': ['Sie können verwaltete Dateien im '
|
||||
'entsprechenden Bereich anzeigen und '
|
||||
'herunterladen.'],
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie den '
|
||||
'entsprechenden Personen- oder Gruppenraum '
|
||||
'aus.',
|
||||
'Navigieren Sie in Ordnern, sortieren Sie '
|
||||
'die Liste oder verwenden Sie ein Pfad- / '
|
||||
'Namensmuster, um den beabsichtigten Inhalt '
|
||||
'zu finden.',
|
||||
'Überprüfen Sie den angezeigten Besitzer, '
|
||||
'Pfad, Größe, Prüfsumme und Versionsdetails.',
|
||||
'Laden Sie eine aktuelle Dateiversion '
|
||||
'herunter oder wählen Sie mehrere Dateien '
|
||||
'aus und wählen Sie ZIP herunterladen.'],
|
||||
'verification': 'Bestätigen Sie die heruntergeladenen '
|
||||
'Namen und vergleichen Sie, wo '
|
||||
'Integrität wichtig ist, die '
|
||||
'Dateibytes mit der angezeigten '
|
||||
'Prüfsumme.'},
|
||||
'files.workflow.organize-managed-files': {'outcome': 'Der ausgewählte Inhalt hat den '
|
||||
'beabsichtigten verwalteten Eigentümer und '
|
||||
'logischen Pfad.',
|
||||
'prerequisites': ['Sie können verwaltete Dateien '
|
||||
'anzeigen und organisieren.',
|
||||
'Sie haben Schreib- oder '
|
||||
'Eigentümerzugriff auf alle von der '
|
||||
'Operation verwendeten Quellelemente '
|
||||
'und Zielbereiche; die globale '
|
||||
'Organisationsberechtigung allein '
|
||||
'gewährt keinen Ressourcenzugriff.'],
|
||||
'steps': ['Öffnen Sie Dateien und wählen Sie den '
|
||||
'persönlichen oder Gruppenraum zum '
|
||||
'Organisieren aus.',
|
||||
'Erstellen Sie die erforderlichen Zielordner '
|
||||
'oder wählen Sie die Dateien und Ordner aus, '
|
||||
'um sie umzubenennen, zu verschieben oder zu '
|
||||
'kopieren.',
|
||||
'Wählen Sie das Ziel aus und lösen Sie jeden '
|
||||
'Zielkonflikt explizit.',
|
||||
'Wenden Sie die Operation an und öffnen Sie '
|
||||
'den Zielordner erneut.'],
|
||||
'verification': 'Bestätigen Sie jeden resultierenden '
|
||||
'Pfad und Besitzer; für eine Bewegung '
|
||||
'bestätigen Sie auch, dass der alte '
|
||||
'Pfad verschwunden ist, und für eine '
|
||||
'Kopie bestätigen Sie, dass die Quelle '
|
||||
'verbleibt.'},
|
||||
'files.workflow.restore-retain-and-purge': {'limitations': ['Automatische zeitbasierte '
|
||||
'Bereinigungsplanung ist nicht '
|
||||
'enthalten; ein autorisierter '
|
||||
'handelnde Person startet Vorschau, '
|
||||
'Ausführung und Blob-Sammlung.',
|
||||
'Der Verschlüsselungsumschlag eines '
|
||||
'geschützten Blobs folgt der eigenen '
|
||||
'Aufbewahrungs- und '
|
||||
'Schlüssel-Verwahrungsrichtlinie des '
|
||||
'Verschlüsselungsmoduls.'],
|
||||
'outcome': 'Berechtigte Metadaten und nicht '
|
||||
'referenzierte Bytes werden mit separat '
|
||||
'überprüfbaren Richtlinien-, Audit- und '
|
||||
'Wiederherstellungsnachweisen gelöscht.',
|
||||
'prerequisites': ['Die handelnde Person hat die '
|
||||
'eindeutige Wiederherstellungs-, '
|
||||
'Aufbewahrungs- oder '
|
||||
'Bereinigungsberechtigung, die für '
|
||||
'die beabsichtigte Aktion '
|
||||
'erforderlich ist.',
|
||||
'Ein Bereinigungsziel ist bereits '
|
||||
'sanft gelöscht und alle abhängigen '
|
||||
'Nachweise oder Aktien wurden '
|
||||
'rechtmäßig gelöst.'],
|
||||
'steps': ['Festlegen oder Überprüfen von '
|
||||
'Aufbewahrungs- und Haltezustand unter '
|
||||
'Verwendung der angezeigten '
|
||||
'Lifecycle-Revision.',
|
||||
'Überprüfen Sie zum Wiederherstellen, ob '
|
||||
'der ursprüngliche Pfad oder das '
|
||||
'ursprüngliche Label frei ist, und stellen '
|
||||
'Sie die gleiche Ressource wieder her.',
|
||||
'Erstellen Sie zum Löschen eine begrenzte '
|
||||
'Purge-Vorschau und lösen Sie jeden '
|
||||
'benannten Blocker auf.',
|
||||
'Führen Sie die passende Vorschau mit einem '
|
||||
'idempotency-Schlüssel, PURGE-Bestätigung '
|
||||
'und Genehmigungsreferenz aus.',
|
||||
'Führen Sie eine begrenzte '
|
||||
'Blob-Garbage-Sammlung aus und prüfen Sie '
|
||||
'alle Wiederherstellungs- oder '
|
||||
'Ergebnis-unbekannten Operationen in Ops.'],
|
||||
'verification': 'Überprüfen Sie, ob die '
|
||||
'Asset-/Versionszeilen fehlen, der '
|
||||
'Blob bleibt referenziert, nicht '
|
||||
'referenzierte Bytes fehlen nach GC '
|
||||
'und jede Operation hat eine gültige '
|
||||
'Wiederherstellungsnachweiskette.'},
|
||||
'files.workflow.share-managed-files': {'limitations': ['Kampagnenziel-Zuschüsse werden '
|
||||
'normalerweise durch die '
|
||||
'Kampagnenintegration erstellt und nicht '
|
||||
'manuell im Dateidialog ausgewählt.',
|
||||
'Ein Benutzer kann den Zugriff über einen '
|
||||
'anderen aktiven direkten Zuschuss oder '
|
||||
'Besitzpfad behalten, nachdem eine Aktie '
|
||||
'widerrufen wurde.',
|
||||
'Ausgewählte benutzererklärungen erfordern '
|
||||
'die richtlinienberechtigung und sind nur '
|
||||
'diagnosen; sie geben sich nicht als '
|
||||
'benutzer aus oder gewähren dateizugriff.'],
|
||||
'outcome': 'Direkter Zugriff hat die angeforderte '
|
||||
'Berechtigung und Lebensdauer, während das '
|
||||
'verwaltete Asset seinen Eigentümer behält.',
|
||||
'prerequisites': ['Sie haben die Freigabeberechtigung für '
|
||||
'Dateien und besitzen die Datei oder '
|
||||
'verwalten Dateiräume für den aktiven '
|
||||
'Mandanten.',
|
||||
'Das beabsichtigte Benutzer-, Gruppen-, '
|
||||
'Mandant- oder Kampagnenziel existiert '
|
||||
'und ist aktiv.'],
|
||||
'steps': ['Wählen Sie eine verwaltete Datei aus, wählen '
|
||||
'Sie Shares verwalten und überprüfen Sie die '
|
||||
'effektiven und historischen direkten Zuschüsse.',
|
||||
'Wählen Sie den beabsichtigten Benutzer, die '
|
||||
'Gruppe oder den Mandanten aus und wählen Sie '
|
||||
'den Zugriff lesen, schreiben oder verwalten '
|
||||
'sowie einen optionalen Ablauf.',
|
||||
'Geben oder aktualisieren Sie die Freigabe, ohne '
|
||||
'den Dateibesitz zu ändern.',
|
||||
'Widerrufen Sie einen Zuschuss, wenn er nicht '
|
||||
'mehr benötigt wird; wiederholter Widerruf ist '
|
||||
'ein No-Op.'],
|
||||
'verification': 'Testen Sie einen beabsichtigten und einen '
|
||||
'verweigerten Pfad, verfallen oder '
|
||||
'widerrufen Sie die Gewährung und stellen '
|
||||
'Sie sicher, dass nur noch unabhängige '
|
||||
'Zugangspfade verbleiben.'}}
|
||||
File diff suppressed because it is too large
Load Diff
+119
@@ -0,0 +1,119 @@
|
||||
"""file connector spaces
|
||||
|
||||
Revision ID: 4f5a6b7c8d9e
|
||||
Revises: 2e3f4a5b6c7d
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "4f5a6b7c8d9e"
|
||||
down_revision = "2e3f4a5b6c7d"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(tables: set[str]) -> str:
|
||||
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
tables = set(inspector.get_table_names())
|
||||
scope_fk_target = _scope_fk_target(tables)
|
||||
if "file_connector_spaces" not in tables:
|
||||
op.create_table(
|
||||
"file_connector_spaces",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("owner_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("owner_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("owner_group_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("label", sa.String(length=255), nullable=False),
|
||||
sa.Column("connector_profile_id", sa.String(length=255), nullable=False),
|
||||
sa.Column("provider", sa.String(length=50), nullable=False),
|
||||
sa.Column("library_id", sa.String(length=255), nullable=True),
|
||||
sa.Column("remote_path", sa.String(length=1000), nullable=False),
|
||||
sa.Column("sync_mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("read_only", sa.Boolean(), nullable=False),
|
||||
sa.Column("is_active", sa.Boolean(), nullable=False),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_spaces_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["owner_group_id"], ["access_groups.id"], name=op.f("fk_file_connector_spaces_owner_group_id_groups"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["owner_user_id"], ["access_users.id"], name=op.f("fk_file_connector_spaces_owner_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_spaces_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_spaces")),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_spaces")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
"connector_profile_id",
|
||||
"provider",
|
||||
"is_active",
|
||||
"created_by_user_id",
|
||||
"deleted_at",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_spaces_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_spaces", [column], unique=False)
|
||||
if "ix_file_connector_spaces_owner" not in indexes:
|
||||
op.create_index(
|
||||
"ix_file_connector_spaces_owner",
|
||||
"file_connector_spaces",
|
||||
["tenant_id", "owner_type", "owner_user_id", "owner_group_id"],
|
||||
unique=False,
|
||||
)
|
||||
if "uq_file_connector_spaces_active_user_label" not in indexes:
|
||||
op.create_index(
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"file_connector_spaces",
|
||||
["tenant_id", "owner_user_id", "label"],
|
||||
unique=True,
|
||||
sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
)
|
||||
if "uq_file_connector_spaces_active_group_label" not in indexes:
|
||||
op.create_index(
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"file_connector_spaces",
|
||||
["tenant_id", "owner_group_id", "label"],
|
||||
unique=True,
|
||||
sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_spaces" not in inspector.get_table_names():
|
||||
return
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_spaces")}
|
||||
for name in (
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"ix_file_connector_spaces_owner",
|
||||
op.f("ix_file_connector_spaces_deleted_at"),
|
||||
op.f("ix_file_connector_spaces_created_by_user_id"),
|
||||
op.f("ix_file_connector_spaces_is_active"),
|
||||
op.f("ix_file_connector_spaces_provider"),
|
||||
op.f("ix_file_connector_spaces_connector_profile_id"),
|
||||
op.f("ix_file_connector_spaces_owner_group_id"),
|
||||
op.f("ix_file_connector_spaces_owner_user_id"),
|
||||
op.f("ix_file_connector_spaces_owner_type"),
|
||||
op.f("ix_file_connector_spaces_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_spaces")
|
||||
op.drop_table("file_connector_spaces")
|
||||
+94
@@ -0,0 +1,94 @@
|
||||
"""file connector profiles
|
||||
|
||||
Revision ID: 5a6b7c8d9e0f
|
||||
Revises: 4f5a6b7c8d9e
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "5a6b7c8d9e0f"
|
||||
down_revision = "4f5a6b7c8d9e"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(inspector) -> str:
|
||||
tables = set(inspector.get_table_names())
|
||||
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
scope_fk_target = _scope_fk_target(inspector)
|
||||
if "file_connector_profiles" not in inspector.get_table_names():
|
||||
op.create_table(
|
||||
"file_connector_profiles",
|
||||
sa.Column("id", sa.String(length=255), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("label", sa.String(length=255), nullable=False),
|
||||
sa.Column("provider", sa.String(length=50), nullable=False),
|
||||
sa.Column("endpoint_url", sa.String(length=1000), nullable=True),
|
||||
sa.Column("base_path", sa.String(length=1000), nullable=True),
|
||||
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||
sa.Column("credential_mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("username", sa.String(length=320), nullable=True),
|
||||
sa.Column("password_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("token_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("password_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("token_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("secret_ref", sa.String(length=1000), nullable=True),
|
||||
sa.Column("capabilities", sa.JSON(), nullable=True),
|
||||
sa.Column("policy", sa.JSON(), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_profiles_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_profiles_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_profiles_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_profiles")),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
"provider",
|
||||
"enabled",
|
||||
"created_by_user_id",
|
||||
"updated_by_user_id",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_profiles_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_profiles", [column], unique=False)
|
||||
if "ix_file_connector_profiles_scope" not in indexes:
|
||||
op.create_index("ix_file_connector_profiles_scope", "file_connector_profiles", ["scope_type", "scope_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_profiles" not in inspector.get_table_names():
|
||||
return
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
for name in (
|
||||
"ix_file_connector_profiles_scope",
|
||||
op.f("ix_file_connector_profiles_updated_by_user_id"),
|
||||
op.f("ix_file_connector_profiles_created_by_user_id"),
|
||||
op.f("ix_file_connector_profiles_enabled"),
|
||||
op.f("ix_file_connector_profiles_provider"),
|
||||
op.f("ix_file_connector_profiles_scope_id"),
|
||||
op.f("ix_file_connector_profiles_scope_type"),
|
||||
op.f("ix_file_connector_profiles_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_profiles")
|
||||
op.drop_table("file_connector_profiles")
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
"""file connector credentials
|
||||
|
||||
Revision ID: 6b7c8d9e0f1a
|
||||
Revises: 5a6b7c8d9e0f
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "6b7c8d9e0f1a"
|
||||
down_revision = "5a6b7c8d9e0f"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(table_names: set[str]) -> str:
|
||||
return "core_scopes.id" if "core_scopes" in table_names else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
table_names = set(inspector.get_table_names())
|
||||
scope_fk_target = _scope_fk_target(table_names)
|
||||
|
||||
if "file_connector_profiles" in table_names:
|
||||
columns = {item["name"] for item in inspector.get_columns("file_connector_profiles")}
|
||||
if "credential_profile_id" not in columns:
|
||||
op.add_column("file_connector_profiles", sa.Column("credential_profile_id", sa.String(length=255), nullable=True))
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
index_name = op.f("ix_file_connector_profiles_credential_profile_id")
|
||||
if index_name not in indexes:
|
||||
op.create_index(index_name, "file_connector_profiles", ["credential_profile_id"], unique=False)
|
||||
|
||||
if "file_connector_credentials" not in table_names:
|
||||
op.create_table(
|
||||
"file_connector_credentials",
|
||||
sa.Column("id", sa.String(length=255), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("label", sa.String(length=255), nullable=False),
|
||||
sa.Column("provider", sa.String(length=50), nullable=True),
|
||||
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||
sa.Column("credential_mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("username", sa.String(length=320), nullable=True),
|
||||
sa.Column("password_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("token_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("password_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("token_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("secret_ref", sa.String(length=1000), nullable=True),
|
||||
sa.Column("policy", sa.JSON(), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_credentials_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_credentials_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_credentials_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_credentials")),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_credentials")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
"provider",
|
||||
"enabled",
|
||||
"created_by_user_id",
|
||||
"updated_by_user_id",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_credentials_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_credentials", [column], unique=False)
|
||||
if "ix_file_connector_credentials_scope" not in indexes:
|
||||
op.create_index("ix_file_connector_credentials_scope", "file_connector_credentials", ["scope_type", "scope_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_credentials" in inspector.get_table_names():
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_credentials")}
|
||||
for name in (
|
||||
"ix_file_connector_credentials_scope",
|
||||
op.f("ix_file_connector_credentials_updated_by_user_id"),
|
||||
op.f("ix_file_connector_credentials_created_by_user_id"),
|
||||
op.f("ix_file_connector_credentials_enabled"),
|
||||
op.f("ix_file_connector_credentials_provider"),
|
||||
op.f("ix_file_connector_credentials_scope_id"),
|
||||
op.f("ix_file_connector_credentials_scope_type"),
|
||||
op.f("ix_file_connector_credentials_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_credentials")
|
||||
op.drop_table("file_connector_credentials")
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_profiles" not in inspector.get_table_names():
|
||||
return
|
||||
profile_indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
profile_index_name = op.f("ix_file_connector_profiles_credential_profile_id")
|
||||
if profile_index_name in profile_indexes:
|
||||
op.drop_index(profile_index_name, table_name="file_connector_profiles")
|
||||
profile_columns = {item["name"] for item in inspector.get_columns("file_connector_profiles")}
|
||||
if "credential_profile_id" in profile_columns:
|
||||
op.drop_column("file_connector_profiles", "credential_profile_id")
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
"""file connector policies
|
||||
|
||||
Revision ID: a7b8c9d0e1f3
|
||||
Revises: 6b7c8d9e0f1a
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "a7b8c9d0e1f3"
|
||||
down_revision = "6b7c8d9e0f1a"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(inspector) -> str:
|
||||
tables = set(inspector.get_table_names())
|
||||
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
scope_fk_target = _scope_fk_target(inspector)
|
||||
if "file_connector_policies" not in inspector.get_table_names():
|
||||
op.create_table(
|
||||
"file_connector_policies",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("policy", sa.JSON(), nullable=False),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_policies_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_policies_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_policies_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_policies")),
|
||||
sa.UniqueConstraint("tenant_id", "scope_type", "scope_id", name="uq_file_connector_policies_scope"),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_policies")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
"created_by_user_id",
|
||||
"updated_by_user_id",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_policies_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_policies", [column], unique=False)
|
||||
if "ix_file_connector_policies_scope" not in indexes:
|
||||
op.create_index("ix_file_connector_policies_scope", "file_connector_policies", ["scope_type", "scope_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_policies" not in inspector.get_table_names():
|
||||
return
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_policies")}
|
||||
for name in (
|
||||
"ix_file_connector_policies_scope",
|
||||
op.f("ix_file_connector_policies_updated_by_user_id"),
|
||||
op.f("ix_file_connector_policies_created_by_user_id"),
|
||||
op.f("ix_file_connector_policies_scope_id"),
|
||||
op.f("ix_file_connector_policies_scope_type"),
|
||||
op.f("ix_file_connector_policies_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_policies")
|
||||
op.drop_table("file_connector_policies")
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
"""add purpose-bound Form evidence upload grants
|
||||
|
||||
Revision ID: a2b3c4d5e6f8
|
||||
Revises: f1a2b3c4d5e7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "a2b3c4d5e6f8"
|
||||
down_revision = "f1a2b3c4d5e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"file_form_evidence_grants",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("form_instance_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("form_definition_id", sa.String(length=255), nullable=False),
|
||||
sa.Column("form_definition_revision", sa.String(length=255), nullable=False),
|
||||
sa.Column("token_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||
sa.Column("request_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("custodian_user_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("evidence_kind", sa.String(length=30), nullable=False),
|
||||
sa.Column("purpose", sa.String(length=500), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("max_size_bytes", sa.Integer(), nullable=False),
|
||||
sa.Column("allowed_content_types", sa.JSON(), nullable=False),
|
||||
sa.Column("file_asset_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("file_version_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("uploaded_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["custodian_user_id"],
|
||||
["access_users.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["file_asset_id"],
|
||||
["file_assets.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["file_version_id"],
|
||||
["file_versions.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_file_form_evidence_grants_idempotency",
|
||||
),
|
||||
sa.UniqueConstraint("token_sha256"),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_tenant_id",
|
||||
"file_form_evidence_grants",
|
||||
["tenant_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_form_instance_id",
|
||||
"file_form_evidence_grants",
|
||||
["form_instance_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_custodian_user_id",
|
||||
"file_form_evidence_grants",
|
||||
["custodian_user_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_status",
|
||||
"file_form_evidence_grants",
|
||||
["status"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_expires_at",
|
||||
"file_form_evidence_grants",
|
||||
["expires_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_file_asset_id",
|
||||
"file_form_evidence_grants",
|
||||
["file_asset_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_file_version_id",
|
||||
"file_form_evidence_grants",
|
||||
["file_version_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
"file_form_evidence_grants",
|
||||
[
|
||||
"tenant_id",
|
||||
"form_instance_id",
|
||||
"form_definition_id",
|
||||
"form_definition_revision",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_file_version_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_file_asset_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_expires_at",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_status",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_custodian_user_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_form_instance_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_tenant_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_table("file_form_evidence_grants")
|
||||
@@ -0,0 +1,52 @@
|
||||
"""govern managed file retention and legal holds
|
||||
|
||||
Revision ID: a2b3c4d5e6f9
|
||||
Revises: f1a2b3c4d5e7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "a2b3c4d5e6f9"
|
||||
down_revision = "f1a2b3c4d5e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_assets") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("retained_until", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"legal_hold", sa.Boolean(), nullable=False, server_default=sa.false()
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"lifecycle_revision", sa.Integer(), nullable=False, server_default="1"
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("lifecycle_reason", sa.String(length=500), nullable=True)
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_assets_retained_until"), ["retained_until"], unique=False
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_assets_legal_hold"), ["legal_hold"], unique=False
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_assets") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_assets_legal_hold"))
|
||||
batch_op.drop_index(op.f("ix_file_assets_retained_until"))
|
||||
batch_op.drop_column("lifecycle_reason")
|
||||
batch_op.drop_column("lifecycle_revision")
|
||||
batch_op.drop_column("legal_hold")
|
||||
batch_op.drop_column("retained_until")
|
||||
@@ -0,0 +1,158 @@
|
||||
"""v0.1.7 files baseline
|
||||
|
||||
Revision ID: a7b8c9d0e1f3
|
||||
Revises: None
|
||||
Create Date: 2026-07-11 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = 'a7b8c9d0e1f3'
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = '4f2a9c8e7b6d'
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table('file_connector_credentials',
|
||||
sa.Column('id', sa.String(length=255), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('label', sa.String(length=255), nullable=False),
|
||||
sa.Column('provider', sa.String(length=50), nullable=True),
|
||||
sa.Column('enabled', sa.Boolean(), nullable=False),
|
||||
sa.Column('credential_mode', sa.String(length=30), nullable=False),
|
||||
sa.Column('username', sa.String(length=320), nullable=True),
|
||||
sa.Column('password_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('token_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('password_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('token_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
|
||||
sa.Column('policy', sa.JSON(), nullable=True),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_credentials_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_credentials'))
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_credentials_created_by_user_id'), 'file_connector_credentials', ['created_by_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_enabled'), 'file_connector_credentials', ['enabled'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_provider'), 'file_connector_credentials', ['provider'], unique=False)
|
||||
op.create_index('ix_file_connector_credentials_scope', 'file_connector_credentials', ['scope_type', 'scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_scope_id'), 'file_connector_credentials', ['scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_scope_type'), 'file_connector_credentials', ['scope_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_tenant_id'), 'file_connector_credentials', ['tenant_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_updated_by_user_id'), 'file_connector_credentials', ['updated_by_user_id'], unique=False)
|
||||
op.create_table('file_connector_policies',
|
||||
sa.Column('id', sa.String(length=36), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('policy', sa.JSON(), nullable=False),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_policies_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_policies')),
|
||||
sa.UniqueConstraint('tenant_id', 'scope_type', 'scope_id', name='uq_file_connector_policies_scope')
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_policies_created_by_user_id'), 'file_connector_policies', ['created_by_user_id'], unique=False)
|
||||
op.create_index('ix_file_connector_policies_scope', 'file_connector_policies', ['scope_type', 'scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_scope_id'), 'file_connector_policies', ['scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_scope_type'), 'file_connector_policies', ['scope_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_tenant_id'), 'file_connector_policies', ['tenant_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_updated_by_user_id'), 'file_connector_policies', ['updated_by_user_id'], unique=False)
|
||||
op.create_table('file_connector_profiles',
|
||||
sa.Column('id', sa.String(length=255), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('label', sa.String(length=255), nullable=False),
|
||||
sa.Column('provider', sa.String(length=50), nullable=False),
|
||||
sa.Column('endpoint_url', sa.String(length=1000), nullable=True),
|
||||
sa.Column('base_path', sa.String(length=1000), nullable=True),
|
||||
sa.Column('enabled', sa.Boolean(), nullable=False),
|
||||
sa.Column('credential_profile_id', sa.String(length=255), nullable=True),
|
||||
sa.Column('credential_mode', sa.String(length=30), nullable=False),
|
||||
sa.Column('username', sa.String(length=320), nullable=True),
|
||||
sa.Column('password_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('token_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('password_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('token_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
|
||||
sa.Column('capabilities', sa.JSON(), nullable=True),
|
||||
sa.Column('policy', sa.JSON(), nullable=True),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_profiles_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_profiles'))
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_profiles_created_by_user_id'), 'file_connector_profiles', ['created_by_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_credential_profile_id'), 'file_connector_profiles', ['credential_profile_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_enabled'), 'file_connector_profiles', ['enabled'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_provider'), 'file_connector_profiles', ['provider'], unique=False)
|
||||
op.create_index('ix_file_connector_profiles_scope', 'file_connector_profiles', ['scope_type', 'scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_scope_id'), 'file_connector_profiles', ['scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_scope_type'), 'file_connector_profiles', ['scope_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_tenant_id'), 'file_connector_profiles', ['tenant_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_updated_by_user_id'), 'file_connector_profiles', ['updated_by_user_id'], unique=False)
|
||||
op.create_table('file_connector_spaces',
|
||||
sa.Column('id', sa.String(length=36), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=False),
|
||||
sa.Column('owner_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('owner_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('owner_group_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('label', sa.String(length=255), nullable=False),
|
||||
sa.Column('connector_profile_id', sa.String(length=255), nullable=False),
|
||||
sa.Column('provider', sa.String(length=50), nullable=False),
|
||||
sa.Column('library_id', sa.String(length=255), nullable=True),
|
||||
sa.Column('remote_path', sa.String(length=1000), nullable=False),
|
||||
sa.Column('sync_mode', sa.String(length=30), nullable=False),
|
||||
sa.Column('read_only', sa.Boolean(), nullable=False),
|
||||
sa.Column('is_active', sa.Boolean(), nullable=False),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('deleted_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['owner_group_id'], ['access_groups.id'], name=op.f('fk_file_connector_spaces_owner_group_id_access_groups'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_owner_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_spaces_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_spaces'))
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_spaces_connector_profile_id'), 'file_connector_spaces', ['connector_profile_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_created_by_user_id'), 'file_connector_spaces', ['created_by_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_deleted_at'), 'file_connector_spaces', ['deleted_at'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_is_active'), 'file_connector_spaces', ['is_active'], unique=False)
|
||||
op.create_index('ix_file_connector_spaces_owner', 'file_connector_spaces', ['tenant_id', 'owner_type', 'owner_user_id', 'owner_group_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_owner_group_id'), 'file_connector_spaces', ['owner_group_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_owner_type'), 'file_connector_spaces', ['owner_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_owner_user_id'), 'file_connector_spaces', ['owner_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_provider'), 'file_connector_spaces', ['provider'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_tenant_id'), 'file_connector_spaces', ['tenant_id'], unique=False)
|
||||
op.create_index('uq_file_connector_spaces_active_group_label', 'file_connector_spaces', ['tenant_id', 'owner_group_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"))
|
||||
op.create_index('uq_file_connector_spaces_active_user_label', 'file_connector_spaces', ['tenant_id', 'owner_user_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table('file_connector_spaces')
|
||||
op.drop_table('file_connector_profiles')
|
||||
op.drop_table('file_connector_policies')
|
||||
op.drop_table('file_connector_credentials')
|
||||
@@ -0,0 +1,53 @@
|
||||
"""file share lifecycle
|
||||
|
||||
Revision ID: b8c9d0e1f2a4
|
||||
Revises: a7b8c9d0e1f3
|
||||
Create Date: 2026-07-30 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "b8c9d0e1f2a4"
|
||||
down_revision = "a7b8c9d0e1f3"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_shares") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("revoked_by_user_id", sa.String(length=36), nullable=True)
|
||||
)
|
||||
batch_op.create_foreign_key(
|
||||
op.f("fk_file_shares_revoked_by_user_id_access_users"),
|
||||
"access_users",
|
||||
["revoked_by_user_id"],
|
||||
["id"],
|
||||
ondelete="SET NULL",
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_shares_expires_at"), ["expires_at"], unique=False
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_shares_revoked_by_user_id"),
|
||||
["revoked_by_user_id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_shares") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_shares_revoked_by_user_id"))
|
||||
batch_op.drop_index(op.f("ix_file_shares_expires_at"))
|
||||
batch_op.drop_constraint(
|
||||
op.f("fk_file_shares_revoked_by_user_id_access_users"),
|
||||
type_="foreignkey",
|
||||
)
|
||||
batch_op.drop_column("revoked_by_user_id")
|
||||
batch_op.drop_column("expires_at")
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
"""file integrity reconciliation
|
||||
|
||||
Revision ID: c9d0e1f2a3b5
|
||||
Revises: b8c9d0e1f2a4
|
||||
Create Date: 2026-07-30 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "c9d0e1f2a3b5"
|
||||
down_revision = "b8c9d0e1f2a4"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"integrity_status",
|
||||
sa.String(length=30),
|
||||
nullable=False,
|
||||
server_default="unchecked",
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("integrity_checked_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("integrity_failure", sa.String(length=100), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_integrity_status"),
|
||||
["integrity_status"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_integrity_checked_at"),
|
||||
["integrity_checked_at"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_quarantined_at"),
|
||||
["quarantined_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"file_integrity_scans",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("storage_backend", sa.String(length=50), nullable=False),
|
||||
sa.Column("storage_prefix", sa.String(length=1000), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("phase", sa.String(length=30), nullable=False),
|
||||
sa.Column("verify_checksums", sa.Boolean(), nullable=False),
|
||||
sa.Column("batch_size", sa.Integer(), nullable=False),
|
||||
sa.Column("blob_cursor", sa.String(length=36), nullable=True),
|
||||
sa.Column("object_cursor", sa.String(length=1000), nullable=True),
|
||||
sa.Column("scanned_blob_count", sa.Integer(), nullable=False),
|
||||
sa.Column("verified_blob_count", sa.Integer(), nullable=False),
|
||||
sa.Column("quarantined_blob_count", sa.Integer(), nullable=False),
|
||||
sa.Column("scanned_object_count", sa.Integer(), nullable=False),
|
||||
sa.Column("orphan_object_count", sa.Integer(), nullable=False),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_error", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["created_by_user_id"],
|
||||
["access_users.id"],
|
||||
name=op.f(
|
||||
"fk_file_integrity_scans_created_by_user_id_access_users"
|
||||
),
|
||||
ondelete="SET NULL",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_scans")),
|
||||
)
|
||||
for column in ("tenant_id", "status", "created_by_user_id"):
|
||||
op.create_index(
|
||||
op.f(f"ix_file_integrity_scans_{column}"),
|
||||
"file_integrity_scans",
|
||||
[column],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"file_integrity_findings",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("scan_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("kind", sa.String(length=40), nullable=False),
|
||||
sa.Column("state", sa.String(length=30), nullable=False),
|
||||
sa.Column("blob_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("storage_key", sa.String(length=1000), nullable=False),
|
||||
sa.Column("expected_size_bytes", sa.Integer(), nullable=True),
|
||||
sa.Column("observed_size_bytes", sa.Integer(), nullable=True),
|
||||
sa.Column("expected_checksum_sha256", sa.String(length=64), nullable=True),
|
||||
sa.Column("observed_checksum_sha256", sa.String(length=64), nullable=True),
|
||||
sa.Column("resolved_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("resolved_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["blob_id"],
|
||||
["file_blobs.id"],
|
||||
name=op.f("fk_file_integrity_findings_blob_id_file_blobs"),
|
||||
ondelete="SET NULL",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["resolved_by_user_id"],
|
||||
["access_users.id"],
|
||||
name=op.f(
|
||||
"fk_file_integrity_findings_resolved_by_user_id_access_users"
|
||||
),
|
||||
ondelete="SET NULL",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["scan_id"],
|
||||
["file_integrity_scans.id"],
|
||||
name=op.f(
|
||||
"fk_file_integrity_findings_scan_id_file_integrity_scans"
|
||||
),
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_findings")),
|
||||
)
|
||||
for column in (
|
||||
"scan_id",
|
||||
"tenant_id",
|
||||
"kind",
|
||||
"state",
|
||||
"blob_id",
|
||||
"resolved_by_user_id",
|
||||
):
|
||||
op.create_index(
|
||||
op.f(f"ix_file_integrity_findings_{column}"),
|
||||
"file_integrity_findings",
|
||||
[column],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_integrity_findings_scan_state",
|
||||
"file_integrity_findings",
|
||||
["scan_id", "state"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("file_integrity_findings")
|
||||
op.drop_table("file_integrity_scans")
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_blobs_quarantined_at"))
|
||||
batch_op.drop_index(op.f("ix_file_blobs_integrity_checked_at"))
|
||||
batch_op.drop_index(op.f("ix_file_blobs_integrity_status"))
|
||||
batch_op.drop_column("quarantined_at")
|
||||
batch_op.drop_column("integrity_failure")
|
||||
batch_op.drop_column("integrity_checked_at")
|
||||
batch_op.drop_column("integrity_status")
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
"""file content-protection metadata
|
||||
|
||||
Revision ID: d0e1f2a3b4c6
|
||||
Revises: c9d0e1f2a3b5
|
||||
Create Date: 2026-08-02 00:00:00.000000
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "d0e1f2a3b4c6"
|
||||
down_revision = "c9d0e1f2a3b5"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"protection_discriminator",
|
||||
sa.String(length=320),
|
||||
nullable=False,
|
||||
server_default="plaintext",
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("encryption_envelope_id", sa.String(length=255), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("storage_checksum_sha256", sa.String(length=64), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("storage_size_bytes", sa.Integer(), nullable=True)
|
||||
)
|
||||
batch_op.drop_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size",
|
||||
type_="unique",
|
||||
)
|
||||
batch_op.create_unique_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size_protection",
|
||||
[
|
||||
"tenant_id",
|
||||
"checksum_sha256",
|
||||
"size_bytes",
|
||||
"protection_discriminator",
|
||||
],
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_protection_discriminator"),
|
||||
["protection_discriminator"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_encryption_envelope_id"),
|
||||
["encryption_envelope_id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_blobs_encryption_envelope_id"))
|
||||
batch_op.drop_index(op.f("ix_file_blobs_protection_discriminator"))
|
||||
batch_op.drop_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size_protection",
|
||||
type_="unique",
|
||||
)
|
||||
batch_op.create_unique_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size",
|
||||
["tenant_id", "checksum_sha256", "size_bytes"],
|
||||
)
|
||||
batch_op.drop_column("storage_size_bytes")
|
||||
batch_op.drop_column("storage_checksum_sha256")
|
||||
batch_op.drop_column("encryption_envelope_id")
|
||||
batch_op.drop_column("protection_discriminator")
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
"""add stale-action revisions to Files integrity operations
|
||||
|
||||
Revision ID: f1a2b3c4d5e7
|
||||
Revises: d0e1f2a3b4c6
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "f1a2b3c4d5e7"
|
||||
down_revision = "d0e1f2a3b4c6"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_integrity_scans") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("revision", sa.Integer(), nullable=False, server_default="1")
|
||||
)
|
||||
with op.batch_alter_table("file_integrity_findings") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("revision", sa.Integer(), nullable=False, server_default="1")
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_integrity_findings") as batch_op:
|
||||
batch_op.drop_column("revision")
|
||||
with op.batch_alter_table("file_integrity_scans") as batch_op:
|
||||
batch_op.drop_column("revision")
|
||||
@@ -0,0 +1,67 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import secrets
|
||||
from uuid import uuid4
|
||||
|
||||
from govoplan_core.core.operations import OperationalCheck
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
|
||||
|
||||
def managed_storage_roundtrip_check() -> OperationalCheck:
|
||||
"""Exercise the configured managed store without retaining probe data."""
|
||||
|
||||
backend = get_storage_backend()
|
||||
key = f".govoplan-health/probes/{uuid4().hex}.bin"
|
||||
payload = secrets.token_bytes(64)
|
||||
expected_digest = hashlib.sha256(payload).hexdigest()
|
||||
delete_error: Exception | None = None
|
||||
try:
|
||||
backend.put_bytes(key, payload, content_type="application/octet-stream")
|
||||
stored = backend.get_bytes(key)
|
||||
info = backend.stat(key)
|
||||
if info.size_bytes != len(payload):
|
||||
raise StorageBackendError("Managed storage returned an unexpected object size")
|
||||
if hashlib.sha256(stored).hexdigest() != expected_digest:
|
||||
raise StorageBackendError("Managed storage returned different bytes than were written")
|
||||
except Exception as exc: # noqa: BLE001 - operational boundary reports provider failures.
|
||||
return OperationalCheck(
|
||||
id="files.managed_storage_roundtrip",
|
||||
label="Managed file storage",
|
||||
state="error",
|
||||
detail=(
|
||||
"The configured managed file store failed a bounded write/read/stat/delete "
|
||||
f"probe ({type(exc).__name__})."
|
||||
),
|
||||
readiness_critical=True,
|
||||
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
backend.delete(key)
|
||||
except Exception as exc: # noqa: BLE001 - reported below when the data probe passed.
|
||||
delete_error = exc
|
||||
|
||||
if delete_error is not None:
|
||||
return OperationalCheck(
|
||||
id="files.managed_storage_roundtrip",
|
||||
label="Managed file storage",
|
||||
state="error",
|
||||
detail=(
|
||||
"Managed file bytes round-tripped, but probe cleanup failed "
|
||||
f"({type(delete_error).__name__})."
|
||||
),
|
||||
readiness_critical=True,
|
||||
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||
)
|
||||
return OperationalCheck(
|
||||
id="files.managed_storage_roundtrip",
|
||||
label="Managed file storage",
|
||||
state="ok",
|
||||
detail="The configured managed file store passed write, read, stat, integrity, and delete checks.",
|
||||
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||
)
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.provider_governance import (
|
||||
ExternalProviderRuntimeState,
|
||||
ExternalProviderStateContext,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileConnectorProfile, FileConnectorSpace
|
||||
from govoplan_files.backend.storage.connector_providers import (
|
||||
ConnectorProviderDescriptor,
|
||||
connector_provider_descriptors,
|
||||
)
|
||||
|
||||
|
||||
REMOTE_STORAGE_PROVIDER_ID = "files.remote_storage"
|
||||
|
||||
|
||||
def remote_storage_provider_states(
|
||||
context: ExternalProviderStateContext,
|
||||
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||
if not isinstance(context.session, Session):
|
||||
raise RuntimeError("Files provider state requires a database session.")
|
||||
statement = select(FileConnectorProfile)
|
||||
if context.tenant_id is not None:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
FileConnectorProfile.tenant_id.is_(None),
|
||||
FileConnectorProfile.tenant_id == context.tenant_id,
|
||||
)
|
||||
)
|
||||
profiles = tuple(
|
||||
context.session.scalars(
|
||||
statement.order_by(
|
||||
FileConnectorProfile.tenant_id,
|
||||
FileConnectorProfile.id,
|
||||
).limit(context.max_items + 1)
|
||||
)
|
||||
)
|
||||
if not profiles:
|
||||
return ()
|
||||
|
||||
profile_ids = tuple(item.id for item in profiles)
|
||||
space_statement = select(FileConnectorSpace).where(
|
||||
FileConnectorSpace.connector_profile_id.in_(profile_ids),
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
)
|
||||
if context.tenant_id is not None:
|
||||
space_statement = space_statement.where(
|
||||
FileConnectorSpace.tenant_id == context.tenant_id
|
||||
)
|
||||
spaces_by_profile: dict[str, list[FileConnectorSpace]] = defaultdict(list)
|
||||
for space in context.session.scalars(space_statement):
|
||||
spaces_by_profile[space.connector_profile_id].append(space)
|
||||
|
||||
descriptors = {
|
||||
item.provider: item for item in connector_provider_descriptors()
|
||||
}
|
||||
observed_at = datetime.now(UTC)
|
||||
return tuple(
|
||||
_profile_state(
|
||||
profile,
|
||||
spaces=spaces_by_profile.get(profile.id, []),
|
||||
descriptor=descriptors.get(profile.provider),
|
||||
observed_at=observed_at,
|
||||
)
|
||||
for profile in profiles
|
||||
)
|
||||
|
||||
|
||||
def _profile_state(
|
||||
profile: FileConnectorProfile,
|
||||
*,
|
||||
spaces: list[FileConnectorSpace],
|
||||
descriptor: ConnectorProviderDescriptor | None,
|
||||
observed_at: datetime,
|
||||
) -> ExternalProviderRuntimeState:
|
||||
active_spaces = tuple(item for item in spaces if item.is_active)
|
||||
active = bool(profile.enabled)
|
||||
implementation_ready = bool(
|
||||
descriptor is not None and descriptor.implemented and descriptor.installed
|
||||
)
|
||||
health = (
|
||||
"inactive"
|
||||
if not active
|
||||
else "error"
|
||||
if not implementation_ready
|
||||
else "unknown"
|
||||
)
|
||||
recovery = (
|
||||
"not_applicable"
|
||||
if not active
|
||||
else "unsupported"
|
||||
if not implementation_ready
|
||||
else "attention"
|
||||
)
|
||||
detail = (
|
||||
"Remote-storage connector profile is disabled."
|
||||
if not active
|
||||
else "The configured provider is not available in this runtime."
|
||||
if not implementation_ready
|
||||
else "Software support is available; no live remote health observation is retained."
|
||||
)
|
||||
return ExternalProviderRuntimeState(
|
||||
provider_id=REMOTE_STORAGE_PROVIDER_ID,
|
||||
binding_ref=f"files:connector-profile:{profile.id}",
|
||||
authority_mode="external_mirror",
|
||||
observed_at=observed_at,
|
||||
configured=True,
|
||||
active=active,
|
||||
health=health,
|
||||
freshness="unknown" if active else "not_applicable",
|
||||
conflict="not_applicable",
|
||||
recovery=recovery,
|
||||
detail=detail,
|
||||
metrics={
|
||||
"provider": profile.provider,
|
||||
"configured_spaces": len(spaces),
|
||||
"active_spaces": len(active_spaces),
|
||||
"write_requested_spaces": sum(
|
||||
1 for item in active_spaces if not item.read_only
|
||||
),
|
||||
"software_implemented": bool(descriptor and descriptor.implemented),
|
||||
"optional_dependency_available": bool(descriptor and descriptor.installed),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["REMOTE_STORAGE_PROVIDER_ID", "remote_storage_provider_states"]
|
||||
@@ -0,0 +1,113 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from urllib.parse import quote
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.records import (
|
||||
RecordContractError,
|
||||
RecordSourceLocator,
|
||||
RecordSourceReference,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileBlob, FileVersion
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import get_asset_for_user
|
||||
|
||||
|
||||
CAPABILITY_RECORD_SOURCE_FILES = "records.source.files"
|
||||
|
||||
|
||||
class FilesRecordSource:
|
||||
provider_id = "files"
|
||||
|
||||
def resource_types(self) -> Sequence[str]:
|
||||
return ("file_version",)
|
||||
|
||||
def resolve(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
locator: RecordSourceLocator,
|
||||
purpose: str,
|
||||
) -> RecordSourceReference:
|
||||
if not isinstance(session, Session):
|
||||
raise RecordContractError(
|
||||
"Files record references require a database session."
|
||||
)
|
||||
tenant_id = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not tenant_id or locator.tenant_id != tenant_id:
|
||||
raise RecordContractError("Files record references cannot cross tenants.")
|
||||
if locator.source_module != "files" or locator.resource_type != "file_version":
|
||||
raise RecordContractError("Unsupported Files record source type.")
|
||||
if not str(purpose or "").strip():
|
||||
raise RecordContractError("Files record references require a purpose.")
|
||||
if not hasattr(principal, "has") or not (
|
||||
principal.has("files:file:read") or principal.has("files:file:admin")
|
||||
):
|
||||
raise RecordContractError("Current Files read permission is required.")
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(
|
||||
getattr(user, "id", "") or getattr(principal, "membership_id", "") or ""
|
||||
).strip()
|
||||
if not user_id:
|
||||
raise RecordContractError(
|
||||
"Files record references require a tenant user principal."
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=locator.resource_id,
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise RecordContractError(str(exc)) from exc
|
||||
version_query = session.query(FileVersion).filter(
|
||||
FileVersion.tenant_id == tenant_id,
|
||||
FileVersion.file_asset_id == asset.id,
|
||||
)
|
||||
revision = locator.source_revision.strip()
|
||||
version = version_query.filter(FileVersion.id == revision).one_or_none()
|
||||
if version is None:
|
||||
raise RecordContractError("The exact file version does not exist.")
|
||||
blob = session.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != tenant_id:
|
||||
raise RecordContractError(
|
||||
"The exact file version has no managed content object."
|
||||
)
|
||||
if blob.quarantined_at is not None or blob.integrity_status == "failed":
|
||||
raise RecordContractError(
|
||||
"The exact file version failed the current integrity gate."
|
||||
)
|
||||
return RecordSourceReference(
|
||||
locator=locator,
|
||||
label=version.filename_at_upload,
|
||||
authority_mode="external_authoritative",
|
||||
content_sha256=version.checksum_sha256,
|
||||
content_type=version.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
recorded_at=version.created_at,
|
||||
launch_url=(
|
||||
f"/files?fileId={quote(asset.id, safe='')}&versionId={quote(version.id, safe='')}"
|
||||
),
|
||||
metadata={
|
||||
"display_path": version.display_path_at_upload,
|
||||
"version_number": version.version_number,
|
||||
"integrity_status": blob.integrity_status,
|
||||
"protection": blob.protection_discriminator,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def create_files_record_source(_context: object) -> FilesRecordSource:
|
||||
return FilesRecordSource()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CAPABILITY_RECORD_SOURCE_FILES",
|
||||
"FilesRecordSource",
|
||||
"create_files_record_source",
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,722 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from io import BytesIO
|
||||
from typing import Literal
|
||||
from urllib.parse import quote
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, UploadFile, status
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
from fastapi import APIRouter
|
||||
|
||||
from govoplan_core.auth.dependencies import ApiPrincipal, has_scope, require_scope
|
||||
from govoplan_core.core.optional import reraise_unless_missing_package
|
||||
from govoplan_files.backend.schemas import (
|
||||
ArchiveRequest,
|
||||
BulkFileShareRequest,
|
||||
BulkFileShareResponse,
|
||||
BulkDeleteRequest,
|
||||
BulkDeleteResponse,
|
||||
ConflictResolutionRequest,
|
||||
FileAssetResponse,
|
||||
FileFolderCreateRequest,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderDeleteResponse,
|
||||
FileFolderResponse,
|
||||
FileFoldersResponse,
|
||||
FileListResponse,
|
||||
FileShareRequest,
|
||||
FileShareResponse,
|
||||
FileSpaceResponse,
|
||||
FileSpacesResponse,
|
||||
FileUploadResponse,
|
||||
PatternMatchResponse,
|
||||
PatternResolveRequest,
|
||||
PatternResolveResponse,
|
||||
RenamePreviewItem,
|
||||
RenameRequest,
|
||||
RenameResponse,
|
||||
TransferRequest,
|
||||
TransferResponse,
|
||||
_conflict_resolutions,
|
||||
from govoplan_files.backend.routes.assets import router as assets_router
|
||||
from govoplan_files.backend.routes.connector_io import router as connector_io_router
|
||||
from govoplan_files.backend.routes.connector_profiles import (
|
||||
router as connector_profiles_router,
|
||||
)
|
||||
from govoplan_core.db.models import Group, UserGroupMembership
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError, filename_from_path, normalize_logical_path
|
||||
from govoplan_files.backend.storage.access import ensure_group_access, user_group_ids
|
||||
from govoplan_files.backend.storage.archives import create_zip_file, extract_zip_upload
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
asset_is_audit_relevant,
|
||||
create_file_asset,
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
list_assets_for_user,
|
||||
read_asset_bytes,
|
||||
share_file,
|
||||
share_files,
|
||||
soft_delete_assets,
|
||||
from govoplan_files.backend.routes.connector_settings import (
|
||||
router as connector_settings_router,
|
||||
)
|
||||
from govoplan_files.backend.storage.folders import create_folder, list_folders_for_user, soft_delete_folder
|
||||
from govoplan_files.backend.storage.search import resolve_patterns
|
||||
from govoplan_files.backend.storage.transfers import rename_selection, transfer_selection
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
from govoplan_files.backend.routes.folders import router as folders_router
|
||||
from govoplan_files.backend.routes.form_evidence import router as form_evidence_router
|
||||
from govoplan_files.backend.routes.integrity import router as integrity_router
|
||||
from govoplan_files.backend.routes.lifecycle import router as lifecycle_router
|
||||
from govoplan_files.backend.routes.listing import router as listing_router
|
||||
from govoplan_files.backend.routes.shares import router as shares_router
|
||||
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||
|
||||
|
||||
def _campaign_models():
|
||||
try:
|
||||
from govoplan_campaign.backend.db.models import Campaign, CampaignShare
|
||||
except ModuleNotFoundError as exc:
|
||||
reraise_unless_missing_package(exc, "govoplan_campaign")
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Campaign module is not installed") from exc
|
||||
return Campaign, CampaignShare
|
||||
|
||||
|
||||
def _is_admin(principal: ApiPrincipal) -> bool:
|
||||
return has_scope(principal, "files:file:admin")
|
||||
|
||||
|
||||
async def _read_limited_upload(upload: UploadFile, *, max_bytes: int) -> bytes:
|
||||
data = await upload.read(max_bytes + 1)
|
||||
if len(data) > max_bytes:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
|
||||
detail=f"Upload exceeds limit of {max_bytes} bytes",
|
||||
)
|
||||
return data
|
||||
|
||||
|
||||
|
||||
|
||||
def _cleanup_temp_file(path: str) -> None:
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def _attachment_disposition(filename: str) -> str:
|
||||
safe = filename.replace("\\", "_").replace("/", "_").replace("\r", "_").replace("\n", "_").strip() or "download"
|
||||
ascii_name = "".join(
|
||||
char if 32 <= ord(char) < 127 and char not in {'"', "\\", ";"} else "_"
|
||||
for char in safe
|
||||
).strip() or "download"
|
||||
encoded = quote(safe, safe="")
|
||||
return f'attachment; filename="{ascii_name}"; filename*=UTF-8' + "''" + encoded
|
||||
|
||||
|
||||
def _ensure_campaign_file_access(session: Session, principal: ApiPrincipal, campaign_id: str | None) -> None:
|
||||
if not campaign_id:
|
||||
return
|
||||
if not has_scope(principal, "campaigns:campaign:read"):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Missing scope: campaign:read")
|
||||
Campaign, CampaignShare = _campaign_models()
|
||||
campaign = session.get(Campaign, campaign_id)
|
||||
if not campaign or campaign.tenant_id != principal.tenant_id:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Campaign not found")
|
||||
if has_scope(principal, "tenant:*"):
|
||||
return
|
||||
if campaign.owner_user_id == principal.user.id:
|
||||
return
|
||||
group_ids = {
|
||||
row[0]
|
||||
for row in session.query(UserGroupMembership.group_id)
|
||||
.filter(UserGroupMembership.tenant_id == principal.tenant_id, UserGroupMembership.user_id == principal.user.id)
|
||||
.all()
|
||||
}
|
||||
if campaign.owner_group_id and campaign.owner_group_id in group_ids:
|
||||
return
|
||||
share = (
|
||||
session.query(CampaignShare)
|
||||
.filter(
|
||||
CampaignShare.tenant_id == principal.tenant_id,
|
||||
CampaignShare.campaign_id == campaign.id,
|
||||
CampaignShare.revoked_at.is_(None),
|
||||
or_(
|
||||
(CampaignShare.target_type == "user") & (CampaignShare.target_id == principal.user.id),
|
||||
(CampaignShare.target_type == "group") & (CampaignShare.target_id.in_(group_ids)),
|
||||
),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if share is None:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No access to this campaign")
|
||||
|
||||
|
||||
def _http_error(exc: Exception, *, not_found: bool = False) -> HTTPException:
|
||||
code = status.HTTP_404_NOT_FOUND if not_found else status.HTTP_400_BAD_REQUEST
|
||||
return HTTPException(status_code=code, detail=str(exc))
|
||||
|
||||
|
||||
def _owner_id(asset: FileAsset) -> str:
|
||||
return asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id # type: ignore[return-value]
|
||||
|
||||
|
||||
def _asset_response(session: Session, asset: FileAsset, *, include_shares: bool = False) -> FileAssetResponse:
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
shares: list[FileShareResponse] = []
|
||||
if include_shares:
|
||||
rows = session.query(FileShare).filter(FileShare.file_asset_id == asset.id).order_by(FileShare.created_at.desc()).all()
|
||||
shares = [
|
||||
FileShareResponse(
|
||||
id=row.id,
|
||||
target_type=row.target_type,
|
||||
target_id=row.target_id,
|
||||
permission=row.permission,
|
||||
created_at=row.created_at.isoformat(),
|
||||
revoked_at=row.revoked_at.isoformat() if row.revoked_at else None,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
return FileAssetResponse(
|
||||
id=asset.id,
|
||||
tenant_id=asset.tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_owner_id(asset),
|
||||
display_path=asset.display_path,
|
||||
filename=asset.filename,
|
||||
description=asset.description,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
version_id=version.id,
|
||||
created_at=asset.created_at.isoformat(),
|
||||
updated_at=asset.updated_at.isoformat(),
|
||||
deleted_at=asset.deleted_at.isoformat() if asset.deleted_at else None,
|
||||
audit_relevant=asset_is_audit_relevant(session, asset),
|
||||
metadata=asset.metadata_ or {},
|
||||
shares=shares,
|
||||
)
|
||||
|
||||
|
||||
def _folder_owner_id(folder: FileFolder) -> str:
|
||||
return folder.owner_user_id if folder.owner_type == "user" else folder.owner_group_id # type: ignore[return-value]
|
||||
|
||||
|
||||
def _folder_response(folder: FileFolder) -> FileFolderResponse:
|
||||
return FileFolderResponse(
|
||||
id=folder.id,
|
||||
tenant_id=folder.tenant_id,
|
||||
owner_type=folder.owner_type,
|
||||
owner_id=_folder_owner_id(folder),
|
||||
path=folder.path,
|
||||
created_at=folder.created_at.isoformat(),
|
||||
updated_at=folder.updated_at.isoformat(),
|
||||
deleted_at=folder.deleted_at.isoformat() if folder.deleted_at else None,
|
||||
)
|
||||
|
||||
|
||||
def _ensure_list_owner_access(session: Session, principal: ApiPrincipal, owner_type: str | None, owner_id: str | None) -> None:
|
||||
if not owner_type:
|
||||
return
|
||||
if owner_type == "user" and owner_id and owner_id != principal.user.id and not _is_admin(principal):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No access to this user file space")
|
||||
if owner_type == "group" and owner_id:
|
||||
try:
|
||||
ensure_group_access(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
group_id=owner_id,
|
||||
user_id=principal.user.id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/spaces", response_model=FileSpacesResponse)
|
||||
def list_file_spaces(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
router = APIRouter()
|
||||
for workflow_router in (
|
||||
spaces_router,
|
||||
folders_router,
|
||||
form_evidence_router,
|
||||
integrity_router,
|
||||
lifecycle_router,
|
||||
listing_router,
|
||||
uploads_router,
|
||||
connector_settings_router,
|
||||
connector_io_router,
|
||||
connector_profiles_router,
|
||||
assets_router,
|
||||
shares_router,
|
||||
transfers_router,
|
||||
):
|
||||
spaces = [
|
||||
FileSpaceResponse(
|
||||
id=f"user:{principal.user.id}",
|
||||
label="My files",
|
||||
owner_type="user",
|
||||
owner_id=principal.user.id,
|
||||
description="Files owned by your user account.",
|
||||
)
|
||||
]
|
||||
group_ids = user_group_ids(session, tenant_id=principal.tenant_id, user_id=principal.user.id, include_admin_groups=_is_admin(principal))
|
||||
if group_ids:
|
||||
groups = session.query(Group).filter(Group.tenant_id == principal.tenant_id, Group.id.in_(group_ids)).order_by(Group.name.asc()).all()
|
||||
spaces.extend(
|
||||
FileSpaceResponse(
|
||||
id=f"group:{group.id}",
|
||||
label=f"{group.name} files",
|
||||
owner_type="group",
|
||||
owner_id=group.id,
|
||||
description="Files owned by this group.",
|
||||
)
|
||||
for group in groups
|
||||
)
|
||||
return FileSpacesResponse(spaces=spaces)
|
||||
|
||||
|
||||
@router.get("/folders", response_model=FileFoldersResponse)
|
||||
def list_file_folders(
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
folders = list_folders_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileFoldersResponse(folders=[_folder_response(folder) for folder in folders])
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders", response_model=FileFolderResponse)
|
||||
def create_file_folder(
|
||||
payload: FileFolderCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
folder = create_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return _folder_response(folder)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
|
||||
def delete_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
deleted_folders, deleted_files = soft_delete_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderDeleteResponse(deleted_folders=deleted_folders, deleted_files=deleted_files)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get("", response_model=FileListResponse)
|
||||
def list_files(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(files=[_asset_response(session, asset, include_shares=True) for asset in assets])
|
||||
|
||||
|
||||
@router.post("/upload", response_model=FileUploadResponse)
|
||||
async def upload_files(
|
||||
files: list[UploadFile] = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
unpack_zip: bool = Form(default=False),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(default="reject"),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
uploaded_assets: list[FileAsset] = []
|
||||
try:
|
||||
raw_resolutions = json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
upload_resolutions = _conflict_resolutions([ConflictResolutionRequest(**item) for item in raw_resolutions])
|
||||
for upload in files:
|
||||
filename = upload.filename or "file"
|
||||
content_type = upload.content_type or None
|
||||
upload_limit = settings.file_upload_zip_max_bytes if unpack_zip and filename.lower().endswith(".zip") else settings.file_upload_max_bytes
|
||||
data = await _read_limited_upload(upload, max_bytes=upload_limit)
|
||||
if unpack_zip and filename.lower().endswith(".zip"):
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=data,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
is_admin=_is_admin(principal),
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
uploaded_assets.extend(item.asset for item in extracted)
|
||||
continue
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder=path,
|
||||
content_type=content_type,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
uploaded_assets.append(stored.asset)
|
||||
session.commit()
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(files=[_asset_response(session, asset, include_shares=True) for asset in uploaded_assets])
|
||||
|
||||
|
||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||
async def upload_zip(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(default="reject"),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
data = await _read_limited_upload(file, max_bytes=settings.file_upload_zip_max_bytes)
|
||||
target_owner = owner_id or principal.user.id
|
||||
try:
|
||||
raw_resolutions = json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
upload_resolutions = _conflict_resolutions([ConflictResolutionRequest(**item) for item in raw_resolutions])
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=data,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
is_admin=_is_admin(principal),
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
session.commit()
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(files=[_asset_response(session, item.asset, include_shares=True) for item in extracted])
|
||||
|
||||
|
||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||
def get_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.get("/{file_id}/download")
|
||||
def download_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
data, _, blob = read_asset_bytes(session, asset)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
|
||||
return StreamingResponse(BytesIO(data), media_type=blob.content_type or "application/octet-stream", headers=headers)
|
||||
|
||||
|
||||
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
||||
def delete_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
count = soft_delete_assets(session, [asset])
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
|
||||
def bulk_delete_files(
|
||||
payload: BulkDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
count = soft_delete_assets(session, assets)
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/{file_id}/shares", response_model=FileShareResponse)
|
||||
def create_share(
|
||||
file_id: str,
|
||||
payload: FileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
share = share_file(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset=asset,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
)
|
||||
session.commit()
|
||||
return FileShareResponse(
|
||||
id=share.id,
|
||||
target_type=share.target_type,
|
||||
target_id=share.target_id,
|
||||
permission=share.permission,
|
||||
created_at=share.created_at.isoformat(),
|
||||
revoked_at=share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-shares", response_model=BulkFileShareResponse)
|
||||
def create_bulk_shares(
|
||||
payload: BulkFileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
file_ids = list(dict.fromkeys(payload.file_ids))
|
||||
assets = [
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
for file_id in file_ids
|
||||
]
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
assets=assets,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
)
|
||||
session.commit()
|
||||
return BulkFileShareResponse(
|
||||
shared_count=len(shares),
|
||||
shares=[
|
||||
FileShareResponse(
|
||||
id=share.id,
|
||||
target_type=share.target_type,
|
||||
target_id=share.target_id,
|
||||
permission=share.permission,
|
||||
created_at=share.created_at.isoformat(),
|
||||
revoked_at=share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
)
|
||||
for share in shares
|
||||
],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
|
||||
@router.post("/bulk-rename", response_model=RenameResponse)
|
||||
def bulk_rename(
|
||||
payload: RenameRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
plan = rename_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
mode=payload.mode,
|
||||
new_name=payload.new_name,
|
||||
find=payload.find,
|
||||
replacement=payload.replacement,
|
||||
prefix=payload.prefix,
|
||||
suffix=payload.suffix,
|
||||
recursive=payload.recursive,
|
||||
dry_run=payload.dry_run,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if not payload.dry_run:
|
||||
session.commit()
|
||||
return RenameResponse(
|
||||
dry_run=payload.dry_run,
|
||||
items=[
|
||||
RenamePreviewItem(
|
||||
kind=item.kind,
|
||||
id=item.id,
|
||||
file_id=item.id if item.kind == "file" else None,
|
||||
folder_path=item.old_path if item.kind == "folder" else None,
|
||||
old_path=item.old_path,
|
||||
new_path=item.new_path,
|
||||
)
|
||||
for item in plan
|
||||
],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/transfer", response_model=TransferResponse)
|
||||
def transfer_files(
|
||||
payload: TransferRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
files, folders = transfer_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
operation=payload.operation,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
source_owner_type=payload.source_owner_type,
|
||||
source_owner_id=payload.source_owner_id,
|
||||
target_owner_type=payload.target_owner_type,
|
||||
target_owner_id=payload.target_owner_id,
|
||||
target_folder=payload.target_folder,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return TransferResponse(operation=payload.operation, files=files, folders=folders)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/archive.zip")
|
||||
def download_archive(
|
||||
payload: ArchiveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="multimailer-files-", suffix=".zip", delete=False)
|
||||
tmp_path = tmp.name
|
||||
tmp.close()
|
||||
try:
|
||||
create_zip_file(session, assets, tmp_path)
|
||||
except Exception:
|
||||
_cleanup_temp_file(tmp_path)
|
||||
raise
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
filename = filename_from_path(normalize_logical_path(payload.filename, fallback_filename="files.zip"))
|
||||
headers = {"Content-Disposition": _attachment_disposition(filename)}
|
||||
return FileResponse(tmp_path, media_type="application/zip", headers=headers, background=BackgroundTask(_cleanup_temp_file, tmp_path))
|
||||
|
||||
|
||||
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
|
||||
def resolve_file_patterns(
|
||||
payload: PatternResolveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
try:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
campaign_id=payload.campaign_id,
|
||||
path_prefix=payload.path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
resolved, unmatched = resolve_patterns(assets, payload.patterns, base_path=payload.path_prefix, case_sensitive=payload.case_sensitive)
|
||||
return PatternResolveResponse(
|
||||
patterns=[PatternMatchResponse(pattern=item.pattern, matches=[_asset_response(session, asset) for asset in item.matches]) for item in resolved],
|
||||
unmatched=[_asset_response(session, asset) for asset in unmatched] if payload.include_unmatched else [],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
router.routes.extend(workflow_router.routes)
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Focused HTTP route modules for the Files API."""
|
||||
@@ -0,0 +1,175 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from io import BytesIO
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi.responses import StreamingResponse
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
BulkDeleteRequest,
|
||||
BulkDeleteResponse,
|
||||
FileAssetResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
read_asset_version_bytes,
|
||||
soft_delete_assets,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_attachment_disposition,
|
||||
_audit_connector_event,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||
def get_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.get("/{file_id}/download")
|
||||
def download_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
_audit_connector_event(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.accessed",
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
operation="download",
|
||||
commit=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
|
||||
return StreamingResponse(
|
||||
BytesIO(data),
|
||||
media_type=blob.content_type or "application/octet-stream",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{file_id}/versions/{version_id}/download")
|
||||
def download_file_version(
|
||||
file_id: str,
|
||||
version_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_version_bytes(session, asset, version_id)
|
||||
_audit_connector_event(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.accessed",
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
operation="download-version",
|
||||
commit=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {
|
||||
"Content-Disposition": _attachment_disposition(
|
||||
version.filename_at_upload or asset.filename
|
||||
)
|
||||
}
|
||||
return StreamingResponse(
|
||||
BytesIO(data),
|
||||
media_type=blob.content_type or "application/octet-stream",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
||||
def delete_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
count = soft_delete_assets(session, [asset])
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
|
||||
def bulk_delete_files(
|
||||
payload: BulkDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
count = soft_delete_assets(session, assets)
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,631 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorBrowseItem,
|
||||
FileConnectorBrowseResponse,
|
||||
FileConnectorFolderSyncItemResponse,
|
||||
FileConnectorFolderSyncRequest,
|
||||
FileConnectorFolderSyncResponse,
|
||||
FileConnectorFolderSyncSummary,
|
||||
FileConnectorImportRequest,
|
||||
FileConnectorSyncResponse,
|
||||
FileConnectorWriteRequest,
|
||||
FileConnectorWriteResponse,
|
||||
FileUploadResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_imports import (
|
||||
ConnectorImportError,
|
||||
ConnectorImportUnsupported,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_folder_sync import (
|
||||
connector_relative_path,
|
||||
discover_connector_folder,
|
||||
join_connector_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
ConnectorPolicyDenied,
|
||||
connector_policy_decision,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
sync_file_asset_from_source,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_spaces import (
|
||||
connector_space_owner_id,
|
||||
get_connector_space_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_writes import write_connector_file
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_audit_connector_imports,
|
||||
_audit_connector_sync,
|
||||
_connector_browse_next_token,
|
||||
_connector_policy_error,
|
||||
_connector_space_policy_decision,
|
||||
_download_connector_payload,
|
||||
_ensure_campaign_file_access,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
_visible_connector_profile,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/sync",
|
||||
response_model=FileConnectorFolderSyncResponse,
|
||||
)
|
||||
def sync_connector_space_folder(
|
||||
space_id: str,
|
||||
payload: FileConnectorFolderSyncRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
try:
|
||||
if space.sync_mode != "manual":
|
||||
raise FileStorageError("This connector space is not configured for manual sync")
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
remote_path = join_connector_path(space.remote_path, payload.path)
|
||||
target_folder = join_connector_path(payload.target_folder)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=space.library_id,
|
||||
remote_path=remote_path,
|
||||
operation="sync",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
discovery = discover_connector_folder(
|
||||
profile,
|
||||
path=remote_path,
|
||||
library_id=space.library_id,
|
||||
recursive=payload.recursive,
|
||||
max_files=payload.max_files,
|
||||
max_depth=payload.max_depth,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (ConnectorBrowseError, FileStorageError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
owner_id = connector_space_owner_id(space)
|
||||
items: list[FileConnectorFolderSyncItemResponse] = [
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=skipped.path,
|
||||
action="skipped",
|
||||
detail=skipped.reason,
|
||||
)
|
||||
for skipped in discovery.skipped
|
||||
]
|
||||
counts = {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"skipped": len(discovery.skipped),
|
||||
"conflicts": 0,
|
||||
"policy_denied": 0,
|
||||
"failed": 0,
|
||||
}
|
||||
|
||||
for source in discovery.files:
|
||||
source_path = normalize_connector_browse_path(source.path)
|
||||
try:
|
||||
relative_path = connector_relative_path(
|
||||
space_root=space.remote_path,
|
||||
item_path=source_path,
|
||||
)
|
||||
target_path = join_connector_path(target_folder, relative_path)
|
||||
except (ConnectorBrowseError, ValueError) as exc:
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
action="failed",
|
||||
detail=str(exc),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
try:
|
||||
with session.begin_nested():
|
||||
file_payload = FileConnectorImportRequest(
|
||||
library_id=space.library_id or "",
|
||||
path=source_path,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=owner_id,
|
||||
target_path=target_path,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
source_revision=source.etag,
|
||||
metadata={
|
||||
**dict(source.metadata),
|
||||
**payload.metadata,
|
||||
"connector_space_id": space.id,
|
||||
"browse_name": source.name,
|
||||
"browse_path": source_path,
|
||||
"browse_modified_at": source.modified_at,
|
||||
"browse_etag": source.etag,
|
||||
"folder_sync": True,
|
||||
},
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, file_payload, operation="sync"
|
||||
)
|
||||
stored, sync_action, previous_version_id = (
|
||||
sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
display_path=target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
)
|
||||
_audit_connector_sync(
|
||||
session,
|
||||
principal,
|
||||
stored.asset,
|
||||
sync_action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
)
|
||||
file_response = _asset_response(
|
||||
session, stored.asset, include_shares=True
|
||||
)
|
||||
counts[sync_action] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action=sync_action,
|
||||
file=file_response,
|
||||
previous_version_id=previous_version_id,
|
||||
current_version_id=stored.version.id,
|
||||
source_revision=file_response.source_revision,
|
||||
)
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
counts["policy_denied"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action="policy_denied",
|
||||
source_revision=source.etag,
|
||||
detail=str(exc),
|
||||
policy_decision=exc.decision.to_dict(),
|
||||
)
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
detail = str(exc)
|
||||
if detail.startswith("Skipped upload target:"):
|
||||
action = "skipped"
|
||||
counts["skipped"] += 1
|
||||
elif detail.startswith("Target file already exists:"):
|
||||
action = "conflict"
|
||||
counts["conflicts"] += 1
|
||||
else:
|
||||
action = "failed"
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action=action,
|
||||
source_revision=source.etag,
|
||||
detail=detail,
|
||||
)
|
||||
)
|
||||
except (
|
||||
ConnectorImportError,
|
||||
UnsafeFilePathError,
|
||||
OSError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action="failed",
|
||||
source_revision=source.etag,
|
||||
detail=str(exc),
|
||||
)
|
||||
)
|
||||
|
||||
summary = FileConnectorFolderSyncSummary(
|
||||
discovered=len(discovery.files),
|
||||
**counts,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.folder_synced",
|
||||
object_type="file_connector_space",
|
||||
object_id=space.id,
|
||||
details={
|
||||
"connector_profile_id": profile.id,
|
||||
"provider": profile.provider,
|
||||
"library_id": space.library_id,
|
||||
"remote_path": remote_path,
|
||||
"target_folder": target_folder,
|
||||
"recursive": payload.recursive,
|
||||
"truncated": discovery.truncated,
|
||||
"summary": summary.model_dump(),
|
||||
"results": [
|
||||
{
|
||||
"source_path": item.source_path,
|
||||
"target_path": item.target_path,
|
||||
"action": item.action,
|
||||
"file_id": item.file.id if item.file else None,
|
||||
"current_version_id": item.current_version_id,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorFolderSyncResponse(
|
||||
connector_space_id=space.id,
|
||||
connector_profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
remote_path=remote_path,
|
||||
target_folder=target_folder,
|
||||
recursive=payload.recursive,
|
||||
truncated=discovery.truncated,
|
||||
summary=summary,
|
||||
items=items,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/write-back",
|
||||
response_model=FileConnectorWriteResponse,
|
||||
)
|
||||
def write_back_connector_file(
|
||||
space_id: str,
|
||||
payload: FileConnectorWriteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:connector:write")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if space.read_only:
|
||||
raise FileStorageError("This connector space is read-only")
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
requested_path = normalize_connector_browse_path(payload.remote_path)
|
||||
remote_path = normalize_connector_browse_path(
|
||||
"/".join(part for part in (space.remote_path, requested_path) if part)
|
||||
)
|
||||
decision = connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path=remote_path,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation="write",
|
||||
),
|
||||
profile.policy_sources,
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=payload.file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
connector_library_id = space.library_id
|
||||
content_type = blob.content_type
|
||||
# Close the read snapshot before the independent recovery transaction
|
||||
# records authority for the external effect. This avoids upgrading an
|
||||
# older SQLite read snapshot after the ledger commit.
|
||||
session.commit()
|
||||
result = write_connector_file(
|
||||
profile,
|
||||
tenant_id=principal.tenant_id,
|
||||
library_id=connector_library_id,
|
||||
remote_path=remote_path,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
expected_revision=payload.expected_revision,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.written",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={
|
||||
"file_version_id": version.id,
|
||||
"file_blob_id": blob.id,
|
||||
"checksum_sha256": blob.checksum_sha256,
|
||||
"connector_space_id": space.id,
|
||||
"connector_profile_id": profile.id,
|
||||
"remote_path": remote_path,
|
||||
"recovery_operation_id": result.recovery_operation_id,
|
||||
"recovery_status": result.status,
|
||||
"revision": result.revision,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorWriteResponse(
|
||||
recovery_operation_id=result.recovery_operation_id,
|
||||
status=result.status,
|
||||
replayed=result.replayed,
|
||||
provider=result.provider,
|
||||
remote_path=result.remote_path,
|
||||
revision=result.revision,
|
||||
checksum_sha256=result.checksum_sha256,
|
||||
size_bytes=result.size_bytes,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ConnectorBrowseError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles/{profile_id}/import", response_model=FileUploadResponse
|
||||
)
|
||||
def import_connector_file(
|
||||
profile_id: str,
|
||||
payload: FileConnectorImportRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
if payload.campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, profile_id, campaign_id=payload.campaign_id
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, payload, operation="import"
|
||||
)
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
folder=payload.target_folder,
|
||||
display_path=payload.target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
campaign_id=payload.campaign_id,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_audit_connector_imports(session, principal, [stored.asset])
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorImportUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (
|
||||
ConnectorImportError,
|
||||
FileStorageError,
|
||||
UnsafeFilePathError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(
|
||||
files=[_asset_response(session, stored.asset, include_shares=True)]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles/{profile_id}/sync", response_model=FileConnectorSyncResponse
|
||||
)
|
||||
def sync_connector_file(
|
||||
profile_id: str,
|
||||
payload: FileConnectorImportRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
if payload.campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, profile_id, campaign_id=payload.campaign_id
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, payload, operation="sync"
|
||||
)
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
stored, sync_action, previous_version_id = sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
folder=payload.target_folder,
|
||||
display_path=payload.target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
campaign_id=payload.campaign_id,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_audit_connector_sync(
|
||||
session,
|
||||
principal,
|
||||
stored.asset,
|
||||
sync_action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
)
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorImportUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (
|
||||
ConnectorImportError,
|
||||
FileStorageError,
|
||||
UnsafeFilePathError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorSyncResponse(
|
||||
file=_asset_response(session, stored.asset, include_shares=True),
|
||||
action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
current_version_id=stored.version.id,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/profiles/{profile_id}/browse",
|
||||
response_model=FileConnectorBrowseResponse,
|
||||
)
|
||||
def browse_connector_profile_items(
|
||||
profile_id: str,
|
||||
path: str | None = None,
|
||||
library_id: str | None = None,
|
||||
continuation_token: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
"system:settings:read",
|
||||
"admin:settings:read",
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, profile_id, campaign_id=campaign_id
|
||||
)
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
decision = connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path=browse_path,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation="browse",
|
||||
),
|
||||
profile.policy_sources,
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
items = browse_connector_profile(
|
||||
profile,
|
||||
path=browse_path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (ConnectorBrowseError, OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorBrowseResponse(
|
||||
profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
path=browse_path,
|
||||
library_id=library_id,
|
||||
next_continuation_token=_connector_browse_next_token(items),
|
||||
has_more=any(bool(item.metadata.get("listing_truncated")) for item in items),
|
||||
decision=decision.to_dict(),
|
||||
items=[FileConnectorBrowseItem(**item.to_response()) for item in items],
|
||||
)
|
||||
@@ -0,0 +1,261 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorProfileResponse,
|
||||
FileConnectorProfileUpdateRequest,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||
delete_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
connector_profile_from_row,
|
||||
get_connector_profile_row,
|
||||
update_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
_can_read_disabled_connector_profiles,
|
||||
_connector_policy_error,
|
||||
_credential_row_for_profile,
|
||||
_ensure_connector_configuration_allowed,
|
||||
_ensure_connector_local_policy_allowed,
|
||||
_http_error,
|
||||
_record_connector_settings_change,
|
||||
_require_connector_profile_write,
|
||||
_visible_connector_profile,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||
)
|
||||
def get_connector_profile(
|
||||
profile_id: str,
|
||||
campaign_id: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
"system:settings:read",
|
||||
"admin:settings:read",
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
profile = _visible_connector_profile(
|
||||
session,
|
||||
principal,
|
||||
profile_id,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorProfileResponse(**profile.to_response())
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||
)
|
||||
def update_connector_profile(
|
||||
profile_id: str,
|
||||
payload: FileConnectorProfileUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_profile_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
profile_id=profile_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_profile_write(principal, row.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
credential_profile_id = (
|
||||
payload.credential_profile_id
|
||||
if payload.credential_profile_id is not None
|
||||
else row.credential_profile_id
|
||||
)
|
||||
provider = payload.provider if payload.provider is not None else row.provider
|
||||
credential_row = _credential_row_for_profile(
|
||||
session,
|
||||
principal,
|
||||
credential_profile_id=credential_profile_id,
|
||||
provider=provider,
|
||||
profile_id=row.id,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=row.id,
|
||||
credential_id=credential_profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=connector_effective_endpoint_url(
|
||||
provider=provider,
|
||||
endpoint_url=payload.endpoint_url
|
||||
if payload.endpoint_url is not None
|
||||
else row.endpoint_url,
|
||||
metadata=payload.metadata
|
||||
if payload.metadata is not None
|
||||
else row.metadata_,
|
||||
),
|
||||
base_path=payload.base_path
|
||||
if payload.base_path is not None
|
||||
else row.base_path,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
operation="configure",
|
||||
)
|
||||
if payload.policy is not None:
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
update_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
enabled=payload.enabled,
|
||||
credential_profile_id=payload.credential_profile_id,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username if credentials else None,
|
||||
password=credentials.password if credentials else None,
|
||||
token=credentials.token if credentials else None,
|
||||
password_env=credentials.password_env if credentials else None,
|
||||
token_env=credentials.token_env if credentials else None,
|
||||
secret_ref=credentials.secret_ref if credentials else None,
|
||||
clear_password=payload.clear_password,
|
||||
clear_token=payload.clear_token,
|
||||
capabilities=payload.capabilities,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return FileConnectorProfileResponse(
|
||||
**connector_profile_from_row(
|
||||
row, credential_row=credential_row
|
||||
).to_response()
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||
)
|
||||
def deactivate_connector_profile(
|
||||
profile_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_profile_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
profile_id=profile_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_profile_write(principal, row.scope_type)
|
||||
try:
|
||||
changed = delete_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason="api_delete",
|
||||
user_id=principal.user.id,
|
||||
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||
)
|
||||
if changed:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="deleted",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return FileConnectorProfileResponse(
|
||||
**connector_profile_from_row(row).to_response()
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
@@ -0,0 +1,852 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorCredentialCreateRequest,
|
||||
FileConnectorCredentialResponse,
|
||||
FileConnectorCredentialsResponse,
|
||||
FileConnectorCredentialUpdateRequest,
|
||||
FileConnectorDiscoveryRequest,
|
||||
FileConnectorDiscoveryResponse,
|
||||
FileConnectorSettingsDeltaResponse,
|
||||
FileConnectorPolicyEvaluateRequest,
|
||||
FileConnectorPolicyEvaluateResponse,
|
||||
FileConnectorPolicyResponse,
|
||||
FileConnectorPolicyUpdateRequest,
|
||||
FileConnectorProfileCreateRequest,
|
||||
FileConnectorProfileResponse,
|
||||
FileConnectorProfilesResponse,
|
||||
FileConnectorProviderResponse,
|
||||
FileConnectorProvidersResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
create_connector_credential_row,
|
||||
get_connector_credential_row,
|
||||
list_database_connector_credentials,
|
||||
update_connector_credential_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||
delete_connector_credential_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
browse_connector_profile,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
reject_api_controlled_deployment_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
connector_profile_from_row,
|
||||
create_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_providers import (
|
||||
connector_provider_descriptors,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
ConnectorPolicyDenied,
|
||||
connector_policy_decision,
|
||||
connector_policy_sources_from_payload,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
connector_policy_response,
|
||||
set_connector_policy,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
FILES_CONNECTOR_POLICY_RESOURCE,
|
||||
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
_audit_connector_discovery_attempt,
|
||||
_can_read_disabled_connector_profiles,
|
||||
_connector_credential_response,
|
||||
_connector_policy_error,
|
||||
_credential_row_for_profile,
|
||||
_discovery_profile_from_payload,
|
||||
_ensure_campaign_file_access,
|
||||
_ensure_connector_configuration_allowed,
|
||||
_ensure_connector_credential_configuration_allowed,
|
||||
_ensure_connector_local_policy_allowed,
|
||||
_file_connector_policy_resource_id,
|
||||
_file_connector_settings_entries,
|
||||
_http_error,
|
||||
_record_connector_settings_change,
|
||||
_require_connector_credential_write,
|
||||
_require_connector_policy_read,
|
||||
_require_connector_profile_write,
|
||||
_same_endpoint,
|
||||
_visible_connector_profiles,
|
||||
_webdav_discovery_candidates,
|
||||
)
|
||||
from govoplan_files.backend.services.connector_settings_delta import (
|
||||
_full_file_connector_settings_delta_response,
|
||||
_incremental_file_connector_settings_delta_response,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-policy/evaluate", response_model=FileConnectorPolicyEvaluateResponse
|
||||
)
|
||||
def evaluate_connector_policy(
|
||||
payload: FileConnectorPolicyEvaluateRequest,
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope("files:file:read", "files:file:upload", "files:file:download")
|
||||
),
|
||||
):
|
||||
del principal
|
||||
sources = connector_policy_sources_from_payload(
|
||||
[item.model_dump(mode="json") for item in payload.policy_sources]
|
||||
)
|
||||
request = ConnectorAccessRequest.from_provenance(
|
||||
payload.source_provenance.model_dump(mode="json", exclude_none=True),
|
||||
operation=payload.operation,
|
||||
)
|
||||
return FileConnectorPolicyEvaluateResponse(
|
||||
decision=connector_policy_decision(request, sources).to_dict()
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/settings/delta", response_model=FileConnectorSettingsDeltaResponse
|
||||
)
|
||||
def connector_settings_delta(
|
||||
scope_type: str = Query(default="tenant"),
|
||||
scope_id: str | None = Query(default=None),
|
||||
provider: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
include_inactive: bool = False,
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
since: str | None = None,
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
scope_type = scope_type.strip().casefold()
|
||||
_require_connector_policy_read(principal, scope_type)
|
||||
try:
|
||||
if since is None:
|
||||
return _full_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
entries, has_more = _file_connector_settings_entries(
|
||||
session, tenant_id=principal.tenant_id, since=since, limit=limit
|
||||
)
|
||||
if entries is None:
|
||||
return _full_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
return _incremental_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
entries=entries,
|
||||
has_more=has_more,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get("/connectors/providers", response_model=FileConnectorProvidersResponse)
|
||||
def list_connector_providers(
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope("files:file:read", "files:file:upload", "files:file:admin")
|
||||
),
|
||||
):
|
||||
del principal
|
||||
return FileConnectorProvidersResponse(
|
||||
providers=[
|
||||
FileConnectorProviderResponse(**item.to_response())
|
||||
for item in connector_provider_descriptors()
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post("/connectors/discover", response_model=FileConnectorDiscoveryResponse)
|
||||
def discover_connector_endpoint(
|
||||
payload: FileConnectorDiscoveryRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=payload.credentials.password_env,
|
||||
token_env=payload.credentials.token_env,
|
||||
secret_ref=payload.credentials.secret_ref,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
if payload.provider not in {"webdav", "nextcloud"}:
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=None,
|
||||
base_path=payload.base_path,
|
||||
status="unsupported",
|
||||
message=f"Discovery is not implemented for {payload.provider} connectors yet",
|
||||
)
|
||||
candidates: list[dict[str, str]] = []
|
||||
for endpoint_url in _webdav_discovery_candidates(payload):
|
||||
profile = _discovery_profile_from_payload(
|
||||
payload, endpoint_url, principal=principal
|
||||
)
|
||||
try:
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=None,
|
||||
credential_id=None,
|
||||
provider=profile.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
scope_type="tenant",
|
||||
scope_id=principal.tenant_id,
|
||||
operation="discover",
|
||||
)
|
||||
_audit_connector_discovery_attempt(
|
||||
session,
|
||||
principal,
|
||||
provider=profile.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
)
|
||||
browse_connector_profile(profile, path=payload.base_path or "")
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
OSError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
message = str(exc)
|
||||
if "credentials were rejected" in message.casefold():
|
||||
if payload.require_valid_credentials:
|
||||
candidates.append(
|
||||
{
|
||||
"endpoint_url": endpoint_url,
|
||||
"status": "credentials_rejected",
|
||||
"message": "The endpoint exists, but the credentials were rejected.",
|
||||
}
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
status="credentials_rejected",
|
||||
message="The endpoint was found, but login failed with these credentials.",
|
||||
candidates=candidates,
|
||||
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||
)
|
||||
candidates.append(
|
||||
{
|
||||
"endpoint_url": endpoint_url,
|
||||
"status": "found",
|
||||
"message": "The endpoint exists, but credentials are required or were rejected.",
|
||||
}
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
status="found",
|
||||
message="The endpoint was found. Add working credentials before saving or testing the connection.",
|
||||
candidates=candidates,
|
||||
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||
)
|
||||
candidates.append(
|
||||
{"endpoint_url": endpoint_url, "status": "failed", "message": message}
|
||||
)
|
||||
continue
|
||||
status_value = (
|
||||
"usable" if _same_endpoint(endpoint_url, payload.endpoint_url) else "found"
|
||||
)
|
||||
message = (
|
||||
"The supplied URL is directly usable."
|
||||
if status_value == "usable"
|
||||
else "A usable connector endpoint was discovered."
|
||||
)
|
||||
candidates.append(
|
||||
{"endpoint_url": endpoint_url, "status": status_value, "message": message}
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
status=status_value,
|
||||
message=message,
|
||||
candidates=candidates,
|
||||
metadata={"discovered_by": "webdav-propfind"},
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=None,
|
||||
base_path=payload.base_path,
|
||||
status="not_found",
|
||||
message="No usable WebDAV endpoint was found for this server URL.",
|
||||
candidates=candidates,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/connectors/credentials", response_model=FileConnectorCredentialsResponse)
|
||||
def list_connector_credentials(
|
||||
provider: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
provider_norm = provider.strip().casefold() if provider else None
|
||||
credentials = list_database_connector_credentials(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
)
|
||||
return FileConnectorCredentialsResponse(
|
||||
credentials=[
|
||||
FileConnectorCredentialResponse(**credential.to_response())
|
||||
for credential in credentials
|
||||
if provider_norm is None or credential.provider in {None, provider_norm}
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
|
||||
)
|
||||
def read_connector_policy(
|
||||
scope_type: str,
|
||||
scope_id: str | None = Query(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_policy_read(principal, scope_type)
|
||||
try:
|
||||
return FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.put(
|
||||
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
|
||||
)
|
||||
def write_connector_policy(
|
||||
scope_type: str,
|
||||
payload: FileConnectorPolicyUpdateRequest,
|
||||
scope_id: str | None = Query(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_profile_write(principal, scope_type)
|
||||
try:
|
||||
set_connector_policy(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_POLICY_RESOURCE,
|
||||
resource_id=_file_connector_policy_resource_id(scope_type, scope_id),
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=None
|
||||
if scope_type.strip().casefold() == "system"
|
||||
else principal.tenant_id,
|
||||
payload={"scope_type": scope_type.strip().casefold(), "scope_id": scope_id},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/credentials",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_connector_credential(
|
||||
payload: FileConnectorCredentialCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_credential_write(principal, payload.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
_ensure_connector_credential_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
credential_id=payload.id,
|
||||
provider=payload.provider,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
operation="configure_credentials",
|
||||
)
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
row = create_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
credential_id=payload.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
enabled=payload.enabled,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username,
|
||||
password=credentials.password,
|
||||
token=credentials.token,
|
||||
password_env=credentials.password_env,
|
||||
token_env=credentials.token_env,
|
||||
secret_ref=credentials.secret_ref,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="created",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/credentials/{credential_id}",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
)
|
||||
def get_connector_credential(
|
||||
credential_id: str,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
credential_id=credential_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
)
|
||||
return _connector_credential_response(row)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/connectors/credentials/{credential_id}",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
)
|
||||
def update_connector_credential(
|
||||
credential_id: str,
|
||||
payload: FileConnectorCredentialUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
credential_id=credential_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_credential_write(principal, row.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
provider = payload.provider if payload.provider is not None else row.provider
|
||||
_ensure_connector_credential_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
credential_id=row.id,
|
||||
provider=provider,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
operation="configure_credentials",
|
||||
)
|
||||
if payload.policy is not None:
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
update_connector_credential_row(
|
||||
session,
|
||||
row,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
enabled=payload.enabled,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username if credentials else None,
|
||||
password=credentials.password if credentials else None,
|
||||
token=credentials.token if credentials else None,
|
||||
password_env=credentials.password_env if credentials else None,
|
||||
token_env=credentials.token_env if credentials else None,
|
||||
secret_ref=credentials.secret_ref if credentials else None,
|
||||
clear_password=payload.clear_password,
|
||||
clear_token=payload.clear_token,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/connectors/credentials/{credential_id}",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
)
|
||||
def deactivate_connector_credential(
|
||||
credential_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
credential_id=credential_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_credential_write(principal, row.scope_type)
|
||||
try:
|
||||
deletion = delete_connector_credential_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason="api_delete",
|
||||
user_id=principal.user.id,
|
||||
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||
)
|
||||
if deletion.changed:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="deleted",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
for profile in deletion.affected_profiles:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=profile.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=profile.tenant_id,
|
||||
payload={
|
||||
"scope_type": profile.scope_type,
|
||||
"scope_id": profile.scope_id,
|
||||
"provider": profile.provider,
|
||||
"reason": "credential_deleted",
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
|
||||
def list_connector_profiles(
|
||||
provider: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
"system:settings:read",
|
||||
"admin:settings:read",
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
if campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorProfilesResponse(
|
||||
profiles=[
|
||||
FileConnectorProfileResponse(**profile.to_response())
|
||||
for profile in profiles
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles",
|
||||
response_model=FileConnectorProfileResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_connector_profile(
|
||||
payload: FileConnectorProfileCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_profile_write(principal, payload.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
credential_row = _credential_row_for_profile(
|
||||
session,
|
||||
principal,
|
||||
credential_profile_id=payload.credential_profile_id,
|
||||
provider=payload.provider,
|
||||
profile_id=payload.id,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
)
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=payload.id,
|
||||
credential_id=payload.credential_profile_id,
|
||||
provider=payload.provider,
|
||||
endpoint_url=connector_effective_endpoint_url(
|
||||
provider=payload.provider,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
metadata=payload.metadata,
|
||||
),
|
||||
base_path=payload.base_path,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
operation="configure",
|
||||
)
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
row = create_connector_profile_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
profile_id=payload.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
enabled=payload.enabled,
|
||||
credential_profile_id=payload.credential_profile_id,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username,
|
||||
password=credentials.password,
|
||||
token=credentials.token,
|
||||
password_env=credentials.password_env,
|
||||
token_env=credentials.token_env,
|
||||
secret_ref=credentials.secret_ref,
|
||||
capabilities=payload.capabilities,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="created",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return FileConnectorProfileResponse(
|
||||
**connector_profile_from_row(
|
||||
row, credential_row=credential_row
|
||||
).to_response()
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,151 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileFolderCreateRequest,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderDeleteResponse,
|
||||
FileFolderResponse,
|
||||
FileFoldersResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.folders import (
|
||||
create_folder,
|
||||
list_folders_for_user,
|
||||
list_folders_for_user_window,
|
||||
soft_delete_folder,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_folder_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.services.list_queries import (
|
||||
FOLDERS_LIST_CURSOR_SCOPE,
|
||||
_cursor_page_size,
|
||||
_files_delta_watermark,
|
||||
_folder_cursor_values,
|
||||
_folders_list_fingerprint,
|
||||
_next_folder_list_cursor,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/folders", response_model=FileFoldersResponse)
|
||||
def list_file_folders(
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
watermark = _files_delta_watermark(session, principal.tenant_id)
|
||||
effective_page_size = _cursor_page_size(
|
||||
FOLDERS_LIST_CURSOR_SCOPE, cursor, page_size
|
||||
)
|
||||
if effective_page_size is None:
|
||||
folders = list_folders_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileFoldersResponse(
|
||||
folders=[_folder_response(folder) for folder in folders],
|
||||
watermark=watermark,
|
||||
)
|
||||
fingerprint = _folders_list_fingerprint(
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
page_size=effective_page_size,
|
||||
)
|
||||
after_path, after_id = _folder_cursor_values(cursor, fingerprint=fingerprint)
|
||||
folders, has_more = list_folders_for_user_window(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
page_size=effective_page_size,
|
||||
after_path=after_path,
|
||||
after_id=after_id,
|
||||
)
|
||||
return FileFoldersResponse(
|
||||
folders=[_folder_response(folder) for folder in folders],
|
||||
cursor=cursor,
|
||||
next_cursor=_next_folder_list_cursor(
|
||||
principal,
|
||||
folders,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
page_size=effective_page_size,
|
||||
has_more=has_more,
|
||||
),
|
||||
watermark=watermark,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders", response_model=FileFolderResponse)
|
||||
def create_file_folder(
|
||||
payload: FileFolderCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
folder = create_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return _folder_response(folder)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
|
||||
def delete_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
deleted_folders, deleted_files = soft_delete_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderDeleteResponse(
|
||||
deleted_folders=deleted_folders, deleted_files=deleted_files
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,172 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
|
||||
from fastapi import (
|
||||
APIRouter,
|
||||
Depends,
|
||||
File as FastAPIFile,
|
||||
Header,
|
||||
HTTPException,
|
||||
UploadFile,
|
||||
status,
|
||||
)
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.events import (
|
||||
EventActorRef,
|
||||
EventObjectRef,
|
||||
EventTenantRef,
|
||||
PlatformEvent,
|
||||
emit_platform_event,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import FileFormEvidenceGrant
|
||||
from govoplan_files.backend.route_support import _read_limited_upload
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import create_file_asset
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/form-evidence/upload",
|
||||
response_model=dict[str, object],
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def upload_form_evidence(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
x_form_evidence_token: str = Header(alias="X-Form-Evidence-Token"),
|
||||
session: Session = Depends(get_session),
|
||||
) -> dict[str, object]:
|
||||
now = datetime.now(UTC)
|
||||
try:
|
||||
grant = (
|
||||
session.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.token_sha256
|
||||
== _token_sha256(x_form_evidence_token)
|
||||
)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if grant is None or grant.status != "issued" or _aware(grant.expires_at) <= now:
|
||||
raise _upload_unavailable()
|
||||
content_type = (
|
||||
str(file.content_type or "application/octet-stream")
|
||||
.split(";", 1)[0]
|
||||
.strip()
|
||||
.casefold()
|
||||
)
|
||||
if grant.allowed_content_types and content_type not in set(
|
||||
grant.allowed_content_types
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
|
||||
detail="This Form does not accept the uploaded document type.",
|
||||
)
|
||||
data = _read_limited_upload(file, max_bytes=grant.max_size_bytes)
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=grant.tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=grant.custodian_user_id,
|
||||
user_id=grant.custodian_user_id,
|
||||
filename=file.filename or "form-attachment",
|
||||
data=data,
|
||||
folder=f"Form submissions/{grant.form_instance_id}",
|
||||
content_type=content_type,
|
||||
description="Managed attachment captured through Forms Runtime.",
|
||||
metadata={
|
||||
"form_evidence": {
|
||||
"grant_id": grant.id,
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"form_definition_id": grant.form_definition_id,
|
||||
"form_definition_revision": grant.form_definition_revision,
|
||||
"purpose": grant.purpose,
|
||||
},
|
||||
"source_provenance": {
|
||||
"source_type": "form_evidence",
|
||||
"connector_id": "forms_runtime.evidence.files",
|
||||
"provider": "forms_runtime",
|
||||
"external_id": grant.id,
|
||||
"revision": grant.form_definition_revision,
|
||||
},
|
||||
},
|
||||
conflict_strategy="rename",
|
||||
is_admin=True,
|
||||
)
|
||||
grant.status = "uploaded"
|
||||
grant.file_asset_id = stored.asset.id
|
||||
grant.file_version_id = stored.version.id
|
||||
grant.uploaded_at = now
|
||||
session.add(grant)
|
||||
evidence = EvidenceReference(
|
||||
kind="document",
|
||||
owner_module="files",
|
||||
evidence_id=stored.asset.id,
|
||||
tenant_id=grant.tenant_id,
|
||||
version=stored.version.id,
|
||||
checksum=stored.version.checksum_sha256,
|
||||
source_ref=(f"form_submission:{grant.form_instance_id}:grant:{grant.id}"),
|
||||
responsible_actor_ref=f"user:{grant.custodian_user_id}",
|
||||
captured_at=now,
|
||||
)
|
||||
emit_platform_event(
|
||||
session,
|
||||
PlatformEvent(
|
||||
type="files.form_evidence.uploaded",
|
||||
module_id="files",
|
||||
payload={
|
||||
"grant_id": grant.id,
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"file_asset_id": stored.asset.id,
|
||||
"file_version_id": stored.version.id,
|
||||
"checksum_sha256": stored.version.checksum_sha256,
|
||||
"size_bytes": stored.version.size_bytes,
|
||||
},
|
||||
occurred_at=now,
|
||||
actor=EventActorRef(type="user", id=grant.custodian_user_id),
|
||||
tenant=EventTenantRef(id=grant.tenant_id),
|
||||
resource=EventObjectRef(type="file", id=stored.asset.id),
|
||||
classification="confidential",
|
||||
),
|
||||
)
|
||||
session.commit()
|
||||
except HTTPException:
|
||||
session.rollback()
|
||||
raise
|
||||
except (FileStorageError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
return {
|
||||
"grant_id": grant.id,
|
||||
"evidence": evidence.to_dict(include_inspection=True),
|
||||
}
|
||||
|
||||
|
||||
def _token_sha256(token: str) -> str:
|
||||
clean = str(token or "").strip()
|
||||
if len(clean) < 32:
|
||||
raise _upload_unavailable()
|
||||
return hashlib.sha256(clean.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _upload_unavailable() -> HTTPException:
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="The Form evidence upload is unavailable.",
|
||||
)
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = ["router", "upload_form_evidence"]
|
||||
@@ -0,0 +1,387 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileIntegrityActionRequest,
|
||||
FileIntegrityActionResponse,
|
||||
FileIntegrityFindingResponse,
|
||||
FileIntegrityFindingsResponse,
|
||||
FileIntegrityScanCreateRequest,
|
||||
FileIntegrityScanRunRequest,
|
||||
FileIntegrityScanResponse,
|
||||
FileIntegrityScansResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
cleanup_orphan_finding,
|
||||
create_integrity_scan,
|
||||
mark_integrity_scan_failed,
|
||||
recheck_integrity_finding,
|
||||
run_integrity_scan_batch,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files/integrity", tags=["files-integrity"])
|
||||
|
||||
|
||||
@router.get("/scans", response_model=FileIntegrityScansResponse)
|
||||
def list_integrity_scans(
|
||||
limit: int = Query(default=50, ge=1, le=200),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityScansResponse:
|
||||
rows = (
|
||||
session.query(FileIntegrityScan)
|
||||
.filter(FileIntegrityScan.tenant_id == principal.tenant_id)
|
||||
.order_by(FileIntegrityScan.created_at.desc(), FileIntegrityScan.id.desc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return FileIntegrityScansResponse(
|
||||
scans=[_scan_response(row) for row in rows]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/scans",
|
||||
response_model=FileIntegrityScanResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_scan(
|
||||
payload: FileIntegrityScanCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityScanResponse:
|
||||
try:
|
||||
scan = create_integrity_scan(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
verify_checksums=payload.verify_checksums,
|
||||
batch_size=payload.batch_size,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.integrity.scan_created",
|
||||
object_type="file_integrity_scan",
|
||||
object_id=scan.id,
|
||||
details={
|
||||
"storage_backend": scan.storage_backend,
|
||||
"verify_checksums": scan.verify_checksums,
|
||||
"batch_size": scan.batch_size,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _scan_response(scan)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/scans/{scan_id}/run", response_model=FileIntegrityScanResponse)
|
||||
def run_scan_batch(
|
||||
scan_id: str,
|
||||
payload: FileIntegrityScanRunRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityScanResponse:
|
||||
scan = _scan_for_tenant(
|
||||
session,
|
||||
scan_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
_assert_expected_revision(scan.revision, payload.expected_revision)
|
||||
previous_status = scan.status
|
||||
try:
|
||||
run_integrity_scan_batch(session, scan)
|
||||
if scan.status == "completed" and previous_status != "completed":
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.integrity.scan_completed",
|
||||
object_type="file_integrity_scan",
|
||||
object_id=scan.id,
|
||||
details={
|
||||
"verified_blob_count": scan.verified_blob_count,
|
||||
"quarantined_blob_count": scan.quarantined_blob_count,
|
||||
"orphan_object_count": scan.orphan_object_count,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _scan_response(scan)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
scan = _scan_for_tenant(
|
||||
session,
|
||||
scan_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
mark_integrity_scan_failed(scan, error=exc)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.integrity.scan_failed",
|
||||
object_type="file_integrity_scan",
|
||||
object_id=scan.id,
|
||||
details={"error_type": type(exc).__name__},
|
||||
)
|
||||
session.commit()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get(
|
||||
"/scans/{scan_id}/findings",
|
||||
response_model=FileIntegrityFindingsResponse,
|
||||
)
|
||||
def list_integrity_findings(
|
||||
scan_id: str,
|
||||
state_filter: str | None = Query(default=None, alias="state"),
|
||||
limit: int = Query(default=500, ge=1, le=1000),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityFindingsResponse:
|
||||
scan = _scan_for_tenant(session, scan_id, principal.tenant_id)
|
||||
query = session.query(FileIntegrityFinding).filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.tenant_id == principal.tenant_id,
|
||||
)
|
||||
if state_filter:
|
||||
query = query.filter(FileIntegrityFinding.state == state_filter)
|
||||
rows = (
|
||||
query.order_by(
|
||||
FileIntegrityFinding.created_at.asc(),
|
||||
FileIntegrityFinding.id.asc(),
|
||||
)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return FileIntegrityFindingsResponse(
|
||||
findings=[_finding_response(row) for row in rows]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/findings/{finding_id}/recheck",
|
||||
response_model=FileIntegrityActionResponse,
|
||||
)
|
||||
def recheck_finding(
|
||||
finding_id: str,
|
||||
payload: FileIntegrityActionRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityActionResponse:
|
||||
finding = _finding_for_tenant(
|
||||
session,
|
||||
finding_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
_assert_expected_revision(finding.revision, payload.expected_revision)
|
||||
try:
|
||||
result = recheck_integrity_finding(
|
||||
session,
|
||||
finding,
|
||||
user_id=principal.user.id,
|
||||
dry_run=payload.dry_run,
|
||||
)
|
||||
_audit_integrity_action(session, principal, result)
|
||||
session.commit()
|
||||
return _action_response(result)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/findings/{finding_id}/cleanup",
|
||||
response_model=FileIntegrityActionResponse,
|
||||
)
|
||||
def cleanup_finding(
|
||||
finding_id: str,
|
||||
payload: FileIntegrityActionRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityActionResponse:
|
||||
finding = _finding_for_tenant(
|
||||
session,
|
||||
finding_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
_assert_expected_revision(finding.revision, payload.expected_revision)
|
||||
try:
|
||||
result = cleanup_orphan_finding(
|
||||
session,
|
||||
finding,
|
||||
user_id=principal.user.id,
|
||||
dry_run=payload.dry_run,
|
||||
)
|
||||
_audit_integrity_action(session, principal, result)
|
||||
session.commit()
|
||||
return _action_response(result)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
def _scan_for_tenant(
|
||||
session: Session,
|
||||
scan_id: str,
|
||||
tenant_id: str,
|
||||
*,
|
||||
for_update: bool = False,
|
||||
) -> FileIntegrityScan:
|
||||
query = session.query(FileIntegrityScan).filter(FileIntegrityScan.id == scan_id)
|
||||
if for_update:
|
||||
query = query.populate_existing().with_for_update()
|
||||
scan = query.one_or_none()
|
||||
if scan is None or scan.tenant_id != tenant_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="Integrity scan not found",
|
||||
)
|
||||
return scan
|
||||
|
||||
|
||||
def _finding_for_tenant(
|
||||
session: Session,
|
||||
finding_id: str,
|
||||
tenant_id: str,
|
||||
*,
|
||||
for_update: bool = False,
|
||||
) -> FileIntegrityFinding:
|
||||
query = session.query(FileIntegrityFinding).filter(
|
||||
FileIntegrityFinding.id == finding_id
|
||||
)
|
||||
if for_update:
|
||||
query = query.populate_existing().with_for_update()
|
||||
finding = query.one_or_none()
|
||||
if finding is None or finding.tenant_id != tenant_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="Integrity finding not found",
|
||||
)
|
||||
return finding
|
||||
|
||||
|
||||
def _assert_expected_revision(current: int, expected: int) -> None:
|
||||
if current != expected:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=(
|
||||
"The integrity record changed after it was loaded; reload before "
|
||||
"performing this action."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _audit_integrity_action(session, principal, result) -> None:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=f"files.integrity.{result.action}",
|
||||
object_type="file_integrity_finding",
|
||||
object_id=result.finding.id,
|
||||
details={
|
||||
"scan_id": result.finding.scan_id,
|
||||
"kind": result.finding.kind,
|
||||
"dry_run": result.dry_run,
|
||||
"changed": result.changed,
|
||||
"storage_key_sha256": hashlib.sha256(
|
||||
result.finding.storage_key.encode("utf-8")
|
||||
).hexdigest(),
|
||||
"inspection": result.inspection.kind
|
||||
if result.inspection
|
||||
else None,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _scan_response(scan: FileIntegrityScan) -> FileIntegrityScanResponse:
|
||||
return FileIntegrityScanResponse(
|
||||
id=scan.id,
|
||||
tenant_id=scan.tenant_id,
|
||||
storage_backend=scan.storage_backend,
|
||||
storage_prefix=scan.storage_prefix,
|
||||
status=scan.status,
|
||||
revision=scan.revision,
|
||||
phase=scan.phase,
|
||||
verify_checksums=scan.verify_checksums,
|
||||
batch_size=scan.batch_size,
|
||||
scanned_blob_count=scan.scanned_blob_count,
|
||||
verified_blob_count=scan.verified_blob_count,
|
||||
quarantined_blob_count=scan.quarantined_blob_count,
|
||||
scanned_object_count=scan.scanned_object_count,
|
||||
orphan_object_count=scan.orphan_object_count,
|
||||
created_by_user_id=scan.created_by_user_id,
|
||||
started_at=scan.started_at.isoformat() if scan.started_at else None,
|
||||
completed_at=scan.completed_at.isoformat()
|
||||
if scan.completed_at
|
||||
else None,
|
||||
last_error=scan.last_error,
|
||||
created_at=scan.created_at.isoformat(),
|
||||
updated_at=scan.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
def _finding_response(
|
||||
finding: FileIntegrityFinding,
|
||||
) -> FileIntegrityFindingResponse:
|
||||
return FileIntegrityFindingResponse(
|
||||
id=finding.id,
|
||||
scan_id=finding.scan_id,
|
||||
tenant_id=finding.tenant_id,
|
||||
kind=finding.kind,
|
||||
state=finding.state,
|
||||
revision=finding.revision,
|
||||
blob_id=finding.blob_id,
|
||||
storage_key=finding.storage_key,
|
||||
expected_size_bytes=finding.expected_size_bytes,
|
||||
observed_size_bytes=finding.observed_size_bytes,
|
||||
expected_checksum_sha256=finding.expected_checksum_sha256,
|
||||
observed_checksum_sha256=finding.observed_checksum_sha256,
|
||||
resolved_at=finding.resolved_at.isoformat()
|
||||
if finding.resolved_at
|
||||
else None,
|
||||
resolved_by_user_id=finding.resolved_by_user_id,
|
||||
created_at=finding.created_at.isoformat(),
|
||||
updated_at=finding.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
def _action_response(result) -> FileIntegrityActionResponse:
|
||||
return FileIntegrityActionResponse(
|
||||
action=result.action,
|
||||
changed=result.changed,
|
||||
dry_run=result.dry_run,
|
||||
finding=_finding_response(result.finding),
|
||||
inspection_kind=result.inspection.kind if result.inspection else None,
|
||||
inspection_valid=result.inspection.valid if result.inspection else None,
|
||||
)
|
||||
|
||||
|
||||
def _integrity_http_error(exc: Exception) -> HTTPException:
|
||||
if isinstance(exc, FileStorageError):
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=str(exc),
|
||||
)
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="The configured file storage backend could not complete the integrity operation",
|
||||
)
|
||||
@@ -0,0 +1,297 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_connector_space_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileAssetResponse,
|
||||
FileBlobGcRequest,
|
||||
FileBlobGcResponse,
|
||||
FileConnectorSpaceResponse,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderRestoreResponse,
|
||||
FileLifecycleUpdateRequest,
|
||||
FilePurgeExecuteRequest,
|
||||
FilePurgePreviewItem,
|
||||
FilePurgePreviewRequest,
|
||||
FilePurgePreviewResponse,
|
||||
FilePurgeResponse,
|
||||
FileRestoreResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.lifecycle import (
|
||||
execute_asset_purge,
|
||||
garbage_collect_unreferenced_blobs,
|
||||
get_asset_for_lifecycle,
|
||||
preview_asset_purge,
|
||||
restore_asset,
|
||||
restore_connector_space,
|
||||
restore_folder,
|
||||
set_asset_lifecycle,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files-lifecycle"])
|
||||
|
||||
|
||||
@router.patch("/{file_id}/lifecycle", response_model=FileAssetResponse)
|
||||
def update_file_lifecycle(
|
||||
file_id: str,
|
||||
payload: FileLifecycleUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:retention")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_lifecycle(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
set_asset_lifecycle(
|
||||
session,
|
||||
asset,
|
||||
retained_until=payload.retained_until,
|
||||
legal_hold=payload.legal_hold,
|
||||
reason=payload.reason,
|
||||
expected_revision=payload.expected_revision,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.lifecycle.updated",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={
|
||||
"retained_until": (
|
||||
asset.retained_until.isoformat() if asset.retained_until else None
|
||||
),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
"reason": asset.lifecycle_reason,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/assets/{file_id}/restore", response_model=FileRestoreResponse)
|
||||
def restore_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_lifecycle(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
changed = restore_asset(session, asset)
|
||||
if changed:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.file.restored",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={"display_path": asset.display_path},
|
||||
)
|
||||
session.commit()
|
||||
return FileRestoreResponse(restored_count=1 if changed else 0)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/folders/restore", response_model=FileFolderRestoreResponse)
|
||||
def restore_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
restored_folders, restored_files = restore_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.folder.restored",
|
||||
object_type="file_folder",
|
||||
object_id=payload.path,
|
||||
details={
|
||||
"restored_folders": restored_folders,
|
||||
"restored_files": restored_files,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderRestoreResponse(
|
||||
restored_folders=restored_folders, restored_files=restored_files
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/restore",
|
||||
response_model=FileConnectorSpaceResponse,
|
||||
)
|
||||
def restore_file_connector_space(
|
||||
space_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
space = restore_connector_space(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector_space.restored",
|
||||
object_type="file_connector_space",
|
||||
object_id=space.id,
|
||||
details={"label": space.label},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/purge/preview", response_model=FilePurgePreviewResponse)
|
||||
def preview_file_purge(
|
||||
payload: FilePurgePreviewRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
preview = preview_asset_purge(
|
||||
session, tenant_id=principal.tenant_id, file_ids=payload.file_ids
|
||||
)
|
||||
return FilePurgePreviewResponse(
|
||||
preview_sha256=preview.preview_sha256,
|
||||
eligible=preview.eligible,
|
||||
items=[
|
||||
FilePurgePreviewItem(
|
||||
file_id=item.file_id,
|
||||
filename=item.filename,
|
||||
lifecycle_revision=item.lifecycle_revision,
|
||||
deleted_at=(item.deleted_at.isoformat() if item.deleted_at else None),
|
||||
retained_until=(
|
||||
item.retained_until.isoformat() if item.retained_until else None
|
||||
),
|
||||
legal_hold=item.legal_hold,
|
||||
blockers=list(item.blockers),
|
||||
blob_ids=list(item.blob_ids),
|
||||
)
|
||||
for item in preview.items
|
||||
],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/purge/execute", response_model=FilePurgeResponse)
|
||||
def execute_file_purge(
|
||||
payload: FilePurgeExecuteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
result = execute_asset_purge(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
file_ids=payload.file_ids,
|
||||
preview_sha256=payload.preview_sha256,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
approval_reference=payload.approval_reference,
|
||||
before_commit=lambda assets: audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.files.purged",
|
||||
object_type="file_batch",
|
||||
object_id=payload.preview_sha256,
|
||||
details={
|
||||
"file_ids": [asset.id for asset in assets],
|
||||
"approval_reference": payload.approval_reference,
|
||||
"preview_sha256": payload.preview_sha256,
|
||||
},
|
||||
),
|
||||
)
|
||||
return FilePurgeResponse(
|
||||
recovery_operation_id=result.recovery_operation_id,
|
||||
status=result.status,
|
||||
replayed=result.replayed,
|
||||
purged_files=result.purged_files,
|
||||
released_blobs=result.released_blobs,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/purge/blobs", response_model=FileBlobGcResponse)
|
||||
def garbage_collect_file_blobs(
|
||||
payload: FileBlobGcRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
result = garbage_collect_unreferenced_blobs(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
limit=payload.limit,
|
||||
approval_reference=payload.approval_reference,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.blobs.garbage_collected",
|
||||
object_type="file_blob_batch",
|
||||
object_id=principal.tenant_id,
|
||||
details={
|
||||
"inspected_blobs": result.inspected_blobs,
|
||||
"deleted_blobs": result.deleted_blobs,
|
||||
"unresolved_operation_ids": list(result.unresolved_operation_ids),
|
||||
"approval_reference": payload.approval_reference,
|
||||
},
|
||||
commit=True,
|
||||
)
|
||||
return FileBlobGcResponse(
|
||||
inspected_blobs=result.inspected_blobs,
|
||||
deleted_blobs=result.deleted_blobs,
|
||||
unresolved_operation_ids=list(result.unresolved_operation_ids),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,194 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileDeltaResponse,
|
||||
FileListResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.files import (
|
||||
count_assets_for_user,
|
||||
list_assets_for_user,
|
||||
list_assets_for_user_window,
|
||||
list_recent_assets_for_user,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_list_response,
|
||||
_ensure_campaign_file_access,
|
||||
_ensure_list_owner_access,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.services.list_queries import (
|
||||
FILES_LIST_CURSOR_SCOPE,
|
||||
_cursor_page_size,
|
||||
_file_cursor_values,
|
||||
_files_delta_response,
|
||||
_files_delta_watermark,
|
||||
_files_list_fingerprint,
|
||||
_full_file_delta_response,
|
||||
_next_file_list_cursor,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/delta", response_model=FileDeltaResponse)
|
||||
def files_delta(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
since: str | None = None,
|
||||
limit: int = Query(default=500, ge=1, le=1000),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
if since is None:
|
||||
return _full_file_delta_response(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
return _files_delta_response(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
since=since,
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=FileListResponse)
|
||||
def list_files(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
sort: Literal["path", "recent"] = "path",
|
||||
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
watermark = _files_delta_watermark(session, principal.tenant_id)
|
||||
total = count_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if sort == "recent":
|
||||
if cursor:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="Recent file projections do not accept a path-order cursor.",
|
||||
)
|
||||
recent_limit = page_size or 25
|
||||
assets = list_recent_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
limit=recent_limit,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
watermark=watermark,
|
||||
)
|
||||
effective_page_size = _cursor_page_size(FILES_LIST_CURSOR_SCOPE, cursor, page_size)
|
||||
if effective_page_size is not None:
|
||||
fingerprint = _files_list_fingerprint(
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
page_size=effective_page_size,
|
||||
)
|
||||
after_display_path, after_updated_at, after_id = _file_cursor_values(
|
||||
cursor, fingerprint=fingerprint
|
||||
)
|
||||
assets, has_more = list_assets_for_user_window(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
page_size=effective_page_size,
|
||||
after_display_path=after_display_path,
|
||||
after_updated_at=after_updated_at,
|
||||
after_id=after_id,
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
cursor=cursor,
|
||||
next_cursor=_next_file_list_cursor(
|
||||
principal,
|
||||
assets,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
page_size=effective_page_size,
|
||||
has_more=has_more,
|
||||
),
|
||||
watermark=watermark,
|
||||
)
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
watermark=watermark,
|
||||
)
|
||||
@@ -0,0 +1,330 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.api.v1.schemas import (
|
||||
ReferenceOptionListResponse,
|
||||
ReferenceOptionResponse,
|
||||
)
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.core.references import (
|
||||
access_scope_reference_page,
|
||||
access_scope_reference_provider_available,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import get_registry
|
||||
from govoplan_files.backend.schemas import (
|
||||
BulkFileShareRequest,
|
||||
BulkFileShareResponse,
|
||||
FileShareRequest,
|
||||
FileShareResponse,
|
||||
FileSharesResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
current_file_share_for_target,
|
||||
get_asset_for_share_management,
|
||||
list_file_shares,
|
||||
revoke_file_share,
|
||||
share_file,
|
||||
share_files,
|
||||
)
|
||||
from govoplan_files.backend.route_support import (
|
||||
_file_share_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{file_id}/share-target-options",
|
||||
response_model=ReferenceOptionListResponse,
|
||||
)
|
||||
def search_share_targets(
|
||||
file_id: str,
|
||||
target_type: str,
|
||||
q: str = "",
|
||||
selected: list[str] = Query(default=[]),
|
||||
limit: int = Query(default=50, ge=1, le=200),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
) -> ReferenceOptionListResponse:
|
||||
if target_type not in {"user", "group"}:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="Share target type must be user or group",
|
||||
)
|
||||
try:
|
||||
get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
registry = get_registry()
|
||||
page = access_scope_reference_page(
|
||||
registry,
|
||||
principal,
|
||||
scope_type=target_type,
|
||||
reference_kind="membership" if target_type == "user" else "group",
|
||||
query=q,
|
||||
selected_values=selected,
|
||||
limit=limit,
|
||||
cursor=cursor,
|
||||
administrative=True,
|
||||
session=session,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return ReferenceOptionListResponse(
|
||||
options=[
|
||||
ReferenceOptionResponse(**option.to_dict()) for option in page.options
|
||||
],
|
||||
provider_available=access_scope_reference_provider_available(registry),
|
||||
next_cursor=page.next_cursor,
|
||||
has_more=page.has_more,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{file_id}/shares", response_model=FileSharesResponse)
|
||||
def list_shares(
|
||||
file_id: str,
|
||||
include_inactive: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileSharesResponse(
|
||||
shares=[
|
||||
_file_share_response(share)
|
||||
for share in list_file_shares(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
]
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/{file_id}/shares", response_model=FileShareResponse)
|
||||
def create_share(
|
||||
file_id: str,
|
||||
payload: FileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
previous = current_file_share_for_target(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
)
|
||||
previous_permission = previous.permission if previous else None
|
||||
previous_expiry = previous.expires_at if previous else None
|
||||
share = share_file(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset=asset,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
expires_at=payload.expires_at,
|
||||
)
|
||||
session.flush()
|
||||
action = _share_audit_action(
|
||||
existed=previous is not None,
|
||||
previous_permission=previous_permission,
|
||||
permission=share.permission,
|
||||
previous_expiry=previous_expiry,
|
||||
expiry=share.expires_at,
|
||||
)
|
||||
if action:
|
||||
_audit_share_change(session, principal, share, action=action)
|
||||
session.commit()
|
||||
return _file_share_response(share)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-shares", response_model=BulkFileShareResponse)
|
||||
def create_bulk_shares(
|
||||
payload: BulkFileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
file_ids = list(dict.fromkeys(payload.file_ids))
|
||||
assets = [
|
||||
get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
for file_id in file_ids
|
||||
]
|
||||
previous_by_asset = {
|
||||
asset.id: current_file_share_for_target(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
)
|
||||
for asset in assets
|
||||
}
|
||||
previous_values = {
|
||||
asset_id: (
|
||||
share.permission if share else None,
|
||||
share.expires_at if share else None,
|
||||
)
|
||||
for asset_id, share in previous_by_asset.items()
|
||||
}
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
assets=assets,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
expires_at=payload.expires_at,
|
||||
)
|
||||
session.flush()
|
||||
for share in shares:
|
||||
previous_permission, previous_expiry = previous_values[share.file_asset_id]
|
||||
action = _share_audit_action(
|
||||
existed=previous_by_asset[share.file_asset_id] is not None,
|
||||
previous_permission=previous_permission,
|
||||
permission=share.permission,
|
||||
previous_expiry=previous_expiry,
|
||||
expiry=share.expires_at,
|
||||
)
|
||||
if action:
|
||||
_audit_share_change(session, principal, share, action=action)
|
||||
session.commit()
|
||||
return BulkFileShareResponse(
|
||||
shared_count=len(shares),
|
||||
shares=[_file_share_response(share) for share in shares],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete("/{file_id}/shares/{share_id}", response_model=FileShareResponse)
|
||||
def revoke_share(
|
||||
file_id: str,
|
||||
share_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
share, changed = revoke_file_share(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
share_id=share_id,
|
||||
user_id=principal.user.id,
|
||||
)
|
||||
if changed:
|
||||
_audit_share_change(
|
||||
session, principal, share, action="files.share.revoked"
|
||||
)
|
||||
session.commit()
|
||||
return _file_share_response(share)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
def _share_audit_action(
|
||||
*,
|
||||
existed: bool,
|
||||
previous_permission: str | None,
|
||||
permission: str,
|
||||
previous_expiry: datetime | None,
|
||||
expiry: datetime | None,
|
||||
) -> str | None:
|
||||
if not existed:
|
||||
return "files.share.granted"
|
||||
permission_changed = previous_permission != permission
|
||||
expiry_changed = _normalized_expiry(previous_expiry) != _normalized_expiry(expiry)
|
||||
if permission_changed:
|
||||
return "files.share.changed"
|
||||
if expiry_changed:
|
||||
return "files.share.expiry_changed"
|
||||
return None
|
||||
|
||||
|
||||
def _audit_share_change(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
share,
|
||||
*,
|
||||
action: str,
|
||||
) -> None:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=action,
|
||||
object_type="file_share",
|
||||
object_id=share.id,
|
||||
details={
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
"expires_at": _normalized_expiry(share.expires_at),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _normalized_expiry(value: datetime | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(timezone.utc).isoformat()
|
||||
@@ -0,0 +1,308 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorSpaceCreateRequest,
|
||||
FileConnectorSpaceResponse,
|
||||
FileConnectorSpacesResponse,
|
||||
FileConnectorSpaceUpdateRequest,
|
||||
FileSpaceResponse,
|
||||
FileSpacesResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.access import group_refs_for_ids, user_group_ids
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_spaces import (
|
||||
connector_space_owner_id,
|
||||
create_connector_space,
|
||||
get_connector_space_for_user,
|
||||
list_connector_spaces_for_user,
|
||||
soft_delete_connector_space,
|
||||
update_connector_space,
|
||||
validate_connector_space_write_mode,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
_connector_policy_error,
|
||||
_connector_space_file_space_response,
|
||||
_connector_space_policy_decision,
|
||||
_connector_space_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
_record_connector_settings_change,
|
||||
_visible_connector_profile,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/spaces", response_model=FileSpacesResponse)
|
||||
def list_file_spaces(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
spaces = [
|
||||
FileSpaceResponse(
|
||||
id=f"user:{principal.user.id}",
|
||||
label="My files",
|
||||
owner_type="user",
|
||||
owner_id=principal.user.id,
|
||||
description="Files owned by your user account.",
|
||||
)
|
||||
]
|
||||
group_ids = user_group_ids(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
include_admin_groups=_is_admin(principal),
|
||||
)
|
||||
if group_ids:
|
||||
groups = group_refs_for_ids(tenant_id=principal.tenant_id, group_ids=group_ids)
|
||||
spaces.extend(
|
||||
FileSpaceResponse(
|
||||
id=f"group:{group.id}",
|
||||
label=f"{group.name} files",
|
||||
owner_type="group",
|
||||
owner_id=group.id,
|
||||
description="Files owned by this group.",
|
||||
)
|
||||
for group in groups
|
||||
)
|
||||
connector_spaces = list_connector_spaces_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
spaces.extend(
|
||||
_connector_space_file_space_response(space) for space in connector_spaces
|
||||
)
|
||||
return FileSpacesResponse(spaces=spaces)
|
||||
|
||||
|
||||
@router.get("/connector-spaces", response_model=FileConnectorSpacesResponse)
|
||||
def list_file_connector_spaces(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
include_inactive: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
spaces = list_connector_spaces_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive and _is_admin(principal),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileConnectorSpacesResponse(
|
||||
spaces=[_connector_space_response(space) for space in spaces]
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces",
|
||||
response_model=FileConnectorSpaceResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_file_connector_space(
|
||||
payload: FileConnectorSpaceCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
if payload.owner_type == "group" and not payload.owner_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail="owner_id is required for group connector spaces",
|
||||
)
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
try:
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, payload.connector_profile_id
|
||||
)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
operation="link",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
validate_connector_space_write_mode(profile, read_only=payload.read_only)
|
||||
space = create_connector_space(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
profile=profile,
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
sync_mode=payload.sync_mode,
|
||||
read_only=payload.read_only,
|
||||
metadata=payload.metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
resource_id=space.id,
|
||||
operation="created",
|
||||
principal=principal,
|
||||
tenant_id=space.tenant_id,
|
||||
payload={
|
||||
"owner_type": space.owner_type,
|
||||
"owner_id": connector_space_owner_id(space),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.patch("/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse)
|
||||
def update_file_connector_space(
|
||||
space_id: str,
|
||||
payload: FileConnectorSpaceUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
include_inactive=_is_admin(principal),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
try:
|
||||
if (
|
||||
payload.library_id is not None
|
||||
or payload.remote_path is not None
|
||||
or payload.read_only is not None
|
||||
):
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=payload.library_id
|
||||
if payload.library_id is not None
|
||||
else space.library_id,
|
||||
remote_path=payload.remote_path
|
||||
if payload.remote_path is not None
|
||||
else space.remote_path,
|
||||
operation="link",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
validate_connector_space_write_mode(
|
||||
profile,
|
||||
read_only=(
|
||||
payload.read_only
|
||||
if payload.read_only is not None
|
||||
else space.read_only
|
||||
),
|
||||
)
|
||||
update_connector_space(
|
||||
session,
|
||||
space,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
sync_mode=payload.sync_mode,
|
||||
read_only=payload.read_only,
|
||||
is_active=payload.is_active,
|
||||
metadata=payload.metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
resource_id=space.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=space.tenant_id,
|
||||
payload={
|
||||
"owner_type": space.owner_type,
|
||||
"owner_id": connector_space_owner_id(space),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse
|
||||
)
|
||||
def delete_file_connector_space(
|
||||
space_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
include_inactive=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
soft_delete_connector_space(
|
||||
session, space, user_id=principal.user.id, is_admin=_is_admin(principal)
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
resource_id=space.id,
|
||||
operation="deleted",
|
||||
principal=principal,
|
||||
tenant_id=space.tenant_id,
|
||||
payload={
|
||||
"owner_type": space.owner_type,
|
||||
"owner_id": connector_space_owner_id(space),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
@@ -0,0 +1,213 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi.responses import FileResponse
|
||||
from starlette.background import BackgroundTask
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
ArchiveRequest,
|
||||
PatternMatchResponse,
|
||||
PatternResolveRequest,
|
||||
PatternResolveResponse,
|
||||
RenamePreviewItem,
|
||||
RenameRequest,
|
||||
RenameResponse,
|
||||
TransferRequest,
|
||||
TransferResponse,
|
||||
_conflict_resolutions,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
UnsafeFilePathError,
|
||||
filename_from_path,
|
||||
normalize_logical_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.archives import create_zip_file
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
get_asset_for_user,
|
||||
list_assets_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.search import resolve_patterns
|
||||
from govoplan_files.backend.storage.transfers import (
|
||||
rename_selection,
|
||||
transfer_selection,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_attachment_disposition,
|
||||
_audit_connector_access,
|
||||
_cleanup_temp_file,
|
||||
_ensure_campaign_file_access,
|
||||
_ensure_list_owner_access,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post("/bulk-rename", response_model=RenameResponse)
|
||||
def bulk_rename(
|
||||
payload: RenameRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
plan = rename_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
mode=payload.mode,
|
||||
new_name=payload.new_name,
|
||||
find=payload.find,
|
||||
replacement=payload.replacement,
|
||||
prefix=payload.prefix,
|
||||
suffix=payload.suffix,
|
||||
recursive=payload.recursive,
|
||||
dry_run=payload.dry_run,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if not payload.dry_run:
|
||||
session.commit()
|
||||
return RenameResponse(
|
||||
dry_run=payload.dry_run,
|
||||
items=[
|
||||
RenamePreviewItem(
|
||||
kind=item.kind,
|
||||
id=item.id,
|
||||
file_id=item.id if item.kind == "file" else None,
|
||||
folder_path=item.old_path if item.kind == "folder" else None,
|
||||
old_path=item.old_path,
|
||||
new_path=item.new_path,
|
||||
)
|
||||
for item in plan
|
||||
],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/transfer", response_model=TransferResponse)
|
||||
def transfer_files(
|
||||
payload: TransferRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
files, folders = transfer_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
operation=payload.operation,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
source_owner_type=payload.source_owner_type,
|
||||
source_owner_id=payload.source_owner_id,
|
||||
target_owner_type=payload.target_owner_type,
|
||||
target_owner_id=payload.target_owner_id,
|
||||
target_folder=payload.target_folder,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return TransferResponse(
|
||||
operation=payload.operation, files=files, folders=folders
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/archive.zip")
|
||||
def download_archive(
|
||||
payload: ArchiveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
tmp = tempfile.NamedTemporaryFile(
|
||||
prefix="govoplan-files-", suffix=".zip", delete=False
|
||||
)
|
||||
tmp_path = tmp.name
|
||||
tmp.close()
|
||||
try:
|
||||
create_zip_file(session, assets, tmp_path)
|
||||
except Exception:
|
||||
_cleanup_temp_file(tmp_path)
|
||||
raise
|
||||
_audit_connector_access(session, principal, assets, operation="archive")
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
filename = filename_from_path(
|
||||
normalize_logical_path(payload.filename, fallback_filename="files.zip")
|
||||
)
|
||||
headers = {"Content-Disposition": _attachment_disposition(filename)}
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type="application/zip",
|
||||
headers=headers,
|
||||
background=BackgroundTask(_cleanup_temp_file, tmp_path),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
|
||||
def resolve_file_patterns(
|
||||
payload: PatternResolveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
try:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
campaign_id=payload.campaign_id,
|
||||
path_prefix=payload.path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
resolved, unmatched = resolve_patterns(
|
||||
assets,
|
||||
payload.patterns,
|
||||
base_path=payload.path_prefix,
|
||||
case_sensitive=payload.case_sensitive,
|
||||
)
|
||||
return PatternResolveResponse(
|
||||
patterns=[
|
||||
PatternMatchResponse(
|
||||
pattern=item.pattern,
|
||||
matches=[_asset_response(session, asset) for asset in item.matches],
|
||||
)
|
||||
for item in resolved
|
||||
],
|
||||
unmatched=[_asset_response(session, asset) for asset in unmatched]
|
||||
if payload.include_unmatched
|
||||
else [],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,487 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.security.secrets import (
|
||||
TransientPayloadError,
|
||||
open_transient_payload,
|
||||
seal_transient_payload,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
ArchiveEntryResponse,
|
||||
ArchivePreviewResponse,
|
||||
ConflictResolutionRequest,
|
||||
FileUploadResponse,
|
||||
_conflict_resolutions,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
UnsafeFilePathError,
|
||||
normalize_folder,
|
||||
)
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
archive_format_for_filename,
|
||||
extract_archive_upload,
|
||||
extract_zip_upload,
|
||||
inspect_archive,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_audit_connector_imports,
|
||||
_cleanup_temp_file,
|
||||
_connector_policy_error,
|
||||
_enforce_connector_policy,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
_read_limited_upload,
|
||||
_source_metadata_from_form,
|
||||
_spool_limited_upload_to_temp,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
_ARCHIVE_PREVIEW_PURPOSE = "files.archive-preview.v1"
|
||||
|
||||
|
||||
def _archive_suffix(filename: str) -> str:
|
||||
lowered = filename.casefold()
|
||||
for suffix in (".tar.bz2", ".tar.gz", ".tar.xz", ".tbz2", ".tgz", ".txz", ".tar", ".zip"):
|
||||
if lowered.endswith(suffix):
|
||||
return suffix
|
||||
return ".archive"
|
||||
|
||||
|
||||
def _archive_sha256(path: str) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as source:
|
||||
while chunk := source.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _archive_selected_paths(value: str) -> list[str]:
|
||||
parsed = json.loads(value)
|
||||
if not isinstance(parsed, list) or not parsed:
|
||||
raise FileStorageError("Select at least one archive file or folder")
|
||||
if len(parsed) > settings.file_archive_max_entries:
|
||||
raise FileStorageError(
|
||||
"Archive selection exceeds the configured entry limit"
|
||||
)
|
||||
selected: list[str] = []
|
||||
for item in parsed:
|
||||
if not isinstance(item, str) or not item.strip():
|
||||
raise FileStorageError("Archive selection contains an invalid path")
|
||||
if len(item) > 4096:
|
||||
raise FileStorageError("Archive selection path is too long")
|
||||
selected.append(item)
|
||||
return selected
|
||||
|
||||
|
||||
def _validate_archive_preview_token(
|
||||
token: str,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
path: str,
|
||||
campaign_id: str | None,
|
||||
archive_format: str,
|
||||
archive_sha256: str,
|
||||
) -> None:
|
||||
payload = open_transient_payload(
|
||||
token,
|
||||
ttl_seconds=settings.file_archive_preview_ttl_seconds,
|
||||
)
|
||||
expected = {
|
||||
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||
"tenant_id": tenant_id,
|
||||
"user_id": user_id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"path": path,
|
||||
"campaign_id": campaign_id or "",
|
||||
"archive_format": archive_format,
|
||||
}
|
||||
if any(payload.get(key) != value for key, value in expected.items()):
|
||||
raise FileStorageError(
|
||||
"Archive preview does not match this upload destination"
|
||||
)
|
||||
token_digest = payload.get("archive_sha256")
|
||||
if not isinstance(token_digest, str) or not hmac.compare_digest(
|
||||
token_digest, archive_sha256
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Archive contents changed after preview; preview it again"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/archive-preview", response_model=ArchivePreviewResponse)
|
||||
def preview_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
password: str | None = Form(default=None),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
try:
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
inspection = inspect_archive(
|
||||
archive_path,
|
||||
filename=filename,
|
||||
password=password,
|
||||
max_entries=settings.file_archive_max_entries,
|
||||
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
normalized_path = normalize_folder(path)
|
||||
preview_token = seal_transient_payload(
|
||||
{
|
||||
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||
"tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": target_owner,
|
||||
"path": normalized_path,
|
||||
"campaign_id": campaign_id or "",
|
||||
"archive_format": archive_format,
|
||||
"archive_sha256": digest,
|
||||
}
|
||||
)
|
||||
expires_at = datetime.now(UTC) + timedelta(
|
||||
seconds=settings.file_archive_preview_ttl_seconds
|
||||
)
|
||||
return ArchivePreviewResponse(
|
||||
preview_token=preview_token,
|
||||
archive_format=inspection.archive_format,
|
||||
entries=[
|
||||
ArchiveEntryResponse(
|
||||
path=entry.path,
|
||||
kind=entry.kind,
|
||||
size_bytes=entry.size_bytes,
|
||||
compressed_size_bytes=entry.compressed_size_bytes,
|
||||
encrypted=entry.encrypted,
|
||||
)
|
||||
for entry in inspection.entries
|
||||
],
|
||||
file_count=inspection.file_count,
|
||||
directory_count=inspection.directory_count,
|
||||
expanded_size_bytes=inspection.expanded_size_bytes,
|
||||
compressed_size_bytes=inspection.compressed_size_bytes,
|
||||
requires_password=inspection.requires_password,
|
||||
password_verified=inspection.password_verified,
|
||||
expires_at=expires_at.isoformat(),
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
|
||||
|
||||
@router.post("/archive-confirm", response_model=FileUploadResponse)
|
||||
def confirm_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
preview_token: str = Form(...),
|
||||
selected_paths_json: str = Form(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
password: str | None = Form(default=None),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||
default="reject"
|
||||
),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
source_provenance_json: str | None = Form(default=None),
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
try:
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json)
|
||||
if conflict_resolutions_json
|
||||
else []
|
||||
)
|
||||
upload_resolutions = _conflict_resolutions(
|
||||
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||
)
|
||||
selected_paths = _archive_selected_paths(selected_paths_json)
|
||||
_enforce_connector_policy(
|
||||
source_provenance_json,
|
||||
connector_policy_json,
|
||||
operation="import",
|
||||
)
|
||||
metadata = _source_metadata_from_form(
|
||||
source_provenance_json, source_revision
|
||||
)
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
normalized_path = normalize_folder(path)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
_validate_archive_preview_token(
|
||||
preview_token,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
path=normalized_path,
|
||||
campaign_id=campaign_id,
|
||||
archive_format=archive_format,
|
||||
archive_sha256=digest,
|
||||
)
|
||||
extracted = extract_archive_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
archive_data=archive_path,
|
||||
filename=filename,
|
||||
folder=normalized_path,
|
||||
campaign_id=campaign_id,
|
||||
selected_paths=selected_paths,
|
||||
password=password,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_entries=settings.file_archive_max_entries,
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||
)
|
||||
uploaded_assets = [item.asset for item in extracted]
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
session.commit()
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (
|
||||
FileStorageError,
|
||||
TransientPayloadError,
|
||||
UnsafeFilePathError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
|
||||
|
||||
@router.post("/upload", response_model=FileUploadResponse)
|
||||
def upload_files(
|
||||
files: list[UploadFile] = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
unpack_zip: bool = Form(default=False),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||
default="reject"
|
||||
),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
source_provenance_json: str | None = Form(default=None),
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
uploaded_assets: list[FileAsset] = []
|
||||
try:
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
)
|
||||
upload_resolutions = _conflict_resolutions(
|
||||
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||
)
|
||||
_enforce_connector_policy(
|
||||
source_provenance_json, connector_policy_json, operation="import"
|
||||
)
|
||||
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
|
||||
for upload in files:
|
||||
filename = upload.filename or "file"
|
||||
content_type = upload.content_type or None
|
||||
upload_limit = (
|
||||
settings.file_upload_zip_max_bytes
|
||||
if unpack_zip and filename.lower().endswith(".zip")
|
||||
else settings.file_upload_max_bytes
|
||||
)
|
||||
if unpack_zip and filename.lower().endswith(".zip"):
|
||||
zip_path = _spool_limited_upload_to_temp(
|
||||
upload, max_bytes=upload_limit, suffix=".zip"
|
||||
)
|
||||
try:
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=zip_path,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
finally:
|
||||
_cleanup_temp_file(zip_path)
|
||||
uploaded_assets.extend(item.asset for item in extracted)
|
||||
continue
|
||||
data = _read_limited_upload(upload, max_bytes=upload_limit)
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder=path,
|
||||
content_type=content_type,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
uploaded_assets.append(stored.asset)
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||
def upload_zip(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||
default="reject"
|
||||
),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
source_provenance_json: str | None = Form(default=None),
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
zip_path: str | None = None
|
||||
try:
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
)
|
||||
upload_resolutions = _conflict_resolutions(
|
||||
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||
)
|
||||
_enforce_connector_policy(
|
||||
source_provenance_json, connector_policy_json, operation="import"
|
||||
)
|
||||
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
|
||||
zip_path = _spool_limited_upload_to_temp(
|
||||
file, max_bytes=settings.file_upload_zip_max_bytes, suffix=".zip"
|
||||
)
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=zip_path,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
_audit_connector_imports(session, principal, [item.asset for item in extracted])
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if zip_path:
|
||||
_cleanup_temp_file(zip_path)
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, item.asset, include_shares=True)
|
||||
for item in extracted
|
||||
]
|
||||
)
|
||||
@@ -1,29 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from govoplan_core.core.runtime import ModuleRuntimeState
|
||||
|
||||
_runtime_settings: object | None = None
|
||||
_runtime = ModuleRuntimeState("Files")
|
||||
|
||||
|
||||
def configure_runtime(*, settings: object | None = None) -> None:
|
||||
global _runtime_settings
|
||||
if settings is not None:
|
||||
_runtime_settings = settings
|
||||
|
||||
|
||||
def get_settings() -> object:
|
||||
if _runtime_settings is not None:
|
||||
return _runtime_settings
|
||||
try:
|
||||
from govoplan_core.settings import settings as legacy_settings
|
||||
except ModuleNotFoundError as exc:
|
||||
raise RuntimeError("GovOPlaN Files runtime settings are not configured") from exc
|
||||
return legacy_settings
|
||||
|
||||
|
||||
class SettingsProxy:
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(get_settings(), name)
|
||||
|
||||
|
||||
settings = SettingsProxy()
|
||||
configure_runtime = _runtime.configure_runtime
|
||||
get_registry = _runtime.get_registry
|
||||
get_settings = _runtime.get_settings
|
||||
settings = _runtime.settings
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution
|
||||
|
||||
|
||||
@@ -13,19 +15,168 @@ class FileSpaceResponse(BaseModel):
|
||||
owner_type: Literal["user", "group"]
|
||||
owner_id: str
|
||||
description: str | None = None
|
||||
space_type: Literal["managed", "connector"] = "managed"
|
||||
connector_space_id: str | None = None
|
||||
connector_profile_id: str | None = None
|
||||
provider: str | None = None
|
||||
library_id: str | None = None
|
||||
remote_path: str | None = None
|
||||
sync_mode: str | None = None
|
||||
read_only: bool = False
|
||||
|
||||
|
||||
class FileSpacesResponse(BaseModel):
|
||||
spaces: list[FileSpaceResponse]
|
||||
|
||||
|
||||
class FileConnectorSpaceCreateRequest(BaseModel):
|
||||
owner_type: Literal["user", "group"] = "user"
|
||||
owner_id: str | None = None
|
||||
label: str
|
||||
connector_profile_id: str
|
||||
library_id: str | None = None
|
||||
remote_path: str = ""
|
||||
sync_mode: Literal["manual"] = "manual"
|
||||
read_only: bool = True
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorSpaceUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
library_id: str | None = None
|
||||
remote_path: str | None = None
|
||||
sync_mode: Literal["manual"] | None = None
|
||||
read_only: bool | None = None
|
||||
is_active: bool | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorSpaceResponse(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
owner_type: Literal["user", "group"]
|
||||
owner_id: str
|
||||
label: str
|
||||
connector_profile_id: str
|
||||
provider: str
|
||||
library_id: str | None = None
|
||||
remote_path: str = ""
|
||||
sync_mode: str
|
||||
read_only: bool
|
||||
is_active: bool
|
||||
created_at: str
|
||||
updated_at: str
|
||||
deleted_at: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorSpacesResponse(BaseModel):
|
||||
spaces: list[FileConnectorSpaceResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileShareResponse(BaseModel):
|
||||
id: str
|
||||
file_asset_id: str
|
||||
target_type: str
|
||||
target_id: str
|
||||
permission: str
|
||||
created_by_user_id: str | None = None
|
||||
created_at: str
|
||||
expires_at: str | None = None
|
||||
revoked_at: str | None = None
|
||||
revoked_by_user_id: str | None = None
|
||||
active: bool
|
||||
|
||||
|
||||
class FileSharesResponse(BaseModel):
|
||||
shares: list[FileShareResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileIntegrityScanCreateRequest(BaseModel):
|
||||
verify_checksums: bool = True
|
||||
batch_size: int = Field(default=100, ge=1, le=1000)
|
||||
|
||||
|
||||
class FileIntegrityScanResponse(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
storage_backend: str
|
||||
storage_prefix: str
|
||||
status: str
|
||||
revision: int
|
||||
phase: str
|
||||
verify_checksums: bool
|
||||
batch_size: int
|
||||
scanned_blob_count: int
|
||||
verified_blob_count: int
|
||||
quarantined_blob_count: int
|
||||
scanned_object_count: int
|
||||
orphan_object_count: int
|
||||
created_by_user_id: str | None = None
|
||||
started_at: str | None = None
|
||||
completed_at: str | None = None
|
||||
last_error: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class FileIntegrityScansResponse(BaseModel):
|
||||
scans: list[FileIntegrityScanResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileIntegrityFindingResponse(BaseModel):
|
||||
id: str
|
||||
scan_id: str
|
||||
tenant_id: str
|
||||
kind: str
|
||||
state: str
|
||||
revision: int
|
||||
blob_id: str | None = None
|
||||
storage_key: str
|
||||
expected_size_bytes: int | None = None
|
||||
observed_size_bytes: int | None = None
|
||||
expected_checksum_sha256: str | None = None
|
||||
observed_checksum_sha256: str | None = None
|
||||
resolved_at: str | None = None
|
||||
resolved_by_user_id: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class FileIntegrityFindingsResponse(BaseModel):
|
||||
findings: list[FileIntegrityFindingResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileIntegrityActionRequest(BaseModel):
|
||||
dry_run: bool = True
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileIntegrityScanRunRequest(BaseModel):
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileIntegrityActionResponse(BaseModel):
|
||||
action: str
|
||||
changed: bool
|
||||
dry_run: bool
|
||||
finding: FileIntegrityFindingResponse
|
||||
inspection_kind: str | None = None
|
||||
inspection_valid: bool | None = None
|
||||
|
||||
|
||||
class FileSourceProvenance(BaseModel):
|
||||
source_type: str | None = None
|
||||
connector_id: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
external_path: str | None = None
|
||||
external_url: str | None = None
|
||||
revision: str | None = None
|
||||
revision_label: str | None = None
|
||||
observed_at: str | None = None
|
||||
imported_at: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileAssetResponse(BaseModel):
|
||||
@@ -43,8 +194,14 @@ class FileAssetResponse(BaseModel):
|
||||
created_at: str
|
||||
updated_at: str
|
||||
deleted_at: str | None = None
|
||||
retained_until: str | None = None
|
||||
legal_hold: bool = False
|
||||
lifecycle_revision: int = 1
|
||||
lifecycle_reason: str | None = None
|
||||
audit_relevant: bool = False
|
||||
metadata: dict[str, Any] | None = None
|
||||
source_provenance: FileSourceProvenance | None = None
|
||||
source_revision: str | None = None
|
||||
shares: list[FileShareResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
@@ -61,6 +218,9 @@ class FileFolderResponse(BaseModel):
|
||||
|
||||
class FileFoldersResponse(BaseModel):
|
||||
folders: list[FileFolderResponse]
|
||||
cursor: str | None = None
|
||||
next_cursor: str | None = None
|
||||
watermark: str | None = None
|
||||
|
||||
|
||||
class FileFolderCreateRequest(BaseModel):
|
||||
@@ -83,12 +243,376 @@ class FileFolderDeleteResponse(BaseModel):
|
||||
|
||||
class FileListResponse(BaseModel):
|
||||
files: list[FileAssetResponse]
|
||||
total: int
|
||||
cursor: str | None = None
|
||||
next_cursor: str | None = None
|
||||
watermark: str | None = None
|
||||
|
||||
|
||||
class FileDeltaResponse(BaseModel):
|
||||
files: list[FileAssetResponse] = Field(default_factory=list)
|
||||
folders: list[FileFolderResponse] = Field(default_factory=list)
|
||||
deleted: list[DeltaDeletedItem] = Field(default_factory=list)
|
||||
watermark: str | None = None
|
||||
has_more: bool = False
|
||||
full: bool = False
|
||||
|
||||
|
||||
class FileUploadResponse(BaseModel):
|
||||
files: list[FileAssetResponse]
|
||||
|
||||
|
||||
class ArchiveEntryResponse(BaseModel):
|
||||
path: str
|
||||
kind: Literal["file", "directory"]
|
||||
size_bytes: int
|
||||
compressed_size_bytes: int | None = None
|
||||
encrypted: bool = False
|
||||
|
||||
|
||||
class ArchivePreviewResponse(BaseModel):
|
||||
preview_token: str
|
||||
archive_format: str
|
||||
entries: list[ArchiveEntryResponse] = Field(default_factory=list)
|
||||
file_count: int
|
||||
directory_count: int
|
||||
expanded_size_bytes: int
|
||||
compressed_size_bytes: int
|
||||
requires_password: bool
|
||||
password_verified: bool
|
||||
expires_at: str
|
||||
|
||||
|
||||
class FileConnectorPolicySource(BaseModel):
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||
scope_id: str | None = None
|
||||
label: str | None = None
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorPolicyEvaluateRequest(BaseModel):
|
||||
source_provenance: FileSourceProvenance
|
||||
operation: str = "access"
|
||||
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorPolicyEvaluateResponse(BaseModel):
|
||||
decision: dict[str, Any]
|
||||
|
||||
|
||||
class FileConnectorPolicyUpdateRequest(BaseModel):
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorPolicyStepResponse(BaseModel):
|
||||
scope_type: str
|
||||
scope_id: str | None = None
|
||||
path: str
|
||||
label: str
|
||||
applied_fields: list[str] = Field(default_factory=list)
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorPolicyResponse(BaseModel):
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"]
|
||||
scope_id: str | None = None
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
effective_policy: dict[str, Any] | None = None
|
||||
parent_policy: dict[str, Any] | None = None
|
||||
effective_policy_sources: list[FileConnectorPolicyStepResponse] = Field(default_factory=list)
|
||||
parent_policy_sources: list[FileConnectorPolicyStepResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorProfileResponse(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: str
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||
scope_id: str | None = None
|
||||
source_path: str
|
||||
credential_profile_id: str | None = None
|
||||
credential_profile_label: str | None = None
|
||||
credential_mode: str = "none"
|
||||
credential_source: str | None = None
|
||||
credentials_configured: bool = False
|
||||
username: str | None = None
|
||||
capabilities: list[str] = Field(default_factory=list)
|
||||
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
source_kind: str = "settings"
|
||||
|
||||
|
||||
class FileConnectorProfilesResponse(BaseModel):
|
||||
profiles: list[FileConnectorProfileResponse]
|
||||
|
||||
|
||||
class FileConnectorCredentialResponse(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: str | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||
scope_id: str | None = None
|
||||
source_path: str
|
||||
credential_mode: str = "none"
|
||||
credential_secret_source: str | None = None
|
||||
credentials_configured: bool = False
|
||||
username: str | None = None
|
||||
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
source_kind: str = "database"
|
||||
|
||||
|
||||
class FileConnectorCredentialsResponse(BaseModel):
|
||||
credentials: list[FileConnectorCredentialResponse]
|
||||
|
||||
|
||||
class FileConnectorSettingsDeltaResponse(BaseModel):
|
||||
profiles: list[FileConnectorProfileResponse] = Field(default_factory=list)
|
||||
credentials: list[FileConnectorCredentialResponse] = Field(default_factory=list)
|
||||
spaces: list[FileConnectorSpaceResponse] = Field(default_factory=list)
|
||||
policy: FileConnectorPolicyResponse | None = None
|
||||
changed_sections: list[str] = Field(default_factory=list)
|
||||
deleted: list[DeltaDeletedItem] = Field(default_factory=list)
|
||||
watermark: str | None = None
|
||||
has_more: bool = False
|
||||
full: bool = False
|
||||
|
||||
|
||||
class FileConnectorProfileCredentialsRequest(BaseModel):
|
||||
username: str | None = None
|
||||
password: str | None = None
|
||||
token: str | None = None
|
||||
password_env: str | None = None
|
||||
token_env: str | None = None
|
||||
secret_ref: str | None = None
|
||||
|
||||
|
||||
class FileConnectorDiscoveryCandidate(BaseModel):
|
||||
endpoint_url: str
|
||||
status: Literal["failed", "usable", "found", "credentials_rejected"]
|
||||
message: str
|
||||
|
||||
|
||||
class FileConnectorDiscoveryRequest(BaseModel):
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||
endpoint_url: str
|
||||
base_path: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
require_valid_credentials: bool = False
|
||||
|
||||
|
||||
class FileConnectorDiscoveryResponse(BaseModel):
|
||||
provider: str
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
status: Literal["usable", "found", "credentials_rejected", "not_found", "unsupported"]
|
||||
message: str
|
||||
candidates: list[FileConnectorDiscoveryCandidate] = Field(default_factory=list)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorProfileCreateRequest(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||
scope_id: str | None = None
|
||||
credential_profile_id: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||
capabilities: list[str] = Field(default_factory=lambda: ["browse", "import", "sync"])
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorProfileUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool | None = None
|
||||
credential_profile_id: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||
clear_password: bool = False
|
||||
clear_token: bool = False
|
||||
capabilities: list[str] | None = None
|
||||
policy: dict[str, Any] | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorCredentialCreateRequest(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||
scope_id: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorCredentialUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
enabled: bool | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||
clear_password: bool = False
|
||||
clear_token: bool = False
|
||||
policy: dict[str, Any] | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorProviderResponse(BaseModel):
|
||||
provider: str
|
||||
label: str
|
||||
protocol: str
|
||||
implemented: bool
|
||||
installed: bool
|
||||
browse_supported: bool
|
||||
import_supported: bool
|
||||
write_supported: bool = False
|
||||
optional_dependency: str | None = None
|
||||
permission_model: str
|
||||
sync_strategy: str
|
||||
conflict_strategy: str
|
||||
preview_strategy: str
|
||||
audit_events: list[str] = Field(default_factory=list)
|
||||
notes: str | None = None
|
||||
|
||||
|
||||
class FileConnectorProvidersResponse(BaseModel):
|
||||
providers: list[FileConnectorProviderResponse]
|
||||
|
||||
|
||||
class FileConnectorBrowseItem(BaseModel):
|
||||
kind: Literal["library", "folder", "file"]
|
||||
name: str
|
||||
path: str
|
||||
external_id: str | None = None
|
||||
external_url: str | None = None
|
||||
size_bytes: int | None = None
|
||||
content_type: str | None = None
|
||||
modified_at: str | None = None
|
||||
etag: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorBrowseResponse(BaseModel):
|
||||
profile_id: str
|
||||
provider: str
|
||||
path: str = ""
|
||||
library_id: str | None = None
|
||||
read_only: bool = True
|
||||
next_continuation_token: str | None = None
|
||||
has_more: bool = False
|
||||
decision: dict[str, Any]
|
||||
items: list[FileConnectorBrowseItem]
|
||||
|
||||
|
||||
class FileConnectorImportRequest(BaseModel):
|
||||
library_id: str
|
||||
path: str
|
||||
owner_type: Literal["user", "group"] = "user"
|
||||
owner_id: str | None = None
|
||||
target_folder: str | None = None
|
||||
target_path: str | None = None
|
||||
campaign_id: str | None = None
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename", "skip"] = "reject"
|
||||
source_revision: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorSyncResponse(BaseModel):
|
||||
file: FileAssetResponse
|
||||
action: Literal["created", "updated", "unchanged"]
|
||||
previous_version_id: str | None = None
|
||||
current_version_id: str
|
||||
|
||||
|
||||
class FileConnectorFolderSyncRequest(BaseModel):
|
||||
path: str = ""
|
||||
target_folder: str | None = None
|
||||
recursive: bool = True
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename", "skip"] = "skip"
|
||||
max_files: int = Field(default=100, ge=1, le=500)
|
||||
max_depth: int = Field(default=12, ge=0, le=50)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorFolderSyncItemResponse(BaseModel):
|
||||
source_path: str
|
||||
target_path: str | None = None
|
||||
action: Literal[
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"skipped",
|
||||
"conflict",
|
||||
"policy_denied",
|
||||
"failed",
|
||||
]
|
||||
file: FileAssetResponse | None = None
|
||||
previous_version_id: str | None = None
|
||||
current_version_id: str | None = None
|
||||
source_revision: str | None = None
|
||||
detail: str | None = None
|
||||
policy_decision: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorFolderSyncSummary(BaseModel):
|
||||
discovered: int = 0
|
||||
created: int = 0
|
||||
updated: int = 0
|
||||
unchanged: int = 0
|
||||
skipped: int = 0
|
||||
conflicts: int = 0
|
||||
policy_denied: int = 0
|
||||
failed: int = 0
|
||||
|
||||
|
||||
class FileConnectorFolderSyncResponse(BaseModel):
|
||||
connector_space_id: str
|
||||
connector_profile_id: str
|
||||
provider: str
|
||||
remote_path: str
|
||||
target_folder: str
|
||||
recursive: bool
|
||||
truncated: bool = False
|
||||
summary: FileConnectorFolderSyncSummary
|
||||
items: list[FileConnectorFolderSyncItemResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorWriteRequest(BaseModel):
|
||||
file_id: str
|
||||
remote_path: str
|
||||
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||
expected_revision: str | None = Field(default=None, max_length=500)
|
||||
|
||||
|
||||
class FileConnectorWriteResponse(BaseModel):
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool = False
|
||||
provider: str
|
||||
remote_path: str
|
||||
revision: str | None = None
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
|
||||
|
||||
class BulkDeleteRequest(BaseModel):
|
||||
file_ids: list[str]
|
||||
|
||||
@@ -97,6 +621,69 @@ class BulkDeleteResponse(BaseModel):
|
||||
deleted_count: int
|
||||
|
||||
|
||||
class FileLifecycleUpdateRequest(BaseModel):
|
||||
retained_until: datetime | None = None
|
||||
legal_hold: bool = False
|
||||
reason: str = Field(min_length=1, max_length=500)
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileRestoreResponse(BaseModel):
|
||||
restored_count: int
|
||||
|
||||
|
||||
class FileFolderRestoreResponse(BaseModel):
|
||||
restored_folders: int
|
||||
restored_files: int
|
||||
|
||||
|
||||
class FilePurgePreviewRequest(BaseModel):
|
||||
file_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class FilePurgePreviewItem(BaseModel):
|
||||
file_id: str
|
||||
filename: str
|
||||
lifecycle_revision: int
|
||||
deleted_at: str | None = None
|
||||
retained_until: str | None = None
|
||||
legal_hold: bool = False
|
||||
blockers: list[str] = Field(default_factory=list)
|
||||
blob_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FilePurgePreviewResponse(BaseModel):
|
||||
preview_sha256: str
|
||||
eligible: bool
|
||||
items: list[FilePurgePreviewItem]
|
||||
|
||||
|
||||
class FilePurgeExecuteRequest(FilePurgePreviewRequest):
|
||||
preview_sha256: str = Field(min_length=64, max_length=64)
|
||||
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||
confirmation: Literal["PURGE"]
|
||||
|
||||
|
||||
class FilePurgeResponse(BaseModel):
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool = False
|
||||
purged_files: int = 0
|
||||
released_blobs: int = 0
|
||||
|
||||
|
||||
class FileBlobGcRequest(BaseModel):
|
||||
limit: int = Field(default=25, ge=1, le=100)
|
||||
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||
|
||||
|
||||
class FileBlobGcResponse(BaseModel):
|
||||
inspected_blobs: int
|
||||
deleted_blobs: int
|
||||
unresolved_operation_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ConflictResolutionRequest(BaseModel):
|
||||
target_path: str
|
||||
action: Literal["overwrite", "rename", "skip"]
|
||||
@@ -111,6 +698,7 @@ class FileShareRequest(BaseModel):
|
||||
target_type: Literal["user", "group", "campaign", "tenant"]
|
||||
target_id: str
|
||||
permission: Literal["read", "write", "manage"] = "read"
|
||||
expires_at: datetime | None = None
|
||||
|
||||
|
||||
class BulkFileShareRequest(BaseModel):
|
||||
@@ -118,6 +706,7 @@ class BulkFileShareRequest(BaseModel):
|
||||
target_type: Literal["user", "group", "campaign", "tenant"]
|
||||
target_id: str
|
||||
permission: Literal["read", "write", "manage"] = "read"
|
||||
expires_at: datetime | None = None
|
||||
|
||||
|
||||
class BulkFileShareResponse(BaseModel):
|
||||
@@ -128,8 +717,8 @@ class BulkFileShareResponse(BaseModel):
|
||||
class RenameRequest(BaseModel):
|
||||
file_ids: list[str] = Field(default_factory=list)
|
||||
folder_paths: list[str] = Field(default_factory=list)
|
||||
owner_type: Literal["user", "group"] | None = None
|
||||
owner_id: str | None = None
|
||||
owner_type: Literal["user", "group"]
|
||||
owner_id: str
|
||||
mode: Literal["direct", "prefix", "suffix", "replace"]
|
||||
new_name: str | None = None
|
||||
find: str | None = None
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from urllib.parse import quote
|
||||
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.events import PlatformEvent
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_core.core.search import (
|
||||
SearchAuthorizationRequest,
|
||||
SearchBackfillPage,
|
||||
SearchBackfillRequest,
|
||||
SearchDocument,
|
||||
SearchIndexChange,
|
||||
SearchResourceReference,
|
||||
SearchResourceType,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
PROVIDER_ID = "files.objects"
|
||||
RESOURCE_MODELS = {
|
||||
"file": FileAsset,
|
||||
"folder": FileFolder,
|
||||
}
|
||||
READ_SCOPE = "files:file:read"
|
||||
ADMIN_SCOPE = "files:file:admin"
|
||||
|
||||
|
||||
class FilesSearchSource:
|
||||
def resource_types(self) -> Sequence[SearchResourceType]:
|
||||
return (
|
||||
SearchResourceType(
|
||||
provider_id=PROVIDER_ID,
|
||||
module_id="files",
|
||||
resource_type="file",
|
||||
label="Files",
|
||||
requires_authorization_recheck=True,
|
||||
),
|
||||
SearchResourceType(
|
||||
provider_id=PROVIDER_ID,
|
||||
module_id="files",
|
||||
resource_type="folder",
|
||||
label="File folders",
|
||||
requires_authorization_recheck=True,
|
||||
),
|
||||
)
|
||||
|
||||
def backfill(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
request: SearchBackfillRequest,
|
||||
) -> SearchBackfillPage:
|
||||
db = _session(session)
|
||||
model = _model(request.provider_id, request.resource_type)
|
||||
statement = select(model).where(
|
||||
model.tenant_id == request.tenant_id,
|
||||
model.deleted_at.is_(None),
|
||||
)
|
||||
if request.cursor:
|
||||
statement = statement.where(model.id > request.cursor)
|
||||
rows = list(
|
||||
db.scalars(
|
||||
statement.order_by(model.id).limit(request.limit + 1)
|
||||
)
|
||||
)
|
||||
has_more = len(rows) > request.limit
|
||||
selected = rows[: request.limit]
|
||||
shares = (
|
||||
_shares_by_asset(db, selected)
|
||||
if request.resource_type == "file"
|
||||
else {}
|
||||
)
|
||||
high_watermark = db.scalar(
|
||||
select(func.max(model.updated_at)).where(
|
||||
model.tenant_id == request.tenant_id,
|
||||
model.deleted_at.is_(None),
|
||||
)
|
||||
)
|
||||
return SearchBackfillPage(
|
||||
documents=tuple(
|
||||
_document(
|
||||
row,
|
||||
resource_type=request.resource_type,
|
||||
shares=shares.get(row.id, ()),
|
||||
)
|
||||
for row in selected
|
||||
),
|
||||
next_cursor=selected[-1].id if has_more and selected else None,
|
||||
complete=not has_more,
|
||||
high_watermark=(
|
||||
high_watermark.isoformat()
|
||||
if high_watermark is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def authorize(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
requests: Sequence[SearchAuthorizationRequest],
|
||||
) -> Mapping[str, bool]:
|
||||
decisions = {item.reference.key: False for item in requests}
|
||||
if not isinstance(principal, ApiPrincipal) or not (
|
||||
principal.has(READ_SCOPE) or principal.has(ADMIN_SCOPE)
|
||||
):
|
||||
return decisions
|
||||
db = _session(session)
|
||||
for request in requests:
|
||||
reference = request.reference
|
||||
if (
|
||||
reference.tenant_id != principal.tenant_id
|
||||
or reference.module_id != "files"
|
||||
or reference.resource_type not in RESOURCE_MODELS
|
||||
):
|
||||
continue
|
||||
decisions[reference.key] = _can_read(
|
||||
db,
|
||||
principal,
|
||||
reference=reference,
|
||||
)
|
||||
return decisions
|
||||
|
||||
def index_changes_for_event(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
event: PlatformEvent,
|
||||
delivery_key: str,
|
||||
) -> Sequence[SearchIndexChange]:
|
||||
if (
|
||||
event.module_id != "files"
|
||||
or event.tenant is None
|
||||
or event.resource is None
|
||||
or event.resource.id is None
|
||||
or event.resource.type not in RESOURCE_MODELS
|
||||
):
|
||||
return ()
|
||||
db = _session(session)
|
||||
reference = SearchResourceReference(
|
||||
tenant_id=event.tenant.id,
|
||||
module_id="files",
|
||||
resource_type=event.resource.type,
|
||||
resource_id=event.resource.id,
|
||||
)
|
||||
model = RESOURCE_MODELS[event.resource.type]
|
||||
row = db.get(model, event.resource.id)
|
||||
deleted = row is None or row.tenant_id != event.tenant.id or row.deleted_at is not None
|
||||
cursor = event.event_id
|
||||
document = None
|
||||
if not deleted:
|
||||
shares = (
|
||||
tuple(_active_shares(db, row.id))
|
||||
if event.resource.type == "file"
|
||||
else ()
|
||||
)
|
||||
document = _document(
|
||||
row,
|
||||
resource_type=event.resource.type,
|
||||
shares=shares,
|
||||
change_cursor=cursor,
|
||||
)
|
||||
return (
|
||||
SearchIndexChange(
|
||||
change_id=f"{delivery_key}:{PROVIDER_ID}:{event.resource.type}",
|
||||
provider_id=PROVIDER_ID,
|
||||
kind="delete" if deleted else "upsert",
|
||||
reference=reference,
|
||||
source_revision=(
|
||||
document.source_revision if document is not None else cursor
|
||||
),
|
||||
cursor=cursor,
|
||||
document=document,
|
||||
occurred_at=event.occurred_at,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def create_files_search_source(_context: ModuleContext) -> FilesSearchSource:
|
||||
return FilesSearchSource()
|
||||
|
||||
|
||||
def _model(provider_id: str, resource_type: str):
|
||||
if provider_id != PROVIDER_ID or resource_type not in RESOURCE_MODELS:
|
||||
raise ValueError("Unsupported Files search source.")
|
||||
return RESOURCE_MODELS[resource_type]
|
||||
|
||||
|
||||
def _document(
|
||||
row: FileAsset | FileFolder,
|
||||
*,
|
||||
resource_type: str,
|
||||
shares: Sequence[FileShare] = (),
|
||||
change_cursor: str | None = None,
|
||||
) -> SearchDocument:
|
||||
is_file = isinstance(row, FileAsset)
|
||||
title = row.filename if is_file else (row.path.rsplit("/", 1)[-1] or row.path)
|
||||
path = row.display_path if is_file else row.path
|
||||
owner_id = row.owner_user_id if row.owner_type == "user" else row.owner_group_id
|
||||
tokens = [f"scope:{READ_SCOPE}", f"scope:{ADMIN_SCOPE}"]
|
||||
if owner_id:
|
||||
tokens.append(
|
||||
f"membership:{owner_id}"
|
||||
if row.owner_type == "user"
|
||||
else f"group:{owner_id}"
|
||||
)
|
||||
for share in shares:
|
||||
prefix = "membership" if share.target_type == "user" else share.target_type
|
||||
if prefix in {"membership", "group", "tenant"}:
|
||||
tokens.append(f"{prefix}:{share.target_id}")
|
||||
updated_at = row.updated_at or row.created_at
|
||||
revision = (
|
||||
f"{row.current_version_id or 'none'}:{updated_at.isoformat()}"
|
||||
if is_file
|
||||
else updated_at.isoformat()
|
||||
)
|
||||
return SearchDocument(
|
||||
tenant_id=row.tenant_id,
|
||||
module_id="files",
|
||||
provider_id=PROVIDER_ID,
|
||||
resource_type=resource_type,
|
||||
resource_id=row.id,
|
||||
title=title,
|
||||
url=f"/files?{resource_type}Id={quote(row.id, safe='')}",
|
||||
summary=((row.description or "") if is_file else path)[:4000] or None,
|
||||
body=" ".join(
|
||||
value for value in (path, row.description if is_file else None) if value
|
||||
)[:200_000],
|
||||
keywords=(path[:200], row.owner_type[:200]),
|
||||
visibility="restricted",
|
||||
acl_tokens=tuple(dict.fromkeys(tokens)),
|
||||
metadata={
|
||||
"path": path,
|
||||
"owner_type": row.owner_type,
|
||||
"current_version_id": row.current_version_id if is_file else None,
|
||||
},
|
||||
source_revision=revision,
|
||||
change_cursor=change_cursor,
|
||||
source_updated_at=updated_at,
|
||||
requires_authorization_recheck=True,
|
||||
)
|
||||
|
||||
|
||||
def _can_read(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
reference: SearchResourceReference,
|
||||
) -> bool:
|
||||
model = RESOURCE_MODELS[reference.resource_type]
|
||||
row = session.get(model, reference.resource_id)
|
||||
if row is None or row.tenant_id != principal.tenant_id or row.deleted_at is not None:
|
||||
return False
|
||||
if principal.has(ADMIN_SCOPE):
|
||||
return True
|
||||
user_id = str(getattr(principal.user, "id", "") or principal.membership_id or "")
|
||||
if row.owner_type == "user" and row.owner_user_id == user_id:
|
||||
return True
|
||||
if row.owner_type == "group" and row.owner_group_id in principal.group_ids:
|
||||
return True
|
||||
if reference.resource_type != "file":
|
||||
return False
|
||||
target_clauses = [
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
(FileShare.target_type == "tenant")
|
||||
& (FileShare.target_id == principal.tenant_id),
|
||||
]
|
||||
if principal.group_ids:
|
||||
target_clauses.append(
|
||||
(FileShare.target_type == "group")
|
||||
& (FileShare.target_id.in_(tuple(principal.group_ids)))
|
||||
)
|
||||
return session.scalar(
|
||||
select(FileShare.id).where(
|
||||
FileShare.tenant_id == principal.tenant_id,
|
||||
FileShare.file_asset_id == row.id,
|
||||
effective_file_share_clause(),
|
||||
or_(*target_clauses),
|
||||
).limit(1)
|
||||
) is not None
|
||||
|
||||
|
||||
def _active_shares(session: Session, asset_id: str) -> Sequence[FileShare]:
|
||||
return tuple(
|
||||
session.scalars(
|
||||
select(FileShare).where(
|
||||
FileShare.file_asset_id == asset_id,
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _shares_by_asset(
|
||||
session: Session,
|
||||
rows: Sequence[FileAsset | FileFolder],
|
||||
) -> dict[str, tuple[FileShare, ...]]:
|
||||
asset_ids = [row.id for row in rows if isinstance(row, FileAsset)]
|
||||
grouped: dict[str, list[FileShare]] = {asset_id: [] for asset_id in asset_ids}
|
||||
if not asset_ids:
|
||||
return {}
|
||||
for share in session.scalars(
|
||||
select(FileShare).where(
|
||||
FileShare.file_asset_id.in_(asset_ids),
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
):
|
||||
grouped.setdefault(share.file_asset_id, []).append(share)
|
||||
return {key: tuple(value) for key, value in grouped.items()}
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files search requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FilesSearchSource",
|
||||
"PROVIDER_ID",
|
||||
"create_files_search_source",
|
||||
]
|
||||
@@ -0,0 +1 @@
|
||||
"""Query and response assembly services used by Files routes."""
|
||||
@@ -0,0 +1,292 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||
from govoplan_core.core.change_sequence import (
|
||||
ChangeSequenceEntry,
|
||||
)
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorCredentialResponse,
|
||||
FileConnectorSettingsDeltaResponse,
|
||||
FileConnectorPolicyResponse,
|
||||
FileConnectorProfileResponse,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileConnectorSpace
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
ConnectorCredential,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
connector_policy_response,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
_can_read_disabled_connector_profiles,
|
||||
_connector_deleted_item,
|
||||
_connector_space_response,
|
||||
_ensure_campaign_file_access,
|
||||
_file_connector_settings_response_watermark,
|
||||
_file_connector_settings_watermark,
|
||||
_visible_connector_credentials,
|
||||
_visible_connector_profiles,
|
||||
_visible_connector_spaces,
|
||||
)
|
||||
|
||||
|
||||
def _full_file_connector_settings_delta_response(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
provider: str | None,
|
||||
campaign_id: str | None,
|
||||
include_disabled: bool,
|
||||
include_inactive: bool,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> FileConnectorSettingsDeltaResponse:
|
||||
if campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
credentials = _visible_connector_credentials(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
spaces = _visible_connector_spaces(
|
||||
session,
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
return FileConnectorSettingsDeltaResponse(
|
||||
profiles=[
|
||||
FileConnectorProfileResponse(**profile.to_response())
|
||||
for profile in profiles
|
||||
],
|
||||
credentials=[
|
||||
FileConnectorCredentialResponse(**credential.to_response())
|
||||
for credential in credentials
|
||||
],
|
||||
spaces=[_connector_space_response(space) for space in spaces],
|
||||
policy=FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
),
|
||||
changed_sections=["profiles", "credentials", "spaces", "policy"],
|
||||
deleted=[],
|
||||
watermark=_file_connector_settings_watermark(
|
||||
session, tenant_id=principal.tenant_id
|
||||
),
|
||||
has_more=False,
|
||||
full=True,
|
||||
)
|
||||
|
||||
|
||||
def _changed_file_connector_setting_ids(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
) -> tuple[set[str], set[str], set[str], bool]:
|
||||
changed_profile_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION
|
||||
and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
}
|
||||
changed_credential_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION
|
||||
and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
}
|
||||
changed_space_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION
|
||||
and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
}
|
||||
policy_changed = any(
|
||||
entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries
|
||||
)
|
||||
return (
|
||||
changed_profile_ids,
|
||||
changed_credential_ids,
|
||||
changed_space_ids,
|
||||
policy_changed,
|
||||
)
|
||||
|
||||
|
||||
def _file_connector_settings_changed_sections(
|
||||
*,
|
||||
changed_profile_ids: set[str],
|
||||
changed_credential_ids: set[str],
|
||||
changed_space_ids: set[str],
|
||||
policy_changed: bool,
|
||||
profiles: list[ConnectorProfile],
|
||||
) -> list[str]:
|
||||
changed_sections = []
|
||||
if changed_profile_ids or any(
|
||||
profile.credential_profile_id
|
||||
and profile.credential_profile_id in changed_credential_ids
|
||||
for profile in profiles
|
||||
):
|
||||
changed_sections.append("profiles")
|
||||
if changed_credential_ids:
|
||||
changed_sections.append("credentials")
|
||||
if changed_space_ids:
|
||||
changed_sections.append("spaces")
|
||||
if policy_changed:
|
||||
changed_sections.append("policy")
|
||||
return changed_sections
|
||||
|
||||
|
||||
def _file_connector_settings_deleted_items(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
*,
|
||||
visible_profiles: dict[str, ConnectorProfile],
|
||||
visible_credentials: dict[str, ConnectorCredential],
|
||||
visible_spaces: dict[str, FileConnectorSpace],
|
||||
) -> list[DeltaDeletedItem]:
|
||||
return [
|
||||
_connector_deleted_item(entry)
|
||||
for entry in entries
|
||||
if (
|
||||
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
and entry.resource_id not in visible_profiles
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
and entry.resource_id not in visible_credentials
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
and entry.resource_id not in visible_spaces
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _incremental_file_connector_settings_delta_response(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
entries: list[ChangeSequenceEntry],
|
||||
has_more: bool,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
provider: str | None,
|
||||
campaign_id: str | None,
|
||||
include_disabled: bool,
|
||||
include_inactive: bool,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> FileConnectorSettingsDeltaResponse:
|
||||
changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed = (
|
||||
_changed_file_connector_setting_ids(entries)
|
||||
)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
visible_profiles = {
|
||||
profile.id: profile
|
||||
for profile in profiles
|
||||
if profile.id in changed_profile_ids
|
||||
or (
|
||||
profile.credential_profile_id
|
||||
and profile.credential_profile_id in changed_credential_ids
|
||||
)
|
||||
}
|
||||
credentials = _visible_connector_credentials(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
visible_credentials = {
|
||||
credential.id: credential
|
||||
for credential in credentials
|
||||
if credential.id in changed_credential_ids
|
||||
}
|
||||
spaces = _visible_connector_spaces(
|
||||
session,
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
visible_spaces = {
|
||||
space.id: space for space in spaces if space.id in changed_space_ids
|
||||
}
|
||||
return FileConnectorSettingsDeltaResponse(
|
||||
profiles=[
|
||||
FileConnectorProfileResponse(**profile.to_response())
|
||||
for profile in visible_profiles.values()
|
||||
],
|
||||
credentials=[
|
||||
FileConnectorCredentialResponse(**credential.to_response())
|
||||
for credential in visible_credentials.values()
|
||||
],
|
||||
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
|
||||
policy=FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
if policy_changed
|
||||
else None,
|
||||
changed_sections=_file_connector_settings_changed_sections(
|
||||
changed_profile_ids=changed_profile_ids,
|
||||
changed_credential_ids=changed_credential_ids,
|
||||
changed_space_ids=changed_space_ids,
|
||||
policy_changed=policy_changed,
|
||||
profiles=profiles,
|
||||
),
|
||||
deleted=_file_connector_settings_deleted_items(
|
||||
entries,
|
||||
visible_profiles=visible_profiles,
|
||||
visible_credentials=visible_credentials,
|
||||
visible_spaces=visible_spaces,
|
||||
),
|
||||
watermark=_file_connector_settings_response_watermark(
|
||||
session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more
|
||||
),
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
)
|
||||
@@ -0,0 +1,620 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||
from govoplan_core.core.change_sequence import (
|
||||
ChangeSequenceEntry,
|
||||
decode_sequence_watermark,
|
||||
encode_sequence_watermark,
|
||||
max_sequence_id,
|
||||
sequence_entries_since,
|
||||
sequence_watermark_is_expired,
|
||||
)
|
||||
from govoplan_core.core.pagination import (
|
||||
KeysetCursorError,
|
||||
decode_keyset_cursor,
|
||||
encode_keyset_cursor,
|
||||
keyset_query_fingerprint,
|
||||
)
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_ASSETS_COLLECTION,
|
||||
FILES_FOLDERS_COLLECTION,
|
||||
FILES_MODULE_ID,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileDeltaResponse,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_files.backend.storage.access import user_group_ids
|
||||
from govoplan_files.backend.storage.files import (
|
||||
list_assets_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.folders import list_folders_for_user
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_list_response,
|
||||
_folder_response,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
_FILES_DELTA_COLLECTIONS = (FILES_ASSETS_COLLECTION, FILES_FOLDERS_COLLECTION)
|
||||
FILES_LIST_CURSOR_SCOPE = "files.list.v1"
|
||||
FOLDERS_LIST_CURSOR_SCOPE = "files.folders.list.v1"
|
||||
DEFAULT_FILE_LIST_PAGE_SIZE = 500
|
||||
|
||||
|
||||
def _cursor_http_error(exc: Exception) -> HTTPException:
|
||||
return HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc))
|
||||
|
||||
|
||||
def _cursor_page_size(
|
||||
scope: str, cursor: str | None, explicit_page_size: int | None
|
||||
) -> int | None:
|
||||
if explicit_page_size is not None:
|
||||
return explicit_page_size
|
||||
if not cursor:
|
||||
return DEFAULT_FILE_LIST_PAGE_SIZE
|
||||
try:
|
||||
values = decode_keyset_cursor(scope, cursor)
|
||||
except KeysetCursorError as exc:
|
||||
raise _cursor_http_error(exc) from exc
|
||||
raw_page_size = values.get("page_size")
|
||||
if not isinstance(raw_page_size, int) or raw_page_size < 1 or raw_page_size > 1000:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
)
|
||||
return raw_page_size
|
||||
|
||||
|
||||
def _files_list_fingerprint(
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None,
|
||||
audit_relevant: bool | None,
|
||||
page_size: int,
|
||||
) -> str:
|
||||
return keyset_query_fingerprint(
|
||||
FILES_LIST_CURSOR_SCOPE,
|
||||
{
|
||||
"tenant_id": principal.tenant_id,
|
||||
"actor": "admin" if _is_admin(principal) else principal.user.id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"campaign_id": campaign_id,
|
||||
"path_prefix": path_prefix or "",
|
||||
"campaign_usage": campaign_usage or "",
|
||||
"audit_relevant": audit_relevant,
|
||||
"sort": "display_path.asc,updated_at.desc,id.asc",
|
||||
"page_size": page_size,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _folders_list_fingerprint(
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
page_size: int,
|
||||
) -> str:
|
||||
return keyset_query_fingerprint(
|
||||
FOLDERS_LIST_CURSOR_SCOPE,
|
||||
{
|
||||
"tenant_id": principal.tenant_id,
|
||||
"actor": "admin" if _is_admin(principal) else principal.user.id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"sort": "path.asc,id.asc",
|
||||
"page_size": page_size,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _file_cursor_values(
|
||||
cursor: str | None, *, fingerprint: str
|
||||
) -> tuple[str | None, datetime | None, str | None]:
|
||||
try:
|
||||
values = decode_keyset_cursor(
|
||||
FILES_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
|
||||
)
|
||||
except KeysetCursorError as exc:
|
||||
raise _cursor_http_error(exc) from exc
|
||||
if values is None:
|
||||
return None, None, None
|
||||
display_path = values.get("display_path")
|
||||
updated_at = values.get("updated_at")
|
||||
asset_id = values.get("id")
|
||||
if (
|
||||
not isinstance(display_path, str)
|
||||
or not isinstance(updated_at, str)
|
||||
or not isinstance(asset_id, str)
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
)
|
||||
try:
|
||||
parsed_updated_at = datetime.fromisoformat(updated_at)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
) from exc
|
||||
return display_path, parsed_updated_at, asset_id
|
||||
|
||||
|
||||
def _folder_cursor_values(
|
||||
cursor: str | None, *, fingerprint: str
|
||||
) -> tuple[str | None, str | None]:
|
||||
try:
|
||||
values = decode_keyset_cursor(
|
||||
FOLDERS_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
|
||||
)
|
||||
except KeysetCursorError as exc:
|
||||
raise _cursor_http_error(exc) from exc
|
||||
if values is None:
|
||||
return None, None
|
||||
folder_path = values.get("path")
|
||||
folder_id = values.get("id")
|
||||
if not isinstance(folder_path, str) or not isinstance(folder_id, str):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
)
|
||||
return folder_path, folder_id
|
||||
|
||||
|
||||
def _next_file_list_cursor(
|
||||
principal: ApiPrincipal,
|
||||
assets: list[FileAsset],
|
||||
*,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None,
|
||||
audit_relevant: bool | None,
|
||||
page_size: int,
|
||||
has_more: bool,
|
||||
) -> str | None:
|
||||
if not has_more or not assets:
|
||||
return None
|
||||
last = assets[-1]
|
||||
return encode_keyset_cursor(
|
||||
FILES_LIST_CURSOR_SCOPE,
|
||||
fingerprint=_files_list_fingerprint(
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
page_size=page_size,
|
||||
),
|
||||
values={
|
||||
"display_path": last.display_path,
|
||||
"updated_at": last.updated_at,
|
||||
"id": last.id,
|
||||
"page_size": page_size,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _next_folder_list_cursor(
|
||||
principal: ApiPrincipal,
|
||||
folders: list[FileFolder],
|
||||
*,
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
page_size: int,
|
||||
has_more: bool,
|
||||
) -> str | None:
|
||||
if not has_more or not folders:
|
||||
return None
|
||||
last = folders[-1]
|
||||
return encode_keyset_cursor(
|
||||
FOLDERS_LIST_CURSOR_SCOPE,
|
||||
fingerprint=_folders_list_fingerprint(
|
||||
principal, owner_type=owner_type, owner_id=owner_id, page_size=page_size
|
||||
),
|
||||
values={"path": last.path, "id": last.id, "page_size": page_size},
|
||||
)
|
||||
|
||||
|
||||
def _files_delta_watermark(session: Session, tenant_id: str) -> str:
|
||||
return encode_sequence_watermark(
|
||||
max_sequence_id(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collections=_FILES_DELTA_COLLECTIONS,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _full_file_delta_response(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> FileDeltaResponse:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
folders = _visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
return FileDeltaResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
folders=[_folder_response(folder) for folder in folders],
|
||||
deleted=[],
|
||||
watermark=_files_delta_watermark(session, principal.tenant_id),
|
||||
has_more=False,
|
||||
full=True,
|
||||
)
|
||||
|
||||
|
||||
def _visible_folders_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> list[FileFolder]:
|
||||
if not owner_type or not owner_id:
|
||||
return []
|
||||
folders = list_folders_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if not path_prefix:
|
||||
return folders
|
||||
normalized = path_prefix.strip().strip("/")
|
||||
if not normalized:
|
||||
return folders
|
||||
return [
|
||||
folder
|
||||
for folder in folders
|
||||
if folder.path == normalized or folder.path.startswith(f"{normalized}/")
|
||||
]
|
||||
|
||||
|
||||
def _entry_path_matches(
|
||||
entry_payload: dict[str, object], path_prefix: str | None
|
||||
) -> bool:
|
||||
if not path_prefix:
|
||||
return True
|
||||
normalized = path_prefix.strip().strip("/")
|
||||
if not normalized:
|
||||
return True
|
||||
for key in ("path", "previous_path"):
|
||||
value = entry_payload.get(key)
|
||||
if isinstance(value, str) and (
|
||||
value == normalized or value.startswith(f"{normalized}/")
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _entry_owner_matches(
|
||||
entry_payload: dict[str, object],
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> bool:
|
||||
if not owner_type or not owner_id:
|
||||
return True
|
||||
return (
|
||||
entry_payload.get("owner_type") == owner_type
|
||||
and entry_payload.get("owner_id") == owner_id
|
||||
)
|
||||
|
||||
|
||||
def _entry_campaign_matches(session: Session, entry, campaign_id: str | None) -> bool:
|
||||
if not campaign_id:
|
||||
return True
|
||||
payload = entry.payload or {}
|
||||
if (
|
||||
payload.get("share_target_type") == "campaign"
|
||||
and payload.get("share_target_id") == campaign_id
|
||||
):
|
||||
return True
|
||||
if entry.resource_type != "file":
|
||||
return False
|
||||
return (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == entry.tenant_id,
|
||||
FileShare.file_asset_id == entry.resource_id,
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _principal_group_ids_for_delta(
|
||||
session: Session, principal: ApiPrincipal
|
||||
) -> set[str]:
|
||||
cache = session.info.setdefault("files_delta_group_ids", {})
|
||||
key = (principal.tenant_id, principal.user.id)
|
||||
if key not in cache:
|
||||
cache[key] = set(
|
||||
user_group_ids(
|
||||
session, tenant_id=principal.tenant_id, user_id=principal.user.id
|
||||
)
|
||||
)
|
||||
return cache[key]
|
||||
|
||||
|
||||
def _entry_subject_matches_principal(
|
||||
session: Session, principal: ApiPrincipal, entry_payload: dict[str, object]
|
||||
) -> bool:
|
||||
if _is_admin(principal):
|
||||
return True
|
||||
owner_type = entry_payload.get("owner_type")
|
||||
owner_id = entry_payload.get("owner_id")
|
||||
if owner_type == "user" and owner_id == principal.user.id:
|
||||
return True
|
||||
if owner_type == "group" and isinstance(owner_id, str):
|
||||
if owner_id in _principal_group_ids_for_delta(session, principal):
|
||||
return True
|
||||
share_target_type = entry_payload.get("share_target_type")
|
||||
share_target_id = entry_payload.get("share_target_id")
|
||||
if share_target_type == "user" and share_target_id == principal.user.id:
|
||||
return True
|
||||
if share_target_type == "tenant" and share_target_id == principal.tenant_id:
|
||||
return True
|
||||
if share_target_type == "group" and isinstance(share_target_id, str):
|
||||
if share_target_id in _principal_group_ids_for_delta(session, principal):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _entry_matches_delta_scope(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
entry,
|
||||
*,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> bool:
|
||||
payload = entry.payload or {}
|
||||
if (
|
||||
not owner_type
|
||||
and not campaign_id
|
||||
and not _entry_subject_matches_principal(session, principal, payload)
|
||||
):
|
||||
return False
|
||||
return (
|
||||
_entry_owner_matches(payload, owner_type, owner_id)
|
||||
and _entry_path_matches(payload, path_prefix)
|
||||
and _entry_campaign_matches(session, entry, campaign_id)
|
||||
)
|
||||
|
||||
|
||||
def _decode_files_delta_watermark(since: str) -> int:
|
||||
try:
|
||||
return decode_sequence_watermark(since)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)
|
||||
) from exc
|
||||
|
||||
|
||||
def _changed_delta_resource_ids(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
) -> tuple[list[str], list[str]]:
|
||||
file_ids = list(
|
||||
dict.fromkeys(
|
||||
entry.resource_id for entry in entries if entry.resource_type == "file"
|
||||
)
|
||||
)
|
||||
folder_ids = list(
|
||||
dict.fromkeys(
|
||||
entry.resource_id for entry in entries if entry.resource_type == "folder"
|
||||
)
|
||||
)
|
||||
return file_ids, folder_ids
|
||||
|
||||
|
||||
def _visible_assets_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
changed_file_ids: list[str],
|
||||
) -> dict[str, FileAsset]:
|
||||
return {
|
||||
asset.id: asset
|
||||
for asset in list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if asset.id in changed_file_ids
|
||||
}
|
||||
|
||||
|
||||
def _changed_visible_folders_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
path_prefix: str | None,
|
||||
changed_folder_ids: list[str],
|
||||
) -> dict[str, FileFolder]:
|
||||
return {
|
||||
folder.id: folder
|
||||
for folder in _visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
if folder.id in changed_folder_ids
|
||||
}
|
||||
|
||||
|
||||
def _deleted_delta_items(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
entries: list[ChangeSequenceEntry],
|
||||
visible_assets: dict[str, FileAsset],
|
||||
visible_folders: dict[str, FileFolder],
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> list[DeltaDeletedItem]:
|
||||
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
|
||||
for entry in entries:
|
||||
if entry.resource_type == "file" and entry.resource_id in visible_assets:
|
||||
continue
|
||||
if entry.resource_type == "folder" and entry.resource_id in visible_folders:
|
||||
continue
|
||||
if not _entry_matches_delta_scope(
|
||||
session,
|
||||
principal,
|
||||
entry,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
):
|
||||
continue
|
||||
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
|
||||
id=entry.resource_id,
|
||||
resource_type=entry.resource_type,
|
||||
revision=encode_sequence_watermark(entry.id),
|
||||
deleted_at=entry.created_at if entry.operation == "deleted" else None,
|
||||
)
|
||||
return list(deleted.values())
|
||||
|
||||
|
||||
def _files_delta_response(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
since: str,
|
||||
limit: int,
|
||||
) -> FileDeltaResponse:
|
||||
since_sequence = _decode_files_delta_watermark(since)
|
||||
if sequence_watermark_is_expired(
|
||||
session,
|
||||
since=since_sequence,
|
||||
tenant_id=principal.tenant_id,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collections=_FILES_DELTA_COLLECTIONS,
|
||||
):
|
||||
return _full_file_delta_response(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
|
||||
entries_plus_one = sequence_entries_since(
|
||||
session,
|
||||
since=since_sequence,
|
||||
tenant_id=principal.tenant_id,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collections=_FILES_DELTA_COLLECTIONS,
|
||||
limit=limit + 1,
|
||||
)
|
||||
has_more = len(entries_plus_one) > limit
|
||||
entries = entries_plus_one[:limit]
|
||||
|
||||
changed_file_ids, changed_folder_ids = _changed_delta_resource_ids(entries)
|
||||
visible_assets = _visible_assets_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
changed_file_ids=changed_file_ids,
|
||||
)
|
||||
visible_folders = _changed_visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
changed_folder_ids=changed_folder_ids,
|
||||
)
|
||||
deleted = _deleted_delta_items(
|
||||
session,
|
||||
principal=principal,
|
||||
entries=entries,
|
||||
visible_assets=visible_assets,
|
||||
visible_folders=visible_folders,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
|
||||
watermark = (
|
||||
encode_sequence_watermark(entries[-1].id)
|
||||
if has_more and entries
|
||||
else _files_delta_watermark(session, principal.tenant_id)
|
||||
)
|
||||
return FileDeltaResponse(
|
||||
files=_asset_list_response(
|
||||
session, list(visible_assets.values()), include_shares=True
|
||||
),
|
||||
folders=[_folder_response(folder) for folder in visible_folders.values()],
|
||||
deleted=deleted,
|
||||
watermark=watermark,
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
)
|
||||
@@ -2,33 +2,50 @@ from __future__ import annotations
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.db.models import Group, UserGroupMembership
|
||||
from govoplan_core.core.access import (
|
||||
CAPABILITY_ACCESS_DIRECTORY,
|
||||
AccessDirectory,
|
||||
GroupRef,
|
||||
)
|
||||
from govoplan_core.core.runtime import get_registry
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
def _access_directory() -> AccessDirectory:
|
||||
registry = get_registry()
|
||||
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||
raise FileStorageError("Access directory capability is not configured")
|
||||
capability = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||
if not isinstance(capability, AccessDirectory):
|
||||
raise FileStorageError("Access directory capability is invalid")
|
||||
return capability
|
||||
|
||||
|
||||
def group_refs_for_ids(*, tenant_id: str, group_ids: list[str]) -> list[GroupRef]:
|
||||
if not group_ids:
|
||||
return []
|
||||
groups = _access_directory().get_groups(group_ids)
|
||||
return sorted(
|
||||
[group for group in groups.values() if group.tenant_id == tenant_id],
|
||||
key=lambda group: group.name.casefold(),
|
||||
)
|
||||
|
||||
|
||||
def user_group_ids(session: Session, *, tenant_id: str, user_id: str, include_admin_groups: bool = False) -> list[str]:
|
||||
del session
|
||||
directory = _access_directory()
|
||||
if include_admin_groups:
|
||||
return [row.id for row in session.query(Group).filter(Group.tenant_id == tenant_id).order_by(Group.name.asc()).all()]
|
||||
return [
|
||||
row.group_id
|
||||
for row in session.query(UserGroupMembership)
|
||||
.filter(UserGroupMembership.tenant_id == tenant_id, UserGroupMembership.user_id == user_id)
|
||||
.all()
|
||||
]
|
||||
return [group.id for group in directory.groups_for_tenant(tenant_id)]
|
||||
return [group.id for group in directory.groups_for_user(user_id, tenant_id=tenant_id)]
|
||||
|
||||
|
||||
def ensure_group_access(session: Session, *, tenant_id: str, group_id: str, user_id: str, is_admin: bool = False) -> None:
|
||||
group = session.get(Group, group_id)
|
||||
group = _access_directory().get_group(group_id)
|
||||
if not group or group.tenant_id != tenant_id:
|
||||
raise FileStorageError("Group not found")
|
||||
if is_admin:
|
||||
return
|
||||
membership = (
|
||||
session.query(UserGroupMembership)
|
||||
.filter(UserGroupMembership.tenant_id == tenant_id, UserGroupMembership.user_id == user_id, UserGroupMembership.group_id == group_id)
|
||||
.one_or_none()
|
||||
)
|
||||
if membership is None:
|
||||
if group_id not in user_group_ids(session, tenant_id=tenant_id, user_id=user_id):
|
||||
raise FileStorageError("No access to this group file space")
|
||||
|
||||
|
||||
@@ -42,3 +59,21 @@ def ensure_owner_access(session: Session, *, tenant_id: str, owner_type: str, ow
|
||||
ensure_group_access(session, tenant_id=tenant_id, group_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
return
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def ensure_share_target_exists(*, tenant_id: str, target_type: str, target_id: str) -> None:
|
||||
target_type = target_type.lower().strip()
|
||||
if target_type == "user":
|
||||
user = _access_directory().get_user(target_id)
|
||||
if not user or user.tenant_id != tenant_id or user.status != "active":
|
||||
raise FileStorageError("User not found")
|
||||
return
|
||||
if target_type == "group":
|
||||
group = _access_directory().get_group(target_id)
|
||||
if not group or group.tenant_id != tenant_id or group.status != "active":
|
||||
raise FileStorageError("Group not found")
|
||||
return
|
||||
if target_type == "tenant":
|
||||
if target_id != tenant_id:
|
||||
raise FileStorageError("Tenant not found")
|
||||
return
|
||||
|
||||
@@ -1,53 +1,93 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import mimetypes
|
||||
import re
|
||||
import stat
|
||||
import tarfile
|
||||
import zipfile
|
||||
from dataclasses import dataclass
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
from os import PathLike
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any, BinaryIO, Iterable, Literal
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
||||
from govoplan_files.backend.storage.files import create_file_asset, current_version_and_blob
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, normalize_folder, normalize_logical_path
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import (
|
||||
FileConflictResolution,
|
||||
FileStorageError,
|
||||
UploadedStoredFile,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
current_versions_and_blobs,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
filename_from_path,
|
||||
normalize_folder,
|
||||
normalize_logical_path,
|
||||
)
|
||||
|
||||
|
||||
_ZIP_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_ARCHIVE_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:")
|
||||
ARCHIVE_UPLOAD_MAX_ENTRIES = 10_000
|
||||
# Kept for callers and documentation using the former ZIP-specific name.
|
||||
ZIP_UPLOAD_MAX_FILES = ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||
SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||
".tar.bz2",
|
||||
".tar.gz",
|
||||
".tar.xz",
|
||||
".tbz2",
|
||||
".tgz",
|
||||
".txz",
|
||||
".tar",
|
||||
".zip",
|
||||
)
|
||||
|
||||
|
||||
def _read_zip_member(
|
||||
archive: zipfile.ZipFile,
|
||||
info: zipfile.ZipInfo,
|
||||
*,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
current_total: int,
|
||||
) -> tuple[bytes, int]:
|
||||
parts: list[bytes] = []
|
||||
actual_size = 0
|
||||
with archive.open(info) as source:
|
||||
while True:
|
||||
read_size = min(_ZIP_READ_CHUNK_SIZE, max_file_bytes + 1 - actual_size)
|
||||
chunk = source.read(read_size)
|
||||
if not chunk:
|
||||
break
|
||||
actual_size += len(chunk)
|
||||
if actual_size > max_file_bytes:
|
||||
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
|
||||
if current_total + actual_size > max_total_bytes:
|
||||
raise FileStorageError("ZIP is too large after extraction")
|
||||
parts.append(chunk)
|
||||
return b"".join(parts), current_total + actual_size
|
||||
class ArchivePasswordError(FileStorageError):
|
||||
pass
|
||||
|
||||
|
||||
def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path: str | Path) -> None:
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ArchiveEntry:
|
||||
path: str
|
||||
kind: Literal["file", "directory"]
|
||||
size_bytes: int
|
||||
compressed_size_bytes: int | None = None
|
||||
encrypted: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ArchiveInspection:
|
||||
archive_format: str
|
||||
entries: tuple[ArchiveEntry, ...]
|
||||
file_count: int
|
||||
directory_count: int
|
||||
expanded_size_bytes: int
|
||||
compressed_size_bytes: int
|
||||
requires_password: bool
|
||||
password_verified: bool
|
||||
|
||||
|
||||
def create_zip_file(
|
||||
session: Session, assets: Iterable[FileAsset], output_path: str | Path
|
||||
) -> None:
|
||||
backend = get_storage_backend()
|
||||
with zipfile.ZipFile(output_path, mode="w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for asset in assets:
|
||||
_version, blob = current_version_and_blob(session, asset)
|
||||
asset_list = list(assets)
|
||||
version_blobs = current_versions_and_blobs(session, asset_list)
|
||||
with zipfile.ZipFile(
|
||||
output_path, mode="w", compression=zipfile.ZIP_DEFLATED
|
||||
) as archive:
|
||||
for asset in asset_list:
|
||||
_version, blob = version_blobs[asset.id]
|
||||
info = zipfile.ZipInfo(asset.display_path)
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
with archive.open(info, "w") as member:
|
||||
@@ -59,6 +99,147 @@ def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
|
||||
|
||||
def archive_format_for_filename(filename: str) -> str:
|
||||
lowered = filename.strip().casefold()
|
||||
if lowered.endswith(".zip"):
|
||||
return "zip"
|
||||
if lowered.endswith((".tar.gz", ".tgz")):
|
||||
return "tar.gz"
|
||||
if lowered.endswith((".tar.bz2", ".tbz2")):
|
||||
return "tar.bz2"
|
||||
if lowered.endswith((".tar.xz", ".txz")):
|
||||
return "tar.xz"
|
||||
if lowered.endswith(".tar"):
|
||||
return "tar"
|
||||
raise FileStorageError(
|
||||
"Unsupported archive format. Use ZIP, TAR, TAR.GZ, TAR.BZ2, or TAR.XZ."
|
||||
)
|
||||
|
||||
|
||||
def is_supported_archive_filename(filename: str) -> bool:
|
||||
try:
|
||||
archive_format_for_filename(filename)
|
||||
except FileStorageError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def inspect_archive(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
filename: str,
|
||||
password: str | None = None,
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> ArchiveInspection:
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
compressed_size = _archive_size(archive_data)
|
||||
if archive_format == "zip":
|
||||
entries, requires_password, password_verified = _inspect_zip(
|
||||
archive_data,
|
||||
password=password,
|
||||
)
|
||||
else:
|
||||
entries = _inspect_tar(archive_data)
|
||||
requires_password = False
|
||||
password_verified = True
|
||||
_validate_archive_limits(
|
||||
entries,
|
||||
compressed_size=compressed_size,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
complete_entries = _with_derived_directories(entries)
|
||||
return ArchiveInspection(
|
||||
archive_format=archive_format,
|
||||
entries=tuple(complete_entries),
|
||||
file_count=sum(entry.kind == "file" for entry in complete_entries),
|
||||
directory_count=sum(
|
||||
entry.kind == "directory" for entry in complete_entries
|
||||
),
|
||||
expanded_size_bytes=sum(
|
||||
entry.size_bytes for entry in entries if entry.kind == "file"
|
||||
),
|
||||
compressed_size_bytes=compressed_size,
|
||||
requires_password=requires_password,
|
||||
password_verified=password_verified,
|
||||
)
|
||||
|
||||
|
||||
def extract_archive_upload(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
filename: str,
|
||||
folder: str | None,
|
||||
campaign_id: str | None,
|
||||
selected_paths: Iterable[str] | None = None,
|
||||
password: str | None = None,
|
||||
conflict_strategy: str = "reject",
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
encryption_vault_id: str | None = None,
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_file_bytes: int = 50 * 1024 * 1024,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> list[UploadedStoredFile]:
|
||||
inspection = inspect_archive(
|
||||
archive_data,
|
||||
filename=filename,
|
||||
password=password,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
if inspection.requires_password and not inspection.password_verified:
|
||||
raise ArchivePasswordError("Archive password is required")
|
||||
selected_files = _selected_file_paths(inspection.entries, selected_paths)
|
||||
if not selected_files:
|
||||
raise FileStorageError("Select at least one archive file to import")
|
||||
actual_total_limit = min(
|
||||
max_expanded_bytes,
|
||||
inspection.compressed_size_bytes * max_expansion_ratio,
|
||||
)
|
||||
if inspection.archive_format == "zip":
|
||||
members = _read_selected_zip_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
password=password,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
else:
|
||||
members = _read_selected_tar_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
return _store_archive_members(
|
||||
session,
|
||||
members=members,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
folder=folder,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
|
||||
|
||||
def extract_zip_upload(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -66,59 +247,416 @@ def extract_zip_upload(
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
zip_data: bytes,
|
||||
zip_data: bytes | str | PathLike[str],
|
||||
folder: str | None,
|
||||
campaign_id: str | None,
|
||||
conflict_strategy: str = "reject",
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
max_files: int = 1000,
|
||||
encryption_vault_id: str | None = None,
|
||||
max_files: int = ZIP_UPLOAD_MAX_FILES,
|
||||
max_file_bytes: int = 50 * 1024 * 1024,
|
||||
max_total_bytes: int = 250 * 1024 * 1024,
|
||||
) -> list[UploadedStoredFile]:
|
||||
uploaded: list[UploadedStoredFile] = []
|
||||
total = 0
|
||||
base_folder = normalize_folder(folder)
|
||||
"""Backward-compatible wrapper for the original immediate ZIP endpoint."""
|
||||
|
||||
return extract_archive_upload(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
archive_data=zip_data,
|
||||
filename="archive.zip",
|
||||
folder=folder,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_entries=max_files,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_expanded_bytes=max_total_bytes,
|
||||
)
|
||||
|
||||
|
||||
def _inspect_zip(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
password: str | None,
|
||||
) -> tuple[list[ArchiveEntry], bool, bool]:
|
||||
try:
|
||||
with zipfile.ZipFile(BytesIO(zip_data)) as archive:
|
||||
infos = [info for info in archive.infolist() if not info.is_dir()]
|
||||
if len(infos) > max_files:
|
||||
raise FileStorageError(f"ZIP contains too many files (limit {max_files})")
|
||||
for info in infos:
|
||||
if info.flag_bits & 0x1:
|
||||
raise FileStorageError("Encrypted ZIP uploads are not supported")
|
||||
if info.file_size < 0:
|
||||
raise FileStorageError("Invalid ZIP member")
|
||||
if info.file_size > max_file_bytes:
|
||||
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
|
||||
if total + info.file_size > max_total_bytes:
|
||||
raise FileStorageError("ZIP is too large after extraction")
|
||||
inner_path = normalize_logical_path(info.filename)
|
||||
target_path = f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
data, total = _read_zip_member(
|
||||
archive,
|
||||
info,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
)
|
||||
uploaded.append(
|
||||
create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
filename=filename_from_path(inner_path),
|
||||
data=data,
|
||||
display_path=target_path,
|
||||
content_type=mimetypes.guess_type(inner_path)[0] or "application/octet-stream",
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
is_admin=is_admin,
|
||||
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||
infos = archive.infolist()
|
||||
entries = [_zip_entry(info) for info in infos]
|
||||
encrypted_info = next(
|
||||
(
|
||||
info
|
||||
for info in infos
|
||||
if not info.is_dir() and bool(info.flag_bits & 0x1)
|
||||
),
|
||||
None,
|
||||
)
|
||||
if encrypted_info is None:
|
||||
return entries, False, True
|
||||
if not password:
|
||||
return entries, True, False
|
||||
try:
|
||||
with archive.open(
|
||||
encrypted_info, pwd=password.encode("utf-8")
|
||||
) as member:
|
||||
member.read(1)
|
||||
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise ArchivePasswordError("Archive password is incorrect") from exc
|
||||
return entries, True, True
|
||||
except ArchivePasswordError:
|
||||
raise
|
||||
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise FileStorageError("Invalid ZIP upload") from exc
|
||||
|
||||
|
||||
def _inspect_tar(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
) -> list[ArchiveEntry]:
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
entries: list[ArchiveEntry] = []
|
||||
for member in archive.getmembers():
|
||||
path = _safe_member_path(member.name)
|
||||
if member.isdir():
|
||||
entries.append(
|
||||
ArchiveEntry(path=path, kind="directory", size_bytes=0)
|
||||
)
|
||||
continue
|
||||
if not member.isfile():
|
||||
raise FileStorageError(
|
||||
f"Archive member {member.name!r} is not a regular file or directory"
|
||||
)
|
||||
entries.append(
|
||||
ArchiveEntry(
|
||||
path=path,
|
||||
kind="file",
|
||||
size_bytes=max(0, int(member.size)),
|
||||
)
|
||||
)
|
||||
except zipfile.BadZipFile as exc:
|
||||
raise FileStorageError("Invalid ZIP upload") from exc
|
||||
return entries
|
||||
except FileStorageError:
|
||||
raise
|
||||
except (OSError, tarfile.TarError) as exc:
|
||||
raise FileStorageError("Invalid TAR upload") from exc
|
||||
|
||||
|
||||
def _zip_entry(info: zipfile.ZipInfo) -> ArchiveEntry:
|
||||
path = _safe_member_path(info.filename)
|
||||
unix_mode = (info.external_attr >> 16) & 0xFFFF
|
||||
file_type = stat.S_IFMT(unix_mode)
|
||||
if file_type and not (
|
||||
stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)
|
||||
):
|
||||
raise FileStorageError(
|
||||
f"Archive member {info.filename!r} is not a regular file or directory"
|
||||
)
|
||||
if info.file_size < 0 or info.compress_size < 0:
|
||||
raise FileStorageError(f"Archive member {info.filename!r} has invalid size metadata")
|
||||
return ArchiveEntry(
|
||||
path=path,
|
||||
kind="directory" if info.is_dir() else "file",
|
||||
size_bytes=0 if info.is_dir() else int(info.file_size),
|
||||
compressed_size_bytes=(
|
||||
None if info.is_dir() else int(info.compress_size)
|
||||
),
|
||||
encrypted=bool(info.flag_bits & 0x1),
|
||||
)
|
||||
|
||||
|
||||
def _validate_archive_limits(
|
||||
entries: list[ArchiveEntry],
|
||||
*,
|
||||
compressed_size: int,
|
||||
max_entries: int,
|
||||
max_expanded_bytes: int,
|
||||
max_expansion_ratio: int,
|
||||
) -> None:
|
||||
if len(entries) > max_entries:
|
||||
raise FileStorageError(
|
||||
f"Archive contains too many entries (limit {max_entries})"
|
||||
)
|
||||
seen: dict[str, str] = {}
|
||||
expanded_size = 0
|
||||
for entry in entries:
|
||||
previous_kind = seen.get(entry.path)
|
||||
if previous_kind is not None:
|
||||
raise FileStorageError(
|
||||
f"Archive contains duplicate path {entry.path!r}"
|
||||
)
|
||||
seen[entry.path] = entry.kind
|
||||
if entry.kind == "file":
|
||||
expanded_size += entry.size_bytes
|
||||
if expanded_size > max_expanded_bytes:
|
||||
raise FileStorageError(
|
||||
"Archive is too large after extraction "
|
||||
f"(limit {max_expanded_bytes} bytes)"
|
||||
)
|
||||
if expanded_size and (
|
||||
compressed_size <= 0
|
||||
or expanded_size > compressed_size * max_expansion_ratio
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Archive expansion ratio exceeds "
|
||||
f"{max_expansion_ratio}:1"
|
||||
)
|
||||
|
||||
|
||||
def _with_derived_directories(
|
||||
entries: list[ArchiveEntry],
|
||||
) -> list[ArchiveEntry]:
|
||||
by_path = {entry.path: entry for entry in entries}
|
||||
for entry in entries:
|
||||
parent = PurePosixPath(entry.path).parent
|
||||
while str(parent) not in {"", "."}:
|
||||
path = str(parent)
|
||||
existing = by_path.get(path)
|
||||
if existing and existing.kind != "directory":
|
||||
raise FileStorageError(
|
||||
f"Archive path {path!r} is both a file and a directory"
|
||||
)
|
||||
by_path.setdefault(
|
||||
path,
|
||||
ArchiveEntry(path=path, kind="directory", size_bytes=0),
|
||||
)
|
||||
parent = parent.parent
|
||||
return sorted(
|
||||
by_path.values(),
|
||||
key=lambda entry: (
|
||||
tuple(entry.path.casefold().split("/")),
|
||||
entry.kind != "directory",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _selected_file_paths(
|
||||
entries: tuple[ArchiveEntry, ...],
|
||||
selected_paths: Iterable[str] | None,
|
||||
) -> set[str]:
|
||||
file_paths = {entry.path for entry in entries if entry.kind == "file"}
|
||||
if selected_paths is None:
|
||||
return file_paths
|
||||
known = {entry.path: entry for entry in entries}
|
||||
normalized = {_safe_member_path(path) for path in selected_paths}
|
||||
unknown = normalized - known.keys()
|
||||
if unknown:
|
||||
raise FileStorageError(
|
||||
f"Archive selection contains unknown path {sorted(unknown)[0]!r}"
|
||||
)
|
||||
selected_files: set[str] = set()
|
||||
for path in normalized:
|
||||
entry = known[path]
|
||||
if entry.kind == "file":
|
||||
selected_files.add(path)
|
||||
continue
|
||||
prefix = f"{path}/"
|
||||
selected_files.update(
|
||||
file_path
|
||||
for file_path in file_paths
|
||||
if file_path.startswith(prefix)
|
||||
)
|
||||
return selected_files
|
||||
|
||||
|
||||
def _read_selected_zip_members(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
selected_files: set[str],
|
||||
password: str | None,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
total = 0
|
||||
try:
|
||||
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||
for info in archive.infolist():
|
||||
if info.is_dir():
|
||||
continue
|
||||
path = _safe_member_path(info.filename)
|
||||
if path not in selected_files:
|
||||
continue
|
||||
pwd = password.encode("utf-8") if password else None
|
||||
try:
|
||||
with archive.open(info, pwd=pwd) as source:
|
||||
data, total = _read_member(
|
||||
source,
|
||||
path=path,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
)
|
||||
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||
if info.flag_bits & 0x1:
|
||||
raise ArchivePasswordError(
|
||||
"Archive password is incorrect"
|
||||
) from exc
|
||||
raise
|
||||
result.append((path, data))
|
||||
except (FileStorageError, ArchivePasswordError):
|
||||
raise
|
||||
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise FileStorageError("ZIP extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_selected_tar_members(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
selected_files: set[str],
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
total = 0
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
for member in archive.getmembers():
|
||||
if not member.isfile():
|
||||
continue
|
||||
path = _safe_member_path(member.name)
|
||||
if path not in selected_files:
|
||||
continue
|
||||
source = archive.extractfile(member)
|
||||
if source is None:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} could not be read"
|
||||
)
|
||||
with source:
|
||||
data, total = _read_member(
|
||||
source,
|
||||
path=path,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
)
|
||||
result.append((path, data))
|
||||
except FileStorageError:
|
||||
raise
|
||||
except (OSError, tarfile.TarError) as exc:
|
||||
raise FileStorageError("TAR extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_member(
|
||||
source: BinaryIO,
|
||||
*,
|
||||
path: str,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
current_total: int,
|
||||
) -> tuple[bytes, int]:
|
||||
parts: list[bytes] = []
|
||||
actual_size = 0
|
||||
while True:
|
||||
read_size = min(
|
||||
_ARCHIVE_READ_CHUNK_SIZE,
|
||||
max_file_bytes + 1 - actual_size,
|
||||
)
|
||||
chunk = source.read(read_size)
|
||||
if not chunk:
|
||||
break
|
||||
actual_size += len(chunk)
|
||||
if actual_size > max_file_bytes:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} exceeds per-file limit"
|
||||
)
|
||||
if current_total + actual_size > max_total_bytes:
|
||||
raise FileStorageError("Archive is too large after extraction")
|
||||
parts.append(chunk)
|
||||
return b"".join(parts), current_total + actual_size
|
||||
|
||||
|
||||
def _store_archive_members(
|
||||
session: Session,
|
||||
*,
|
||||
members: Iterable[tuple[str, bytes]],
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
folder: str | None,
|
||||
campaign_id: str | None,
|
||||
conflict_strategy: str,
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None,
|
||||
metadata: dict[str, Any] | None,
|
||||
is_admin: bool,
|
||||
encryption_vault_id: str | None,
|
||||
) -> list[UploadedStoredFile]:
|
||||
uploaded: list[UploadedStoredFile] = []
|
||||
base_folder = normalize_folder(folder)
|
||||
for inner_path, data in members:
|
||||
target_path = (
|
||||
f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
)
|
||||
uploaded.append(
|
||||
create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
filename=filename_from_path(inner_path),
|
||||
data=data,
|
||||
display_path=target_path,
|
||||
content_type=mimetypes.guess_type(inner_path)[0]
|
||||
or "application/octet-stream",
|
||||
metadata=metadata,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
)
|
||||
return uploaded
|
||||
|
||||
|
||||
def _safe_member_path(value: str) -> str:
|
||||
raw = str(value or "").replace("\\", "/").strip()
|
||||
if (
|
||||
not raw
|
||||
or "\x00" in raw
|
||||
or raw.startswith("/")
|
||||
or _WINDOWS_DRIVE_RE.match(raw)
|
||||
):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
if any(part == ".." for part in raw.split("/")):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
try:
|
||||
return normalize_logical_path(raw.rstrip("/"))
|
||||
except ValueError as exc:
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}") from exc
|
||||
|
||||
|
||||
def _archive_size(archive_data: bytes | str | PathLike[str]) -> int:
|
||||
if isinstance(archive_data, bytes):
|
||||
return len(archive_data)
|
||||
try:
|
||||
return Path(archive_data).stat().st_size
|
||||
except OSError as exc:
|
||||
raise FileStorageError("Archive upload could not be read") from exc
|
||||
|
||||
|
||||
def _archive_source(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
) -> BytesIO | str | PathLike[str]:
|
||||
return BytesIO(archive_data) if isinstance(archive_data, bytes) else archive_data
|
||||
|
||||
|
||||
def _open_tar(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
) -> tarfile.TarFile:
|
||||
if isinstance(archive_data, bytes):
|
||||
return tarfile.open(fileobj=BytesIO(archive_data), mode="r:*")
|
||||
try:
|
||||
return tarfile.open(name=archive_data, mode="r:*")
|
||||
except OSError as exc:
|
||||
raise FileStorageError("Archive upload could not be read") from exc
|
||||
|
||||
@@ -1,160 +1,32 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Protocol
|
||||
from govoplan_core.core.object_storage import (
|
||||
LocalFilesystemStorageBackend,
|
||||
S3StorageBackend,
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectInfo,
|
||||
StorageObjectMissing,
|
||||
StorageObjectPage,
|
||||
configured_storage_backend,
|
||||
)
|
||||
|
||||
from govoplan_files.backend.runtime import settings
|
||||
|
||||
|
||||
class StorageBackendError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class StorageBackend(Protocol):
|
||||
name: str
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None: ...
|
||||
def get_bytes(self, key: str) -> bytes: ...
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]: ...
|
||||
def delete(self, key: str) -> None: ...
|
||||
def exists(self, key: str) -> bool: ...
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class LocalFilesystemStorageBackend:
|
||||
root: Path
|
||||
fallback_roots: tuple[Path, ...] = field(default_factory=tuple)
|
||||
name: str = "local"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.root = self.root.expanduser().resolve()
|
||||
self.fallback_roots = tuple(root.expanduser().resolve() for root in self.fallback_roots if root)
|
||||
self.root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _path_for_root(self, root: Path, key: str) -> Path:
|
||||
path = (root / key).resolve()
|
||||
if not path.is_relative_to(root):
|
||||
raise StorageBackendError("Storage key escapes local storage root")
|
||||
return path
|
||||
|
||||
def _path(self, key: str) -> Path:
|
||||
return self._path_for_root(self.root, key)
|
||||
|
||||
def _readable_path(self, key: str) -> Path:
|
||||
primary = self._path(key)
|
||||
if primary.exists() and primary.is_file():
|
||||
return primary
|
||||
for root in self.fallback_roots:
|
||||
candidate = self._path_for_root(root, key)
|
||||
if candidate.exists() and candidate.is_file():
|
||||
return candidate
|
||||
raise StorageBackendError("Stored object does not exist")
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
||||
path = self._path(key)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
|
||||
def get_bytes(self, key: str) -> bytes:
|
||||
return self._readable_path(key).read_bytes()
|
||||
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
||||
path = self._readable_path(key)
|
||||
with path.open("rb") as handle:
|
||||
while True:
|
||||
chunk = handle.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
yield chunk
|
||||
|
||||
def delete(self, key: str) -> None:
|
||||
path = self._path(key)
|
||||
if path.exists() and path.is_file():
|
||||
path.unlink()
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
try:
|
||||
self._readable_path(key)
|
||||
except StorageBackendError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class S3StorageBackend:
|
||||
bucket: str
|
||||
endpoint_url: str
|
||||
region_name: str
|
||||
access_key_id: str
|
||||
secret_access_key: str
|
||||
name: str = "s3"
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
try:
|
||||
import boto3
|
||||
except ModuleNotFoundError as exc:
|
||||
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=self.endpoint_url,
|
||||
region_name=self.region_name,
|
||||
aws_access_key_id=self.access_key_id,
|
||||
aws_secret_access_key=self.secret_access_key,
|
||||
)
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
||||
kwargs = {"Bucket": self.bucket, "Key": key, "Body": data}
|
||||
if content_type:
|
||||
kwargs["ContentType"] = content_type
|
||||
self.client.put_object(**kwargs)
|
||||
|
||||
def get_bytes(self, key: str) -> bytes:
|
||||
try:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
return obj["Body"].read()
|
||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
||||
try:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
body = obj["Body"]
|
||||
while True:
|
||||
chunk = body.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
yield chunk
|
||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
|
||||
def delete(self, key: str) -> None:
|
||||
self.client.delete_object(Bucket=self.bucket, Key=key)
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
try:
|
||||
self.client.head_object(Bucket=self.bucket, Key=key)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _fallback_roots() -> tuple[Path, ...]:
|
||||
raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
|
||||
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())
|
||||
|
||||
|
||||
def get_storage_backend() -> StorageBackend:
|
||||
backend = settings.file_storage_backend.lower().strip()
|
||||
if backend in {"local", "filesystem", "fs"}:
|
||||
return LocalFilesystemStorageBackend(Path(settings.file_storage_local_root), fallback_roots=_fallback_roots())
|
||||
if backend in {"s3", "garage"}:
|
||||
return S3StorageBackend(
|
||||
bucket=settings.file_storage_s3_bucket or settings.s3_bucket,
|
||||
endpoint_url=settings.file_storage_s3_endpoint_url or settings.s3_endpoint_url,
|
||||
region_name=settings.file_storage_s3_region or settings.s3_region,
|
||||
access_key_id=settings.file_storage_s3_access_key_id or settings.s3_access_key_id,
|
||||
secret_access_key=settings.file_storage_s3_secret_access_key or settings.s3_secret_access_key,
|
||||
)
|
||||
raise StorageBackendError(f"Unsupported file storage backend: {settings.file_storage_backend}")
|
||||
"""Return the deployment-owned backend for Files-managed objects."""
|
||||
|
||||
return configured_storage_backend(settings)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"LocalFilesystemStorageBackend",
|
||||
"S3StorageBackend",
|
||||
"StorageBackend",
|
||||
"StorageBackendError",
|
||||
"StorageObjectInfo",
|
||||
"StorageObjectMissing",
|
||||
"StorageObjectPage",
|
||||
"get_storage_backend",
|
||||
]
|
||||
|
||||
@@ -12,9 +12,17 @@ from typing import Any, Iterator
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob, list_assets_for_user, read_asset_bytes
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
|
||||
from govoplan_files.backend.storage.backends import StorageBackend, get_storage_backend
|
||||
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
ensure_blob_is_readable,
|
||||
read_verified_blob_bytes,
|
||||
)
|
||||
|
||||
|
||||
MANAGED_SOURCE_PREFIX = "managed:"
|
||||
@@ -34,10 +42,17 @@ class ManagedAttachmentFile:
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
content_type: str | None
|
||||
linked_to_campaign: bool = True
|
||||
source_provenance: dict[str, Any] | None = None
|
||||
source_revision: str | None = None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
payload = asdict(self)
|
||||
payload.pop("local_path", None)
|
||||
if payload.get("source_provenance") is None:
|
||||
payload.pop("source_provenance", None)
|
||||
if payload.get("source_revision") is None:
|
||||
payload.pop("source_revision", None)
|
||||
return payload
|
||||
|
||||
|
||||
@@ -47,6 +62,7 @@ class PreparedCampaignSnapshot:
|
||||
raw_json: dict[str, Any]
|
||||
managed_files_by_local_path: dict[str, ManagedAttachmentFile]
|
||||
shared_assets: list[FileAsset]
|
||||
candidate_assets: list[FileAsset]
|
||||
|
||||
|
||||
def parse_managed_source(value: object) -> tuple[str, str] | None:
|
||||
@@ -108,6 +124,99 @@ def _iter_rule_dicts(attachments: dict[str, Any], raw_json: dict[str, Any]):
|
||||
yield rule
|
||||
|
||||
|
||||
def _managed_base_sources(raw_json: dict[str, Any]) -> list[tuple[str, str, str]]:
|
||||
attachments = raw_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
return []
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
return []
|
||||
sources: list[tuple[str, str, str]] = []
|
||||
seen: set[tuple[str, str, str]] = set()
|
||||
for item in base_paths:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
key = (owner_type, owner_id, logical_root)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
sources.append(key)
|
||||
return sources
|
||||
|
||||
|
||||
def _campaign_linked_asset_ids(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
asset_ids: list[str],
|
||||
) -> set[str]:
|
||||
if not asset_ids:
|
||||
return set()
|
||||
linked: set[str] = set()
|
||||
for offset in range(0, len(asset_ids), 900):
|
||||
chunk = asset_ids[offset : offset + 900]
|
||||
rows = (
|
||||
session.query(FileShare.file_asset_id)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id.in_(chunk),
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
linked.update(row[0] for row in rows)
|
||||
return linked
|
||||
|
||||
|
||||
def _candidate_assets_for_managed_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
raw_json: dict[str, Any],
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
shared_assets: list[FileAsset],
|
||||
) -> tuple[list[FileAsset], set[str]]:
|
||||
assets_by_id: dict[str, FileAsset] = {asset.id: asset for asset in shared_assets}
|
||||
for owner_type, owner_id, logical_root in _managed_base_sources(raw_json):
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
for asset in list_assets_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=logical_root or None,
|
||||
is_admin=is_admin,
|
||||
):
|
||||
assets_by_id.setdefault(asset.id, asset)
|
||||
candidate_assets = sorted(assets_by_id.values(), key=lambda asset: (asset.display_path, asset.updated_at, asset.id))
|
||||
linked_ids = _campaign_linked_asset_ids(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
asset_ids=[asset.id for asset in candidate_assets],
|
||||
)
|
||||
return candidate_assets, linked_ids
|
||||
|
||||
|
||||
def _selected_base_path(
|
||||
rule: dict[str, Any],
|
||||
prepared_by_id: dict[str, tuple[str, str]],
|
||||
@@ -127,6 +236,175 @@ def _selected_base_path(
|
||||
return None
|
||||
|
||||
|
||||
def _prepared_attachment_config(raw_json: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any], list[Any]]:
|
||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||
attachments = prepared_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
attachments = {}
|
||||
prepared_json["attachments"] = attachments
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
base_paths = []
|
||||
return prepared_json, attachments, base_paths
|
||||
|
||||
|
||||
def _campaign_snapshot_assets(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
prepared_json: dict[str, Any],
|
||||
include_unlinked_candidates: bool,
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
) -> tuple[list[FileAsset], list[FileAsset], set[str]]:
|
||||
shared_assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id="",
|
||||
campaign_id=campaign_id,
|
||||
is_admin=True,
|
||||
)
|
||||
if not include_unlinked_candidates:
|
||||
return shared_assets, shared_assets, {asset.id for asset in shared_assets}
|
||||
candidate_assets, linked_asset_ids = _candidate_assets_for_managed_sources(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
raw_json=prepared_json,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
shared_assets=shared_assets,
|
||||
)
|
||||
return shared_assets, candidate_assets, linked_asset_ids
|
||||
|
||||
|
||||
def _assets_grouped_by_owner(assets: list[FileAsset]) -> dict[tuple[str, str], list[FileAsset]]:
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||
for asset in assets:
|
||||
owner_id = _asset_owner_id(asset)
|
||||
if owner_id:
|
||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||
return assets_by_owner
|
||||
|
||||
|
||||
def _materialize_managed_asset(
|
||||
asset: FileAsset,
|
||||
*,
|
||||
owner_id: str,
|
||||
logical_root: str,
|
||||
local_root: Path,
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
backend: StorageBackend | None,
|
||||
include_bytes: bool,
|
||||
linked_asset_ids: set[str],
|
||||
) -> tuple[str, ManagedAttachmentFile] | None:
|
||||
relative_path = _relative_asset_path(asset, logical_root)
|
||||
if not relative_path:
|
||||
return None
|
||||
target = _safe_local_target(local_root, relative_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
version, blob = version_blobs[asset.id]
|
||||
ensure_blob_is_readable(blob)
|
||||
if include_bytes:
|
||||
data = read_verified_blob_bytes(blob, backend=backend) if backend else b""
|
||||
target.write_bytes(data)
|
||||
else:
|
||||
target.touch()
|
||||
local_key = str(target.resolve())
|
||||
return local_key, ManagedAttachmentFile(
|
||||
local_path=local_key,
|
||||
asset_id=asset.id,
|
||||
version_id=version.id,
|
||||
blob_id=blob.id,
|
||||
display_path=asset.display_path,
|
||||
relative_path=normalize_logical_path(relative_path),
|
||||
filename=asset.filename,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=owner_id,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
linked_to_campaign=asset.id in linked_asset_ids,
|
||||
source_provenance=source_provenance_from_metadata(asset.metadata_),
|
||||
source_revision=source_revision_from_metadata(asset.metadata_),
|
||||
)
|
||||
|
||||
|
||||
def _prepare_managed_base_paths(
|
||||
base_paths: list[Any],
|
||||
*,
|
||||
materialized_root: Path,
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]],
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
backend: StorageBackend | None,
|
||||
include_bytes: bool,
|
||||
linked_asset_ids: set[str],
|
||||
) -> tuple[
|
||||
dict[str, ManagedAttachmentFile],
|
||||
dict[str, tuple[str, str]],
|
||||
dict[str, list[tuple[str, str]]],
|
||||
tuple[str, str] | None,
|
||||
]:
|
||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||
first_prepared: tuple[str, str] | None = None
|
||||
for index, item in enumerate(base_paths):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||
local_root.mkdir(parents=True, exist_ok=True)
|
||||
local_root_string = str(local_root.resolve())
|
||||
prepared = (base_path_id, local_root_string)
|
||||
prepared_by_id[base_path_id] = prepared
|
||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||
if first_prepared is None:
|
||||
first_prepared = prepared
|
||||
item["path"] = local_root_string
|
||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||
materialized = _materialize_managed_asset(
|
||||
asset,
|
||||
owner_id=owner_id,
|
||||
logical_root=logical_root,
|
||||
local_root=local_root,
|
||||
version_blobs=version_blobs,
|
||||
backend=backend,
|
||||
include_bytes=include_bytes,
|
||||
linked_asset_ids=linked_asset_ids,
|
||||
)
|
||||
if materialized is not None:
|
||||
local_key, managed_file = materialized
|
||||
manifest[local_key] = managed_file
|
||||
return manifest, prepared_by_id, prepared_by_old_path, first_prepared
|
||||
|
||||
|
||||
def _rewrite_managed_attachment_rules(
|
||||
attachments: dict[str, Any],
|
||||
prepared_json: dict[str, Any],
|
||||
*,
|
||||
prepared_by_id: dict[str, tuple[str, str]],
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]],
|
||||
first_prepared: tuple[str, str] | None,
|
||||
) -> None:
|
||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||
if selected is None:
|
||||
continue
|
||||
base_path_id, local_root_string = selected
|
||||
rule["base_path_id"] = base_path_id
|
||||
rule["base_dir"] = local_root_string
|
||||
if first_prepared is not None:
|
||||
attachments["base_path"] = first_prepared[1]
|
||||
|
||||
|
||||
def prepare_campaign_snapshot(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -135,6 +413,9 @@ def prepare_campaign_snapshot(
|
||||
raw_json: dict[str, Any],
|
||||
destination: Path,
|
||||
include_bytes: bool,
|
||||
include_unlinked_candidates: bool = False,
|
||||
user_id: str = "",
|
||||
is_admin: bool = False,
|
||||
) -> PreparedCampaignSnapshot:
|
||||
"""Create a temporary file-oriented campaign snapshot for managed attachments.
|
||||
|
||||
@@ -150,94 +431,35 @@ def prepare_campaign_snapshot(
|
||||
materialized_root = destination / "managed-attachments"
|
||||
materialized_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||
attachments = prepared_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
attachments = {}
|
||||
prepared_json["attachments"] = attachments
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
base_paths = []
|
||||
|
||||
shared_assets = list_assets_for_user(
|
||||
prepared_json, attachments, base_paths = _prepared_attachment_config(raw_json)
|
||||
shared_assets, candidate_assets, linked_asset_ids = _campaign_snapshot_assets(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id="",
|
||||
campaign_id=campaign_id,
|
||||
is_admin=True,
|
||||
prepared_json=prepared_json,
|
||||
include_unlinked_candidates=include_unlinked_candidates,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
assets_by_owner = _assets_grouped_by_owner(candidate_assets)
|
||||
version_blobs = current_versions_and_blobs(session, candidate_assets)
|
||||
backend = get_storage_backend() if include_bytes else None
|
||||
manifest, prepared_by_id, prepared_by_old_path, first_prepared = _prepare_managed_base_paths(
|
||||
base_paths,
|
||||
materialized_root=materialized_root,
|
||||
assets_by_owner=assets_by_owner,
|
||||
version_blobs=version_blobs,
|
||||
backend=backend,
|
||||
include_bytes=include_bytes,
|
||||
linked_asset_ids=linked_asset_ids,
|
||||
)
|
||||
_rewrite_managed_attachment_rules(
|
||||
attachments,
|
||||
prepared_json,
|
||||
prepared_by_id=prepared_by_id,
|
||||
prepared_by_old_path=prepared_by_old_path,
|
||||
first_prepared=first_prepared,
|
||||
)
|
||||
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||
for asset in shared_assets:
|
||||
owner_id = _asset_owner_id(asset)
|
||||
if owner_id:
|
||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||
|
||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||
first_prepared: tuple[str, str] | None = None
|
||||
|
||||
for index, item in enumerate(base_paths):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||
local_root.mkdir(parents=True, exist_ok=True)
|
||||
local_root_string = str(local_root.resolve())
|
||||
|
||||
prepared = (base_path_id, local_root_string)
|
||||
prepared_by_id[base_path_id] = prepared
|
||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||
if first_prepared is None:
|
||||
first_prepared = prepared
|
||||
|
||||
item["path"] = local_root_string
|
||||
|
||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||
relative_path = _relative_asset_path(asset, logical_root)
|
||||
if not relative_path:
|
||||
continue
|
||||
target = _safe_local_target(local_root, relative_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if include_bytes:
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
target.write_bytes(data)
|
||||
else:
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
target.touch()
|
||||
local_key = str(target.resolve())
|
||||
manifest[local_key] = ManagedAttachmentFile(
|
||||
local_path=local_key,
|
||||
asset_id=asset.id,
|
||||
version_id=version.id,
|
||||
blob_id=blob.id,
|
||||
display_path=asset.display_path,
|
||||
relative_path=normalize_logical_path(relative_path),
|
||||
filename=asset.filename,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=owner_id,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
)
|
||||
|
||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||
if selected is None:
|
||||
continue
|
||||
base_path_id, local_root_string = selected
|
||||
rule["base_path_id"] = base_path_id
|
||||
rule["base_dir"] = local_root_string
|
||||
|
||||
if first_prepared is not None:
|
||||
attachments["base_path"] = first_prepared[1]
|
||||
|
||||
snapshot_path = destination / "campaign.json"
|
||||
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
@@ -246,6 +468,7 @@ def prepare_campaign_snapshot(
|
||||
raw_json=prepared_json,
|
||||
managed_files_by_local_path=manifest,
|
||||
shared_assets=shared_assets,
|
||||
candidate_assets=candidate_assets,
|
||||
)
|
||||
|
||||
|
||||
@@ -257,7 +480,10 @@ def prepared_campaign_snapshot(
|
||||
campaign_id: str,
|
||||
raw_json: dict[str, Any],
|
||||
include_bytes: bool,
|
||||
prefix: str = "multimailer-managed-campaign-",
|
||||
prefix: str = "govoplan-managed-campaign-",
|
||||
include_unlinked_candidates: bool = False,
|
||||
user_id: str = "",
|
||||
is_admin: bool = False,
|
||||
) -> Iterator[PreparedCampaignSnapshot]:
|
||||
temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
|
||||
try:
|
||||
@@ -268,6 +494,9 @@ def prepared_campaign_snapshot(
|
||||
raw_json=raw_json,
|
||||
destination=temp_dir,
|
||||
include_bytes=include_bytes,
|
||||
include_unlinked_candidates=include_unlinked_candidates,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
finally:
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
@@ -285,6 +514,53 @@ def managed_match_payloads(
|
||||
return payloads
|
||||
|
||||
|
||||
def share_assets_with_campaign(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
file_ids: list[str],
|
||||
user_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> list[dict[str, Any]]:
|
||||
unique_ids = list(dict.fromkeys(file_id for file_id in file_ids if file_id))
|
||||
if not unique_ids:
|
||||
return []
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
for file_id in unique_ids
|
||||
]
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
assets=assets,
|
||||
target_type="campaign",
|
||||
target_id=campaign_id,
|
||||
permission="read",
|
||||
user_id=user_id,
|
||||
)
|
||||
session.flush()
|
||||
return [
|
||||
{
|
||||
"id": share.id,
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
"created_at": share.created_at.isoformat() if share.created_at else None,
|
||||
"revoked_at": share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
}
|
||||
for share in shares
|
||||
]
|
||||
|
||||
|
||||
def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
|
||||
"""Return an attachment summary without temporary materialization paths.
|
||||
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePosixPath
|
||||
from typing import TYPE_CHECKING, Iterable
|
||||
from typing import Any, Iterable
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileVersion
|
||||
from govoplan_files.backend.storage.common import utcnow
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob, list_assets_for_user
|
||||
from govoplan_files.backend.storage.files import current_versions_and_blobs, list_assets_for_user
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from govoplan_campaign.backend.db.models import CampaignJob
|
||||
CampaignJobLike = Any
|
||||
|
||||
|
||||
def _candidate_match_keys(raw_match: str) -> set[str]:
|
||||
@@ -25,11 +25,20 @@ def _candidate_match_keys(raw_match: str) -> set[str]:
|
||||
AttachmentUseKey = tuple[str, str, str, str]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _AttachmentBatchRefs:
|
||||
managed_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||
asset_ids: set[str] = field(default_factory=set)
|
||||
version_ids: set[str] = field(default_factory=set)
|
||||
blob_ids: set[str] = field(default_factory=set)
|
||||
|
||||
|
||||
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
|
||||
return job_id, file_version_id, filename_used, stage
|
||||
|
||||
|
||||
def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJob], *, stage: str) -> set[AttachmentUseKey]:
|
||||
def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJobLike], *, stage: str) -> set[AttachmentUseKey]:
|
||||
job_ids = {job.id for job in jobs if job.id}
|
||||
if not job_ids:
|
||||
return set()
|
||||
@@ -72,13 +81,13 @@ def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJob], *, s
|
||||
return keys
|
||||
|
||||
|
||||
def _known_use_keys(session: Session, job: CampaignJob, *, stage: str) -> set[AttachmentUseKey]:
|
||||
def _known_use_keys(session: Session, job: CampaignJobLike, *, stage: str) -> set[AttachmentUseKey]:
|
||||
return _known_use_keys_for_jobs(session, [job], stage=stage)
|
||||
|
||||
|
||||
def _add_use(
|
||||
session: Session,
|
||||
job: CampaignJob,
|
||||
job: CampaignJobLike,
|
||||
*,
|
||||
asset: FileAsset,
|
||||
version: FileVersion,
|
||||
@@ -118,7 +127,7 @@ def _add_use(
|
||||
|
||||
def record_campaign_attachment_uses_for_jobs(
|
||||
session: Session,
|
||||
jobs: Iterable[CampaignJob],
|
||||
jobs: Iterable[CampaignJobLike],
|
||||
*,
|
||||
stage: str = "built",
|
||||
) -> None:
|
||||
@@ -135,62 +144,91 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
if not job_list:
|
||||
return
|
||||
|
||||
managed_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
||||
refs = _collect_attachment_batch_refs(job_list)
|
||||
job_by_id = {job.id: job for job in job_list}
|
||||
asset_ids: set[str] = set()
|
||||
version_ids: set[str] = set()
|
||||
blob_ids: set[str] = set()
|
||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||
assets_by_id, versions_by_id, blobs_by_id = _load_managed_attachment_entities(session, refs)
|
||||
|
||||
for job in job_list:
|
||||
_record_managed_attachment_uses(
|
||||
session,
|
||||
job_by_id=job_by_id,
|
||||
managed_by_job=refs.managed_by_job,
|
||||
assets_by_id=assets_by_id,
|
||||
versions_by_id=versions_by_id,
|
||||
blobs_by_id=blobs_by_id,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
_record_fallback_attachment_uses(
|
||||
session,
|
||||
job_by_id=job_by_id,
|
||||
fallback_attachments_by_job=refs.fallback_attachments_by_job,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def _collect_attachment_batch_refs(jobs: list[CampaignJobLike]) -> _AttachmentBatchRefs:
|
||||
refs = _AttachmentBatchRefs()
|
||||
for job in jobs:
|
||||
attachments = job.resolved_attachments or []
|
||||
if not isinstance(attachments, list):
|
||||
continue
|
||||
for attachment in attachments:
|
||||
if not isinstance(attachment, dict):
|
||||
continue
|
||||
managed_matches = attachment.get("managed_matches")
|
||||
if isinstance(managed_matches, list) and managed_matches:
|
||||
for item in managed_matches:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
asset_id = str(item.get("asset_id") or "")
|
||||
version_id = str(item.get("version_id") or "")
|
||||
blob_id = str(item.get("blob_id") or "")
|
||||
if not asset_id or not version_id or not blob_id:
|
||||
continue
|
||||
managed_by_job[job.id].append(item)
|
||||
asset_ids.add(asset_id)
|
||||
version_ids.add(version_id)
|
||||
blob_ids.add(blob_id)
|
||||
else:
|
||||
fallback_attachments_by_job[job.id].append(attachment)
|
||||
if not _collect_managed_attachment_refs(refs, job.id, attachment):
|
||||
refs.fallback_attachments_by_job[job.id].append(attachment)
|
||||
return refs
|
||||
|
||||
assets_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileAsset).filter(FileAsset.id.in_(asset_ids)).all()
|
||||
} if asset_ids else {}
|
||||
versions_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileVersion).filter(FileVersion.id.in_(version_ids)).all()
|
||||
} if version_ids else {}
|
||||
blobs_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileBlob).filter(FileBlob.id.in_(blob_ids)).all()
|
||||
} if blob_ids else {}
|
||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||
|
||||
def _collect_managed_attachment_refs(refs: _AttachmentBatchRefs, job_id: str, attachment: dict[str, object]) -> bool:
|
||||
managed_matches = attachment.get("managed_matches")
|
||||
if not isinstance(managed_matches, list) or not managed_matches:
|
||||
return False
|
||||
for item in managed_matches:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
asset_id = str(item.get("asset_id") or "")
|
||||
version_id = str(item.get("version_id") or "")
|
||||
blob_id = str(item.get("blob_id") or "")
|
||||
if not asset_id or not version_id or not blob_id:
|
||||
continue
|
||||
refs.managed_by_job[job_id].append(item)
|
||||
refs.asset_ids.add(asset_id)
|
||||
refs.version_ids.add(version_id)
|
||||
refs.blob_ids.add(blob_id)
|
||||
return True
|
||||
|
||||
|
||||
def _load_managed_attachment_entities(
|
||||
session: Session,
|
||||
refs: _AttachmentBatchRefs,
|
||||
) -> tuple[dict[str, FileAsset], dict[str, FileVersion], dict[str, FileBlob]]:
|
||||
assets_by_id = {item.id: item for item in session.query(FileAsset).filter(FileAsset.id.in_(refs.asset_ids)).all()} if refs.asset_ids else {}
|
||||
versions_by_id = {item.id: item for item in session.query(FileVersion).filter(FileVersion.id.in_(refs.version_ids)).all()} if refs.version_ids else {}
|
||||
blobs_by_id = {item.id: item for item in session.query(FileBlob).filter(FileBlob.id.in_(refs.blob_ids)).all()} if refs.blob_ids else {}
|
||||
return assets_by_id, versions_by_id, blobs_by_id
|
||||
|
||||
|
||||
def _record_managed_attachment_uses(
|
||||
session: Session,
|
||||
*,
|
||||
job_by_id: dict[str, CampaignJobLike],
|
||||
managed_by_job: dict[str, list[dict[str, object]]],
|
||||
assets_by_id: dict[str, FileAsset],
|
||||
versions_by_id: dict[str, FileVersion],
|
||||
blobs_by_id: dict[str, FileBlob],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
for job_id, items in managed_by_job.items():
|
||||
job = job_by_id[job_id]
|
||||
for item in items:
|
||||
asset = assets_by_id.get(str(item.get("asset_id") or ""))
|
||||
version = versions_by_id.get(str(item.get("version_id") or ""))
|
||||
blob = blobs_by_id.get(str(item.get("blob_id") or ""))
|
||||
if not asset or not version or not blob:
|
||||
continue
|
||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||
continue
|
||||
if version.file_asset_id != asset.id or version.blob_id != blob.id:
|
||||
if not _managed_attachment_entities_match_job(job, asset, version, blob):
|
||||
continue
|
||||
_add_use(
|
||||
session,
|
||||
@@ -203,53 +241,109 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def _managed_attachment_entities_match_job(
|
||||
job: CampaignJobLike,
|
||||
asset: FileAsset | None,
|
||||
version: FileVersion | None,
|
||||
blob: FileBlob | None,
|
||||
) -> bool:
|
||||
if not asset or not version or not blob:
|
||||
return False
|
||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||
return False
|
||||
return version.file_asset_id == asset.id and version.blob_id == blob.id
|
||||
|
||||
|
||||
def _record_fallback_attachment_uses(
|
||||
session: Session,
|
||||
*,
|
||||
job_by_id: dict[str, CampaignJobLike],
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
|
||||
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {}
|
||||
for job_id, attachments in fallback_attachments_by_job.items():
|
||||
job = job_by_id[job_id]
|
||||
campaign_key = (job.tenant_id, job.campaign_id)
|
||||
by_key = assets_by_campaign.get(campaign_key)
|
||||
if by_key is None:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=job.tenant_id,
|
||||
user_id="",
|
||||
campaign_id=job.campaign_id,
|
||||
is_admin=True,
|
||||
)
|
||||
by_key = {}
|
||||
for asset in assets:
|
||||
by_key[asset.display_path.strip("/")] = asset
|
||||
by_key.setdefault(asset.filename, asset)
|
||||
assets_by_campaign[campaign_key] = by_key
|
||||
|
||||
by_key, version_blobs = _fallback_campaign_assets(
|
||||
session,
|
||||
job,
|
||||
campaign_key=campaign_key,
|
||||
assets_by_campaign=assets_by_campaign,
|
||||
version_blobs_by_campaign=version_blobs_by_campaign,
|
||||
)
|
||||
for attachment in attachments:
|
||||
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
||||
for raw in matches:
|
||||
if not isinstance(raw, str):
|
||||
continue
|
||||
asset = next((by_key[key] for key in _candidate_match_keys(raw) if key in by_key), None)
|
||||
if not asset:
|
||||
continue
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
_add_use(
|
||||
session,
|
||||
job,
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
filename_used=asset.filename,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
_record_fallback_attachment(session, job, attachment, by_key=by_key, version_blobs=version_blobs, stage=stage, known_keys=known_keys)
|
||||
|
||||
|
||||
def record_campaign_attachment_uses_for_job(session: Session, job: CampaignJob, *, stage: str = "built") -> None:
|
||||
def _fallback_campaign_assets(
|
||||
session: Session,
|
||||
job: CampaignJobLike,
|
||||
*,
|
||||
campaign_key: tuple[str, str],
|
||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]],
|
||||
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]],
|
||||
) -> tuple[dict[str, FileAsset], dict[str, tuple[FileVersion, FileBlob]]]:
|
||||
by_key = assets_by_campaign.get(campaign_key)
|
||||
if by_key is None:
|
||||
assets = list_assets_for_user(session, tenant_id=job.tenant_id, user_id="", campaign_id=job.campaign_id, is_admin=True)
|
||||
by_key = _asset_lookup_by_path_and_filename(assets)
|
||||
assets_by_campaign[campaign_key] = by_key
|
||||
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
|
||||
return by_key, version_blobs_by_campaign[campaign_key]
|
||||
|
||||
|
||||
def _asset_lookup_by_path_and_filename(assets: list[FileAsset]) -> dict[str, FileAsset]:
|
||||
by_key: dict[str, FileAsset] = {}
|
||||
for asset in assets:
|
||||
by_key[asset.display_path.strip("/")] = asset
|
||||
by_key.setdefault(asset.filename, asset)
|
||||
return by_key
|
||||
|
||||
|
||||
def _record_fallback_attachment(
|
||||
session: Session,
|
||||
job: CampaignJobLike,
|
||||
attachment: dict[str, object],
|
||||
*,
|
||||
by_key: dict[str, FileAsset],
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
||||
for raw in matches:
|
||||
if not isinstance(raw, str):
|
||||
continue
|
||||
asset = next((by_key[key] for key in _candidate_match_keys(raw) if key in by_key), None)
|
||||
if not asset:
|
||||
continue
|
||||
version_blob = version_blobs.get(asset.id)
|
||||
if not version_blob:
|
||||
continue
|
||||
version, blob = version_blob
|
||||
_add_use(
|
||||
session,
|
||||
job,
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
filename_used=asset.filename,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def record_campaign_attachment_uses_for_job(session: Session, job: CampaignJobLike, *, stage: str = "built") -> None:
|
||||
"""Record immutable managed file versions used by one built/sent job."""
|
||||
|
||||
record_campaign_attachment_uses_for_jobs(session, [job], stage=stage)
|
||||
|
||||
|
||||
def mark_job_attachment_uses_sent(session: Session, job: CampaignJob) -> None:
|
||||
def mark_job_attachment_uses_sent(session: Session, job: CampaignJobLike) -> None:
|
||||
record_campaign_attachment_uses_for_job(session, job, stage="built")
|
||||
# Sessions use autoflush=False. Flush any compatibility-built evidence so
|
||||
# the following query can copy it to the sent stage in the same call.
|
||||
|
||||
@@ -0,0 +1,923 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from email.utils import parsedate_to_datetime
|
||||
from importlib import import_module
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
from urllib.parse import quote, unquote, urljoin, urlsplit
|
||||
import xml.etree.ElementTree as ET # nosec B405 - typing/element creation only; parsing uses defusedxml below.
|
||||
|
||||
from defusedxml import ElementTree as SafeElementTree
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
validate_outbound_host,
|
||||
validate_outbound_http_url,
|
||||
)
|
||||
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
ConnectorDeploymentConfigurationError,
|
||||
connector_ca_bundle_path,
|
||||
connector_secret_env_value,
|
||||
validate_connector_tls_metadata,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.sdk_peer_pinning import (
|
||||
SdkPeerPinningError,
|
||||
create_pinned_s3_client,
|
||||
install_pinned_smb_transport,
|
||||
pinned_smb_connection_cache,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorBrowseError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectorBrowseUnsupported(ConnectorBrowseError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorBrowseItem:
|
||||
kind: str
|
||||
name: str
|
||||
path: str
|
||||
external_id: str | None = None
|
||||
external_url: str | None = None
|
||||
size_bytes: int | None = None
|
||||
content_type: str | None = None
|
||||
modified_at: str | None = None
|
||||
etag: str | None = None
|
||||
metadata: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"name": self.name,
|
||||
"path": self.path,
|
||||
"external_id": self.external_id,
|
||||
"external_url": self.external_url,
|
||||
"size_bytes": self.size_bytes,
|
||||
"content_type": self.content_type,
|
||||
"modified_at": self.modified_at,
|
||||
"etag": self.etag,
|
||||
"metadata": dict(self.metadata),
|
||||
}
|
||||
|
||||
|
||||
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None, continuation_token: str | None = None) -> list[ConnectorBrowseItem]:
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
static_items = _static_listing(profile, path=browse_path, library_id=library_id)
|
||||
if static_items is not None:
|
||||
return static_items
|
||||
if profile.provider == "seafile":
|
||||
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||
return _browse_webdav(profile, path=browse_path)
|
||||
return _browse_seafile(profile, path=browse_path, library_id=library_id)
|
||||
if profile.provider in {"webdav", "nextcloud"} or _metadata_string(profile, "browse_protocol") == "webdav":
|
||||
return _browse_webdav(profile, path=browse_path)
|
||||
if profile.provider == "smb":
|
||||
return _browse_smb(profile, path=browse_path)
|
||||
if profile.provider == "s3":
|
||||
return _browse_s3(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
|
||||
raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet")
|
||||
|
||||
|
||||
def parse_webdav_multistatus(*, root_url: str, current_path: str, payload: str | bytes) -> list[ConnectorBrowseItem]:
|
||||
try:
|
||||
root = SafeElementTree.fromstring(payload)
|
||||
except SafeElementTree.ParseError as exc:
|
||||
raise ConnectorBrowseError("Connector returned invalid WebDAV XML") from exc
|
||||
root_path = _url_path_root(root_url)
|
||||
current = normalize_connector_browse_path(current_path)
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for response in root.findall("{DAV:}response"):
|
||||
href = response.findtext("{DAV:}href")
|
||||
if not href:
|
||||
continue
|
||||
relative_path = _relative_href_path(root_path, href)
|
||||
if relative_path == current:
|
||||
continue
|
||||
if current and not relative_path.startswith(current.rstrip("/") + "/"):
|
||||
continue
|
||||
parent = relative_path.rsplit("/", 1)[0] if "/" in relative_path else ""
|
||||
if parent != current:
|
||||
continue
|
||||
prop = _webdav_prop(response)
|
||||
is_collection = prop.find("{DAV:}resourcetype/{DAV:}collection") is not None if prop is not None else False
|
||||
name = _webdav_display_name(prop) or _path_name(relative_path)
|
||||
if not name:
|
||||
continue
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder" if is_collection else "file",
|
||||
name=name,
|
||||
path=relative_path,
|
||||
external_id=_text(prop, "{DAV:}getetag") or relative_path,
|
||||
size_bytes=None if is_collection else _int(_text(prop, "{DAV:}getcontentlength")),
|
||||
content_type=None if is_collection else _text(prop, "{DAV:}getcontenttype"),
|
||||
modified_at=_http_date(_text(prop, "{DAV:}getlastmodified")),
|
||||
etag=_text(prop, "{DAV:}getetag"),
|
||||
)
|
||||
)
|
||||
return sorted(items, key=lambda item: (item.kind != "library", item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def seafile_libraries_from_payload(payload: object) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(payload, list):
|
||||
raise ConnectorBrowseError("Seafile library response must be a list")
|
||||
libraries = [_seafile_library_item(item) for item in payload if isinstance(item, Mapping)]
|
||||
return sorted(libraries, key=lambda item: item.name.casefold())
|
||||
|
||||
|
||||
def seafile_directory_items_from_payload(payload: object, *, path: str | None = None) -> list[ConnectorBrowseItem]:
|
||||
if payload == "uptodate":
|
||||
return []
|
||||
if not isinstance(payload, list):
|
||||
raise ConnectorBrowseError("Seafile directory response must be a list")
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
items = [_seafile_directory_item(item, parent_path=browse_path) for item in payload if isinstance(item, Mapping)]
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _browse_seafile(profile: ConnectorProfile, *, path: str, library_id: str | None) -> list[ConnectorBrowseItem]:
|
||||
repo_id, dir_path = _seafile_repo_and_path(path=path, library_id=library_id)
|
||||
headers = _seafile_headers(profile)
|
||||
if not repo_id:
|
||||
payload = _request_json("GET", _seafile_url(profile, "api2/repos/"), headers=headers)
|
||||
return seafile_libraries_from_payload(payload)
|
||||
payload = _request_json(
|
||||
"GET",
|
||||
_seafile_url(profile, f"api2/repos/{quote(repo_id, safe='')}/dir/"),
|
||||
headers=headers,
|
||||
params={"p": "/" + dir_path if dir_path else "/"},
|
||||
)
|
||||
return seafile_directory_items_from_payload(payload, path=dir_path)
|
||||
|
||||
|
||||
def _browse_webdav(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||
root_url = _metadata_string(profile, "webdav_endpoint_url") or profile.endpoint_url
|
||||
if not root_url:
|
||||
raise ConnectorBrowseError("Connector profile does not define an endpoint URL")
|
||||
url = _webdav_url(root_url, path)
|
||||
headers = {"Depth": "1", "Content-Type": "application/xml; charset=utf-8"}
|
||||
auth: tuple[str, str] | None = None
|
||||
password = _profile_password(profile)
|
||||
token = _profile_token(profile)
|
||||
if profile.username and password:
|
||||
auth = (profile.username, password)
|
||||
elif token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref connector credentials need a runtime secret resolver before live browsing")
|
||||
body = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<propfind xmlns="DAV:">
|
||||
<prop>
|
||||
<displayname />
|
||||
<resourcetype />
|
||||
<getcontentlength />
|
||||
<getcontenttype />
|
||||
<getlastmodified />
|
||||
<getetag />
|
||||
</prop>
|
||||
</propfind>"""
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
"PROPFIND",
|
||||
url,
|
||||
headers=headers,
|
||||
content=body,
|
||||
auth=auth,
|
||||
timeout=15.0,
|
||||
label="WebDAV browse",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||
if response.status_code not in {200, 207}:
|
||||
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.content)
|
||||
|
||||
|
||||
def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||
location = _smb_location(profile)
|
||||
unc_path = _smb_unc_path(location, path)
|
||||
smbclient = _smbclient_module()
|
||||
try:
|
||||
entries = smbclient.scandir(unc_path, **_smb_client_kwargs(profile, location))
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"SMB connector browse failed: {exc}") from exc
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
try:
|
||||
with entries as iterator:
|
||||
for entry in iterator:
|
||||
name = _clean(getattr(entry, "name", None))
|
||||
if not name or name in {".", ".."}:
|
||||
continue
|
||||
try:
|
||||
is_dir = bool(entry.is_dir())
|
||||
except Exception:
|
||||
is_dir = False
|
||||
stat_result = _smb_entry_stat(entry)
|
||||
item_path = _join_browse_path(path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder" if is_dir else "file",
|
||||
name=name,
|
||||
path=item_path,
|
||||
external_id=f"{location.share}:{item_path}",
|
||||
size_bytes=None if is_dir else _smb_stat_size(stat_result),
|
||||
content_type=None if is_dir else mimetypes.guess_type(name)[0],
|
||||
modified_at=_smb_stat_modified_at(stat_result),
|
||||
etag=_smb_stat_revision(stat_result),
|
||||
metadata={
|
||||
"share": location.share,
|
||||
**({"server": location.server} if _metadata_bool(profile, "expose_server_metadata", default=False) else {}),
|
||||
},
|
||||
)
|
||||
)
|
||||
except ConnectorBrowseError:
|
||||
raise
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"SMB connector browse failed: {exc}") from exc
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _browse_s3(profile: ConnectorProfile, *, path: str, library_id: str | None, continuation_token: str | None) -> list[ConnectorBrowseItem]:
|
||||
client = _s3_client(profile)
|
||||
try:
|
||||
return _browse_s3_with_client(
|
||||
client,
|
||||
profile=profile,
|
||||
path=path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
def _browse_s3_with_client(
|
||||
client: Any,
|
||||
*,
|
||||
profile: ConnectorProfile,
|
||||
path: str,
|
||||
library_id: str | None,
|
||||
continuation_token: str | None,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
if not bucket:
|
||||
try:
|
||||
payload = client.list_buckets()
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||
buckets = payload.get("Buckets") if isinstance(payload, Mapping) else None
|
||||
if not isinstance(buckets, list):
|
||||
raise ConnectorBrowseError("S3 connector returned invalid bucket list")
|
||||
return sorted(
|
||||
(
|
||||
ConnectorBrowseItem(
|
||||
kind="library",
|
||||
name=_clean(item.get("Name")) or "",
|
||||
path=_clean(item.get("Name")) or "",
|
||||
external_id=_clean(item.get("Name")),
|
||||
modified_at=_timestamp(item.get("CreationDate")),
|
||||
metadata={"bucket": _clean(item.get("Name"))},
|
||||
)
|
||||
for item in buckets
|
||||
if isinstance(item, Mapping) and _clean(item.get("Name"))
|
||||
),
|
||||
key=lambda item: item.name.casefold(),
|
||||
)
|
||||
prefix = _s3_object_key(profile, path, directory=True)
|
||||
params: dict[str, object] = {
|
||||
"Bucket": bucket,
|
||||
"Prefix": prefix,
|
||||
"Delimiter": "/",
|
||||
"MaxKeys": _s3_max_keys(profile),
|
||||
}
|
||||
next_page_request = _clean(continuation_token) or _metadata_string(profile, "continuation_token")
|
||||
if next_page_request:
|
||||
params["ContinuationToken"] = next_page_request
|
||||
try:
|
||||
payload = client.list_objects_v2(**params)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||
if not isinstance(payload, Mapping):
|
||||
raise ConnectorBrowseError("S3 connector returned invalid object listing")
|
||||
items = [
|
||||
*_s3_prefix_items(bucket=bucket, browse_path=path, base_prefix=prefix, prefixes=payload.get("CommonPrefixes")),
|
||||
*_s3_object_items(bucket=bucket, browse_path=path, base_prefix=prefix, objects=payload.get("Contents")),
|
||||
]
|
||||
next_token = _clean(payload.get("NextContinuationToken"))
|
||||
if next_token and items:
|
||||
last = items[-1]
|
||||
items[-1] = ConnectorBrowseItem(
|
||||
kind=last.kind,
|
||||
name=last.name,
|
||||
path=last.path,
|
||||
external_id=last.external_id,
|
||||
external_url=last.external_url,
|
||||
size_bytes=last.size_bytes,
|
||||
content_type=last.content_type,
|
||||
modified_at=last.modified_at,
|
||||
etag=last.etag,
|
||||
metadata={**dict(last.metadata), "next_continuation_token": next_token, "listing_truncated": True},
|
||||
)
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _s3_client(profile: ConnectorProfile) -> Any:
|
||||
if profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref S3 credentials need a runtime secret resolver before live browsing")
|
||||
if profile.endpoint_url:
|
||||
try:
|
||||
endpoint_url = validate_outbound_http_url(
|
||||
profile.endpoint_url,
|
||||
label="S3 connector endpoint",
|
||||
)
|
||||
except OutboundHttpError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
else:
|
||||
endpoint_url = None
|
||||
try:
|
||||
config_module = import_module("botocore.config")
|
||||
unsigned = import_module("botocore").UNSIGNED
|
||||
except ImportError as exc:
|
||||
raise ConnectorBrowseUnsupported("S3 connector browsing requires the optional boto3 dependency") from exc
|
||||
kwargs: dict[str, object] = {}
|
||||
if endpoint_url:
|
||||
kwargs["endpoint_url"] = endpoint_url
|
||||
region = _metadata_string(profile, "region") or _metadata_string(profile, "aws_region")
|
||||
if region:
|
||||
kwargs["region_name"] = region
|
||||
access_key = profile.username or _metadata_env(profile, "access_key_id_env") or _metadata_string(profile, "access_key_id")
|
||||
secret_key = _profile_password(profile) or _metadata_env(profile, "secret_access_key_env")
|
||||
session_token = _profile_token(profile) or _metadata_env(profile, "session_token_env")
|
||||
if access_key:
|
||||
kwargs["aws_access_key_id"] = access_key
|
||||
if secret_key:
|
||||
kwargs["aws_secret_access_key"] = secret_key
|
||||
if session_token:
|
||||
kwargs["aws_session_token"] = session_token
|
||||
verify = _s3_verify(profile)
|
||||
if verify is not None:
|
||||
kwargs["verify"] = verify
|
||||
addressing_style = _s3_addressing_style(profile)
|
||||
config_values: dict[str, object] = {
|
||||
"proxies": {},
|
||||
"retries": {"mode": "standard", "max_attempts": 4},
|
||||
}
|
||||
if addressing_style:
|
||||
config_values["s3"] = {"addressing_style": addressing_style}
|
||||
if bool(access_key) != bool(secret_key):
|
||||
raise ConnectorBrowseError("S3 connectors require both an access key and a secret key")
|
||||
if not access_key:
|
||||
config_values["signature_version"] = unsigned
|
||||
kwargs["config"] = config_module.Config(**config_values)
|
||||
try:
|
||||
return create_pinned_s3_client(**kwargs)
|
||||
except SdkPeerPinningError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector could not be initialized: {exc}") from exc
|
||||
|
||||
|
||||
def _s3_bucket(profile: ConnectorProfile, library_id: str | None = None) -> str | None:
|
||||
return _metadata_string(profile, "bucket") or _metadata_string(profile, "bucket_name") or _clean(library_id)
|
||||
|
||||
|
||||
def _s3_object_key(profile: ConnectorProfile, path: str, *, directory: bool = False) -> str:
|
||||
base_prefix = normalize_connector_browse_path(profile.base_path or _metadata_string(profile, "base_prefix"))
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
key = "/".join(part for part in (base_prefix, browse_path) if part)
|
||||
if directory and key:
|
||||
return key.rstrip("/") + "/"
|
||||
return key
|
||||
|
||||
|
||||
def _s3_prefix_items(
|
||||
*,
|
||||
bucket: str,
|
||||
browse_path: str,
|
||||
base_prefix: str,
|
||||
prefixes: object,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(prefixes, list):
|
||||
return []
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for item in prefixes:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
key = _clean(item.get("Prefix"))
|
||||
if not key:
|
||||
continue
|
||||
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||
name = _path_name(relative)
|
||||
if not name:
|
||||
continue
|
||||
path = _join_browse_path(browse_path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder",
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=f"{bucket}:{key}",
|
||||
metadata={"bucket": bucket, "key": key},
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _s3_object_items(
|
||||
*,
|
||||
bucket: str,
|
||||
browse_path: str,
|
||||
base_prefix: str,
|
||||
objects: object,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(objects, list):
|
||||
return []
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for item in objects:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
key = _clean(item.get("Key"))
|
||||
if not key or key == base_prefix:
|
||||
continue
|
||||
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||
name = _path_name(relative)
|
||||
if not name:
|
||||
continue
|
||||
path = _join_browse_path(browse_path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="file",
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=f"{bucket}:{key}",
|
||||
size_bytes=_int(item.get("Size")),
|
||||
content_type=mimetypes.guess_type(name)[0],
|
||||
modified_at=_timestamp(item.get("LastModified")),
|
||||
etag=_clean(item.get("ETag")),
|
||||
metadata={
|
||||
"bucket": bucket,
|
||||
"key": key,
|
||||
**({"storage_class": item["StorageClass"]} if "StorageClass" in item else {}),
|
||||
},
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _s3_relative_key(key: str, *, base_prefix: str) -> str:
|
||||
clean_key = key.rstrip("/")
|
||||
clean_base = base_prefix.rstrip("/")
|
||||
if clean_base and clean_key.startswith(f"{clean_base}/"):
|
||||
return clean_key[len(clean_base) + 1 :]
|
||||
return clean_key
|
||||
|
||||
|
||||
def _s3_max_keys(profile: ConnectorProfile) -> int:
|
||||
configured = _int(profile.metadata.get("max_keys"))
|
||||
if configured is None:
|
||||
return 1000
|
||||
return max(1, min(configured, 1000))
|
||||
|
||||
|
||||
def _s3_verify(profile: ConnectorProfile) -> bool | str | None:
|
||||
try:
|
||||
validate_connector_tls_metadata(profile.metadata)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
ca_bundle = _metadata_string(profile, "ca_bundle")
|
||||
if ca_bundle:
|
||||
try:
|
||||
return connector_ca_bundle_path(ca_bundle)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
if "verify_tls" in profile.metadata:
|
||||
return _metadata_bool(profile, "verify_tls", default=True)
|
||||
if "tls_verify" in profile.metadata:
|
||||
return _metadata_bool(profile, "tls_verify", default=True)
|
||||
return None
|
||||
|
||||
|
||||
def _s3_addressing_style(profile: ConnectorProfile) -> str | None:
|
||||
style = _metadata_string(profile, "addressing_style")
|
||||
if style in {"path", "virtual", "auto"}:
|
||||
return style
|
||||
if _metadata_bool(profile, "path_style", default=False):
|
||||
return "path"
|
||||
return None
|
||||
|
||||
|
||||
def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]:
|
||||
token = _seafile_token(profile)
|
||||
return {"Authorization": f"Token {token}", "Accept": "application/json"}
|
||||
|
||||
|
||||
def _seafile_token(profile: ConnectorProfile) -> str:
|
||||
token = _profile_token(profile)
|
||||
if token:
|
||||
return token
|
||||
password = _profile_password(profile)
|
||||
if profile.username and password:
|
||||
payload = _request_json(
|
||||
"POST",
|
||||
_seafile_url(profile, "api2/auth-token/"),
|
||||
data={"username": profile.username, "password": password},
|
||||
)
|
||||
if not isinstance(payload, Mapping) or not _clean(payload.get("token")):
|
||||
raise ConnectorBrowseError("Seafile did not return an account token")
|
||||
return _clean(payload.get("token")) or ""
|
||||
if profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref Seafile credentials need a runtime secret resolver before live browsing")
|
||||
raise ConnectorBrowseError("Seafile connector profiles require token credentials or username plus password credentials")
|
||||
|
||||
|
||||
def _seafile_url(profile: ConnectorProfile, suffix: str) -> str:
|
||||
if not profile.endpoint_url:
|
||||
raise ConnectorBrowseError("Seafile connector profile does not define endpoint_url")
|
||||
return urljoin(profile.endpoint_url.rstrip("/") + "/", suffix.lstrip("/"))
|
||||
|
||||
|
||||
def _request_json(
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
headers: Mapping[str, str] | None = None,
|
||||
params: Mapping[str, str] | None = None,
|
||||
data: Mapping[str, str] | None = None,
|
||||
) -> object:
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
method,
|
||||
url,
|
||||
headers=dict(headers or {}),
|
||||
params=params,
|
||||
data=data,
|
||||
timeout=15.0,
|
||||
label="Seafile API",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||
if response.status_code not in {200, 201}:
|
||||
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||
try:
|
||||
return json.loads(response.content)
|
||||
except (UnicodeDecodeError, ValueError) as exc:
|
||||
raise ConnectorBrowseError("Connector returned invalid JSON") from exc
|
||||
|
||||
|
||||
def _seafile_repo_and_path(*, path: str, library_id: str | None) -> tuple[str | None, str]:
|
||||
repo_id = _clean(library_id)
|
||||
if repo_id:
|
||||
return repo_id, path
|
||||
if not path:
|
||||
return None, ""
|
||||
first, _, rest = path.partition("/")
|
||||
return first, rest
|
||||
|
||||
|
||||
def _seafile_library_item(value: Mapping[str, Any]) -> ConnectorBrowseItem:
|
||||
repo_id = _clean(value.get("id") or value.get("repo_id"))
|
||||
name = _clean(value.get("name") or value.get("repo_name") or repo_id)
|
||||
if not repo_id or not name:
|
||||
raise ConnectorBrowseError("Seafile library entries require id and name")
|
||||
return ConnectorBrowseItem(
|
||||
kind="library",
|
||||
name=name,
|
||||
path=repo_id,
|
||||
external_id=repo_id,
|
||||
size_bytes=_int(value.get("size") or value.get("repo_size")),
|
||||
modified_at=_timestamp(value.get("mtime")),
|
||||
metadata={
|
||||
key: value[key]
|
||||
for key in ("type", "permission", "encrypted", "owner", "file_count")
|
||||
if key in value
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _seafile_directory_item(value: Mapping[str, Any], *, parent_path: str) -> ConnectorBrowseItem:
|
||||
name = _clean(value.get("name") or value.get("obj_name"))
|
||||
if not name:
|
||||
raise ConnectorBrowseError("Seafile directory entries require name")
|
||||
item_type = str(value.get("type") or ("dir" if value.get("is_dir") else "file")).casefold()
|
||||
kind = "folder" if item_type in {"dir", "folder"} else "file"
|
||||
path = _join_browse_path(parent_path, name)
|
||||
content_type = None if kind == "folder" else mimetypes.guess_type(name)[0]
|
||||
return ConnectorBrowseItem(
|
||||
kind=kind,
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=_clean(value.get("id") or value.get("obj_id")),
|
||||
size_bytes=None if kind == "folder" else _int(value.get("size")),
|
||||
content_type=content_type,
|
||||
modified_at=_timestamp(value.get("mtime") or value.get("modified")),
|
||||
etag=_clean(value.get("id") or value.get("obj_id")),
|
||||
metadata={
|
||||
key: value[key]
|
||||
for key in ("permission", "modifier_email", "modifier_name")
|
||||
if key in value
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _static_listing(profile: ConnectorProfile, *, path: str, library_id: str | None) -> list[ConnectorBrowseItem] | None:
|
||||
listing = profile.metadata.get("static_listing")
|
||||
if listing is None:
|
||||
return None
|
||||
if isinstance(listing, list):
|
||||
raw_items = listing if path == "" else []
|
||||
elif isinstance(listing, Mapping):
|
||||
key = library_id or path
|
||||
raw_items = listing.get(key)
|
||||
if raw_items is None and key == "":
|
||||
raw_items = listing.get("/")
|
||||
else:
|
||||
raise ConnectorBrowseError("Connector static_listing metadata must be a list or object")
|
||||
if raw_items is None:
|
||||
return []
|
||||
if not isinstance(raw_items, list):
|
||||
raise ConnectorBrowseError("Connector static_listing entries must be lists")
|
||||
return sorted(
|
||||
[_static_item(item, parent_path=path) for item in raw_items if isinstance(item, Mapping)],
|
||||
key=lambda item: (item.kind != "library", item.kind != "folder", item.name.casefold(), item.path.casefold()),
|
||||
)
|
||||
|
||||
|
||||
def _static_item(value: Mapping[str, Any], *, parent_path: str) -> ConnectorBrowseItem:
|
||||
kind = str(value.get("kind") or "file").strip().casefold()
|
||||
if kind not in {"library", "folder", "file"}:
|
||||
raise ConnectorBrowseError("Connector browse items must be library, folder or file")
|
||||
name = str(value.get("name") or value.get("path") or "").strip()
|
||||
if not name:
|
||||
raise ConnectorBrowseError("Connector browse items require name")
|
||||
path = normalize_connector_browse_path(value.get("path"))
|
||||
if not path:
|
||||
path = _join_browse_path(parent_path, name)
|
||||
return ConnectorBrowseItem(
|
||||
kind=kind,
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=_clean(value.get("external_id")),
|
||||
external_url=_clean(value.get("external_url")),
|
||||
size_bytes=_int(value.get("size_bytes")),
|
||||
content_type=_clean(value.get("content_type")),
|
||||
modified_at=_clean(value.get("modified_at")),
|
||||
etag=_clean(value.get("etag")),
|
||||
metadata=value.get("metadata") if isinstance(value.get("metadata"), Mapping) else {},
|
||||
)
|
||||
|
||||
|
||||
def _webdav_url(root_url: str, path: str) -> str:
|
||||
base = root_url if root_url.endswith("/") else root_url + "/"
|
||||
if not path:
|
||||
return base
|
||||
quoted = "/".join(quote(part, safe="") for part in path.split("/") if part)
|
||||
return urljoin(base, quoted + "/")
|
||||
|
||||
|
||||
def _url_path_root(root_url: str) -> str:
|
||||
path = unquote(urlsplit(root_url).path or "/")
|
||||
return path if path.endswith("/") else path + "/"
|
||||
|
||||
|
||||
def _relative_href_path(root_path: str, href: str) -> str:
|
||||
href_path = unquote(urlsplit(href).path or href).strip()
|
||||
if href_path.startswith(root_path):
|
||||
return normalize_connector_browse_path(href_path[len(root_path):])
|
||||
return normalize_connector_browse_path(href_path.rsplit("/", 1)[-1])
|
||||
|
||||
|
||||
def _webdav_prop(response: ET.Element) -> ET.Element:
|
||||
prop = response.find("{DAV:}propstat/{DAV:}prop")
|
||||
if prop is not None:
|
||||
return prop
|
||||
prop = response.find(".//{DAV:}prop")
|
||||
return prop if prop is not None else ET.Element("prop")
|
||||
|
||||
|
||||
def _webdav_display_name(prop: ET.Element | None) -> str | None:
|
||||
return _clean(_text(prop, "{DAV:}displayname"))
|
||||
|
||||
|
||||
def _text(prop: ET.Element | None, tag: str) -> str | None:
|
||||
if prop is None:
|
||||
return None
|
||||
child = prop.find(tag)
|
||||
return child.text.strip() if child is not None and child.text else None
|
||||
|
||||
|
||||
def _http_date(value: str | None) -> str | None:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return parsedate_to_datetime(value).isoformat()
|
||||
except (TypeError, ValueError):
|
||||
return value
|
||||
|
||||
|
||||
def _timestamp(value: object) -> str | None:
|
||||
number = _int(value)
|
||||
if number is None:
|
||||
return _clean(value)
|
||||
return datetime.fromtimestamp(number, tz=timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _metadata_string(profile: ConnectorProfile, key: str) -> str | None:
|
||||
return _clean(profile.metadata.get(key))
|
||||
|
||||
|
||||
def _metadata_bool(profile: ConnectorProfile, key: str, *, default: bool = False) -> bool:
|
||||
value = profile.metadata.get(key)
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
return str(value).strip().casefold() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _metadata_env(profile: ConnectorProfile, key: str) -> str | None:
|
||||
env_name = _metadata_string(profile, key)
|
||||
if not env_name:
|
||||
return None
|
||||
return _env_required(env_name, profile)
|
||||
|
||||
|
||||
def _env_required(name: str, profile: ConnectorProfile) -> str:
|
||||
try:
|
||||
return connector_secret_env_value(name, source_kind=profile.source_kind)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(f"Connector profile {profile.id}: {exc}") from exc
|
||||
|
||||
|
||||
def normalize_connector_browse_path(value: object) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
path = str(value).replace("\\", "/").strip().strip("/")
|
||||
parts = [part for part in path.split("/") if part and part not in {"."}]
|
||||
if any(part == ".." for part in parts):
|
||||
raise ConnectorBrowseError("Connector browse paths cannot contain parent directory segments")
|
||||
return "/".join(parts)
|
||||
|
||||
|
||||
def _join_browse_path(parent: str, name: str) -> str:
|
||||
child = normalize_connector_browse_path(name)
|
||||
if not parent:
|
||||
return child
|
||||
return f"{parent.rstrip('/')}/{child}"
|
||||
|
||||
|
||||
def _path_name(path: str) -> str:
|
||||
return path.rstrip("/").rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def _int(value: object) -> int | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _SmbLocation:
|
||||
server: str
|
||||
share: str
|
||||
port: int
|
||||
root_path: str = ""
|
||||
|
||||
|
||||
def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
|
||||
if not profile.endpoint_url:
|
||||
raise ConnectorBrowseError("SMB connector profile does not define endpoint_url")
|
||||
parsed = urlsplit(profile.endpoint_url)
|
||||
if parsed.scheme.casefold() != "smb":
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must use smb://")
|
||||
server = _clean(parsed.hostname)
|
||||
if not server:
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must include a server")
|
||||
port = parsed.port or _int(profile.metadata.get("port")) or 445
|
||||
try:
|
||||
validate_outbound_host(server, port=port, label="SMB connector endpoint")
|
||||
except OutboundHttpError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part]
|
||||
if not path_parts:
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must include a share name")
|
||||
root_parts = path_parts[1:]
|
||||
if profile.base_path:
|
||||
root_parts.extend(normalize_connector_browse_path(profile.base_path).split("/"))
|
||||
return _SmbLocation(
|
||||
server=server,
|
||||
share=path_parts[0],
|
||||
port=port,
|
||||
root_path=normalize_connector_browse_path("/".join(root_parts)),
|
||||
)
|
||||
|
||||
|
||||
def _smb_unc_path(location: _SmbLocation, path: str) -> str:
|
||||
parts = [part for part in (location.root_path, normalize_connector_browse_path(path)) if part]
|
||||
suffix = "\\".join(part.replace("/", "\\") for part in parts)
|
||||
base = f"\\\\{location.server}\\{location.share}"
|
||||
return f"{base}\\{suffix}" if suffix else base
|
||||
|
||||
|
||||
def _smb_client_kwargs(profile: ConnectorProfile, location: _SmbLocation) -> dict[str, object]:
|
||||
kwargs: dict[str, object] = {
|
||||
"port": location.port,
|
||||
"connection_cache": pinned_smb_connection_cache(),
|
||||
"require_signing": _metadata_bool(profile, "require_signing", default=True),
|
||||
"auth_protocol": _metadata_string(profile, "auth_protocol") or "ntlm",
|
||||
}
|
||||
if "encrypt" in profile.metadata:
|
||||
kwargs["encrypt"] = _metadata_bool(profile, "encrypt", default=False)
|
||||
password = _profile_password(profile)
|
||||
token = _profile_token(profile)
|
||||
if profile.username and password:
|
||||
kwargs["username"] = profile.username
|
||||
kwargs["password"] = password
|
||||
elif token:
|
||||
raise ConnectorBrowseError("SMB connector profiles do not support bearer-token credentials")
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref SMB credentials need a runtime secret resolver before live browsing")
|
||||
return kwargs
|
||||
|
||||
|
||||
def _profile_password(profile: ConnectorProfile) -> str | None:
|
||||
if profile.password_value:
|
||||
return profile.password_value
|
||||
if profile.password_env:
|
||||
return _env_required(profile.password_env, profile)
|
||||
return None
|
||||
|
||||
|
||||
def _profile_token(profile: ConnectorProfile) -> str | None:
|
||||
if profile.token_value:
|
||||
return profile.token_value
|
||||
if profile.token_env:
|
||||
return _env_required(profile.token_env, profile)
|
||||
return None
|
||||
|
||||
|
||||
def _smbclient_module() -> Any:
|
||||
try:
|
||||
return install_pinned_smb_transport(import_module("smbclient"))
|
||||
except ImportError as exc:
|
||||
raise ConnectorBrowseUnsupported("SMB connector browsing requires the optional smbprotocol dependency") from exc
|
||||
except SdkPeerPinningError as exc:
|
||||
raise ConnectorBrowseUnsupported(str(exc)) from exc
|
||||
|
||||
|
||||
def _smb_entry_stat(entry: object) -> object | None:
|
||||
try:
|
||||
return entry.stat() # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _smb_stat_size(stat_result: object | None) -> int | None:
|
||||
return _int(getattr(stat_result, "st_size", None))
|
||||
|
||||
|
||||
def _smb_stat_modified_at(stat_result: object | None) -> str | None:
|
||||
value = getattr(stat_result, "st_mtime", None)
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromtimestamp(float(value), tz=timezone.utc).isoformat()
|
||||
except (TypeError, ValueError, OSError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _smb_stat_revision(stat_result: object | None) -> str | None:
|
||||
mtime_ns = getattr(stat_result, "st_mtime_ns", None)
|
||||
size = getattr(stat_result, "st_size", None)
|
||||
if mtime_ns is not None:
|
||||
return f"{mtime_ns}:{size or 0}"
|
||||
modified = getattr(stat_result, "st_mtime", None)
|
||||
if modified is not None:
|
||||
return f"{modified}:{size or 0}"
|
||||
return None
|
||||
@@ -0,0 +1,337 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import inspect
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_event
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorCredentialDeletionResult:
|
||||
changed: bool
|
||||
affected_profiles: tuple[FileConnectorProfile, ...] = ()
|
||||
|
||||
|
||||
def delete_connector_credential_row(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> ConnectorCredentialDeletionResult:
|
||||
"""Scrub a credential tombstone and disable every profile that used it.
|
||||
|
||||
``secret_ref`` predates a Files-owned secret-provider contract. It may be
|
||||
shared or deployment-owned, so it is detached locally and explicitly
|
||||
audited without ever being passed to a provider delete operation.
|
||||
"""
|
||||
|
||||
dependent_profiles = (
|
||||
tuple(
|
||||
session.query(FileConnectorProfile)
|
||||
.filter(FileConnectorProfile.credential_profile_id == row.id)
|
||||
.order_by(FileConnectorProfile.id.asc())
|
||||
.all()
|
||||
)
|
||||
if inspect(session.get_bind()).has_table(FileConnectorProfile.__tablename__)
|
||||
else ()
|
||||
)
|
||||
affected_profiles: list[FileConnectorProfile] = []
|
||||
for profile in dependent_profiles:
|
||||
if _delete_connector_profile_row(
|
||||
session,
|
||||
profile,
|
||||
deletion_reason="credential_deleted",
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
):
|
||||
affected_profiles.append(profile)
|
||||
|
||||
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||
removed_reference_kinds = _credential_reference_kinds(row)
|
||||
removed_metadata = bool(row.metadata_)
|
||||
changed = _credential_row_requires_deletion(row)
|
||||
if changed:
|
||||
_scrub_credential_row(row, user_id=user_id)
|
||||
session.add(row)
|
||||
_audit_credential_deletion(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
deleted_secret_kinds=deleted_secret_kinds,
|
||||
removed_reference_kinds=removed_reference_kinds,
|
||||
removed_metadata=removed_metadata,
|
||||
affected_profile_count=len(affected_profiles),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
session.flush()
|
||||
return ConnectorCredentialDeletionResult(
|
||||
changed=changed or bool(affected_profiles),
|
||||
affected_profiles=tuple(affected_profiles),
|
||||
)
|
||||
|
||||
|
||||
def delete_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> bool:
|
||||
changed = _delete_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
session.flush()
|
||||
return changed
|
||||
|
||||
|
||||
def delete_connector_credentials_for_retirement(session: Session) -> int:
|
||||
"""Scrub and audit stored connector material before destructive retirement.
|
||||
|
||||
Legacy external references are detached and audited as non-owned. Files
|
||||
never sends those arbitrary references to a provider delete operation.
|
||||
"""
|
||||
|
||||
inspector = inspect(session.get_bind())
|
||||
profiles = (
|
||||
session.query(FileConnectorProfile).order_by(FileConnectorProfile.id.asc()).all()
|
||||
if inspector.has_table(FileConnectorProfile.__tablename__)
|
||||
else []
|
||||
)
|
||||
credentials = (
|
||||
session.query(FileConnectorCredential).order_by(FileConnectorCredential.id.asc()).all()
|
||||
if inspector.has_table(FileConnectorCredential.__tablename__)
|
||||
else []
|
||||
)
|
||||
deleted = 0
|
||||
for profile in profiles:
|
||||
if not _profile_row_has_credential_material(profile):
|
||||
continue
|
||||
if _delete_connector_profile_row(
|
||||
session,
|
||||
profile,
|
||||
deletion_reason="module_data_retired",
|
||||
):
|
||||
deleted += 1
|
||||
for credential in credentials:
|
||||
if not _credential_row_has_material(credential):
|
||||
continue
|
||||
result = delete_connector_credential_row(
|
||||
session,
|
||||
credential,
|
||||
deletion_reason="module_data_retired",
|
||||
)
|
||||
if result.changed:
|
||||
deleted += 1
|
||||
session.flush()
|
||||
return deleted
|
||||
|
||||
|
||||
def _delete_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> bool:
|
||||
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||
removed_reference_kinds = _profile_reference_kinds(row)
|
||||
removed_metadata = bool(row.metadata_)
|
||||
changed = _profile_row_requires_deletion(row)
|
||||
if not changed:
|
||||
return False
|
||||
_scrub_profile_row(row, user_id=user_id)
|
||||
session.add(row)
|
||||
_audit_profile_deletion(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
deleted_secret_kinds=deleted_secret_kinds,
|
||||
removed_reference_kinds=removed_reference_kinds,
|
||||
removed_metadata=removed_metadata,
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def _scrub_credential_row(row: FileConnectorCredential, *, user_id: str | None) -> None:
|
||||
row.enabled = False
|
||||
row.credential_mode = "none"
|
||||
row.username = None
|
||||
row.password_encrypted = None
|
||||
row.token_encrypted = None
|
||||
row.password_env = None
|
||||
row.token_env = None
|
||||
row.secret_ref = None
|
||||
row.metadata_ = {}
|
||||
row.updated_by_user_id = user_id
|
||||
|
||||
|
||||
def _scrub_profile_row(row: FileConnectorProfile, *, user_id: str | None) -> None:
|
||||
row.enabled = False
|
||||
row.credential_profile_id = None
|
||||
row.credential_mode = "none"
|
||||
row.username = None
|
||||
row.password_encrypted = None
|
||||
row.token_encrypted = None
|
||||
row.password_env = None
|
||||
row.token_env = None
|
||||
row.secret_ref = None
|
||||
row.metadata_ = {}
|
||||
row.updated_by_user_id = user_id
|
||||
|
||||
|
||||
def _credential_row_requires_deletion(row: FileConnectorCredential) -> bool:
|
||||
return bool(row.enabled or _credential_row_has_material(row))
|
||||
|
||||
|
||||
def _profile_row_requires_deletion(row: FileConnectorProfile) -> bool:
|
||||
return bool(row.enabled or _profile_row_has_credential_material(row))
|
||||
|
||||
|
||||
def _credential_row_has_material(row: FileConnectorCredential) -> bool:
|
||||
return bool(
|
||||
row.username
|
||||
or row.password_encrypted
|
||||
or row.token_encrypted
|
||||
or row.password_env
|
||||
or row.token_env
|
||||
or row.secret_ref
|
||||
or row.metadata_
|
||||
or row.credential_mode not in {"", "none", "anonymous"}
|
||||
)
|
||||
|
||||
|
||||
def _profile_row_has_credential_material(row: FileConnectorProfile) -> bool:
|
||||
return bool(
|
||||
row.credential_profile_id
|
||||
or row.username
|
||||
or row.password_encrypted
|
||||
or row.token_encrypted
|
||||
or row.password_env
|
||||
or row.token_env
|
||||
or row.secret_ref
|
||||
or row.metadata_
|
||||
or row.credential_mode not in {"", "none", "anonymous"}
|
||||
)
|
||||
|
||||
|
||||
def _encrypted_secret_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||
kinds: list[str] = []
|
||||
if row.password_encrypted:
|
||||
kinds.append("password")
|
||||
if row.token_encrypted:
|
||||
kinds.append("token")
|
||||
return kinds
|
||||
|
||||
|
||||
def _credential_reference_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||
kinds: list[str] = []
|
||||
if row.password_env:
|
||||
kinds.append("password_env")
|
||||
if row.token_env:
|
||||
kinds.append("token_env")
|
||||
if row.secret_ref:
|
||||
kinds.append("unowned_external_secret_ref")
|
||||
return kinds
|
||||
|
||||
|
||||
def _profile_reference_kinds(row: FileConnectorProfile) -> list[str]:
|
||||
kinds = _credential_reference_kinds(row)
|
||||
if row.credential_profile_id:
|
||||
kinds.append("credential_profile")
|
||||
return kinds
|
||||
|
||||
|
||||
def _storage_backend(deleted_secret_kinds: list[str], removed_reference_kinds: list[str]) -> str:
|
||||
if "unowned_external_secret_ref" in removed_reference_kinds:
|
||||
return "unowned_external_reference_detached"
|
||||
if deleted_secret_kinds:
|
||||
return "encrypted_database"
|
||||
if removed_reference_kinds:
|
||||
return "reference_only"
|
||||
return "none"
|
||||
|
||||
|
||||
def _audit_credential_deletion(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
deleted_secret_kinds: list[str],
|
||||
removed_reference_kinds: list[str],
|
||||
removed_metadata: bool,
|
||||
affected_profile_count: int,
|
||||
user_id: str | None,
|
||||
api_key_id: str | None,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=row.tenant_id,
|
||||
scope=_audit_scope(row.scope_type),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
action="files.connector_credential_deleted",
|
||||
object_type="file_connector_credential",
|
||||
object_id=row.id,
|
||||
details={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||
"deleted_secret_kinds": deleted_secret_kinds,
|
||||
"removed_reference_kinds": removed_reference_kinds,
|
||||
"removed_metadata": removed_metadata,
|
||||
"affected_profile_count": affected_profile_count,
|
||||
"deletion_reason": deletion_reason,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _audit_profile_deletion(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
deleted_secret_kinds: list[str],
|
||||
removed_reference_kinds: list[str],
|
||||
removed_metadata: bool,
|
||||
user_id: str | None,
|
||||
api_key_id: str | None,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=row.tenant_id,
|
||||
scope=_audit_scope(row.scope_type),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
action="files.connector_profile_deleted",
|
||||
object_type="file_connector_profile",
|
||||
object_id=row.id,
|
||||
details={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||
"deleted_secret_kinds": deleted_secret_kinds,
|
||||
"removed_reference_kinds": removed_reference_kinds,
|
||||
"removed_metadata": removed_metadata,
|
||||
"deletion_reason": deletion_reason,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _audit_scope(scope_type: str) -> str:
|
||||
return "system" if scope_type == "system" else "tenant"
|
||||
@@ -0,0 +1,540 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import inspect
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||
from govoplan_core.security.credential_envelopes import (
|
||||
CredentialAccessContext,
|
||||
CredentialEnvelope,
|
||||
CredentialEnvelopeError,
|
||||
ResolvedCredentialEnvelope,
|
||||
list_credential_envelopes,
|
||||
resolve_credential_envelope,
|
||||
)
|
||||
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
|
||||
|
||||
CORE_CREDENTIAL_ENVELOPE_PREFIX = "credential-envelope:"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorCredential:
|
||||
id: str
|
||||
label: str
|
||||
scope_type: str
|
||||
scope_id: str | None = None
|
||||
provider: str | None = None
|
||||
enabled: bool = True
|
||||
credential_mode: str = "none"
|
||||
username: str | None = None
|
||||
password_env: str | None = None
|
||||
token_env: str | None = None
|
||||
secret_ref: str | None = None
|
||||
password_value: str | None = None
|
||||
token_value: str | None = None
|
||||
policy_sources: tuple[ConnectorPolicySource, ...] = ()
|
||||
metadata: Mapping[str, Any] | None = None
|
||||
source_kind: str = "database"
|
||||
has_secret: bool = False
|
||||
|
||||
@property
|
||||
def source_path(self) -> str:
|
||||
return policy_source_path(self.scope_type, self.scope_id)
|
||||
|
||||
@property
|
||||
def credential_secret_source(self) -> str | None:
|
||||
if self.secret_ref:
|
||||
return "secret_ref"
|
||||
if self.password_value or self.token_value:
|
||||
return "stored"
|
||||
if self.token_env:
|
||||
return "token_env"
|
||||
if self.password_env:
|
||||
return "password_env"
|
||||
return None
|
||||
|
||||
@property
|
||||
def credentials_configured(self) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
|
||||
return True
|
||||
# Environment references are deliberately unavailable to API-managed
|
||||
# credential rows. Legacy rows remain visible but fail closed.
|
||||
return bool(self.has_secret or self.secret_ref or self.password_value or self.token_value)
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
"id": self.id,
|
||||
"label": self.label,
|
||||
"provider": self.provider,
|
||||
"enabled": self.enabled,
|
||||
"scope_type": self.scope_type,
|
||||
"scope_id": self.scope_id,
|
||||
"source_path": self.source_path,
|
||||
"credential_mode": self.credential_mode,
|
||||
"credential_secret_source": self.credential_secret_source,
|
||||
"credentials_configured": self.credentials_configured,
|
||||
"username": self.username,
|
||||
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||
"metadata": dict(self.metadata or {}),
|
||||
"source_kind": self.source_kind,
|
||||
}
|
||||
|
||||
|
||||
def list_database_connector_credentials(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[ConnectorCredential]:
|
||||
local = [
|
||||
connector_credential_from_row(row)
|
||||
for row in list_connector_credential_rows(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
]
|
||||
return [
|
||||
*local,
|
||||
*list_reusable_connector_credentials(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def reusable_credential_reference(credential_id: str) -> str:
|
||||
return f"{CORE_CREDENTIAL_ENVELOPE_PREFIX}{credential_id}"
|
||||
|
||||
|
||||
def reusable_credential_id(credential_ref: str | None) -> str | None:
|
||||
if not credential_ref or not credential_ref.startswith(CORE_CREDENTIAL_ENVELOPE_PREFIX):
|
||||
return None
|
||||
value = credential_ref.removeprefix(CORE_CREDENTIAL_ENVELOPE_PREFIX).strip()
|
||||
return value or None
|
||||
|
||||
|
||||
def file_credential_context(
|
||||
*,
|
||||
tenant_id: str,
|
||||
profile_id: str | None = None,
|
||||
scope_type: str = "tenant",
|
||||
scope_id: str | None = None,
|
||||
administrative: bool = False,
|
||||
) -> CredentialAccessContext:
|
||||
return CredentialAccessContext(
|
||||
tenant_id=tenant_id,
|
||||
user_id=scope_id if scope_type == "user" else None,
|
||||
group_ids=frozenset({scope_id}) if scope_type == "group" and scope_id else frozenset(),
|
||||
target_scope_type=scope_type,
|
||||
target_scope_id=scope_id or tenant_id,
|
||||
module_id="files",
|
||||
server_ref=f"files:{profile_id}" if profile_id else None,
|
||||
administrative=administrative,
|
||||
)
|
||||
|
||||
|
||||
def list_reusable_connector_credentials(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[ConnectorCredential]:
|
||||
if not inspect(session.get_bind()).has_table(CredentialEnvelope.__tablename__):
|
||||
return []
|
||||
context = file_credential_context(tenant_id=tenant_id, administrative=True)
|
||||
return [
|
||||
connector_credential_from_envelope(row)
|
||||
for row in list_credential_envelopes(
|
||||
session,
|
||||
context=context,
|
||||
include_inactive=include_disabled,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def resolve_reusable_connector_credential(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
credential_ref: str,
|
||||
profile_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
) -> ConnectorCredential:
|
||||
credential_id = reusable_credential_id(credential_ref)
|
||||
if credential_id is None:
|
||||
raise FileStorageError("Reusable credential reference is invalid")
|
||||
try:
|
||||
resolved = resolve_credential_envelope(
|
||||
session,
|
||||
credential_id=credential_id,
|
||||
context=file_credential_context(
|
||||
tenant_id=tenant_id,
|
||||
profile_id=profile_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
),
|
||||
)
|
||||
except CredentialEnvelopeError as exc:
|
||||
raise FileStorageError("Reusable credential is unavailable to this file connection") from exc
|
||||
return connector_credential_from_resolved_envelope(
|
||||
resolved,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
|
||||
|
||||
def connector_credential_from_envelope(row: CredentialEnvelope) -> ConnectorCredential:
|
||||
return ConnectorCredential(
|
||||
id=reusable_credential_reference(row.id),
|
||||
label=row.name,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
enabled=row.is_active,
|
||||
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_keys),
|
||||
username=_clean_public_value(row.public_data, "username"),
|
||||
source_kind="credential_envelope",
|
||||
has_secret=bool(row.secret_data_encrypted),
|
||||
metadata={
|
||||
"credential_envelope_id": row.id,
|
||||
"credential_kind": row.credential_kind,
|
||||
"allowed_modules": list(row.allowed_modules or []),
|
||||
"allowed_server_refs": list(row.allowed_server_refs or []),
|
||||
"inherit_to_lower_scopes": bool(row.inherit_to_lower_scopes),
|
||||
"revision": row.revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def connector_credential_from_resolved_envelope(
|
||||
row: ResolvedCredentialEnvelope,
|
||||
*,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
) -> ConnectorCredential:
|
||||
return ConnectorCredential(
|
||||
id=reusable_credential_reference(row.id),
|
||||
label=row.name,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
enabled=True,
|
||||
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_data),
|
||||
username=_clean_public_value(row.public_data, "username"),
|
||||
password_value=_first_secret(row.secret_data, "password", "secret"),
|
||||
token_value=_first_secret(
|
||||
row.secret_data,
|
||||
"access_token",
|
||||
"bearer_token",
|
||||
"token",
|
||||
"api_key",
|
||||
),
|
||||
source_kind="credential_envelope",
|
||||
has_secret=bool(row.secret_data),
|
||||
metadata={
|
||||
"credential_envelope_id": row.id,
|
||||
"credential_kind": row.credential_kind,
|
||||
"inherit_to_lower_scopes": True,
|
||||
"revision": row.revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def list_connector_credential_rows(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[FileConnectorCredential]:
|
||||
query = session.query(FileConnectorCredential).filter(
|
||||
(FileConnectorCredential.scope_type == "system")
|
||||
| (FileConnectorCredential.tenant_id == tenant_id)
|
||||
)
|
||||
if not include_disabled:
|
||||
query = query.filter(FileConnectorCredential.enabled.is_(True))
|
||||
return query.order_by(FileConnectorCredential.scope_type.asc(), FileConnectorCredential.label.asc()).all()
|
||||
|
||||
|
||||
def connector_credential_from_row(row: FileConnectorCredential) -> ConnectorCredential:
|
||||
policy_sources = []
|
||||
if row.policy:
|
||||
policy_sources = connector_policy_sources_from_payload({
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"label": row.label,
|
||||
"policy": row.policy,
|
||||
})
|
||||
return ConnectorCredential(
|
||||
id=row.id,
|
||||
label=row.label,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
provider=row.provider,
|
||||
enabled=row.enabled,
|
||||
credential_mode=row.credential_mode,
|
||||
username=row.username,
|
||||
password_env=row.password_env,
|
||||
token_env=row.token_env,
|
||||
secret_ref=row.secret_ref,
|
||||
password_value=decrypt_secret(row.password_encrypted),
|
||||
token_value=decrypt_secret(row.token_encrypted),
|
||||
policy_sources=tuple(policy_sources),
|
||||
metadata=dict(row.metadata_ or {}),
|
||||
)
|
||||
|
||||
|
||||
def get_connector_credential_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
credential_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> FileConnectorCredential:
|
||||
row = session.get(FileConnectorCredential, credential_id)
|
||||
if row is None or (row.scope_type != "system" and row.tenant_id != tenant_id):
|
||||
raise FileStorageError("Connector credential not found")
|
||||
if not include_disabled and not row.enabled:
|
||||
raise FileStorageError("Connector credential not found")
|
||||
return row
|
||||
|
||||
|
||||
def credential_rows_by_id(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
credential_ids: set[str],
|
||||
include_disabled: bool = False,
|
||||
) -> dict[str, FileConnectorCredential]:
|
||||
if not credential_ids:
|
||||
return {}
|
||||
rows = session.query(FileConnectorCredential).filter(FileConnectorCredential.id.in_(credential_ids)).all()
|
||||
result: dict[str, FileConnectorCredential] = {}
|
||||
for row in rows:
|
||||
if row.scope_type != "system" and row.tenant_id != tenant_id:
|
||||
continue
|
||||
if not include_disabled and not row.enabled:
|
||||
continue
|
||||
result[row.id] = row
|
||||
return result
|
||||
|
||||
|
||||
def create_connector_credential_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str | None,
|
||||
credential_id: str,
|
||||
label: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
provider: str | None = None,
|
||||
enabled: bool = True,
|
||||
credential_mode: str = "none",
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> FileConnectorCredential:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
clean_id = _normalize_id(credential_id)
|
||||
if session.get(FileConnectorCredential, clean_id) is not None:
|
||||
raise FileStorageError(f"Connector credential already exists: {clean_id}")
|
||||
clean_scope_type, clean_scope_id, row_tenant_id = _normalize_scope(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
row = FileConnectorCredential(
|
||||
id=clean_id,
|
||||
tenant_id=row_tenant_id,
|
||||
scope_type=clean_scope_type,
|
||||
scope_id=clean_scope_id,
|
||||
label=_normalize_label(label),
|
||||
provider=_normalize_provider(provider),
|
||||
enabled=bool(enabled),
|
||||
credential_mode=_normalize_credential_mode(credential_mode),
|
||||
username=_clean(username),
|
||||
password_encrypted=encrypt_secret(_clean(password)),
|
||||
token_encrypted=encrypt_secret(_clean(token)),
|
||||
password_env=_clean(password_env),
|
||||
token_env=_clean(token_env),
|
||||
secret_ref=_clean(secret_ref),
|
||||
policy=dict(policy or {}),
|
||||
metadata_=dict(metadata or {}),
|
||||
created_by_user_id=user_id,
|
||||
updated_by_user_id=user_id,
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def update_connector_credential_row(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
user_id: str | None,
|
||||
label: str | None = None,
|
||||
provider: str | None = None,
|
||||
enabled: bool | None = None,
|
||||
credential_mode: str | None = None,
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
clear_password: bool = False,
|
||||
clear_token: bool = False,
|
||||
) -> FileConnectorCredential:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
|
||||
if _clean(row.secret_ref):
|
||||
raise FileStorageError(
|
||||
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
if label is not None:
|
||||
row.label = _normalize_label(label)
|
||||
if provider is not None:
|
||||
row.provider = _normalize_provider(provider)
|
||||
if enabled is not None:
|
||||
row.enabled = bool(enabled)
|
||||
if credential_mode is not None:
|
||||
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||
if username is not None:
|
||||
row.username = _clean(username)
|
||||
if password is not None:
|
||||
row.password_encrypted = encrypt_secret(_clean(password))
|
||||
elif clear_password:
|
||||
row.password_encrypted = None
|
||||
if token is not None:
|
||||
row.token_encrypted = encrypt_secret(_clean(token))
|
||||
elif clear_token:
|
||||
row.token_encrypted = None
|
||||
if password_env is not None:
|
||||
row.password_env = _clean(password_env)
|
||||
if token_env is not None:
|
||||
row.token_env = _clean(token_env)
|
||||
if secret_ref is not None:
|
||||
row.secret_ref = _clean(secret_ref)
|
||||
if policy is not None:
|
||||
row.policy = dict(policy)
|
||||
if metadata is not None:
|
||||
row.metadata_ = dict(metadata)
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return "system", None, None
|
||||
if clean_scope_type == "tenant":
|
||||
return "tenant", tenant_id, tenant_id
|
||||
if clean_scope_type in {"user", "group", "campaign"}:
|
||||
if not clean_scope_id:
|
||||
raise FileStorageError(f"{clean_scope_type.capitalize()} connector credentials require scope_id")
|
||||
return clean_scope_type, clean_scope_id, tenant_id
|
||||
raise FileStorageError("Unsupported connector credential scope")
|
||||
|
||||
|
||||
def _normalize_id(value: str) -> str:
|
||||
clean = _clean(value)
|
||||
if not clean:
|
||||
raise FileStorageError("Connector credential id is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector credential id is too long")
|
||||
if any(char.isspace() for char in clean):
|
||||
raise FileStorageError("Connector credential id cannot contain whitespace")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_label(value: str) -> str:
|
||||
clean = value.strip()
|
||||
if not clean:
|
||||
raise FileStorageError("Connector credential label is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector credential label is too long")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_provider(value: str | None) -> str | None:
|
||||
clean = _clean(value)
|
||||
if clean is None:
|
||||
return None
|
||||
clean = clean.casefold()
|
||||
if clean not in supported_connector_providers():
|
||||
raise FileStorageError(f"Unsupported connector credential provider: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_credential_mode(value: str) -> str:
|
||||
clean = value.strip().casefold() or "none"
|
||||
if clean not in {"none", "anonymous", "basic", "token", "secret_ref"}:
|
||||
raise FileStorageError(f"Unsupported connector credential mode: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||
return {
|
||||
"scope_type": source.scope_type,
|
||||
"scope_id": source.scope_id,
|
||||
"label": source.label,
|
||||
"policy": dict(source.policy or {}),
|
||||
}
|
||||
|
||||
|
||||
def _envelope_credential_mode(kind: str, secret_values: Mapping[str, Any] | list[str]) -> str:
|
||||
keys = {str(key) for key in secret_values}
|
||||
if kind in {"token", "oauth2", "api_key"} or keys.intersection(
|
||||
{"access_token", "bearer_token", "token", "api_key"}
|
||||
):
|
||||
return "token"
|
||||
return "basic"
|
||||
|
||||
|
||||
def _clean_public_value(values: Mapping[str, Any] | None, key: str) -> str | None:
|
||||
return _clean((values or {}).get(key))
|
||||
|
||||
|
||||
def _first_secret(values: Mapping[str, Any], *keys: str) -> str | None:
|
||||
for key in keys:
|
||||
value = _clean(values.get(key))
|
||||
if value is not None:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,241 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
_SECRET_ENV_ALLOWLIST = "GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST" # noqa: S105 # nosec B105 - configuration key.
|
||||
_CA_BUNDLE_ALLOWLIST = "GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST"
|
||||
_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||
_SECRET_ENV_METADATA_KEYS = frozenset(
|
||||
{
|
||||
"access_key_id_env",
|
||||
"secret_access_key_env",
|
||||
"session_token_env",
|
||||
}
|
||||
)
|
||||
_SECRET_VALUE_METADATA_KEYS = frozenset(
|
||||
{
|
||||
"password",
|
||||
"token",
|
||||
"access_token",
|
||||
"auth_token",
|
||||
"bearer_token",
|
||||
"refresh_token",
|
||||
"api_key",
|
||||
"access_key",
|
||||
"access_key_id",
|
||||
"secret_key",
|
||||
"secret_access_key",
|
||||
"session_token",
|
||||
}
|
||||
)
|
||||
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
|
||||
|
||||
|
||||
class ConnectorDeploymentConfigurationError(ValueError):
|
||||
"""Raised when connector data crosses a deployment-owned trust boundary."""
|
||||
|
||||
|
||||
def connector_secret_env_value(name: str, *, source_kind: str) -> str:
|
||||
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||
value = os.environ.get(clean_name)
|
||||
if value is None or value == "":
|
||||
raise ConnectorDeploymentConfigurationError("Connector credential environment variable is not configured")
|
||||
return value
|
||||
|
||||
|
||||
def connector_secret_env_available(name: str | None, *, source_kind: str) -> bool:
|
||||
if not name:
|
||||
return False
|
||||
try:
|
||||
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||
except ConnectorDeploymentConfigurationError:
|
||||
return False
|
||||
return bool(os.environ.get(clean_name))
|
||||
|
||||
|
||||
def validate_deployment_connector_references(
|
||||
*,
|
||||
source_kind: str,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Validate references in deployment-owned connector configuration."""
|
||||
|
||||
for name in (password_env, token_env):
|
||||
if name:
|
||||
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||
for key in _SECRET_ENV_METADATA_KEYS:
|
||||
name = _clean((metadata or {}).get(key))
|
||||
if name:
|
||||
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||
validate_connector_tls_metadata(metadata)
|
||||
|
||||
|
||||
def reject_api_controlled_deployment_references(
|
||||
*,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Reject process-secret selectors controlled through tenant-facing APIs."""
|
||||
|
||||
if _clean(password_env) or _clean(token_env):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||
)
|
||||
if _clean(secret_ref):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"External secret references cannot be managed through the Files API until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
for key, value in _metadata_entries(metadata or {}):
|
||||
clean_key = str(key).strip().casefold()
|
||||
if _clean(value) and (clean_key in _SECRET_ENV_METADATA_KEYS or clean_key.endswith("_env")):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||
)
|
||||
if _clean(value) and (
|
||||
clean_key in _SECRET_VALUE_METADATA_KEYS
|
||||
or clean_key.endswith(("_password", "_secret", "_api_key"))
|
||||
):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Connector credential values must use the dedicated encrypted credential fields, not metadata"
|
||||
)
|
||||
validate_connector_tls_metadata(metadata)
|
||||
|
||||
|
||||
def _metadata_entries(value: object) -> list[tuple[str, object]]:
|
||||
entries: list[tuple[str, object]] = []
|
||||
if isinstance(value, Mapping):
|
||||
for key, item in value.items():
|
||||
entries.append((str(key), item))
|
||||
entries.extend(_metadata_entries(item))
|
||||
elif isinstance(value, (list, tuple)):
|
||||
for item in value:
|
||||
entries.extend(_metadata_entries(item))
|
||||
return entries
|
||||
|
||||
|
||||
def connector_ca_bundle_path(value: str) -> str:
|
||||
clean_value = _clean(value)
|
||||
if not clean_value:
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path is empty")
|
||||
candidate = Path(clean_value)
|
||||
if not candidate.is_absolute():
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle paths must be absolute")
|
||||
try:
|
||||
resolved = candidate.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path does not exist") from exc
|
||||
allowed = _allowed_ca_bundle_paths()
|
||||
if resolved not in allowed:
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
f"Connector CA bundle path is not listed in {_CA_BUNDLE_ALLOWLIST}"
|
||||
)
|
||||
if not resolved.is_file():
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path must be a regular file")
|
||||
return str(resolved)
|
||||
|
||||
|
||||
def validate_connector_tls_metadata(metadata: Mapping[str, Any] | None) -> None:
|
||||
values = metadata or {}
|
||||
ca_bundle = _clean(values.get("ca_bundle"))
|
||||
if ca_bundle:
|
||||
connector_ca_bundle_path(ca_bundle)
|
||||
for key in ("verify_tls", "tls_verify"):
|
||||
if key in values and not _as_bool(values.get(key), default=True) and not _development_runtime():
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Connector TLS certificate verification may only be disabled in dev/test environments"
|
||||
)
|
||||
|
||||
|
||||
def connector_effective_endpoint_url(
|
||||
*,
|
||||
provider: str | None,
|
||||
endpoint_url: str | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> str | None:
|
||||
"""Return the endpoint that connector I/O will actually use."""
|
||||
|
||||
clean_provider = (provider or "").strip().casefold()
|
||||
values = metadata or {}
|
||||
webdav_url = _clean(values.get("webdav_endpoint_url"))
|
||||
browse_protocol = (_clean(values.get("browse_protocol")) or "").casefold()
|
||||
if webdav_url and (clean_provider in {"seafile", "webdav", "nextcloud"} or browse_protocol == "webdav"):
|
||||
return webdav_url
|
||||
return _clean(endpoint_url)
|
||||
|
||||
|
||||
def _validate_secret_env_reference(name: str, *, source_kind: str) -> str:
|
||||
if source_kind.strip().casefold() != "settings":
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be used by deployment-owned connector configuration"
|
||||
)
|
||||
clean_name = name.strip()
|
||||
if not _ENV_NAME.fullmatch(clean_name):
|
||||
raise ConnectorDeploymentConfigurationError("Connector credential environment variable name is invalid")
|
||||
if clean_name not in _allowed_secret_env_names():
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
f"Connector credential environment variable is not listed in {_SECRET_ENV_ALLOWLIST}"
|
||||
)
|
||||
return clean_name
|
||||
|
||||
|
||||
def _allowed_secret_env_names() -> frozenset[str]:
|
||||
raw = os.environ.get(_SECRET_ENV_ALLOWLIST, "")
|
||||
names = frozenset(item.strip() for item in raw.split(",") if item.strip())
|
||||
invalid = sorted(name for name in names if not _ENV_NAME.fullmatch(name))
|
||||
if invalid:
|
||||
raise ConnectorDeploymentConfigurationError(f"{_SECRET_ENV_ALLOWLIST} contains an invalid variable name")
|
||||
return names
|
||||
|
||||
|
||||
def _allowed_ca_bundle_paths() -> frozenset[Path]:
|
||||
raw = os.environ.get(_CA_BUNDLE_ALLOWLIST, "")
|
||||
paths: set[Path] = set()
|
||||
for item in (part.strip() for part in raw.split(",")):
|
||||
if not item:
|
||||
continue
|
||||
path = Path(item)
|
||||
if not path.is_absolute():
|
||||
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} requires absolute paths")
|
||||
try:
|
||||
paths.add(path.resolve(strict=True))
|
||||
except OSError as exc:
|
||||
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} contains a missing path") from exc
|
||||
return frozenset(paths)
|
||||
|
||||
|
||||
def _development_runtime() -> bool:
|
||||
app_env = os.environ.get("APP_ENV", "dev").strip().casefold()
|
||||
install_profile = os.environ.get("GOVOPLAN_INSTALL_PROFILE", "").strip().casefold()
|
||||
return app_env in _DEVELOPMENT_ENVIRONMENTS and (
|
||||
not install_profile or install_profile in _DEVELOPMENT_ENVIRONMENTS
|
||||
)
|
||||
|
||||
|
||||
def _as_bool(value: object, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
clean = str(value).strip().casefold()
|
||||
if clean in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if clean in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
raise ConnectorDeploymentConfigurationError("Connector TLS verification setting must be true or false")
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
clean = str(value).strip()
|
||||
return clean or None
|
||||
@@ -0,0 +1,139 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseItem,
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorFolderSkip:
|
||||
path: str
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ConnectorFolderDiscovery:
|
||||
files: list[ConnectorBrowseItem] = field(default_factory=list)
|
||||
skipped: list[ConnectorFolderSkip] = field(default_factory=list)
|
||||
truncated: bool = False
|
||||
|
||||
|
||||
def discover_connector_folder(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
path: str,
|
||||
library_id: str | None,
|
||||
recursive: bool,
|
||||
max_files: int,
|
||||
max_depth: int,
|
||||
) -> ConnectorFolderDiscovery:
|
||||
"""Discover a bounded, deterministic set of files below one connector folder."""
|
||||
|
||||
root_path = normalize_connector_browse_path(path)
|
||||
queue: deque[tuple[str, int]] = deque([(root_path, 0)])
|
||||
visited_folders: set[str] = set()
|
||||
seen_files: set[tuple[str, str]] = set()
|
||||
result = ConnectorFolderDiscovery()
|
||||
|
||||
while queue:
|
||||
folder_path, depth = queue.popleft()
|
||||
if folder_path in visited_folders:
|
||||
continue
|
||||
visited_folders.add(folder_path)
|
||||
|
||||
continuation_token: str | None = None
|
||||
seen_tokens: set[str] = set()
|
||||
while True:
|
||||
items = browse_connector_profile(
|
||||
profile,
|
||||
path=folder_path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
for item in items:
|
||||
item_path = normalize_connector_browse_path(item.path)
|
||||
if item.kind == "file":
|
||||
identity = (str(item.external_id or ""), item_path)
|
||||
if identity in seen_files:
|
||||
continue
|
||||
if len(result.files) >= max_files:
|
||||
result.truncated = True
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path or folder_path,
|
||||
reason=f"The manual sync limit of {max_files} files was reached.",
|
||||
)
|
||||
)
|
||||
return result
|
||||
seen_files.add(identity)
|
||||
result.files.append(item)
|
||||
continue
|
||||
|
||||
if item.kind == "folder":
|
||||
if not recursive:
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason="Subfolder skipped because recursive sync is disabled.",
|
||||
)
|
||||
)
|
||||
elif depth >= max_depth:
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason=f"Subfolder skipped at the configured depth limit of {max_depth}.",
|
||||
)
|
||||
)
|
||||
else:
|
||||
queue.append((item_path, depth + 1))
|
||||
continue
|
||||
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason=f"Connector item kind {item.kind!r} cannot be synchronized as a file.",
|
||||
)
|
||||
)
|
||||
|
||||
next_token = _next_continuation_token(items)
|
||||
if not next_token or next_token in seen_tokens:
|
||||
break
|
||||
seen_tokens.add(next_token)
|
||||
continuation_token = next_token
|
||||
|
||||
result.files.sort(key=lambda item: normalize_connector_browse_path(item.path).casefold())
|
||||
result.skipped.sort(key=lambda item: (item.path.casefold(), item.reason))
|
||||
return result
|
||||
|
||||
|
||||
def connector_relative_path(*, space_root: str, item_path: str) -> str:
|
||||
root = normalize_connector_browse_path(space_root)
|
||||
item = normalize_connector_browse_path(item_path)
|
||||
if not root:
|
||||
return item
|
||||
if item == root:
|
||||
return item.rsplit("/", 1)[-1]
|
||||
prefix = f"{root}/"
|
||||
if not item.startswith(prefix):
|
||||
raise ValueError("Connector returned a file outside the linked connector-space root")
|
||||
return item[len(prefix) :]
|
||||
|
||||
|
||||
def join_connector_path(*parts: str | None) -> str:
|
||||
return normalize_connector_browse_path(
|
||||
"/".join(str(part or "").strip("/\\") for part in parts if str(part or "").strip("/\\"))
|
||||
)
|
||||
|
||||
|
||||
def _next_continuation_token(items: list[ConnectorBrowseItem]) -> str | None:
|
||||
for item in reversed(items):
|
||||
value = item.metadata.get("next_continuation_token")
|
||||
if value is not None and str(value).strip():
|
||||
return str(value).strip()
|
||||
return None
|
||||
@@ -0,0 +1,358 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.security.outbound_http import response_limit
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
_clean,
|
||||
_metadata_string,
|
||||
_profile_password,
|
||||
_profile_token,
|
||||
_request_json,
|
||||
_smb_client_kwargs,
|
||||
_smb_location,
|
||||
_smb_stat_modified_at,
|
||||
_smb_stat_revision,
|
||||
_smb_stat_size,
|
||||
_smb_unc_path,
|
||||
_smbclient_module,
|
||||
_s3_bucket,
|
||||
_s3_client,
|
||||
_s3_object_key,
|
||||
_seafile_headers,
|
||||
_seafile_url,
|
||||
_webdav_url,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.paths import filename_from_path
|
||||
|
||||
|
||||
class ConnectorImportError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectorImportUnsupported(ConnectorImportError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorDownloadedFile:
|
||||
filename: str
|
||||
data: bytes
|
||||
content_type: str | None = None
|
||||
revision: str | None = None
|
||||
external_id: str | None = None
|
||||
external_url: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
def read_connector_file(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
library_id: str,
|
||||
path: str,
|
||||
max_bytes: int,
|
||||
) -> ConnectorDownloadedFile:
|
||||
max_bytes = min(max_bytes, response_limit("file"))
|
||||
if profile.provider == "seafile":
|
||||
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||
return _read_seafile_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||
if profile.provider in {"webdav", "nextcloud"}:
|
||||
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||
if profile.provider == "smb":
|
||||
return _read_smb_file(profile, path=path, max_bytes=max_bytes)
|
||||
if profile.provider == "s3":
|
||||
return _read_s3_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||
raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet")
|
||||
|
||||
|
||||
def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
repo_id = _clean(library_id)
|
||||
if not repo_id:
|
||||
raise ConnectorImportError("Seafile import requires library_id")
|
||||
file_path = "/" + normalize_connector_browse_path(path)
|
||||
if file_path == "/":
|
||||
raise ConnectorImportError("Seafile import requires a file path")
|
||||
try:
|
||||
headers = _seafile_headers(profile)
|
||||
detail = _request_json(
|
||||
"GET",
|
||||
_seafile_url(profile, f"api2/repos/{repo_id}/file/detail/"),
|
||||
headers=headers,
|
||||
params={"p": file_path},
|
||||
)
|
||||
if isinstance(detail, dict):
|
||||
size = _int(detail.get("size"))
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||
download_url = _request_json(
|
||||
"GET",
|
||||
_seafile_url(profile, f"api2/repos/{repo_id}/file/"),
|
||||
headers=headers,
|
||||
params={"p": file_path, "reuse": "1"},
|
||||
)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
if not isinstance(download_url, str) or not download_url.strip():
|
||||
raise ConnectorImportError("Seafile did not return a file download URL")
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
"GET",
|
||||
download_url,
|
||||
timeout=30.0,
|
||||
kind="file",
|
||||
max_bytes=max_bytes,
|
||||
label="Seafile file download",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc
|
||||
if response.status_code != 200:
|
||||
raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}")
|
||||
data = response.content
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||
detail = detail if isinstance(detail, dict) else {}
|
||||
filename = filename_from_path(str(detail.get("name") or path))
|
||||
content_type = response.headers.get("content-type") or mimetypes.guess_type(filename)[0]
|
||||
external_id = f"{repo_id}:{normalize_connector_browse_path(path)}"
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=_clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified")),
|
||||
external_id=external_id,
|
||||
external_url=download_url,
|
||||
metadata={
|
||||
"library_id": repo_id,
|
||||
"library_path": normalize_connector_browse_path(path),
|
||||
"size": len(data),
|
||||
**({key: detail[key] for key in ("permission", "last_modified", "mtime", "uploader_email", "last_modifier_email") if key in detail}),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
root_url = _metadata_string(profile, "webdav_endpoint_url") or profile.endpoint_url
|
||||
if not root_url:
|
||||
raise ConnectorImportError("WebDAV connector profile does not define endpoint_url")
|
||||
file_path = normalize_connector_browse_path(path)
|
||||
if not file_path:
|
||||
raise ConnectorImportError("WebDAV import requires a file path")
|
||||
url = _webdav_url(root_url, file_path).rstrip("/")
|
||||
headers: dict[str, str] = {}
|
||||
auth: tuple[str, str] | None = None
|
||||
password = _profile_password(profile)
|
||||
token = _profile_token(profile)
|
||||
if profile.username and password:
|
||||
auth = (profile.username, password)
|
||||
elif token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import")
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
"GET",
|
||||
url,
|
||||
headers=headers,
|
||||
auth=auth,
|
||||
timeout=30.0,
|
||||
kind="file",
|
||||
max_bytes=max_bytes,
|
||||
label="WebDAV file download",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorImportError("Connector credentials were rejected")
|
||||
if response.status_code != 200:
|
||||
raise ConnectorImportError(f"WebDAV file download failed with HTTP {response.status_code}")
|
||||
length = _int(response.headers.get("content-length"))
|
||||
if length is not None and length > max_bytes:
|
||||
raise ConnectorImportError(f"WebDAV file exceeds limit of {max_bytes} bytes")
|
||||
data = response.content
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"WebDAV file exceeds limit of {max_bytes} bytes")
|
||||
filename = filename_from_path(file_path)
|
||||
revision = _clean(response.headers.get("etag") or response.headers.get("last-modified"))
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=response.headers.get("content-type") or mimetypes.guess_type(filename)[0],
|
||||
revision=revision,
|
||||
external_id=file_path,
|
||||
external_url=url,
|
||||
metadata={"path": file_path, "size": len(data)},
|
||||
)
|
||||
|
||||
|
||||
def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
file_path = normalize_connector_browse_path(path)
|
||||
if not file_path:
|
||||
raise ConnectorImportError("SMB import requires a file path")
|
||||
try:
|
||||
location = _smb_location(profile)
|
||||
unc_path = _smb_unc_path(location, file_path)
|
||||
smbclient = _smbclient_module()
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
size = _smb_stat_size(stat_result)
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
with smbclient.open_file(unc_path, mode="rb", **kwargs) as handle:
|
||||
data = handle.read(max_bytes + 1)
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
except ConnectorImportError:
|
||||
raise
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"SMB file download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
filename = filename_from_path(file_path)
|
||||
revision = _smb_stat_revision(stat_result)
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=mimetypes.guess_type(filename)[0],
|
||||
revision=revision,
|
||||
external_id=f"{location.share}:{file_path}",
|
||||
external_url=f"smb://{location.server}:{location.port}/{location.share}/{file_path}",
|
||||
metadata={
|
||||
"share": location.share,
|
||||
"path": file_path,
|
||||
"size": len(data),
|
||||
"modified_at": _smb_stat_modified_at(stat_result),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _s3_import_client(profile: ConnectorProfile) -> Any:
|
||||
try:
|
||||
return _s3_client(profile)
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
|
||||
|
||||
def _s3_object_detail(client: Any, *, bucket: str, key: str, max_bytes: int) -> dict[str, Any]:
|
||||
try:
|
||||
detail = client.head_object(Bucket=bucket, Key=key)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"S3 object metadata lookup failed: {exc}") from exc
|
||||
if not isinstance(detail, dict):
|
||||
raise ConnectorImportError("S3 connector returned invalid object metadata")
|
||||
size = _int(detail.get("ContentLength"))
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
return detail
|
||||
|
||||
|
||||
def _download_s3_object(
|
||||
client: Any,
|
||||
*,
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
max_bytes: int,
|
||||
) -> tuple[Any, bytes]:
|
||||
request: dict[str, object] = {"Bucket": bucket, "Key": key}
|
||||
if version_id:
|
||||
request["VersionId"] = version_id
|
||||
try:
|
||||
response = client.get_object(**request)
|
||||
body = response.get("Body")
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
return response, data
|
||||
|
||||
|
||||
def _s3_download_metadata(
|
||||
detail: dict[str, Any],
|
||||
*,
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
etag: str | None,
|
||||
size: int,
|
||||
) -> dict[str, Any]:
|
||||
metadata: dict[str, Any] = {
|
||||
"bucket": bucket,
|
||||
"key": key,
|
||||
"version_id": version_id,
|
||||
"etag": etag,
|
||||
"size": size,
|
||||
}
|
||||
for source_key, target_key in (
|
||||
("ChecksumSHA256", "checksum_sha256"),
|
||||
("ChecksumCRC32", "checksum_crc32"),
|
||||
("StorageClass", "storage_class"),
|
||||
):
|
||||
if source_key in detail:
|
||||
metadata[target_key] = detail[source_key]
|
||||
return metadata
|
||||
|
||||
|
||||
def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
if not bucket:
|
||||
raise ConnectorImportError("S3 import requires a bucket in library_id or profile metadata")
|
||||
key = _s3_object_key(profile, path)
|
||||
if not key:
|
||||
raise ConnectorImportError("S3 import requires an object key")
|
||||
client = _s3_import_client(profile)
|
||||
try:
|
||||
detail = _s3_object_detail(client, bucket=bucket, key=key, max_bytes=max_bytes)
|
||||
version_id = _clean(detail.get("VersionId"))
|
||||
response, data = _download_s3_object(
|
||||
client,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
|
||||
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
|
||||
filename = filename_from_path(key)
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=version_id or etag or _clean(detail.get("LastModified")),
|
||||
external_id=f"{bucket}:{key}",
|
||||
external_url=f"s3://{bucket}/{key}",
|
||||
metadata=_s3_download_metadata(
|
||||
detail,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
etag=etag,
|
||||
size=len(data),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _int(value: object) -> int | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
@@ -0,0 +1,310 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from fnmatch import fnmatchcase
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.core.policy import PolicyDecision, PolicySourceStep, normalize_policy_scope_type
|
||||
|
||||
|
||||
_ALLOW_KEYS = ("allow", "allowlist", "whitelist")
|
||||
_DENY_KEYS = ("deny", "denylist", "blacklist")
|
||||
|
||||
_FIELD_ALIASES = {
|
||||
"connectors": ("connectors", "connector_ids", "connector_id"),
|
||||
"credentials": ("credentials", "credential_ids", "credential_id"),
|
||||
"providers": ("providers", "provider"),
|
||||
"external_ids": ("external_ids", "external_id"),
|
||||
"external_paths": ("external_paths", "paths", "path_prefixes", "external_path"),
|
||||
"external_urls": ("external_urls", "urls", "external_url"),
|
||||
}
|
||||
|
||||
CONNECTOR_POLICY_FIELDS = tuple(_FIELD_ALIASES)
|
||||
|
||||
|
||||
class ConnectorPolicyDenied(RuntimeError):
|
||||
def __init__(self, decision: PolicyDecision) -> None:
|
||||
super().__init__(decision.reason or "Connector policy denied")
|
||||
self.decision = decision
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorAccessRequest:
|
||||
connector_id: str | None = None
|
||||
credential_id: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
external_path: str | None = None
|
||||
external_url: str | None = None
|
||||
operation: str = "access"
|
||||
|
||||
@classmethod
|
||||
def from_provenance(cls, value: Mapping[str, Any] | None, *, operation: str = "access") -> "ConnectorAccessRequest":
|
||||
payload = value or {}
|
||||
return cls(
|
||||
connector_id=_clean(payload.get("connector_id")),
|
||||
credential_id=_clean(payload.get("credential_id") or payload.get("connector_credential_id")),
|
||||
provider=_clean(payload.get("provider") or payload.get("source_type")),
|
||||
external_id=_clean(payload.get("external_id")),
|
||||
external_path=_clean(payload.get("external_path")),
|
||||
external_url=_clean(payload.get("external_url")),
|
||||
operation=_clean(operation) or "access",
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict[str, str | None]:
|
||||
return {
|
||||
"connector_id": self.connector_id,
|
||||
"credential_id": self.credential_id,
|
||||
"provider": self.provider,
|
||||
"external_id": self.external_id,
|
||||
"external_path": self.external_path,
|
||||
"external_url": self.external_url,
|
||||
"operation": self.operation,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorPolicySource:
|
||||
scope_type: str
|
||||
label: str
|
||||
scope_id: str | None = None
|
||||
policy: Mapping[str, Any] | None = None
|
||||
|
||||
def source_step(self, applied_fields: Iterable[str]) -> PolicySourceStep:
|
||||
return PolicySourceStep(
|
||||
scope_type=normalize_policy_scope_type(self.scope_type),
|
||||
scope_id=self.scope_id,
|
||||
label=self.label,
|
||||
applied_fields=tuple(dict.fromkeys(applied_fields)),
|
||||
policy=dict(self.policy or {}),
|
||||
)
|
||||
|
||||
|
||||
def connector_policy_sources_from_payload(value: object) -> list[ConnectorPolicySource]:
|
||||
if value is None or value == "":
|
||||
return []
|
||||
if isinstance(value, Mapping):
|
||||
raw_sources = value.get("sources")
|
||||
if raw_sources is None:
|
||||
raw_sources = [value]
|
||||
elif isinstance(value, list):
|
||||
raw_sources = value
|
||||
else:
|
||||
raise ValueError("connector_policy must be a JSON object or list")
|
||||
if not isinstance(raw_sources, list):
|
||||
raise ValueError("connector_policy.sources must be a list")
|
||||
return [_source_from_mapping(item) for item in raw_sources if isinstance(item, Mapping)]
|
||||
|
||||
|
||||
def connector_policy_applied_fields(policy: Mapping[str, Any] | None) -> tuple[str, ...]:
|
||||
return _applied_fields(_policy_mapping(policy))
|
||||
|
||||
|
||||
def filtered_connector_policy_source(source: ConnectorPolicySource, fields: Iterable[str]) -> ConnectorPolicySource:
|
||||
allowed_fields = {field for field in fields if field in _FIELD_ALIASES}
|
||||
if not allowed_fields:
|
||||
return ConnectorPolicySource(scope_type=source.scope_type, scope_id=source.scope_id, label=source.label, policy={})
|
||||
policy = _policy_mapping(source.policy)
|
||||
filtered: dict[str, Any] = {}
|
||||
for prefix, keys in (("allow", _ALLOW_KEYS), ("deny", _DENY_KEYS)):
|
||||
rules = _merged_rule(policy, keys)
|
||||
selected = {field: _string_list(rules.get(field)) for field in allowed_fields}
|
||||
selected = {field: values for field, values in selected.items() if values}
|
||||
if selected:
|
||||
filtered[prefix] = selected
|
||||
return ConnectorPolicySource(scope_type=source.scope_type, scope_id=source.scope_id, label=source.label, policy=filtered)
|
||||
|
||||
|
||||
def connector_policy_sources_for_fields(
|
||||
sources: Iterable[ConnectorPolicySource],
|
||||
fields: Iterable[str],
|
||||
) -> list[ConnectorPolicySource]:
|
||||
return [filtered_connector_policy_source(source, fields) for source in sources]
|
||||
|
||||
|
||||
def connector_policy_decision(
|
||||
request: ConnectorAccessRequest,
|
||||
sources: Iterable[ConnectorPolicySource],
|
||||
) -> PolicyDecision:
|
||||
source_list = list(sources)
|
||||
source_steps: list[PolicySourceStep] = []
|
||||
deny_matches: list[dict[str, Any]] = []
|
||||
allow_misses: list[dict[str, Any]] = []
|
||||
|
||||
for source in source_list:
|
||||
policy = _policy_mapping(source.policy)
|
||||
applied_fields = _applied_fields(policy)
|
||||
if applied_fields:
|
||||
source_steps.append(source.source_step(applied_fields))
|
||||
|
||||
deny_policy = _merged_rule(policy, _DENY_KEYS)
|
||||
for field, patterns in _rules_by_field(deny_policy).items():
|
||||
if _matches_field(request, field, patterns):
|
||||
deny_matches.append({
|
||||
"scope": source.source_step((f"deny.{field}",)).to_dict(),
|
||||
"field": field,
|
||||
"patterns": patterns,
|
||||
})
|
||||
|
||||
allow_policy = _merged_rule(policy, _ALLOW_KEYS)
|
||||
for field, patterns in _rules_by_field(allow_policy).items():
|
||||
if not _matches_field(request, field, patterns):
|
||||
allow_misses.append({
|
||||
"scope": source.source_step((f"allow.{field}",)).to_dict(),
|
||||
"field": field,
|
||||
"patterns": patterns,
|
||||
})
|
||||
|
||||
details = {
|
||||
"request": request.to_dict(),
|
||||
"deny_matches": deny_matches,
|
||||
"allow_misses": allow_misses,
|
||||
}
|
||||
if deny_matches:
|
||||
return PolicyDecision(
|
||||
allowed=False,
|
||||
reason="Connector access is denied by a blacklist rule.",
|
||||
source_path=tuple(source_steps),
|
||||
requirements=("connector_policy_denylist",),
|
||||
details=details,
|
||||
)
|
||||
if allow_misses:
|
||||
return PolicyDecision(
|
||||
allowed=False,
|
||||
reason="Connector access is not allowed by the configured whitelist.",
|
||||
source_path=tuple(source_steps),
|
||||
requirements=("connector_policy_allowlist",),
|
||||
details=details,
|
||||
)
|
||||
return PolicyDecision(
|
||||
allowed=True,
|
||||
reason=None,
|
||||
source_path=tuple(source_steps),
|
||||
requirements=(),
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
def ensure_connector_policy_allows(
|
||||
request: ConnectorAccessRequest,
|
||||
sources: Iterable[ConnectorPolicySource],
|
||||
) -> PolicyDecision:
|
||||
decision = connector_policy_decision(request, sources)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
return decision
|
||||
|
||||
|
||||
def _source_from_mapping(value: Mapping[str, Any]) -> ConnectorPolicySource:
|
||||
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||
scope_id = _clean(value.get("scope_id"))
|
||||
if scope_type == "system":
|
||||
scope_id = None
|
||||
elif not scope_id:
|
||||
raise ValueError(f"{scope_type} connector policy sources require scope_id")
|
||||
label = _clean(value.get("label")) or scope_type.capitalize()
|
||||
policy = value.get("policy")
|
||||
if policy is None:
|
||||
policy = {key: value[key] for key in (*_ALLOW_KEYS, *_DENY_KEYS) if key in value}
|
||||
if policy is not None and not isinstance(policy, Mapping):
|
||||
raise ValueError("connector policy source policy must be a JSON object")
|
||||
return ConnectorPolicySource(scope_type=scope_type, scope_id=scope_id, label=label, policy=policy or {})
|
||||
|
||||
|
||||
def _policy_mapping(value: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||
return value if isinstance(value, Mapping) else {}
|
||||
|
||||
|
||||
def _merged_rule(policy: Mapping[str, Any], keys: Iterable[str]) -> dict[str, Any]:
|
||||
merged: dict[str, Any] = {}
|
||||
for key in keys:
|
||||
raw = policy.get(key)
|
||||
if isinstance(raw, Mapping):
|
||||
merged.update(raw)
|
||||
return merged
|
||||
|
||||
|
||||
def _rules_by_field(value: Mapping[str, Any]) -> dict[str, list[str]]:
|
||||
result: dict[str, list[str]] = {}
|
||||
for field, aliases in _FIELD_ALIASES.items():
|
||||
items: list[str] = []
|
||||
for alias in aliases:
|
||||
items.extend(_string_list(value.get(alias)))
|
||||
if items:
|
||||
result[field] = list(dict.fromkeys(items))
|
||||
return result
|
||||
|
||||
|
||||
def _applied_fields(policy: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
fields: list[str] = []
|
||||
for prefix, keys in (("allow", _ALLOW_KEYS), ("deny", _DENY_KEYS)):
|
||||
rules = _merged_rule(policy, keys)
|
||||
fields.extend(f"{prefix}.{field}" for field in _rules_by_field(rules))
|
||||
return tuple(dict.fromkeys(fields))
|
||||
|
||||
|
||||
def _matches_field(request: ConnectorAccessRequest, field: str, patterns: list[str]) -> bool:
|
||||
if field == "connectors":
|
||||
return _matches_reference(request.connector_id, patterns)
|
||||
if field == "credentials":
|
||||
return _matches_reference(request.credential_id, patterns)
|
||||
if field == "providers":
|
||||
return _matches_reference(request.provider, patterns)
|
||||
if field == "external_ids":
|
||||
return _matches_glob(request.external_id, patterns)
|
||||
if field == "external_paths":
|
||||
return _matches_path(request.external_path, patterns)
|
||||
if field == "external_urls":
|
||||
return _matches_glob(request.external_url, patterns)
|
||||
return False
|
||||
|
||||
|
||||
def _matches_reference(value: str | None, patterns: list[str]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
clean = value.casefold()
|
||||
return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
|
||||
|
||||
|
||||
def _matches_glob(value: str | None, patterns: list[str]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
clean = value.casefold()
|
||||
return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
|
||||
|
||||
|
||||
def _matches_path(value: str | None, patterns: list[str]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
clean = value.replace("\\", "/").strip()
|
||||
for pattern in patterns:
|
||||
normalized = pattern.replace("\\", "/").strip()
|
||||
if normalized == "*":
|
||||
return True
|
||||
if any(token in normalized for token in "*?[]"):
|
||||
if fnmatchcase(clean.casefold(), normalized.casefold()):
|
||||
return True
|
||||
continue
|
||||
if clean.casefold() == normalized.casefold() or clean.casefold().startswith(normalized.rstrip("/").casefold() + "/"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
values = [value]
|
||||
elif isinstance(value, Iterable) and not isinstance(value, (bytes, bytearray, Mapping)):
|
||||
values = [str(item) for item in value]
|
||||
else:
|
||||
values = [str(value)]
|
||||
return [item.strip() for item in values if item.strip()]
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,334 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type
|
||||
from govoplan_files.backend.db.models import FileConnectorPolicy
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
CONNECTOR_POLICY_FIELDS,
|
||||
ConnectorPolicySource,
|
||||
connector_policy_applied_fields,
|
||||
)
|
||||
|
||||
_RULE_GROUPS = ("allow", "deny")
|
||||
_FIELD_ALIASES = {
|
||||
"connectors": ("connectors", "connector_ids", "connector_id"),
|
||||
"credentials": ("credentials", "credential_ids", "credential_id"),
|
||||
"providers": ("providers", "provider"),
|
||||
"external_ids": ("external_ids", "external_id"),
|
||||
"external_paths": ("external_paths", "paths", "path_prefixes", "external_path"),
|
||||
"external_urls": ("external_urls", "urls", "external_url"),
|
||||
}
|
||||
|
||||
|
||||
def get_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
return _normalize_connector_policy(row.policy if row is not None else None)
|
||||
|
||||
|
||||
def set_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str | None,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
normalized = _normalize_connector_policy(policy)
|
||||
parent = parent_connector_policy(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
if parent is not None:
|
||||
_validate_lower_level_limits(parent, normalized)
|
||||
row_tenant_id, row_scope_type, row_scope_id = _policy_scope_key(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=row_scope_type, scope_id=row_scope_id)
|
||||
if row is None:
|
||||
row = FileConnectorPolicy(
|
||||
tenant_id=row_tenant_id,
|
||||
scope_type=row_scope_type,
|
||||
scope_id=row_scope_id,
|
||||
policy=normalized,
|
||||
created_by_user_id=user_id,
|
||||
updated_by_user_id=user_id,
|
||||
)
|
||||
else:
|
||||
row.policy = normalized
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return normalized
|
||||
|
||||
|
||||
def connector_policy_response(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
policy = get_connector_policy(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
effective_sources = effective_connector_policy_sources(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
parent_policy = parent_connector_policy(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
parent_sources = parent_connector_policy_sources(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
return {
|
||||
"scope_type": clean_scope_type,
|
||||
"scope_id": clean_scope_id,
|
||||
"policy": policy,
|
||||
"effective_policy": merge_connector_policies(source.policy for source in effective_sources),
|
||||
"parent_policy": parent_policy,
|
||||
"effective_policy_sources": [_source_step(source) for source in effective_sources],
|
||||
"parent_policy_sources": [_source_step(source) for source in parent_sources],
|
||||
}
|
||||
|
||||
|
||||
def effective_connector_policy_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> list[ConnectorPolicySource]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
sources: list[ConnectorPolicySource] = []
|
||||
for source_scope_type, source_scope_id in _policy_source_chain(tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id):
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=source_scope_type, scope_id=source_scope_id)
|
||||
if row is None:
|
||||
continue
|
||||
policy = _normalize_connector_policy(row.policy)
|
||||
if _policy_is_empty(policy):
|
||||
continue
|
||||
sources.append(
|
||||
ConnectorPolicySource(
|
||||
scope_type=source_scope_type,
|
||||
scope_id=source_scope_id,
|
||||
label=_policy_source_label(source_scope_type),
|
||||
policy=policy,
|
||||
)
|
||||
)
|
||||
return sources
|
||||
|
||||
|
||||
def parent_connector_policy_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> list[ConnectorPolicySource]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
chain = _policy_source_chain(tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
parent_chain = chain[:-1] if chain else []
|
||||
sources: list[ConnectorPolicySource] = []
|
||||
for source_scope_type, source_scope_id in parent_chain:
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=source_scope_type, scope_id=source_scope_id)
|
||||
if row is None:
|
||||
continue
|
||||
policy = _normalize_connector_policy(row.policy)
|
||||
if _policy_is_empty(policy):
|
||||
continue
|
||||
sources.append(
|
||||
ConnectorPolicySource(
|
||||
scope_type=source_scope_type,
|
||||
scope_id=source_scope_id,
|
||||
label=_policy_source_label(source_scope_type),
|
||||
policy=policy,
|
||||
)
|
||||
)
|
||||
return sources
|
||||
|
||||
|
||||
def parent_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
sources = parent_connector_policy_sources(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
if not sources:
|
||||
return None
|
||||
return merge_connector_policies(source.policy for source in sources)
|
||||
|
||||
|
||||
def validate_connector_policy_allowed_by_parent(parent_policy: Mapping[str, Any] | None, local_policy: Mapping[str, Any] | None) -> None:
|
||||
if parent_policy is None:
|
||||
return
|
||||
_validate_lower_level_limits(_normalize_connector_policy(parent_policy), _normalize_connector_policy(local_policy))
|
||||
|
||||
|
||||
def merge_connector_policies(policies: Iterable[Mapping[str, Any] | None]) -> dict[str, Any]:
|
||||
result = _empty_connector_policy()
|
||||
for policy in policies:
|
||||
normalized = _normalize_connector_policy(policy)
|
||||
for group in _RULE_GROUPS:
|
||||
rules = normalized.get(group) or {}
|
||||
if not isinstance(rules, Mapping):
|
||||
continue
|
||||
target = result[group]
|
||||
for field in CONNECTOR_POLICY_FIELDS:
|
||||
values = _string_list(rules.get(field))
|
||||
if values:
|
||||
target[field] = _unique([*target.get(field, []), *values])
|
||||
lower = normalized.get("allow_lower_level_limits") or {}
|
||||
if isinstance(lower, Mapping):
|
||||
for key, value in lower.items():
|
||||
if isinstance(value, bool):
|
||||
result["allow_lower_level_limits"][str(key)] = value
|
||||
return _compact_connector_policy(result, keep_lower=True)
|
||||
|
||||
|
||||
def _connector_policy_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> FileConnectorPolicy | None:
|
||||
row_tenant_id, row_scope_type, row_scope_id = _policy_scope_key(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
query = session.query(FileConnectorPolicy).filter(FileConnectorPolicy.scope_type == row_scope_type)
|
||||
query = query.filter(FileConnectorPolicy.tenant_id.is_(None) if row_tenant_id is None else FileConnectorPolicy.tenant_id == row_tenant_id)
|
||||
query = query.filter(FileConnectorPolicy.scope_id.is_(None) if row_scope_id is None else FileConnectorPolicy.scope_id == row_scope_id)
|
||||
return query.order_by(FileConnectorPolicy.created_at.asc()).first()
|
||||
|
||||
|
||||
def _policy_scope_key(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str | None, str, str | None]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return None, "system", None
|
||||
return tenant_id, clean_scope_type, clean_scope_id
|
||||
|
||||
|
||||
def _policy_scope_ref(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return "system", None
|
||||
if clean_scope_type == "tenant":
|
||||
if clean_scope_id and clean_scope_id != tenant_id:
|
||||
raise FileStorageError("Tenant connector policy scope does not belong to the active tenant")
|
||||
return "tenant", tenant_id
|
||||
if clean_scope_type in {"user", "group", "campaign"}:
|
||||
if not clean_scope_id:
|
||||
raise FileStorageError(f"{clean_scope_type.capitalize()} connector policy requires scope_id")
|
||||
return clean_scope_type, clean_scope_id
|
||||
raise FileStorageError("Unsupported connector policy scope")
|
||||
|
||||
|
||||
def _policy_source_chain(*, tenant_id: str, scope_type: str, scope_id: str | None) -> list[tuple[str, str | None]]:
|
||||
chain: list[tuple[str, str | None]] = [("system", None)]
|
||||
if scope_type == "system":
|
||||
return chain
|
||||
chain.append(("tenant", tenant_id))
|
||||
if scope_type == "tenant":
|
||||
return chain
|
||||
chain.append((scope_type, scope_id))
|
||||
return chain
|
||||
|
||||
|
||||
def _normalize_connector_policy(policy: Mapping[str, Any] | None) -> dict[str, Any]:
|
||||
if policy is None:
|
||||
return {}
|
||||
if not isinstance(policy, Mapping):
|
||||
raise FileStorageError("Connector policy must be a JSON object")
|
||||
normalized = _empty_connector_policy()
|
||||
for group in _RULE_GROUPS:
|
||||
raw = policy.get(group)
|
||||
if raw is None:
|
||||
raw = policy.get(f"{group}list")
|
||||
if raw is None:
|
||||
raw = policy.get("whitelist" if group == "allow" else "blacklist")
|
||||
if raw is not None and not isinstance(raw, Mapping):
|
||||
raise FileStorageError(f"Connector policy {group} rules must be an object")
|
||||
for field, aliases in _FIELD_ALIASES.items():
|
||||
values: list[str] = []
|
||||
if isinstance(raw, Mapping):
|
||||
for alias in aliases:
|
||||
values.extend(_string_list(raw.get(alias)))
|
||||
if values:
|
||||
normalized[group][field] = _unique(values)
|
||||
lower = policy.get("allow_lower_level_limits")
|
||||
if lower is not None:
|
||||
if not isinstance(lower, Mapping):
|
||||
raise FileStorageError("Connector policy allow_lower_level_limits must be an object")
|
||||
for key, value in lower.items():
|
||||
clean_key = str(key).strip()
|
||||
if clean_key and isinstance(value, bool):
|
||||
normalized["allow_lower_level_limits"][clean_key] = value
|
||||
return _compact_connector_policy(normalized, keep_lower=True)
|
||||
|
||||
|
||||
def _empty_connector_policy() -> dict[str, Any]:
|
||||
return {"allow": {}, "deny": {}, "allow_lower_level_limits": {}}
|
||||
|
||||
|
||||
def _compact_connector_policy(policy: dict[str, Any], *, keep_lower: bool = False) -> dict[str, Any]:
|
||||
compact: dict[str, Any] = {}
|
||||
for group in _RULE_GROUPS:
|
||||
rules = {field: _unique(_string_list(values)) for field, values in (policy.get(group) or {}).items()}
|
||||
rules = {field: values for field, values in rules.items() if values}
|
||||
if rules:
|
||||
compact[group] = rules
|
||||
lower = policy.get("allow_lower_level_limits") or {}
|
||||
if isinstance(lower, Mapping):
|
||||
lower_compact = {str(key): bool(value) for key, value in lower.items() if isinstance(value, bool)}
|
||||
if lower_compact or keep_lower:
|
||||
compact["allow_lower_level_limits"] = lower_compact
|
||||
return compact
|
||||
|
||||
|
||||
def _policy_is_empty(policy: Mapping[str, Any]) -> bool:
|
||||
return not connector_policy_applied_fields(policy) and not bool(policy.get("allow_lower_level_limits"))
|
||||
|
||||
|
||||
def _source_step(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||
return source.source_step(connector_policy_applied_fields(source.policy)).to_dict()
|
||||
|
||||
|
||||
def _policy_source_label(scope_type: str) -> str:
|
||||
if scope_type == "system":
|
||||
return "System connector policy"
|
||||
if scope_type == "tenant":
|
||||
return "Tenant connector policy"
|
||||
return f"{scope_type.capitalize()} connector policy"
|
||||
|
||||
|
||||
def _validate_lower_level_limits(parent_policy: Mapping[str, Any], local_policy: Mapping[str, Any]) -> None:
|
||||
limits = parent_policy.get("allow_lower_level_limits")
|
||||
if not isinstance(limits, Mapping):
|
||||
return
|
||||
for field in connector_policy_applied_fields(local_policy):
|
||||
allowed = limits.get(field)
|
||||
if allowed is False:
|
||||
raise FileStorageError(f"Parent connector policy does not allow lower-level {field} limits")
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
clean = value.strip()
|
||||
return [clean] if clean else []
|
||||
if isinstance(value, Iterable) and not isinstance(value, (str, bytes, bytearray, Mapping)):
|
||||
return [str(item).strip() for item in value if str(item).strip()]
|
||||
return [str(value).strip()] if str(value).strip() else []
|
||||
|
||||
|
||||
def _unique(values: Iterable[str]) -> list[str]:
|
||||
return list(dict.fromkeys(value for value in values if value))
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,470 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type
|
||||
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
ConnectorCredential,
|
||||
connector_credential_from_row,
|
||||
credential_rows_by_id,
|
||||
resolve_reusable_connector_credential,
|
||||
reusable_credential_id,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
|
||||
|
||||
|
||||
def list_database_connector_profiles(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||
) -> list[ConnectorProfile]:
|
||||
profiles, _profile_ids = select_database_connector_profiles(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
row_visible=row_visible,
|
||||
)
|
||||
return profiles
|
||||
|
||||
|
||||
def select_database_connector_profiles(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||
) -> tuple[list[ConnectorProfile], set[str]]:
|
||||
"""Return visible profiles and every database id that shadows settings."""
|
||||
|
||||
query = session.query(FileConnectorProfile).filter(
|
||||
(FileConnectorProfile.scope_type == "system")
|
||||
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||
)
|
||||
if not include_disabled:
|
||||
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||
profile_ids = {row.id for row in rows}
|
||||
if row_visible is not None:
|
||||
rows = [row for row in rows if row_visible(row)]
|
||||
credential_ids = {
|
||||
_clean(row.credential_profile_id)
|
||||
for row in rows
|
||||
if _clean(row.credential_profile_id) and not reusable_credential_id(row.credential_profile_id)
|
||||
}
|
||||
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
|
||||
return (
|
||||
[
|
||||
connector_profile_from_row(
|
||||
row,
|
||||
credential_row=(
|
||||
_resolve_profile_reusable_credential(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
row=row,
|
||||
)
|
||||
if reusable_credential_id(row.credential_profile_id)
|
||||
else credentials.get(row.credential_profile_id or "")
|
||||
),
|
||||
)
|
||||
for row in rows
|
||||
],
|
||||
profile_ids,
|
||||
)
|
||||
|
||||
|
||||
def list_connector_profile_rows(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[FileConnectorProfile]:
|
||||
query = session.query(FileConnectorProfile).filter(
|
||||
(FileConnectorProfile.scope_type == "system")
|
||||
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||
)
|
||||
if not include_disabled:
|
||||
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||
return query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||
|
||||
|
||||
def connector_profile_from_row(
|
||||
row: FileConnectorProfile,
|
||||
credential_row: FileConnectorCredential | ConnectorCredential | None = None,
|
||||
) -> ConnectorProfile:
|
||||
policy_sources = []
|
||||
if row.policy:
|
||||
policy_sources = connector_policy_sources_from_payload({
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"label": row.label,
|
||||
"policy": row.policy,
|
||||
})
|
||||
credential = (
|
||||
credential_row
|
||||
if isinstance(credential_row, ConnectorCredential)
|
||||
else connector_credential_from_row(credential_row)
|
||||
if credential_row is not None
|
||||
else None
|
||||
)
|
||||
if credential:
|
||||
policy_sources.extend(credential.policy_sources)
|
||||
metadata = dict(row.metadata_ or {})
|
||||
if reusable_credential_id(row.credential_profile_id) and credential is None:
|
||||
metadata["credential_unavailable"] = True
|
||||
return ConnectorProfile(
|
||||
id=row.id,
|
||||
label=row.label,
|
||||
provider=row.provider,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
endpoint_url=row.endpoint_url,
|
||||
base_path=row.base_path,
|
||||
enabled=row.enabled,
|
||||
credential_profile_id=row.credential_profile_id,
|
||||
credential_profile_label=credential.label if credential else None,
|
||||
credential_mode=credential.credential_mode if credential else row.credential_mode,
|
||||
username=credential.username if credential else row.username,
|
||||
password_env=credential.password_env if credential else row.password_env,
|
||||
token_env=credential.token_env if credential else row.token_env,
|
||||
secret_ref=credential.secret_ref if credential else row.secret_ref,
|
||||
password_value=credential.password_value if credential else decrypt_secret(row.password_encrypted),
|
||||
token_value=credential.token_value if credential else decrypt_secret(row.token_encrypted),
|
||||
capabilities=tuple(_string_list(row.capabilities)),
|
||||
policy_sources=tuple(policy_sources),
|
||||
metadata=metadata,
|
||||
source_kind="database",
|
||||
)
|
||||
|
||||
|
||||
def _resolve_profile_reusable_credential(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
row: FileConnectorProfile,
|
||||
) -> ConnectorCredential | None:
|
||||
try:
|
||||
return resolve_reusable_connector_credential(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
credential_ref=row.credential_profile_id or "",
|
||||
profile_id=row.id,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
)
|
||||
except FileStorageError:
|
||||
return None
|
||||
|
||||
|
||||
def get_connector_profile_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
profile_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> FileConnectorProfile:
|
||||
row = session.get(FileConnectorProfile, profile_id)
|
||||
if row is None or (row.scope_type != "system" and row.tenant_id != tenant_id):
|
||||
raise FileStorageError("Connector profile not found")
|
||||
if not include_disabled and not row.enabled:
|
||||
raise FileStorageError("Connector profile not found")
|
||||
return row
|
||||
|
||||
|
||||
def create_connector_profile_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str | None,
|
||||
profile_id: str,
|
||||
label: str,
|
||||
provider: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
endpoint_url: str | None = None,
|
||||
base_path: str | None = None,
|
||||
enabled: bool = True,
|
||||
credential_profile_id: str | None = None,
|
||||
credential_mode: str = "none",
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
capabilities: list[str] | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> FileConnectorProfile:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
clean_id = _normalize_profile_id(profile_id)
|
||||
if session.get(FileConnectorProfile, clean_id) is not None:
|
||||
raise FileStorageError(f"Connector profile already exists: {clean_id}")
|
||||
clean_scope_type, clean_scope_id, row_tenant_id = _normalize_scope(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
row = FileConnectorProfile(
|
||||
id=clean_id,
|
||||
tenant_id=row_tenant_id,
|
||||
scope_type=clean_scope_type,
|
||||
scope_id=clean_scope_id,
|
||||
label=_normalize_label(label),
|
||||
provider=_normalize_provider(provider),
|
||||
endpoint_url=_clean(endpoint_url),
|
||||
base_path=_clean(base_path),
|
||||
enabled=bool(enabled),
|
||||
credential_profile_id=_clean(credential_profile_id),
|
||||
credential_mode=_normalize_credential_mode(credential_mode),
|
||||
username=_clean(username),
|
||||
password_encrypted=encrypt_secret(_clean(password)),
|
||||
token_encrypted=encrypt_secret(_clean(token)),
|
||||
password_env=_clean(password_env),
|
||||
token_env=_clean(token_env),
|
||||
secret_ref=_clean(secret_ref),
|
||||
capabilities=_string_list(capabilities),
|
||||
policy=dict(policy or {}),
|
||||
metadata_=dict(metadata or {}),
|
||||
created_by_user_id=user_id,
|
||||
updated_by_user_id=user_id,
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def update_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
user_id: str | None,
|
||||
label: str | None = None,
|
||||
provider: str | None = None,
|
||||
endpoint_url: str | None = None,
|
||||
base_path: str | None = None,
|
||||
enabled: bool | None = None,
|
||||
credential_profile_id: str | None = None,
|
||||
credential_mode: str | None = None,
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
capabilities: list[str] | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
clear_password: bool = False,
|
||||
clear_token: bool = False,
|
||||
) -> FileConnectorProfile:
|
||||
_validate_profile_update_references(
|
||||
row,
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
_update_profile_connection_fields(
|
||||
row,
|
||||
label=label,
|
||||
provider=provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=base_path,
|
||||
enabled=enabled,
|
||||
credential_profile_id=credential_profile_id,
|
||||
credential_mode=credential_mode,
|
||||
)
|
||||
_update_profile_credential_fields(
|
||||
row,
|
||||
username=username,
|
||||
password=password,
|
||||
token=token,
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
clear_password=clear_password,
|
||||
clear_token=clear_token,
|
||||
)
|
||||
_update_profile_governance_fields(
|
||||
row,
|
||||
capabilities=capabilities,
|
||||
policy=policy,
|
||||
metadata=metadata,
|
||||
)
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def _validate_profile_update_references(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
password_env: str | None,
|
||||
token_env: str | None,
|
||||
secret_ref: str | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> None:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
if secret_ref is None or _clean(secret_ref) == _clean(row.secret_ref):
|
||||
return
|
||||
if _clean(row.secret_ref):
|
||||
raise FileStorageError(
|
||||
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
|
||||
|
||||
def _update_profile_connection_fields(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
label: str | None,
|
||||
provider: str | None,
|
||||
endpoint_url: str | None,
|
||||
base_path: str | None,
|
||||
enabled: bool | None,
|
||||
credential_profile_id: str | None,
|
||||
credential_mode: str | None,
|
||||
) -> None:
|
||||
if label is not None:
|
||||
row.label = _normalize_label(label)
|
||||
if provider is not None:
|
||||
row.provider = _normalize_provider(provider)
|
||||
if endpoint_url is not None:
|
||||
row.endpoint_url = _clean(endpoint_url)
|
||||
if base_path is not None:
|
||||
row.base_path = _clean(base_path)
|
||||
if enabled is not None:
|
||||
row.enabled = bool(enabled)
|
||||
if credential_profile_id is not None:
|
||||
row.credential_profile_id = _clean(credential_profile_id)
|
||||
if credential_mode is not None:
|
||||
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||
|
||||
|
||||
def _update_profile_credential_fields(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
username: str | None,
|
||||
password: str | None,
|
||||
token: str | None,
|
||||
password_env: str | None,
|
||||
token_env: str | None,
|
||||
secret_ref: str | None,
|
||||
clear_password: bool,
|
||||
clear_token: bool,
|
||||
) -> None:
|
||||
if username is not None:
|
||||
row.username = _clean(username)
|
||||
if password is not None:
|
||||
row.password_encrypted = encrypt_secret(_clean(password))
|
||||
elif clear_password:
|
||||
row.password_encrypted = None
|
||||
if token is not None:
|
||||
row.token_encrypted = encrypt_secret(_clean(token))
|
||||
elif clear_token:
|
||||
row.token_encrypted = None
|
||||
if password_env is not None:
|
||||
row.password_env = _clean(password_env)
|
||||
if token_env is not None:
|
||||
row.token_env = _clean(token_env)
|
||||
if secret_ref is not None:
|
||||
row.secret_ref = _clean(secret_ref)
|
||||
|
||||
|
||||
def _update_profile_governance_fields(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
capabilities: list[str] | None,
|
||||
policy: Mapping[str, Any] | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> None:
|
||||
if capabilities is not None:
|
||||
row.capabilities = _string_list(capabilities)
|
||||
if policy is not None:
|
||||
row.policy = dict(policy)
|
||||
if metadata is not None:
|
||||
row.metadata_ = dict(metadata)
|
||||
|
||||
|
||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return "system", None, None
|
||||
if clean_scope_type == "tenant":
|
||||
return "tenant", tenant_id, tenant_id
|
||||
if clean_scope_type in {"user", "group", "campaign"}:
|
||||
if not clean_scope_id:
|
||||
raise FileStorageError(f"{clean_scope_type.capitalize()} connector profiles require scope_id")
|
||||
return clean_scope_type, clean_scope_id, tenant_id
|
||||
raise FileStorageError("Unsupported connector profile scope")
|
||||
|
||||
|
||||
def _normalize_profile_id(value: str) -> str:
|
||||
clean = _clean(value)
|
||||
if not clean:
|
||||
raise FileStorageError("Connector profile id is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector profile id is too long")
|
||||
if any(char.isspace() for char in clean):
|
||||
raise FileStorageError("Connector profile id cannot contain whitespace")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_label(value: str) -> str:
|
||||
clean = value.strip()
|
||||
if not clean:
|
||||
raise FileStorageError("Connector profile label is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector profile label is too long")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_provider(value: str) -> str:
|
||||
clean = value.strip().casefold()
|
||||
if clean not in supported_connector_providers():
|
||||
raise FileStorageError(f"Unsupported connector provider: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_credential_mode(value: str) -> str:
|
||||
clean = value.strip().casefold() or "none"
|
||||
if clean not in {"none", "anonymous", "basic", "token", "secret_ref"}:
|
||||
raise FileStorageError(f"Unsupported connector credential mode: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
values = value.split(",")
|
||||
elif isinstance(value, (list, tuple, set)):
|
||||
values = value
|
||||
else:
|
||||
values = [value]
|
||||
return [str(item).strip() for item in values if str(item).strip()]
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,343 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_secret_env_available,
|
||||
validate_deployment_connector_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||
|
||||
|
||||
_ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON")
|
||||
_ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE")
|
||||
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"}
|
||||
_INLINE_SECRET_FIELDS = (
|
||||
"password",
|
||||
"token",
|
||||
"api_key",
|
||||
"access_key",
|
||||
"access_key_id",
|
||||
"secret_key",
|
||||
"secret_access_key",
|
||||
"session_token",
|
||||
)
|
||||
|
||||
|
||||
def supported_connector_providers() -> set[str]:
|
||||
return set(_SUPPORTED_PROVIDERS)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorProfile:
|
||||
id: str
|
||||
label: str
|
||||
provider: str
|
||||
scope_type: str = "system"
|
||||
scope_id: str | None = None
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
credential_profile_id: str | None = None
|
||||
credential_profile_label: str | None = None
|
||||
credential_mode: str = "none"
|
||||
username: str | None = None
|
||||
password_env: str | None = None
|
||||
token_env: str | None = None
|
||||
secret_ref: str | None = None
|
||||
password_value: str | None = field(default=None, repr=False)
|
||||
token_value: str | None = field(default=None, repr=False)
|
||||
has_inline_secret: bool = False
|
||||
capabilities: tuple[str, ...] = ()
|
||||
policy_sources: tuple[ConnectorPolicySource, ...] = ()
|
||||
metadata: Mapping[str, Any] = field(default_factory=dict)
|
||||
source_kind: str = "settings"
|
||||
|
||||
@property
|
||||
def source_path(self) -> str:
|
||||
return policy_source_path(self.scope_type, self.scope_id)
|
||||
|
||||
@property
|
||||
def credential_source(self) -> str | None:
|
||||
if self.credential_profile_id:
|
||||
return "credential_profile"
|
||||
if self.secret_ref:
|
||||
return "secret_ref"
|
||||
if self.token_value or self.password_value:
|
||||
return "stored"
|
||||
if self.token_env:
|
||||
return "token_env"
|
||||
if self.password_env:
|
||||
return "password_env"
|
||||
if self.has_inline_secret:
|
||||
return "inline"
|
||||
return None
|
||||
|
||||
@property
|
||||
def credentials_configured(self) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
mode = self.credential_mode.casefold()
|
||||
if mode in {"", "none", "anonymous"}:
|
||||
return True
|
||||
if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value:
|
||||
return True
|
||||
if self.token_env:
|
||||
return connector_secret_env_available(self.token_env, source_kind=self.source_kind)
|
||||
if self.password_env:
|
||||
return connector_secret_env_available(self.password_env, source_kind=self.source_kind)
|
||||
return False
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
"id": self.id,
|
||||
"label": self.label,
|
||||
"provider": self.provider,
|
||||
"endpoint_url": self.endpoint_url,
|
||||
"base_path": self.base_path,
|
||||
"enabled": self.enabled,
|
||||
"scope_type": self.scope_type,
|
||||
"scope_id": self.scope_id,
|
||||
"source_path": self.source_path,
|
||||
"credential_profile_id": self.credential_profile_id,
|
||||
"credential_profile_label": self.credential_profile_label,
|
||||
"credential_mode": self.credential_mode,
|
||||
"credential_source": self.credential_source,
|
||||
"credentials_configured": self.credentials_configured,
|
||||
"username": self.username,
|
||||
"capabilities": list(self.capabilities),
|
||||
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||
"metadata": _response_metadata(self.metadata),
|
||||
"source_kind": self.source_kind,
|
||||
}
|
||||
|
||||
|
||||
def connector_profiles_from_settings(settings: object | None = None) -> list[ConnectorProfile]:
|
||||
raw = _raw_profiles_payload(settings)
|
||||
if raw in (None, ""):
|
||||
return []
|
||||
payload = json.loads(raw) if isinstance(raw, str) else raw
|
||||
return connector_profiles_from_payload(payload)
|
||||
|
||||
|
||||
def connector_profiles_from_payload(value: object) -> list[ConnectorProfile]:
|
||||
if isinstance(value, Mapping):
|
||||
raw_profiles = value.get("profiles", [])
|
||||
elif isinstance(value, list):
|
||||
raw_profiles = value
|
||||
else:
|
||||
raise ValueError("Connector profiles must be a JSON object with profiles or a list")
|
||||
if not isinstance(raw_profiles, list):
|
||||
raise ValueError("Connector profiles payload requires a profiles list")
|
||||
return [_profile_from_mapping(item) for item in raw_profiles if isinstance(item, Mapping)]
|
||||
|
||||
|
||||
def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
|
||||
profile_id = _profile_id_from_mapping(value)
|
||||
provider = _provider_from_mapping(value)
|
||||
scope_type, scope_id = _scope_from_mapping(value)
|
||||
credentials = _credentials_from_mapping(value)
|
||||
mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none"
|
||||
profile = _profile_from_normalized_mapping(
|
||||
value,
|
||||
profile_id=profile_id,
|
||||
provider=provider,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
credential_mode=mode,
|
||||
credentials=credentials,
|
||||
)
|
||||
result = ConnectorProfile(
|
||||
id=profile.id,
|
||||
label=profile.label,
|
||||
provider=profile.provider,
|
||||
scope_type=profile.scope_type,
|
||||
scope_id=profile.scope_id,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
enabled=profile.enabled,
|
||||
credential_profile_id=profile.credential_profile_id,
|
||||
credential_profile_label=profile.credential_profile_label,
|
||||
credential_mode=profile.credential_mode,
|
||||
username=profile.username,
|
||||
password_env=profile.password_env,
|
||||
token_env=profile.token_env,
|
||||
secret_ref=profile.secret_ref,
|
||||
password_value=profile.password_value,
|
||||
token_value=profile.token_value,
|
||||
has_inline_secret=profile.has_inline_secret,
|
||||
capabilities=profile.capabilities,
|
||||
policy_sources=tuple(_policy_sources_from_profile(value, profile)),
|
||||
metadata=profile.metadata,
|
||||
source_kind="settings",
|
||||
)
|
||||
validate_deployment_connector_references(
|
||||
source_kind=result.source_kind,
|
||||
password_env=result.password_env,
|
||||
token_env=result.token_env,
|
||||
metadata=result.metadata,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _profile_id_from_mapping(value: Mapping[str, Any]) -> str:
|
||||
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
|
||||
if not profile_id:
|
||||
raise ValueError("Connector profiles require id")
|
||||
return profile_id
|
||||
|
||||
|
||||
def _provider_from_mapping(value: Mapping[str, Any]) -> str:
|
||||
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
|
||||
if provider not in _SUPPORTED_PROVIDERS:
|
||||
raise ValueError(f"Unsupported connector provider: {provider}")
|
||||
return provider
|
||||
|
||||
|
||||
def _scope_from_mapping(value: Mapping[str, Any]) -> tuple[str, str | None]:
|
||||
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||
scope_id = _clean(value.get("scope_id"))
|
||||
if scope_type == "system":
|
||||
return scope_type, None
|
||||
if not scope_id:
|
||||
raise ValueError(f"{scope_type} connector profiles require scope_id")
|
||||
return scope_type, scope_id
|
||||
|
||||
|
||||
def _credentials_from_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||
credentials = value.get("credentials")
|
||||
return credentials if isinstance(credentials, Mapping) else {}
|
||||
|
||||
|
||||
def _profile_from_normalized_mapping(
|
||||
value: Mapping[str, Any],
|
||||
*,
|
||||
profile_id: str,
|
||||
provider: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
credential_mode: str,
|
||||
credentials: Mapping[str, Any],
|
||||
) -> ConnectorProfile:
|
||||
return ConnectorProfile(
|
||||
id=profile_id,
|
||||
label=_clean(value.get("label") or value.get("name")) or profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
|
||||
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
|
||||
enabled=_bool(value.get("enabled"), default=True),
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
|
||||
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
|
||||
credential_mode=credential_mode,
|
||||
username=_clean(value.get("username") or credentials.get("username")),
|
||||
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
|
||||
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
|
||||
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
|
||||
password_value=_clean(value.get("password") or credentials.get("password")),
|
||||
token_value=_clean(value.get("token") or credentials.get("token")),
|
||||
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
|
||||
capabilities=tuple(_string_list(value.get("capabilities"))),
|
||||
metadata=_public_metadata(value.get("metadata")),
|
||||
)
|
||||
|
||||
|
||||
def _raw_profiles_payload(settings: object | None) -> object:
|
||||
for key in _ENV_JSON_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return value
|
||||
for key in _ENV_FILE_KEYS:
|
||||
path = os.environ.get(key)
|
||||
if path:
|
||||
with open(path, encoding="utf-8") as handle:
|
||||
return handle.read()
|
||||
if settings is not None:
|
||||
value = getattr(settings, "files_connector_profiles_json", None)
|
||||
if value:
|
||||
return value
|
||||
value = getattr(settings, "files_connector_profiles", None)
|
||||
if value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _policy_sources_from_profile(value: Mapping[str, Any], profile: ConnectorProfile) -> list[ConnectorPolicySource]:
|
||||
raw_sources: list[Mapping[str, Any]] = []
|
||||
policy = value.get("policy")
|
||||
if isinstance(policy, Mapping):
|
||||
raw_sources.append({
|
||||
"scope_type": profile.scope_type,
|
||||
"scope_id": profile.scope_id,
|
||||
"label": profile.label,
|
||||
"policy": policy,
|
||||
})
|
||||
configured_sources = value.get("policy_sources") or value.get("connector_policy_sources")
|
||||
if configured_sources is not None:
|
||||
raw = connector_policy_sources_from_payload(configured_sources)
|
||||
raw_sources.extend(_policy_source_response(source) for source in raw)
|
||||
return connector_policy_sources_from_payload(raw_sources)
|
||||
|
||||
|
||||
def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||
return {
|
||||
"scope_type": source.scope_type,
|
||||
"scope_id": source.scope_id,
|
||||
"label": source.label,
|
||||
"policy": dict(source.policy or {}),
|
||||
}
|
||||
|
||||
|
||||
def _public_metadata(value: object) -> Mapping[str, Any]:
|
||||
if not isinstance(value, Mapping):
|
||||
return {}
|
||||
return {
|
||||
str(key): item
|
||||
for key, item in value.items()
|
||||
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||
}
|
||||
|
||||
|
||||
def _response_metadata(value: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
str(key): item
|
||||
for key, item in value.items()
|
||||
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||
and not str(key).strip().casefold().endswith("_env")
|
||||
and str(key).strip().casefold() != "ca_bundle"
|
||||
}
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
values: Iterable[object] = value.split(",")
|
||||
elif isinstance(value, Iterable) and not isinstance(value, (bytes, bytearray, Mapping)):
|
||||
values = value
|
||||
else:
|
||||
values = (value,)
|
||||
return [str(item).strip() for item in values if str(item).strip()]
|
||||
|
||||
|
||||
def _bool(value: object, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return value.strip().casefold() not in {"0", "false", "no", "off"}
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,191 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from importlib.util import find_spec
|
||||
|
||||
|
||||
CONNECTOR_PROVIDER_CAPABILITY = "files.connector.providers"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorProviderDescriptor:
|
||||
provider: str
|
||||
label: str
|
||||
protocol: str
|
||||
implemented: bool
|
||||
browse_supported: bool
|
||||
import_supported: bool
|
||||
optional_dependency: str | None
|
||||
permission_model: str
|
||||
sync_strategy: str
|
||||
conflict_strategy: str
|
||||
preview_strategy: str
|
||||
audit_events: tuple[str, ...]
|
||||
notes: str | None = None
|
||||
|
||||
def to_response(self) -> dict[str, object]:
|
||||
return {
|
||||
"provider": self.provider,
|
||||
"label": self.label,
|
||||
"protocol": self.protocol,
|
||||
"implemented": self.implemented,
|
||||
"installed": self.installed,
|
||||
"browse_supported": self.browse_supported,
|
||||
"import_supported": self.import_supported,
|
||||
"write_supported": self.provider == "s3",
|
||||
"optional_dependency": self.optional_dependency,
|
||||
"permission_model": self.permission_model,
|
||||
"sync_strategy": self.sync_strategy,
|
||||
"conflict_strategy": self.conflict_strategy,
|
||||
"preview_strategy": self.preview_strategy,
|
||||
"audit_events": list(self.audit_events),
|
||||
"notes": self.notes,
|
||||
}
|
||||
|
||||
@property
|
||||
def installed(self) -> bool:
|
||||
if self.optional_dependency is None:
|
||||
return self.implemented
|
||||
return find_spec(self.optional_dependency) is not None
|
||||
|
||||
|
||||
def connector_provider_descriptors() -> tuple[ConnectorProviderDescriptor, ...]:
|
||||
freeze_model = "Imported or synced connector files become managed GovOPlaN files with frozen blob/version provenance."
|
||||
governed_permissions = "GovOPlaN profile scope and connector policy are enforced before browse/import/sync; remote ACLs are still evaluated by the upstream service."
|
||||
return (
|
||||
ConnectorProviderDescriptor(
|
||||
provider="seafile",
|
||||
label="Seafile",
|
||||
protocol="seafile-api",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Native account-token API for libraries, directories, file detail, and download-link import; WebDAV opt-in remains available.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="nextcloud",
|
||||
label="Nextcloud",
|
||||
protocol="webdav",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand WebDAV browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Uses the configured Nextcloud WebDAV endpoint, basic auth, or bearer token profile credentials.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="webdav",
|
||||
label="WebDAV",
|
||||
protocol="webdav",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand WebDAV browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Generic WebDAV provider for standards-compatible stores.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="smb",
|
||||
label="SMB",
|
||||
protocol="smb",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency="smbprotocol",
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand browse/import/sync over administrator-declared shares; sync updates managed versions and never mutates the remote share.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after download.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the share.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Initial sessions, reconnects, aliases, and DFS referral targets use the Files-owned pinned smbprotocol transport and the deployment-wide private-network policy.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="s3",
|
||||
label="S3-compatible object storage",
|
||||
protocol="s3-api",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency="boto3",
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand bucket/prefix browse and object import/sync; sync updates managed versions and never mutates the remote bucket.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after object download.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the bucket.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Every botocore connection pool uses the Files-owned pinned transport, including retries, redirects, endpoint discovery, and virtual-host aliases; outbound proxies and ambient credential discovery are disabled.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="sharepoint",
|
||||
label="SharePoint",
|
||||
protocol="microsoft-graph/sharepoint",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph or SharePoint credentials.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from SharePoint.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Provider/profile key is reserved; live browsing/import still needs Graph/SharePoint authentication and paging implementation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="onedrive",
|
||||
label="OneDrive",
|
||||
protocol="microsoft-graph/onedrive",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph credentials.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from OneDrive.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Provider/profile key is reserved; live browsing/import still needs Graph drive selection, OAuth/app registration, and paging implementation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="nfs",
|
||||
label="NFS",
|
||||
protocol="nfs",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned optional provider over administrator-mounted paths or a sidecar service.",
|
||||
conflict_strategy="Will use managed import conflict_strategy handling after download.",
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from the mount.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Kept optional because NFS availability is deployment and host-kernel dependent.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="local",
|
||||
label="Local filesystem",
|
||||
protocol="file",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model="Local connector access should be restricted to administrator-declared roots plus GovOPlaN profile and policy checks.",
|
||||
sync_strategy="Planned optional provider; managed storage local backend already exists separately.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Separate from the existing managed-file local storage backend.",
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,264 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import false, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileConnectorSpace
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, user_group_ids
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.connector_browse import normalize_connector_browse_path
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
SYNC_MODES = {"manual"}
|
||||
WRITABLE_PROVIDERS = {"s3"}
|
||||
|
||||
|
||||
def connector_space_owner_id(space: FileConnectorSpace) -> str:
|
||||
return space.owner_user_id if space.owner_type == "user" else space.owner_group_id # type: ignore[return-value]
|
||||
|
||||
|
||||
def create_connector_space(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
label: str,
|
||||
profile: ConnectorProfile,
|
||||
library_id: str | None = None,
|
||||
remote_path: str | None = None,
|
||||
sync_mode: str = "manual",
|
||||
read_only: bool = True,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
label = _normalize_label(label)
|
||||
sync_mode = _normalize_sync_mode(sync_mode)
|
||||
validate_connector_space_write_mode(profile, read_only=read_only)
|
||||
remote_path = normalize_connector_browse_path(remote_path)
|
||||
library_id = _clean_optional(library_id)
|
||||
|
||||
existing = _owned_query(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id).filter(
|
||||
FileConnectorSpace.label == label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if existing is not None:
|
||||
raise FileStorageError(f"Connector space already exists: {label}")
|
||||
|
||||
space = FileConnectorSpace(
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_user_id=owner_id if owner_type == "user" else None,
|
||||
owner_group_id=owner_id if owner_type == "group" else None,
|
||||
label=label,
|
||||
connector_profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
library_id=library_id,
|
||||
remote_path=remote_path,
|
||||
sync_mode=sync_mode,
|
||||
read_only=read_only,
|
||||
is_active=True,
|
||||
created_by_user_id=user_id,
|
||||
metadata_=dict(metadata or {}),
|
||||
)
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def list_connector_spaces_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
include_inactive: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileConnectorSpace]:
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == tenant_id,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
)
|
||||
if not include_inactive:
|
||||
query = query.filter(FileConnectorSpace.is_active.is_(True))
|
||||
|
||||
if owner_type:
|
||||
if not owner_id:
|
||||
raise FileStorageError("owner_id is required when owner_type is set")
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
query = _owner_filter(query, owner_type, owner_id)
|
||||
elif not is_admin:
|
||||
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||
query = query.filter(
|
||||
or_(
|
||||
FileConnectorSpace.owner_user_id == user_id,
|
||||
FileConnectorSpace.owner_group_id.in_(group_ids) if group_ids else false(),
|
||||
)
|
||||
)
|
||||
|
||||
return query.order_by(FileConnectorSpace.owner_type.asc(), FileConnectorSpace.label.asc()).all()
|
||||
|
||||
|
||||
def get_connector_space_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
space_id: str,
|
||||
include_inactive: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.id == space_id,
|
||||
FileConnectorSpace.tenant_id == tenant_id,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
)
|
||||
if not include_inactive:
|
||||
query = query.filter(FileConnectorSpace.is_active.is_(True))
|
||||
space = query.one_or_none()
|
||||
if space is None:
|
||||
raise FileStorageError("Connector space not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return space
|
||||
|
||||
|
||||
def update_connector_space(
|
||||
session: Session,
|
||||
space: FileConnectorSpace,
|
||||
*,
|
||||
user_id: str,
|
||||
label: str | None = None,
|
||||
library_id: str | None = None,
|
||||
remote_path: str | None = None,
|
||||
sync_mode: str | None = None,
|
||||
read_only: bool | None = None,
|
||||
is_active: bool | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=space.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if label is not None:
|
||||
new_label = _normalize_label(label)
|
||||
existing = _owned_query(
|
||||
session,
|
||||
tenant_id=space.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
).filter(
|
||||
FileConnectorSpace.id != space.id,
|
||||
FileConnectorSpace.label == new_label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if existing is not None:
|
||||
raise FileStorageError(f"Connector space already exists: {new_label}")
|
||||
space.label = new_label
|
||||
if library_id is not None:
|
||||
space.library_id = _clean_optional(library_id)
|
||||
if remote_path is not None:
|
||||
space.remote_path = normalize_connector_browse_path(remote_path)
|
||||
if sync_mode is not None:
|
||||
space.sync_mode = _normalize_sync_mode(sync_mode)
|
||||
if read_only is not None:
|
||||
space.read_only = bool(read_only)
|
||||
if is_active is not None:
|
||||
space.is_active = bool(is_active)
|
||||
if metadata is not None:
|
||||
space.metadata_ = dict(metadata)
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def validate_connector_space_write_mode(
|
||||
profile: ConnectorProfile, *, read_only: bool
|
||||
) -> None:
|
||||
if read_only:
|
||||
return
|
||||
if profile.provider not in WRITABLE_PROVIDERS or "write" not in profile.capabilities:
|
||||
raise FileStorageError(
|
||||
"Two-way mode requires an S3 connection with the write capability enabled"
|
||||
)
|
||||
|
||||
|
||||
def soft_delete_connector_space(
|
||||
session: Session,
|
||||
space: FileConnectorSpace,
|
||||
*,
|
||||
user_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=space.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
space.deleted_at = utcnow()
|
||||
space.is_active = False
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def _owned_query(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == tenant_id,
|
||||
FileConnectorSpace.owner_type == owner_type,
|
||||
)
|
||||
return _owner_filter(query, owner_type, owner_id)
|
||||
|
||||
|
||||
def _owner_filter(query, owner_type: str, owner_id: str):
|
||||
if owner_type == "user":
|
||||
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def _normalize_label(value: str) -> str:
|
||||
label = value.strip()
|
||||
if not label:
|
||||
raise FileStorageError("Connector space label is required")
|
||||
if len(label) > 255:
|
||||
raise FileStorageError("Connector space label is too long")
|
||||
return label
|
||||
|
||||
|
||||
def _normalize_sync_mode(value: str) -> str:
|
||||
mode = value.strip().casefold()
|
||||
if mode not in SYNC_MODES:
|
||||
raise FileStorageError(f"Unsupported connector space sync mode: {value}")
|
||||
return mode
|
||||
|
||||
|
||||
def _clean_optional(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,213 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterable
|
||||
from dataclasses import replace
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
select_database_connector_profiles,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import (
|
||||
ConnectorProfile,
|
||||
connector_profiles_from_settings,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
connector_policy_decision,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
effective_connector_policy_sources,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_providers import (
|
||||
connector_provider_descriptors,
|
||||
)
|
||||
|
||||
|
||||
CampaignVisibility = Callable[[str], bool]
|
||||
|
||||
|
||||
def visible_connector_profiles_for_actor(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: Iterable[str],
|
||||
settings: object | None,
|
||||
provider: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
campaign_visible: CampaignVisibility | None = None,
|
||||
include_disabled: bool = False,
|
||||
include_admin_scopes: bool = False,
|
||||
include_effective_policy: bool = True,
|
||||
) -> list[ConnectorProfile]:
|
||||
"""Return the same actor-filtered connector set used by API and docs surfaces."""
|
||||
|
||||
provider_norm = provider.strip().casefold() if provider else None
|
||||
actor_group_ids = {str(group_id) for group_id in group_ids if str(group_id)}
|
||||
database_profiles, database_profile_ids = select_database_connector_profiles(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
row_visible=lambda row: (
|
||||
include_admin_scopes
|
||||
and row.scope_type in {"user", "group", "campaign"}
|
||||
) or _scope_visible_to_actor(
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=actor_group_ids,
|
||||
campaign_id=campaign_id,
|
||||
campaign_visible=campaign_visible,
|
||||
),
|
||||
)
|
||||
configured_profiles = connector_profiles_from_settings(settings)
|
||||
profiles_by_id: dict[str, ConnectorProfile] = {}
|
||||
for profile in database_profiles:
|
||||
if profile.id not in profiles_by_id:
|
||||
profiles_by_id[profile.id] = profile
|
||||
for profile in configured_profiles:
|
||||
if profile.id not in database_profile_ids and profile.id not in profiles_by_id:
|
||||
profiles_by_id[profile.id] = profile
|
||||
|
||||
visible: list[ConnectorProfile] = []
|
||||
for profile in profiles_by_id.values():
|
||||
if not (profile.enabled or include_disabled):
|
||||
continue
|
||||
if provider_norm is not None and profile.provider != provider_norm:
|
||||
continue
|
||||
admin_scope_visible = include_admin_scopes and profile.scope_type in {
|
||||
"user",
|
||||
"group",
|
||||
"campaign",
|
||||
}
|
||||
if not admin_scope_visible and not _profile_visible_to_actor(
|
||||
profile,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=actor_group_ids,
|
||||
campaign_id=campaign_id,
|
||||
campaign_visible=campaign_visible,
|
||||
):
|
||||
continue
|
||||
visible.append(
|
||||
_with_effective_connector_policy(
|
||||
session, tenant_id=tenant_id, profile=profile
|
||||
)
|
||||
if include_effective_policy
|
||||
else profile
|
||||
)
|
||||
return visible
|
||||
|
||||
|
||||
def connector_profile_usable_for_import(profile: ConnectorProfile) -> bool:
|
||||
"""Whether a visible profile can safely offer the current live browse/import task."""
|
||||
|
||||
effective_endpoint_url = connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
)
|
||||
if (
|
||||
not profile.enabled
|
||||
or not effective_endpoint_url
|
||||
or not profile.credentials_configured
|
||||
or bool(profile.secret_ref)
|
||||
):
|
||||
return False
|
||||
descriptor = next(
|
||||
(
|
||||
item
|
||||
for item in connector_provider_descriptors()
|
||||
if item.provider == profile.provider
|
||||
),
|
||||
None,
|
||||
)
|
||||
if descriptor is None:
|
||||
return False
|
||||
if not (
|
||||
descriptor.implemented
|
||||
and descriptor.installed
|
||||
and descriptor.browse_supported
|
||||
and descriptor.import_supported
|
||||
):
|
||||
return False
|
||||
# Prove that the initial root browse performed by the current Files UI is
|
||||
# policy-allowed. A later selected remote path/item is checked again.
|
||||
return connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path="",
|
||||
external_url=effective_endpoint_url,
|
||||
operation="import",
|
||||
),
|
||||
profile.policy_sources,
|
||||
).allowed
|
||||
|
||||
|
||||
def _profile_visible_to_actor(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: set[str],
|
||||
campaign_id: str | None,
|
||||
campaign_visible: CampaignVisibility | None,
|
||||
) -> bool:
|
||||
return _scope_visible_to_actor(
|
||||
scope_type=profile.scope_type,
|
||||
scope_id=profile.scope_id,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=group_ids,
|
||||
campaign_id=campaign_id,
|
||||
campaign_visible=campaign_visible,
|
||||
)
|
||||
|
||||
|
||||
def _scope_visible_to_actor(
|
||||
*,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: set[str],
|
||||
campaign_id: str | None,
|
||||
campaign_visible: CampaignVisibility | None,
|
||||
) -> bool:
|
||||
if scope_type == "system":
|
||||
return True
|
||||
if scope_type == "tenant":
|
||||
return scope_id == tenant_id
|
||||
if scope_type == "user":
|
||||
return scope_id == user_id
|
||||
if scope_type == "group":
|
||||
return bool(scope_id and scope_id in group_ids)
|
||||
if scope_type != "campaign" or not scope_id:
|
||||
return False
|
||||
if campaign_id and scope_id != campaign_id:
|
||||
return False
|
||||
return bool(campaign_visible and campaign_visible(scope_id))
|
||||
|
||||
|
||||
def _with_effective_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
profile: ConnectorProfile,
|
||||
) -> ConnectorProfile:
|
||||
sources = effective_connector_policy_sources(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
scope_type=profile.scope_type,
|
||||
scope_id=profile.scope_id,
|
||||
)
|
||||
if not sources:
|
||||
return profile
|
||||
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))
|
||||
@@ -0,0 +1,438 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
from typing import Any, Mapping
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
_clean,
|
||||
_s3_bucket,
|
||||
_s3_client,
|
||||
_s3_object_key,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorWriteResult:
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool
|
||||
provider: str
|
||||
remote_path: str
|
||||
revision: str | None
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _HeadProbe:
|
||||
observed: Mapping[str, Any] | None
|
||||
verified: bool
|
||||
exception_type: str | None = None
|
||||
|
||||
|
||||
def write_connector_file(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
tenant_id: str,
|
||||
library_id: str | None,
|
||||
remote_path: str,
|
||||
data: bytes,
|
||||
content_type: str | None,
|
||||
expected_revision: str | None,
|
||||
idempotency_key: str,
|
||||
) -> ConnectorWriteResult:
|
||||
if profile.provider != "s3" or "write" not in profile.capabilities:
|
||||
raise FileStorageError(
|
||||
"Remote writes require an S3 connection with the write capability enabled"
|
||||
)
|
||||
try:
|
||||
normalized_path = normalize_connector_browse_path(remote_path)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if not normalized_path:
|
||||
raise FileStorageError("Remote write requires an object path")
|
||||
try:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
key = _s3_object_key(profile, normalized_path)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if not bucket or not key:
|
||||
raise FileStorageError("S3 remote write requires a bucket and object path")
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
target_digest = hashlib.sha256(f"{bucket}:{key}".encode("utf-8")).hexdigest()
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"connector_profile_id": profile.id,
|
||||
"provider": profile.provider,
|
||||
"remote_target_sha256": target_digest,
|
||||
"content_sha256": checksum,
|
||||
"size_bytes": len(data),
|
||||
"expected_revision": expected_revision,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="connector-s3-write",
|
||||
idempotency_key=f"files-connector-write:{idempotency_key}",
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the connector space is explicitly configured for two-way writes",
|
||||
"the connection and inherited policy allow the exact remote path",
|
||||
"a conditional create or expected remote revision prevents blind overwrite",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"inspect provider metadata for the request and content digests",
|
||||
"resolve success only for an exact digest match or retry after verified absence",
|
||||
),
|
||||
verification_steps=(
|
||||
"read S3 object metadata after the write",
|
||||
"match the recorded request and content digests without downloading content",
|
||||
),
|
||||
),
|
||||
precondition_evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"content_sha256": checksum,
|
||||
"size_bytes": len(data),
|
||||
"conditional_write": True,
|
||||
},
|
||||
lease_resource_key=f"files:connector:{tenant_id}:{profile.id}:{target_digest[:40]}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_connector_object",
|
||||
resource_id=target_digest,
|
||||
metadata={
|
||||
"resources": ["postgresql", "s3-connector"],
|
||||
"provider": "s3",
|
||||
"connector_profile_id": profile.id,
|
||||
},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"This remote object is owned by another write or unresolved recovery operation"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; the remote object was not changed"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=started.status,
|
||||
replayed=True,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=None,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
operation = started.operation
|
||||
try:
|
||||
client = _s3_client(profile)
|
||||
except (ConnectorBrowseError, ConnectorBrowseUnsupported) as exc:
|
||||
operation.reject(
|
||||
summary="The S3 client was unavailable before the remote effect",
|
||||
evidence=_verified(
|
||||
{"external_effect_started": False}, exception_type=type(exc).__name__
|
||||
),
|
||||
)
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
try:
|
||||
try:
|
||||
before = _head_object(client, bucket=bucket, key=key)
|
||||
except Exception as exc:
|
||||
operation.reject(
|
||||
summary="Remote preconditions could not be inspected before the effect",
|
||||
evidence=_verified(
|
||||
{"external_effect_started": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise FileStorageError("S3 remote revision lookup failed") from exc
|
||||
expected = _normalize_revision(expected_revision)
|
||||
if before is not None and expected is None:
|
||||
operation.reject(
|
||||
summary="A blind remote overwrite was rejected",
|
||||
evidence=_verified(
|
||||
{"remote_object_exists": True, "expected_revision_supplied": False}
|
||||
),
|
||||
)
|
||||
raise FileStorageError(
|
||||
"The remote object already exists; reload it and supply its expected revision"
|
||||
)
|
||||
if expected is not None and (
|
||||
before is None or expected not in _observed_revisions(before)
|
||||
):
|
||||
operation.reject(
|
||||
summary="The expected remote revision did not match",
|
||||
evidence=_verified(
|
||||
{
|
||||
"remote_object_exists": before is not None,
|
||||
"expected_revision_matches": False,
|
||||
}
|
||||
),
|
||||
)
|
||||
raise FileStorageError("The remote object changed; reload before writing")
|
||||
params: dict[str, object] = {
|
||||
"Bucket": bucket,
|
||||
"Key": key,
|
||||
"Body": data,
|
||||
"Metadata": {
|
||||
"govoplan-sha256": checksum,
|
||||
"govoplan-request-id": started.operation_id,
|
||||
},
|
||||
}
|
||||
if content_type:
|
||||
params["ContentType"] = content_type
|
||||
if before is None:
|
||||
params["IfNoneMatch"] = "*"
|
||||
else:
|
||||
params["IfMatch"] = str(before.get("ETag") or expected_revision or "")
|
||||
try:
|
||||
client.put_object(**params)
|
||||
except Exception as exc:
|
||||
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||
if probe.verified and _matches_effect(
|
||||
probe.observed,
|
||||
checksum=checksum,
|
||||
operation_id=started.operation_id,
|
||||
):
|
||||
operation.succeed(
|
||||
evidence=_success_evidence(
|
||||
probe.observed,
|
||||
checksum=checksum,
|
||||
operation_id=started.operation_id,
|
||||
)
|
||||
)
|
||||
return _result(
|
||||
started.operation_id,
|
||||
normalized_path,
|
||||
checksum,
|
||||
len(data),
|
||||
probe.observed,
|
||||
)
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The S3 write outcome requires reconciliation",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
"exception_type": type(exc).__name__,
|
||||
},
|
||||
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=_revision(probe.observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||
if not probe.verified:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The S3 write returned but provider evidence could not be queried",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
},
|
||||
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=None,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
observed = probe.observed
|
||||
if not _matches_effect(
|
||||
observed, checksum=checksum, operation_id=started.operation_id
|
||||
):
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="The S3 write returned but exact provider evidence did not match",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
},
|
||||
failure_summary="Reconcile the remote object before another write",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=_revision(observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
operation.succeed(
|
||||
evidence=_success_evidence(
|
||||
observed, checksum=checksum, operation_id=started.operation_id
|
||||
)
|
||||
)
|
||||
return _result(
|
||||
started.operation_id,
|
||||
normalized_path,
|
||||
checksum,
|
||||
len(data),
|
||||
observed,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
def _head_object(client: Any, *, bucket: str, key: str) -> Mapping[str, Any] | None:
|
||||
try:
|
||||
response = client.head_object(Bucket=bucket, Key=key)
|
||||
except Exception as exc:
|
||||
if _is_not_found(exc):
|
||||
return None
|
||||
raise
|
||||
if not isinstance(response, Mapping):
|
||||
raise FileStorageError("S3 connector returned invalid object metadata")
|
||||
return response
|
||||
|
||||
|
||||
def _probe_head_object(client: Any, *, bucket: str, key: str) -> _HeadProbe:
|
||||
try:
|
||||
return _HeadProbe(
|
||||
observed=_head_object(client, bucket=bucket, key=key), verified=True
|
||||
)
|
||||
except Exception as exc:
|
||||
return _HeadProbe(
|
||||
observed=None,
|
||||
verified=False,
|
||||
exception_type=type(exc).__name__,
|
||||
)
|
||||
|
||||
|
||||
def _is_not_found(exc: Exception) -> bool:
|
||||
response = getattr(exc, "response", None)
|
||||
if not isinstance(response, Mapping):
|
||||
return False
|
||||
error = response.get("Error")
|
||||
code = error.get("Code") if isinstance(error, Mapping) else None
|
||||
status = response.get("ResponseMetadata")
|
||||
http_status = status.get("HTTPStatusCode") if isinstance(status, Mapping) else None
|
||||
return str(code).casefold() in {"404", "nosuchkey", "notfound"} or http_status == 404
|
||||
|
||||
|
||||
def _metadata(observed: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||
value = observed.get("Metadata") if observed else None
|
||||
return value if isinstance(value, Mapping) else {}
|
||||
|
||||
|
||||
def _matches_effect(
|
||||
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||
) -> bool:
|
||||
metadata = _metadata(observed)
|
||||
return bool(
|
||||
observed is not None
|
||||
and _clean(metadata.get("govoplan-sha256")) == checksum
|
||||
and _clean(metadata.get("govoplan-request-id")) == operation_id
|
||||
)
|
||||
|
||||
|
||||
def _revision(observed: Mapping[str, Any] | None) -> str | None:
|
||||
if observed is None:
|
||||
return None
|
||||
return _clean(observed.get("VersionId") or observed.get("ETag"))
|
||||
|
||||
|
||||
def _normalize_revision(value: str | None) -> str | None:
|
||||
cleaned = _clean(value)
|
||||
return cleaned.strip('"') if cleaned else None
|
||||
|
||||
|
||||
def _observed_revisions(observed: Mapping[str, Any]) -> set[str]:
|
||||
return {
|
||||
normalized
|
||||
for value in (observed.get("VersionId"), observed.get("ETag"))
|
||||
if (normalized := _normalize_revision(_clean(value))) is not None
|
||||
}
|
||||
|
||||
|
||||
def _success_evidence(
|
||||
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||
) -> dict[str, object]:
|
||||
return _verified(
|
||||
{
|
||||
"remote_object_present": observed is not None,
|
||||
"content_digest_matches": _clean(_metadata(observed).get("govoplan-sha256")) == checksum,
|
||||
"request_marker_matches": _clean(_metadata(observed).get("govoplan-request-id")) == operation_id,
|
||||
},
|
||||
revision=_revision(observed),
|
||||
content_sha256=checksum,
|
||||
)
|
||||
|
||||
|
||||
def _public_observation(probe: _HeadProbe) -> dict[str, object]:
|
||||
observed = probe.observed
|
||||
return {
|
||||
"probe_verified": probe.verified,
|
||||
"probe_exception_type": probe.exception_type,
|
||||
"present": observed is not None,
|
||||
"revision": _revision(observed),
|
||||
"has_content_digest": bool(_clean(_metadata(observed).get("govoplan-sha256"))),
|
||||
"has_request_marker": bool(_clean(_metadata(observed).get("govoplan-request-id"))),
|
||||
}
|
||||
|
||||
|
||||
def _verified(checks: dict[str, object], **details: object) -> dict[str, object]:
|
||||
return {"verified": True, "checks": checks, **details}
|
||||
|
||||
|
||||
def _result(
|
||||
operation_id: str,
|
||||
remote_path: str,
|
||||
checksum: str,
|
||||
size_bytes: int,
|
||||
observed: Mapping[str, Any] | None,
|
||||
) -> ConnectorWriteResult:
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=operation_id,
|
||||
status=RecoveryStatus.SUCCEEDED.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=remote_path,
|
||||
revision=_revision(observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=size_bytes,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["ConnectorWriteResult", "write_connector_file"]
|
||||
@@ -0,0 +1,93 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from govoplan_core.core.encryption import (
|
||||
ContentProtectionRequest,
|
||||
ContentUnprotectionRequest,
|
||||
ProtectedContent,
|
||||
encryption_content_cipher,
|
||||
)
|
||||
from govoplan_files.backend.runtime import get_registry
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
FILES_PROTECTION_PROFILE = "files-server-envelope-v1"
|
||||
|
||||
|
||||
def protect_blob_content(
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
blob_id: str,
|
||||
vault_id: str,
|
||||
ciphertext_ref: str,
|
||||
plaintext: bytes,
|
||||
actor_id: str,
|
||||
content_type: str | None,
|
||||
) -> ProtectedContent:
|
||||
capability = encryption_content_cipher(get_registry())
|
||||
if capability is None:
|
||||
raise FileStorageError(
|
||||
"File encryption was requested, but the Encryption module is unavailable."
|
||||
)
|
||||
try:
|
||||
return capability.protect_content(
|
||||
session,
|
||||
request=ContentProtectionRequest(
|
||||
tenant_id=tenant_id,
|
||||
owner_module="files",
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
profile_id=FILES_PROTECTION_PROFILE,
|
||||
vault_id=vault_id,
|
||||
ciphertext_ref=ciphertext_ref,
|
||||
plaintext=plaintext,
|
||||
policy_decision_ref="files:explicit-vault-selection:v1",
|
||||
idempotency_key=f"file-blob:{blob_id}:content:v1",
|
||||
actor_id=actor_id,
|
||||
metadata={
|
||||
"content_type": content_type or "application/octet-stream",
|
||||
},
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
raise FileStorageError(
|
||||
"Managed file content could not be protected by the configured vault."
|
||||
) from exc
|
||||
|
||||
|
||||
def unprotect_blob_content(
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
blob_id: str,
|
||||
envelope_id: str,
|
||||
ciphertext: bytes,
|
||||
) -> bytes:
|
||||
capability = encryption_content_cipher(get_registry())
|
||||
if capability is None:
|
||||
raise FileStorageError(
|
||||
"This file is encrypted and cannot be read while Encryption is unavailable."
|
||||
)
|
||||
try:
|
||||
return capability.unprotect_content(
|
||||
session,
|
||||
request=ContentUnprotectionRequest(
|
||||
tenant_id=tenant_id,
|
||||
owner_module="files",
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
envelope_id=envelope_id,
|
||||
ciphertext=ciphertext,
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
raise FileStorageError(
|
||||
"Managed file content could not be opened with its protection envelope."
|
||||
) from exc
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FILES_PROTECTION_PROFILE",
|
||||
"protect_blob_content",
|
||||
"unprotect_blob_content",
|
||||
]
|
||||
@@ -2,30 +2,47 @@ from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from datetime import datetime
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any, Iterable
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy import and_, exists, func, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.db.models import Group, Tenant, User
|
||||
from govoplan_core.core.optional import reraise_unless_missing_package
|
||||
from govoplan_core.core.campaigns import CAPABILITY_CAMPAIGNS_ACCESS, CampaignAccessProvider
|
||||
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileShare, FileVersion
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, user_group_ids
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
||||
from govoplan_files.backend.runtime import get_registry, settings
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
QUARANTINED_BLOB_STATUSES,
|
||||
read_verified_blob_bytes,
|
||||
)
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
def _campaign_model():
|
||||
try:
|
||||
from govoplan_campaign.backend.db.models import Campaign
|
||||
except ModuleNotFoundError as exc:
|
||||
reraise_unless_missing_package(exc, "govoplan_campaign")
|
||||
raise FileStorageError("Campaign module is not installed") from exc
|
||||
return Campaign
|
||||
def _campaign_access_provider() -> CampaignAccessProvider:
|
||||
registry = get_registry()
|
||||
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||
raise FileStorageError("Campaign module is not installed")
|
||||
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||
if not isinstance(capability, CampaignAccessProvider):
|
||||
raise FileStorageError("Campaign access capability is invalid")
|
||||
return capability
|
||||
|
||||
|
||||
def _ensure_campaign_exists(session: Session, *, tenant_id: str, campaign_id: str) -> None:
|
||||
if not _campaign_access_provider().campaign_exists(session, tenant_id=tenant_id, campaign_id=campaign_id):
|
||||
raise FileStorageError("Campaign not found")
|
||||
|
||||
|
||||
def _asset_query_for_owner(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
||||
@@ -45,8 +62,9 @@ def _storage_backend_name() -> str:
|
||||
return settings.file_storage_backend.lower().strip()
|
||||
|
||||
|
||||
def _storage_key(*, tenant_id: str, checksum: str, filename: str) -> str:
|
||||
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}-{safe_storage_component(filename)}"
|
||||
def _storage_key(*, tenant_id: str, checksum: str) -> str:
|
||||
# Object locators remain opaque so recovery evidence never persists names.
|
||||
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}.blob"
|
||||
|
||||
|
||||
def _get_or_create_blob(
|
||||
@@ -56,40 +74,144 @@ def _get_or_create_blob(
|
||||
data: bytes,
|
||||
filename: str,
|
||||
content_type: str | None,
|
||||
actor_id: str,
|
||||
encryption_vault_id: str | None = None,
|
||||
) -> FileBlob:
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
size = len(data)
|
||||
vault_id = str(encryption_vault_id or "").strip() or None
|
||||
protection_discriminator = f"vault:{vault_id}" if vault_id else "plaintext"
|
||||
blob = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size)
|
||||
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size, FileBlob.protection_discriminator == protection_discriminator)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if blob:
|
||||
backend = get_storage_backend()
|
||||
if not backend.exists(blob.storage_key):
|
||||
repair_required = (
|
||||
blob.integrity_status in QUARANTINED_BLOB_STATUSES
|
||||
or blob.quarantined_at is not None
|
||||
)
|
||||
try:
|
||||
backend.stat(blob.storage_key)
|
||||
except StorageObjectMissing:
|
||||
repair_required = True
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if repair_required:
|
||||
repair_token = hashlib.sha256(
|
||||
repr(
|
||||
(
|
||||
blob.integrity_status,
|
||||
blob.integrity_checked_at,
|
||||
blob.quarantined_at,
|
||||
blob.storage_checksum_sha256,
|
||||
)
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
recovery = begin_blob_write_recovery(
|
||||
session,
|
||||
backend=backend,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob.id,
|
||||
storage_key=blob.storage_key,
|
||||
semantic_checksum_sha256=checksum,
|
||||
semantic_size_bytes=size,
|
||||
protection_discriminator=protection_discriminator,
|
||||
created_new=False,
|
||||
repair_token=repair_token,
|
||||
)
|
||||
stored_data = data
|
||||
expected_envelope_id = blob.encryption_envelope_id
|
||||
if vault_id:
|
||||
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||
|
||||
protected = protect_blob_content(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob.id,
|
||||
vault_id=vault_id,
|
||||
ciphertext_ref=blob.storage_key,
|
||||
plaintext=data,
|
||||
actor_id=actor_id,
|
||||
content_type=content_type,
|
||||
)
|
||||
if blob.encryption_envelope_id not in {None, protected.envelope.envelope_id}:
|
||||
raise FileStorageError("The existing encrypted blob has another protection envelope.")
|
||||
stored_data = protected.ciphertext
|
||||
blob.encryption_envelope_id = protected.envelope.envelope_id
|
||||
expected_envelope_id = protected.envelope.envelope_id
|
||||
blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest()
|
||||
blob.storage_size_bytes = len(stored_data)
|
||||
recovery.prepare_stored_bytes(
|
||||
stored_data,
|
||||
envelope_id=expected_envelope_id,
|
||||
)
|
||||
try:
|
||||
backend.put_bytes(blob.storage_key, data, content_type=content_type)
|
||||
backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
blob.integrity_status = "verified"
|
||||
blob.integrity_checked_at = utcnow()
|
||||
blob.integrity_failure = None
|
||||
blob.quarantined_at = None
|
||||
blob.ref_count += 1
|
||||
session.add(blob)
|
||||
return blob
|
||||
|
||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum, filename=filename)
|
||||
blob_id = str(uuid4())
|
||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
|
||||
backend = get_storage_backend()
|
||||
recovery = begin_blob_write_recovery(
|
||||
session,
|
||||
backend=backend,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob_id,
|
||||
storage_key=storage_key,
|
||||
semantic_checksum_sha256=checksum,
|
||||
semantic_size_bytes=size,
|
||||
protection_discriminator=protection_discriminator,
|
||||
created_new=True,
|
||||
)
|
||||
stored_data = data
|
||||
envelope_id = None
|
||||
if vault_id:
|
||||
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||
|
||||
protected = protect_blob_content(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob_id,
|
||||
vault_id=vault_id,
|
||||
ciphertext_ref=storage_key,
|
||||
plaintext=data,
|
||||
actor_id=actor_id,
|
||||
content_type=content_type,
|
||||
)
|
||||
stored_data = protected.ciphertext
|
||||
envelope_id = protected.envelope.envelope_id
|
||||
recovery.prepare_stored_bytes(stored_data, envelope_id=envelope_id)
|
||||
try:
|
||||
backend.put_bytes(storage_key, data, content_type=content_type)
|
||||
backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
blob = FileBlob(
|
||||
id=blob_id,
|
||||
tenant_id=tenant_id,
|
||||
storage_backend=_storage_backend_name(),
|
||||
storage_bucket=_storage_bucket_name(),
|
||||
storage_key=storage_key,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=size,
|
||||
protection_discriminator=protection_discriminator,
|
||||
encryption_envelope_id=envelope_id,
|
||||
storage_checksum_sha256=hashlib.sha256(stored_data).hexdigest() if vault_id else None,
|
||||
storage_size_bytes=len(stored_data) if vault_id else None,
|
||||
content_type=content_type,
|
||||
ref_count=1,
|
||||
integrity_status="verified",
|
||||
integrity_checked_at=utcnow(),
|
||||
)
|
||||
session.add(blob)
|
||||
session.flush()
|
||||
@@ -114,6 +236,7 @@ def create_file_asset(
|
||||
conflict_strategy: str = "reject",
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||
is_admin: bool = False,
|
||||
encryption_vault_id: str | None = None,
|
||||
) -> UploadedStoredFile:
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
@@ -141,7 +264,7 @@ def create_file_asset(
|
||||
elif action == "rename":
|
||||
logical_path = _next_available_logical_path(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, desired_path=logical_path)
|
||||
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type)
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=encryption_vault_id)
|
||||
asset = FileAsset(
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
@@ -176,6 +299,150 @@ def create_file_asset(
|
||||
return UploadedStoredFile(asset=asset, version=version, blob=blob)
|
||||
|
||||
|
||||
def sync_file_asset_from_source(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
filename: str,
|
||||
data: bytes,
|
||||
metadata: dict[str, Any],
|
||||
folder: str | None = None,
|
||||
display_path: str | None = None,
|
||||
content_type: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
conflict_strategy: str = "rename",
|
||||
is_admin: bool = False,
|
||||
) -> tuple[UploadedStoredFile, str, str | None]:
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
provenance = source_provenance_from_metadata(metadata)
|
||||
if not provenance:
|
||||
raise FileStorageError("Connector sync requires source provenance")
|
||||
existing = find_asset_by_source(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
source_provenance=provenance,
|
||||
)
|
||||
if existing is None:
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder=folder,
|
||||
display_path=display_path,
|
||||
content_type=content_type,
|
||||
metadata=metadata,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return stored, "created", None
|
||||
|
||||
previous_version_id = existing.current_version_id
|
||||
stored, action = update_file_asset_content(
|
||||
session,
|
||||
existing,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
metadata=metadata,
|
||||
)
|
||||
if campaign_id:
|
||||
share_file(session, tenant_id=tenant_id, asset=existing, target_type="campaign", target_id=campaign_id, permission="read", user_id=user_id)
|
||||
return stored, action, previous_version_id
|
||||
|
||||
|
||||
def find_asset_by_source(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
source_provenance: dict[str, Any],
|
||||
) -> FileAsset | None:
|
||||
wanted = _source_identity(source_provenance)
|
||||
if wanted is None:
|
||||
return None
|
||||
assets = (
|
||||
_asset_query_for_owner(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id)
|
||||
.filter(FileAsset.deleted_at.is_(None))
|
||||
.order_by(FileAsset.updated_at.desc())
|
||||
.all()
|
||||
)
|
||||
for asset in assets:
|
||||
if _source_identity(source_provenance_from_metadata(asset.metadata_ or {})) == wanted:
|
||||
return asset
|
||||
return None
|
||||
|
||||
|
||||
def update_file_asset_content(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
filename: str,
|
||||
data: bytes,
|
||||
content_type: str | None,
|
||||
metadata: dict[str, Any],
|
||||
encryption_vault_id: str | None = None,
|
||||
) -> tuple[UploadedStoredFile, str]:
|
||||
if asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||
raise FileStorageError("File not found")
|
||||
safe_filename = filename_from_path(normalize_logical_path(filename, fallback_filename="file"))
|
||||
if not content_type:
|
||||
content_type = mimetypes.guess_type(safe_filename)[0] or "application/octet-stream"
|
||||
current_version, current_blob = current_version_and_blob(session, asset)
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
asset.metadata_ = metadata
|
||||
session.add(asset)
|
||||
inherited_vault_id = encryption_vault_id
|
||||
if inherited_vault_id is None and current_blob.encryption_envelope_id:
|
||||
prefix = "vault:"
|
||||
if current_blob.protection_discriminator.startswith(prefix):
|
||||
inherited_vault_id = current_blob.protection_discriminator[len(prefix) :]
|
||||
inherited_vault_id = str(inherited_vault_id or "").strip() or None
|
||||
target_protection = (
|
||||
f"vault:{inherited_vault_id}" if inherited_vault_id else "plaintext"
|
||||
)
|
||||
if (
|
||||
current_blob.checksum_sha256 == checksum
|
||||
and current_blob.size_bytes == len(data)
|
||||
and current_blob.protection_discriminator == target_protection
|
||||
):
|
||||
return UploadedStoredFile(asset=asset, version=current_version, blob=current_blob), "unchanged"
|
||||
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=inherited_vault_id)
|
||||
version = FileVersion(
|
||||
tenant_id=tenant_id,
|
||||
file_asset_id=asset.id,
|
||||
blob_id=blob.id,
|
||||
version_number=_next_version_number(session, asset.id),
|
||||
filename_at_upload=safe_filename,
|
||||
display_path_at_upload=asset.display_path,
|
||||
content_type=content_type,
|
||||
size_bytes=blob.size_bytes,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
created_by_user_id=user_id,
|
||||
)
|
||||
session.add(version)
|
||||
session.flush()
|
||||
asset.current_version_id = version.id
|
||||
session.add(asset)
|
||||
return UploadedStoredFile(asset=asset, version=version, blob=blob), "updated"
|
||||
|
||||
|
||||
def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_id: str, require_write: bool = False, is_admin: bool = False) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||
@@ -192,7 +459,7 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset.id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
effective_file_share_clause(),
|
||||
FileShare.permission.in_(permission_values),
|
||||
or_(
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
@@ -207,6 +474,32 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
|
||||
return asset
|
||||
|
||||
|
||||
def get_asset_for_share_management(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
asset_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||
raise FileStorageError("File not found")
|
||||
if is_admin:
|
||||
return asset
|
||||
owns_asset = (
|
||||
asset.owner_type == "user"
|
||||
and asset.owner_user_id == user_id
|
||||
) or (
|
||||
asset.owner_type == "group"
|
||||
and asset.owner_group_id
|
||||
in user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||
)
|
||||
if not owns_asset:
|
||||
raise FileStorageError("Only file owners and administrators can manage shares")
|
||||
return asset
|
||||
|
||||
|
||||
def list_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -216,9 +509,119 @@ def list_assets_for_user(
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileAsset]:
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).all()
|
||||
|
||||
|
||||
def list_recent_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
limit: int,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileAsset]:
|
||||
"""Return a bounded recent projection through the normal access query."""
|
||||
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return (
|
||||
query.order_by(FileAsset.updated_at.desc(), FileAsset.id.asc())
|
||||
.limit(max(1, limit))
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def list_assets_for_user_window(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
page_size: int,
|
||||
after_display_path: str | None = None,
|
||||
after_updated_at=None,
|
||||
after_id: str | None = None,
|
||||
) -> tuple[list[FileAsset], bool]:
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if after_display_path is not None and after_updated_at is not None and after_id:
|
||||
query = query.filter(
|
||||
or_(
|
||||
FileAsset.display_path > after_display_path,
|
||||
and_(FileAsset.display_path == after_display_path, FileAsset.updated_at < after_updated_at),
|
||||
and_(FileAsset.display_path == after_display_path, FileAsset.updated_at == after_updated_at, FileAsset.id > after_id),
|
||||
)
|
||||
)
|
||||
rows = query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).limit(page_size + 1).all()
|
||||
return rows[:page_size], len(rows) > page_size
|
||||
|
||||
|
||||
def _asset_visibility_query_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
):
|
||||
query = session.query(FileAsset).filter(FileAsset.tenant_id == tenant_id)
|
||||
if not include_deleted:
|
||||
query = query.filter(FileAsset.deleted_at.is_(None))
|
||||
@@ -229,28 +632,95 @@ def list_assets_for_user(
|
||||
if owner_type == "group" and owner_id:
|
||||
query = query.filter(FileAsset.owner_group_id == owner_id)
|
||||
if campaign_id:
|
||||
query = query.join(FileShare, FileShare.file_asset_id == FileAsset.id).filter(
|
||||
campaign_share = exists().where(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == FileAsset.id,
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
query = query.filter(campaign_share)
|
||||
elif not is_admin and not owner_type:
|
||||
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||
query = query.outerjoin(FileShare, FileShare.file_asset_id == FileAsset.id).filter(
|
||||
active_share = exists().where(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == FileAsset.id,
|
||||
effective_file_share_clause(),
|
||||
or_(
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
(FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
|
||||
(FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
|
||||
)
|
||||
)
|
||||
query = query.filter(
|
||||
or_(
|
||||
(FileAsset.owner_type == "user") & (FileAsset.owner_user_id == user_id),
|
||||
(FileAsset.owner_type == "group") & (FileAsset.owner_group_id.in_(group_ids)),
|
||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
|
||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
|
||||
active_share,
|
||||
)
|
||||
)
|
||||
if path_prefix:
|
||||
prefix = normalize_folder(path_prefix)
|
||||
if prefix:
|
||||
query = query.filter(FileAsset.display_path.like(f"{prefix}/%"))
|
||||
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc()).all()
|
||||
campaign_share_exists = exists().where(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == FileAsset.id,
|
||||
FileShare.target_type == "campaign",
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
campaign_use_exists = exists().where(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == FileAsset.id,
|
||||
)
|
||||
if campaign_usage == "linked":
|
||||
query = query.filter(or_(campaign_share_exists, campaign_use_exists))
|
||||
elif campaign_usage == "unlinked":
|
||||
query = query.filter(~or_(campaign_share_exists, campaign_use_exists))
|
||||
|
||||
if audit_relevant is not None:
|
||||
sent_use_exists = exists().where(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == FileAsset.id,
|
||||
CampaignAttachmentUse.use_stage == "sent",
|
||||
)
|
||||
query = query.filter(sent_use_exists if audit_relevant else ~sent_use_exists)
|
||||
return query
|
||||
|
||||
|
||||
def count_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> int:
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return int(
|
||||
query.order_by(None)
|
||||
.with_entities(func.count(func.distinct(FileAsset.id)))
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
|
||||
|
||||
def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVersion, FileBlob]:
|
||||
@@ -265,13 +735,51 @@ def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVe
|
||||
return version, blob
|
||||
|
||||
|
||||
def current_versions_and_blobs(session: Session, assets: Iterable[FileAsset]) -> dict[str, tuple[FileVersion, FileBlob]]:
|
||||
asset_list = list(assets)
|
||||
if not asset_list:
|
||||
return {}
|
||||
version_ids = [asset.current_version_id for asset in asset_list if asset.current_version_id]
|
||||
if len(version_ids) != len(asset_list):
|
||||
raise FileStorageError("File has no current version")
|
||||
|
||||
rows: list[tuple[FileVersion, FileBlob]] = []
|
||||
for chunk in _chunks(version_ids):
|
||||
rows.extend(
|
||||
session.query(FileVersion, FileBlob)
|
||||
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
|
||||
.filter(FileVersion.id.in_(chunk))
|
||||
.all()
|
||||
)
|
||||
by_version_id = {version.id: (version, blob) for version, blob in rows}
|
||||
result: dict[str, tuple[FileVersion, FileBlob]] = {}
|
||||
for asset in asset_list:
|
||||
version_blob = by_version_id.get(asset.current_version_id or "")
|
||||
if not version_blob:
|
||||
raise FileStorageError("File version not found")
|
||||
result[asset.id] = version_blob
|
||||
return result
|
||||
|
||||
|
||||
def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVersion, FileBlob]:
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
backend = get_storage_backend()
|
||||
try:
|
||||
return backend.get_bytes(blob.storage_key), version, blob
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||
|
||||
|
||||
def read_asset_version_bytes(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
version_id: str,
|
||||
) -> tuple[bytes, FileVersion, FileBlob]:
|
||||
version = session.get(FileVersion, version_id)
|
||||
if version is None or version.file_asset_id != asset.id:
|
||||
raise FileStorageError("File version not found")
|
||||
blob = session.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != asset.tenant_id:
|
||||
raise FileStorageError("File blob not found")
|
||||
backend = get_storage_backend()
|
||||
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||
|
||||
|
||||
def share_file(
|
||||
@@ -283,6 +791,7 @@ def share_file(
|
||||
target_id: str,
|
||||
permission: str,
|
||||
user_id: str,
|
||||
expires_at: datetime | None = None,
|
||||
) -> FileShare:
|
||||
target_type = target_type.lower().strip()
|
||||
permission = permission.lower().strip()
|
||||
@@ -292,23 +801,11 @@ def share_file(
|
||||
raise FileStorageError("Unsupported share target")
|
||||
if permission not in {"read", "write", "manage"}:
|
||||
raise FileStorageError("Unsupported file permission")
|
||||
if target_type == "user":
|
||||
target_user = session.get(User, target_id)
|
||||
if not target_user or target_user.tenant_id != tenant_id or not target_user.is_active:
|
||||
raise FileStorageError("User not found")
|
||||
if target_type == "group":
|
||||
group = session.get(Group, target_id)
|
||||
if not group or group.tenant_id != tenant_id or not group.is_active:
|
||||
raise FileStorageError("Group not found")
|
||||
if target_type == "tenant":
|
||||
tenant = session.get(Tenant, target_id)
|
||||
if target_id != tenant_id or not tenant or not tenant.is_active:
|
||||
raise FileStorageError("Tenant not found")
|
||||
_validate_share_expiry(expires_at)
|
||||
if target_type in {"user", "group", "tenant"}:
|
||||
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
|
||||
if target_type == "campaign":
|
||||
Campaign = _campaign_model()
|
||||
campaign = session.get(Campaign, target_id)
|
||||
if not campaign or campaign.tenant_id != tenant_id:
|
||||
raise FileStorageError("Campaign not found")
|
||||
_ensure_campaign_exists(session, tenant_id=tenant_id, campaign_id=target_id)
|
||||
existing = (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
@@ -322,6 +819,7 @@ def share_file(
|
||||
)
|
||||
if existing:
|
||||
existing.permission = permission
|
||||
existing.expires_at = expires_at
|
||||
session.add(existing)
|
||||
return existing
|
||||
share = FileShare(
|
||||
@@ -331,6 +829,7 @@ def share_file(
|
||||
target_id=target_id,
|
||||
permission=permission,
|
||||
created_by_user_id=user_id,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
session.add(share)
|
||||
return share
|
||||
@@ -345,6 +844,7 @@ def share_files(
|
||||
target_id: str,
|
||||
permission: str,
|
||||
user_id: str,
|
||||
expires_at: datetime | None = None,
|
||||
) -> list[FileShare]:
|
||||
target_type = target_type.lower().strip()
|
||||
permission = permission.lower().strip()
|
||||
@@ -352,23 +852,11 @@ def share_files(
|
||||
raise FileStorageError("Unsupported share target")
|
||||
if permission not in {"read", "write", "manage"}:
|
||||
raise FileStorageError("Unsupported file permission")
|
||||
if target_type == "user":
|
||||
target_user = session.get(User, target_id)
|
||||
if not target_user or target_user.tenant_id != tenant_id or not target_user.is_active:
|
||||
raise FileStorageError("User not found")
|
||||
if target_type == "group":
|
||||
group = session.get(Group, target_id)
|
||||
if not group or group.tenant_id != tenant_id or not group.is_active:
|
||||
raise FileStorageError("Group not found")
|
||||
if target_type == "tenant":
|
||||
tenant = session.get(Tenant, target_id)
|
||||
if target_id != tenant_id or not tenant or not tenant.is_active:
|
||||
raise FileStorageError("Tenant not found")
|
||||
_validate_share_expiry(expires_at)
|
||||
if target_type in {"user", "group", "tenant"}:
|
||||
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
|
||||
if target_type == "campaign":
|
||||
Campaign = _campaign_model()
|
||||
campaign = session.get(Campaign, target_id)
|
||||
if not campaign or campaign.tenant_id != tenant_id:
|
||||
raise FileStorageError("Campaign not found")
|
||||
_ensure_campaign_exists(session, tenant_id=tenant_id, campaign_id=target_id)
|
||||
|
||||
asset_list = list(assets)
|
||||
if not asset_list:
|
||||
@@ -392,6 +880,7 @@ def share_files(
|
||||
existing = existing_by_asset.get(asset.id)
|
||||
if existing:
|
||||
existing.permission = permission
|
||||
existing.expires_at = expires_at
|
||||
session.add(existing)
|
||||
shares.append(existing)
|
||||
continue
|
||||
@@ -402,12 +891,88 @@ def share_files(
|
||||
target_id=target_id,
|
||||
permission=permission,
|
||||
created_by_user_id=user_id,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
session.add(share)
|
||||
shares.append(share)
|
||||
return shares
|
||||
|
||||
|
||||
def list_file_shares(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
asset_id: str,
|
||||
include_inactive: bool = False,
|
||||
) -> list[FileShare]:
|
||||
query = session.query(FileShare).filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset_id,
|
||||
)
|
||||
if not include_inactive:
|
||||
query = query.filter(effective_file_share_clause())
|
||||
return query.order_by(FileShare.created_at.desc(), FileShare.id.desc()).all()
|
||||
|
||||
|
||||
def revoke_file_share(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
asset_id: str,
|
||||
share_id: str,
|
||||
user_id: str,
|
||||
) -> tuple[FileShare, bool]:
|
||||
share = (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
FileShare.id == share_id,
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if share is None:
|
||||
raise FileStorageError("File share not found")
|
||||
if share.revoked_at is not None:
|
||||
return share, False
|
||||
share.revoked_at = utcnow()
|
||||
share.revoked_by_user_id = user_id
|
||||
session.add(share)
|
||||
return share, True
|
||||
|
||||
|
||||
def current_file_share_for_target(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
asset_id: str,
|
||||
target_type: str,
|
||||
target_id: str,
|
||||
) -> FileShare | None:
|
||||
return (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset_id,
|
||||
FileShare.target_type == target_type.lower().strip(),
|
||||
FileShare.target_id == target_id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
)
|
||||
.order_by(FileShare.created_at.desc(), FileShare.id.desc())
|
||||
.first()
|
||||
)
|
||||
|
||||
|
||||
def _validate_share_expiry(expires_at: datetime | None) -> None:
|
||||
if expires_at is None:
|
||||
return
|
||||
from govoplan_files.backend.storage.share_state import file_share_is_active
|
||||
|
||||
candidate = FileShare(expires_at=expires_at)
|
||||
if not file_share_is_active(candidate):
|
||||
raise FileStorageError("File share expiry must be in the future")
|
||||
|
||||
|
||||
|
||||
def soft_delete_assets(session: Session, assets: Iterable[FileAsset]) -> int:
|
||||
count = 0
|
||||
@@ -437,6 +1002,11 @@ def _asset_owner_id(asset: FileAsset) -> str:
|
||||
raise FileStorageError("File has no valid owner")
|
||||
|
||||
|
||||
def _chunks(values: list[str], size: int = 900):
|
||||
for index in range(0, len(values), size):
|
||||
yield values[index:index + size]
|
||||
|
||||
|
||||
def _active_asset_exists(session: Session, *, tenant_id: str, owner_type: str, owner_id: str, path: str, exclude_asset_id: str | None = None) -> bool:
|
||||
return _active_asset_at_path(
|
||||
session,
|
||||
@@ -538,11 +1108,44 @@ def _resolution_by_path(conflict_resolutions: Iterable[FileConflictResolution] |
|
||||
|
||||
def _normalize_conflict_strategy(strategy: str | None) -> str:
|
||||
normalized = (strategy or "reject").lower().strip()
|
||||
if normalized not in {"reject", "overwrite", "rename"}:
|
||||
if normalized not in {"reject", "overwrite", "rename", "skip"}:
|
||||
raise FileStorageError("Unsupported conflict strategy")
|
||||
return normalized
|
||||
|
||||
|
||||
def _next_version_number(session: Session, asset_id: str) -> int:
|
||||
row = (
|
||||
session.query(FileVersion.version_number)
|
||||
.filter(FileVersion.file_asset_id == asset_id)
|
||||
.order_by(FileVersion.version_number.desc())
|
||||
.first()
|
||||
)
|
||||
return (int(row[0]) if row else 0) + 1
|
||||
|
||||
|
||||
def _source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||
if not provenance:
|
||||
return None
|
||||
connector_id = _clean_identity(provenance.get("connector_id"))
|
||||
provider = _clean_identity(provenance.get("provider"))
|
||||
external_id = _clean_identity(provenance.get("external_id"))
|
||||
if connector_id and external_id:
|
||||
return ("external_id", connector_id, provider, external_id)
|
||||
external_path = _clean_identity(provenance.get("external_path"))
|
||||
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||
library_id = _clean_identity(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||
if connector_id and external_path:
|
||||
return ("external_path", connector_id, provider, library_id, external_path)
|
||||
return None
|
||||
|
||||
|
||||
def _clean_identity(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _copy_asset_to_path(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
|
||||
@@ -68,13 +68,63 @@ def list_folders_for_user(
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileFolder]:
|
||||
query = _folder_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return query.order_by(FileFolder.path.asc(), FileFolder.id.asc()).all()
|
||||
|
||||
|
||||
def list_folders_for_user_window(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
page_size: int,
|
||||
after_path: str | None = None,
|
||||
after_id: str | None = None,
|
||||
) -> tuple[list[FileFolder], bool]:
|
||||
query = _folder_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if after_path is not None and after_id:
|
||||
query = query.filter(or_(FileFolder.path > after_path, (FileFolder.path == after_path) & (FileFolder.id > after_id)))
|
||||
rows = query.order_by(FileFolder.path.asc(), FileFolder.id.asc()).limit(page_size + 1).all()
|
||||
return rows[:page_size], len(rows) > page_size
|
||||
|
||||
|
||||
def _folder_visibility_query_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
):
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
query = session.query(FileFolder).filter(FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type)
|
||||
query = _owner_filter(query, owner_type, owner_id)
|
||||
if not include_deleted:
|
||||
query = query.filter(FileFolder.deleted_at.is_(None))
|
||||
return query.order_by(FileFolder.path.asc()).all()
|
||||
return query
|
||||
|
||||
|
||||
def soft_delete_folder(
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import ssl
|
||||
import select
|
||||
from collections.abc import Mapping
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterator
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
bounded_chunks_bytes,
|
||||
create_outbound_connection,
|
||||
response_limit,
|
||||
validate_outbound_http_url,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorHttpError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorHttpResponse:
|
||||
status_code: int
|
||||
headers: Mapping[str, str]
|
||||
content: bytes
|
||||
|
||||
|
||||
_HTTPCORE_TRANSPORT_ERRORS = (
|
||||
httpcore.TimeoutException,
|
||||
httpcore.NetworkError,
|
||||
httpcore.ProtocolError,
|
||||
httpcore.ProxyError,
|
||||
httpcore.UnsupportedProtocol,
|
||||
)
|
||||
|
||||
|
||||
class _SocketNetworkStream(httpcore.NetworkStream):
|
||||
"""Public httpcore NetworkStream adapter around an approved socket."""
|
||||
|
||||
def __init__(self, sock: socket.socket) -> None:
|
||||
self._socket = sock
|
||||
|
||||
def read(self, max_bytes: int, timeout: float | None = None) -> bytes:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
return self._socket.recv(max_bytes)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.ReadTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
raise httpcore.ReadError(str(exc)) from exc
|
||||
|
||||
def write(self, buffer: bytes, timeout: float | None = None) -> None:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
self._socket.sendall(buffer)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.WriteTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
raise httpcore.WriteError(str(exc)) from exc
|
||||
|
||||
def close(self) -> None:
|
||||
self._socket.close()
|
||||
|
||||
def start_tls(
|
||||
self,
|
||||
ssl_context: ssl.SSLContext,
|
||||
server_hostname: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> httpcore.NetworkStream:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
tls_socket = ssl_context.wrap_socket(self._socket, server_hostname=server_hostname)
|
||||
except socket.timeout as exc:
|
||||
self.close()
|
||||
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
self.close()
|
||||
raise httpcore.ConnectError(str(exc)) from exc
|
||||
return _SocketNetworkStream(tls_socket)
|
||||
|
||||
def get_extra_info(self, info: str) -> Any:
|
||||
if info == "ssl_object" and isinstance(self._socket, ssl.SSLSocket):
|
||||
return self._socket
|
||||
if info == "client_addr":
|
||||
return self._socket.getsockname()
|
||||
if info == "server_addr":
|
||||
return self._socket.getpeername()
|
||||
if info == "socket":
|
||||
return self._socket
|
||||
if info == "is_readable":
|
||||
try:
|
||||
return bool(select.select([self._socket], [], [], 0)[0])
|
||||
except (OSError, ValueError):
|
||||
return True
|
||||
return None
|
||||
|
||||
|
||||
class _OutboundPolicyNetworkBackend(httpcore.NetworkBackend):
|
||||
def connect_tcp(
|
||||
self,
|
||||
host: str,
|
||||
port: int,
|
||||
timeout: float | None = None,
|
||||
local_address: str | None = None,
|
||||
socket_options: Any = None,
|
||||
) -> httpcore.NetworkStream:
|
||||
source_address = None if local_address is None else (local_address, 0)
|
||||
try:
|
||||
sock = create_outbound_connection(
|
||||
host,
|
||||
port,
|
||||
timeout=timeout,
|
||||
source_address=source_address,
|
||||
socket_options=socket_options,
|
||||
label="File connector HTTP endpoint",
|
||||
)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||
except (OSError, OutboundHttpError) as exc:
|
||||
raise httpcore.ConnectError(str(exc)) from exc
|
||||
return _SocketNetworkStream(sock)
|
||||
|
||||
def connect_unix_socket(self, path: str, timeout: float | None = None, socket_options: Any = None): # type: ignore[no-untyped-def]
|
||||
del path, timeout, socket_options
|
||||
raise httpcore.ConnectError("Unix sockets are not supported for file connectors")
|
||||
|
||||
|
||||
class _HttpcoreResponseStream(httpx.SyncByteStream):
|
||||
def __init__(self, stream: Any, *, request: httpx.Request) -> None:
|
||||
self._stream = stream
|
||||
self._request = request
|
||||
|
||||
def __iter__(self): # type: ignore[no-untyped-def]
|
||||
try:
|
||||
yield from self._stream
|
||||
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||
raise httpx.TransportError(str(exc), request=self._request) from exc
|
||||
|
||||
def close(self) -> None:
|
||||
if hasattr(self._stream, "close"):
|
||||
self._stream.close()
|
||||
|
||||
|
||||
class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
|
||||
def __init__(self) -> None:
|
||||
self._connection_pool = httpcore.ConnectionPool(
|
||||
ssl_context=httpx.create_ssl_context(verify=True, trust_env=False),
|
||||
max_connections=100,
|
||||
max_keepalive_connections=20,
|
||||
keepalive_expiry=5.0,
|
||||
network_backend=_OutboundPolicyNetworkBackend(),
|
||||
)
|
||||
|
||||
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||
core_request = httpcore.Request(
|
||||
method=request.method,
|
||||
url=httpcore.URL(
|
||||
scheme=request.url.raw_scheme,
|
||||
host=request.url.raw_host,
|
||||
port=request.url.port,
|
||||
target=request.url.raw_path,
|
||||
),
|
||||
headers=request.headers.raw,
|
||||
content=request.stream,
|
||||
extensions=request.extensions,
|
||||
)
|
||||
try:
|
||||
response = self._connection_pool.handle_request(core_request)
|
||||
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||
raise httpx.TransportError(str(exc), request=request) from exc
|
||||
return httpx.Response(
|
||||
status_code=response.status,
|
||||
headers=response.headers,
|
||||
stream=_HttpcoreResponseStream(response.stream, request=request),
|
||||
extensions=response.extensions,
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
self._connection_pool.close()
|
||||
|
||||
|
||||
_CONNECTOR_HTTP_CLIENT = httpx.Client(
|
||||
transport=_OutboundPolicyHTTPTransport(),
|
||||
follow_redirects=False,
|
||||
timeout=15.0,
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
|
||||
timeout = kwargs.pop("timeout", 15.0)
|
||||
with _CONNECTOR_HTTP_CLIENT.stream(
|
||||
method,
|
||||
url,
|
||||
timeout=timeout,
|
||||
**kwargs,
|
||||
) as response:
|
||||
yield response
|
||||
|
||||
|
||||
def request_connector_bytes(
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
headers: Mapping[str, str] | None = None,
|
||||
params: Mapping[str, str] | None = None,
|
||||
data: Mapping[str, str] | bytes | str | None = None,
|
||||
content: bytes | str | None = None,
|
||||
auth: Any = None,
|
||||
timeout: float = 15.0,
|
||||
kind: str = "structured",
|
||||
max_bytes: int | None = None,
|
||||
label: str = "File connector",
|
||||
) -> ConnectorHttpResponse:
|
||||
try:
|
||||
validated_url = validate_outbound_http_url(url, label=f"{label} URL")
|
||||
effective_limit = response_limit(kind) if max_bytes is None else min(int(max_bytes), response_limit(kind))
|
||||
with _stream_connector_request(
|
||||
method,
|
||||
validated_url,
|
||||
headers=dict(headers or {}),
|
||||
params=params,
|
||||
data=data,
|
||||
content=content,
|
||||
auth=auth,
|
||||
timeout=timeout,
|
||||
) as response:
|
||||
body = bounded_chunks_bytes(
|
||||
response.iter_bytes(),
|
||||
headers=response.headers,
|
||||
max_bytes=effective_limit,
|
||||
kind=kind,
|
||||
label=f"{label} response",
|
||||
)
|
||||
return ConnectorHttpResponse(
|
||||
status_code=response.status_code,
|
||||
headers=dict(response.headers),
|
||||
content=body,
|
||||
)
|
||||
except (httpx.HTTPError, OutboundHttpError, ValueError) as exc:
|
||||
raise ConnectorHttpError(str(exc)) from exc
|
||||
@@ -0,0 +1,550 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy.orm import Session, object_session
|
||||
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileBlob,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
)
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.recovery import (
|
||||
begin_orphan_cleanup_recovery,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
|
||||
|
||||
TERMINAL_SCAN_STATUSES = {"completed", "cancelled"}
|
||||
QUARANTINED_BLOB_STATUSES = {
|
||||
"missing",
|
||||
"size_mismatch",
|
||||
"checksum_mismatch",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BlobInspection:
|
||||
valid: bool
|
||||
kind: str
|
||||
observed_size_bytes: int | None = None
|
||||
observed_checksum_sha256: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class IntegrityActionResult:
|
||||
action: str
|
||||
changed: bool
|
||||
dry_run: bool
|
||||
finding: FileIntegrityFinding
|
||||
inspection: BlobInspection | None = None
|
||||
|
||||
|
||||
def storage_prefix_for_tenant(tenant_id: str) -> str:
|
||||
return f"tenants/{tenant_id}/files/"
|
||||
|
||||
|
||||
def create_integrity_scan(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
verify_checksums: bool = True,
|
||||
batch_size: int = 100,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> FileIntegrityScan:
|
||||
active_backend = backend or get_storage_backend()
|
||||
scan = FileIntegrityScan(
|
||||
tenant_id=tenant_id,
|
||||
storage_backend=active_backend.name,
|
||||
storage_prefix=storage_prefix_for_tenant(tenant_id),
|
||||
verify_checksums=verify_checksums,
|
||||
batch_size=max(1, min(int(batch_size), 1000)),
|
||||
created_by_user_id=user_id,
|
||||
)
|
||||
session.add(scan)
|
||||
session.flush()
|
||||
return scan
|
||||
|
||||
|
||||
def run_integrity_scan_batch(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
*,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> FileIntegrityScan:
|
||||
if scan.status in TERMINAL_SCAN_STATUSES:
|
||||
return scan
|
||||
active_backend = backend or get_storage_backend()
|
||||
if active_backend.name != scan.storage_backend:
|
||||
raise FileStorageError(
|
||||
"The configured storage backend changed after this integrity scan started"
|
||||
)
|
||||
if scan.started_at is None:
|
||||
scan.started_at = utcnow()
|
||||
scan.status = "running"
|
||||
scan.last_error = None
|
||||
if scan.phase == "blobs":
|
||||
_scan_blob_batch(session, scan, active_backend)
|
||||
elif scan.phase == "objects":
|
||||
_scan_object_batch(session, scan, active_backend)
|
||||
else:
|
||||
scan.phase = "completed"
|
||||
scan.status = "completed"
|
||||
scan.completed_at = utcnow()
|
||||
scan.revision += 1
|
||||
session.add(scan)
|
||||
return scan
|
||||
|
||||
|
||||
def mark_integrity_scan_failed(
|
||||
scan: FileIntegrityScan,
|
||||
*,
|
||||
error: Exception,
|
||||
) -> None:
|
||||
scan.status = "failed"
|
||||
scan.last_error = type(error).__name__[:255]
|
||||
scan.revision += 1
|
||||
|
||||
|
||||
def inspect_blob(
|
||||
blob: FileBlob,
|
||||
*,
|
||||
backend: StorageBackend,
|
||||
verify_checksum: bool = True,
|
||||
) -> BlobInspection:
|
||||
expected_size = blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes
|
||||
expected_checksum = blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256
|
||||
try:
|
||||
info = backend.stat(blob.storage_key)
|
||||
except StorageObjectMissing:
|
||||
return BlobInspection(valid=False, kind="missing")
|
||||
if info.size_bytes != expected_size:
|
||||
return BlobInspection(
|
||||
valid=False,
|
||||
kind="size_mismatch",
|
||||
observed_size_bytes=info.size_bytes,
|
||||
)
|
||||
if not verify_checksum:
|
||||
return BlobInspection(
|
||||
valid=True,
|
||||
kind="verified",
|
||||
observed_size_bytes=info.size_bytes,
|
||||
)
|
||||
digest = hashlib.sha256()
|
||||
observed_size = 0
|
||||
for chunk in backend.iter_bytes(blob.storage_key):
|
||||
observed_size += len(chunk)
|
||||
digest.update(chunk)
|
||||
observed_checksum = digest.hexdigest()
|
||||
if observed_size != expected_size:
|
||||
return BlobInspection(
|
||||
valid=False,
|
||||
kind="size_mismatch",
|
||||
observed_size_bytes=observed_size,
|
||||
observed_checksum_sha256=observed_checksum,
|
||||
)
|
||||
if observed_checksum != expected_checksum:
|
||||
return BlobInspection(
|
||||
valid=False,
|
||||
kind="checksum_mismatch",
|
||||
observed_size_bytes=observed_size,
|
||||
observed_checksum_sha256=observed_checksum,
|
||||
)
|
||||
return BlobInspection(
|
||||
valid=True,
|
||||
kind="verified",
|
||||
observed_size_bytes=observed_size,
|
||||
observed_checksum_sha256=observed_checksum,
|
||||
)
|
||||
|
||||
|
||||
def apply_blob_inspection(
|
||||
session: Session,
|
||||
blob: FileBlob,
|
||||
inspection: BlobInspection,
|
||||
*,
|
||||
checked_at=None,
|
||||
) -> None:
|
||||
timestamp = checked_at or utcnow()
|
||||
blob.integrity_checked_at = timestamp
|
||||
if inspection.valid:
|
||||
blob.integrity_status = "verified"
|
||||
blob.integrity_failure = None
|
||||
blob.quarantined_at = None
|
||||
else:
|
||||
blob.integrity_status = inspection.kind
|
||||
blob.integrity_failure = inspection.kind
|
||||
blob.quarantined_at = blob.quarantined_at or timestamp
|
||||
session.add(blob)
|
||||
|
||||
|
||||
def ensure_blob_is_readable(blob: FileBlob) -> None:
|
||||
if blob.integrity_status in QUARANTINED_BLOB_STATUSES or blob.quarantined_at:
|
||||
raise FileStorageError(
|
||||
"Managed file content is quarantined because integrity verification failed"
|
||||
)
|
||||
|
||||
|
||||
def read_verified_blob_bytes(
|
||||
blob: FileBlob,
|
||||
*,
|
||||
backend: StorageBackend,
|
||||
) -> bytes:
|
||||
ensure_blob_is_readable(blob)
|
||||
try:
|
||||
data = backend.get_bytes(blob.storage_key)
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError("Managed file content is not available") from exc
|
||||
expected_storage_size = blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes
|
||||
expected_storage_checksum = blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256
|
||||
if len(data) != expected_storage_size:
|
||||
raise FileStorageError(
|
||||
"Managed file content failed its recorded size verification"
|
||||
)
|
||||
if hashlib.sha256(data).hexdigest() != expected_storage_checksum:
|
||||
raise FileStorageError(
|
||||
"Managed file content failed its recorded checksum verification"
|
||||
)
|
||||
if blob.encryption_envelope_id:
|
||||
session = object_session(blob)
|
||||
if session is None:
|
||||
raise FileStorageError("Encrypted managed content requires an attached database session")
|
||||
from govoplan_files.backend.storage.content_protection import unprotect_blob_content
|
||||
|
||||
data = unprotect_blob_content(
|
||||
session,
|
||||
tenant_id=blob.tenant_id,
|
||||
blob_id=blob.id,
|
||||
envelope_id=blob.encryption_envelope_id,
|
||||
ciphertext=data,
|
||||
)
|
||||
if len(data) != blob.size_bytes:
|
||||
raise FileStorageError("Decrypted managed file content failed its recorded size verification")
|
||||
if hashlib.sha256(data).hexdigest() != blob.checksum_sha256:
|
||||
raise FileStorageError("Decrypted managed file content failed its recorded checksum verification")
|
||||
return data
|
||||
|
||||
|
||||
def recheck_integrity_finding(
|
||||
session: Session,
|
||||
finding: FileIntegrityFinding,
|
||||
*,
|
||||
user_id: str,
|
||||
dry_run: bool,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> IntegrityActionResult:
|
||||
if finding.kind == "orphan_object" or not finding.blob_id:
|
||||
raise FileStorageError("Only blob integrity findings can be rechecked")
|
||||
blob = session.get(FileBlob, finding.blob_id)
|
||||
if blob is None or blob.tenant_id != finding.tenant_id:
|
||||
raise FileStorageError("Integrity finding blob no longer exists")
|
||||
active_backend = backend or get_storage_backend()
|
||||
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||
if scan is None or scan.storage_backend != active_backend.name:
|
||||
raise FileStorageError(
|
||||
"The configured storage backend does not match the integrity finding"
|
||||
)
|
||||
inspection = inspect_blob(
|
||||
blob,
|
||||
backend=active_backend,
|
||||
verify_checksum=True,
|
||||
)
|
||||
changed = False
|
||||
if not dry_run:
|
||||
previous = (
|
||||
blob.integrity_status,
|
||||
blob.quarantined_at,
|
||||
finding.state,
|
||||
)
|
||||
apply_blob_inspection(session, blob, inspection)
|
||||
_update_finding_from_inspection(finding, inspection)
|
||||
if inspection.valid:
|
||||
finding.state = "resolved"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.resolved_by_user_id = user_id
|
||||
session.add(finding)
|
||||
finding.revision += 1
|
||||
changed = previous != (
|
||||
blob.integrity_status,
|
||||
blob.quarantined_at,
|
||||
finding.state,
|
||||
)
|
||||
return IntegrityActionResult(
|
||||
action="recheck",
|
||||
changed=changed,
|
||||
dry_run=dry_run,
|
||||
finding=finding,
|
||||
inspection=inspection,
|
||||
)
|
||||
|
||||
|
||||
def cleanup_orphan_finding(
|
||||
session: Session,
|
||||
finding: FileIntegrityFinding,
|
||||
*,
|
||||
user_id: str,
|
||||
dry_run: bool,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> IntegrityActionResult:
|
||||
if finding.kind != "orphan_object":
|
||||
raise FileStorageError("Only orphan-object findings can be cleaned up")
|
||||
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||
if scan is None or scan.tenant_id != finding.tenant_id:
|
||||
raise FileStorageError("Integrity scan not found")
|
||||
if not finding.storage_key.startswith(scan.storage_prefix):
|
||||
raise FileStorageError("Orphan cleanup is outside the scan storage scope")
|
||||
referenced = (
|
||||
session.query(FileBlob.id)
|
||||
.filter(
|
||||
FileBlob.tenant_id == finding.tenant_id,
|
||||
FileBlob.storage_backend == scan.storage_backend,
|
||||
FileBlob.storage_key == finding.storage_key,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if referenced:
|
||||
if not dry_run and finding.state != "resolved":
|
||||
finding.state = "resolved"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.resolved_by_user_id = user_id
|
||||
finding.revision += 1
|
||||
session.add(finding)
|
||||
return IntegrityActionResult(
|
||||
action="retained_referenced",
|
||||
changed=True,
|
||||
dry_run=False,
|
||||
finding=finding,
|
||||
)
|
||||
return IntegrityActionResult(
|
||||
action="retained_referenced",
|
||||
changed=False,
|
||||
dry_run=dry_run,
|
||||
finding=finding,
|
||||
)
|
||||
if finding.state == "deleted":
|
||||
return IntegrityActionResult(
|
||||
action="already_deleted",
|
||||
changed=False,
|
||||
dry_run=dry_run,
|
||||
finding=finding,
|
||||
)
|
||||
if dry_run:
|
||||
return IntegrityActionResult(
|
||||
action="would_delete",
|
||||
changed=False,
|
||||
dry_run=True,
|
||||
finding=finding,
|
||||
)
|
||||
active_backend = backend or get_storage_backend()
|
||||
if active_backend.name != scan.storage_backend:
|
||||
raise FileStorageError(
|
||||
"The configured storage backend does not match the integrity finding"
|
||||
)
|
||||
begin_orphan_cleanup_recovery(
|
||||
session,
|
||||
finding,
|
||||
backend=active_backend,
|
||||
user_id=user_id,
|
||||
)
|
||||
try:
|
||||
active_backend.stat(finding.storage_key)
|
||||
except StorageObjectMissing:
|
||||
action = "already_absent"
|
||||
else:
|
||||
active_backend.delete(finding.storage_key)
|
||||
action = "deleted"
|
||||
finding.state = "deleted"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.resolved_by_user_id = user_id
|
||||
finding.revision += 1
|
||||
session.add(finding)
|
||||
return IntegrityActionResult(
|
||||
action=action,
|
||||
changed=True,
|
||||
dry_run=False,
|
||||
finding=finding,
|
||||
)
|
||||
|
||||
|
||||
def _scan_blob_batch(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
backend: StorageBackend,
|
||||
) -> None:
|
||||
query = session.query(FileBlob).filter(
|
||||
FileBlob.tenant_id == scan.tenant_id,
|
||||
FileBlob.storage_backend == scan.storage_backend,
|
||||
)
|
||||
if scan.blob_cursor:
|
||||
query = query.filter(FileBlob.id > scan.blob_cursor)
|
||||
blobs = query.order_by(FileBlob.id.asc()).limit(scan.batch_size).all()
|
||||
for blob in blobs:
|
||||
inspection = inspect_blob(
|
||||
blob,
|
||||
backend=backend,
|
||||
verify_checksum=scan.verify_checksums,
|
||||
)
|
||||
apply_blob_inspection(session, blob, inspection)
|
||||
scan.scanned_blob_count += 1
|
||||
if inspection.valid:
|
||||
scan.verified_blob_count += 1
|
||||
_resolve_scan_blob_findings(session, scan, blob)
|
||||
else:
|
||||
scan.quarantined_blob_count += 1
|
||||
_record_blob_finding(session, scan, blob, inspection)
|
||||
scan.blob_cursor = blob.id
|
||||
if len(blobs) < scan.batch_size:
|
||||
scan.phase = "objects"
|
||||
scan.object_cursor = None
|
||||
|
||||
|
||||
def _scan_object_batch(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
backend: StorageBackend,
|
||||
) -> None:
|
||||
page = backend.list_objects(
|
||||
prefix=scan.storage_prefix,
|
||||
after=scan.object_cursor,
|
||||
limit=scan.batch_size,
|
||||
)
|
||||
keys = [item.key for item in page.objects]
|
||||
referenced_keys = {
|
||||
row[0]
|
||||
for row in session.query(FileBlob.storage_key)
|
||||
.filter(
|
||||
FileBlob.tenant_id == scan.tenant_id,
|
||||
FileBlob.storage_backend == scan.storage_backend,
|
||||
FileBlob.storage_key.in_(keys),
|
||||
)
|
||||
.all()
|
||||
} if keys else set()
|
||||
for item in page.objects:
|
||||
scan.scanned_object_count += 1
|
||||
if item.key in referenced_keys:
|
||||
continue
|
||||
scan.orphan_object_count += 1
|
||||
_record_orphan_finding(
|
||||
session,
|
||||
scan,
|
||||
storage_key=item.key,
|
||||
size_bytes=item.size_bytes,
|
||||
)
|
||||
scan.object_cursor = page.next_cursor
|
||||
if page.next_cursor is None:
|
||||
scan.phase = "completed"
|
||||
scan.status = "completed"
|
||||
scan.completed_at = utcnow()
|
||||
|
||||
|
||||
def _record_blob_finding(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
blob: FileBlob,
|
||||
inspection: BlobInspection,
|
||||
) -> FileIntegrityFinding:
|
||||
finding = (
|
||||
session.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.blob_id == blob.id,
|
||||
FileIntegrityFinding.kind == inspection.kind,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if finding is None:
|
||||
finding = FileIntegrityFinding(
|
||||
scan_id=scan.id,
|
||||
tenant_id=scan.tenant_id,
|
||||
kind=inspection.kind,
|
||||
blob_id=blob.id,
|
||||
storage_key=blob.storage_key,
|
||||
expected_size_bytes=blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes,
|
||||
expected_checksum_sha256=blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256,
|
||||
)
|
||||
else:
|
||||
finding.revision += 1
|
||||
_update_finding_from_inspection(finding, inspection)
|
||||
session.add(finding)
|
||||
return finding
|
||||
|
||||
|
||||
def _record_orphan_finding(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
*,
|
||||
storage_key: str,
|
||||
size_bytes: int,
|
||||
) -> FileIntegrityFinding:
|
||||
finding = (
|
||||
session.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.kind == "orphan_object",
|
||||
FileIntegrityFinding.storage_key == storage_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if finding is None:
|
||||
finding = FileIntegrityFinding(
|
||||
scan_id=scan.id,
|
||||
tenant_id=scan.tenant_id,
|
||||
kind="orphan_object",
|
||||
storage_key=storage_key,
|
||||
observed_size_bytes=size_bytes,
|
||||
)
|
||||
session.add(finding)
|
||||
return finding
|
||||
|
||||
|
||||
def _resolve_scan_blob_findings(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
blob: FileBlob,
|
||||
) -> None:
|
||||
for finding in (
|
||||
session.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.blob_id == blob.id,
|
||||
FileIntegrityFinding.state == "open",
|
||||
)
|
||||
.all()
|
||||
):
|
||||
finding.state = "resolved"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.revision += 1
|
||||
session.add(finding)
|
||||
|
||||
|
||||
def _update_finding_from_inspection(
|
||||
finding: FileIntegrityFinding,
|
||||
inspection: BlobInspection,
|
||||
) -> None:
|
||||
finding.observed_size_bytes = inspection.observed_size_bytes
|
||||
finding.observed_checksum_sha256 = inspection.observed_checksum_sha256
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BlobInspection",
|
||||
"IntegrityActionResult",
|
||||
"QUARANTINED_BLOB_STATUSES",
|
||||
"apply_blob_inspection",
|
||||
"cleanup_orphan_finding",
|
||||
"create_integrity_scan",
|
||||
"ensure_blob_is_readable",
|
||||
"inspect_blob",
|
||||
"mark_integrity_scan_failed",
|
||||
"read_verified_blob_bytes",
|
||||
"recheck_integrity_finding",
|
||||
"run_integrity_scan_batch",
|
||||
"storage_prefix_for_tenant",
|
||||
]
|
||||
@@ -0,0 +1,796 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Callable, Iterable
|
||||
|
||||
from sqlalchemy import func, inspect, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.connector_spaces import connector_space_owner_id
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgePreviewItem:
|
||||
file_id: str
|
||||
filename: str
|
||||
lifecycle_revision: int
|
||||
deleted_at: datetime | None
|
||||
retained_until: datetime | None
|
||||
legal_hold: bool
|
||||
blockers: tuple[str, ...]
|
||||
blob_ids: tuple[str, ...]
|
||||
|
||||
def digest_payload(self) -> dict[str, object]:
|
||||
return {
|
||||
"file_id": self.file_id,
|
||||
"filename": self.filename,
|
||||
"lifecycle_revision": self.lifecycle_revision,
|
||||
"deleted_at": _iso(self.deleted_at),
|
||||
"retained_until": _iso(self.retained_until),
|
||||
"legal_hold": self.legal_hold,
|
||||
"blockers": list(self.blockers),
|
||||
"blob_ids": list(self.blob_ids),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgePreview:
|
||||
preview_sha256: str
|
||||
items: tuple[PurgePreviewItem, ...]
|
||||
|
||||
@property
|
||||
def eligible(self) -> bool:
|
||||
return all(not item.blockers for item in self.items)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgeResult:
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool
|
||||
purged_files: int
|
||||
released_blobs: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BlobGcResult:
|
||||
inspected_blobs: int
|
||||
deleted_blobs: int
|
||||
unresolved_operation_ids: tuple[str, ...]
|
||||
|
||||
|
||||
def set_asset_lifecycle(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
retained_until: datetime | None,
|
||||
legal_hold: bool,
|
||||
reason: str,
|
||||
expected_revision: int,
|
||||
) -> FileAsset:
|
||||
if asset.lifecycle_revision != expected_revision:
|
||||
raise FileStorageError("File lifecycle settings changed; reload before saving")
|
||||
asset.retained_until = retained_until
|
||||
asset.legal_hold = bool(legal_hold)
|
||||
asset.lifecycle_reason = reason.strip()
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return asset
|
||||
|
||||
|
||||
def get_asset_for_lifecycle(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
asset_id: str,
|
||||
is_admin: bool,
|
||||
) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if asset is None or asset.tenant_id != tenant_id:
|
||||
raise FileStorageError("File not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_asset_owner_id(asset),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return asset
|
||||
|
||||
|
||||
def restore_asset(session: Session, asset: FileAsset) -> bool:
|
||||
if asset.deleted_at is None:
|
||||
return False
|
||||
collision = _asset_owner_query(session, asset).filter(
|
||||
FileAsset.id != asset.id,
|
||||
FileAsset.display_path == asset.display_path,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
"The file cannot be restored because its path is already in use"
|
||||
)
|
||||
asset.deleted_at = None
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return True
|
||||
|
||||
|
||||
def restore_folder(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
path: str,
|
||||
recursive: bool,
|
||||
is_admin: bool,
|
||||
) -> tuple[int, int]:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
normalized = normalize_folder(path)
|
||||
if not normalized:
|
||||
raise FileStorageError("Folder path is required")
|
||||
prefix = f"{normalized}/"
|
||||
folders = _folder_owner_query(
|
||||
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||
).filter(FileFolder.deleted_at.is_not(None))
|
||||
assets = _asset_owner_query_by_values(
|
||||
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||
).filter(FileAsset.deleted_at.is_not(None))
|
||||
if recursive:
|
||||
folders = folders.filter(
|
||||
(FileFolder.path == normalized) | FileFolder.path.like(f"{prefix}%")
|
||||
)
|
||||
assets = assets.filter(FileAsset.display_path.like(f"{prefix}%"))
|
||||
else:
|
||||
folders = folders.filter(FileFolder.path == normalized)
|
||||
assets = assets.filter(False)
|
||||
folder_rows = folders.order_by(FileFolder.path.asc()).all()
|
||||
asset_rows = assets.order_by(FileAsset.display_path.asc()).all()
|
||||
if not folder_rows:
|
||||
raise FileStorageError("Deleted folder not found")
|
||||
for folder in folder_rows:
|
||||
collision = _folder_owner_query(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
).filter(
|
||||
FileFolder.id != folder.id,
|
||||
FileFolder.path == folder.path,
|
||||
FileFolder.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
f"The folder cannot be restored because its path is in use: {folder.path}"
|
||||
)
|
||||
for asset in asset_rows:
|
||||
collision = _asset_owner_query(session, asset).filter(
|
||||
FileAsset.id != asset.id,
|
||||
FileAsset.display_path == asset.display_path,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
f"A file cannot be restored because its path is in use: {asset.display_path}"
|
||||
)
|
||||
for folder in folder_rows:
|
||||
folder.deleted_at = None
|
||||
session.add(folder)
|
||||
for asset in asset_rows:
|
||||
asset.deleted_at = None
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return len(folder_rows), len(asset_rows)
|
||||
|
||||
|
||||
def restore_connector_space(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
space_id: str,
|
||||
is_admin: bool,
|
||||
) -> FileConnectorSpace:
|
||||
space = session.get(FileConnectorSpace, space_id)
|
||||
if space is None or space.tenant_id != tenant_id or space.deleted_at is None:
|
||||
raise FileStorageError("Deleted connector space not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
collision = _connector_space_owner_query(session, space).filter(
|
||||
FileConnectorSpace.id != space.id,
|
||||
FileConnectorSpace.label == space.label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
"The connector space cannot be restored because its label is in use"
|
||||
)
|
||||
space.deleted_at = None
|
||||
space.is_active = True
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def preview_asset_purge(
|
||||
session: Session, *, tenant_id: str, file_ids: Iterable[str]
|
||||
) -> PurgePreview:
|
||||
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||
if not normalized_ids or len(normalized_ids) > 100:
|
||||
raise FileStorageError("Purge preview must contain between 1 and 100 files")
|
||||
assets = (
|
||||
session.query(FileAsset)
|
||||
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||
.order_by(FileAsset.id.asc())
|
||||
.all()
|
||||
)
|
||||
by_id = {asset.id: asset for asset in assets}
|
||||
items: list[PurgePreviewItem] = []
|
||||
for file_id in normalized_ids:
|
||||
asset = by_id.get(file_id)
|
||||
if asset is None:
|
||||
items.append(
|
||||
PurgePreviewItem(
|
||||
file_id=file_id,
|
||||
filename="",
|
||||
lifecycle_revision=0,
|
||||
deleted_at=None,
|
||||
retained_until=None,
|
||||
legal_hold=False,
|
||||
blockers=("not_found",),
|
||||
blob_ids=(),
|
||||
)
|
||||
)
|
||||
continue
|
||||
blockers = _asset_purge_blockers(session, asset)
|
||||
blob_ids = tuple(
|
||||
sorted(
|
||||
row[0]
|
||||
for row in session.query(FileVersion.blob_id)
|
||||
.filter(FileVersion.file_asset_id == asset.id)
|
||||
.distinct()
|
||||
.all()
|
||||
)
|
||||
)
|
||||
items.append(
|
||||
PurgePreviewItem(
|
||||
file_id=asset.id,
|
||||
filename=asset.filename,
|
||||
lifecycle_revision=asset.lifecycle_revision,
|
||||
deleted_at=asset.deleted_at,
|
||||
retained_until=asset.retained_until,
|
||||
legal_hold=asset.legal_hold,
|
||||
blockers=tuple(blockers),
|
||||
blob_ids=blob_ids,
|
||||
)
|
||||
)
|
||||
payload = [item.digest_payload() for item in items]
|
||||
digest = hashlib.sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
).hexdigest()
|
||||
return PurgePreview(preview_sha256=digest, items=tuple(items))
|
||||
|
||||
|
||||
def execute_asset_purge(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
file_ids: Iterable[str],
|
||||
preview_sha256: str,
|
||||
idempotency_key: str,
|
||||
approval_reference: str,
|
||||
before_commit: Callable[[list[FileAsset]], None] | None = None,
|
||||
) -> PurgeResult:
|
||||
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||
if not normalized_ids or len(normalized_ids) > 100:
|
||||
raise FileStorageError("Purge must contain between 1 and 100 files")
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"file_ids": normalized_ids,
|
||||
"preview_sha256": preview_sha256,
|
||||
"approval_reference": approval_reference,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="asset-hard-purge",
|
||||
idempotency_key=f"files-asset-purge:{idempotency_key}",
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.IRREVERSIBLE,
|
||||
preconditions=(
|
||||
"the actor holds the dedicated Files purge permission",
|
||||
"the signed-off preview still matches current lifecycle state",
|
||||
"every target is soft-deleted and free of retention or legal-hold blockers",
|
||||
),
|
||||
verification_steps=(
|
||||
"verify every target FileAsset and FileVersion row is absent",
|
||||
"recalculate retained FileBlob reference counts before later garbage collection",
|
||||
),
|
||||
approval_reference=approval_reference,
|
||||
),
|
||||
precondition_evidence={
|
||||
"preview_sha256": preview_sha256,
|
||||
"target_count": len(normalized_ids),
|
||||
},
|
||||
lease_resource_key=f"files:purge:{tenant_id}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_asset_batch",
|
||||
resource_id=preview_sha256,
|
||||
metadata={"resources": ["postgresql"], "bounded_target_count": len(normalized_ids)},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"Another purge or unresolved retry owns the tenant purge fence"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError("The Files recovery ledger is unavailable; nothing was purged") from exc
|
||||
if started.replayed or started.operation is None:
|
||||
return PurgeResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=started.status,
|
||||
replayed=True,
|
||||
purged_files=0,
|
||||
released_blobs=0,
|
||||
)
|
||||
operation = started.operation
|
||||
try:
|
||||
preview = preview_asset_purge(session, tenant_id=tenant_id, file_ids=normalized_ids)
|
||||
if preview.preview_sha256 != preview_sha256:
|
||||
operation.reject(
|
||||
summary="The purge preview became stale before execution",
|
||||
evidence=_verified_evidence(
|
||||
{"preview_matches": False},
|
||||
expected_preview_sha256=preview_sha256,
|
||||
observed_preview_sha256=preview.preview_sha256,
|
||||
),
|
||||
)
|
||||
raise FileStorageError("The purge preview is stale; preview again")
|
||||
if not preview.eligible:
|
||||
operation.reject(
|
||||
summary="The purge was blocked by current lifecycle policy",
|
||||
evidence=_verified_evidence(
|
||||
{"lifecycle_policy_allows": False},
|
||||
blocked_files=[
|
||||
{"file_id": item.file_id, "blockers": list(item.blockers)}
|
||||
for item in preview.items
|
||||
if item.blockers
|
||||
]
|
||||
),
|
||||
)
|
||||
raise FileStorageError("One or more files are not eligible for purge")
|
||||
assets = (
|
||||
session.query(FileAsset)
|
||||
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||
.with_for_update()
|
||||
.all()
|
||||
)
|
||||
blob_ids = sorted({blob_id for item in preview.items for blob_id in item.blob_ids})
|
||||
if before_commit is not None:
|
||||
before_commit(assets)
|
||||
session.query(FileShare).filter(
|
||||
FileShare.file_asset_id.in_(normalized_ids)
|
||||
).delete(synchronize_session=False)
|
||||
session.query(FileVersion).filter(
|
||||
FileVersion.file_asset_id.in_(normalized_ids)
|
||||
).delete(synchronize_session=False)
|
||||
for asset in assets:
|
||||
session.delete(asset)
|
||||
session.flush()
|
||||
released_blobs = 0
|
||||
for blob_id in blob_ids:
|
||||
references = int(
|
||||
session.query(func.count(FileVersion.id))
|
||||
.filter(FileVersion.blob_id == blob_id)
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
blob = session.get(FileBlob, blob_id)
|
||||
if blob is not None:
|
||||
blob.ref_count = references
|
||||
session.add(blob)
|
||||
if references == 0:
|
||||
released_blobs += 1
|
||||
operation.commit_verified_success(
|
||||
session,
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"assets_absent": True,
|
||||
"versions_absent": True,
|
||||
"blob_reference_counts_recalculated": True,
|
||||
},
|
||||
purged_file_ids=normalized_ids,
|
||||
purged_files=len(assets),
|
||||
released_blobs=released_blobs,
|
||||
preview_sha256=preview_sha256,
|
||||
),
|
||||
)
|
||||
return PurgeResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.SUCCEEDED.value,
|
||||
replayed=False,
|
||||
purged_files=len(assets),
|
||||
released_blobs=released_blobs,
|
||||
)
|
||||
except FileStorageError:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.release_unresolved()
|
||||
raise
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.reject(
|
||||
summary="The purge failed before its database transaction committed",
|
||||
evidence=_verified_evidence(
|
||||
{"database_transaction_committed": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
def garbage_collect_unreferenced_blobs(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
limit: int,
|
||||
approval_reference: str,
|
||||
) -> BlobGcResult:
|
||||
candidate_rows = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.tenant_id == tenant_id)
|
||||
.filter(
|
||||
or_(FileBlob.retained_until.is_(None), FileBlob.retained_until <= utcnow())
|
||||
)
|
||||
.filter(
|
||||
~session.query(FileVersion.id)
|
||||
.filter(FileVersion.blob_id == FileBlob.id)
|
||||
.exists()
|
||||
)
|
||||
.order_by(FileBlob.created_at.asc(), FileBlob.id.asc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
candidates = tuple(
|
||||
(row.id, row.storage_key) for row in candidate_rows
|
||||
)
|
||||
# Close the read snapshot before the independent recovery transaction takes
|
||||
# its durable SQLite/PostgreSQL write lock.
|
||||
session.commit()
|
||||
deleted = 0
|
||||
unresolved: list[str] = []
|
||||
backend = get_storage_backend()
|
||||
for candidate_id, storage_key in candidates:
|
||||
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"blob_id": candidate_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"approval_reference": approval_reference,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="blob-garbage-collection",
|
||||
idempotency_key=(
|
||||
f"files-blob-gc:{candidate_id}:{key_digest[:12]}:"
|
||||
f"{hashlib.sha256(approval_reference.encode('utf-8')).hexdigest()[:12]}"
|
||||
),
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the caller holds dedicated Files purge authority",
|
||||
"a fresh database check shows no FileVersion reference",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"verify the object is absent",
|
||||
"delete the unreferenced FileBlob row after absence is proven",
|
||||
),
|
||||
verification_steps=(
|
||||
"recheck FileVersion references while holding the blob fence",
|
||||
"probe the exact managed storage key after deletion",
|
||||
),
|
||||
approval_reference=approval_reference,
|
||||
),
|
||||
precondition_evidence={
|
||||
"blob_id": candidate_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"observed_reference_count": 0,
|
||||
},
|
||||
lease_resource_key=f"files:blob:{tenant_id}:{candidate_id}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=candidate_id,
|
||||
metadata={"resources": ["postgresql", "object-storage"], "storage_backend": backend.name},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict):
|
||||
continue
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no blob bytes were deleted"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
continue
|
||||
operation = started.operation
|
||||
try:
|
||||
blob = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.id == candidate_id, FileBlob.tenant_id == tenant_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
references = int(
|
||||
session.query(func.count(FileVersion.id))
|
||||
.filter(FileVersion.blob_id == candidate_id)
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
blob_retained = bool(
|
||||
blob is not None
|
||||
and blob.retained_until is not None
|
||||
and _as_utc(blob.retained_until) > utcnow()
|
||||
)
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.reject(
|
||||
summary="Blob eligibility could not be rechecked before deletion",
|
||||
evidence=_verified_evidence(
|
||||
{"external_effect_started": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise FileStorageError(
|
||||
"Blob eligibility could not be rechecked; no bytes were deleted"
|
||||
) from exc
|
||||
if blob is None or references or blob_retained:
|
||||
operation.reject(
|
||||
summary="Blob garbage collection was no longer eligible",
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"blob_present": blob is not None,
|
||||
"reference_count_zero": references == 0,
|
||||
"blob_retention_expired": not blob_retained,
|
||||
},
|
||||
reference_count=references,
|
||||
),
|
||||
)
|
||||
session.rollback()
|
||||
continue
|
||||
try:
|
||||
backend.delete(blob.storage_key)
|
||||
object_present = backend.exists(blob.storage_key)
|
||||
except (StorageBackendError, OSError) as exc:
|
||||
try:
|
||||
object_present = backend.exists(blob.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
object_present = None
|
||||
session.rollback()
|
||||
if object_present is True:
|
||||
operation.fail(
|
||||
summary="The unreferenced blob object was retained",
|
||||
evidence={"object_present": True, "exception_type": type(exc).__name__},
|
||||
)
|
||||
else:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The blob deletion outcome could not be verified",
|
||||
evidence={"object_present": object_present, "exception_type": type(exc).__name__},
|
||||
failure_summary="Reconcile the exact blob key before retrying garbage collection",
|
||||
)
|
||||
unresolved.append(started.operation_id)
|
||||
continue
|
||||
if object_present:
|
||||
session.rollback()
|
||||
operation.fail(
|
||||
summary="The object store did not delete the unreferenced blob",
|
||||
evidence={"object_present": True},
|
||||
)
|
||||
continue
|
||||
try:
|
||||
session.delete(blob)
|
||||
session.flush()
|
||||
operation.commit_verified_success(
|
||||
session,
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"object_absent": True,
|
||||
"database_blob_absent": True,
|
||||
"reference_count_zero": True,
|
||||
},
|
||||
blob_id=candidate_id,
|
||||
storage_key_sha256=key_digest,
|
||||
object_present=False,
|
||||
database_blob_present=False,
|
||||
reference_count=0,
|
||||
),
|
||||
)
|
||||
deleted += 1
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="The object is absent but blob metadata still requires forward recovery",
|
||||
evidence={"object_present": False, "exception_type": type(exc).__name__},
|
||||
failure_summary="Delete the FileBlob row only after rechecking all FileVersion references",
|
||||
)
|
||||
unresolved.append(started.operation_id)
|
||||
return BlobGcResult(
|
||||
inspected_blobs=len(candidates),
|
||||
deleted_blobs=deleted,
|
||||
unresolved_operation_ids=tuple(unresolved),
|
||||
)
|
||||
|
||||
|
||||
def _asset_purge_blockers(session: Session, asset: FileAsset) -> list[str]:
|
||||
blockers: list[str] = []
|
||||
if asset.deleted_at is None:
|
||||
blockers.append("not_soft_deleted")
|
||||
if asset.legal_hold:
|
||||
blockers.append("legal_hold")
|
||||
if asset.retained_until is not None and _as_utc(asset.retained_until) > utcnow():
|
||||
blockers.append("retention_active")
|
||||
if _table_exists(session, CampaignAttachmentUse.__tablename__) and session.query(
|
||||
CampaignAttachmentUse.id
|
||||
).filter(CampaignAttachmentUse.file_asset_id == asset.id).first() is not None:
|
||||
blockers.append("campaign_evidence")
|
||||
if _table_exists(session, FileFormEvidenceGrant.__tablename__) and session.query(
|
||||
FileFormEvidenceGrant.id
|
||||
).filter(FileFormEvidenceGrant.file_asset_id == asset.id).first() is not None:
|
||||
blockers.append("form_evidence")
|
||||
if session.query(FileShare.id).filter(
|
||||
FileShare.file_asset_id == asset.id,
|
||||
effective_file_share_clause(),
|
||||
).first() is not None:
|
||||
blockers.append("active_share")
|
||||
return blockers
|
||||
|
||||
|
||||
def _asset_owner_id(asset: FileAsset) -> str:
|
||||
owner_id = asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id
|
||||
if not owner_id:
|
||||
raise FileStorageError("File has no valid owner")
|
||||
return owner_id
|
||||
|
||||
|
||||
def _asset_owner_query(session: Session, asset: FileAsset):
|
||||
return _asset_owner_query_by_values(
|
||||
session,
|
||||
tenant_id=asset.tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_asset_owner_id(asset),
|
||||
)
|
||||
|
||||
|
||||
def _asset_owner_query_by_values(
|
||||
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||
):
|
||||
query = session.query(FileAsset).filter(
|
||||
FileAsset.tenant_id == tenant_id, FileAsset.owner_type == owner_type
|
||||
)
|
||||
if owner_type == "user":
|
||||
return query.filter(FileAsset.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileAsset.owner_group_id == owner_id)
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def _folder_owner_query(
|
||||
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||
):
|
||||
query = session.query(FileFolder).filter(
|
||||
FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type
|
||||
)
|
||||
if owner_type == "user":
|
||||
return query.filter(FileFolder.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileFolder.owner_group_id == owner_id)
|
||||
raise FileStorageError("Folders must be owned by a user or group")
|
||||
|
||||
|
||||
def _connector_space_owner_query(session: Session, space: FileConnectorSpace):
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == space.tenant_id,
|
||||
FileConnectorSpace.owner_type == space.owner_type,
|
||||
)
|
||||
owner_id = connector_space_owner_id(space)
|
||||
if space.owner_type == "user":
|
||||
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||
|
||||
|
||||
def _as_utc(value: datetime) -> datetime:
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
return _as_utc(value).isoformat() if value is not None else None
|
||||
|
||||
|
||||
def _table_exists(session: Session, table_name: str) -> bool:
|
||||
return bool(session.bind is not None and inspect(session.bind).has_table(table_name))
|
||||
|
||||
|
||||
def _verified_evidence(
|
||||
checks: dict[str, object], **details: object
|
||||
) -> dict[str, object]:
|
||||
return {"verified": True, "checks": checks, **details}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BlobGcResult",
|
||||
"PurgePreview",
|
||||
"PurgePreviewItem",
|
||||
"PurgeResult",
|
||||
"execute_asset_purge",
|
||||
"garbage_collect_unreferenced_blobs",
|
||||
"get_asset_for_lifecycle",
|
||||
"preview_asset_purge",
|
||||
"restore_asset",
|
||||
"restore_connector_space",
|
||||
"restore_folder",
|
||||
"set_asset_lifecycle",
|
||||
]
|
||||
@@ -0,0 +1,90 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
|
||||
SOURCE_PROVENANCE_METADATA_KEY = "source_provenance"
|
||||
SOURCE_REVISION_METADATA_KEY = "source_revision"
|
||||
|
||||
_PROVENANCE_STRING_FIELDS = {
|
||||
"source_type",
|
||||
"connector_id",
|
||||
"provider",
|
||||
"external_id",
|
||||
"external_path",
|
||||
"external_url",
|
||||
"revision",
|
||||
"revision_label",
|
||||
"observed_at",
|
||||
"imported_at",
|
||||
}
|
||||
|
||||
|
||||
def normalize_source_revision(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def normalize_source_provenance(value: object, *, source_revision: object = None) -> dict[str, Any] | None:
|
||||
if value is None:
|
||||
revision = normalize_source_revision(source_revision)
|
||||
return {"revision": revision} if revision else None
|
||||
if not isinstance(value, Mapping):
|
||||
raise ValueError("source_provenance must be a JSON object")
|
||||
|
||||
payload: dict[str, Any] = {}
|
||||
for field in _PROVENANCE_STRING_FIELDS:
|
||||
normalized = normalize_source_revision(value.get(field))
|
||||
if normalized:
|
||||
payload[field] = normalized
|
||||
|
||||
extra = value.get("metadata")
|
||||
if isinstance(extra, Mapping) and extra:
|
||||
payload["metadata"] = dict(extra)
|
||||
elif extra is not None and extra != "":
|
||||
raise ValueError("source_provenance.metadata must be a JSON object")
|
||||
|
||||
revision = normalize_source_revision(source_revision)
|
||||
if revision:
|
||||
payload["revision"] = revision
|
||||
return payload or None
|
||||
|
||||
|
||||
def source_metadata(
|
||||
*,
|
||||
existing: Mapping[str, Any] | None = None,
|
||||
source_provenance: object = None,
|
||||
source_revision: object = None,
|
||||
) -> dict[str, Any] | None:
|
||||
payload = dict(existing or {})
|
||||
normalized_provenance = normalize_source_provenance(source_provenance, source_revision=source_revision)
|
||||
normalized_revision = normalize_source_revision(source_revision)
|
||||
if normalized_provenance:
|
||||
payload[SOURCE_PROVENANCE_METADATA_KEY] = normalized_provenance
|
||||
if normalized_revision:
|
||||
payload[SOURCE_REVISION_METADATA_KEY] = normalized_revision
|
||||
elif normalized_provenance and normalized_provenance.get("revision"):
|
||||
payload[SOURCE_REVISION_METADATA_KEY] = str(normalized_provenance["revision"])
|
||||
return payload or None
|
||||
|
||||
|
||||
def source_provenance_from_metadata(metadata: Mapping[str, Any] | None) -> dict[str, Any] | None:
|
||||
if not isinstance(metadata, Mapping):
|
||||
return None
|
||||
value = metadata.get(SOURCE_PROVENANCE_METADATA_KEY)
|
||||
if isinstance(value, Mapping):
|
||||
return dict(value)
|
||||
return None
|
||||
|
||||
|
||||
def source_revision_from_metadata(metadata: Mapping[str, Any] | None) -> str | None:
|
||||
if not isinstance(metadata, Mapping):
|
||||
return None
|
||||
revision = normalize_source_revision(metadata.get(SOURCE_REVISION_METADATA_KEY))
|
||||
if revision:
|
||||
return revision
|
||||
provenance = source_provenance_from_metadata(metadata)
|
||||
return normalize_source_revision(provenance.get("revision") if provenance else None)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user