Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a8c3192c3a | ||
|
|
ce4eaefbfd | ||
|
|
6c3cf1c55e | ||
|
|
95aef18955 | ||
|
|
a9c7a7a40c | ||
|
|
e1361427b8 | ||
|
|
ad55d47645 | ||
|
|
21122e058e | ||
|
|
25140fb0b6 | ||
|
|
e9561a87e1 | ||
|
|
9d7d3f96ec | ||
|
|
44ba0ec9c1 | ||
|
|
1e511e4c15 | ||
|
|
cb37079465 | ||
|
|
04e2bae190 | ||
|
|
aa9f90c71e | ||
|
|
60c3d14f08 | ||
|
|
3a3360650f | ||
|
|
c9f8e9262e | ||
|
|
edee29af51 | ||
|
|
0293e49def | ||
|
|
aaa364b8d4 | ||
|
|
4bb17c0f4c | ||
|
|
92e649477f | ||
|
|
0c68e904cf | ||
|
|
8ec31b16ee | ||
|
|
7e6be4b017 | ||
|
|
04882f1628 | ||
|
|
d8ae506ff8 | ||
|
|
6baf2a421b | ||
|
|
b993d8e31a | ||
|
|
359c4e9570 | ||
|
|
233ce40983 | ||
|
|
f084a0f3a9 | ||
|
|
b752dea610 | ||
|
|
159a012833 | ||
|
|
835eacfc5d | ||
|
|
85606d5580 | ||
|
|
632cc6cf7d | ||
|
|
86a905a3a7 | ||
|
|
5b868272b9 | ||
|
|
93eea839c8 | ||
|
|
b667e7ff0e | ||
|
|
0fd8f6e02a | ||
|
|
2b34f6e305 | ||
|
|
58af1a20a7 | ||
|
|
4722161592 | ||
|
|
1444ba80a0 | ||
|
|
02ef83ecee | ||
|
|
1069f85796 | ||
|
|
1401c78c8a | ||
|
|
b6109245a7 | ||
|
|
f5d40b23c2 | ||
|
|
94ea629635 | ||
|
|
6c8a8c655d | ||
|
|
062ad5ddfb | ||
|
|
9adfa91e74 | ||
|
|
65d8ed80b5 | ||
|
|
cffe161f29 | ||
|
|
5248e7de4a | ||
|
|
d5d0df792b | ||
|
|
15ade8df75 | ||
|
|
f3c485ef61 | ||
|
|
0e36b20a14 | ||
|
|
06e6e7191b | ||
|
|
3449cbc8a5 | ||
|
|
92950af6f4 | ||
|
|
8826cf2890 | ||
|
|
f2dfb6c90e | ||
|
|
3bc1d3489e | ||
|
|
d1051293b2 | ||
|
|
8c5f149f07 | ||
|
|
f3210234d3 | ||
|
|
b8b395e8b5 | ||
|
|
8bf7296bf5 | ||
|
|
620fdda2fc | ||
|
|
58c0441763 | ||
|
|
ba7fb2def7 | ||
|
|
4e7c77135c | ||
|
|
13fd7fc3bd | ||
|
|
18e6c3eb9b | ||
|
|
a5c24fe73e | ||
|
|
ee5b06b1e7 | ||
|
|
a7895ad394 |
@@ -0,0 +1,35 @@
|
|||||||
|
---
|
||||||
|
name: "Bug"
|
||||||
|
about: "Report a reproducible defect, regression, or incorrect behavior"
|
||||||
|
title: "[Bug] "
|
||||||
|
labels:
|
||||||
|
- type/bug
|
||||||
|
- status/triage
|
||||||
|
- module/files
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Repository:
|
||||||
|
- Area/module:
|
||||||
|
- Affected version or commit:
|
||||||
|
|
||||||
|
## Behavior
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
|
||||||
|
Actual:
|
||||||
|
|
||||||
|
## Reproduction
|
||||||
|
|
||||||
|
1.
|
||||||
|
2.
|
||||||
|
3.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
Logs, screenshots, traces, or failing test output:
|
||||||
|
|
||||||
|
## Verification Target
|
||||||
|
|
||||||
|
Command or workflow that should pass when fixed:
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
blank_issues_enabled: false
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
---
|
||||||
|
name: "Docs / workflow"
|
||||||
|
about: "Request documentation, process, or developer workflow changes"
|
||||||
|
title: "[Docs] "
|
||||||
|
labels:
|
||||||
|
- type/docs
|
||||||
|
- status/triage
|
||||||
|
- module/files
|
||||||
|
- area/docs
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Repository:
|
||||||
|
- Document or workflow:
|
||||||
|
|
||||||
|
## Current State
|
||||||
|
|
||||||
|
What is missing, unclear, duplicated, or stale?
|
||||||
|
|
||||||
|
## Desired State
|
||||||
|
|
||||||
|
What should the docs or workflow make clear?
|
||||||
|
|
||||||
|
## Verification Target
|
||||||
|
|
||||||
|
How should this be checked?
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
---
|
||||||
|
name: "Feature"
|
||||||
|
about: "Propose new user-visible behavior or platform capability"
|
||||||
|
title: "[Feature] "
|
||||||
|
labels:
|
||||||
|
- type/feature
|
||||||
|
- status/triage
|
||||||
|
- module/files
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
What user, operator, or developer problem should this solve?
|
||||||
|
|
||||||
|
## Proposed Capability
|
||||||
|
|
||||||
|
What should exist when this is done?
|
||||||
|
|
||||||
|
## Ownership
|
||||||
|
|
||||||
|
- Owning repository:
|
||||||
|
- Related module repositories:
|
||||||
|
- Extension point or integration boundary:
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [ ]
|
||||||
|
- [ ]
|
||||||
|
|
||||||
|
## Verification Target
|
||||||
|
|
||||||
|
Command, scenario, or UI flow that should prove completion:
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
---
|
||||||
|
name: "Task"
|
||||||
|
about: "Track implementation, maintenance, or migration work"
|
||||||
|
title: "[Task] "
|
||||||
|
labels:
|
||||||
|
- type/task
|
||||||
|
- status/triage
|
||||||
|
- module/files
|
||||||
|
---
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
What needs to be completed?
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Owning repository:
|
||||||
|
- In-scope:
|
||||||
|
- Out-of-scope:
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [ ]
|
||||||
|
- [ ]
|
||||||
|
|
||||||
|
## Verification Target
|
||||||
|
|
||||||
|
Command or manual check:
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
name: "Tech debt"
|
||||||
|
about: "Track cleanup, refactoring, risk reduction, or deferred engineering work"
|
||||||
|
title: "[Debt] "
|
||||||
|
labels:
|
||||||
|
- type/debt
|
||||||
|
- status/triage
|
||||||
|
- module/files
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Cost
|
||||||
|
|
||||||
|
What does this make harder, riskier, slower, or more fragile?
|
||||||
|
|
||||||
|
## Desired Shape
|
||||||
|
|
||||||
|
What should the code, tests, or architecture look like afterwards?
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
What behavior, compatibility, or module boundary must be preserved?
|
||||||
|
|
||||||
|
## Verification Target
|
||||||
|
|
||||||
|
Focused checks that should pass:
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
## Issue
|
||||||
|
|
||||||
|
Closes #
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
-
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
-
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
Follow-up issues:
|
||||||
@@ -0,0 +1,270 @@
|
|||||||
|
name: Module Package Release
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- "v*"
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
release_tag:
|
||||||
|
description: Existing protected version tag to publish
|
||||||
|
required: true
|
||||||
|
type: string
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
publish-packages:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
env:
|
||||||
|
GITEA_REPOSITORY: ${{ gitea.repository }}
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
|
||||||
|
with:
|
||||||
|
node-version: "22"
|
||||||
|
- name: Select and validate protected release tag
|
||||||
|
shell: bash
|
||||||
|
env:
|
||||||
|
REQUESTED_TAG: ${{ inputs.release_tag }}
|
||||||
|
TRIGGER_TAG: ${{ gitea.ref_name }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
tag="${REQUESTED_TAG:-$TRIGGER_TAG}"
|
||||||
|
case "$tag" in
|
||||||
|
v[0-9]*.[0-9]*.[0-9]*) ;;
|
||||||
|
*) echo "Release tag must start with a SemVer-shaped vX.Y.Z value" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
git fetch --force origin "refs/tags/$tag:refs/tags/$tag" refs/heads/main:refs/remotes/origin/main
|
||||||
|
tag_commit="$(git rev-list -n 1 "$tag")"
|
||||||
|
git merge-base --is-ancestor "$tag_commit" refs/remotes/origin/main || {
|
||||||
|
echo "Release tag is not contained in main" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
git checkout --detach "$tag"
|
||||||
|
printf 'RELEASE_TAG=%s\n' "$tag" >> "$GITEA_ENV"
|
||||||
|
printf 'SOURCE_DATE_EPOCH=%s\n' "$(git show -s --format=%ct HEAD)" >> "$GITEA_ENV"
|
||||||
|
- name: Validate package versions
|
||||||
|
run: |
|
||||||
|
python - <<'PY'
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import tomllib
|
||||||
|
|
||||||
|
tag = os.environ["RELEASE_TAG"]
|
||||||
|
expected = tag.removeprefix("v")
|
||||||
|
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
|
||||||
|
if project.get("version") != expected:
|
||||||
|
raise SystemExit(f"pyproject version {project.get('version')!r} does not match {tag}")
|
||||||
|
if re.fullmatch(r"govoplan-[a-z0-9-]+", str(project.get("name", ""))) is None:
|
||||||
|
raise SystemExit("Python distribution name must use the govoplan-* namespace")
|
||||||
|
webui = Path("webui/package.json")
|
||||||
|
if webui.is_file():
|
||||||
|
package = json.loads(webui.read_text(encoding="utf-8"))
|
||||||
|
if package.get("version") != expected:
|
||||||
|
raise SystemExit(f"WebUI version {package.get('version')!r} does not match {tag}")
|
||||||
|
if re.fullmatch(r"@govoplan/[a-z0-9-]+-webui", str(package.get("name", ""))) is None:
|
||||||
|
raise SystemExit("WebUI package name must use the @govoplan/*-webui namespace")
|
||||||
|
release = Path("webui/package.release.json")
|
||||||
|
if release.is_file():
|
||||||
|
release_package = json.loads(release.read_text(encoding="utf-8"))
|
||||||
|
if (
|
||||||
|
release_package.get("name") != package.get("name")
|
||||||
|
or release_package.get("version") != expected
|
||||||
|
):
|
||||||
|
raise SystemExit("WebUI release package identity does not match package.json and the release tag")
|
||||||
|
PY
|
||||||
|
- name: Build immutable package artifacts
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python -m pip install --disable-pip-version-check build==1.5.0 twine==7.0.0
|
||||||
|
rm -rf dist .package-webui
|
||||||
|
python -m build --wheel --outdir dist
|
||||||
|
python -m twine check dist/*.whl
|
||||||
|
if [[ -f webui/package.json ]]; then
|
||||||
|
mkdir .package-webui
|
||||||
|
cp -a webui/. .package-webui/
|
||||||
|
rm -rf .package-webui/node_modules .package-webui/dist
|
||||||
|
if [[ -f .package-webui/package.release.json ]]; then
|
||||||
|
cp .package-webui/package.release.json .package-webui/package.json
|
||||||
|
fi
|
||||||
|
node <<'NODE'
|
||||||
|
const fs = require("node:fs");
|
||||||
|
const path = ".package-webui/package.json";
|
||||||
|
const packageJson = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||||
|
const groups = ["dependencies", "optionalDependencies", "peerDependencies"];
|
||||||
|
for (const group of groups) {
|
||||||
|
for (const [name, specifier] of Object.entries(packageJson[group] || {})) {
|
||||||
|
if (!name.startsWith("@govoplan/")) continue;
|
||||||
|
if (typeof specifier !== "string") {
|
||||||
|
throw new Error(`${group}.${name} must use a string version`);
|
||||||
|
}
|
||||||
|
const packageSlug = name.slice("@govoplan/".length);
|
||||||
|
if (!packageSlug.endsWith("-webui")) {
|
||||||
|
throw new Error(`${group}.${name} is outside the WebUI package namespace`);
|
||||||
|
}
|
||||||
|
const repository = `govoplan-${packageSlug.slice(0, -"-webui".length)}`;
|
||||||
|
const escapedRepository = repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||||
|
const gitTag = specifier.match(
|
||||||
|
new RegExp(
|
||||||
|
`^git\\+(?:ssh://git@|https://)git\\.add-ideas\\.de/(?:GovOPlaN|add-ideas)/${escapedRepository}\\.git#v([0-9]+\\.[0-9]+\\.[0-9]+)$`,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
if (gitTag) {
|
||||||
|
packageJson[group][name] = gitTag[1];
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (specifier.startsWith("file:") || specifier.startsWith("git+")) {
|
||||||
|
throw new Error(
|
||||||
|
`${group}.${name} must resolve to an exact registry version for publication`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
delete packageJson.private;
|
||||||
|
fs.writeFileSync(path, `${JSON.stringify(packageJson, null, 2)}\n`);
|
||||||
|
NODE
|
||||||
|
npm pkg delete private --prefix .package-webui
|
||||||
|
(cd .package-webui && npm pack --ignore-scripts --pack-destination ../dist)
|
||||||
|
fi
|
||||||
|
python - <<'PY'
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
artifacts = []
|
||||||
|
for path in sorted(Path("dist").iterdir()):
|
||||||
|
if path.suffix not in {".whl", ".tgz"}:
|
||||||
|
continue
|
||||||
|
digest = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
|
artifacts.append({"filename": path.name, "sha256": digest, "size": path.stat().st_size})
|
||||||
|
payload = {
|
||||||
|
"schema_version": "1",
|
||||||
|
"repository": os.environ["GITEA_REPOSITORY"],
|
||||||
|
"tag": os.environ["RELEASE_TAG"],
|
||||||
|
"commit": subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip(),
|
||||||
|
"artifacts": artifacts,
|
||||||
|
}
|
||||||
|
Path("dist/package-artifacts.json").write_text(
|
||||||
|
json.dumps(payload, indent=2, sort_keys=True) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
PY
|
||||||
|
- name: Retain package hash evidence
|
||||||
|
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
|
||||||
|
with:
|
||||||
|
name: module-packages-${{ gitea.ref_name }}
|
||||||
|
path: dist/package-artifacts.json
|
||||||
|
- name: Check immutable registry state
|
||||||
|
shell: bash
|
||||||
|
env:
|
||||||
|
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
test -n "$PACKAGE_TOKEN"
|
||||||
|
python - <<'PY'
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import tomllib
|
||||||
|
from urllib.error import HTTPError
|
||||||
|
from urllib.parse import quote
|
||||||
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
|
api_root = "https://git.add-ideas.de/api/v1/packages/GovOPlaN"
|
||||||
|
token = os.environ["PACKAGE_TOKEN"]
|
||||||
|
|
||||||
|
def should_publish(kind, name, version, path):
|
||||||
|
package_url = "/".join(
|
||||||
|
(api_root, kind, quote(name, safe=""), quote(version, safe=""), "files")
|
||||||
|
)
|
||||||
|
request = Request(
|
||||||
|
package_url,
|
||||||
|
headers={"Accept": "application/json", "Authorization": f"token {token}"},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=30) as response:
|
||||||
|
files = json.load(response)
|
||||||
|
except HTTPError as exc:
|
||||||
|
if exc.code == 404:
|
||||||
|
print(f"{kind} package {name}=={version} is not published yet")
|
||||||
|
return True
|
||||||
|
raise
|
||||||
|
if not isinstance(files, list) or len(files) != 1:
|
||||||
|
raise SystemExit(
|
||||||
|
f"immutable {kind} package {name}=={version} has an unexpected file set"
|
||||||
|
)
|
||||||
|
expected_sha256 = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
|
if files[0].get("sha256") != expected_sha256:
|
||||||
|
raise SystemExit(
|
||||||
|
f"immutable {kind} package {name}=={version} already exists with a different SHA-256"
|
||||||
|
)
|
||||||
|
print(f"verified existing {kind} package {name}=={version} ({expected_sha256})")
|
||||||
|
return False
|
||||||
|
|
||||||
|
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
|
||||||
|
wheels = tuple(Path("dist").glob("*.whl"))
|
||||||
|
if len(wheels) != 1:
|
||||||
|
raise SystemExit("release build must contain exactly one wheel")
|
||||||
|
publish_pypi = should_publish(
|
||||||
|
"pypi", str(project["name"]), str(project["version"]), wheels[0]
|
||||||
|
)
|
||||||
|
|
||||||
|
tarballs = tuple(Path("dist").glob("*.tgz"))
|
||||||
|
if len(tarballs) > 1:
|
||||||
|
raise SystemExit("release build must contain at most one npm package")
|
||||||
|
publish_npm = False
|
||||||
|
if tarballs:
|
||||||
|
webui = json.loads(
|
||||||
|
Path(".package-webui/package.json").read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
publish_npm = should_publish(
|
||||||
|
"npm", str(webui["name"]), str(webui["version"]), tarballs[0]
|
||||||
|
)
|
||||||
|
|
||||||
|
with Path(os.environ["GITEA_ENV"]).open("a", encoding="utf-8") as env_file:
|
||||||
|
env_file.write(f"PUBLISH_PYPI={int(publish_pypi)}\n")
|
||||||
|
env_file.write(f"PUBLISH_NPM={int(publish_npm)}\n")
|
||||||
|
PY
|
||||||
|
- name: Publish wheel and WebUI package
|
||||||
|
shell: bash
|
||||||
|
env:
|
||||||
|
PACKAGE_USERNAME: ${{ secrets.GOVOPLAN_PACKAGE_USERNAME }}
|
||||||
|
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
test -n "$PACKAGE_USERNAME"
|
||||||
|
test -n "$PACKAGE_TOKEN"
|
||||||
|
if [[ "$PUBLISH_PYPI" == 1 ]]; then
|
||||||
|
TWINE_USERNAME="$PACKAGE_USERNAME" TWINE_PASSWORD="$PACKAGE_TOKEN" \
|
||||||
|
python -m twine upload --non-interactive \
|
||||||
|
--repository-url https://git.add-ideas.de/api/packages/GovOPlaN/pypi \
|
||||||
|
dist/*.whl
|
||||||
|
else
|
||||||
|
echo "Exact wheel is already present; skipping immutable retry."
|
||||||
|
fi
|
||||||
|
shopt -s nullglob
|
||||||
|
webui_packages=(dist/*.tgz)
|
||||||
|
if (( ${#webui_packages[@]} )) && [[ "$PUBLISH_NPM" == 1 ]]; then
|
||||||
|
npmrc="$(mktemp)"
|
||||||
|
trap 'rm -f "$npmrc"' EXIT
|
||||||
|
chmod 600 "$npmrc"
|
||||||
|
printf '%s\n' \
|
||||||
|
'@govoplan:registry=https://git.add-ideas.de/api/packages/GovOPlaN/npm/' \
|
||||||
|
"//git.add-ideas.de/api/packages/GovOPlaN/npm/:_authToken=$PACKAGE_TOKEN" \
|
||||||
|
> "$npmrc"
|
||||||
|
NPM_CONFIG_USERCONFIG="$npmrc" npm publish "./${webui_packages[0]}" \
|
||||||
|
--ignore-scripts --access public \
|
||||||
|
--registry https://git.add-ideas.de/api/packages/GovOPlaN/npm/
|
||||||
|
elif (( ${#webui_packages[@]} )); then
|
||||||
|
echo "Exact WebUI package is already present; skipping immutable retry."
|
||||||
|
fi
|
||||||
+18
@@ -327,3 +327,21 @@ dist
|
|||||||
# Built Visual Studio Code Extensions
|
# Built Visual Studio Code Extensions
|
||||||
*.vsix
|
*.vsix
|
||||||
|
|
||||||
|
# GovOPlaN shared ignore rules from govoplan-core
|
||||||
|
# Local WebUI test/build scratch directories
|
||||||
|
.component-test-build/
|
||||||
|
.module-test-build/
|
||||||
|
.policy-test-build/
|
||||||
|
.template-preview-test-build/
|
||||||
|
.import-test-build/
|
||||||
|
webui/.component-test-build/
|
||||||
|
webui/.module-test-build/
|
||||||
|
webui/.policy-test-build/
|
||||||
|
webui/.template-preview-test-build/
|
||||||
|
webui/.import-test-build/
|
||||||
|
*.db
|
||||||
|
# GovOPlaN local runtime state
|
||||||
|
runtime/
|
||||||
|
# GovOPlaN WebUI test output
|
||||||
|
webui/.module-test-build/
|
||||||
|
webui/.component-test-build/
|
||||||
|
|||||||
@@ -1,5 +1,11 @@
|
|||||||
# GovOPlaN Files Codex Guide
|
# GovOPlaN Files Codex Guide
|
||||||
|
|
||||||
|
## Documentation Contract
|
||||||
|
|
||||||
|
- Treat documentation as part of every behavior change. Update this module's manifest-driven `DocumentationTopic` contributions for affected user and administrator behavior.
|
||||||
|
- Keep feature content here; `govoplan-docs` projects it without importing Files internals.
|
||||||
|
- Maintain a static user/admin baseline and run `/mnt/DATA/git/govoplan/tools/checks/check-manifest-shapes.py` after behavior or manifest changes.
|
||||||
|
|
||||||
## Scope
|
## Scope
|
||||||
|
|
||||||
This repository owns the `files` module: managed file storage APIs, file metadata, shares, uploads/downloads, folder and pattern helpers, backend module manifest, and `@govoplan/files-webui`.
|
This repository owns the `files` module: managed file storage APIs, file metadata, shares, uploads/downloads, folder and pattern helpers, backend module manifest, and `@govoplan/files-webui`.
|
||||||
@@ -18,8 +24,8 @@ cd /mnt/DATA/git/govoplan-core
|
|||||||
For combined checks, run:
|
For combined checks, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /mnt/DATA/git/govoplan-core
|
cd /mnt/DATA/git/govoplan
|
||||||
./scripts/check-focused.sh
|
tools/checks/check-focused.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
For WebUI permutation checks that include files:
|
For WebUI permutation checks that include files:
|
||||||
|
|||||||
@@ -1,5 +1,9 @@
|
|||||||
# govoplan-files
|
# govoplan-files
|
||||||
|
|
||||||
|
<!-- govoplan-repository-type:start -->
|
||||||
|
**Repository type:** module (domain).
|
||||||
|
<!-- govoplan-repository-type:end -->
|
||||||
|
|
||||||
GovOPlaN Files is the managed file module. It bundles backend storage APIs and the Files WebUI package so file features can be installed as one module.
|
GovOPlaN Files is the managed file module. It bundles backend storage APIs and the Files WebUI package so file features can be installed as one module.
|
||||||
|
|
||||||
## Ownership
|
## Ownership
|
||||||
@@ -46,10 +50,166 @@ Frontend package:
|
|||||||
@govoplan/files-webui
|
@govoplan/files-webui
|
||||||
```
|
```
|
||||||
|
|
||||||
The campaign module can integrate with files when both modules are installed, for example for managed attachment selection and campaign file sharing.
|
The campaign module can integrate with files when both modules are installed,
|
||||||
|
for example for managed attachment selection and campaign file sharing. Files
|
||||||
|
does not import campaign internals; campaign share/existence checks use the core
|
||||||
|
`campaigns.access` capability registered by the campaign module.
|
||||||
|
|
||||||
|
Files also publishes the optional `privacy.dsar.files` capability. The Core
|
||||||
|
data-subject workflow can use it to collect bounded, tenant-scoped file,
|
||||||
|
version, share, folder, evidence, and non-secret configuration metadata for a
|
||||||
|
direct membership subject. The provider may revoke a subject-targeted share or
|
||||||
|
detach mutable actor references idempotently, but it never exports credential
|
||||||
|
material or raw bytes and never bypasses Files retention, legal hold, evidence,
|
||||||
|
purge approval, audit, or recovery controls. See the handbook's data-subject
|
||||||
|
request coverage section for the review and erasure boundary.
|
||||||
|
|
||||||
Platform RBAC and governance rules are documented in `govoplan-core/docs/`.
|
Platform RBAC and governance rules are documented in `govoplan-core/docs/`.
|
||||||
|
|
||||||
|
Managed files can carry source provenance for connector and import workflows.
|
||||||
|
Upload callers may provide `source_provenance_json` and `source_revision`; the
|
||||||
|
module stores those values under file metadata, returns normalized
|
||||||
|
`source_provenance` and `source_revision` fields in file responses, and carries
|
||||||
|
them into managed campaign attachment matches for frozen execution evidence.
|
||||||
|
|
||||||
|
Connector policy preflight is available through
|
||||||
|
`POST /api/v1/files/connector-policy/evaluate`, and provenance-bearing uploads
|
||||||
|
can pass `connector_policy_json` to enforce the same policy before file content
|
||||||
|
is read. Policy payloads contain ordered `sources`; each source has
|
||||||
|
`scope_type`, optional `scope_id`, optional `label`, and a `policy` object. The
|
||||||
|
policy object supports `allow`/`allowlist`/`whitelist` and
|
||||||
|
`deny`/`denylist`/`blacklist` rules for `connectors`, `providers`,
|
||||||
|
`external_ids`, `external_paths`, and `external_urls`. Deny rules win across the
|
||||||
|
hierarchy; allow rules narrow access at each source that defines them.
|
||||||
|
|
||||||
|
Connector endpoint settings are exposed through governed connector profiles.
|
||||||
|
Profiles can be supplied as JSON through
|
||||||
|
`GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through
|
||||||
|
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`,
|
||||||
|
`endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and
|
||||||
|
deployment-owned credential references such as `password_env`, `token_env`, or
|
||||||
|
`secret_ref`. Environment references require an exact name in the deployment-wide
|
||||||
|
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
|
||||||
|
process environment variables and may use only Files-owned encrypted password or
|
||||||
|
token values. API-created `secret_ref` values fail closed until Files has an
|
||||||
|
ownership contract that can confirm provider-side deletion. Legacy external
|
||||||
|
references are treated as non-owned: deleting a profile or credential detaches
|
||||||
|
and audits the reference but never passes it to an arbitrary secret provider.
|
||||||
|
Profile and credential deletion immediately clears encrypted values, credential
|
||||||
|
identities, deployment references, and private metadata in the same transaction
|
||||||
|
as the non-secret audit record; an audit failure rolls the deletion back.
|
||||||
|
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
|
||||||
|
current principal (system, tenant, user, group, or accessible campaign scope) and
|
||||||
|
redacts secret values and environment variable names. Use the returned
|
||||||
|
`policy_sources` with connector policy preflight before importing files.
|
||||||
|
`GET /api/v1/files/connectors/providers` exposes provider descriptors for
|
||||||
|
Seafile, Nextcloud, WebDAV, SMB, NFS, and local filesystem connectors. The
|
||||||
|
descriptor declares implementation status, optional dependencies, permission
|
||||||
|
mapping, sync/index strategy, conflict handling, preview behavior, and audit
|
||||||
|
events so provider coverage remains visible without forcing every optional
|
||||||
|
protocol dependency to be installed.
|
||||||
|
`GET /api/v1/files/connectors/profiles/{profile_id}/browse` provides read-only
|
||||||
|
connector browsing. Browse entries use a shared `library`/`folder`/`file` shape,
|
||||||
|
run profile policy with the `browse` operation, and support Seafile,
|
||||||
|
WebDAV/Nextcloud, and SMB when the optional `smb` extra is installed. Seafile
|
||||||
|
profiles browse libraries and directories via Seafile's read-only API; profiles
|
||||||
|
can still opt into WebDAV browsing by setting `metadata.webdav_endpoint_url` or
|
||||||
|
`metadata.browse_protocol` to `webdav`. S3-compatible profiles support bucket
|
||||||
|
and prefix browsing when the optional `s3` extra is installed.
|
||||||
|
`POST /api/v1/files/connectors/profiles/{profile_id}/import` imports a Seafile
|
||||||
|
or WebDAV/Nextcloud/SMB file into managed storage through the same governance,
|
||||||
|
conflict handling, source provenance, and connector audit path as direct
|
||||||
|
uploads. The Seafile provider uses account-token auth and the native file
|
||||||
|
download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET`
|
||||||
|
requests against the configured WebDAV endpoint. SMB profiles use
|
||||||
|
`smb://server[:port]/share[/path]` endpoints and deployment-owned or encrypted
|
||||||
|
stored credentials through `smbprotocol`. Files installs a pinned transport for
|
||||||
|
every initial session, reconnect, alias, and DFS referral target. The deployment
|
||||||
|
private-network policy is re-evaluated immediately before each socket opens.
|
||||||
|
|
||||||
|
S3 connector browse/import binds botocore HTTP and HTTPS pools to the same pinned
|
||||||
|
socket policy. Retries, redirects, endpoint discovery, bucket aliases, and new
|
||||||
|
connections are therefore revalidated while TLS keeps the configured hostname
|
||||||
|
for SNI and certificate checks. Connector clients do not use outbound proxies or
|
||||||
|
ambient AWS credential discovery; configure credentials on the governed profile,
|
||||||
|
or explicitly use an anonymous profile for public objects. Keep connector spaces
|
||||||
|
read-only unless a remote write is explicitly needed.
|
||||||
|
An S3 profile with the `write` capability can back a two-way connector space.
|
||||||
|
`POST /api/v1/files/connector-spaces/{space_id}/write-back` then conditionally
|
||||||
|
creates or replaces one remote object from a managed file, records durable
|
||||||
|
recovery intent before the provider effect, and verifies request/content markers
|
||||||
|
afterward. Automatic remote delete, rename, move, and ACL propagation remain
|
||||||
|
disabled; other connector providers remain read-only.
|
||||||
|
An incompatible SDK upgrade fails closed before a usable client/session is
|
||||||
|
returned.
|
||||||
|
|
||||||
|
Durable platform storage has a separate deployment boundary: installer-owned
|
||||||
|
Garage is accepted only at its exact generated endpoint, while an
|
||||||
|
operator-selected external backend requires a clean HTTPS origin and
|
||||||
|
`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`. That flag is deployment configuration,
|
||||||
|
cannot be supplied through a Files connector profile, and does not replace
|
||||||
|
operator responsibility for DNS, certificates, egress, bucket policy,
|
||||||
|
versioning, and recovery.
|
||||||
|
|
||||||
|
The actual local/S3 backend implementation is owned by Core so Files, Campaign,
|
||||||
|
and workers resolve the same object namespace. Files owns file metadata and key
|
||||||
|
layout. Node-local storage is supported only for `local` or one-host
|
||||||
|
`host-shared` profiles; multi-host `shared` deployments require S3.
|
||||||
|
|
||||||
|
Destructive Files-module retirement applies the same credential lifecycle before
|
||||||
|
dropping tables. Every remaining Files-owned encrypted connector secret is
|
||||||
|
scrubbed and audited first, while legacy non-owned external references are
|
||||||
|
detached and identified as such in the audit record. Retirement does not claim
|
||||||
|
or attempt provider-side deletion for references Files cannot prove it owns.
|
||||||
|
|
||||||
|
Local connector development assets live in `dev/connectors/`. The compose stack
|
||||||
|
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
|
||||||
|
manual interoperability testing.
|
||||||
|
|
||||||
|
Connector and collaboration ownership boundaries are documented in
|
||||||
|
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
|
||||||
|
The role-adaptive user, administration, integration, and operator guide is the
|
||||||
|
[Files handbook](docs/FILES_HANDBOOK.md).
|
||||||
|
The Files route, connector surfaces, consequence classes, and pattern-language
|
||||||
|
verification are recorded in
|
||||||
|
[Files interface pattern migration](docs/INTERFACE_PATTERN_MIGRATION.md).
|
||||||
|
|
||||||
|
Archive imports use a two-phase preview and confirmation flow for ZIP, TAR,
|
||||||
|
TAR.GZ, TAR.BZ2, and TAR.XZ. Requests are spooled to bounded temporary files;
|
||||||
|
the server validates paths, entry count, expanded size, and expansion ratio,
|
||||||
|
then returns a 30-minute tenant/user-bound preview token. Confirmation reuploads
|
||||||
|
the original archive and stores only the selected members. Password-protected
|
||||||
|
ZIP passwords remain request-only and are never included in the preview token.
|
||||||
|
|
||||||
|
Managed blob writes and applied orphan cleanup use Core's durable recovery
|
||||||
|
ledger. On PostgreSQL, intent, request digests, recovery mode, and a distributed
|
||||||
|
lease are committed before physical storage effects; the Files session commit
|
||||||
|
verifies database and streamed object evidence, while rollback compensates only
|
||||||
|
a newly reserved unreferenced key. Development SQLite records blob intent in
|
||||||
|
the caller transaction to avoid its second-writer deadlock, then verifies on
|
||||||
|
commit or reconstructs compensation evidence after handled rollback. Because a
|
||||||
|
hard loss before that commit can leave an unrecorded object, SQLite requires a
|
||||||
|
complete integrity scan after a crash and is not a production recovery profile.
|
||||||
|
New object keys are opaque and do not retain the uploaded filename. Uncertain
|
||||||
|
or mismatched effects remain visible through Ops.
|
||||||
|
|
||||||
|
Operators with `files:file:admin` can run bounded, resumable integrity scans in
|
||||||
|
Administration. Scan batches and finding actions carry monotonic revisions;
|
||||||
|
stale resume, recheck, or cleanup requests fail before touching object storage.
|
||||||
|
Orphan cleanup always requires a dry-run preview followed by separate
|
||||||
|
confirmation and records recovery-ledger evidence.
|
||||||
|
|
||||||
|
Bulk rename and transfer APIs are owner-scoped: callers must provide the active
|
||||||
|
user or group file space with `owner_type` and `owner_id`. The storage layer
|
||||||
|
keeps a named legacy file-only helper for historical callers that lack owner
|
||||||
|
context, and regression tests cover its write-access checks.
|
||||||
|
|
||||||
|
Optional producer modules can persist generated output through the provider-
|
||||||
|
neutral `files.artifact_store` capability. Files remains responsible for upload
|
||||||
|
authorization, ownership, path normalization, versioning, and blob storage;
|
||||||
|
producers receive stable file/version references without importing Files
|
||||||
|
internals. See [Generated Artifact Store](docs/GENERATED_ARTIFACT_STORE.md).
|
||||||
|
|
||||||
## Release packaging
|
## Release packaging
|
||||||
|
|
||||||
The repository root includes a `package.json` for git-based WebUI installs. It exports the package `@govoplan/files-webui` from `webui/src` so release builds can depend on tagged git refs instead of local `file:` paths.
|
The repository root includes a `package.json` for git-based WebUI installs. It exports the package `@govoplan/files-webui` from `webui/src` so release builds can depend on tagged git refs instead of local `file:` paths.
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
NEXTCLOUD_HOST_PORT=9081
|
||||||
|
NEXTCLOUD_DB_ROOT_PASSWORD=govoplan-nextcloud-root
|
||||||
|
NEXTCLOUD_DB_PASSWORD=govoplan-nextcloud
|
||||||
|
NEXTCLOUD_ADMIN_USER=admin
|
||||||
|
NEXTCLOUD_ADMIN_PASSWORD=govoplan-nextcloud-admin
|
||||||
|
|
||||||
|
SEAFILE_HOST_PORT=9082
|
||||||
|
SEAFILE_MYSQL_ROOT_PASSWORD=govoplan-seafile-root
|
||||||
|
SEAFILE_ADMIN_EMAIL=admin@example.local
|
||||||
|
SEAFILE_ADMIN_PASSWORD=govoplan-seafile-admin
|
||||||
|
|
||||||
|
WEBDAV_HOST_PORT=9083
|
||||||
|
WEBDAV_USER=govoplan
|
||||||
|
WEBDAV_PASSWORD=govoplan-webdav
|
||||||
|
|
||||||
|
SMB_HOST_PORT=1445
|
||||||
|
SMB_SHARE_NAME=files
|
||||||
|
SMB_USER=govoplan
|
||||||
|
SMB_PASSWORD=govoplan-smb
|
||||||
|
|
||||||
|
MINIO_API_HOST_PORT=9000
|
||||||
|
MINIO_CONSOLE_HOST_PORT=9001
|
||||||
|
MINIO_ROOT_USER=govoplan
|
||||||
|
MINIO_ROOT_PASSWORD=govoplan-minio
|
||||||
|
MINIO_BUCKET=govoplan
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
# Connector Dev Stack
|
||||||
|
|
||||||
|
This directory keeps local Docker Compose assets for GovOPlaN file connector
|
||||||
|
development. The stack is intentionally separate from production deployment and
|
||||||
|
binds services to localhost high ports.
|
||||||
|
|
||||||
|
## Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||||
|
cp .env.example .env
|
||||||
|
mkdir -p data/smb data/webdav
|
||||||
|
docker compose up -d nextcloud nextcloud-db webdav smb minio
|
||||||
|
docker compose up -d seafile-db seafile-memcached seafile
|
||||||
|
```
|
||||||
|
|
||||||
|
If the SMB service was already running before a compose change, recreate it so
|
||||||
|
the image and share configuration are applied:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose rm -sf smb
|
||||||
|
docker compose up -d --build smb
|
||||||
|
```
|
||||||
|
|
||||||
|
Endpoints:
|
||||||
|
|
||||||
|
- Nextcloud: `http://127.0.0.1:9081`, WebDAV root
|
||||||
|
`http://127.0.0.1:9081/remote.php/dav/files/admin/`
|
||||||
|
- Seafile: `http://127.0.0.1:9082`, WebDAV root
|
||||||
|
`http://127.0.0.1:9082/seafdav/`
|
||||||
|
- WebDAV: `http://127.0.0.1:9083`
|
||||||
|
- SMB: `smb://127.0.0.1:1445/files`
|
||||||
|
- MinIO/S3 API: `http://127.0.0.1:9000`, console
|
||||||
|
`http://127.0.0.1:9001`
|
||||||
|
|
||||||
|
The local fixture data under `data/` is ignored by git.
|
||||||
|
|
||||||
|
## GovOPlaN Profile Config
|
||||||
|
|
||||||
|
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
|
||||||
|
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. This deployment-owned configuration
|
||||||
|
may reference environment variables only when their exact names are listed in
|
||||||
|
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; profile API responses expose only the
|
||||||
|
credential source and configured state.
|
||||||
|
|
||||||
|
Example local profile file:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"profiles": [
|
||||||
|
{
|
||||||
|
"id": "dev-seafile",
|
||||||
|
"label": "Dev Seafile",
|
||||||
|
"provider": "seafile",
|
||||||
|
"endpoint_url": "http://127.0.0.1:9082",
|
||||||
|
"scope_type": "system",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": "admin@example.local",
|
||||||
|
"password_env": "SEAFILE_ADMIN_PASSWORD",
|
||||||
|
"capabilities": ["browse", "import"],
|
||||||
|
"metadata": { "webdav_endpoint_url": "http://127.0.0.1:9082/seafdav/" },
|
||||||
|
"policy": { "allow": { "providers": ["seafile"] } }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-nextcloud",
|
||||||
|
"label": "Dev Nextcloud",
|
||||||
|
"provider": "nextcloud",
|
||||||
|
"endpoint_url": "http://127.0.0.1:9081/remote.php/dav/files/admin/",
|
||||||
|
"scope_type": "system",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": "admin",
|
||||||
|
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
|
||||||
|
"capabilities": ["browse", "import"],
|
||||||
|
"policy": { "allow": { "providers": ["nextcloud", "webdav"] } }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-webdav",
|
||||||
|
"label": "Dev WebDAV",
|
||||||
|
"provider": "webdav",
|
||||||
|
"endpoint_url": "http://127.0.0.1:9083",
|
||||||
|
"scope_type": "system",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": "govoplan",
|
||||||
|
"password_env": "WEBDAV_PASSWORD",
|
||||||
|
"capabilities": ["browse", "import"],
|
||||||
|
"policy": { "allow": { "providers": ["webdav"] } }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-smb",
|
||||||
|
"label": "Dev SMB",
|
||||||
|
"provider": "smb",
|
||||||
|
"endpoint_url": "smb://127.0.0.1:1445/files",
|
||||||
|
"scope_type": "system",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": "govoplan",
|
||||||
|
"password_env": "SMB_PASSWORD",
|
||||||
|
"capabilities": ["browse", "import"],
|
||||||
|
"policy": { "allow": { "providers": ["smb"] } }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-s3",
|
||||||
|
"label": "Dev MinIO",
|
||||||
|
"provider": "s3",
|
||||||
|
"endpoint_url": "http://127.0.0.1:9000",
|
||||||
|
"base_path": "",
|
||||||
|
"scope_type": "system",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": "govoplan",
|
||||||
|
"password_env": "MINIO_ROOT_PASSWORD",
|
||||||
|
"capabilities": ["browse", "import"],
|
||||||
|
"metadata": {
|
||||||
|
"bucket": "govoplan",
|
||||||
|
"region": "us-east-1",
|
||||||
|
"path_style": true,
|
||||||
|
"verify_tls": false
|
||||||
|
},
|
||||||
|
"policy": { "allow": { "providers": ["s3"] } }
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Start GovOPlaN with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
|
||||||
|
export GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST=SEAFILE_ADMIN_PASSWORD,NEXTCLOUD_ADMIN_PASSWORD,WEBDAV_PASSWORD,SMB_PASSWORD,MINIO_ROOT_PASSWORD
|
||||||
|
export GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true
|
||||||
|
```
|
||||||
|
|
||||||
|
## Smoke Test
|
||||||
|
|
||||||
|
Run the connector helper smoke checks from an environment where
|
||||||
|
`govoplan-files` and `httpx` are importable:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||||
|
/mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The script reads `.env` from this directory when present and seeds tiny WebDAV,
|
||||||
|
Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud are browsed and imported
|
||||||
|
through the pinned connector transport. SMB and S3 product access deliberately
|
||||||
|
fails closed until their SDK transports support peer pinning, so their fixtures
|
||||||
|
are retained for transport development and reported as expected optional skips.
|
||||||
|
The `--require-smb` and `--require-s3` switches are useful only while developing
|
||||||
|
that transport support and currently make the smoke check fail by design.
|
||||||
|
|
||||||
|
SMB smoke checks need the optional Python dependency in the environment running
|
||||||
|
the script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
|
||||||
|
```
|
||||||
|
|
||||||
|
S3 smoke checks need the optional boto3 dependency in the environment running
|
||||||
|
the script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[s3]
|
||||||
|
```
|
||||||
|
|
||||||
|
The SMB service is built from `dev/connectors/smb/` so the development share is
|
||||||
|
deterministic: one `files` share backed by `data/smb`, with the credentials from
|
||||||
|
`.env`.
|
||||||
|
|
||||||
|
The SMB image runs as the non-root `govoplan` user with UID/GID `1000` and
|
||||||
|
listens on unprivileged container port `1445`. The default host endpoint remains
|
||||||
|
`smb://127.0.0.1:1445/files`. `SMB_USER` must stay `govoplan` unless the image is
|
||||||
|
rebuilt with a matching user; `SMB_PORT` must be `1024` or higher. `SMB_PASSWORD`
|
||||||
|
is applied when the image is built, so rebuild the `smb` service after changing
|
||||||
|
it in `.env`.
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
*
|
||||||
|
!.gitignore
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
name: govoplan-files-connectors
|
||||||
|
|
||||||
|
services:
|
||||||
|
nextcloud-db:
|
||||||
|
image: mariadb:10.11
|
||||||
|
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
MYSQL_ROOT_PASSWORD: ${NEXTCLOUD_DB_ROOT_PASSWORD:-govoplan-nextcloud-root}
|
||||||
|
MYSQL_DATABASE: nextcloud
|
||||||
|
MYSQL_USER: nextcloud
|
||||||
|
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
|
||||||
|
volumes:
|
||||||
|
- nextcloud-db:/var/lib/mysql
|
||||||
|
|
||||||
|
nextcloud:
|
||||||
|
image: nextcloud:apache
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- nextcloud-db
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:${NEXTCLOUD_HOST_PORT:-9081}:80"
|
||||||
|
environment:
|
||||||
|
MYSQL_HOST: nextcloud-db
|
||||||
|
MYSQL_DATABASE: nextcloud
|
||||||
|
MYSQL_USER: nextcloud
|
||||||
|
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
|
||||||
|
NEXTCLOUD_ADMIN_USER: ${NEXTCLOUD_ADMIN_USER:-admin}
|
||||||
|
NEXTCLOUD_ADMIN_PASSWORD: ${NEXTCLOUD_ADMIN_PASSWORD:-govoplan-nextcloud-admin}
|
||||||
|
NEXTCLOUD_TRUSTED_DOMAINS: "localhost 127.0.0.1"
|
||||||
|
volumes:
|
||||||
|
- nextcloud:/var/www/html
|
||||||
|
|
||||||
|
seafile-db:
|
||||||
|
image: mariadb:10.11
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
MYSQL_ROOT_PASSWORD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
|
||||||
|
MARIADB_AUTO_UPGRADE: "1"
|
||||||
|
MYSQL_LOG_CONSOLE: "true"
|
||||||
|
volumes:
|
||||||
|
- seafile-db:/var/lib/mysql
|
||||||
|
|
||||||
|
seafile-memcached:
|
||||||
|
image: memcached:1.6
|
||||||
|
restart: unless-stopped
|
||||||
|
command: memcached -m 256
|
||||||
|
|
||||||
|
seafile:
|
||||||
|
image: seafileltd/seafile-mc:11.0-latest
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- seafile-db
|
||||||
|
- seafile-memcached
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:${SEAFILE_HOST_PORT:-9082}:80"
|
||||||
|
environment:
|
||||||
|
DB_HOST: seafile-db
|
||||||
|
DB_ROOT_PASSWD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
|
||||||
|
TIME_ZONE: Etc/UTC
|
||||||
|
SEAFILE_ADMIN_EMAIL: ${SEAFILE_ADMIN_EMAIL:-admin@example.local}
|
||||||
|
SEAFILE_ADMIN_PASSWORD: ${SEAFILE_ADMIN_PASSWORD:-govoplan-seafile-admin}
|
||||||
|
SEAFILE_SERVER_LETSENCRYPT: "false"
|
||||||
|
SEAFILE_SERVER_HOSTNAME: "127.0.0.1:${SEAFILE_HOST_PORT:-9082}"
|
||||||
|
volumes:
|
||||||
|
- seafile-data:/shared
|
||||||
|
|
||||||
|
webdav:
|
||||||
|
image: rclone/rclone:latest
|
||||||
|
restart: unless-stopped
|
||||||
|
command:
|
||||||
|
- serve
|
||||||
|
- webdav
|
||||||
|
- /data
|
||||||
|
- --addr
|
||||||
|
- :8080
|
||||||
|
- --user
|
||||||
|
- ${WEBDAV_USER:-govoplan}
|
||||||
|
- --pass
|
||||||
|
- ${WEBDAV_PASSWORD:-govoplan-webdav}
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:${WEBDAV_HOST_PORT:-9083}:8080"
|
||||||
|
volumes:
|
||||||
|
- ./data/webdav:/data
|
||||||
|
|
||||||
|
smb:
|
||||||
|
build:
|
||||||
|
context: ./smb
|
||||||
|
args:
|
||||||
|
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
|
||||||
|
image: govoplan-files-samba-dev:latest
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
|
||||||
|
SMB_USER: ${SMB_USER:-govoplan}
|
||||||
|
SMB_PORT: ${SMB_PORT:-1445}
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:${SMB_HOST_PORT:-1445}:${SMB_PORT:-1445}"
|
||||||
|
volumes:
|
||||||
|
- ./data/smb:/storage
|
||||||
|
|
||||||
|
minio:
|
||||||
|
image: minio/minio:latest
|
||||||
|
restart: unless-stopped
|
||||||
|
command: server /data --console-address ":9001"
|
||||||
|
environment:
|
||||||
|
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-govoplan}
|
||||||
|
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-govoplan-minio}
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:${MINIO_API_HOST_PORT:-9000}:9000"
|
||||||
|
- "127.0.0.1:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"
|
||||||
|
volumes:
|
||||||
|
- minio:/data
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
nextcloud-db:
|
||||||
|
nextcloud:
|
||||||
|
seafile-db:
|
||||||
|
seafile-data:
|
||||||
|
minio:
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
FROM alpine:3.20
|
||||||
|
|
||||||
|
ARG SMB_PASSWORD=govoplan-smb
|
||||||
|
|
||||||
|
RUN apk add --no-cache samba-server samba-common-tools \
|
||||||
|
&& addgroup -S -g 1000 govoplan \
|
||||||
|
&& adduser -S -D -H -h /nonexistent -s /sbin/nologin -u 1000 -G govoplan govoplan \
|
||||||
|
&& mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba /var/log/samba/cores \
|
||||||
|
&& printf '%s\n' \
|
||||||
|
'[global]' \
|
||||||
|
' passdb backend = tdbsam' \
|
||||||
|
' private dir = /var/lib/samba/private' \
|
||||||
|
' lock directory = /run/samba' \
|
||||||
|
' state directory = /var/lib/samba' \
|
||||||
|
' cache directory = /var/cache/samba' \
|
||||||
|
' pid directory = /run/samba' \
|
||||||
|
> /etc/samba/smb.conf \
|
||||||
|
&& printf '%s\n%s\n' "$SMB_PASSWORD" "$SMB_PASSWORD" | smbpasswd -s -a govoplan \
|
||||||
|
&& smbpasswd -e govoplan \
|
||||||
|
&& chown -R govoplan:govoplan /storage /var/lib/samba /var/cache/samba /run/samba /etc/samba /var/log/samba
|
||||||
|
|
||||||
|
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
|
||||||
|
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
|
||||||
|
|
||||||
|
EXPOSE 1445
|
||||||
|
|
||||||
|
USER govoplan:govoplan
|
||||||
|
|
||||||
|
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
share_name="${SMB_SHARE_NAME:-files}"
|
||||||
|
user_name="${SMB_USER:-govoplan}"
|
||||||
|
smb_port="${SMB_PORT:-1445}"
|
||||||
|
runtime_user="$(id -un)"
|
||||||
|
runtime_group="$(id -gn)"
|
||||||
|
|
||||||
|
case "$share_name" in
|
||||||
|
"" | *[!A-Za-z0-9_.-]*)
|
||||||
|
printf 'SMB_SHARE_NAME must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
case "$user_name" in
|
||||||
|
"" | *[!A-Za-z0-9_.-]*)
|
||||||
|
printf 'SMB_USER must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
case "$smb_port" in
|
||||||
|
"" | *[!0-9]*)
|
||||||
|
printf 'SMB_PORT must be numeric.\n' >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [ "$smb_port" -lt 1024 ]; then
|
||||||
|
printf 'SMB_PORT must be >= 1024 because the dev Samba container runs as a non-root user.\n' >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$user_name" != "$runtime_user" ]; then
|
||||||
|
printf 'SMB_USER=%s is not supported by the non-root dev image; use %s or rebuild the image with a matching user.\n' "$user_name" "$runtime_user" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba
|
||||||
|
|
||||||
|
cat > /etc/samba/smb.conf <<EOF
|
||||||
|
[global]
|
||||||
|
server role = standalone server
|
||||||
|
workgroup = WORKGROUP
|
||||||
|
security = user
|
||||||
|
map to guest = Never
|
||||||
|
guest account = $runtime_user
|
||||||
|
server min protocol = SMB2
|
||||||
|
server signing = mandatory
|
||||||
|
smb ports = $smb_port
|
||||||
|
load printers = no
|
||||||
|
printing = bsd
|
||||||
|
disable spoolss = yes
|
||||||
|
log level = 1
|
||||||
|
passdb backend = tdbsam
|
||||||
|
private dir = /var/lib/samba/private
|
||||||
|
lock directory = /run/samba
|
||||||
|
state directory = /var/lib/samba
|
||||||
|
cache directory = /var/cache/samba
|
||||||
|
pid directory = /run/samba
|
||||||
|
|
||||||
|
[$share_name]
|
||||||
|
path = /storage
|
||||||
|
browseable = yes
|
||||||
|
read only = no
|
||||||
|
guest ok = no
|
||||||
|
valid users = $user_name
|
||||||
|
force user = $runtime_user
|
||||||
|
force group = $runtime_group
|
||||||
|
create mask = 0664
|
||||||
|
directory mask = 0775
|
||||||
|
EOF
|
||||||
|
|
||||||
|
if ! pdbedit -L -u "$user_name" >/dev/null 2>&1; then
|
||||||
|
printf 'Samba user %s is missing from passdb. Rebuild the smb image so the non-root passdb is seeded.\n' "$user_name" >&2
|
||||||
|
printf 'Run: docker compose build --no-cache smb && docker compose up -d smb\n' >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec smbd --foreground --no-process-group --debug-stdout -p "$smb_port"
|
||||||
@@ -0,0 +1,282 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import socket
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from govoplan_files.backend.storage.connector_browse import browse_connector_profile
|
||||||
|
from govoplan_files.backend.storage.connector_imports import read_connector_file
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import connector_profiles_from_payload
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
|
||||||
|
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
|
||||||
|
parser.add_argument("--require-s3", action="store_true", help="Fail if the S3 connector cannot browse/import.")
|
||||||
|
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
_load_dotenv()
|
||||||
|
_default_env()
|
||||||
|
preflight_failures = _preflight_services(require_smb=args.require_smb, require_s3=args.require_s3)
|
||||||
|
if preflight_failures:
|
||||||
|
for failure in preflight_failures:
|
||||||
|
print(f"FAIL {failure}")
|
||||||
|
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb minio")
|
||||||
|
return 1
|
||||||
|
_seed_webdav_fixture()
|
||||||
|
_seed_smb_fixture()
|
||||||
|
try:
|
||||||
|
_seed_nextcloud_fixture()
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
print(f"FAIL dev-nextcloud seed failed: {exc}")
|
||||||
|
return 1
|
||||||
|
try:
|
||||||
|
_seed_s3_fixture()
|
||||||
|
except Exception as exc:
|
||||||
|
if args.require_s3:
|
||||||
|
print(f"FAIL dev-s3 seed failed: {exc}")
|
||||||
|
return 1
|
||||||
|
print(f"SKIP dev-s3 seed failed: {exc}")
|
||||||
|
|
||||||
|
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
|
||||||
|
if args.debug_smb:
|
||||||
|
_debug_smb(profiles["dev-smb"])
|
||||||
|
failures: list[str] = []
|
||||||
|
for profile_id, folder, file_path, expected in (
|
||||||
|
("dev-webdav", "GovOPlaN", "GovOPlaN/webdav-live.txt", "webdav live fixture"),
|
||||||
|
("dev-nextcloud", "GovOPlaN", "GovOPlaN/nextcloud-live.txt", "nextcloud live fixture"),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
_exercise_profile(profiles[profile_id], folder=folder, file_path=file_path, expected=expected)
|
||||||
|
except Exception as exc:
|
||||||
|
failures.append(f"{profile_id}: {exc}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
_exercise_profile(profiles["dev-smb"], folder="GovOPlaN", file_path="GovOPlaN/smb-live.txt", expected="smb live fixture")
|
||||||
|
except Exception as exc:
|
||||||
|
message = f"dev-smb: {exc}"
|
||||||
|
if args.require_smb:
|
||||||
|
failures.append(message)
|
||||||
|
else:
|
||||||
|
print(f"SKIP {message}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
_exercise_profile(profiles["dev-s3"], folder="GovOPlaN", file_path="GovOPlaN/s3-live.txt", expected="s3 live fixture")
|
||||||
|
except Exception as exc:
|
||||||
|
message = f"dev-s3: {exc}"
|
||||||
|
if args.require_s3:
|
||||||
|
failures.append(message)
|
||||||
|
else:
|
||||||
|
print(f"SKIP {message}")
|
||||||
|
|
||||||
|
if failures:
|
||||||
|
for failure in failures:
|
||||||
|
print(f"FAIL {failure}")
|
||||||
|
return 1
|
||||||
|
print("OK connector dev stack smoke checks passed")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _default_env() -> None:
|
||||||
|
defaults = {
|
||||||
|
"NEXTCLOUD_ADMIN_USER": "admin",
|
||||||
|
"NEXTCLOUD_ADMIN_PASSWORD": "govoplan-nextcloud-admin",
|
||||||
|
"WEBDAV_USER": "govoplan",
|
||||||
|
"WEBDAV_PASSWORD": "govoplan-webdav",
|
||||||
|
"SMB_SHARE_NAME": "files",
|
||||||
|
"SMB_USER": "govoplan",
|
||||||
|
"SMB_PASSWORD": "govoplan-smb",
|
||||||
|
"MINIO_ROOT_USER": "govoplan",
|
||||||
|
"MINIO_ROOT_PASSWORD": "govoplan-minio",
|
||||||
|
"MINIO_BUCKET": "govoplan",
|
||||||
|
}
|
||||||
|
for key, value in defaults.items():
|
||||||
|
os.environ.setdefault(key, value)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_dotenv() -> None:
|
||||||
|
path = ROOT / ".env"
|
||||||
|
if not path.exists():
|
||||||
|
return
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines():
|
||||||
|
text = line.strip()
|
||||||
|
if not text or text.startswith("#") or "=" not in text:
|
||||||
|
continue
|
||||||
|
key, value = text.split("=", 1)
|
||||||
|
key = key.strip()
|
||||||
|
if not key or key in os.environ:
|
||||||
|
continue
|
||||||
|
os.environ[key] = value.strip().strip("\"'")
|
||||||
|
|
||||||
|
|
||||||
|
def _preflight_services(*, require_smb: bool, require_s3: bool) -> list[str]:
|
||||||
|
failures: list[str] = []
|
||||||
|
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
|
||||||
|
try:
|
||||||
|
response = httpx.get(nextcloud_url, timeout=3.0)
|
||||||
|
if response.status_code != 200:
|
||||||
|
failures.append(f"dev-nextcloud expected HTTP 200 at {nextcloud_url}, got HTTP {response.status_code}")
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
failures.append(f"dev-nextcloud is not reachable at {nextcloud_url}: {exc}")
|
||||||
|
|
||||||
|
webdav_url = f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/"
|
||||||
|
try:
|
||||||
|
response = httpx.get(webdav_url, timeout=3.0)
|
||||||
|
if response.status_code not in {200, 401}:
|
||||||
|
failures.append(f"dev-webdav expected HTTP 200/401 at {webdav_url}, got HTTP {response.status_code}")
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
failures.append(f"dev-webdav is not reachable at {webdav_url}: {exc}")
|
||||||
|
|
||||||
|
if require_smb:
|
||||||
|
smb_port = int(os.getenv("SMB_HOST_PORT", "1445"))
|
||||||
|
try:
|
||||||
|
with socket.create_connection(("127.0.0.1", smb_port), timeout=3.0):
|
||||||
|
pass
|
||||||
|
except OSError as exc:
|
||||||
|
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
|
||||||
|
if require_s3:
|
||||||
|
minio_url = f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}/minio/health/live"
|
||||||
|
try:
|
||||||
|
response = httpx.get(minio_url, timeout=3.0)
|
||||||
|
if response.status_code != 200:
|
||||||
|
failures.append(f"dev-s3 expected HTTP 200 at {minio_url}, got HTTP {response.status_code}")
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
failures.append(f"dev-s3 is not reachable at {minio_url}: {exc}")
|
||||||
|
return failures
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_payloads() -> list[dict[str, object]]:
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"id": "dev-webdav",
|
||||||
|
"provider": "webdav",
|
||||||
|
"endpoint_url": f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": os.getenv("WEBDAV_USER", "govoplan"),
|
||||||
|
"password_env": "WEBDAV_PASSWORD",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-nextcloud",
|
||||||
|
"provider": "nextcloud",
|
||||||
|
"endpoint_url": f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": os.getenv("NEXTCLOUD_ADMIN_USER", "admin"),
|
||||||
|
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-smb",
|
||||||
|
"provider": "smb",
|
||||||
|
"endpoint_url": f"smb://127.0.0.1:{os.getenv('SMB_HOST_PORT', '1445')}/{os.getenv('SMB_SHARE_NAME', 'files')}",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": os.getenv("SMB_USER", "govoplan"),
|
||||||
|
"password_env": "SMB_PASSWORD",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "dev-s3",
|
||||||
|
"provider": "s3",
|
||||||
|
"endpoint_url": f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||||
|
"credential_mode": "basic",
|
||||||
|
"username": os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||||
|
"password_env": "MINIO_ROOT_PASSWORD",
|
||||||
|
"metadata": {
|
||||||
|
"bucket": os.getenv("MINIO_BUCKET", "govoplan"),
|
||||||
|
"region": "us-east-1",
|
||||||
|
"path_style": True,
|
||||||
|
"verify_tls": False,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _exercise_profile(profile, *, folder: str, file_path: str, expected: str) -> None:
|
||||||
|
items = browse_connector_profile(profile, path=folder)
|
||||||
|
paths = {item.path for item in items}
|
||||||
|
if file_path not in paths:
|
||||||
|
raise RuntimeError(f"{file_path!r} not found; saw {sorted(paths)!r}")
|
||||||
|
downloaded = read_connector_file(profile, library_id="", path=file_path, max_bytes=1024 * 1024)
|
||||||
|
text = downloaded.data.decode("utf-8").strip()
|
||||||
|
if text != expected:
|
||||||
|
raise RuntimeError(f"{file_path!r} content mismatch: {text!r}")
|
||||||
|
print(f"OK {profile.id} browse/import {file_path}")
|
||||||
|
|
||||||
|
|
||||||
|
def _debug_smb(profile) -> None:
|
||||||
|
try:
|
||||||
|
import smbclient
|
||||||
|
except ImportError as exc:
|
||||||
|
print(f"DEBUG smbclient import failed: {exc}")
|
||||||
|
return
|
||||||
|
from govoplan_files.backend.storage.connector_browse import _smb_client_kwargs, _smb_location, _smb_unc_path
|
||||||
|
|
||||||
|
location = _smb_location(profile)
|
||||||
|
kwargs = _smb_client_kwargs(profile, location)
|
||||||
|
print(f"DEBUG SMB endpoint=//{location.server}:{location.port}/{location.share} root_path={location.root_path!r}")
|
||||||
|
print(f"DEBUG SMB user={kwargs.get('username')!r} require_signing={kwargs.get('require_signing')!r} auth_protocol={kwargs.get('auth_protocol')!r}")
|
||||||
|
try:
|
||||||
|
smbclient.reset_connection_cache()
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"DEBUG SMB reset cache failed: {exc}")
|
||||||
|
for path in (_smb_unc_path(location, ""), _smb_unc_path(location, "GovOPlaN")):
|
||||||
|
try:
|
||||||
|
print(f"DEBUG SMB list {path}: {smbclient.listdir(path, **kwargs)}")
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"DEBUG SMB list {path} failed: {type(exc).__name__}: {exc}")
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_webdav_fixture() -> None:
|
||||||
|
path = ROOT / "data" / "webdav" / "GovOPlaN"
|
||||||
|
path.mkdir(parents=True, exist_ok=True)
|
||||||
|
(path / "webdav-live.txt").write_text("webdav live fixture\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_smb_fixture() -> None:
|
||||||
|
path = ROOT / "data" / "smb" / "GovOPlaN"
|
||||||
|
path.mkdir(parents=True, exist_ok=True)
|
||||||
|
(path / "smb-live.txt").write_text("smb live fixture\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_nextcloud_fixture() -> None:
|
||||||
|
base_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/GovOPlaN"
|
||||||
|
auth = (os.getenv("NEXTCLOUD_ADMIN_USER", "admin"), os.getenv("NEXTCLOUD_ADMIN_PASSWORD", "govoplan-nextcloud-admin"))
|
||||||
|
response = httpx.request("MKCOL", base_url, auth=auth, timeout=15.0)
|
||||||
|
if response.status_code not in {201, 405}:
|
||||||
|
raise RuntimeError(f"Nextcloud MKCOL failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||||
|
response = httpx.put(f"{base_url}/nextcloud-live.txt", content=b"nextcloud live fixture\n", auth=auth, timeout=15.0)
|
||||||
|
if response.status_code not in {200, 201, 204}:
|
||||||
|
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_s3_fixture() -> None:
|
||||||
|
try:
|
||||||
|
import boto3
|
||||||
|
from botocore.config import Config
|
||||||
|
from botocore.exceptions import ClientError
|
||||||
|
except ImportError as exc:
|
||||||
|
raise RuntimeError("boto3 is not installed; install govoplan-files[s3]") from exc
|
||||||
|
bucket = os.getenv("MINIO_BUCKET", "govoplan")
|
||||||
|
client = boto3.client(
|
||||||
|
"s3",
|
||||||
|
endpoint_url=f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||||
|
aws_access_key_id=os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||||
|
aws_secret_access_key=os.getenv("MINIO_ROOT_PASSWORD", "govoplan-minio"),
|
||||||
|
region_name="us-east-1",
|
||||||
|
config=Config(s3={"addressing_style": "path"}),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
client.create_bucket(Bucket=bucket)
|
||||||
|
except ClientError as exc:
|
||||||
|
code = exc.response.get("Error", {}).get("Code")
|
||||||
|
if code not in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
|
||||||
|
raise
|
||||||
|
client.put_object(Bucket=bucket, Key="GovOPlaN/s3-live.txt", Body=b"s3 live fixture\n", ContentType="text/plain")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# File Connector Boundary
|
||||||
|
|
||||||
|
GovOPlaN Files owns the managed-file boundary: frozen blobs, versions, shares,
|
||||||
|
campaign attachment evidence, provenance, connector policy, connector profiles,
|
||||||
|
and read-only browse/import APIs that turn external files into managed
|
||||||
|
GovOPlaN files.
|
||||||
|
|
||||||
|
The built-in connector layer is intentionally on-demand. It does not run remote
|
||||||
|
indexing, background sync, remote mutation, or upstream permission management.
|
||||||
|
Connectors may browse an external source, import one selected file, freeze it as
|
||||||
|
a managed file, record provenance, and audit the access.
|
||||||
|
|
||||||
|
## Built-In Baseline
|
||||||
|
|
||||||
|
The files module may keep small baseline providers when they are needed for
|
||||||
|
normal product workflows and can share the same governance model:
|
||||||
|
|
||||||
|
- Seafile through the native read/download API
|
||||||
|
- Nextcloud through WebDAV
|
||||||
|
- generic WebDAV
|
||||||
|
- SMB through the optional `smb` extra
|
||||||
|
- S3-compatible stores through the optional `s3` extra
|
||||||
|
|
||||||
|
These providers are surfaced through connector descriptors at
|
||||||
|
`GET /api/v1/files/connectors/providers`. Provider descriptors declare whether
|
||||||
|
the provider is implemented, whether its optional dependency is installed, and
|
||||||
|
which browse/import behaviors are available.
|
||||||
|
|
||||||
|
## Separate Connector Module Candidates
|
||||||
|
|
||||||
|
A separate connector module becomes appropriate when integration needs exceed
|
||||||
|
on-demand browse/import:
|
||||||
|
|
||||||
|
- background indexing or synchronization
|
||||||
|
- bidirectional remote mutation
|
||||||
|
- long-running transfer workers
|
||||||
|
- provider-specific credential lifecycle or OAuth flows
|
||||||
|
- DMS/eAkte metadata models, registers, retention, or filing plans
|
||||||
|
- S3-compatible object store administration
|
||||||
|
- NFS host-mount lifecycle or sidecar coordination
|
||||||
|
- provider-specific WebUI administration beyond profile fields
|
||||||
|
|
||||||
|
In that model, `govoplan-files` should keep the managed-file import contract,
|
||||||
|
policy checks, provenance model, and audit events. A connector module should own
|
||||||
|
provider-specific discovery, synchronization, credentials, health checks, and
|
||||||
|
any remote write behavior.
|
||||||
|
|
||||||
|
## Provider Responsibilities
|
||||||
|
|
||||||
|
Every provider must:
|
||||||
|
|
||||||
|
- enforce GovOPlaN profile visibility and connector policy before browse/import
|
||||||
|
- keep credentials as encrypted values or scoped secret references, never API
|
||||||
|
response values; process-environment references are allowed only in
|
||||||
|
deployment-owned profiles with an exact deployment allowlist
|
||||||
|
- import external files into managed storage before they are used in campaigns
|
||||||
|
or workflows
|
||||||
|
- preserve source provenance and revision metadata
|
||||||
|
- emit connector audit events for imported or accessed files
|
||||||
|
- treat remote ACLs as upstream checks, not as a replacement for GovOPlaN policy
|
||||||
|
- use a transport that pins every connection to a policy-validated DNS/IP answer
|
||||||
|
and revalidates redirects; SDK transports without that guarantee fail closed
|
||||||
|
|
||||||
|
SMB initial connections, reconnects, aliases, and DFS referral targets are
|
||||||
|
created through a Files-owned pinned `smbprotocol` transport. S3 HTTP/HTTPS pools
|
||||||
|
use the equivalent botocore adapter for every connection selected by retries,
|
||||||
|
redirects, endpoint discovery, and virtual-host addressing. Both adapters apply
|
||||||
|
the deployment-wide private-network policy at socket creation. S3 keeps the
|
||||||
|
configured hostname for TLS SNI and certificate verification, but does not use
|
||||||
|
outbound proxies or ambient AWS credential discovery. An incompatible optional
|
||||||
|
SDK release fails closed before a usable session or client is returned.
|
||||||
|
|
||||||
|
## Non-Goals For Files
|
||||||
|
|
||||||
|
Files does not own collaborative editing, comments, document review workflows,
|
||||||
|
remote lock orchestration, DMS records management, or external system data
|
||||||
|
models. Those belong to future document, workflow, DMS, or connector modules and
|
||||||
|
should integrate through capabilities and managed-file imports.
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
# Connector Spaces
|
||||||
|
|
||||||
|
GovOPlaN Files should treat external file shares as two separate product
|
||||||
|
objects:
|
||||||
|
|
||||||
|
1. A governed connection profile defines a reusable remote endpoint such as a
|
||||||
|
WebDAV server, Nextcloud account, Seafile server, or SMB share. Profiles are
|
||||||
|
administered in settings at system or tenant scope, with the same
|
||||||
|
inheritance and limit semantics used by mail-server profiles.
|
||||||
|
2. A governed credential profile defines reusable authentication material for
|
||||||
|
one provider or for any compatible provider. Credentials are administered
|
||||||
|
separately from connection profiles and can carry their own allow/deny
|
||||||
|
policy.
|
||||||
|
3. A linked file space binds one concrete remote folder or library path from an
|
||||||
|
allowed profile to a user or group. Linked spaces appear beside "My files"
|
||||||
|
and group file spaces in the files module.
|
||||||
|
|
||||||
|
This keeps secrets, endpoint governance, and tenant limits in settings, while
|
||||||
|
keeping user/group workspaces and concrete folder choices in the files module.
|
||||||
|
|
||||||
|
## Model
|
||||||
|
|
||||||
|
Connection profile:
|
||||||
|
|
||||||
|
- provider: `seafile`, `nextcloud`, `webdav`, `smb`, or a future provider
|
||||||
|
- endpoint URL and optional base path
|
||||||
|
- scope: `system` or `tenant` for administered profiles; user/group profiles
|
||||||
|
may be allowed later only when policy explicitly permits them
|
||||||
|
- optional credential profile id
|
||||||
|
- capabilities: browse, sync, import, optional write when a provider supports it
|
||||||
|
- profile-local policy for allow/deny rules
|
||||||
|
|
||||||
|
Credential profile:
|
||||||
|
|
||||||
|
- provider: a specific provider or any provider
|
||||||
|
- scope: `system` or `tenant` for administered credentials
|
||||||
|
- credential mode: anonymous, scoped secret reference, or encrypted stored
|
||||||
|
password/token; environment references are reserved for deployment-owned JSON
|
||||||
|
profiles and exact deployment allowlisting
|
||||||
|
- username and redacted secret configuration
|
||||||
|
- credential-local policy for allow/deny rules
|
||||||
|
|
||||||
|
Connector policy:
|
||||||
|
|
||||||
|
- system policy is the baseline
|
||||||
|
- tenant policy inherits system policy and may narrow it unless system allows
|
||||||
|
lower-level relaxation
|
||||||
|
- future user/group policy may narrow tenant policy for self-service links
|
||||||
|
- policies can allow or block providers, profile ids, endpoint URLs, external
|
||||||
|
path prefixes, credential ids, credential inheritance, and local linked-space
|
||||||
|
creation
|
||||||
|
|
||||||
|
Linked connector space:
|
||||||
|
|
||||||
|
- owner type: user or group
|
||||||
|
- display name
|
||||||
|
- connector profile id
|
||||||
|
- remote library id or share id
|
||||||
|
- remote root path
|
||||||
|
- sync mode: manual initially; background sync is future work
|
||||||
|
- read-only flag from provider/policy
|
||||||
|
- active/deleted state
|
||||||
|
|
||||||
|
The linked space should be addressable as a normal file space in the files UI.
|
||||||
|
For the first implementation slice, actions may use the existing connector
|
||||||
|
browse/sync APIs and managed file storage. A linked space can therefore browse
|
||||||
|
the remote folder and sync selected files into managed storage. Later slices can
|
||||||
|
add a native remote listing view or background sync jobs.
|
||||||
|
|
||||||
|
## Policy Semantics
|
||||||
|
|
||||||
|
Use mail-profile terminology because administrators already see it there:
|
||||||
|
|
||||||
|
- must use: lower scopes are restricted to selected profile ids or providers
|
||||||
|
- can use: lower scopes may choose from inherited allowed profiles
|
||||||
|
- shall not use anything outside: endpoint URLs and path prefixes are enforced
|
||||||
|
by deny/allow rules before browse, import, or sync
|
||||||
|
- may create local links: controls whether users/groups can add linked spaces
|
||||||
|
from inherited profiles
|
||||||
|
|
||||||
|
Connector policy should provide an explainable effective policy response with
|
||||||
|
source path entries: system, tenant, user, group, campaign when applicable.
|
||||||
|
|
||||||
|
## Current State
|
||||||
|
|
||||||
|
Implemented:
|
||||||
|
|
||||||
|
- provider descriptors for Seafile, Nextcloud, WebDAV, and SMB
|
||||||
|
- database-backed connector profiles with system and tenant scope
|
||||||
|
- database-backed connector credentials with system and tenant scope
|
||||||
|
- encrypted stored password/token support for database credentials
|
||||||
|
- JSON/environment-defined connector profiles with system, tenant, user, group,
|
||||||
|
and campaign visibility
|
||||||
|
- connector allow/deny policy enforcement before browse/import/sync, including
|
||||||
|
provider, connection profile, credential profile, and path checks
|
||||||
|
- read-only browse endpoints
|
||||||
|
- import and sync into managed files with provenance and revision metadata
|
||||||
|
- audit events for connector import, sync, and access
|
||||||
|
- settings/admin UI sections for system and tenant file connections and
|
||||||
|
credentials
|
||||||
|
- linked connector-space rows owned by users or groups
|
||||||
|
- file-space API responses that include linked connector spaces
|
||||||
|
- Files UI entry point to create linked connector spaces from a browsed remote
|
||||||
|
profile/folder
|
||||||
|
- Files UI sync dialog for choosing a profile, browsing a remote folder, and
|
||||||
|
syncing a selected file into a managed destination folder
|
||||||
|
- Files UI connector-space view with provider/read-only/manual-sync state
|
||||||
|
- Docker dev stack smoke checks for WebDAV, Nextcloud, and SMB
|
||||||
|
|
||||||
|
Missing:
|
||||||
|
|
||||||
|
- explicit connector profile policy rows equivalent to mail-profile policies
|
||||||
|
- effective connector policy/explain UI equivalent to mail-profile policies
|
||||||
|
- edit/manage actions for existing linked connector spaces
|
||||||
|
- optional background sync or remote-write semantics
|
||||||
|
|
||||||
|
## Implementation Phases
|
||||||
|
|
||||||
|
1. Persist governed connector profiles and policies.
|
||||||
|
- `file_connector_profiles` exists for system and tenant profiles.
|
||||||
|
- `file_connector_credentials` exists for system and tenant credentials.
|
||||||
|
- Environment JSON profiles remain bootstrap/compatibility profiles.
|
||||||
|
- Profile and credential CRUD endpoints exist for database-backed records.
|
||||||
|
- Remaining: `file_connector_policies` plus effective policy/explain output.
|
||||||
|
|
||||||
|
2. Add linked connector spaces.
|
||||||
|
- `file_connector_spaces` exists with owner user/group, profile id, library
|
||||||
|
id, remote path, display label, sync mode, and active state.
|
||||||
|
- Connector policy is enforced before creating or using a link.
|
||||||
|
- Linked connector spaces are returned from `/api/v1/files/spaces`.
|
||||||
|
|
||||||
|
3. Surface linked spaces in the Files UI.
|
||||||
|
- Linked spaces appear beside managed user/group spaces.
|
||||||
|
- The add-space dialog browses an allowed profile and links the current
|
||||||
|
remote folder/library root.
|
||||||
|
- The connector browser/sync flow is reused when a linked space is open.
|
||||||
|
- Remaining: edit/manage actions for existing linked spaces.
|
||||||
|
|
||||||
|
4. Add sync orchestration.
|
||||||
|
- Manual sync selected file is already available.
|
||||||
|
- Add folder-level manual sync.
|
||||||
|
- Add optional scheduled/background sync with conflict reporting.
|
||||||
|
|
||||||
|
5. Add provider-specific expansion.
|
||||||
|
- OAuth/secret-store credentials.
|
||||||
|
- Remote write and delete only where policy and provider support it.
|
||||||
|
- DMS/eAkte metadata integrations in a separate connector or documents
|
||||||
|
module.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# Document Collaboration Boundary
|
||||||
|
|
||||||
|
GovOPlaN Files is the governed managed-file module. It stores uploaded/imported
|
||||||
|
blobs, versions, shares, provenance, ZIP imports/exports, connector imports, and
|
||||||
|
campaign attachment evidence. It should stay reliable, auditable, and simple.
|
||||||
|
|
||||||
|
Collaborative document behavior should be owned by a future documents module or
|
||||||
|
by provider-specific connector modules when the behavior is delegated to systems
|
||||||
|
such as Nextcloud, Seafile, Collabora, OnlyOffice, OpenDesk, or DMS/eAkte
|
||||||
|
platforms.
|
||||||
|
|
||||||
|
## Files Owns
|
||||||
|
|
||||||
|
- managed blobs and immutable version evidence
|
||||||
|
- owner-scoped user/group file spaces
|
||||||
|
- shares and access checks for managed files
|
||||||
|
- connector browse/import into managed storage
|
||||||
|
- source provenance, source revision, and audit events
|
||||||
|
- freeze-before-send campaign attachment behavior
|
||||||
|
- previews generated from managed file versions
|
||||||
|
|
||||||
|
## Documents Or Connectors Should Own
|
||||||
|
|
||||||
|
- co-editing sessions and editor launch URLs
|
||||||
|
- comments, suggestions, document tasks, and review state
|
||||||
|
- check-in/check-out, remote locks, and lock timeouts
|
||||||
|
- semantic document versions beyond blob versions
|
||||||
|
- template merge flows and generated-document lifecycle
|
||||||
|
- external DMS metadata, filing plans, record categories, and retention classes
|
||||||
|
- provider-specific sync state and conflict resolution
|
||||||
|
- collaborative presence and notification behavior
|
||||||
|
|
||||||
|
## Integration Shape
|
||||||
|
|
||||||
|
The documents layer should integrate with files through stable contracts:
|
||||||
|
|
||||||
|
- create or import a managed file version for evidence points
|
||||||
|
- attach source provenance when a file came from an external editor or DMS
|
||||||
|
- ask files for read/download access rather than importing files internals
|
||||||
|
- emit workflow/audit events when a collaborative document reaches a governed
|
||||||
|
state such as reviewed, approved, frozen, sent, or archived
|
||||||
|
- use connector providers for provider-specific browse/import/write behavior
|
||||||
|
|
||||||
|
Files may expose links to a document or connector module, but it should not own
|
||||||
|
the collaboration state machine. A collaborative document can produce many
|
||||||
|
working states; GovOPlaN Files should persist the governed snapshots that other
|
||||||
|
modules can safely use as evidence.
|
||||||
|
|
||||||
|
## Practical Rule
|
||||||
|
|
||||||
|
If a feature is about storing, sharing, importing, downloading, or freezing a
|
||||||
|
file, it belongs in `govoplan-files`. If a feature is about people jointly
|
||||||
|
editing, reviewing, locking, commenting on, or routing a document through a
|
||||||
|
semantic process, it belongs in a documents/workflow/DMS integration module.
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
# Generated Artifact Store
|
||||||
|
|
||||||
|
Files implements Core's optional `files.artifact_store` capability for modules
|
||||||
|
that generate deterministic output without owning file storage.
|
||||||
|
|
||||||
|
The producer supplies bytes, filename, content type, destination folder,
|
||||||
|
optional idempotency key, and bounded non-secret provenance. Files applies the
|
||||||
|
actor's `files:file:upload` permission, tenant/user ownership, path rules,
|
||||||
|
versioning, configured blob backend, and conflict behavior. An idempotency key
|
||||||
|
is represented as source provenance so an unchanged retry does not create an
|
||||||
|
unrelated file version.
|
||||||
|
|
||||||
|
The shared Files session owns finalization. Before a new managed object is
|
||||||
|
written, Files commits a lease-fenced Core recovery operation containing only
|
||||||
|
identifiers and digests. The caller's eventual session commit independently
|
||||||
|
verifies both `FileBlob` metadata and stored bytes; rollback compensates only an
|
||||||
|
unreferenced key. Producers must therefore complete the supplied transaction
|
||||||
|
normally and must not bypass or replace Files session lifecycle handling.
|
||||||
|
|
||||||
|
The response contains only file/version identifiers, display path, media type,
|
||||||
|
size, digest, and storage provenance. Producers must not put credentials,
|
||||||
|
tokens, or rendered plaintext into metadata. Storing an artifact proves Files
|
||||||
|
accepted it; it does not prove printing, mailing, or any other external effect.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Files Interface Pattern Migration
|
||||||
|
|
||||||
|
This inventory records the Files-owned part of the GovOPlaN interface pattern
|
||||||
|
language. Core owns the shell and shared components; Files owns the composition
|
||||||
|
and consequences described here.
|
||||||
|
|
||||||
|
## Surface inventory
|
||||||
|
|
||||||
|
| Surface | Primary task | Archetype | Consequence | Pattern evidence |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| `/files` space and folder panes | Browse managed and connected content without losing location | Directory/explorer | Low for navigation; medium for exposing filenames and provenance | Full-height two-pane workspace, bounded panes, stable selection and contextual Help Center link |
|
||||||
|
| `/files` toolbar and property filters | Find and act on the current selection | Explorer actions and local filtering | Medium for upload, move, copy, share and synchronization; high for delete | Actions remain beside the affected list, disabled controls explain permission/state/selection blockers, destructive work uses `ConfirmDialog` |
|
||||||
|
| Upload/archive, transfer, rename and connector-import dialogs | Supply and review one bounded change | Adaptive create/edit or guided import | Medium to high because files, paths and external bytes change | Shared `Dialog`, `FileDropZone`, validation, conflict review, unsaved inputs and explicit confirmation |
|
||||||
|
| File share dialog | Inspect and change access | Review/decision | High because another actor gains access | Shared dialog, access explanation, stable row actions and destructive confirmation |
|
||||||
|
| System/tenant/group/user connector surfaces | Compare connections, credentials and effective policy | Administration/configuration | High because endpoints, secrets and inherited policy control external access | Shared `ConnectionTree`, adaptive forms, `ActionBlockerHint`, policy provenance and contextual admin help |
|
||||||
|
| Connection and credential editors | Create or edit one governed endpoint or secret | Adaptive create/edit | High because a saved change may enable remote access | Relevant fields only, typed credential controls, discovery/test, advanced compatibility section, unsaved-change guard and disabled-save reasons |
|
||||||
|
| Connector policy card | Narrow inherited connector access | Effective-policy editor | High because deny/allow changes affect lower scopes | Typed reference selectors, deny precedence warning, effective source evidence and permission blocker |
|
||||||
|
| `files.widget.spaces` | See available managed/connected spaces and open Files | Dashboard widget | Low; names and provider state may still be sensitive | Shared loading/alert/status components, bounded item count, permission-filtered contribution |
|
||||||
|
| Files chooser capability used by another module | Select a managed snapshot without importing Files internals | Directory chooser | Medium because the exact selected version becomes another module's input | Shared dialog/confirmation, capability boundary and exact file/version evidence |
|
||||||
|
|
||||||
|
## State and consequence contract
|
||||||
|
|
||||||
|
- Loading, errors, success, empty results, access explanations and confirmation
|
||||||
|
use Core components. Files does not reproduce the application shell.
|
||||||
|
- A connector that comes from deployment settings remains visible but read-only;
|
||||||
|
its action explains that bootstrap configuration and a restart are required.
|
||||||
|
- Missing permission, target, selection, endpoint, or compatible provider is an
|
||||||
|
explained disabled state. It is not represented only by color or absence.
|
||||||
|
- Provider metadata JSON is an expert compatibility escape hatch inside the
|
||||||
|
collapsed shared advanced-options component. Ordinary connector setup uses
|
||||||
|
typed provider, endpoint, credential, capability and policy controls.
|
||||||
|
- Endpoint discovery and credential tests are explicit and report their result;
|
||||||
|
they do not save the draft. Save remains the only committing action.
|
||||||
|
- Connector/profile disable and managed-file delete remain confirmed actions and
|
||||||
|
state their immediate effect. Soft deletion must not be described as purge.
|
||||||
|
- External connector data, paths and credential references are rendered only in
|
||||||
|
already-authorized administration or explorer contexts. Secret values are
|
||||||
|
never returned for rendering.
|
||||||
|
|
||||||
|
## Accessibility and responsive evidence
|
||||||
|
|
||||||
|
Shared `Dialog` owns focus entry, Escape handling and focus return. Form and
|
||||||
|
toolbar DOM order is the keyboard order; disabled-action tooltips are themselves
|
||||||
|
focusable and expose the reason. The connector form uses semantic sections and
|
||||||
|
labels, status is textual as well as colored, and result alerts are announced by
|
||||||
|
the shared alert component. The explorer collapses to one column below 1050 px;
|
||||||
|
connector forms and action rows collapse below 760 px while preserving source
|
||||||
|
order. Long provider choices scroll inside the segmented control rather than
|
||||||
|
expanding the page.
|
||||||
|
|
||||||
|
The focused structural test guards these contracts, optional-module boundaries,
|
||||||
|
confirmation, contextual help, advanced-only JSON and responsive rules. Core's
|
||||||
|
TypeScript build, structural localization audit, module-permutation suite and
|
||||||
|
full-product bundle check provide the integration gates.
|
||||||
|
|
||||||
+8
-8
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@govoplan/files-webui",
|
"name": "@govoplan/files-webui",
|
||||||
"version": "0.1.1",
|
"version": "0.1.19",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "webui/src/index.ts",
|
"main": "webui/src/index.ts",
|
||||||
@@ -19,14 +19,14 @@
|
|||||||
"LICENSE"
|
"LICENSE"
|
||||||
],
|
],
|
||||||
"peerDependencies": {
|
"peerDependencies": {
|
||||||
"@govoplan/core-webui": "^0.1.1",
|
"@vitejs/plugin-react": "^5.2.0",
|
||||||
"lucide-react": "^0.555.0",
|
"vite": "^7.3.6",
|
||||||
"react": "^19.0.0",
|
|
||||||
"react-dom": "^19.0.0",
|
|
||||||
"react-router-dom": "^7.1.1",
|
|
||||||
"@vitejs/plugin-react": "^4.3.4",
|
|
||||||
"typescript": "^5.7.2",
|
"typescript": "^5.7.2",
|
||||||
"vite": "^6.0.6"
|
"react": ">=19.2.7 <20",
|
||||||
|
"react-dom": ">=19.2.7 <20",
|
||||||
|
"react-router": ">=8.3.0 <9",
|
||||||
|
"lucide-react": "^1.23.0",
|
||||||
|
"@govoplan/core-webui": "^0.1.18"
|
||||||
},
|
},
|
||||||
"peerDependenciesMeta": {
|
"peerDependenciesMeta": {
|
||||||
"@govoplan/core-webui": {
|
"@govoplan/core-webui": {
|
||||||
|
|||||||
+13
-2
@@ -4,14 +4,25 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "govoplan-files"
|
name = "govoplan-files"
|
||||||
version = "0.1.2"
|
version = "0.1.19"
|
||||||
description = "GovOPlaN files module with backend and WebUI integration."
|
description = "GovOPlaN files module with backend and WebUI integration."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
license = { file = "LICENSE" }
|
license = { file = "LICENSE" }
|
||||||
authors = [{ name = "GovOPlaN" }]
|
authors = [{ name = "GovOPlaN" }]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"govoplan-core>=0.1.2",
|
"govoplan-core>=0.1.19",
|
||||||
|
"defusedxml>=0.7,<1",
|
||||||
|
"pyzipper>=0.3.6,<1",
|
||||||
|
"python-multipart>=0.0.31,<1",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
s3 = [
|
||||||
|
"boto3>=1.34,<2",
|
||||||
|
]
|
||||||
|
smb = [
|
||||||
|
"smbprotocol>=1.13",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.setuptools.packages.find]
|
[tool.setuptools.packages.find]
|
||||||
|
|||||||
@@ -1,17 +1,68 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import binascii
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from sqlalchemy import or_
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.audit.logging import audit_from_principal
|
||||||
|
from govoplan_core.auth import ApiPrincipal, has_scope
|
||||||
|
from govoplan_core.core.access import AccessDecisionProvenance, PrincipalRef
|
||||||
|
from govoplan_core.core.files import FileAccessProvider
|
||||||
|
from govoplan_core.core.files import (
|
||||||
|
ManagedTabularFile,
|
||||||
|
ManagedTabularFileAccessError,
|
||||||
|
ManagedTabularFileContent,
|
||||||
|
ManagedTabularFileNotFoundError,
|
||||||
|
ManagedTabularFileProvider,
|
||||||
|
ManagedTabularFileUnavailableError,
|
||||||
|
ManagedTabularFileValidationError,
|
||||||
|
PostboxFileReferenceRef,
|
||||||
|
PostboxFileReferenceRequest,
|
||||||
|
PostboxFileReferenceProvider,
|
||||||
|
ManagedArtifactRef,
|
||||||
|
ManagedArtifactStore,
|
||||||
|
ManagedArtifactWriteRequest,
|
||||||
|
)
|
||||||
from govoplan_core.core.modules import ModuleContext
|
from govoplan_core.core.modules import ModuleContext
|
||||||
|
from govoplan_core.security.module_permissions import scopes_grant_compatible
|
||||||
|
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileFolder, FileShare, FileVersion
|
||||||
from govoplan_files.backend.runtime import configure_runtime
|
from govoplan_files.backend.runtime import configure_runtime
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
from govoplan_files.backend.storage.campaign_attachments import (
|
from govoplan_files.backend.storage.campaign_attachments import (
|
||||||
annotate_built_messages_with_managed_files,
|
annotate_built_messages_with_managed_files,
|
||||||
managed_match_payloads,
|
managed_match_payloads,
|
||||||
prepared_campaign_snapshot,
|
prepared_campaign_snapshot,
|
||||||
public_attachment_summary_payload,
|
public_attachment_summary_payload,
|
||||||
|
share_assets_with_campaign,
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
|
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
|
||||||
from govoplan_files.backend.storage.files import current_version_and_blob
|
from govoplan_files.backend.storage.files import (
|
||||||
|
create_file_asset,
|
||||||
|
current_version_and_blob,
|
||||||
|
get_asset_for_user,
|
||||||
|
list_recent_assets_for_user,
|
||||||
|
read_asset_version_bytes,
|
||||||
|
sync_file_asset_from_source,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.paths import normalize_folder
|
||||||
|
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||||
|
|
||||||
|
|
||||||
|
VIRTUAL_FOLDER_RESOURCE_PREFIX = "virtual-folder:v1"
|
||||||
|
|
||||||
|
WRITE_ACTIONS = {
|
||||||
|
"files:file:upload",
|
||||||
|
"files:file:organize",
|
||||||
|
"files:file:share",
|
||||||
|
"files:file:delete",
|
||||||
|
"files:upload",
|
||||||
|
"files:organize",
|
||||||
|
"files:share",
|
||||||
|
"files:delete",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
class FilesCampaignCapability:
|
class FilesCampaignCapability:
|
||||||
@@ -21,6 +72,7 @@ class FilesCampaignCapability:
|
|||||||
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
|
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
|
||||||
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
|
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
|
||||||
current_version_and_blob = staticmethod(current_version_and_blob)
|
current_version_and_blob = staticmethod(current_version_and_blob)
|
||||||
|
share_assets_with_campaign = staticmethod(share_assets_with_campaign)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:
|
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:
|
||||||
@@ -30,5 +82,679 @@ class FilesCampaignCapability:
|
|||||||
|
|
||||||
|
|
||||||
def campaign_capability(context: ModuleContext) -> FilesCampaignCapability:
|
def campaign_capability(context: ModuleContext) -> FilesCampaignCapability:
|
||||||
configure_runtime(settings=context.settings)
|
configure_runtime(registry=context.registry, settings=context.settings)
|
||||||
return FilesCampaignCapability()
|
return FilesCampaignCapability()
|
||||||
|
|
||||||
|
|
||||||
|
class FilesArtifactStore(ManagedArtifactStore):
|
||||||
|
def store_artifact(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
request: ManagedArtifactWriteRequest,
|
||||||
|
) -> ManagedArtifactRef:
|
||||||
|
if not hasattr(session, "query") or not hasattr(session, "flush"):
|
||||||
|
raise TypeError("Files artifact storage requires a SQLAlchemy session.")
|
||||||
|
if not hasattr(principal, "has") or not principal.has("files:file:upload"):
|
||||||
|
raise PermissionError("Managed artifact storage requires files:file:upload.")
|
||||||
|
user = getattr(principal, "user", None)
|
||||||
|
user_id = str(getattr(user, "id", "") or "")
|
||||||
|
tenant_id = str(getattr(principal, "tenant_id", "") or "")
|
||||||
|
if not user_id or not tenant_id:
|
||||||
|
raise PermissionError("Managed artifact storage requires a tenant user principal.")
|
||||||
|
metadata = dict(request.metadata)
|
||||||
|
if request.idempotency_key:
|
||||||
|
metadata["source_provenance"] = {
|
||||||
|
"source_type": "generated_artifact",
|
||||||
|
"connector_id": "files.artifact_store",
|
||||||
|
"provider": str(metadata.get("producer_module") or "platform"),
|
||||||
|
"external_id": request.idempotency_key,
|
||||||
|
"revision": str(metadata.get("output_sha256") or "") or None,
|
||||||
|
}
|
||||||
|
stored, _action, _previous_version_id = sync_file_asset_from_source(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type="user",
|
||||||
|
owner_id=user_id,
|
||||||
|
user_id=user_id,
|
||||||
|
filename=request.filename,
|
||||||
|
data=request.payload,
|
||||||
|
metadata=metadata,
|
||||||
|
folder=request.folder,
|
||||||
|
content_type=request.content_type,
|
||||||
|
conflict_strategy="rename",
|
||||||
|
is_admin=principal.has("files:file:admin"),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
stored = create_file_asset(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type="user",
|
||||||
|
owner_id=user_id,
|
||||||
|
user_id=user_id,
|
||||||
|
filename=request.filename,
|
||||||
|
data=request.payload,
|
||||||
|
folder=request.folder,
|
||||||
|
content_type=request.content_type,
|
||||||
|
description=request.description,
|
||||||
|
metadata=metadata,
|
||||||
|
conflict_strategy="rename",
|
||||||
|
is_admin=principal.has("files:file:admin"),
|
||||||
|
)
|
||||||
|
return ManagedArtifactRef(
|
||||||
|
file_asset_id=stored.asset.id,
|
||||||
|
file_version_id=stored.version.id,
|
||||||
|
filename=stored.version.filename_at_upload,
|
||||||
|
display_path=stored.asset.display_path,
|
||||||
|
content_type=stored.version.content_type or request.content_type,
|
||||||
|
size_bytes=stored.version.size_bytes,
|
||||||
|
sha256=stored.version.checksum_sha256,
|
||||||
|
provenance={
|
||||||
|
"module": "files",
|
||||||
|
"owner_type": stored.asset.owner_type,
|
||||||
|
"managed": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def artifact_store_capability(context: ModuleContext) -> FilesArtifactStore:
|
||||||
|
configure_runtime(registry=context.registry, settings=context.settings)
|
||||||
|
return FilesArtifactStore()
|
||||||
|
|
||||||
|
|
||||||
|
class FilesPostboxReferenceService(PostboxFileReferenceProvider):
|
||||||
|
"""Resolve Postbox evidence without turning message access into file access."""
|
||||||
|
|
||||||
|
def resolve_postbox_references(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
requests: tuple[PostboxFileReferenceRequest, ...],
|
||||||
|
) -> tuple[PostboxFileReferenceRef, ...]:
|
||||||
|
if not hasattr(session, "get"):
|
||||||
|
raise TypeError("Postbox file resolution requires a SQLAlchemy session.")
|
||||||
|
principal_tenant = str(getattr(principal, "tenant_id", "") or "")
|
||||||
|
can_download = bool(
|
||||||
|
hasattr(principal, "has") and principal.has("files:file:download")
|
||||||
|
)
|
||||||
|
user = getattr(principal, "user", None)
|
||||||
|
user_id = str(getattr(user, "id", "") or "")
|
||||||
|
is_admin = bool(
|
||||||
|
hasattr(principal, "has") and principal.has("files:file:admin")
|
||||||
|
)
|
||||||
|
results: list[PostboxFileReferenceRef] = []
|
||||||
|
for request in requests:
|
||||||
|
if principal_tenant != tenant_id:
|
||||||
|
results.append(_unavailable_postbox_reference(request, "tenant_mismatch"))
|
||||||
|
continue
|
||||||
|
if not can_download or not user_id:
|
||||||
|
results.append(
|
||||||
|
_unavailable_postbox_reference(
|
||||||
|
request,
|
||||||
|
"download_permission_missing",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
reference_type = request.reference_type.strip().casefold()
|
||||||
|
version: FileVersion | None = None
|
||||||
|
if reference_type in {"file", "file_asset", "files:file"}:
|
||||||
|
results.append(
|
||||||
|
_unavailable_postbox_reference(request, "exact_version_required")
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
elif reference_type in {"file_version", "files:file_version"}:
|
||||||
|
version = session.get(FileVersion, request.reference_id) # type: ignore[attr-defined]
|
||||||
|
asset = (
|
||||||
|
session.get(FileAsset, version.file_asset_id) # type: ignore[attr-defined]
|
||||||
|
if version is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
results.append(
|
||||||
|
_unavailable_postbox_reference(
|
||||||
|
request,
|
||||||
|
"unsupported_reference_type",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if (
|
||||||
|
asset is None
|
||||||
|
or version is None
|
||||||
|
or asset.tenant_id != tenant_id
|
||||||
|
or version.tenant_id != tenant_id
|
||||||
|
or asset.deleted_at is not None
|
||||||
|
):
|
||||||
|
results.append(_unavailable_postbox_reference(request, "file_not_found"))
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
get_asset_for_user(
|
||||||
|
session, # type: ignore[arg-type]
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
asset_id=asset.id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
except FileStorageError:
|
||||||
|
results.append(
|
||||||
|
_unavailable_postbox_reference(request, "file_access_denied")
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
blob = session.get(FileBlob, version.blob_id) # type: ignore[attr-defined]
|
||||||
|
if blob is None or blob.tenant_id != tenant_id:
|
||||||
|
results.append(
|
||||||
|
_unavailable_postbox_reference(request, "file_payload_missing")
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
results.append(
|
||||||
|
PostboxFileReferenceRef(
|
||||||
|
reference_type=request.reference_type,
|
||||||
|
reference_id=request.reference_id,
|
||||||
|
available=True,
|
||||||
|
reason_code="available",
|
||||||
|
file_asset_id=asset.id,
|
||||||
|
file_version_id=version.id,
|
||||||
|
filename=version.filename_at_upload or asset.filename,
|
||||||
|
content_type=version.content_type or blob.content_type,
|
||||||
|
size_bytes=version.size_bytes,
|
||||||
|
sha256=version.checksum_sha256,
|
||||||
|
download_path=(
|
||||||
|
f"/api/v1/files/{asset.id}/versions/{version.id}/download"
|
||||||
|
),
|
||||||
|
provenance={
|
||||||
|
"module": "files",
|
||||||
|
"postbox_id": request.postbox_id,
|
||||||
|
"message_id": request.message_id,
|
||||||
|
"display_path": asset.display_path,
|
||||||
|
"exact_version": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(results)
|
||||||
|
|
||||||
|
|
||||||
|
def _unavailable_postbox_reference(
|
||||||
|
request: PostboxFileReferenceRequest,
|
||||||
|
reason_code: str,
|
||||||
|
) -> PostboxFileReferenceRef:
|
||||||
|
return PostboxFileReferenceRef(
|
||||||
|
reference_type=request.reference_type,
|
||||||
|
reference_id=request.reference_id,
|
||||||
|
available=False,
|
||||||
|
reason_code=reason_code,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def postbox_reference_capability(
|
||||||
|
context: ModuleContext,
|
||||||
|
) -> FilesPostboxReferenceService:
|
||||||
|
configure_runtime(registry=context.registry, settings=context.settings)
|
||||||
|
return FilesPostboxReferenceService()
|
||||||
|
|
||||||
|
|
||||||
|
class FilesAccessService(FileAccessProvider):
|
||||||
|
def explain_resource_provenance(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
*,
|
||||||
|
resource_type: str,
|
||||||
|
resource_id: str,
|
||||||
|
action: str,
|
||||||
|
) -> tuple[AccessDecisionProvenance, ...]:
|
||||||
|
normalized_type = resource_type.lower().strip()
|
||||||
|
if normalized_type in {"file", "file_asset", "files:file"}:
|
||||||
|
return self._explain_file(session, principal, resource_id=resource_id, action=action)
|
||||||
|
if normalized_type in {"folder", "file_folder", "files:folder"}:
|
||||||
|
return self._explain_folder(session, principal, resource_id=resource_id, action=action)
|
||||||
|
return ()
|
||||||
|
|
||||||
|
def _explain_file(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
*,
|
||||||
|
resource_id: str,
|
||||||
|
action: str,
|
||||||
|
) -> tuple[AccessDecisionProvenance, ...]:
|
||||||
|
asset = session.get(FileAsset, resource_id) # type: ignore[attr-defined]
|
||||||
|
if asset is None or (principal.tenant_id and asset.tenant_id != principal.tenant_id):
|
||||||
|
return (_missing_resource("file", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||||
|
items = [_file_resource(asset)]
|
||||||
|
items.extend(_owner_provenance(asset.owner_type, _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id), principal, source="files.owner"))
|
||||||
|
items.extend(_admin_provenance(principal, "files:file:admin", source="files.admin_scope"))
|
||||||
|
permission_values = {"write", "manage"} if _requires_write_share(action) else {"read", "write", "manage"}
|
||||||
|
shares = (
|
||||||
|
session.query(FileShare) # type: ignore[attr-defined]
|
||||||
|
.filter(
|
||||||
|
FileShare.tenant_id == asset.tenant_id,
|
||||||
|
FileShare.file_asset_id == asset.id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
FileShare.permission.in_(sorted(permission_values)),
|
||||||
|
or_(
|
||||||
|
(FileShare.target_type == "user") & (FileShare.target_id == principal.membership_id),
|
||||||
|
(FileShare.target_type == "group") & (FileShare.target_id.in_(sorted(principal.group_ids))),
|
||||||
|
(FileShare.target_type == "tenant") & (FileShare.target_id == asset.tenant_id),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.order_by(FileShare.target_type.asc(), FileShare.target_id.asc())
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
for share in shares:
|
||||||
|
items.append(_share_provenance(share, source="files.share"))
|
||||||
|
return tuple(items)
|
||||||
|
|
||||||
|
def _explain_folder(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
*,
|
||||||
|
resource_id: str,
|
||||||
|
action: str,
|
||||||
|
) -> tuple[AccessDecisionProvenance, ...]:
|
||||||
|
del action
|
||||||
|
folder = session.get(FileFolder, resource_id) # type: ignore[attr-defined]
|
||||||
|
if folder is None:
|
||||||
|
return self._explain_virtual_folder(session, principal, resource_id=resource_id)
|
||||||
|
if principal.tenant_id and folder.tenant_id != principal.tenant_id:
|
||||||
|
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||||
|
return tuple([
|
||||||
|
AccessDecisionProvenance(
|
||||||
|
kind="resource",
|
||||||
|
id=folder.id,
|
||||||
|
label=folder.path,
|
||||||
|
tenant_id=folder.tenant_id,
|
||||||
|
source="files.folder",
|
||||||
|
details={
|
||||||
|
"resource_type": "folder",
|
||||||
|
"path": folder.path,
|
||||||
|
"owner_type": folder.owner_type,
|
||||||
|
"deleted": folder.deleted_at is not None,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
*_owner_provenance(folder.owner_type, _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id), principal, source="files.owner"),
|
||||||
|
*_admin_provenance(principal, "files:file:admin", source="files.admin_scope"),
|
||||||
|
])
|
||||||
|
|
||||||
|
def _explain_virtual_folder(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
*,
|
||||||
|
resource_id: str,
|
||||||
|
) -> tuple[AccessDecisionProvenance, ...]:
|
||||||
|
folder_ref = parse_virtual_folder_resource_id(resource_id)
|
||||||
|
if folder_ref is None:
|
||||||
|
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||||
|
tenant_id, owner_type, owner_id, path = folder_ref
|
||||||
|
if principal.tenant_id and tenant_id != principal.tenant_id:
|
||||||
|
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||||
|
child_prefix = f"{path}/"
|
||||||
|
owner_filter = FileAsset.owner_user_id == owner_id if owner_type == "user" else FileAsset.owner_group_id == owner_id
|
||||||
|
child = (
|
||||||
|
session.query(FileAsset.id) # type: ignore[attr-defined]
|
||||||
|
.filter(
|
||||||
|
FileAsset.tenant_id == tenant_id,
|
||||||
|
FileAsset.owner_type == owner_type,
|
||||||
|
owner_filter,
|
||||||
|
FileAsset.deleted_at.is_(None),
|
||||||
|
FileAsset.display_path.like(f"{child_prefix}%"),
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
if child is None:
|
||||||
|
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||||
|
return tuple([
|
||||||
|
AccessDecisionProvenance(
|
||||||
|
kind="resource",
|
||||||
|
id=resource_id,
|
||||||
|
label=path,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
source="files.virtual_folder",
|
||||||
|
details={
|
||||||
|
"resource_type": "folder",
|
||||||
|
"path": path,
|
||||||
|
"owner_type": owner_type,
|
||||||
|
"owner_id": owner_id,
|
||||||
|
"virtual": True,
|
||||||
|
"deleted": False,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
*_owner_provenance(owner_type, owner_id, principal, source="files.owner"),
|
||||||
|
*_admin_provenance(principal, "files:file:admin", source="files.admin_scope"),
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def access_capability(context: ModuleContext) -> FilesAccessService:
|
||||||
|
configure_runtime(registry=context.registry, settings=context.settings)
|
||||||
|
return FilesAccessService()
|
||||||
|
|
||||||
|
|
||||||
|
class FilesManagedTabularFileService(ManagedTabularFileProvider):
|
||||||
|
"""Expose authorized immutable CSV/XLSX versions to optional consumers."""
|
||||||
|
|
||||||
|
def list_tabular_files(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
query: str = "",
|
||||||
|
limit: int = 100,
|
||||||
|
) -> tuple[ManagedTabularFile, ...]:
|
||||||
|
db, api_principal, user_id = _tabular_context(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
required_scope="files:file:read",
|
||||||
|
)
|
||||||
|
normalized_query = str(query or "").strip().casefold()
|
||||||
|
requested_limit = max(1, min(int(limit), 100))
|
||||||
|
assets = list_recent_assets_for_user(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
limit=min(500, requested_limit * 5),
|
||||||
|
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||||
|
)
|
||||||
|
results: list[ManagedTabularFile] = []
|
||||||
|
for asset in assets:
|
||||||
|
if normalized_query and normalized_query not in (
|
||||||
|
f"{asset.filename} {asset.display_path} {asset.description or ''}"
|
||||||
|
).casefold():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
version, blob = current_version_and_blob(db, asset)
|
||||||
|
except FileStorageError:
|
||||||
|
continue
|
||||||
|
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||||
|
continue
|
||||||
|
results.append(_managed_tabular_file(asset, version, blob))
|
||||||
|
if len(results) >= requested_limit:
|
||||||
|
break
|
||||||
|
return tuple(results)
|
||||||
|
|
||||||
|
def get_tabular_file(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
file_asset_id: str,
|
||||||
|
file_version_id: str | None = None,
|
||||||
|
) -> ManagedTabularFile | None:
|
||||||
|
db, api_principal, user_id = _tabular_context(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
required_scope="files:file:read",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
asset_id=file_asset_id,
|
||||||
|
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||||
|
)
|
||||||
|
except FileStorageError:
|
||||||
|
return None
|
||||||
|
if file_version_id:
|
||||||
|
version = db.get(FileVersion, file_version_id)
|
||||||
|
if (
|
||||||
|
version is None
|
||||||
|
or version.file_asset_id != asset.id
|
||||||
|
or version.tenant_id != api_principal.tenant_id
|
||||||
|
):
|
||||||
|
return None
|
||||||
|
blob = db.get(FileBlob, version.blob_id)
|
||||||
|
if blob is None or blob.tenant_id != api_principal.tenant_id:
|
||||||
|
return None
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
version, blob = current_version_and_blob(db, asset)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise ManagedTabularFileUnavailableError(
|
||||||
|
"Managed file version metadata is unavailable."
|
||||||
|
) from exc
|
||||||
|
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||||
|
return None
|
||||||
|
return _managed_tabular_file(asset, version, blob)
|
||||||
|
|
||||||
|
def read_tabular_file(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
file_asset_id: str,
|
||||||
|
file_version_id: str,
|
||||||
|
max_bytes: int,
|
||||||
|
) -> ManagedTabularFileContent:
|
||||||
|
db, api_principal, user_id = _tabular_context(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
required_scope="files:file:download",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
asset_id=file_asset_id,
|
||||||
|
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise ManagedTabularFileNotFoundError(
|
||||||
|
"Managed tabular file not found."
|
||||||
|
) from exc
|
||||||
|
metadata = self.get_tabular_file(
|
||||||
|
db,
|
||||||
|
api_principal,
|
||||||
|
file_asset_id=asset.id,
|
||||||
|
file_version_id=file_version_id,
|
||||||
|
)
|
||||||
|
if metadata is None:
|
||||||
|
raise ManagedTabularFileNotFoundError(
|
||||||
|
"Managed tabular file version not found."
|
||||||
|
)
|
||||||
|
effective_max = max(1, int(max_bytes))
|
||||||
|
if metadata.size_bytes > effective_max:
|
||||||
|
raise ManagedTabularFileValidationError(
|
||||||
|
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
payload, version, blob = read_asset_version_bytes(
|
||||||
|
db,
|
||||||
|
asset,
|
||||||
|
file_version_id,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise ManagedTabularFileUnavailableError(
|
||||||
|
"Managed tabular file content is unavailable or failed integrity verification."
|
||||||
|
) from exc
|
||||||
|
if len(payload) > effective_max:
|
||||||
|
raise ManagedTabularFileValidationError(
|
||||||
|
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||||
|
)
|
||||||
|
result = _managed_tabular_file(asset, version, blob)
|
||||||
|
audit_from_principal(
|
||||||
|
db,
|
||||||
|
api_principal,
|
||||||
|
action="files.tabular_content.read",
|
||||||
|
object_type="file_version",
|
||||||
|
object_id=version.id,
|
||||||
|
details={
|
||||||
|
"file_asset_id": asset.id,
|
||||||
|
"size_bytes": version.size_bytes,
|
||||||
|
"checksum_sha256": version.checksum_sha256,
|
||||||
|
"consumer": "tabular_content",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return ManagedTabularFileContent(file=result, payload=payload)
|
||||||
|
|
||||||
|
|
||||||
|
def managed_tabular_file_capability(
|
||||||
|
context: ModuleContext,
|
||||||
|
) -> FilesManagedTabularFileService:
|
||||||
|
configure_runtime(registry=context.registry, settings=context.settings)
|
||||||
|
return FilesManagedTabularFileService()
|
||||||
|
|
||||||
|
|
||||||
|
def _tabular_context(
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
required_scope: str,
|
||||||
|
) -> tuple[Session, ApiPrincipal, str]:
|
||||||
|
if not isinstance(session, Session):
|
||||||
|
raise TypeError("Managed tabular file access requires a SQLAlchemy session.")
|
||||||
|
if not isinstance(principal, ApiPrincipal):
|
||||||
|
raise ManagedTabularFileAccessError(
|
||||||
|
"Managed tabular file access requires a tenant API principal."
|
||||||
|
)
|
||||||
|
if not (
|
||||||
|
has_scope(principal, required_scope)
|
||||||
|
or has_scope(principal, "files:file:admin")
|
||||||
|
):
|
||||||
|
raise ManagedTabularFileAccessError(
|
||||||
|
f"Managed tabular file access requires {required_scope}."
|
||||||
|
)
|
||||||
|
user_id = str(getattr(principal.user, "id", "") or "").strip()
|
||||||
|
if not principal.tenant_id or not user_id:
|
||||||
|
raise ManagedTabularFileAccessError(
|
||||||
|
"Managed tabular file access requires a tenant user principal."
|
||||||
|
)
|
||||||
|
return session, principal, user_id
|
||||||
|
|
||||||
|
|
||||||
|
def _managed_tabular_file(
|
||||||
|
asset: FileAsset,
|
||||||
|
version: FileVersion,
|
||||||
|
blob: FileBlob,
|
||||||
|
) -> ManagedTabularFile:
|
||||||
|
return ManagedTabularFile(
|
||||||
|
file_asset_id=asset.id,
|
||||||
|
file_version_id=version.id,
|
||||||
|
filename=version.filename_at_upload or asset.filename,
|
||||||
|
display_path=version.display_path_at_upload or asset.display_path,
|
||||||
|
content_type=version.content_type or blob.content_type,
|
||||||
|
size_bytes=version.size_bytes,
|
||||||
|
sha256=version.checksum_sha256,
|
||||||
|
updated_at=version.created_at,
|
||||||
|
current_version=asset.current_version_id == version.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_tabular_file(filename: str, content_type: str | None) -> bool:
|
||||||
|
normalized_name = str(filename or "").strip().casefold()
|
||||||
|
normalized_type = str(content_type or "").split(";", 1)[0].strip().casefold()
|
||||||
|
return normalized_name.endswith((".csv", ".xlsx")) or normalized_type in {
|
||||||
|
"text/csv",
|
||||||
|
"application/csv",
|
||||||
|
"application/vnd.ms-excel",
|
||||||
|
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def virtual_folder_resource_id(*, tenant_id: str, owner_type: str, owner_id: str, path: str) -> str:
|
||||||
|
normalized_path = normalize_folder(path)
|
||||||
|
encoded_path = base64.urlsafe_b64encode(normalized_path.encode("utf-8")).decode("ascii").rstrip("=")
|
||||||
|
return f"{VIRTUAL_FOLDER_RESOURCE_PREFIX}:{tenant_id}:{owner_type}:{owner_id}:{encoded_path}"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_virtual_folder_resource_id(resource_id: str) -> tuple[str, str, str, str] | None:
|
||||||
|
parts = resource_id.split(":", 5)
|
||||||
|
if len(parts) != 6 or f"{parts[0]}:{parts[1]}" != VIRTUAL_FOLDER_RESOURCE_PREFIX:
|
||||||
|
return None
|
||||||
|
_, _, tenant_id, owner_type, owner_id, encoded_path = parts
|
||||||
|
if owner_type not in {"user", "group"} or not tenant_id or not owner_id or not encoded_path:
|
||||||
|
return None
|
||||||
|
padding = "=" * (-len(encoded_path) % 4)
|
||||||
|
try:
|
||||||
|
decoded_path = base64.urlsafe_b64decode(f"{encoded_path}{padding}").decode("utf-8")
|
||||||
|
path = normalize_folder(decoded_path)
|
||||||
|
except (binascii.Error, UnicodeDecodeError, ValueError):
|
||||||
|
return None
|
||||||
|
if not path:
|
||||||
|
return None
|
||||||
|
return tenant_id, owner_type, owner_id, path
|
||||||
|
|
||||||
|
|
||||||
|
def _file_resource(asset: FileAsset) -> AccessDecisionProvenance:
|
||||||
|
return AccessDecisionProvenance(
|
||||||
|
kind="resource",
|
||||||
|
id=asset.id,
|
||||||
|
label=asset.filename,
|
||||||
|
tenant_id=asset.tenant_id,
|
||||||
|
source="files.file",
|
||||||
|
details={
|
||||||
|
"resource_type": "file",
|
||||||
|
"display_path": asset.display_path,
|
||||||
|
"owner_type": asset.owner_type,
|
||||||
|
"deleted": asset.deleted_at is not None,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _missing_resource(resource_type: str, resource_id: str, tenant_id: str | None, *, source: str) -> AccessDecisionProvenance:
|
||||||
|
return AccessDecisionProvenance(
|
||||||
|
kind="resource",
|
||||||
|
id=resource_id,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
source=source,
|
||||||
|
details={"resource_type": resource_type, "found": False},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||||
|
return owner_user_id if owner_type == "user" else owner_group_id
|
||||||
|
|
||||||
|
|
||||||
|
def _owner_provenance(owner_type: str, owner_id: str | None, principal: PrincipalRef, *, source: str) -> tuple[AccessDecisionProvenance, ...]:
|
||||||
|
if owner_id is None:
|
||||||
|
return ()
|
||||||
|
matches_user = owner_type == "user" and owner_id == principal.membership_id
|
||||||
|
matches_group = owner_type == "group" and owner_id in principal.group_ids
|
||||||
|
if not (matches_user or matches_group):
|
||||||
|
return ()
|
||||||
|
return (
|
||||||
|
AccessDecisionProvenance(
|
||||||
|
kind="owner",
|
||||||
|
id=owner_id,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
source=source,
|
||||||
|
details={"owner_type": owner_type},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _admin_provenance(principal: PrincipalRef, required_scope: str, *, source: str) -> tuple[AccessDecisionProvenance, ...]:
|
||||||
|
if not scopes_grant_compatible(principal.scopes, required_scope):
|
||||||
|
return ()
|
||||||
|
return (
|
||||||
|
AccessDecisionProvenance(
|
||||||
|
kind="policy",
|
||||||
|
id=required_scope,
|
||||||
|
label=required_scope,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
source=source,
|
||||||
|
details={"grant": "tenant_admin"},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _share_provenance(share: FileShare, *, source: str) -> AccessDecisionProvenance:
|
||||||
|
return AccessDecisionProvenance(
|
||||||
|
kind="share",
|
||||||
|
id=share.id,
|
||||||
|
label=share.permission,
|
||||||
|
tenant_id=share.tenant_id,
|
||||||
|
source=source,
|
||||||
|
details={
|
||||||
|
"target_type": share.target_type,
|
||||||
|
"target_id": share.target_id,
|
||||||
|
"permission": share.permission,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _requires_write_share(action: str) -> bool:
|
||||||
|
return action in WRITE_ACTIONS
|
||||||
|
|||||||
@@ -0,0 +1,186 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from sqlalchemy import event
|
||||||
|
from sqlalchemy.orm import Session as OrmSession
|
||||||
|
|
||||||
|
from govoplan_core.core.change_sequence import record_change
|
||||||
|
from govoplan_core.core.sqlalchemy_change_tracking import (
|
||||||
|
ensure_object_id,
|
||||||
|
has_attr_changes,
|
||||||
|
object_state,
|
||||||
|
operation_for_soft_deletable,
|
||||||
|
previous_value,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid
|
||||||
|
|
||||||
|
FILES_MODULE_ID = "files"
|
||||||
|
FILES_ASSETS_COLLECTION = "files.assets"
|
||||||
|
FILES_FOLDERS_COLLECTION = "files.folders"
|
||||||
|
FILES_CONNECTOR_PROFILES_COLLECTION = "files.connector_profiles"
|
||||||
|
FILES_CONNECTOR_CREDENTIALS_COLLECTION = "files.connector_credentials"
|
||||||
|
FILES_CONNECTOR_POLICIES_COLLECTION = "files.connector_policies"
|
||||||
|
FILES_CONNECTOR_SPACES_COLLECTION = "files.connector_spaces"
|
||||||
|
|
||||||
|
_REGISTERED = False
|
||||||
|
|
||||||
|
|
||||||
|
def register_files_change_tracking() -> None:
|
||||||
|
global _REGISTERED
|
||||||
|
if _REGISTERED:
|
||||||
|
return
|
||||||
|
event.listen(OrmSession, "before_flush", _record_files_changes)
|
||||||
|
_REGISTERED = True
|
||||||
|
|
||||||
|
|
||||||
|
def _record_files_changes(session: OrmSession, _flush_context: object, _instances: object) -> None:
|
||||||
|
for obj in tuple(session.new) + tuple(session.dirty):
|
||||||
|
if isinstance(obj, FileAsset):
|
||||||
|
_record_asset_change(session, obj)
|
||||||
|
elif isinstance(obj, FileFolder):
|
||||||
|
_record_folder_change(session, obj)
|
||||||
|
elif isinstance(obj, FileShare):
|
||||||
|
_record_share_visibility_change(session, obj)
|
||||||
|
|
||||||
|
|
||||||
|
def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||||
|
operation = operation_for_soft_deletable(
|
||||||
|
asset,
|
||||||
|
changed_attrs=(
|
||||||
|
"owner_type",
|
||||||
|
"owner_user_id",
|
||||||
|
"owner_group_id",
|
||||||
|
"current_version_id",
|
||||||
|
"display_path",
|
||||||
|
"filename",
|
||||||
|
"description",
|
||||||
|
"deleted_at",
|
||||||
|
"retained_until",
|
||||||
|
"legal_hold",
|
||||||
|
"lifecycle_revision",
|
||||||
|
"lifecycle_reason",
|
||||||
|
"metadata_",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if operation is None:
|
||||||
|
return
|
||||||
|
resource_id = _ensure_id(asset)
|
||||||
|
record_change(
|
||||||
|
session,
|
||||||
|
module_id=FILES_MODULE_ID,
|
||||||
|
collection=FILES_ASSETS_COLLECTION,
|
||||||
|
resource_type="file",
|
||||||
|
resource_id=resource_id,
|
||||||
|
operation=operation,
|
||||||
|
tenant_id=asset.tenant_id,
|
||||||
|
actor_type="user" if asset.created_by_user_id else None,
|
||||||
|
actor_id=asset.created_by_user_id,
|
||||||
|
payload={
|
||||||
|
"owner_type": asset.owner_type,
|
||||||
|
"owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id),
|
||||||
|
"path": asset.display_path,
|
||||||
|
"previous_path": previous_value(asset, "display_path"),
|
||||||
|
"filename": asset.filename,
|
||||||
|
"deleted_at": _isoformat(asset.deleted_at),
|
||||||
|
"retained_until": _isoformat(asset.retained_until),
|
||||||
|
"legal_hold": asset.legal_hold,
|
||||||
|
"lifecycle_revision": asset.lifecycle_revision,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
|
||||||
|
operation = operation_for_soft_deletable(
|
||||||
|
folder,
|
||||||
|
changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"),
|
||||||
|
)
|
||||||
|
if operation is None:
|
||||||
|
return
|
||||||
|
resource_id = _ensure_id(folder)
|
||||||
|
record_change(
|
||||||
|
session,
|
||||||
|
module_id=FILES_MODULE_ID,
|
||||||
|
collection=FILES_FOLDERS_COLLECTION,
|
||||||
|
resource_type="folder",
|
||||||
|
resource_id=resource_id,
|
||||||
|
operation=operation,
|
||||||
|
tenant_id=folder.tenant_id,
|
||||||
|
actor_type="user" if folder.created_by_user_id else None,
|
||||||
|
actor_id=folder.created_by_user_id,
|
||||||
|
payload={
|
||||||
|
"owner_type": folder.owner_type,
|
||||||
|
"owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id),
|
||||||
|
"path": folder.path,
|
||||||
|
"previous_path": previous_value(folder, "path"),
|
||||||
|
"deleted_at": _isoformat(folder.deleted_at),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None:
|
||||||
|
state = object_state(share)
|
||||||
|
if not share.file_asset_id:
|
||||||
|
return
|
||||||
|
if not state.pending and not has_attr_changes(
|
||||||
|
state,
|
||||||
|
(
|
||||||
|
"file_asset_id",
|
||||||
|
"target_type",
|
||||||
|
"target_id",
|
||||||
|
"permission",
|
||||||
|
"expires_at",
|
||||||
|
"revoked_at",
|
||||||
|
"revoked_by_user_id",
|
||||||
|
),
|
||||||
|
):
|
||||||
|
return
|
||||||
|
_ensure_id(share)
|
||||||
|
record_change(
|
||||||
|
session,
|
||||||
|
module_id=FILES_MODULE_ID,
|
||||||
|
collection=FILES_ASSETS_COLLECTION,
|
||||||
|
resource_type="file",
|
||||||
|
resource_id=share.file_asset_id,
|
||||||
|
operation="updated",
|
||||||
|
tenant_id=share.tenant_id,
|
||||||
|
actor_type="user" if share.created_by_user_id else None,
|
||||||
|
actor_id=share.created_by_user_id,
|
||||||
|
payload={
|
||||||
|
"share_id": share.id,
|
||||||
|
"share_target_type": share.target_type,
|
||||||
|
"share_target_id": share.target_id,
|
||||||
|
"share_permission": share.permission,
|
||||||
|
"share_expires_at": _isoformat(share.expires_at),
|
||||||
|
"share_revoked_at": _isoformat(share.revoked_at),
|
||||||
|
"share_revoked_by_user_id": share.revoked_by_user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_id(obj: object) -> str:
|
||||||
|
return ensure_object_id(obj, new_uuid)
|
||||||
|
|
||||||
|
|
||||||
|
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||||
|
if owner_type == "user":
|
||||||
|
return owner_user_id
|
||||||
|
if owner_type == "group":
|
||||||
|
return owner_group_id
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _isoformat(value: datetime | None) -> str | None:
|
||||||
|
return value.isoformat() if value else None
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"FILES_ASSETS_COLLECTION",
|
||||||
|
"FILES_CONNECTOR_CREDENTIALS_COLLECTION",
|
||||||
|
"FILES_CONNECTOR_POLICIES_COLLECTION",
|
||||||
|
"FILES_CONNECTOR_PROFILES_COLLECTION",
|
||||||
|
"FILES_CONNECTOR_SPACES_COLLECTION",
|
||||||
|
"FILES_FOLDERS_COLLECTION",
|
||||||
|
"FILES_MODULE_ID",
|
||||||
|
"register_files_change_tracking",
|
||||||
|
]
|
||||||
@@ -0,0 +1,600 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from govoplan_core.core.configuration_packages import (
|
||||||
|
ConfigurationApplyResult,
|
||||||
|
ConfigurationDiagnostic,
|
||||||
|
ConfigurationExportResult,
|
||||||
|
ConfigurationExportSelection,
|
||||||
|
ConfigurationPackageFragment,
|
||||||
|
ConfigurationPlanItem,
|
||||||
|
ConfigurationPreflightContext,
|
||||||
|
ConfigurationPreflightResult,
|
||||||
|
ConfigurationProvider,
|
||||||
|
ConfigurationProviderDescription,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.infrastructure_capabilities import (
|
||||||
|
InfrastructureCapability,
|
||||||
|
InfrastructureCapabilityReceipt,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.runtime import settings as runtime_settings
|
||||||
|
|
||||||
|
|
||||||
|
FILES_CONFIGURATION_CAPABILITY = "files.configuration"
|
||||||
|
MANAGED_STORAGE_FRAGMENT = "managed_storage"
|
||||||
|
_PAYLOAD_KEYS = frozenset(
|
||||||
|
{"capability_id", "expected_backend", "expected_source"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FilesConfigurationProvider(ConfigurationProvider):
|
||||||
|
module_id = "files"
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
settings: object | None = None,
|
||||||
|
environment: Mapping[str, str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
self._settings = runtime_settings if settings is None else settings
|
||||||
|
self._environment = os.environ if environment is None else environment
|
||||||
|
|
||||||
|
def describe(self) -> ConfigurationProviderDescription:
|
||||||
|
return ConfigurationProviderDescription(
|
||||||
|
module_id=self.module_id,
|
||||||
|
fragment_types=(MANAGED_STORAGE_FRAGMENT,),
|
||||||
|
schema_refs={
|
||||||
|
MANAGED_STORAGE_FRAGMENT: "govoplan/files/configuration/managed-storage.v1"
|
||||||
|
},
|
||||||
|
exported_scopes=("system",),
|
||||||
|
)
|
||||||
|
|
||||||
|
def preflight(
|
||||||
|
self,
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
context: ConfigurationPreflightContext,
|
||||||
|
) -> ConfigurationPreflightResult:
|
||||||
|
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||||
|
return ConfigurationPreflightResult(
|
||||||
|
diagnostics=(_unsupported(fragment),),
|
||||||
|
plan=(_blocked_plan(fragment, "Fragment type is unsupported."),),
|
||||||
|
)
|
||||||
|
diagnostics, binding_ref = self._binding_diagnostics(fragment, context)
|
||||||
|
blocked = any(item.severity == "blocker" for item in diagnostics)
|
||||||
|
return ConfigurationPreflightResult(
|
||||||
|
diagnostics=tuple(diagnostics),
|
||||||
|
plan=(
|
||||||
|
ConfigurationPlanItem(
|
||||||
|
action="blocked" if blocked else "skip",
|
||||||
|
module_id="files",
|
||||||
|
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||||
|
fragment_id=fragment.fragment_id or binding_ref,
|
||||||
|
summary=(
|
||||||
|
"Managed storage does not match the deployment receipt."
|
||||||
|
if blocked
|
||||||
|
else "Deployment-owned managed storage already matches the receipt; no module state is rewritten."
|
||||||
|
),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def apply(
|
||||||
|
self,
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
supplied_data: Mapping[str, Any],
|
||||||
|
context: ConfigurationPreflightContext,
|
||||||
|
) -> ConfigurationApplyResult:
|
||||||
|
del supplied_data
|
||||||
|
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||||
|
return ConfigurationApplyResult(diagnostics=(_unsupported(fragment),))
|
||||||
|
diagnostics, _binding_ref = self._binding_diagnostics(fragment, context)
|
||||||
|
return ConfigurationApplyResult(
|
||||||
|
diagnostics=tuple(
|
||||||
|
item for item in diagnostics if item.severity == "blocker"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def export(
|
||||||
|
self,
|
||||||
|
selection: ConfigurationExportSelection,
|
||||||
|
context: ConfigurationPreflightContext,
|
||||||
|
) -> ConfigurationExportResult:
|
||||||
|
del selection
|
||||||
|
receipt = context.infrastructure_receipt
|
||||||
|
if context.infrastructure_receipt_error:
|
||||||
|
return ConfigurationExportResult(
|
||||||
|
diagnostics=(
|
||||||
|
_receipt_error(
|
||||||
|
context.infrastructure_receipt_error,
|
||||||
|
object_ref="files.storage",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if receipt is None:
|
||||||
|
return ConfigurationExportResult(
|
||||||
|
diagnostics=(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_receipt_missing",
|
||||||
|
message="Files managed-storage export requires the deployment capability receipt.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref="files.storage",
|
||||||
|
resolution="Mount the installer-generated receipt before exporting deployment configuration.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
capability = receipt.capability("files.storage")
|
||||||
|
if capability is None:
|
||||||
|
return ConfigurationExportResult(
|
||||||
|
diagnostics=(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_capability_missing",
|
||||||
|
message="The deployment receipt does not declare managed file storage.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref="files.storage",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return ConfigurationExportResult(
|
||||||
|
fragments=(
|
||||||
|
ConfigurationPackageFragment(
|
||||||
|
module_id="files",
|
||||||
|
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||||
|
fragment_id=f"{receipt.installation_id}:files.storage",
|
||||||
|
payload={
|
||||||
|
"capability_id": "files.storage",
|
||||||
|
"expected_backend": _expected_backend(capability.source),
|
||||||
|
"expected_source": capability.source,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def health(
|
||||||
|
self,
|
||||||
|
import_result: ConfigurationApplyResult,
|
||||||
|
context: ConfigurationPreflightContext,
|
||||||
|
) -> tuple[ConfigurationDiagnostic, ...]:
|
||||||
|
del context
|
||||||
|
return tuple(
|
||||||
|
item for item in import_result.diagnostics if item.severity == "blocker"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _binding_diagnostics(
|
||||||
|
self,
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
context: ConfigurationPreflightContext,
|
||||||
|
) -> tuple[list[ConfigurationDiagnostic], str]:
|
||||||
|
diagnostics: list[ConfigurationDiagnostic] = []
|
||||||
|
payload = _files_fragment_payload(fragment, diagnostics)
|
||||||
|
if payload is None:
|
||||||
|
return diagnostics, fragment.fragment_id or "files.storage"
|
||||||
|
receipt_state = _files_receipt_capability(
|
||||||
|
fragment,
|
||||||
|
context,
|
||||||
|
payload,
|
||||||
|
diagnostics,
|
||||||
|
)
|
||||||
|
if receipt_state is None:
|
||||||
|
return diagnostics, fragment.fragment_id or "files.storage"
|
||||||
|
_receipt, capability, binding_ref = receipt_state
|
||||||
|
expected_backend = _expected_backend(capability.source)
|
||||||
|
diagnostics.extend(
|
||||||
|
_package_expectation_diagnostics(
|
||||||
|
payload,
|
||||||
|
capability,
|
||||||
|
expected_backend,
|
||||||
|
binding_ref,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
active_backend = _normalized_backend(
|
||||||
|
getattr(self._settings, "file_storage_backend", "local")
|
||||||
|
)
|
||||||
|
if active_backend != expected_backend:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_runtime_mismatch",
|
||||||
|
message=(
|
||||||
|
f"Files runtime uses {active_backend!r}, while the deployment receipt declares {expected_backend!r}."
|
||||||
|
),
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Reconcile deployment environment and receipt before starting or importing Files configuration.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return diagnostics, binding_ref
|
||||||
|
diagnostics.extend(
|
||||||
|
self._runtime_storage_diagnostics(
|
||||||
|
expected_backend,
|
||||||
|
capability,
|
||||||
|
binding_ref,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if "files" not in capability.dependent_modules:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="info",
|
||||||
|
code="infrastructure_consumer_not_declared",
|
||||||
|
message="Files is active but was not selected as a receipt consumer when the deployment plan was generated.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Regenerate the deployment plan so removal-impact inventory includes Files.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return diagnostics, binding_ref
|
||||||
|
|
||||||
|
def _runtime_storage_diagnostics(
|
||||||
|
self,
|
||||||
|
expected_backend: str,
|
||||||
|
capability: InfrastructureCapability,
|
||||||
|
binding_ref: str,
|
||||||
|
) -> list[ConfigurationDiagnostic]:
|
||||||
|
if expected_backend == "s3":
|
||||||
|
return self._s3_diagnostics(
|
||||||
|
capability.endpoint,
|
||||||
|
capability.secret_refs,
|
||||||
|
capability.source,
|
||||||
|
binding_ref,
|
||||||
|
)
|
||||||
|
root = Path(
|
||||||
|
str(getattr(self._settings, "file_storage_local_root", "") or "")
|
||||||
|
)
|
||||||
|
if root.is_absolute():
|
||||||
|
return []
|
||||||
|
return [
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_local_root_not_durable",
|
||||||
|
message="Receipt-bound local file storage requires an absolute deployment-managed path.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Set FILE_STORAGE_LOCAL_ROOT to the mounted persistent-volume path.",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
def _s3_diagnostics(
|
||||||
|
self,
|
||||||
|
endpoint: Mapping[str, object],
|
||||||
|
secret_refs: tuple[str, ...],
|
||||||
|
source: str,
|
||||||
|
object_ref: str,
|
||||||
|
) -> list[ConfigurationDiagnostic]:
|
||||||
|
diagnostics: list[ConfigurationDiagnostic] = []
|
||||||
|
endpoint_url = str(
|
||||||
|
getattr(self._settings, "file_storage_s3_endpoint_url", "")
|
||||||
|
or getattr(self._settings, "s3_endpoint_url", "")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
runtime_endpoint = _redacted_endpoint(
|
||||||
|
endpoint_url,
|
||||||
|
default_port=int(endpoint.get("port") or 443),
|
||||||
|
)
|
||||||
|
receipt_endpoint = {
|
||||||
|
key: endpoint.get(key) for key in ("scheme", "host", "port") if endpoint.get(key) is not None
|
||||||
|
}
|
||||||
|
if receipt_endpoint and runtime_endpoint != receipt_endpoint:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_endpoint_mismatch",
|
||||||
|
message="Files S3 endpoint does not match the sanitized deployment receipt endpoint.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=object_ref,
|
||||||
|
resolution="Reconcile FILE_STORAGE_S3_ENDPOINT_URL and the deployment plan before apply.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
bucket = str(
|
||||||
|
getattr(self._settings, "file_storage_s3_bucket", "")
|
||||||
|
or getattr(self._settings, "s3_bucket", "")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
if not bucket:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_bucket_missing",
|
||||||
|
message="Files S3 storage has no configured bucket.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=object_ref,
|
||||||
|
resolution="Set FILE_STORAGE_S3_BUCKET in the deployment environment.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for reference in secret_refs:
|
||||||
|
variable = reference.removeprefix("env:")
|
||||||
|
if not variable.startswith("FILE_STORAGE_"):
|
||||||
|
continue
|
||||||
|
if not str(self._environment.get(variable, "")).strip():
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_secret_reference_unresolved",
|
||||||
|
message=f"Required storage secret reference {reference} is not available to the runtime.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=object_ref,
|
||||||
|
resolution="Provide the referenced environment secret without copying its value into the package.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
deployment_managed = bool(
|
||||||
|
getattr(
|
||||||
|
self._settings,
|
||||||
|
"file_storage_s3_deployment_managed",
|
||||||
|
False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
endpoint_trusted = bool(
|
||||||
|
getattr(self._settings, "file_storage_s3_endpoint_trusted", False)
|
||||||
|
)
|
||||||
|
if source == "installer-managed-garage" and not deployment_managed:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_management_boundary_mismatch",
|
||||||
|
message="Garage is installer-managed in the receipt but not marked deployment-managed in Files runtime.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=object_ref,
|
||||||
|
resolution="Set FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true for the installer-managed Garage endpoint.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if source == "operator-supplied-s3" and not endpoint_trusted:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_storage_trust_boundary_missing",
|
||||||
|
message="External S3 storage must be explicitly marked as a trusted deployment endpoint.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=object_ref,
|
||||||
|
resolution="Review the endpoint and set FILE_STORAGE_S3_ENDPOINT_TRUSTED=true in deployment configuration.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return diagnostics
|
||||||
|
|
||||||
|
|
||||||
|
def _files_fragment_payload(
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
diagnostics: list[ConfigurationDiagnostic],
|
||||||
|
) -> Mapping[str, Any] | None:
|
||||||
|
payload = fragment.payload
|
||||||
|
if not isinstance(payload, Mapping):
|
||||||
|
diagnostics.append(
|
||||||
|
_invalid(fragment, "Managed-storage payload must be an object.")
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
unknown = sorted(set(payload) - _PAYLOAD_KEYS)
|
||||||
|
if not unknown:
|
||||||
|
return payload
|
||||||
|
secret_like = any(
|
||||||
|
marker in key.casefold()
|
||||||
|
for key in unknown
|
||||||
|
for marker in ("password", "secret", "token", "credential", "access_key")
|
||||||
|
)
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code=(
|
||||||
|
"files_configuration_secret_forbidden"
|
||||||
|
if secret_like
|
||||||
|
else "files_configuration_payload_invalid"
|
||||||
|
),
|
||||||
|
message=(
|
||||||
|
"Managed-storage fragments accept receipt references and non-secret expectations only."
|
||||||
|
if secret_like
|
||||||
|
else f"Managed-storage payload contains unsupported fields: {', '.join(unknown)}."
|
||||||
|
),
|
||||||
|
module_id="files",
|
||||||
|
object_ref=fragment.fragment_id or MANAGED_STORAGE_FRAGMENT,
|
||||||
|
resolution="Keep storage credentials in deployment environment references, never in a configuration package.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _files_receipt_capability(
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
context: ConfigurationPreflightContext,
|
||||||
|
payload: Mapping[str, Any],
|
||||||
|
diagnostics: list[ConfigurationDiagnostic],
|
||||||
|
) -> tuple[
|
||||||
|
InfrastructureCapabilityReceipt,
|
||||||
|
InfrastructureCapability,
|
||||||
|
str,
|
||||||
|
] | None:
|
||||||
|
if context.infrastructure_receipt_error:
|
||||||
|
diagnostics.append(
|
||||||
|
_receipt_error(
|
||||||
|
context.infrastructure_receipt_error,
|
||||||
|
object_ref=fragment.fragment_id or "files.storage",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
receipt = context.infrastructure_receipt
|
||||||
|
if receipt is None:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_receipt_missing",
|
||||||
|
message="Files managed-storage configuration requires the deployment capability receipt.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=fragment.fragment_id or "files.storage",
|
||||||
|
resolution="Mount the installer-generated receipt and rerun preflight.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
binding_ref = fragment.fragment_id or f"{receipt.installation_id}:files.storage"
|
||||||
|
capability_id = _text(payload.get("capability_id")) or "files.storage"
|
||||||
|
if capability_id != "files.storage":
|
||||||
|
diagnostics.append(
|
||||||
|
_invalid(
|
||||||
|
fragment,
|
||||||
|
"Files managed-storage fragments must reference capability 'files.storage'.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
capability = receipt.capability(capability_id)
|
||||||
|
if capability is None:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_capability_missing",
|
||||||
|
message="The deployment receipt does not declare managed file storage.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Regenerate the receipt from a deployment profile that declares managed storage.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
state_diagnostic = _storage_capability_state_diagnostic(capability, binding_ref)
|
||||||
|
if state_diagnostic is not None:
|
||||||
|
diagnostics.append(state_diagnostic)
|
||||||
|
return None
|
||||||
|
return receipt, capability, binding_ref
|
||||||
|
|
||||||
|
|
||||||
|
def _storage_capability_state_diagnostic(
|
||||||
|
capability: InfrastructureCapability,
|
||||||
|
binding_ref: str,
|
||||||
|
) -> ConfigurationDiagnostic | None:
|
||||||
|
if capability.state == "unavailable":
|
||||||
|
return ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_capability_unavailable",
|
||||||
|
message="The deployment receipt states that managed file storage is unavailable.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Select local, managed Garage, or external S3 storage in the deployment profile.",
|
||||||
|
)
|
||||||
|
if capability.state == "available_unconfigured":
|
||||||
|
return ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_capability_unconfigured",
|
||||||
|
message="Managed file storage is available but has not been bound by the deployment runtime.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Complete the deployment-owned storage configuration before importing Files configuration.",
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _package_expectation_diagnostics(
|
||||||
|
payload: Mapping[str, Any],
|
||||||
|
capability: InfrastructureCapability,
|
||||||
|
expected_backend: str,
|
||||||
|
binding_ref: str,
|
||||||
|
) -> list[ConfigurationDiagnostic]:
|
||||||
|
diagnostics: list[ConfigurationDiagnostic] = []
|
||||||
|
package_backend = (
|
||||||
|
_text(payload.get("expected_backend")) or expected_backend
|
||||||
|
).casefold()
|
||||||
|
if package_backend != expected_backend:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_backend_mismatch",
|
||||||
|
message="The package storage backend expectation conflicts with the deployment receipt.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Use the receipt backend or regenerate the deployment package after review.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
package_source = _text(payload.get("expected_source"))
|
||||||
|
if package_source and package_source != capability.source:
|
||||||
|
diagnostics.append(
|
||||||
|
ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_source_mismatch",
|
||||||
|
message="The package storage source expectation conflicts with the deployment receipt.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=binding_ref,
|
||||||
|
resolution="Review the provider replacement and regenerate the package from the active receipt.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return diagnostics
|
||||||
|
|
||||||
|
|
||||||
|
def _expected_backend(source: str) -> str:
|
||||||
|
return "local" if source == "host-local" else "s3"
|
||||||
|
|
||||||
|
|
||||||
|
def _normalized_backend(value: object) -> str:
|
||||||
|
clean = str(value or "local").strip().casefold()
|
||||||
|
if clean in {"local", "filesystem", "fs"}:
|
||||||
|
return "local"
|
||||||
|
if clean in {"s3", "garage"}:
|
||||||
|
return "s3"
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _redacted_endpoint(value: str, *, default_port: int) -> dict[str, object]:
|
||||||
|
try:
|
||||||
|
parsed = urlsplit(value)
|
||||||
|
host = parsed.hostname
|
||||||
|
if not parsed.scheme or not host:
|
||||||
|
return {"reference": "unresolved"}
|
||||||
|
port = parsed.port or default_port
|
||||||
|
except ValueError:
|
||||||
|
return {"reference": "unresolved"}
|
||||||
|
return {"scheme": parsed.scheme, "host": host, "port": port}
|
||||||
|
|
||||||
|
|
||||||
|
def _text(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
|
|
||||||
|
|
||||||
|
def _receipt_error(error: str, *, object_ref: str) -> ConfigurationDiagnostic:
|
||||||
|
return ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="infrastructure_receipt_invalid",
|
||||||
|
message=f"The deployment capability receipt is invalid: {error}",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=object_ref,
|
||||||
|
resolution="Repair or regenerate the deployment receipt before importing Files configuration.",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _invalid(
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
message: str,
|
||||||
|
) -> ConfigurationDiagnostic:
|
||||||
|
return ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="files_configuration_payload_invalid",
|
||||||
|
message=message,
|
||||||
|
module_id="files",
|
||||||
|
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||||
|
resolution="Review the Files configuration-package fragment and rerun preflight.",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _unsupported(fragment: ConfigurationPackageFragment) -> ConfigurationDiagnostic:
|
||||||
|
return ConfigurationDiagnostic(
|
||||||
|
severity="blocker",
|
||||||
|
code="fragment_type_unsupported",
|
||||||
|
message=f"Files configuration does not support fragment type {fragment.fragment_type!r}.",
|
||||||
|
module_id="files",
|
||||||
|
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _blocked_plan(
|
||||||
|
fragment: ConfigurationPackageFragment,
|
||||||
|
summary: str,
|
||||||
|
) -> ConfigurationPlanItem:
|
||||||
|
return ConfigurationPlanItem(
|
||||||
|
action="blocked",
|
||||||
|
module_id="files",
|
||||||
|
fragment_type=fragment.fragment_type,
|
||||||
|
fragment_id=fragment.fragment_id,
|
||||||
|
summary=summary,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["FILES_CONFIGURATION_CAPABILITY", "FilesConfigurationProvider"]
|
||||||
@@ -1 +1,25 @@
|
|||||||
from govoplan_files.backend.db.models import *
|
from govoplan_files.backend.db.models import (
|
||||||
|
CampaignAttachmentUse,
|
||||||
|
FileAsset,
|
||||||
|
FileBlob,
|
||||||
|
FileConnectorCredential,
|
||||||
|
FileConnectorPolicy,
|
||||||
|
FileConnectorProfile,
|
||||||
|
FileConnectorSpace,
|
||||||
|
FileFolder,
|
||||||
|
FileShare,
|
||||||
|
FileVersion,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"CampaignAttachmentUse",
|
||||||
|
"FileAsset",
|
||||||
|
"FileBlob",
|
||||||
|
"FileConnectorCredential",
|
||||||
|
"FileConnectorPolicy",
|
||||||
|
"FileConnectorProfile",
|
||||||
|
"FileConnectorSpace",
|
||||||
|
"FileFolder",
|
||||||
|
"FileShare",
|
||||||
|
"FileVersion",
|
||||||
|
]
|
||||||
|
|||||||
@@ -4,7 +4,18 @@ import uuid
|
|||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from sqlalchemy import DateTime, ForeignKey, Index, Integer, JSON, String, Text, UniqueConstraint, text
|
from sqlalchemy import (
|
||||||
|
Boolean,
|
||||||
|
DateTime,
|
||||||
|
ForeignKey,
|
||||||
|
Index,
|
||||||
|
Integer,
|
||||||
|
JSON,
|
||||||
|
String,
|
||||||
|
Text,
|
||||||
|
UniqueConstraint,
|
||||||
|
text,
|
||||||
|
)
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
from govoplan_core.db.base import Base, TimestampMixin
|
from govoplan_core.db.base import Base, TimestampMixin
|
||||||
@@ -16,18 +27,123 @@ def new_uuid() -> str:
|
|||||||
|
|
||||||
class FileBlob(Base, TimestampMixin):
|
class FileBlob(Base, TimestampMixin):
|
||||||
__tablename__ = "file_blobs"
|
__tablename__ = "file_blobs"
|
||||||
__table_args__ = (UniqueConstraint("tenant_id", "checksum_sha256", "size_bytes", name="uq_file_blobs_tenant_checksum_size"),)
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"checksum_sha256",
|
||||||
|
"size_bytes",
|
||||||
|
"protection_discriminator",
|
||||||
|
name="uq_file_blobs_tenant_checksum_size_protection",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||||
storage_bucket: Mapped[str | None] = mapped_column(String(255))
|
storage_bucket: Mapped[str | None] = mapped_column(String(255))
|
||||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
|
protection_discriminator: Mapped[str] = mapped_column(
|
||||||
|
String(320), default="plaintext", nullable=False, index=True
|
||||||
|
)
|
||||||
|
encryption_envelope_id: Mapped[str | None] = mapped_column(
|
||||||
|
String(255), nullable=True, index=True
|
||||||
|
)
|
||||||
|
storage_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||||
|
String(64), nullable=True
|
||||||
|
)
|
||||||
|
storage_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||||
ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||||
|
integrity_status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="unchecked", nullable=False, index=True
|
||||||
|
)
|
||||||
|
integrity_checked_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
integrity_failure: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||||
|
quarantined_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityScan(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_integrity_scans"
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||||
|
storage_prefix: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||||
|
status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="pending", nullable=False, index=True
|
||||||
|
)
|
||||||
|
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
|
phase: Mapped[str] = mapped_column(String(30), default="blobs", nullable=False)
|
||||||
|
verify_checksums: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=True, nullable=False
|
||||||
|
)
|
||||||
|
batch_size: Mapped[int] = mapped_column(Integer, default=100, nullable=False)
|
||||||
|
blob_cursor: Mapped[str | None] = mapped_column(String(36), nullable=True)
|
||||||
|
object_cursor: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||||
|
scanned_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||||
|
verified_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||||
|
quarantined_blob_count: Mapped[int] = mapped_column(
|
||||||
|
Integer, default=0, nullable=False
|
||||||
|
)
|
||||||
|
scanned_object_count: Mapped[int] = mapped_column(
|
||||||
|
Integer, default=0, nullable=False
|
||||||
|
)
|
||||||
|
orphan_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||||
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
started_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True
|
||||||
|
)
|
||||||
|
completed_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True
|
||||||
|
)
|
||||||
|
last_error: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityFinding(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_integrity_findings"
|
||||||
|
__table_args__ = (
|
||||||
|
Index("ix_file_integrity_findings_scan_state", "scan_id", "state"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
scan_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("file_integrity_scans.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
kind: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||||
|
state: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="open", nullable=False, index=True
|
||||||
|
)
|
||||||
|
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||||
|
blob_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("file_blobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||||
|
expected_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
|
observed_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
|
expected_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||||
|
String(64), nullable=True
|
||||||
|
)
|
||||||
|
observed_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||||
|
String(64), nullable=True
|
||||||
|
)
|
||||||
|
resolved_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True
|
||||||
|
)
|
||||||
|
resolved_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class FileFolder(Base, TimestampMixin):
|
class FileFolder(Base, TimestampMixin):
|
||||||
@@ -35,14 +151,18 @@ class FileFolder(Base, TimestampMixin):
|
|||||||
__table_args__ = (
|
__table_args__ = (
|
||||||
Index(
|
Index(
|
||||||
"uq_file_folders_active_user_path",
|
"uq_file_folders_active_user_path",
|
||||||
"tenant_id", "owner_user_id", "path",
|
"tenant_id",
|
||||||
|
"owner_user_id",
|
||||||
|
"path",
|
||||||
unique=True,
|
unique=True,
|
||||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||||
),
|
),
|
||||||
Index(
|
Index(
|
||||||
"uq_file_folders_active_group_path",
|
"uq_file_folders_active_group_path",
|
||||||
"tenant_id", "owner_group_id", "path",
|
"tenant_id",
|
||||||
|
"owner_group_id",
|
||||||
|
"path",
|
||||||
unique=True,
|
unique=True,
|
||||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||||
@@ -50,91 +170,414 @@ class FileFolder(Base, TimestampMixin):
|
|||||||
)
|
)
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
owner_user_id: Mapped[str | None] = mapped_column(
|
||||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
owner_group_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
)
|
||||||
|
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||||
|
"metadata", JSON, nullable=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class FileAsset(Base, TimestampMixin):
|
class FileAsset(Base, TimestampMixin):
|
||||||
__tablename__ = "file_assets"
|
__tablename__ = "file_assets"
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
owner_user_id: Mapped[str | None] = mapped_column(
|
||||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
current_version_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
)
|
||||||
|
owner_group_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
current_version_id: Mapped[str | None] = mapped_column(
|
||||||
|
String(36), nullable=True, index=True
|
||||||
|
)
|
||||||
display_path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
display_path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||||
filename: Mapped[str] = mapped_column(String(500), nullable=False, index=True)
|
filename: Mapped[str] = mapped_column(String(500), nullable=False, index=True)
|
||||||
description: Mapped[str | None] = mapped_column(Text)
|
description: Mapped[str | None] = mapped_column(Text)
|
||||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
)
|
||||||
|
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
retained_until: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
legal_hold: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=False, nullable=False, index=True
|
||||||
|
)
|
||||||
|
lifecycle_revision: Mapped[int] = mapped_column(
|
||||||
|
Integer, default=1, nullable=False
|
||||||
|
)
|
||||||
|
lifecycle_reason: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||||
|
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||||
|
"metadata", JSON, nullable=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class FileVersion(Base, TimestampMixin):
|
class FileVersion(Base, TimestampMixin):
|
||||||
__tablename__ = "file_versions"
|
__tablename__ = "file_versions"
|
||||||
__table_args__ = (UniqueConstraint("file_asset_id", "version_number", name="uq_file_versions_asset_number"),)
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"file_asset_id", "version_number", name="uq_file_versions_asset_number"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
file_asset_id: Mapped[str] = mapped_column(
|
||||||
blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||||
|
)
|
||||||
|
blob_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||||
|
)
|
||||||
version_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
version_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
filename_at_upload: Mapped[str] = mapped_column(String(500), nullable=False)
|
filename_at_upload: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||||
display_path_at_upload: Mapped[str] = mapped_column(String(1000), nullable=False)
|
display_path_at_upload: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FileFormEvidenceGrant(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_form_evidence_grants"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"idempotency_key",
|
||||||
|
name="uq_file_form_evidence_grants_idempotency",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_file_form_evidence_grants_form",
|
||||||
|
"tenant_id",
|
||||||
|
"form_instance_id",
|
||||||
|
"form_definition_id",
|
||||||
|
"form_definition_revision",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
form_instance_id: Mapped[str] = mapped_column(
|
||||||
|
String(36), nullable=False, index=True
|
||||||
|
)
|
||||||
|
form_definition_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
form_definition_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
token_sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||||
|
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
request_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
custodian_user_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="RESTRICT"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
evidence_kind: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||||
|
purpose: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||||
|
status: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="issued", nullable=False, index=True
|
||||||
|
)
|
||||||
|
expires_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=False, index=True
|
||||||
|
)
|
||||||
|
max_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
|
allowed_content_types: Mapped[list[str]] = mapped_column(
|
||||||
|
JSON, default=list, nullable=False
|
||||||
|
)
|
||||||
|
file_asset_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
file_version_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
uploaded_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True
|
||||||
|
)
|
||||||
|
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True
|
||||||
|
)
|
||||||
|
metadata_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"metadata", JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class FileShare(Base, TimestampMixin):
|
class FileShare(Base, TimestampMixin):
|
||||||
__tablename__ = "file_shares"
|
__tablename__ = "file_shares"
|
||||||
__table_args__ = (UniqueConstraint("file_asset_id", "target_type", "target_id", "revoked_at", name="uq_file_shares_active_target"),)
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"file_asset_id",
|
||||||
|
"target_type",
|
||||||
|
"target_id",
|
||||||
|
"revoked_at",
|
||||||
|
name="uq_file_shares_active_target",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
file_asset_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||||
|
)
|
||||||
target_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
target_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||||
target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False)
|
permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False)
|
||||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
expires_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
revoked_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProfile(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_connector_profiles"
|
||||||
|
__table_args__ = (
|
||||||
|
Index("ix_file_connector_profiles_scope", "scope_type", "scope_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||||
|
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||||
|
scope_type: Mapped[str] = mapped_column(
|
||||||
|
String(20), default="tenant", nullable=False, index=True
|
||||||
|
)
|
||||||
|
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||||
|
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||||
|
endpoint_url: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||||
|
base_path: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||||
|
enabled: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=True, nullable=False, index=True
|
||||||
|
)
|
||||||
|
credential_profile_id: Mapped[str | None] = mapped_column(
|
||||||
|
String(255), nullable=True, index=True
|
||||||
|
)
|
||||||
|
credential_mode: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="none", nullable=False
|
||||||
|
)
|
||||||
|
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||||
|
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
|
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
|
password_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||||
|
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||||
|
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||||
|
capabilities: Mapped[list[str] | None] = mapped_column(JSON, nullable=True)
|
||||||
|
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||||
|
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||||
|
"metadata", JSON, nullable=True
|
||||||
|
)
|
||||||
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorCredential(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_connector_credentials"
|
||||||
|
__table_args__ = (
|
||||||
|
Index("ix_file_connector_credentials_scope", "scope_type", "scope_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||||
|
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||||
|
scope_type: Mapped[str] = mapped_column(
|
||||||
|
String(20), default="tenant", nullable=False, index=True
|
||||||
|
)
|
||||||
|
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||||
|
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
provider: Mapped[str | None] = mapped_column(String(50), nullable=True, index=True)
|
||||||
|
enabled: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=True, nullable=False, index=True
|
||||||
|
)
|
||||||
|
credential_mode: Mapped[str] = mapped_column(
|
||||||
|
String(30), default="none", nullable=False
|
||||||
|
)
|
||||||
|
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||||
|
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
|
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
|
password_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||||
|
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||||
|
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||||
|
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||||
|
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||||
|
"metadata", JSON, nullable=True
|
||||||
|
)
|
||||||
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicy(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_connector_policies"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"scope_type",
|
||||||
|
"scope_id",
|
||||||
|
name="uq_file_connector_policies_scope",
|
||||||
|
),
|
||||||
|
Index("ix_file_connector_policies_scope", "scope_type", "scope_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||||
|
scope_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||||
|
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||||
|
policy: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||||
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSpace(Base, TimestampMixin):
|
||||||
|
__tablename__ = "file_connector_spaces"
|
||||||
|
__table_args__ = (
|
||||||
|
Index(
|
||||||
|
"ix_file_connector_spaces_owner",
|
||||||
|
"tenant_id",
|
||||||
|
"owner_type",
|
||||||
|
"owner_user_id",
|
||||||
|
"owner_group_id",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"uq_file_connector_spaces_active_user_label",
|
||||||
|
"tenant_id",
|
||||||
|
"owner_user_id",
|
||||||
|
"label",
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||||
|
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"uq_file_connector_spaces_active_group_label",
|
||||||
|
"tenant_id",
|
||||||
|
"owner_group_id",
|
||||||
|
"label",
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||||
|
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||||
|
owner_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
owner_group_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
connector_profile_id: Mapped[str] = mapped_column(
|
||||||
|
String(255), nullable=False, index=True
|
||||||
|
)
|
||||||
|
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||||
|
library_id: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||||
|
remote_path: Mapped[str] = mapped_column(String(1000), default="", nullable=False)
|
||||||
|
sync_mode: Mapped[str] = mapped_column(String(30), default="manual", nullable=False)
|
||||||
|
read_only: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||||
|
is_active: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=True, nullable=False, index=True
|
||||||
|
)
|
||||||
|
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||||
|
"metadata", JSON, nullable=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class CampaignAttachmentUse(Base, TimestampMixin):
|
class CampaignAttachmentUse(Base, TimestampMixin):
|
||||||
__tablename__ = "campaign_attachment_uses"
|
__tablename__ = "campaign_attachment_uses"
|
||||||
__table_args__ = (UniqueConstraint("campaign_job_id", "file_version_id", "filename_used", "use_stage", name="uq_campaign_attachment_uses_job_file_stage"),)
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"campaign_job_id",
|
||||||
|
"file_version_id",
|
||||||
|
"filename_used",
|
||||||
|
"use_stage",
|
||||||
|
name="uq_campaign_attachment_uses_job_file_stage",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
campaign_id: Mapped[str] = mapped_column(ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True)
|
campaign_id: Mapped[str] = mapped_column(
|
||||||
campaign_version_id: Mapped[str] = mapped_column(ForeignKey("campaign_versions.id", ondelete="CASCADE"), nullable=False, index=True)
|
ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True
|
||||||
campaign_job_id: Mapped[str | None] = mapped_column(ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True)
|
)
|
||||||
|
campaign_version_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("campaign_versions.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
campaign_job_id: Mapped[str | None] = mapped_column(
|
||||||
|
ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
entry_index: Mapped[int | None] = mapped_column(Integer)
|
entry_index: Mapped[int | None] = mapped_column(Integer)
|
||||||
entry_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
entry_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True)
|
file_asset_id: Mapped[str] = mapped_column(
|
||||||
file_version_id: Mapped[str] = mapped_column(ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True)
|
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||||
file_blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
)
|
||||||
|
file_version_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||||
|
)
|
||||||
|
file_blob_id: Mapped[str] = mapped_column(
|
||||||
|
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||||
|
)
|
||||||
filename_used: Mapped[str] = mapped_column(String(500), nullable=False)
|
filename_used: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||||
use_stage: Mapped[str] = mapped_column(String(20), default="built", nullable=False, index=True)
|
use_stage: Mapped[str] = mapped_column(
|
||||||
used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
String(20), default="built", nullable=False, index=True
|
||||||
|
)
|
||||||
|
used_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"CampaignAttachmentUse",
|
"CampaignAttachmentUse",
|
||||||
"FileAsset",
|
"FileAsset",
|
||||||
"FileBlob",
|
"FileBlob",
|
||||||
|
"FileConnectorCredential",
|
||||||
|
"FileConnectorPolicy",
|
||||||
|
"FileConnectorProfile",
|
||||||
|
"FileConnectorSpace",
|
||||||
"FileFolder",
|
"FileFolder",
|
||||||
|
"FileFormEvidenceGrant",
|
||||||
"FileShare",
|
"FileShare",
|
||||||
"FileVersion",
|
"FileVersion",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -0,0 +1,517 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.campaigns import (
|
||||||
|
CAPABILITY_CAMPAIGNS_ACCESS,
|
||||||
|
CampaignAccessProvider,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.modules import (
|
||||||
|
DocumentationCondition,
|
||||||
|
DocumentationContext,
|
||||||
|
DocumentationLink,
|
||||||
|
DocumentationTopic,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.archives import ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||||
|
from govoplan_files.backend.storage.access import user_group_ids
|
||||||
|
from govoplan_files.backend.storage.connector_visibility import (
|
||||||
|
connector_profile_usable_for_import,
|
||||||
|
visible_connector_profiles_for_actor,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
|
||||||
|
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
|
||||||
|
_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES = 2 * 1024 * 1024 * 1024
|
||||||
|
_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO = 100
|
||||||
|
_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS = 30 * 60
|
||||||
|
_FILES_READ_SCOPE = "files:file:read"
|
||||||
|
_FILES_UPLOAD_SCOPE = "files:file:upload"
|
||||||
|
|
||||||
|
|
||||||
|
def documentation_topics(
|
||||||
|
context: DocumentationContext,
|
||||||
|
) -> tuple[DocumentationTopic, ...]:
|
||||||
|
if context.documentation_type != "user":
|
||||||
|
return ()
|
||||||
|
|
||||||
|
upload_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_upload_max_bytes",
|
||||||
|
default=_DEFAULT_UPLOAD_MAX_BYTES,
|
||||||
|
)
|
||||||
|
zip_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_upload_zip_max_bytes",
|
||||||
|
default=_DEFAULT_ZIP_MAX_BYTES,
|
||||||
|
)
|
||||||
|
archive_expanded_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_archive_max_expanded_bytes",
|
||||||
|
default=_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES,
|
||||||
|
)
|
||||||
|
archive_entry_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_archive_max_entries",
|
||||||
|
default=ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||||
|
)
|
||||||
|
archive_ratio_limit = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_archive_max_expansion_ratio",
|
||||||
|
default=_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO,
|
||||||
|
)
|
||||||
|
archive_preview_ttl = _configured_positive_int(
|
||||||
|
context.settings,
|
||||||
|
"file_archive_preview_ttl_seconds",
|
||||||
|
default=_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS,
|
||||||
|
)
|
||||||
|
topics: list[DocumentationTopic] = []
|
||||||
|
if upload_limit is not None:
|
||||||
|
topics.append(_upload_topic(upload_limit))
|
||||||
|
if (
|
||||||
|
upload_limit is not None
|
||||||
|
and zip_limit is not None
|
||||||
|
and archive_expanded_limit is not None
|
||||||
|
and archive_entry_limit is not None
|
||||||
|
and archive_ratio_limit is not None
|
||||||
|
and archive_preview_ttl is not None
|
||||||
|
):
|
||||||
|
topics.append(
|
||||||
|
_archive_topic(
|
||||||
|
upload_limit,
|
||||||
|
zip_limit,
|
||||||
|
archive_expanded_limit,
|
||||||
|
archive_entry_limit,
|
||||||
|
archive_ratio_limit,
|
||||||
|
archive_preview_ttl,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
topics.append(_connector_import_topic(context))
|
||||||
|
return tuple(topics)
|
||||||
|
|
||||||
|
|
||||||
|
def _upload_topic(max_bytes: int) -> DocumentationTopic:
|
||||||
|
limit = _format_byte_limit(max_bytes)
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.workflow.upload-managed-files",
|
||||||
|
title="Upload managed files",
|
||||||
|
summary=f"Upload files to a personal or accessible group space; each uploaded file may contain at most {limit}.",
|
||||||
|
body=(
|
||||||
|
f"The current deployment accepts at most {limit} for each ordinary upload. "
|
||||||
|
"A name conflict is never resolved silently: reject stops the upload, rename chooses a copy name, and overwrite retires the old asset before creating a new one."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=39,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(
|
||||||
|
label="Files handbook",
|
||||||
|
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||||
|
kind="repository",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
related_modules=("campaigns",),
|
||||||
|
unlocks=(
|
||||||
|
"Users and connected processes can place governed content in managed storage.",
|
||||||
|
),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and upload managed files.",
|
||||||
|
"The destination personal or group space grants this account write access; upload permission alone does not grant access to every space.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and choose My files or an accessible group space.",
|
||||||
|
"Open the intended destination folder and choose Upload, or drag files into the file list.",
|
||||||
|
"For every name conflict, explicitly reject, rename, overwrite, or skip the affected item.",
|
||||||
|
"Wait for the upload to finish, then open the resulting file details.",
|
||||||
|
],
|
||||||
|
"outcome": "Each accepted file is stored as a governed managed asset in the selected space.",
|
||||||
|
"verification": "Confirm the owner, logical path, size, checksum, and current version in Files.",
|
||||||
|
"constraints": [
|
||||||
|
{
|
||||||
|
"id": "ordinary-upload-size",
|
||||||
|
"label": "Maximum size per file",
|
||||||
|
"description": f"The current safe upload limit is {limit} per file.",
|
||||||
|
"values": [limit],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.upload-and-unpack-zip",
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_topic(
|
||||||
|
max_file_bytes: int,
|
||||||
|
max_archive_request_bytes: int,
|
||||||
|
max_expanded_bytes: int,
|
||||||
|
max_entries: int,
|
||||||
|
max_expansion_ratio: int,
|
||||||
|
preview_ttl_seconds: int,
|
||||||
|
) -> DocumentationTopic:
|
||||||
|
member_limit = _format_byte_limit(max_file_bytes)
|
||||||
|
request_limit = _format_byte_limit(max_archive_request_bytes)
|
||||||
|
expanded_limit = _format_byte_limit(max_expanded_bytes)
|
||||||
|
preview_minutes = max(1, preview_ttl_seconds // 60)
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.workflow.upload-and-unpack-zip",
|
||||||
|
title="Preview and unpack an archive",
|
||||||
|
summary=(
|
||||||
|
f"Review and selectively unpack up to {max_entries:,} entries from ZIP or TAR archives before any managed file is created."
|
||||||
|
),
|
||||||
|
body=(
|
||||||
|
f"The deployment accepts ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ requests up to {request_limit}, limits actual expanded data to {expanded_limit}, "
|
||||||
|
f"each member to {member_limit}, the archive to {max_entries:,} entries, and expansion to {max_expansion_ratio}:1. "
|
||||||
|
f"The server-issued preview expires after {preview_minutes} minutes. Password-protected ZIP archives are supported; passwords remain request-only. "
|
||||||
|
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "file_manager", "process_participant"),
|
||||||
|
order=40,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(
|
||||||
|
label="Files handbook",
|
||||||
|
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||||
|
kind="repository",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
unlocks=(
|
||||||
|
"A bounded archive can become governed managed files without trusting archive paths or size declarations.",
|
||||||
|
),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and upload managed files.",
|
||||||
|
"The archive fits the configured request, expansion, size, and entry limits.",
|
||||||
|
"The destination personal or group space grants this account write access.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and choose the managed destination space and folder.",
|
||||||
|
"Enable Preview and unpack archive, then choose or drag one supported archive.",
|
||||||
|
"Review the discovered entries, supply a ZIP password when required, and select the files or folders to import.",
|
||||||
|
"Resolve every destination conflict explicitly.",
|
||||||
|
"Confirm the selection, then wait for extraction and finalization to finish before leaving the page.",
|
||||||
|
],
|
||||||
|
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
|
||||||
|
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
|
||||||
|
"constraints": [
|
||||||
|
{
|
||||||
|
"id": "archive-request-and-total",
|
||||||
|
"label": "Maximum archive request and expanded total",
|
||||||
|
"description": f"The compressed request is limited to {request_limit}; actual expanded data is limited to {expanded_limit}.",
|
||||||
|
"values": [request_limit, expanded_limit],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "archive-member-size",
|
||||||
|
"label": "Maximum extracted member size",
|
||||||
|
"description": f"Each extracted file is limited to {member_limit}.",
|
||||||
|
"values": [member_limit],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "archive-entry-count",
|
||||||
|
"label": "Maximum entry count",
|
||||||
|
"description": f"An archive may contain at most {max_entries:,} declared entries.",
|
||||||
|
"values": [f"{max_entries:,} entries"],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "archive-expansion-ratio",
|
||||||
|
"label": "Maximum expansion ratio",
|
||||||
|
"description": f"Declared and actual output may not exceed {max_expansion_ratio} times the compressed request size.",
|
||||||
|
"values": [f"{max_expansion_ratio}:1"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.workflow.upload-managed-files",
|
||||||
|
"files.workflow.organize-managed-files",
|
||||||
|
"files.workflow.find-and-download-files",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _connector_import_topic(context: DocumentationContext) -> DocumentationTopic:
|
||||||
|
principal = context.principal
|
||||||
|
if not _has_all_scopes(principal, (_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE)):
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"Connector import is not available to this account because both permission to view Files and permission to upload managed files are required."
|
||||||
|
)
|
||||||
|
|
||||||
|
tenant_id = _safe_text_attribute(principal, "tenant_id")
|
||||||
|
user_id = _safe_text_attribute(getattr(principal, "user", None), "id")
|
||||||
|
session = context.session
|
||||||
|
if not tenant_id or not user_id or not isinstance(session, Session):
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
member_group_ids = _actor_group_ids(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
include_admin_groups=False,
|
||||||
|
)
|
||||||
|
connector_group_ids = (
|
||||||
|
_actor_group_ids(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
include_admin_groups=True,
|
||||||
|
)
|
||||||
|
if _has_all_scopes(principal, ("files:file:admin",))
|
||||||
|
else member_group_ids
|
||||||
|
)
|
||||||
|
profiles = visible_connector_profiles_for_actor(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=connector_group_ids,
|
||||||
|
settings=context.settings,
|
||||||
|
campaign_visible=_campaign_visibility(
|
||||||
|
context,
|
||||||
|
session=session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=member_group_ids,
|
||||||
|
),
|
||||||
|
include_effective_policy=True,
|
||||||
|
)
|
||||||
|
usable_profiles = tuple(
|
||||||
|
profile
|
||||||
|
for profile in profiles
|
||||||
|
if connector_profile_usable_for_import(profile)
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||||
|
)
|
||||||
|
if not usable_profiles:
|
||||||
|
return _connector_import_limitation(
|
||||||
|
"No connection visible to this account is currently eligible to offer an enabled, credential-ready, policy-allowed browse/import path through a pinning-safe provider. Ask a Files administrator to configure or authorize one."
|
||||||
|
)
|
||||||
|
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.workflow.import-managed-snapshot",
|
||||||
|
title="Import an external file as a governed snapshot",
|
||||||
|
summary="Browse an authorized connection read-only and import one selected file into managed storage as a frozen, traceable snapshot.",
|
||||||
|
body=(
|
||||||
|
"At least one connection visible to this account is currently eligible to offer the governed browse/import path. "
|
||||||
|
"The selected remote path and item are re-authorized when used, and browse, import, and sync never mutate the remote source. "
|
||||||
|
"The managed snapshot stays unchanged until an explicit manual sync."
|
||||||
|
),
|
||||||
|
layer="configured",
|
||||||
|
documentation_types=("user",),
|
||||||
|
audience=("file_user", "campaign_manager", "report_author"),
|
||||||
|
order=41,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(
|
||||||
|
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||||
|
DocumentationLink(
|
||||||
|
label="Files handbook",
|
||||||
|
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||||
|
kind="repository",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
related_modules=("campaigns",),
|
||||||
|
unlocks=(
|
||||||
|
"Campaigns, reports, and workflows can consume a managed snapshot with stable source evidence.",
|
||||||
|
),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "workflow",
|
||||||
|
"route": "/files",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list", "files.connector-import"],
|
||||||
|
"prerequisites": [
|
||||||
|
"You may view and upload managed files.",
|
||||||
|
"At least one enabled, credential-ready, policy-allowed connection using a pinning-safe provider is visible to this account.",
|
||||||
|
"The selected remote path and item must pass their operation-time policy checks.",
|
||||||
|
"The managed destination space grants this account write access.",
|
||||||
|
],
|
||||||
|
"steps": [
|
||||||
|
"Open Files and choose a managed destination space.",
|
||||||
|
"Choose Sync from connection, select an available connection, and browse to the permitted remote file.",
|
||||||
|
"Import the selected file and resolve any destination conflict explicitly.",
|
||||||
|
"Review the managed file's source and current-version details before using it in another task.",
|
||||||
|
],
|
||||||
|
"current_configuration": [
|
||||||
|
"At least one actor-visible connection is eligible to offer a safe browse/import operation; the selected endpoint, path, and item are still checked when used.",
|
||||||
|
"Every selected remote path and item is re-authorized at operation time.",
|
||||||
|
],
|
||||||
|
"outcome": "The external content is a tenant-managed snapshot with a checksum, exact version, and recorded source context.",
|
||||||
|
"verification": "Reopen the managed file and confirm its recorded source context, source revision when available, checksum, and current version.",
|
||||||
|
"related_topic_ids": [
|
||||||
|
"files.governed-connectors-and-provenance",
|
||||||
|
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||||
|
"files.reference.snapshot-provenance-and-capabilities",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _connector_import_limitation(message: str) -> DocumentationTopic:
|
||||||
|
return DocumentationTopic(
|
||||||
|
id="files.connector-import-unavailable",
|
||||||
|
title="External file import is not currently available",
|
||||||
|
summary=message,
|
||||||
|
body=(
|
||||||
|
f"{message} Files never exposes connection endpoints, storage paths, credential references, or raw connector policies in user documentation."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("user",),
|
||||||
|
order=41,
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
required_modules=("files",),
|
||||||
|
any_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
links=(DocumentationLink(label="Files", href="/files", kind="runtime"),),
|
||||||
|
source_module_id="files",
|
||||||
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
|
"screen": "Files",
|
||||||
|
"help_contexts": ["files.list", "files.connector-import"],
|
||||||
|
"limitations": [message],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _configured_positive_int(
|
||||||
|
settings: object | None, name: str, *, default: int
|
||||||
|
) -> int | None:
|
||||||
|
raw_value = getattr(settings, name, default) if settings is not None else default
|
||||||
|
try:
|
||||||
|
value = int(raw_value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
return value if value > 0 else None
|
||||||
|
|
||||||
|
|
||||||
|
def _format_byte_limit(value: int) -> str:
|
||||||
|
units = ((1024**3, "GiB"), (1024**2, "MiB"), (1024, "KiB"))
|
||||||
|
for divisor, label in units:
|
||||||
|
if value % divisor == 0:
|
||||||
|
return f"{value // divisor:,} {label} ({value:,} bytes)"
|
||||||
|
return f"{value:,} bytes"
|
||||||
|
|
||||||
|
|
||||||
|
def _has_all_scopes(principal: object | None, scopes: tuple[str, ...]) -> bool:
|
||||||
|
checker = getattr(principal, "has", None)
|
||||||
|
if callable(checker):
|
||||||
|
try:
|
||||||
|
return all(bool(checker(scope)) for scope in scopes)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
granted = {str(scope) for scope in getattr(principal, "scopes", ())}
|
||||||
|
return all(scope in granted for scope in scopes)
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_text_attribute(value: object | None, name: str) -> str:
|
||||||
|
try:
|
||||||
|
result = getattr(value, name, "")
|
||||||
|
except Exception:
|
||||||
|
return ""
|
||||||
|
return str(result or "")
|
||||||
|
|
||||||
|
|
||||||
|
def _principal_group_ids(principal: object) -> tuple[str, ...]:
|
||||||
|
try:
|
||||||
|
values = getattr(principal, "group_ids", ())
|
||||||
|
except Exception:
|
||||||
|
return ()
|
||||||
|
return tuple(str(value) for value in values if str(value))
|
||||||
|
|
||||||
|
|
||||||
|
def _actor_group_ids(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: object,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
include_admin_groups: bool,
|
||||||
|
) -> tuple[str, ...]:
|
||||||
|
try:
|
||||||
|
values = user_group_ids(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
include_admin_groups=include_admin_groups,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return _principal_group_ids(principal)
|
||||||
|
return tuple(str(value) for value in values if str(value))
|
||||||
|
|
||||||
|
|
||||||
|
def _campaign_visibility(
|
||||||
|
context: DocumentationContext,
|
||||||
|
*,
|
||||||
|
session: Session,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: tuple[str, ...],
|
||||||
|
):
|
||||||
|
principal = context.principal
|
||||||
|
registry = context.registry
|
||||||
|
if not _has_all_scopes(principal, ("campaigns:campaign:read",)):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
if not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||||
|
return None
|
||||||
|
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
if not isinstance(capability, CampaignAccessProvider):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def campaign_visible(campaign_id: str) -> bool:
|
||||||
|
try:
|
||||||
|
return capability.campaign_exists(
|
||||||
|
session, tenant_id=tenant_id, campaign_id=campaign_id
|
||||||
|
) and capability.can_read_campaign(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=group_ids,
|
||||||
|
tenant_admin=_has_all_scopes(principal, ("tenant:*",)),
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
return campaign_visible
|
||||||
@@ -0,0 +1,879 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from sqlalchemy import inspect, or_
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.dsar import (
|
||||||
|
DsarErasureActionRef,
|
||||||
|
DsarExecutionResultRef,
|
||||||
|
DsarRecordRef,
|
||||||
|
DsarSubjectRef,
|
||||||
|
dsar_capability_name,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
CampaignAttachmentUse,
|
||||||
|
FileAsset,
|
||||||
|
FileConnectorCredential,
|
||||||
|
FileConnectorPolicy,
|
||||||
|
FileConnectorProfile,
|
||||||
|
FileConnectorSpace,
|
||||||
|
FileFolder,
|
||||||
|
FileFormEvidenceGrant,
|
||||||
|
FileIntegrityFinding,
|
||||||
|
FileIntegrityScan,
|
||||||
|
FileShare,
|
||||||
|
FileVersion,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
FILES_DSAR_CAPABILITY = dsar_capability_name("files")
|
||||||
|
_MAX_RECORDS = 5_000
|
||||||
|
|
||||||
|
|
||||||
|
class FilesDsarProvider:
|
||||||
|
provider_id = "files"
|
||||||
|
module_id = "files"
|
||||||
|
|
||||||
|
def search_subject(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject: DsarSubjectRef,
|
||||||
|
) -> Sequence[DsarRecordRef]:
|
||||||
|
db = _session(session)
|
||||||
|
subject_user_id = _subject_user_id(subject)
|
||||||
|
if subject_user_id is None:
|
||||||
|
return ()
|
||||||
|
|
||||||
|
records: list[DsarRecordRef] = []
|
||||||
|
form_evidence_available = _has_table(db, FileFormEvidenceGrant)
|
||||||
|
campaign_evidence_available = _has_table(db, CampaignAttachmentUse)
|
||||||
|
retention_reasons: dict[str, str | None] = {}
|
||||||
|
|
||||||
|
def retention_reason(asset: FileAsset) -> str | None:
|
||||||
|
if asset.id not in retention_reasons:
|
||||||
|
retention_reasons[asset.id] = _asset_retention_reason(
|
||||||
|
db,
|
||||||
|
asset,
|
||||||
|
form_evidence_available=form_evidence_available,
|
||||||
|
campaign_evidence_available=campaign_evidence_available,
|
||||||
|
)
|
||||||
|
return retention_reasons[asset.id]
|
||||||
|
|
||||||
|
def append(record: DsarRecordRef) -> None:
|
||||||
|
if len(records) >= _MAX_RECORDS:
|
||||||
|
raise ValueError(
|
||||||
|
"Files DSAR match limit exceeded; narrow the subject selectors."
|
||||||
|
)
|
||||||
|
records.append(record)
|
||||||
|
|
||||||
|
assets = _bounded_rows(
|
||||||
|
db.query(FileAsset)
|
||||||
|
.filter(
|
||||||
|
FileAsset.tenant_id == tenant_id,
|
||||||
|
or_(
|
||||||
|
FileAsset.owner_user_id == subject_user_id,
|
||||||
|
FileAsset.created_by_user_id == subject_user_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.order_by(FileAsset.id)
|
||||||
|
)
|
||||||
|
for asset in assets:
|
||||||
|
match_fields = _matching_fields(
|
||||||
|
asset,
|
||||||
|
subject_user_id,
|
||||||
|
("owner_user_id", "created_by_user_id"),
|
||||||
|
)
|
||||||
|
asset_retention_reason = retention_reason(asset)
|
||||||
|
append(
|
||||||
|
_record(
|
||||||
|
"file_asset",
|
||||||
|
asset.id,
|
||||||
|
"managed_file",
|
||||||
|
asset.filename,
|
||||||
|
{
|
||||||
|
"match_fields": match_fields,
|
||||||
|
"owner_type": asset.owner_type,
|
||||||
|
"display_path": asset.display_path,
|
||||||
|
"filename": asset.filename,
|
||||||
|
"description": asset.description,
|
||||||
|
"deleted_at": _iso(asset.deleted_at),
|
||||||
|
"retained_until": _iso(asset.retained_until),
|
||||||
|
"legal_hold": asset.legal_hold,
|
||||||
|
"lifecycle_reason": asset.lifecycle_reason,
|
||||||
|
"lifecycle_revision": asset.lifecycle_revision,
|
||||||
|
},
|
||||||
|
observed_at=asset.updated_at,
|
||||||
|
immutable=asset_retention_reason is not None,
|
||||||
|
retention_reason=asset_retention_reason,
|
||||||
|
source_path=f"/files?file={asset.id}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
version_rows = _bounded_rows(
|
||||||
|
db.query(FileVersion, FileAsset)
|
||||||
|
.join(FileAsset, FileAsset.id == FileVersion.file_asset_id)
|
||||||
|
.filter(
|
||||||
|
FileVersion.tenant_id == tenant_id,
|
||||||
|
FileAsset.tenant_id == tenant_id,
|
||||||
|
or_(
|
||||||
|
FileAsset.owner_user_id == subject_user_id,
|
||||||
|
FileVersion.created_by_user_id == subject_user_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.order_by(FileVersion.id)
|
||||||
|
)
|
||||||
|
for version, asset in version_rows:
|
||||||
|
asset_retention_reason = retention_reason(asset)
|
||||||
|
append(
|
||||||
|
_record(
|
||||||
|
"file_version",
|
||||||
|
version.id,
|
||||||
|
"managed_file_version",
|
||||||
|
version.filename_at_upload,
|
||||||
|
{
|
||||||
|
"match_fields": (
|
||||||
|
["asset.owner_user_id"]
|
||||||
|
if asset.owner_user_id == subject_user_id
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
+ (
|
||||||
|
["created_by_user_id"]
|
||||||
|
if version.created_by_user_id == subject_user_id
|
||||||
|
else []
|
||||||
|
),
|
||||||
|
"file_asset_id": version.file_asset_id,
|
||||||
|
"version_number": version.version_number,
|
||||||
|
"filename_at_upload": version.filename_at_upload,
|
||||||
|
"display_path_at_upload": version.display_path_at_upload,
|
||||||
|
"content_type": version.content_type,
|
||||||
|
"size_bytes": version.size_bytes,
|
||||||
|
"checksum_sha256": version.checksum_sha256,
|
||||||
|
"created_at": _iso(version.created_at),
|
||||||
|
},
|
||||||
|
observed_at=version.updated_at,
|
||||||
|
immutable=asset_retention_reason is not None,
|
||||||
|
retention_reason=asset_retention_reason,
|
||||||
|
source_path=(
|
||||||
|
f"/api/v1/files/{version.file_asset_id}/versions/"
|
||||||
|
f"{version.id}/download"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for folder in _bounded_rows(
|
||||||
|
db.query(FileFolder)
|
||||||
|
.filter(
|
||||||
|
FileFolder.tenant_id == tenant_id,
|
||||||
|
or_(
|
||||||
|
FileFolder.owner_user_id == subject_user_id,
|
||||||
|
FileFolder.created_by_user_id == subject_user_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.order_by(FileFolder.id)
|
||||||
|
):
|
||||||
|
append(
|
||||||
|
_record(
|
||||||
|
"file_folder",
|
||||||
|
folder.id,
|
||||||
|
"managed_folder",
|
||||||
|
folder.path,
|
||||||
|
{
|
||||||
|
"match_fields": _matching_fields(
|
||||||
|
folder,
|
||||||
|
subject_user_id,
|
||||||
|
("owner_user_id", "created_by_user_id"),
|
||||||
|
),
|
||||||
|
"owner_type": folder.owner_type,
|
||||||
|
"path": folder.path,
|
||||||
|
"deleted_at": _iso(folder.deleted_at),
|
||||||
|
},
|
||||||
|
observed_at=folder.updated_at,
|
||||||
|
source_path="/files",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for share in _bounded_rows(
|
||||||
|
db.query(FileShare)
|
||||||
|
.filter(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
or_(
|
||||||
|
(FileShare.target_type == "user")
|
||||||
|
& (FileShare.target_id == subject_user_id),
|
||||||
|
FileShare.created_by_user_id == subject_user_id,
|
||||||
|
FileShare.revoked_by_user_id == subject_user_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.order_by(FileShare.id)
|
||||||
|
):
|
||||||
|
target_matches = (
|
||||||
|
share.target_type == "user" and share.target_id == subject_user_id
|
||||||
|
)
|
||||||
|
match_fields = _matching_fields(
|
||||||
|
share,
|
||||||
|
subject_user_id,
|
||||||
|
("created_by_user_id", "revoked_by_user_id"),
|
||||||
|
)
|
||||||
|
if target_matches:
|
||||||
|
match_fields.insert(0, "target_id")
|
||||||
|
append(
|
||||||
|
_record(
|
||||||
|
"file_share",
|
||||||
|
share.id,
|
||||||
|
"file_access_evidence",
|
||||||
|
f"Share for file {share.file_asset_id}",
|
||||||
|
{
|
||||||
|
"match_fields": match_fields,
|
||||||
|
"file_asset_id": share.file_asset_id,
|
||||||
|
"target_type": share.target_type,
|
||||||
|
"target_is_subject": target_matches,
|
||||||
|
"permission": share.permission,
|
||||||
|
"expires_at": _iso(share.expires_at),
|
||||||
|
"revoked_at": _iso(share.revoked_at),
|
||||||
|
},
|
||||||
|
observed_at=share.updated_at,
|
||||||
|
immutable=True,
|
||||||
|
retention_reason=(
|
||||||
|
"File-sharing history is institutional access evidence."
|
||||||
|
),
|
||||||
|
source_path=f"/files?file={share.file_asset_id}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if form_evidence_available:
|
||||||
|
evidence_rows = _bounded_rows(
|
||||||
|
db.query(FileFormEvidenceGrant)
|
||||||
|
.outerjoin(
|
||||||
|
FileAsset,
|
||||||
|
FileAsset.id == FileFormEvidenceGrant.file_asset_id,
|
||||||
|
)
|
||||||
|
.filter(
|
||||||
|
FileFormEvidenceGrant.tenant_id == tenant_id,
|
||||||
|
or_(
|
||||||
|
FileFormEvidenceGrant.custodian_user_id == subject_user_id,
|
||||||
|
FileAsset.owner_user_id == subject_user_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.order_by(FileFormEvidenceGrant.id)
|
||||||
|
)
|
||||||
|
for grant in evidence_rows:
|
||||||
|
append(
|
||||||
|
_record(
|
||||||
|
"file_form_evidence",
|
||||||
|
grant.id,
|
||||||
|
"form_evidence",
|
||||||
|
f"Form evidence {grant.form_definition_id}",
|
||||||
|
{
|
||||||
|
"match_fields": (
|
||||||
|
["custodian_user_id"]
|
||||||
|
if grant.custodian_user_id == subject_user_id
|
||||||
|
else ["asset.owner_user_id"]
|
||||||
|
),
|
||||||
|
"form_instance_id": grant.form_instance_id,
|
||||||
|
"form_definition_id": grant.form_definition_id,
|
||||||
|
"form_definition_revision": grant.form_definition_revision,
|
||||||
|
"evidence_kind": grant.evidence_kind,
|
||||||
|
"purpose": grant.purpose,
|
||||||
|
"status": grant.status,
|
||||||
|
"expires_at": _iso(grant.expires_at),
|
||||||
|
"file_asset_id": grant.file_asset_id,
|
||||||
|
"file_version_id": grant.file_version_id,
|
||||||
|
},
|
||||||
|
observed_at=grant.updated_at,
|
||||||
|
immutable=True,
|
||||||
|
retention_reason=(
|
||||||
|
"Submitted Form attachment evidence follows the owning "
|
||||||
|
"process retention and cannot be erased through Files alone."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if campaign_evidence_available:
|
||||||
|
campaign_rows = _bounded_rows(
|
||||||
|
db.query(CampaignAttachmentUse)
|
||||||
|
.join(FileAsset, FileAsset.id == CampaignAttachmentUse.file_asset_id)
|
||||||
|
.filter(
|
||||||
|
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||||
|
FileAsset.owner_user_id == subject_user_id,
|
||||||
|
)
|
||||||
|
.order_by(CampaignAttachmentUse.id)
|
||||||
|
)
|
||||||
|
for use in campaign_rows:
|
||||||
|
append(
|
||||||
|
_record(
|
||||||
|
"campaign_attachment_use",
|
||||||
|
use.id,
|
||||||
|
"delivery_evidence",
|
||||||
|
use.filename_used,
|
||||||
|
{
|
||||||
|
"match_fields": ["asset.owner_user_id"],
|
||||||
|
"campaign_id": use.campaign_id,
|
||||||
|
"campaign_version_id": use.campaign_version_id,
|
||||||
|
"campaign_job_id": use.campaign_job_id,
|
||||||
|
"file_asset_id": use.file_asset_id,
|
||||||
|
"file_version_id": use.file_version_id,
|
||||||
|
"filename_used": use.filename_used,
|
||||||
|
"checksum_sha256": use.checksum_sha256,
|
||||||
|
"size_bytes": use.size_bytes,
|
||||||
|
"use_stage": use.use_stage,
|
||||||
|
"used_at": _iso(use.used_at),
|
||||||
|
},
|
||||||
|
observed_at=use.updated_at,
|
||||||
|
immutable=True,
|
||||||
|
retention_reason=(
|
||||||
|
"Campaign attachment use is immutable delivery evidence."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
self._append_configuration_references(
|
||||||
|
db,
|
||||||
|
append=append,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_user_id=subject_user_id,
|
||||||
|
)
|
||||||
|
self._append_integrity_references(
|
||||||
|
db,
|
||||||
|
append=append,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_user_id=subject_user_id,
|
||||||
|
)
|
||||||
|
return tuple(records)
|
||||||
|
|
||||||
|
def plan_erasure(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject: DsarSubjectRef,
|
||||||
|
records: Sequence[DsarRecordRef],
|
||||||
|
) -> Sequence[DsarErasureActionRef]:
|
||||||
|
del session
|
||||||
|
subject_user_id = _subject_user_id(subject)
|
||||||
|
if subject_user_id is None:
|
||||||
|
return ()
|
||||||
|
actions: list[DsarErasureActionRef] = []
|
||||||
|
for record in records:
|
||||||
|
if record.provider_id != self.provider_id or record.module_id != self.module_id:
|
||||||
|
raise ValueError("Files DSAR received a foreign provider record.")
|
||||||
|
match_fields = {
|
||||||
|
str(value) for value in record.data.get("match_fields", ())
|
||||||
|
}
|
||||||
|
if record.immutable_evidence:
|
||||||
|
actions.append(
|
||||||
|
_action(
|
||||||
|
f"files:retain:{record.resource_type}:{record.resource_id}",
|
||||||
|
"retain",
|
||||||
|
record,
|
||||||
|
f"Retain {record.title}",
|
||||||
|
record.retention_reason
|
||||||
|
or "Institutional evidence must be retained.",
|
||||||
|
executable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif record.resource_type in {
|
||||||
|
"file_asset",
|
||||||
|
"file_version",
|
||||||
|
"file_folder",
|
||||||
|
}:
|
||||||
|
actions.append(
|
||||||
|
_action(
|
||||||
|
f"files:review:{record.resource_type}:{record.resource_id}",
|
||||||
|
"manual_review",
|
||||||
|
record,
|
||||||
|
f"Review {record.title}",
|
||||||
|
(
|
||||||
|
"Managed file content, names, paths, ownership, and shared "
|
||||||
|
"references require a case decision. Approved byte erasure "
|
||||||
|
"must use the separate Files purge workflow."
|
||||||
|
),
|
||||||
|
executable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
record.resource_type == "file_share"
|
||||||
|
and "target_id" in match_fields
|
||||||
|
and record.data.get("revoked_at") is None
|
||||||
|
):
|
||||||
|
actions.append(
|
||||||
|
_action(
|
||||||
|
f"files:revoke:file_share:{record.resource_id}",
|
||||||
|
"revoke",
|
||||||
|
record,
|
||||||
|
"Revoke active file share",
|
||||||
|
"The active user-targeted share can be revoked without deleting file evidence.",
|
||||||
|
executable=True,
|
||||||
|
metadata={"subject_user_id": subject_user_id},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for field_name in sorted(
|
||||||
|
match_fields.intersection(_detachable_fields(record.resource_type))
|
||||||
|
):
|
||||||
|
actions.append(
|
||||||
|
_action(
|
||||||
|
(
|
||||||
|
f"files:detach:{record.resource_type}:"
|
||||||
|
f"{field_name}:{record.resource_id}"
|
||||||
|
),
|
||||||
|
"detach",
|
||||||
|
record,
|
||||||
|
f"Detach {field_name.replace('_', ' ')}",
|
||||||
|
(
|
||||||
|
"Remove the mutable subject reference while preserving the "
|
||||||
|
"governed resource and DSAR evidence."
|
||||||
|
),
|
||||||
|
executable=True,
|
||||||
|
metadata={
|
||||||
|
"field": field_name,
|
||||||
|
"subject_user_id": subject_user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
action_ids = [action.action_id for action in actions]
|
||||||
|
if len(action_ids) != len(set(action_ids)):
|
||||||
|
raise ValueError("Files DSAR produced duplicate action ids.")
|
||||||
|
return tuple(actions)
|
||||||
|
|
||||||
|
def execute_erasure(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject: DsarSubjectRef,
|
||||||
|
actions: Sequence[DsarErasureActionRef],
|
||||||
|
request_id: str,
|
||||||
|
) -> Sequence[DsarExecutionResultRef]:
|
||||||
|
db = _session(session)
|
||||||
|
subject_user_id = _subject_user_id(subject)
|
||||||
|
if subject_user_id is None:
|
||||||
|
return tuple(
|
||||||
|
_blocked(action, "Files requires a direct membership subject reference.")
|
||||||
|
for action in actions
|
||||||
|
)
|
||||||
|
results: list[DsarExecutionResultRef] = []
|
||||||
|
for action in actions:
|
||||||
|
if (
|
||||||
|
action.provider_id != self.provider_id
|
||||||
|
or action.module_id != self.module_id
|
||||||
|
or action.metadata.get("subject_user_id") != subject_user_id
|
||||||
|
):
|
||||||
|
results.append(_blocked(action, "The Files DSAR action is stale or invalid."))
|
||||||
|
continue
|
||||||
|
if action.action_id.startswith("files:revoke:file_share:"):
|
||||||
|
results.append(
|
||||||
|
_revoke_share(
|
||||||
|
db,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_user_id=subject_user_id,
|
||||||
|
action=action,
|
||||||
|
request_id=request_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif action.action_id.startswith("files:detach:"):
|
||||||
|
results.append(
|
||||||
|
_detach_reference(
|
||||||
|
db,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_user_id=subject_user_id,
|
||||||
|
action=action,
|
||||||
|
request_id=request_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
results.append(_blocked(action, "Files does not execute this action kind."))
|
||||||
|
db.flush()
|
||||||
|
return tuple(results)
|
||||||
|
|
||||||
|
def _append_configuration_references(
|
||||||
|
self,
|
||||||
|
db: Session,
|
||||||
|
*,
|
||||||
|
append: object,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_user_id: str,
|
||||||
|
) -> None:
|
||||||
|
configurations = (
|
||||||
|
(FileConnectorProfile, "connector_profile", ("created_by_user_id", "updated_by_user_id")),
|
||||||
|
(FileConnectorCredential, "connector_credential", ("created_by_user_id", "updated_by_user_id")),
|
||||||
|
(FileConnectorPolicy, "connector_policy", ("created_by_user_id", "updated_by_user_id")),
|
||||||
|
(FileConnectorSpace, "connector_space", ("owner_user_id", "created_by_user_id")),
|
||||||
|
)
|
||||||
|
for model, resource_type, fields in configurations:
|
||||||
|
if not _has_table(db, model):
|
||||||
|
continue
|
||||||
|
conditions = [getattr(model, field) == subject_user_id for field in fields]
|
||||||
|
query = db.query(model).filter(or_(*conditions))
|
||||||
|
if hasattr(model, "tenant_id"):
|
||||||
|
query = query.filter(model.tenant_id == tenant_id)
|
||||||
|
for row in _bounded_rows(query.order_by(model.id)):
|
||||||
|
match_fields = _matching_fields(row, subject_user_id, fields)
|
||||||
|
data: dict[str, object] = {
|
||||||
|
"match_fields": match_fields,
|
||||||
|
"label": getattr(row, "label", None),
|
||||||
|
"provider": getattr(row, "provider", None),
|
||||||
|
}
|
||||||
|
if isinstance(row, FileConnectorCredential):
|
||||||
|
data["credential_mode"] = row.credential_mode
|
||||||
|
elif isinstance(row, FileConnectorSpace):
|
||||||
|
data.update(
|
||||||
|
{
|
||||||
|
"remote_path": row.remote_path,
|
||||||
|
"sync_mode": row.sync_mode,
|
||||||
|
"read_only": row.read_only,
|
||||||
|
"deleted_at": _iso(row.deleted_at),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
append( # type: ignore[operator]
|
||||||
|
_record(
|
||||||
|
resource_type,
|
||||||
|
row.id,
|
||||||
|
"connector_configuration_evidence",
|
||||||
|
getattr(row, "label", None) or resource_type.replace("_", " "),
|
||||||
|
data,
|
||||||
|
observed_at=row.updated_at,
|
||||||
|
immutable=True,
|
||||||
|
retention_reason=(
|
||||||
|
"Connector configuration history is institutional evidence; "
|
||||||
|
"credential secrets are excluded from the DSAR export."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def _append_integrity_references(
|
||||||
|
self,
|
||||||
|
db: Session,
|
||||||
|
*,
|
||||||
|
append: object,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_user_id: str,
|
||||||
|
) -> None:
|
||||||
|
if _has_table(db, FileIntegrityScan):
|
||||||
|
for scan in _bounded_rows(
|
||||||
|
db.query(FileIntegrityScan)
|
||||||
|
.filter(
|
||||||
|
FileIntegrityScan.tenant_id == tenant_id,
|
||||||
|
FileIntegrityScan.created_by_user_id == subject_user_id,
|
||||||
|
)
|
||||||
|
.order_by(FileIntegrityScan.id)
|
||||||
|
):
|
||||||
|
append( # type: ignore[operator]
|
||||||
|
_record(
|
||||||
|
"file_integrity_scan",
|
||||||
|
scan.id,
|
||||||
|
"storage_integrity_evidence",
|
||||||
|
f"Integrity scan {scan.id}",
|
||||||
|
{
|
||||||
|
"match_fields": ["created_by_user_id"],
|
||||||
|
"storage_backend": scan.storage_backend,
|
||||||
|
"status": scan.status,
|
||||||
|
"started_at": _iso(scan.started_at),
|
||||||
|
"completed_at": _iso(scan.completed_at),
|
||||||
|
},
|
||||||
|
observed_at=scan.updated_at,
|
||||||
|
immutable=True,
|
||||||
|
retention_reason="Storage integrity scans are operator evidence.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if _has_table(db, FileIntegrityFinding):
|
||||||
|
for finding in _bounded_rows(
|
||||||
|
db.query(FileIntegrityFinding)
|
||||||
|
.filter(
|
||||||
|
FileIntegrityFinding.tenant_id == tenant_id,
|
||||||
|
FileIntegrityFinding.resolved_by_user_id == subject_user_id,
|
||||||
|
)
|
||||||
|
.order_by(FileIntegrityFinding.id)
|
||||||
|
):
|
||||||
|
append( # type: ignore[operator]
|
||||||
|
_record(
|
||||||
|
"file_integrity_finding",
|
||||||
|
finding.id,
|
||||||
|
"storage_integrity_evidence",
|
||||||
|
f"Integrity finding {finding.kind}",
|
||||||
|
{
|
||||||
|
"match_fields": ["resolved_by_user_id"],
|
||||||
|
"kind": finding.kind,
|
||||||
|
"state": finding.state,
|
||||||
|
"resolved_at": _iso(finding.resolved_at),
|
||||||
|
},
|
||||||
|
observed_at=finding.updated_at,
|
||||||
|
immutable=True,
|
||||||
|
retention_reason="Integrity resolution is operator evidence.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_retention_reason(
|
||||||
|
session: Session,
|
||||||
|
asset: FileAsset,
|
||||||
|
*,
|
||||||
|
form_evidence_available: bool,
|
||||||
|
campaign_evidence_available: bool,
|
||||||
|
) -> str | None:
|
||||||
|
reasons: list[str] = []
|
||||||
|
if asset.legal_hold:
|
||||||
|
reasons.append("The file is under legal hold.")
|
||||||
|
retained_until = _aware(asset.retained_until)
|
||||||
|
if retained_until is not None and retained_until > datetime.now(timezone.utc):
|
||||||
|
reasons.append(f"The file is retained until {retained_until.isoformat()}.")
|
||||||
|
if form_evidence_available:
|
||||||
|
has_form_evidence = (
|
||||||
|
session.query(FileFormEvidenceGrant.id)
|
||||||
|
.filter(
|
||||||
|
FileFormEvidenceGrant.tenant_id == asset.tenant_id,
|
||||||
|
FileFormEvidenceGrant.file_asset_id == asset.id,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
if has_form_evidence:
|
||||||
|
reasons.append("The file is referenced by submitted Form evidence.")
|
||||||
|
if campaign_evidence_available:
|
||||||
|
has_campaign_evidence = (
|
||||||
|
session.query(CampaignAttachmentUse.id)
|
||||||
|
.filter(
|
||||||
|
CampaignAttachmentUse.tenant_id == asset.tenant_id,
|
||||||
|
CampaignAttachmentUse.file_asset_id == asset.id,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
if has_campaign_evidence:
|
||||||
|
reasons.append("The file is referenced by Campaign delivery evidence.")
|
||||||
|
return " ".join(reasons) or None
|
||||||
|
|
||||||
|
|
||||||
|
_DETACHABLE_MODELS: dict[str, tuple[type[object], frozenset[str]]] = {
|
||||||
|
"file_asset": (FileAsset, frozenset({"created_by_user_id"})),
|
||||||
|
"file_version": (FileVersion, frozenset({"created_by_user_id"})),
|
||||||
|
"file_folder": (FileFolder, frozenset({"created_by_user_id"})),
|
||||||
|
"file_share": (
|
||||||
|
FileShare,
|
||||||
|
frozenset({"created_by_user_id", "revoked_by_user_id"}),
|
||||||
|
),
|
||||||
|
"connector_profile": (
|
||||||
|
FileConnectorProfile,
|
||||||
|
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||||
|
),
|
||||||
|
"connector_credential": (
|
||||||
|
FileConnectorCredential,
|
||||||
|
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||||
|
),
|
||||||
|
"connector_policy": (
|
||||||
|
FileConnectorPolicy,
|
||||||
|
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||||
|
),
|
||||||
|
"connector_space": (FileConnectorSpace, frozenset({"created_by_user_id"})),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _detachable_fields(resource_type: str) -> frozenset[str]:
|
||||||
|
entry = _DETACHABLE_MODELS.get(resource_type)
|
||||||
|
return entry[1] if entry is not None else frozenset()
|
||||||
|
|
||||||
|
|
||||||
|
def _revoke_share(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_user_id: str,
|
||||||
|
action: DsarErasureActionRef,
|
||||||
|
request_id: str,
|
||||||
|
) -> DsarExecutionResultRef:
|
||||||
|
row = (
|
||||||
|
session.query(FileShare)
|
||||||
|
.filter(FileShare.id == action.resource_id)
|
||||||
|
.with_for_update()
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
row is None
|
||||||
|
or row.tenant_id != tenant_id
|
||||||
|
or row.target_type != "user"
|
||||||
|
or row.target_id != subject_user_id
|
||||||
|
):
|
||||||
|
return _blocked(action, "The subject-targeted file share is no longer available.")
|
||||||
|
if row.revoked_at is not None:
|
||||||
|
return _result(
|
||||||
|
action,
|
||||||
|
"unchanged",
|
||||||
|
"The file share was already revoked.",
|
||||||
|
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||||
|
)
|
||||||
|
row.revoked_at = datetime.now(timezone.utc)
|
||||||
|
row.revoked_by_user_id = None
|
||||||
|
return _result(
|
||||||
|
action,
|
||||||
|
"executed",
|
||||||
|
"The subject-targeted file share was revoked.",
|
||||||
|
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _detach_reference(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_user_id: str,
|
||||||
|
action: DsarErasureActionRef,
|
||||||
|
request_id: str,
|
||||||
|
) -> DsarExecutionResultRef:
|
||||||
|
entry = _DETACHABLE_MODELS.get(action.resource_type)
|
||||||
|
field_name = str(action.metadata.get("field") or "")
|
||||||
|
if entry is None or field_name not in entry[1]:
|
||||||
|
return _blocked(action, "The requested Files subject reference is not detachable.")
|
||||||
|
model = entry[0]
|
||||||
|
row = (
|
||||||
|
session.query(model)
|
||||||
|
.filter(getattr(model, "id") == action.resource_id)
|
||||||
|
.with_for_update()
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if row is None or getattr(row, "tenant_id", None) != tenant_id:
|
||||||
|
return _blocked(action, "The Files resource is no longer available.")
|
||||||
|
current = getattr(row, field_name)
|
||||||
|
if current is None:
|
||||||
|
return _result(
|
||||||
|
action,
|
||||||
|
"unchanged",
|
||||||
|
"The subject reference was already detached.",
|
||||||
|
{"request_id": request_id, "field": field_name},
|
||||||
|
)
|
||||||
|
if current != subject_user_id:
|
||||||
|
return _blocked(action, "The Files subject reference changed after planning.")
|
||||||
|
setattr(row, field_name, None)
|
||||||
|
return _result(
|
||||||
|
action,
|
||||||
|
"executed",
|
||||||
|
"The mutable subject reference was detached.",
|
||||||
|
{"request_id": request_id, "field": field_name},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _subject_user_id(subject: DsarSubjectRef) -> str | None:
|
||||||
|
candidates: list[str] = []
|
||||||
|
if subject.membership_id:
|
||||||
|
candidates.append(subject.membership_id)
|
||||||
|
for key, value in subject.external_references.items():
|
||||||
|
if key in {
|
||||||
|
"files.user",
|
||||||
|
"files.membership",
|
||||||
|
"access.membership",
|
||||||
|
"membership_id",
|
||||||
|
}:
|
||||||
|
candidates.append(value)
|
||||||
|
normalized = {value.strip() for value in candidates if value.strip()}
|
||||||
|
if len(normalized) != 1:
|
||||||
|
return None
|
||||||
|
return normalized.pop()
|
||||||
|
|
||||||
|
|
||||||
|
def _matching_fields(
|
||||||
|
row: object,
|
||||||
|
subject_user_id: str,
|
||||||
|
fields: Sequence[str],
|
||||||
|
) -> list[str]:
|
||||||
|
return [field for field in fields if getattr(row, field) == subject_user_id]
|
||||||
|
|
||||||
|
|
||||||
|
def _record(
|
||||||
|
resource_type: str,
|
||||||
|
resource_id: str,
|
||||||
|
category: str,
|
||||||
|
title: str,
|
||||||
|
data: dict[str, object],
|
||||||
|
*,
|
||||||
|
observed_at: datetime | None = None,
|
||||||
|
immutable: bool = False,
|
||||||
|
retention_reason: str | None = None,
|
||||||
|
source_path: str | None = None,
|
||||||
|
) -> DsarRecordRef:
|
||||||
|
return DsarRecordRef(
|
||||||
|
provider_id="files",
|
||||||
|
module_id="files",
|
||||||
|
resource_type=resource_type,
|
||||||
|
resource_id=resource_id,
|
||||||
|
category=category,
|
||||||
|
title=title,
|
||||||
|
data=data,
|
||||||
|
observed_at=observed_at,
|
||||||
|
immutable_evidence=immutable,
|
||||||
|
retention_reason=retention_reason,
|
||||||
|
source_path=source_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _action(
|
||||||
|
action_id: str,
|
||||||
|
kind: str,
|
||||||
|
record: DsarRecordRef,
|
||||||
|
title: str,
|
||||||
|
rationale: str,
|
||||||
|
*,
|
||||||
|
executable: bool,
|
||||||
|
metadata: dict[str, object] | None = None,
|
||||||
|
) -> DsarErasureActionRef:
|
||||||
|
return DsarErasureActionRef(
|
||||||
|
action_id=action_id,
|
||||||
|
provider_id="files",
|
||||||
|
module_id="files",
|
||||||
|
kind=kind, # type: ignore[arg-type]
|
||||||
|
resource_type=record.resource_type,
|
||||||
|
resource_id=record.resource_id,
|
||||||
|
title=title,
|
||||||
|
rationale=rationale,
|
||||||
|
executable=executable,
|
||||||
|
metadata=metadata or {},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _result(
|
||||||
|
action: DsarErasureActionRef,
|
||||||
|
status: str,
|
||||||
|
summary: str,
|
||||||
|
evidence: dict[str, object] | None = None,
|
||||||
|
) -> DsarExecutionResultRef:
|
||||||
|
return DsarExecutionResultRef(
|
||||||
|
action_id=action.action_id,
|
||||||
|
status=status, # type: ignore[arg-type]
|
||||||
|
summary=summary,
|
||||||
|
evidence=evidence or {},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _blocked(action: DsarErasureActionRef, summary: str) -> DsarExecutionResultRef:
|
||||||
|
return _result(action, "blocked", summary)
|
||||||
|
|
||||||
|
|
||||||
|
def _session(value: object) -> Session:
|
||||||
|
if not isinstance(value, Session):
|
||||||
|
raise TypeError("Files DSAR provider requires a SQLAlchemy session.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_rows(query: object) -> list[object]:
|
||||||
|
rows = query.limit(_MAX_RECORDS + 1).all() # type: ignore[attr-defined]
|
||||||
|
if len(rows) > _MAX_RECORDS:
|
||||||
|
raise ValueError("Files DSAR match limit exceeded; narrow the subject selectors.")
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _has_table(session: Session, model: type[object]) -> bool:
|
||||||
|
return inspect(session.connection()).has_table(model.__tablename__)
|
||||||
|
|
||||||
|
|
||||||
|
def _aware(value: datetime | None) -> datetime | None:
|
||||||
|
if value is None or value.tzinfo is not None:
|
||||||
|
return value
|
||||||
|
return value.replace(tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def _iso(value: datetime | None) -> str | None:
|
||||||
|
aware = _aware(value)
|
||||||
|
return aware.isoformat() if aware else None
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["FILES_DSAR_CAPABILITY", "FilesDsarProvider"]
|
||||||
@@ -0,0 +1,449 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import secrets
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.access import CAPABILITY_ACCESS_DIRECTORY, AccessDirectory
|
||||||
|
from govoplan_core.core.form_evidence import (
|
||||||
|
FormEvidenceContractError,
|
||||||
|
FormEvidenceGrant,
|
||||||
|
FormEvidenceGrantRequest,
|
||||||
|
FormEvidenceInspection,
|
||||||
|
FormEvidenceInspectionRequest,
|
||||||
|
FormEvidenceState,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.institutional import EvidenceReference
|
||||||
|
from govoplan_core.core.modules import ModuleContext
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
FileAsset,
|
||||||
|
FileBlob,
|
||||||
|
FileFormEvidenceGrant,
|
||||||
|
FileVersion,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.runtime import configure_runtime
|
||||||
|
|
||||||
|
|
||||||
|
PROVIDER_ID = "files"
|
||||||
|
CAPABILITY_FORM_EVIDENCE_FILES = "forms_runtime.evidence.files"
|
||||||
|
MAX_GRANT_TTL = timedelta(minutes=15)
|
||||||
|
|
||||||
|
|
||||||
|
class FilesFormEvidenceProvider:
|
||||||
|
provider_id = PROVIDER_ID
|
||||||
|
|
||||||
|
def __init__(self, registry: object | None, settings: object) -> None:
|
||||||
|
self._registry = registry
|
||||||
|
self._settings = settings
|
||||||
|
|
||||||
|
def supported_kinds(self) -> Sequence[str]:
|
||||||
|
return ("document",)
|
||||||
|
|
||||||
|
def create_upload_grant(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
request: FormEvidenceGrantRequest,
|
||||||
|
) -> FormEvidenceGrant:
|
||||||
|
db = _session(session)
|
||||||
|
_assert_tenant(principal, request.tenant_id)
|
||||||
|
if request.evidence_kind != "document":
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"Files can accept only document evidence for Forms Runtime."
|
||||||
|
)
|
||||||
|
custodian_user_id = _custodian_user_id(request.custodian_ref)
|
||||||
|
self._assert_active_custodian(
|
||||||
|
tenant_id=request.tenant_id,
|
||||||
|
user_id=custodian_user_id,
|
||||||
|
)
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
if request.expires_at <= now:
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"Form evidence upload grant expiry must be in the future."
|
||||||
|
)
|
||||||
|
allowed_content_types = _content_types(request.allowed_content_types)
|
||||||
|
configured_max = int(getattr(self._settings, "file_upload_max_bytes"))
|
||||||
|
max_size_bytes = min(request.max_size_bytes or configured_max, configured_max)
|
||||||
|
request_sha256 = _request_sha256(
|
||||||
|
request,
|
||||||
|
custodian_user_id=custodian_user_id,
|
||||||
|
max_size_bytes=max_size_bytes,
|
||||||
|
allowed_content_types=allowed_content_types,
|
||||||
|
)
|
||||||
|
existing = (
|
||||||
|
db.query(FileFormEvidenceGrant)
|
||||||
|
.filter(
|
||||||
|
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||||
|
FileFormEvidenceGrant.idempotency_key == request.idempotency_key,
|
||||||
|
)
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if existing is not None:
|
||||||
|
if not secrets.compare_digest(existing.request_sha256, request_sha256):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"Form evidence grant idempotency conflict."
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
existing.status in {"expired", "revoked"}
|
||||||
|
or _aware(existing.expires_at) <= now
|
||||||
|
):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"The existing Form evidence upload grant is no longer usable; "
|
||||||
|
"request a new grant with a new idempotency key."
|
||||||
|
)
|
||||||
|
return _grant_response(existing, upload_token=None, replayed=True)
|
||||||
|
|
||||||
|
metadata = _bounded_metadata(request.metadata)
|
||||||
|
remaining_attachments = metadata.get("remaining_attachments")
|
||||||
|
if isinstance(remaining_attachments, int):
|
||||||
|
existing_attachment_ids = set(metadata.get("existing_attachment_ids", ()))
|
||||||
|
active_grants = (
|
||||||
|
db.query(FileFormEvidenceGrant)
|
||||||
|
.filter(
|
||||||
|
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||||
|
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||||
|
FileFormEvidenceGrant.form_definition_id
|
||||||
|
== request.definition_ref.object_id,
|
||||||
|
FileFormEvidenceGrant.form_definition_revision
|
||||||
|
== str(request.definition_ref.version),
|
||||||
|
FileFormEvidenceGrant.status.in_(("issued", "uploaded")),
|
||||||
|
FileFormEvidenceGrant.expires_at > now,
|
||||||
|
)
|
||||||
|
.with_for_update()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
outstanding = sum(
|
||||||
|
1
|
||||||
|
for item in active_grants
|
||||||
|
if not item.file_asset_id
|
||||||
|
or item.file_asset_id not in existing_attachment_ids
|
||||||
|
)
|
||||||
|
if outstanding >= remaining_attachments:
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"This Form already has the maximum number of active attachment uploads."
|
||||||
|
)
|
||||||
|
|
||||||
|
upload_token = secrets.token_urlsafe(32)
|
||||||
|
grant = FileFormEvidenceGrant(
|
||||||
|
tenant_id=request.tenant_id,
|
||||||
|
form_instance_id=request.instance_id,
|
||||||
|
form_definition_id=request.definition_ref.object_id,
|
||||||
|
form_definition_revision=str(request.definition_ref.version),
|
||||||
|
token_sha256=_token_sha256(upload_token),
|
||||||
|
idempotency_key=request.idempotency_key,
|
||||||
|
request_sha256=request_sha256,
|
||||||
|
custodian_user_id=custodian_user_id,
|
||||||
|
evidence_kind=request.evidence_kind,
|
||||||
|
purpose=request.purpose,
|
||||||
|
status="issued",
|
||||||
|
expires_at=min(request.expires_at, now + MAX_GRANT_TTL),
|
||||||
|
max_size_bytes=max_size_bytes,
|
||||||
|
allowed_content_types=list(allowed_content_types),
|
||||||
|
metadata_=metadata,
|
||||||
|
)
|
||||||
|
db.add(grant)
|
||||||
|
db.flush()
|
||||||
|
return _grant_response(grant, upload_token=upload_token, replayed=False)
|
||||||
|
|
||||||
|
def inspect_evidence(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
request: FormEvidenceInspectionRequest,
|
||||||
|
) -> FormEvidenceInspection:
|
||||||
|
db = _session(session)
|
||||||
|
_assert_tenant(principal, request.tenant_id)
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
if request.evidence.owner_module != PROVIDER_ID:
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="rejected",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The evidence owner does not match the Files provider.",
|
||||||
|
)
|
||||||
|
if request.evidence.kind != "document" or not request.evidence.version:
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="rejected",
|
||||||
|
observed_at=now,
|
||||||
|
reason="Files evidence requires an exact document version.",
|
||||||
|
)
|
||||||
|
grant = (
|
||||||
|
db.query(FileFormEvidenceGrant)
|
||||||
|
.filter(
|
||||||
|
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||||
|
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||||
|
FileFormEvidenceGrant.form_definition_id
|
||||||
|
== request.definition_ref.object_id,
|
||||||
|
FileFormEvidenceGrant.form_definition_revision
|
||||||
|
== str(request.definition_ref.version),
|
||||||
|
FileFormEvidenceGrant.file_asset_id == request.evidence.evidence_id,
|
||||||
|
FileFormEvidenceGrant.file_version_id == request.evidence.version,
|
||||||
|
)
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if grant is None:
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="rejected",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The document was not captured for this exact Form submission.",
|
||||||
|
)
|
||||||
|
if grant.status == "revoked":
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="revoked",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The Form evidence grant was revoked.",
|
||||||
|
)
|
||||||
|
if grant.status != "uploaded":
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="pending",
|
||||||
|
observed_at=now,
|
||||||
|
retryable=True,
|
||||||
|
reason="The Form evidence upload has not completed.",
|
||||||
|
)
|
||||||
|
asset = db.get(FileAsset, grant.file_asset_id)
|
||||||
|
version = db.get(FileVersion, grant.file_version_id)
|
||||||
|
blob = db.get(FileBlob, version.blob_id) if version is not None else None
|
||||||
|
if asset is None or version is None or blob is None:
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="unavailable",
|
||||||
|
observed_at=now,
|
||||||
|
retryable=True,
|
||||||
|
reason="The managed document cannot currently be reconstructed.",
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
asset.tenant_id != request.tenant_id
|
||||||
|
or version.tenant_id != request.tenant_id
|
||||||
|
or blob.tenant_id != request.tenant_id
|
||||||
|
or version.file_asset_id != asset.id
|
||||||
|
):
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="rejected",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The managed document crosses an evidence ownership boundary.",
|
||||||
|
)
|
||||||
|
if asset.deleted_at is not None:
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="revoked",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The managed document is no longer active.",
|
||||||
|
)
|
||||||
|
if blob.quarantined_at is not None or blob.integrity_status == "quarantined":
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="rejected",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The managed document failed its integrity gate.",
|
||||||
|
)
|
||||||
|
if blob.integrity_status != "verified":
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="pending",
|
||||||
|
observed_at=now,
|
||||||
|
retryable=True,
|
||||||
|
reason="The managed document is awaiting integrity verification.",
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
not request.evidence.checksum
|
||||||
|
or not secrets.compare_digest(
|
||||||
|
request.evidence.checksum,
|
||||||
|
version.checksum_sha256,
|
||||||
|
)
|
||||||
|
or not secrets.compare_digest(
|
||||||
|
version.checksum_sha256,
|
||||||
|
blob.checksum_sha256,
|
||||||
|
)
|
||||||
|
):
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="rejected",
|
||||||
|
observed_at=now,
|
||||||
|
reason="The managed document checksum does not match the evidence.",
|
||||||
|
)
|
||||||
|
return _inspection(
|
||||||
|
request.evidence,
|
||||||
|
state="accepted",
|
||||||
|
observed_at=now,
|
||||||
|
metadata={
|
||||||
|
"content_type": version.content_type,
|
||||||
|
"size_bytes": version.size_bytes,
|
||||||
|
"integrity_status": blob.integrity_status,
|
||||||
|
"grant_id": grant.id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _assert_active_custodian(self, *, tenant_id: str, user_id: str) -> None:
|
||||||
|
registry = self._registry
|
||||||
|
if registry is None or not hasattr(registry, "has_capability"):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"The Access directory is unavailable for Form evidence custody."
|
||||||
|
)
|
||||||
|
if not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"The Access directory is unavailable for Form evidence custody."
|
||||||
|
)
|
||||||
|
directory = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||||
|
if not isinstance(directory, AccessDirectory):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"The Access directory capability is invalid."
|
||||||
|
)
|
||||||
|
user = directory.get_user(user_id)
|
||||||
|
if user is None or user.tenant_id != tenant_id or user.status != "active":
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"Form evidence requires an active same-tenant custodian."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def create_files_form_evidence_provider(
|
||||||
|
context: ModuleContext,
|
||||||
|
) -> FilesFormEvidenceProvider:
|
||||||
|
configure_runtime(registry=context.registry, settings=context.settings)
|
||||||
|
return FilesFormEvidenceProvider(context.registry, context.settings)
|
||||||
|
|
||||||
|
|
||||||
|
def _grant_response(
|
||||||
|
grant: FileFormEvidenceGrant,
|
||||||
|
*,
|
||||||
|
upload_token: str | None,
|
||||||
|
replayed: bool,
|
||||||
|
) -> FormEvidenceGrant:
|
||||||
|
return FormEvidenceGrant(
|
||||||
|
provider_id=PROVIDER_ID,
|
||||||
|
grant_id=grant.id,
|
||||||
|
upload_token=upload_token,
|
||||||
|
upload_url="/api/v1/files/form-evidence/upload",
|
||||||
|
expires_at=_aware(grant.expires_at),
|
||||||
|
max_size_bytes=grant.max_size_bytes,
|
||||||
|
allowed_content_types=tuple(grant.allowed_content_types),
|
||||||
|
replayed=replayed,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _inspection(
|
||||||
|
reference: EvidenceReference,
|
||||||
|
*,
|
||||||
|
state: FormEvidenceState,
|
||||||
|
observed_at: datetime,
|
||||||
|
retryable: bool = False,
|
||||||
|
reason: str | None = None,
|
||||||
|
metadata: Mapping[str, object] | None = None,
|
||||||
|
) -> FormEvidenceInspection:
|
||||||
|
return FormEvidenceInspection(
|
||||||
|
provider_id=PROVIDER_ID,
|
||||||
|
reference=reference,
|
||||||
|
state=state,
|
||||||
|
observed_at=observed_at,
|
||||||
|
retryable=retryable,
|
||||||
|
reason=reason,
|
||||||
|
metadata=dict(metadata or {}),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _session(value: object) -> Session:
|
||||||
|
if not isinstance(value, Session):
|
||||||
|
raise TypeError("Files Form evidence requires a SQLAlchemy session.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_tenant(principal: object, tenant_id: str) -> None:
|
||||||
|
principal_tenant = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||||
|
if not principal_tenant or principal_tenant != tenant_id:
|
||||||
|
raise PermissionError("Form evidence cannot cross tenants.")
|
||||||
|
|
||||||
|
|
||||||
|
def _custodian_user_id(value: str | None) -> str:
|
||||||
|
clean = str(value or "").strip()
|
||||||
|
if not clean.startswith("user:") or len(clean) <= len("user:"):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"Files Form evidence requires a user custodian."
|
||||||
|
)
|
||||||
|
return clean.removeprefix("user:")
|
||||||
|
|
||||||
|
|
||||||
|
def _content_types(values: Sequence[str]) -> tuple[str, ...]:
|
||||||
|
cleaned = tuple(
|
||||||
|
dict.fromkeys(
|
||||||
|
value.split(";", 1)[0].strip().casefold()
|
||||||
|
for value in values
|
||||||
|
if value.strip()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if len(cleaned) > 50 or any(len(value) > 255 for value in cleaned):
|
||||||
|
raise FormEvidenceContractError(
|
||||||
|
"Form evidence content-type restrictions are too large."
|
||||||
|
)
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_metadata(value: Mapping[str, object]) -> dict[str, object]:
|
||||||
|
remaining = value.get("remaining_attachments")
|
||||||
|
result: dict[str, object] = {}
|
||||||
|
if isinstance(remaining, int) and remaining >= 0:
|
||||||
|
result["remaining_attachments"] = remaining
|
||||||
|
raw_ids = value.get("existing_attachment_ids")
|
||||||
|
if isinstance(raw_ids, (list, tuple)):
|
||||||
|
clean_ids = tuple(
|
||||||
|
dict.fromkeys(
|
||||||
|
item.strip()
|
||||||
|
for item in raw_ids
|
||||||
|
if isinstance(item, str) and item.strip() and len(item.strip()) <= 255
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if len(clean_ids) <= 1000:
|
||||||
|
result["existing_attachment_ids"] = clean_ids
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _request_sha256(
|
||||||
|
request: FormEvidenceGrantRequest,
|
||||||
|
*,
|
||||||
|
custodian_user_id: str,
|
||||||
|
max_size_bytes: int,
|
||||||
|
allowed_content_types: Sequence[str],
|
||||||
|
) -> str:
|
||||||
|
return hashlib.sha256(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"tenant_id": request.tenant_id,
|
||||||
|
"instance_id": request.instance_id,
|
||||||
|
"definition_ref": request.definition_ref.to_dict(),
|
||||||
|
"evidence_kind": request.evidence_kind,
|
||||||
|
"purpose": request.purpose,
|
||||||
|
"expires_at": request.expires_at.isoformat(),
|
||||||
|
"custodian_user_id": custodian_user_id,
|
||||||
|
"max_size_bytes": max_size_bytes,
|
||||||
|
"allowed_content_types": list(allowed_content_types),
|
||||||
|
"metadata": _bounded_metadata(request.metadata),
|
||||||
|
},
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
).encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _token_sha256(token: str) -> str:
|
||||||
|
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _aware(value: datetime) -> datetime:
|
||||||
|
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"CAPABILITY_FORM_EVIDENCE_FILES",
|
||||||
|
"FilesFormEvidenceProvider",
|
||||||
|
"PROVIDER_ID",
|
||||||
|
"create_files_form_evidence_provider",
|
||||||
|
]
|
||||||
File diff suppressed because it is too large
Load Diff
+119
@@ -0,0 +1,119 @@
|
|||||||
|
"""file connector spaces
|
||||||
|
|
||||||
|
Revision ID: 4f5a6b7c8d9e
|
||||||
|
Revises: 2e3f4a5b6c7d
|
||||||
|
Create Date: 2026-07-08 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "4f5a6b7c8d9e"
|
||||||
|
down_revision = "2e3f4a5b6c7d"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_fk_target(tables: set[str]) -> str:
|
||||||
|
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
tables = set(inspector.get_table_names())
|
||||||
|
scope_fk_target = _scope_fk_target(tables)
|
||||||
|
if "file_connector_spaces" not in tables:
|
||||||
|
op.create_table(
|
||||||
|
"file_connector_spaces",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("owner_type", sa.String(length=20), nullable=False),
|
||||||
|
sa.Column("owner_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("owner_group_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("label", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("connector_profile_id", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("provider", sa.String(length=50), nullable=False),
|
||||||
|
sa.Column("library_id", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("remote_path", sa.String(length=1000), nullable=False),
|
||||||
|
sa.Column("sync_mode", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("read_only", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("is_active", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_spaces_created_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.ForeignKeyConstraint(["owner_group_id"], ["access_groups.id"], name=op.f("fk_file_connector_spaces_owner_group_id_groups"), ondelete="SET NULL"),
|
||||||
|
sa.ForeignKeyConstraint(["owner_user_id"], ["access_users.id"], name=op.f("fk_file_connector_spaces_owner_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_spaces_tenant_id_scopes"), ondelete="CASCADE"),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_spaces")),
|
||||||
|
)
|
||||||
|
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_spaces")}
|
||||||
|
for column in (
|
||||||
|
"tenant_id",
|
||||||
|
"owner_type",
|
||||||
|
"owner_user_id",
|
||||||
|
"owner_group_id",
|
||||||
|
"connector_profile_id",
|
||||||
|
"provider",
|
||||||
|
"is_active",
|
||||||
|
"created_by_user_id",
|
||||||
|
"deleted_at",
|
||||||
|
):
|
||||||
|
name = op.f(f"ix_file_connector_spaces_{column}")
|
||||||
|
if name not in indexes:
|
||||||
|
op.create_index(name, "file_connector_spaces", [column], unique=False)
|
||||||
|
if "ix_file_connector_spaces_owner" not in indexes:
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_connector_spaces_owner",
|
||||||
|
"file_connector_spaces",
|
||||||
|
["tenant_id", "owner_type", "owner_user_id", "owner_group_id"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
if "uq_file_connector_spaces_active_user_label" not in indexes:
|
||||||
|
op.create_index(
|
||||||
|
"uq_file_connector_spaces_active_user_label",
|
||||||
|
"file_connector_spaces",
|
||||||
|
["tenant_id", "owner_user_id", "label"],
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||||
|
postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||||
|
)
|
||||||
|
if "uq_file_connector_spaces_active_group_label" not in indexes:
|
||||||
|
op.create_index(
|
||||||
|
"uq_file_connector_spaces_active_group_label",
|
||||||
|
"file_connector_spaces",
|
||||||
|
["tenant_id", "owner_group_id", "label"],
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||||
|
postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
if "file_connector_spaces" not in inspector.get_table_names():
|
||||||
|
return
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_spaces")}
|
||||||
|
for name in (
|
||||||
|
"uq_file_connector_spaces_active_group_label",
|
||||||
|
"uq_file_connector_spaces_active_user_label",
|
||||||
|
"ix_file_connector_spaces_owner",
|
||||||
|
op.f("ix_file_connector_spaces_deleted_at"),
|
||||||
|
op.f("ix_file_connector_spaces_created_by_user_id"),
|
||||||
|
op.f("ix_file_connector_spaces_is_active"),
|
||||||
|
op.f("ix_file_connector_spaces_provider"),
|
||||||
|
op.f("ix_file_connector_spaces_connector_profile_id"),
|
||||||
|
op.f("ix_file_connector_spaces_owner_group_id"),
|
||||||
|
op.f("ix_file_connector_spaces_owner_user_id"),
|
||||||
|
op.f("ix_file_connector_spaces_owner_type"),
|
||||||
|
op.f("ix_file_connector_spaces_tenant_id"),
|
||||||
|
):
|
||||||
|
if name in indexes:
|
||||||
|
op.drop_index(name, table_name="file_connector_spaces")
|
||||||
|
op.drop_table("file_connector_spaces")
|
||||||
+94
@@ -0,0 +1,94 @@
|
|||||||
|
"""file connector profiles
|
||||||
|
|
||||||
|
Revision ID: 5a6b7c8d9e0f
|
||||||
|
Revises: 4f5a6b7c8d9e
|
||||||
|
Create Date: 2026-07-08 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "5a6b7c8d9e0f"
|
||||||
|
down_revision = "4f5a6b7c8d9e"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_fk_target(inspector) -> str:
|
||||||
|
tables = set(inspector.get_table_names())
|
||||||
|
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
scope_fk_target = _scope_fk_target(inspector)
|
||||||
|
if "file_connector_profiles" not in inspector.get_table_names():
|
||||||
|
op.create_table(
|
||||||
|
"file_connector_profiles",
|
||||||
|
sa.Column("id", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||||
|
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("label", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("provider", sa.String(length=50), nullable=False),
|
||||||
|
sa.Column("endpoint_url", sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column("base_path", sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("credential_mode", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("username", sa.String(length=320), nullable=True),
|
||||||
|
sa.Column("password_encrypted", sa.Text(), nullable=True),
|
||||||
|
sa.Column("token_encrypted", sa.Text(), nullable=True),
|
||||||
|
sa.Column("password_env", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("token_env", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("secret_ref", sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column("capabilities", sa.JSON(), nullable=True),
|
||||||
|
sa.Column("policy", sa.JSON(), nullable=True),
|
||||||
|
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||||
|
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_profiles_created_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_profiles_tenant_id_scopes"), ondelete="CASCADE"),
|
||||||
|
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_profiles_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_profiles")),
|
||||||
|
)
|
||||||
|
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||||
|
for column in (
|
||||||
|
"tenant_id",
|
||||||
|
"scope_type",
|
||||||
|
"scope_id",
|
||||||
|
"provider",
|
||||||
|
"enabled",
|
||||||
|
"created_by_user_id",
|
||||||
|
"updated_by_user_id",
|
||||||
|
):
|
||||||
|
name = op.f(f"ix_file_connector_profiles_{column}")
|
||||||
|
if name not in indexes:
|
||||||
|
op.create_index(name, "file_connector_profiles", [column], unique=False)
|
||||||
|
if "ix_file_connector_profiles_scope" not in indexes:
|
||||||
|
op.create_index("ix_file_connector_profiles_scope", "file_connector_profiles", ["scope_type", "scope_id"], unique=False)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
if "file_connector_profiles" not in inspector.get_table_names():
|
||||||
|
return
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||||
|
for name in (
|
||||||
|
"ix_file_connector_profiles_scope",
|
||||||
|
op.f("ix_file_connector_profiles_updated_by_user_id"),
|
||||||
|
op.f("ix_file_connector_profiles_created_by_user_id"),
|
||||||
|
op.f("ix_file_connector_profiles_enabled"),
|
||||||
|
op.f("ix_file_connector_profiles_provider"),
|
||||||
|
op.f("ix_file_connector_profiles_scope_id"),
|
||||||
|
op.f("ix_file_connector_profiles_scope_type"),
|
||||||
|
op.f("ix_file_connector_profiles_tenant_id"),
|
||||||
|
):
|
||||||
|
if name in indexes:
|
||||||
|
op.drop_index(name, table_name="file_connector_profiles")
|
||||||
|
op.drop_table("file_connector_profiles")
|
||||||
+111
@@ -0,0 +1,111 @@
|
|||||||
|
"""file connector credentials
|
||||||
|
|
||||||
|
Revision ID: 6b7c8d9e0f1a
|
||||||
|
Revises: 5a6b7c8d9e0f
|
||||||
|
Create Date: 2026-07-08 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "6b7c8d9e0f1a"
|
||||||
|
down_revision = "5a6b7c8d9e0f"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_fk_target(table_names: set[str]) -> str:
|
||||||
|
return "core_scopes.id" if "core_scopes" in table_names else "tenancy_tenants.id"
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
table_names = set(inspector.get_table_names())
|
||||||
|
scope_fk_target = _scope_fk_target(table_names)
|
||||||
|
|
||||||
|
if "file_connector_profiles" in table_names:
|
||||||
|
columns = {item["name"] for item in inspector.get_columns("file_connector_profiles")}
|
||||||
|
if "credential_profile_id" not in columns:
|
||||||
|
op.add_column("file_connector_profiles", sa.Column("credential_profile_id", sa.String(length=255), nullable=True))
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||||
|
index_name = op.f("ix_file_connector_profiles_credential_profile_id")
|
||||||
|
if index_name not in indexes:
|
||||||
|
op.create_index(index_name, "file_connector_profiles", ["credential_profile_id"], unique=False)
|
||||||
|
|
||||||
|
if "file_connector_credentials" not in table_names:
|
||||||
|
op.create_table(
|
||||||
|
"file_connector_credentials",
|
||||||
|
sa.Column("id", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||||
|
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("label", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("provider", sa.String(length=50), nullable=True),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("credential_mode", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("username", sa.String(length=320), nullable=True),
|
||||||
|
sa.Column("password_encrypted", sa.Text(), nullable=True),
|
||||||
|
sa.Column("token_encrypted", sa.Text(), nullable=True),
|
||||||
|
sa.Column("password_env", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("token_env", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("secret_ref", sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column("policy", sa.JSON(), nullable=True),
|
||||||
|
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||||
|
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_credentials_created_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_credentials_tenant_id_scopes"), ondelete="CASCADE"),
|
||||||
|
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_credentials_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_credentials")),
|
||||||
|
)
|
||||||
|
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_credentials")}
|
||||||
|
for column in (
|
||||||
|
"tenant_id",
|
||||||
|
"scope_type",
|
||||||
|
"scope_id",
|
||||||
|
"provider",
|
||||||
|
"enabled",
|
||||||
|
"created_by_user_id",
|
||||||
|
"updated_by_user_id",
|
||||||
|
):
|
||||||
|
name = op.f(f"ix_file_connector_credentials_{column}")
|
||||||
|
if name not in indexes:
|
||||||
|
op.create_index(name, "file_connector_credentials", [column], unique=False)
|
||||||
|
if "ix_file_connector_credentials_scope" not in indexes:
|
||||||
|
op.create_index("ix_file_connector_credentials_scope", "file_connector_credentials", ["scope_type", "scope_id"], unique=False)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
if "file_connector_credentials" in inspector.get_table_names():
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_credentials")}
|
||||||
|
for name in (
|
||||||
|
"ix_file_connector_credentials_scope",
|
||||||
|
op.f("ix_file_connector_credentials_updated_by_user_id"),
|
||||||
|
op.f("ix_file_connector_credentials_created_by_user_id"),
|
||||||
|
op.f("ix_file_connector_credentials_enabled"),
|
||||||
|
op.f("ix_file_connector_credentials_provider"),
|
||||||
|
op.f("ix_file_connector_credentials_scope_id"),
|
||||||
|
op.f("ix_file_connector_credentials_scope_type"),
|
||||||
|
op.f("ix_file_connector_credentials_tenant_id"),
|
||||||
|
):
|
||||||
|
if name in indexes:
|
||||||
|
op.drop_index(name, table_name="file_connector_credentials")
|
||||||
|
op.drop_table("file_connector_credentials")
|
||||||
|
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
if "file_connector_profiles" not in inspector.get_table_names():
|
||||||
|
return
|
||||||
|
profile_indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||||
|
profile_index_name = op.f("ix_file_connector_profiles_credential_profile_id")
|
||||||
|
if profile_index_name in profile_indexes:
|
||||||
|
op.drop_index(profile_index_name, table_name="file_connector_profiles")
|
||||||
|
profile_columns = {item["name"] for item in inspector.get_columns("file_connector_profiles")}
|
||||||
|
if "credential_profile_id" in profile_columns:
|
||||||
|
op.drop_column("file_connector_profiles", "credential_profile_id")
|
||||||
+77
@@ -0,0 +1,77 @@
|
|||||||
|
"""file connector policies
|
||||||
|
|
||||||
|
Revision ID: a7b8c9d0e1f3
|
||||||
|
Revises: 6b7c8d9e0f1a
|
||||||
|
Create Date: 2026-07-08 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "a7b8c9d0e1f3"
|
||||||
|
down_revision = "6b7c8d9e0f1a"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_fk_target(inspector) -> str:
|
||||||
|
tables = set(inspector.get_table_names())
|
||||||
|
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
scope_fk_target = _scope_fk_target(inspector)
|
||||||
|
if "file_connector_policies" not in inspector.get_table_names():
|
||||||
|
op.create_table(
|
||||||
|
"file_connector_policies",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||||
|
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("policy", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_policies_created_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_policies_tenant_id_scopes"), ondelete="CASCADE"),
|
||||||
|
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_policies_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_policies")),
|
||||||
|
sa.UniqueConstraint("tenant_id", "scope_type", "scope_id", name="uq_file_connector_policies_scope"),
|
||||||
|
)
|
||||||
|
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_policies")}
|
||||||
|
for column in (
|
||||||
|
"tenant_id",
|
||||||
|
"scope_type",
|
||||||
|
"scope_id",
|
||||||
|
"created_by_user_id",
|
||||||
|
"updated_by_user_id",
|
||||||
|
):
|
||||||
|
name = op.f(f"ix_file_connector_policies_{column}")
|
||||||
|
if name not in indexes:
|
||||||
|
op.create_index(name, "file_connector_policies", [column], unique=False)
|
||||||
|
if "ix_file_connector_policies_scope" not in indexes:
|
||||||
|
op.create_index("ix_file_connector_policies_scope", "file_connector_policies", ["scope_type", "scope_id"], unique=False)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
inspector = sa.inspect(op.get_bind())
|
||||||
|
if "file_connector_policies" not in inspector.get_table_names():
|
||||||
|
return
|
||||||
|
indexes = {item["name"] for item in inspector.get_indexes("file_connector_policies")}
|
||||||
|
for name in (
|
||||||
|
"ix_file_connector_policies_scope",
|
||||||
|
op.f("ix_file_connector_policies_updated_by_user_id"),
|
||||||
|
op.f("ix_file_connector_policies_created_by_user_id"),
|
||||||
|
op.f("ix_file_connector_policies_scope_id"),
|
||||||
|
op.f("ix_file_connector_policies_scope_type"),
|
||||||
|
op.f("ix_file_connector_policies_tenant_id"),
|
||||||
|
):
|
||||||
|
if name in indexes:
|
||||||
|
op.drop_index(name, table_name="file_connector_policies")
|
||||||
|
op.drop_table("file_connector_policies")
|
||||||
+147
@@ -0,0 +1,147 @@
|
|||||||
|
"""add purpose-bound Form evidence upload grants
|
||||||
|
|
||||||
|
Revision ID: a2b3c4d5e6f8
|
||||||
|
Revises: f1a2b3c4d5e7
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
|
||||||
|
revision = "a2b3c4d5e6f8"
|
||||||
|
down_revision = "f1a2b3c4d5e7"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("form_instance_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("form_definition_id", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("form_definition_revision", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("token_sha256", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("request_sha256", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("custodian_user_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("evidence_kind", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("purpose", sa.String(length=500), nullable=False),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("max_size_bytes", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("allowed_content_types", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("file_asset_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("file_version_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("uploaded_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("metadata", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["custodian_user_id"],
|
||||||
|
["access_users.id"],
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["file_asset_id"],
|
||||||
|
["file_assets.id"],
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["file_version_id"],
|
||||||
|
["file_versions.id"],
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id"),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"idempotency_key",
|
||||||
|
name="uq_file_form_evidence_grants_idempotency",
|
||||||
|
),
|
||||||
|
sa.UniqueConstraint("token_sha256"),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_tenant_id",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["tenant_id"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_form_instance_id",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["form_instance_id"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_custodian_user_id",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["custodian_user_id"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_status",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["status"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_expires_at",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["expires_at"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_file_asset_id",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["file_asset_id"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_file_version_id",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
["file_version_id"],
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_form_evidence_grants_form",
|
||||||
|
"file_form_evidence_grants",
|
||||||
|
[
|
||||||
|
"tenant_id",
|
||||||
|
"form_instance_id",
|
||||||
|
"form_definition_id",
|
||||||
|
"form_definition_revision",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_form",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_file_version_id",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_file_asset_id",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_expires_at",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_status",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_custodian_user_id",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_form_instance_id",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_index(
|
||||||
|
"ix_file_form_evidence_grants_tenant_id",
|
||||||
|
table_name="file_form_evidence_grants",
|
||||||
|
)
|
||||||
|
op.drop_table("file_form_evidence_grants")
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
"""govern managed file retention and legal holds
|
||||||
|
|
||||||
|
Revision ID: a2b3c4d5e6f9
|
||||||
|
Revises: f1a2b3c4d5e7
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
|
||||||
|
revision = "a2b3c4d5e6f9"
|
||||||
|
down_revision = "f1a2b3c4d5e7"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_assets") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("retained_until", sa.DateTime(timezone=True), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"legal_hold", sa.Boolean(), nullable=False, server_default=sa.false()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"lifecycle_revision", sa.Integer(), nullable=False, server_default="1"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("lifecycle_reason", sa.String(length=500), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_assets_retained_until"), ["retained_until"], unique=False
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_assets_legal_hold"), ["legal_hold"], unique=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_assets") as batch_op:
|
||||||
|
batch_op.drop_index(op.f("ix_file_assets_legal_hold"))
|
||||||
|
batch_op.drop_index(op.f("ix_file_assets_retained_until"))
|
||||||
|
batch_op.drop_column("lifecycle_reason")
|
||||||
|
batch_op.drop_column("lifecycle_revision")
|
||||||
|
batch_op.drop_column("legal_hold")
|
||||||
|
batch_op.drop_column("retained_until")
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
"""v0.1.7 files baseline
|
||||||
|
|
||||||
|
Revision ID: a7b8c9d0e1f3
|
||||||
|
Revises: None
|
||||||
|
Create Date: 2026-07-11 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = 'a7b8c9d0e1f3'
|
||||||
|
down_revision = None
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = '4f2a9c8e7b6d'
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table('file_connector_credentials',
|
||||||
|
sa.Column('id', sa.String(length=255), nullable=False),
|
||||||
|
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||||
|
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('label', sa.String(length=255), nullable=False),
|
||||||
|
sa.Column('provider', sa.String(length=50), nullable=True),
|
||||||
|
sa.Column('enabled', sa.Boolean(), nullable=False),
|
||||||
|
sa.Column('credential_mode', sa.String(length=30), nullable=False),
|
||||||
|
sa.Column('username', sa.String(length=320), nullable=True),
|
||||||
|
sa.Column('password_encrypted', sa.Text(), nullable=True),
|
||||||
|
sa.Column('token_encrypted', sa.Text(), nullable=True),
|
||||||
|
sa.Column('password_env', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('token_env', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column('policy', sa.JSON(), nullable=True),
|
||||||
|
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||||
|
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_credentials_tenant_id_scopes'), ondelete='CASCADE'),
|
||||||
|
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_credentials'))
|
||||||
|
)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_created_by_user_id'), 'file_connector_credentials', ['created_by_user_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_enabled'), 'file_connector_credentials', ['enabled'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_provider'), 'file_connector_credentials', ['provider'], unique=False)
|
||||||
|
op.create_index('ix_file_connector_credentials_scope', 'file_connector_credentials', ['scope_type', 'scope_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_scope_id'), 'file_connector_credentials', ['scope_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_scope_type'), 'file_connector_credentials', ['scope_type'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_tenant_id'), 'file_connector_credentials', ['tenant_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_credentials_updated_by_user_id'), 'file_connector_credentials', ['updated_by_user_id'], unique=False)
|
||||||
|
op.create_table('file_connector_policies',
|
||||||
|
sa.Column('id', sa.String(length=36), nullable=False),
|
||||||
|
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||||
|
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('policy', sa.JSON(), nullable=False),
|
||||||
|
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_policies_tenant_id_scopes'), ondelete='CASCADE'),
|
||||||
|
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_policies')),
|
||||||
|
sa.UniqueConstraint('tenant_id', 'scope_type', 'scope_id', name='uq_file_connector_policies_scope')
|
||||||
|
)
|
||||||
|
op.create_index(op.f('ix_file_connector_policies_created_by_user_id'), 'file_connector_policies', ['created_by_user_id'], unique=False)
|
||||||
|
op.create_index('ix_file_connector_policies_scope', 'file_connector_policies', ['scope_type', 'scope_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_policies_scope_id'), 'file_connector_policies', ['scope_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_policies_scope_type'), 'file_connector_policies', ['scope_type'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_policies_tenant_id'), 'file_connector_policies', ['tenant_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_policies_updated_by_user_id'), 'file_connector_policies', ['updated_by_user_id'], unique=False)
|
||||||
|
op.create_table('file_connector_profiles',
|
||||||
|
sa.Column('id', sa.String(length=255), nullable=False),
|
||||||
|
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||||
|
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('label', sa.String(length=255), nullable=False),
|
||||||
|
sa.Column('provider', sa.String(length=50), nullable=False),
|
||||||
|
sa.Column('endpoint_url', sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column('base_path', sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column('enabled', sa.Boolean(), nullable=False),
|
||||||
|
sa.Column('credential_profile_id', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('credential_mode', sa.String(length=30), nullable=False),
|
||||||
|
sa.Column('username', sa.String(length=320), nullable=True),
|
||||||
|
sa.Column('password_encrypted', sa.Text(), nullable=True),
|
||||||
|
sa.Column('token_encrypted', sa.Text(), nullable=True),
|
||||||
|
sa.Column('password_env', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('token_env', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column('capabilities', sa.JSON(), nullable=True),
|
||||||
|
sa.Column('policy', sa.JSON(), nullable=True),
|
||||||
|
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||||
|
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_profiles_tenant_id_scopes'), ondelete='CASCADE'),
|
||||||
|
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_profiles'))
|
||||||
|
)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_created_by_user_id'), 'file_connector_profiles', ['created_by_user_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_credential_profile_id'), 'file_connector_profiles', ['credential_profile_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_enabled'), 'file_connector_profiles', ['enabled'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_provider'), 'file_connector_profiles', ['provider'], unique=False)
|
||||||
|
op.create_index('ix_file_connector_profiles_scope', 'file_connector_profiles', ['scope_type', 'scope_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_scope_id'), 'file_connector_profiles', ['scope_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_scope_type'), 'file_connector_profiles', ['scope_type'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_tenant_id'), 'file_connector_profiles', ['tenant_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_profiles_updated_by_user_id'), 'file_connector_profiles', ['updated_by_user_id'], unique=False)
|
||||||
|
op.create_table('file_connector_spaces',
|
||||||
|
sa.Column('id', sa.String(length=36), nullable=False),
|
||||||
|
sa.Column('tenant_id', sa.String(length=36), nullable=False),
|
||||||
|
sa.Column('owner_type', sa.String(length=20), nullable=False),
|
||||||
|
sa.Column('owner_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('owner_group_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('label', sa.String(length=255), nullable=False),
|
||||||
|
sa.Column('connector_profile_id', sa.String(length=255), nullable=False),
|
||||||
|
sa.Column('provider', sa.String(length=50), nullable=False),
|
||||||
|
sa.Column('library_id', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('remote_path', sa.String(length=1000), nullable=False),
|
||||||
|
sa.Column('sync_mode', sa.String(length=30), nullable=False),
|
||||||
|
sa.Column('read_only', sa.Boolean(), nullable=False),
|
||||||
|
sa.Column('is_active', sa.Boolean(), nullable=False),
|
||||||
|
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||||
|
sa.Column('deleted_at', sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||||
|
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.ForeignKeyConstraint(['owner_group_id'], ['access_groups.id'], name=op.f('fk_file_connector_spaces_owner_group_id_access_groups'), ondelete='SET NULL'),
|
||||||
|
sa.ForeignKeyConstraint(['owner_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_owner_user_id_access_users'), ondelete='SET NULL'),
|
||||||
|
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_spaces_tenant_id_scopes'), ondelete='CASCADE'),
|
||||||
|
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_spaces'))
|
||||||
|
)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_connector_profile_id'), 'file_connector_spaces', ['connector_profile_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_created_by_user_id'), 'file_connector_spaces', ['created_by_user_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_deleted_at'), 'file_connector_spaces', ['deleted_at'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_is_active'), 'file_connector_spaces', ['is_active'], unique=False)
|
||||||
|
op.create_index('ix_file_connector_spaces_owner', 'file_connector_spaces', ['tenant_id', 'owner_type', 'owner_user_id', 'owner_group_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_owner_group_id'), 'file_connector_spaces', ['owner_group_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_owner_type'), 'file_connector_spaces', ['owner_type'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_owner_user_id'), 'file_connector_spaces', ['owner_user_id'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_provider'), 'file_connector_spaces', ['provider'], unique=False)
|
||||||
|
op.create_index(op.f('ix_file_connector_spaces_tenant_id'), 'file_connector_spaces', ['tenant_id'], unique=False)
|
||||||
|
op.create_index('uq_file_connector_spaces_active_group_label', 'file_connector_spaces', ['tenant_id', 'owner_group_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"))
|
||||||
|
op.create_index('uq_file_connector_spaces_active_user_label', 'file_connector_spaces', ['tenant_id', 'owner_user_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"))
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table('file_connector_spaces')
|
||||||
|
op.drop_table('file_connector_profiles')
|
||||||
|
op.drop_table('file_connector_policies')
|
||||||
|
op.drop_table('file_connector_credentials')
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
"""file share lifecycle
|
||||||
|
|
||||||
|
Revision ID: b8c9d0e1f2a4
|
||||||
|
Revises: a7b8c9d0e1f3
|
||||||
|
Create Date: 2026-07-30 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "b8c9d0e1f2a4"
|
||||||
|
down_revision = "a7b8c9d0e1f3"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_shares") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("revoked_by_user_id", sa.String(length=36), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.create_foreign_key(
|
||||||
|
op.f("fk_file_shares_revoked_by_user_id_access_users"),
|
||||||
|
"access_users",
|
||||||
|
["revoked_by_user_id"],
|
||||||
|
["id"],
|
||||||
|
ondelete="SET NULL",
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_shares_expires_at"), ["expires_at"], unique=False
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_shares_revoked_by_user_id"),
|
||||||
|
["revoked_by_user_id"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_shares") as batch_op:
|
||||||
|
batch_op.drop_index(op.f("ix_file_shares_revoked_by_user_id"))
|
||||||
|
batch_op.drop_index(op.f("ix_file_shares_expires_at"))
|
||||||
|
batch_op.drop_constraint(
|
||||||
|
op.f("fk_file_shares_revoked_by_user_id_access_users"),
|
||||||
|
type_="foreignkey",
|
||||||
|
)
|
||||||
|
batch_op.drop_column("revoked_by_user_id")
|
||||||
|
batch_op.drop_column("expires_at")
|
||||||
+168
@@ -0,0 +1,168 @@
|
|||||||
|
"""file integrity reconciliation
|
||||||
|
|
||||||
|
Revision ID: c9d0e1f2a3b5
|
||||||
|
Revises: b8c9d0e1f2a4
|
||||||
|
Create Date: 2026-07-30 00:00:00.000000
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "c9d0e1f2a3b5"
|
||||||
|
down_revision = "b8c9d0e1f2a4"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_blobs") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"integrity_status",
|
||||||
|
sa.String(length=30),
|
||||||
|
nullable=False,
|
||||||
|
server_default="unchecked",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("integrity_checked_at", sa.DateTime(timezone=True), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("integrity_failure", sa.String(length=100), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_blobs_integrity_status"),
|
||||||
|
["integrity_status"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_blobs_integrity_checked_at"),
|
||||||
|
["integrity_checked_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_blobs_quarantined_at"),
|
||||||
|
["quarantined_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"file_integrity_scans",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("storage_backend", sa.String(length=50), nullable=False),
|
||||||
|
sa.Column("storage_prefix", sa.String(length=1000), nullable=False),
|
||||||
|
sa.Column("status", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("phase", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("verify_checksums", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("batch_size", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("blob_cursor", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("object_cursor", sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column("scanned_blob_count", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("verified_blob_count", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("quarantined_blob_count", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("scanned_object_count", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("orphan_object_count", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("last_error", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["created_by_user_id"],
|
||||||
|
["access_users.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_file_integrity_scans_created_by_user_id_access_users"
|
||||||
|
),
|
||||||
|
ondelete="SET NULL",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_scans")),
|
||||||
|
)
|
||||||
|
for column in ("tenant_id", "status", "created_by_user_id"):
|
||||||
|
op.create_index(
|
||||||
|
op.f(f"ix_file_integrity_scans_{column}"),
|
||||||
|
"file_integrity_scans",
|
||||||
|
[column],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"file_integrity_findings",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("scan_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("kind", sa.String(length=40), nullable=False),
|
||||||
|
sa.Column("state", sa.String(length=30), nullable=False),
|
||||||
|
sa.Column("blob_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("storage_key", sa.String(length=1000), nullable=False),
|
||||||
|
sa.Column("expected_size_bytes", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("observed_size_bytes", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("expected_checksum_sha256", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("observed_checksum_sha256", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("resolved_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
sa.Column("resolved_by_user_id", sa.String(length=36), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["blob_id"],
|
||||||
|
["file_blobs.id"],
|
||||||
|
name=op.f("fk_file_integrity_findings_blob_id_file_blobs"),
|
||||||
|
ondelete="SET NULL",
|
||||||
|
),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["resolved_by_user_id"],
|
||||||
|
["access_users.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_file_integrity_findings_resolved_by_user_id_access_users"
|
||||||
|
),
|
||||||
|
ondelete="SET NULL",
|
||||||
|
),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["scan_id"],
|
||||||
|
["file_integrity_scans.id"],
|
||||||
|
name=op.f(
|
||||||
|
"fk_file_integrity_findings_scan_id_file_integrity_scans"
|
||||||
|
),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_findings")),
|
||||||
|
)
|
||||||
|
for column in (
|
||||||
|
"scan_id",
|
||||||
|
"tenant_id",
|
||||||
|
"kind",
|
||||||
|
"state",
|
||||||
|
"blob_id",
|
||||||
|
"resolved_by_user_id",
|
||||||
|
):
|
||||||
|
op.create_index(
|
||||||
|
op.f(f"ix_file_integrity_findings_{column}"),
|
||||||
|
"file_integrity_findings",
|
||||||
|
[column],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_file_integrity_findings_scan_state",
|
||||||
|
"file_integrity_findings",
|
||||||
|
["scan_id", "state"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table("file_integrity_findings")
|
||||||
|
op.drop_table("file_integrity_scans")
|
||||||
|
with op.batch_alter_table("file_blobs") as batch_op:
|
||||||
|
batch_op.drop_index(op.f("ix_file_blobs_quarantined_at"))
|
||||||
|
batch_op.drop_index(op.f("ix_file_blobs_integrity_checked_at"))
|
||||||
|
batch_op.drop_index(op.f("ix_file_blobs_integrity_status"))
|
||||||
|
batch_op.drop_column("quarantined_at")
|
||||||
|
batch_op.drop_column("integrity_failure")
|
||||||
|
batch_op.drop_column("integrity_checked_at")
|
||||||
|
batch_op.drop_column("integrity_status")
|
||||||
+79
@@ -0,0 +1,79 @@
|
|||||||
|
"""file content-protection metadata
|
||||||
|
|
||||||
|
Revision ID: d0e1f2a3b4c6
|
||||||
|
Revises: c9d0e1f2a3b5
|
||||||
|
Create Date: 2026-08-02 00:00:00.000000
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "d0e1f2a3b4c6"
|
||||||
|
down_revision = "c9d0e1f2a3b5"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_blobs") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"protection_discriminator",
|
||||||
|
sa.String(length=320),
|
||||||
|
nullable=False,
|
||||||
|
server_default="plaintext",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("encryption_envelope_id", sa.String(length=255), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("storage_checksum_sha256", sa.String(length=64), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("storage_size_bytes", sa.Integer(), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.drop_constraint(
|
||||||
|
"uq_file_blobs_tenant_checksum_size",
|
||||||
|
type_="unique",
|
||||||
|
)
|
||||||
|
batch_op.create_unique_constraint(
|
||||||
|
"uq_file_blobs_tenant_checksum_size_protection",
|
||||||
|
[
|
||||||
|
"tenant_id",
|
||||||
|
"checksum_sha256",
|
||||||
|
"size_bytes",
|
||||||
|
"protection_discriminator",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_blobs_protection_discriminator"),
|
||||||
|
["protection_discriminator"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
batch_op.create_index(
|
||||||
|
op.f("ix_file_blobs_encryption_envelope_id"),
|
||||||
|
["encryption_envelope_id"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_blobs") as batch_op:
|
||||||
|
batch_op.drop_index(op.f("ix_file_blobs_encryption_envelope_id"))
|
||||||
|
batch_op.drop_index(op.f("ix_file_blobs_protection_discriminator"))
|
||||||
|
batch_op.drop_constraint(
|
||||||
|
"uq_file_blobs_tenant_checksum_size_protection",
|
||||||
|
type_="unique",
|
||||||
|
)
|
||||||
|
batch_op.create_unique_constraint(
|
||||||
|
"uq_file_blobs_tenant_checksum_size",
|
||||||
|
["tenant_id", "checksum_sha256", "size_bytes"],
|
||||||
|
)
|
||||||
|
batch_op.drop_column("storage_size_bytes")
|
||||||
|
batch_op.drop_column("storage_checksum_sha256")
|
||||||
|
batch_op.drop_column("encryption_envelope_id")
|
||||||
|
batch_op.drop_column("protection_discriminator")
|
||||||
+34
@@ -0,0 +1,34 @@
|
|||||||
|
"""add stale-action revisions to Files integrity operations
|
||||||
|
|
||||||
|
Revision ID: f1a2b3c4d5e7
|
||||||
|
Revises: d0e1f2a3b4c6
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
|
||||||
|
revision = "f1a2b3c4d5e7"
|
||||||
|
down_revision = "d0e1f2a3b4c6"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_integrity_scans") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("revision", sa.Integer(), nullable=False, server_default="1")
|
||||||
|
)
|
||||||
|
with op.batch_alter_table("file_integrity_findings") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("revision", sa.Integer(), nullable=False, server_default="1")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("file_integrity_findings") as batch_op:
|
||||||
|
batch_op.drop_column("revision")
|
||||||
|
with op.batch_alter_table("file_integrity_scans") as batch_op:
|
||||||
|
batch_op.drop_column("revision")
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import secrets
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
from govoplan_core.core.operations import OperationalCheck
|
||||||
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
StorageBackendError,
|
||||||
|
get_storage_backend,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def managed_storage_roundtrip_check() -> OperationalCheck:
|
||||||
|
"""Exercise the configured managed store without retaining probe data."""
|
||||||
|
|
||||||
|
backend = get_storage_backend()
|
||||||
|
key = f".govoplan-health/probes/{uuid4().hex}.bin"
|
||||||
|
payload = secrets.token_bytes(64)
|
||||||
|
expected_digest = hashlib.sha256(payload).hexdigest()
|
||||||
|
delete_error: Exception | None = None
|
||||||
|
try:
|
||||||
|
backend.put_bytes(key, payload, content_type="application/octet-stream")
|
||||||
|
stored = backend.get_bytes(key)
|
||||||
|
info = backend.stat(key)
|
||||||
|
if info.size_bytes != len(payload):
|
||||||
|
raise StorageBackendError("Managed storage returned an unexpected object size")
|
||||||
|
if hashlib.sha256(stored).hexdigest() != expected_digest:
|
||||||
|
raise StorageBackendError("Managed storage returned different bytes than were written")
|
||||||
|
except Exception as exc: # noqa: BLE001 - operational boundary reports provider failures.
|
||||||
|
return OperationalCheck(
|
||||||
|
id="files.managed_storage_roundtrip",
|
||||||
|
label="Managed file storage",
|
||||||
|
state="error",
|
||||||
|
detail=(
|
||||||
|
"The configured managed file store failed a bounded write/read/stat/delete "
|
||||||
|
f"probe ({type(exc).__name__})."
|
||||||
|
),
|
||||||
|
readiness_critical=True,
|
||||||
|
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
backend.delete(key)
|
||||||
|
except Exception as exc: # noqa: BLE001 - reported below when the data probe passed.
|
||||||
|
delete_error = exc
|
||||||
|
|
||||||
|
if delete_error is not None:
|
||||||
|
return OperationalCheck(
|
||||||
|
id="files.managed_storage_roundtrip",
|
||||||
|
label="Managed file storage",
|
||||||
|
state="error",
|
||||||
|
detail=(
|
||||||
|
"Managed file bytes round-tripped, but probe cleanup failed "
|
||||||
|
f"({type(delete_error).__name__})."
|
||||||
|
),
|
||||||
|
readiness_critical=True,
|
||||||
|
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||||
|
)
|
||||||
|
return OperationalCheck(
|
||||||
|
id="files.managed_storage_roundtrip",
|
||||||
|
label="Managed file storage",
|
||||||
|
state="ok",
|
||||||
|
detail="The configured managed file store passed write, read, stat, integrity, and delete checks.",
|
||||||
|
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||||
|
)
|
||||||
|
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import defaultdict
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
from sqlalchemy import or_, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.provider_governance import (
|
||||||
|
ExternalProviderRuntimeState,
|
||||||
|
ExternalProviderStateContext,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorProfile, FileConnectorSpace
|
||||||
|
from govoplan_files.backend.storage.connector_providers import (
|
||||||
|
ConnectorProviderDescriptor,
|
||||||
|
connector_provider_descriptors,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
REMOTE_STORAGE_PROVIDER_ID = "files.remote_storage"
|
||||||
|
|
||||||
|
|
||||||
|
def remote_storage_provider_states(
|
||||||
|
context: ExternalProviderStateContext,
|
||||||
|
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||||
|
if not isinstance(context.session, Session):
|
||||||
|
raise RuntimeError("Files provider state requires a database session.")
|
||||||
|
statement = select(FileConnectorProfile)
|
||||||
|
if context.tenant_id is not None:
|
||||||
|
statement = statement.where(
|
||||||
|
or_(
|
||||||
|
FileConnectorProfile.tenant_id.is_(None),
|
||||||
|
FileConnectorProfile.tenant_id == context.tenant_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
profiles = tuple(
|
||||||
|
context.session.scalars(
|
||||||
|
statement.order_by(
|
||||||
|
FileConnectorProfile.tenant_id,
|
||||||
|
FileConnectorProfile.id,
|
||||||
|
).limit(context.max_items + 1)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if not profiles:
|
||||||
|
return ()
|
||||||
|
|
||||||
|
profile_ids = tuple(item.id for item in profiles)
|
||||||
|
space_statement = select(FileConnectorSpace).where(
|
||||||
|
FileConnectorSpace.connector_profile_id.in_(profile_ids),
|
||||||
|
FileConnectorSpace.deleted_at.is_(None),
|
||||||
|
)
|
||||||
|
if context.tenant_id is not None:
|
||||||
|
space_statement = space_statement.where(
|
||||||
|
FileConnectorSpace.tenant_id == context.tenant_id
|
||||||
|
)
|
||||||
|
spaces_by_profile: dict[str, list[FileConnectorSpace]] = defaultdict(list)
|
||||||
|
for space in context.session.scalars(space_statement):
|
||||||
|
spaces_by_profile[space.connector_profile_id].append(space)
|
||||||
|
|
||||||
|
descriptors = {
|
||||||
|
item.provider: item for item in connector_provider_descriptors()
|
||||||
|
}
|
||||||
|
observed_at = datetime.now(UTC)
|
||||||
|
return tuple(
|
||||||
|
_profile_state(
|
||||||
|
profile,
|
||||||
|
spaces=spaces_by_profile.get(profile.id, []),
|
||||||
|
descriptor=descriptors.get(profile.provider),
|
||||||
|
observed_at=observed_at,
|
||||||
|
)
|
||||||
|
for profile in profiles
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_state(
|
||||||
|
profile: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
spaces: list[FileConnectorSpace],
|
||||||
|
descriptor: ConnectorProviderDescriptor | None,
|
||||||
|
observed_at: datetime,
|
||||||
|
) -> ExternalProviderRuntimeState:
|
||||||
|
active_spaces = tuple(item for item in spaces if item.is_active)
|
||||||
|
active = bool(profile.enabled)
|
||||||
|
implementation_ready = bool(
|
||||||
|
descriptor is not None and descriptor.implemented and descriptor.installed
|
||||||
|
)
|
||||||
|
health = (
|
||||||
|
"inactive"
|
||||||
|
if not active
|
||||||
|
else "error"
|
||||||
|
if not implementation_ready
|
||||||
|
else "unknown"
|
||||||
|
)
|
||||||
|
recovery = (
|
||||||
|
"not_applicable"
|
||||||
|
if not active
|
||||||
|
else "unsupported"
|
||||||
|
if not implementation_ready
|
||||||
|
else "attention"
|
||||||
|
)
|
||||||
|
detail = (
|
||||||
|
"Remote-storage connector profile is disabled."
|
||||||
|
if not active
|
||||||
|
else "The configured provider is not available in this runtime."
|
||||||
|
if not implementation_ready
|
||||||
|
else "Software support is available; no live remote health observation is retained."
|
||||||
|
)
|
||||||
|
return ExternalProviderRuntimeState(
|
||||||
|
provider_id=REMOTE_STORAGE_PROVIDER_ID,
|
||||||
|
binding_ref=f"files:connector-profile:{profile.id}",
|
||||||
|
authority_mode="external_mirror",
|
||||||
|
observed_at=observed_at,
|
||||||
|
configured=True,
|
||||||
|
active=active,
|
||||||
|
health=health,
|
||||||
|
freshness="unknown" if active else "not_applicable",
|
||||||
|
conflict="not_applicable",
|
||||||
|
recovery=recovery,
|
||||||
|
detail=detail,
|
||||||
|
metrics={
|
||||||
|
"provider": profile.provider,
|
||||||
|
"configured_spaces": len(spaces),
|
||||||
|
"active_spaces": len(active_spaces),
|
||||||
|
"write_requested_spaces": sum(
|
||||||
|
1 for item in active_spaces if not item.read_only
|
||||||
|
),
|
||||||
|
"software_implemented": bool(descriptor and descriptor.implemented),
|
||||||
|
"optional_dependency_available": bool(descriptor and descriptor.installed),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["REMOTE_STORAGE_PROVIDER_ID", "remote_storage_provider_states"]
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.records import (
|
||||||
|
RecordContractError,
|
||||||
|
RecordSourceLocator,
|
||||||
|
RecordSourceReference,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileBlob, FileVersion
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.files import get_asset_for_user
|
||||||
|
|
||||||
|
|
||||||
|
CAPABILITY_RECORD_SOURCE_FILES = "records.source.files"
|
||||||
|
|
||||||
|
|
||||||
|
class FilesRecordSource:
|
||||||
|
provider_id = "files"
|
||||||
|
|
||||||
|
def resource_types(self) -> Sequence[str]:
|
||||||
|
return ("file_version",)
|
||||||
|
|
||||||
|
def resolve(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
locator: RecordSourceLocator,
|
||||||
|
purpose: str,
|
||||||
|
) -> RecordSourceReference:
|
||||||
|
if not isinstance(session, Session):
|
||||||
|
raise RecordContractError(
|
||||||
|
"Files record references require a database session."
|
||||||
|
)
|
||||||
|
tenant_id = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||||
|
if not tenant_id or locator.tenant_id != tenant_id:
|
||||||
|
raise RecordContractError("Files record references cannot cross tenants.")
|
||||||
|
if locator.source_module != "files" or locator.resource_type != "file_version":
|
||||||
|
raise RecordContractError("Unsupported Files record source type.")
|
||||||
|
if not str(purpose or "").strip():
|
||||||
|
raise RecordContractError("Files record references require a purpose.")
|
||||||
|
if not hasattr(principal, "has") or not (
|
||||||
|
principal.has("files:file:read") or principal.has("files:file:admin")
|
||||||
|
):
|
||||||
|
raise RecordContractError("Current Files read permission is required.")
|
||||||
|
user = getattr(principal, "user", None)
|
||||||
|
user_id = str(
|
||||||
|
getattr(user, "id", "") or getattr(principal, "membership_id", "") or ""
|
||||||
|
).strip()
|
||||||
|
if not user_id:
|
||||||
|
raise RecordContractError(
|
||||||
|
"Files record references require a tenant user principal."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
asset_id=locator.resource_id,
|
||||||
|
is_admin=principal.has("files:file:admin"),
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise RecordContractError(str(exc)) from exc
|
||||||
|
version_query = session.query(FileVersion).filter(
|
||||||
|
FileVersion.tenant_id == tenant_id,
|
||||||
|
FileVersion.file_asset_id == asset.id,
|
||||||
|
)
|
||||||
|
revision = locator.source_revision.strip()
|
||||||
|
version = version_query.filter(FileVersion.id == revision).one_or_none()
|
||||||
|
if version is None:
|
||||||
|
raise RecordContractError("The exact file version does not exist.")
|
||||||
|
blob = session.get(FileBlob, version.blob_id)
|
||||||
|
if blob is None or blob.tenant_id != tenant_id:
|
||||||
|
raise RecordContractError(
|
||||||
|
"The exact file version has no managed content object."
|
||||||
|
)
|
||||||
|
if blob.quarantined_at is not None or blob.integrity_status == "failed":
|
||||||
|
raise RecordContractError(
|
||||||
|
"The exact file version failed the current integrity gate."
|
||||||
|
)
|
||||||
|
return RecordSourceReference(
|
||||||
|
locator=locator,
|
||||||
|
label=version.filename_at_upload,
|
||||||
|
authority_mode="external_authoritative",
|
||||||
|
content_sha256=version.checksum_sha256,
|
||||||
|
content_type=version.content_type,
|
||||||
|
size_bytes=version.size_bytes,
|
||||||
|
recorded_at=version.created_at,
|
||||||
|
launch_url=(
|
||||||
|
f"/files?fileId={quote(asset.id, safe='')}&versionId={quote(version.id, safe='')}"
|
||||||
|
),
|
||||||
|
metadata={
|
||||||
|
"display_path": version.display_path_at_upload,
|
||||||
|
"version_number": version.version_number,
|
||||||
|
"integrity_status": blob.integrity_status,
|
||||||
|
"protection": blob.protection_discriminator,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def create_files_record_source(_context: object) -> FilesRecordSource:
|
||||||
|
return FilesRecordSource()
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"CAPABILITY_RECORD_SOURCE_FILES",
|
||||||
|
"FilesRecordSource",
|
||||||
|
"create_files_record_source",
|
||||||
|
]
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,677 +1,40 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
from fastapi import APIRouter
|
||||||
import os
|
|
||||||
import tempfile
|
|
||||||
from io import BytesIO
|
|
||||||
from typing import Literal
|
|
||||||
from urllib.parse import quote
|
|
||||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, UploadFile, status
|
|
||||||
from fastapi.responses import FileResponse, StreamingResponse
|
|
||||||
from starlette.background import BackgroundTask
|
|
||||||
from sqlalchemy import or_
|
|
||||||
from sqlalchemy.orm import Session
|
|
||||||
|
|
||||||
from govoplan_core.auth.dependencies import ApiPrincipal, has_scope, require_scope
|
from govoplan_files.backend.routes.assets import router as assets_router
|
||||||
from govoplan_core.core.optional import reraise_unless_missing_package
|
from govoplan_files.backend.routes.connector_io import router as connector_io_router
|
||||||
from govoplan_files.backend.schemas import (
|
from govoplan_files.backend.routes.connector_profiles import (
|
||||||
ArchiveRequest,
|
router as connector_profiles_router,
|
||||||
BulkDeleteRequest,
|
|
||||||
BulkDeleteResponse,
|
|
||||||
ConflictResolutionRequest,
|
|
||||||
FileAssetResponse,
|
|
||||||
FileFolderCreateRequest,
|
|
||||||
FileFolderDeleteRequest,
|
|
||||||
FileFolderDeleteResponse,
|
|
||||||
FileFolderResponse,
|
|
||||||
FileFoldersResponse,
|
|
||||||
FileListResponse,
|
|
||||||
FileShareRequest,
|
|
||||||
FileShareResponse,
|
|
||||||
FileSpaceResponse,
|
|
||||||
FileSpacesResponse,
|
|
||||||
FileUploadResponse,
|
|
||||||
PatternMatchResponse,
|
|
||||||
PatternResolveRequest,
|
|
||||||
PatternResolveResponse,
|
|
||||||
RenamePreviewItem,
|
|
||||||
RenameRequest,
|
|
||||||
RenameResponse,
|
|
||||||
TransferRequest,
|
|
||||||
TransferResponse,
|
|
||||||
_conflict_resolutions,
|
|
||||||
)
|
)
|
||||||
from govoplan_core.db.models import Group, UserGroupMembership
|
from govoplan_files.backend.routes.connector_settings import (
|
||||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
router as connector_settings_router,
|
||||||
from govoplan_core.db.session import get_session
|
|
||||||
from govoplan_files.backend.runtime import settings
|
|
||||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError, filename_from_path, normalize_logical_path
|
|
||||||
from govoplan_files.backend.storage.access import ensure_group_access, user_group_ids
|
|
||||||
from govoplan_files.backend.storage.archives import create_zip_file, extract_zip_upload
|
|
||||||
from govoplan_files.backend.storage.common import FileStorageError
|
|
||||||
from govoplan_files.backend.storage.files import (
|
|
||||||
asset_is_audit_relevant,
|
|
||||||
create_file_asset,
|
|
||||||
current_version_and_blob,
|
|
||||||
get_asset_for_user,
|
|
||||||
list_assets_for_user,
|
|
||||||
read_asset_bytes,
|
|
||||||
share_file,
|
|
||||||
soft_delete_assets,
|
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.folders import create_folder, list_folders_for_user, soft_delete_folder
|
from govoplan_files.backend.routes.folders import router as folders_router
|
||||||
from govoplan_files.backend.storage.search import resolve_patterns
|
from govoplan_files.backend.routes.form_evidence import router as form_evidence_router
|
||||||
from govoplan_files.backend.storage.transfers import rename_selection, transfer_selection
|
from govoplan_files.backend.routes.integrity import router as integrity_router
|
||||||
|
from govoplan_files.backend.routes.lifecycle import router as lifecycle_router
|
||||||
router = APIRouter(prefix="/files", tags=["files"])
|
from govoplan_files.backend.routes.listing import router as listing_router
|
||||||
|
from govoplan_files.backend.routes.shares import router as shares_router
|
||||||
|
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||||
|
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||||
|
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||||
|
|
||||||
|
|
||||||
def _campaign_models():
|
router = APIRouter()
|
||||||
try:
|
for workflow_router in (
|
||||||
from govoplan_campaign.backend.db.models import Campaign, CampaignShare
|
spaces_router,
|
||||||
except ModuleNotFoundError as exc:
|
folders_router,
|
||||||
reraise_unless_missing_package(exc, "govoplan_campaign")
|
form_evidence_router,
|
||||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Campaign module is not installed") from exc
|
integrity_router,
|
||||||
return Campaign, CampaignShare
|
lifecycle_router,
|
||||||
|
listing_router,
|
||||||
|
uploads_router,
|
||||||
def _is_admin(principal: ApiPrincipal) -> bool:
|
connector_settings_router,
|
||||||
return has_scope(principal, "files:file:admin")
|
connector_io_router,
|
||||||
|
connector_profiles_router,
|
||||||
|
assets_router,
|
||||||
async def _read_limited_upload(upload: UploadFile, *, max_bytes: int) -> bytes:
|
shares_router,
|
||||||
data = await upload.read(max_bytes + 1)
|
transfers_router,
|
||||||
if len(data) > max_bytes:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
|
|
||||||
detail=f"Upload exceeds limit of {max_bytes} bytes",
|
|
||||||
)
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def _cleanup_temp_file(path: str) -> None:
|
|
||||||
try:
|
|
||||||
os.unlink(path)
|
|
||||||
except FileNotFoundError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
def _attachment_disposition(filename: str) -> str:
|
|
||||||
safe = filename.replace("\\", "_").replace("/", "_").replace("\r", "_").replace("\n", "_").strip() or "download"
|
|
||||||
ascii_name = "".join(
|
|
||||||
char if 32 <= ord(char) < 127 and char not in {'"', "\\", ";"} else "_"
|
|
||||||
for char in safe
|
|
||||||
).strip() or "download"
|
|
||||||
encoded = quote(safe, safe="")
|
|
||||||
return f'attachment; filename="{ascii_name}"; filename*=UTF-8' + "''" + encoded
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_campaign_file_access(session: Session, principal: ApiPrincipal, campaign_id: str | None) -> None:
|
|
||||||
if not campaign_id:
|
|
||||||
return
|
|
||||||
if not has_scope(principal, "campaigns:campaign:read"):
|
|
||||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Missing scope: campaign:read")
|
|
||||||
Campaign, CampaignShare = _campaign_models()
|
|
||||||
campaign = session.get(Campaign, campaign_id)
|
|
||||||
if not campaign or campaign.tenant_id != principal.tenant_id:
|
|
||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Campaign not found")
|
|
||||||
if has_scope(principal, "tenant:*"):
|
|
||||||
return
|
|
||||||
if campaign.owner_user_id == principal.user.id:
|
|
||||||
return
|
|
||||||
group_ids = {
|
|
||||||
row[0]
|
|
||||||
for row in session.query(UserGroupMembership.group_id)
|
|
||||||
.filter(UserGroupMembership.tenant_id == principal.tenant_id, UserGroupMembership.user_id == principal.user.id)
|
|
||||||
.all()
|
|
||||||
}
|
|
||||||
if campaign.owner_group_id and campaign.owner_group_id in group_ids:
|
|
||||||
return
|
|
||||||
share = (
|
|
||||||
session.query(CampaignShare)
|
|
||||||
.filter(
|
|
||||||
CampaignShare.tenant_id == principal.tenant_id,
|
|
||||||
CampaignShare.campaign_id == campaign.id,
|
|
||||||
CampaignShare.revoked_at.is_(None),
|
|
||||||
or_(
|
|
||||||
(CampaignShare.target_type == "user") & (CampaignShare.target_id == principal.user.id),
|
|
||||||
(CampaignShare.target_type == "group") & (CampaignShare.target_id.in_(group_ids)),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
if share is None:
|
|
||||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No access to this campaign")
|
|
||||||
|
|
||||||
|
|
||||||
def _http_error(exc: Exception, *, not_found: bool = False) -> HTTPException:
|
|
||||||
code = status.HTTP_404_NOT_FOUND if not_found else status.HTTP_400_BAD_REQUEST
|
|
||||||
return HTTPException(status_code=code, detail=str(exc))
|
|
||||||
|
|
||||||
|
|
||||||
def _owner_id(asset: FileAsset) -> str:
|
|
||||||
return asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id # type: ignore[return-value]
|
|
||||||
|
|
||||||
|
|
||||||
def _asset_response(session: Session, asset: FileAsset, *, include_shares: bool = False) -> FileAssetResponse:
|
|
||||||
version, blob = current_version_and_blob(session, asset)
|
|
||||||
shares: list[FileShareResponse] = []
|
|
||||||
if include_shares:
|
|
||||||
rows = session.query(FileShare).filter(FileShare.file_asset_id == asset.id).order_by(FileShare.created_at.desc()).all()
|
|
||||||
shares = [
|
|
||||||
FileShareResponse(
|
|
||||||
id=row.id,
|
|
||||||
target_type=row.target_type,
|
|
||||||
target_id=row.target_id,
|
|
||||||
permission=row.permission,
|
|
||||||
created_at=row.created_at.isoformat(),
|
|
||||||
revoked_at=row.revoked_at.isoformat() if row.revoked_at else None,
|
|
||||||
)
|
|
||||||
for row in rows
|
|
||||||
]
|
|
||||||
return FileAssetResponse(
|
|
||||||
id=asset.id,
|
|
||||||
tenant_id=asset.tenant_id,
|
|
||||||
owner_type=asset.owner_type,
|
|
||||||
owner_id=_owner_id(asset),
|
|
||||||
display_path=asset.display_path,
|
|
||||||
filename=asset.filename,
|
|
||||||
description=asset.description,
|
|
||||||
size_bytes=blob.size_bytes,
|
|
||||||
content_type=blob.content_type,
|
|
||||||
checksum_sha256=blob.checksum_sha256,
|
|
||||||
version_id=version.id,
|
|
||||||
created_at=asset.created_at.isoformat(),
|
|
||||||
updated_at=asset.updated_at.isoformat(),
|
|
||||||
deleted_at=asset.deleted_at.isoformat() if asset.deleted_at else None,
|
|
||||||
audit_relevant=asset_is_audit_relevant(session, asset),
|
|
||||||
metadata=asset.metadata_ or {},
|
|
||||||
shares=shares,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _folder_owner_id(folder: FileFolder) -> str:
|
|
||||||
return folder.owner_user_id if folder.owner_type == "user" else folder.owner_group_id # type: ignore[return-value]
|
|
||||||
|
|
||||||
|
|
||||||
def _folder_response(folder: FileFolder) -> FileFolderResponse:
|
|
||||||
return FileFolderResponse(
|
|
||||||
id=folder.id,
|
|
||||||
tenant_id=folder.tenant_id,
|
|
||||||
owner_type=folder.owner_type,
|
|
||||||
owner_id=_folder_owner_id(folder),
|
|
||||||
path=folder.path,
|
|
||||||
created_at=folder.created_at.isoformat(),
|
|
||||||
updated_at=folder.updated_at.isoformat(),
|
|
||||||
deleted_at=folder.deleted_at.isoformat() if folder.deleted_at else None,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_list_owner_access(session: Session, principal: ApiPrincipal, owner_type: str | None, owner_id: str | None) -> None:
|
|
||||||
if not owner_type:
|
|
||||||
return
|
|
||||||
if owner_type == "user" and owner_id and owner_id != principal.user.id and not _is_admin(principal):
|
|
||||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No access to this user file space")
|
|
||||||
if owner_type == "group" and owner_id:
|
|
||||||
try:
|
|
||||||
ensure_group_access(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
group_id=owner_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
except FileStorageError as exc:
|
|
||||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=str(exc)) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/spaces", response_model=FileSpacesResponse)
|
|
||||||
def list_file_spaces(
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
|
||||||
):
|
):
|
||||||
spaces = [
|
router.routes.extend(workflow_router.routes)
|
||||||
FileSpaceResponse(
|
|
||||||
id=f"user:{principal.user.id}",
|
|
||||||
label="My files",
|
|
||||||
owner_type="user",
|
|
||||||
owner_id=principal.user.id,
|
|
||||||
description="Files owned by your user account.",
|
|
||||||
)
|
|
||||||
]
|
|
||||||
group_ids = user_group_ids(session, tenant_id=principal.tenant_id, user_id=principal.user.id, include_admin_groups=_is_admin(principal))
|
|
||||||
if group_ids:
|
|
||||||
groups = session.query(Group).filter(Group.tenant_id == principal.tenant_id, Group.id.in_(group_ids)).order_by(Group.name.asc()).all()
|
|
||||||
spaces.extend(
|
|
||||||
FileSpaceResponse(
|
|
||||||
id=f"group:{group.id}",
|
|
||||||
label=f"{group.name} files",
|
|
||||||
owner_type="group",
|
|
||||||
owner_id=group.id,
|
|
||||||
description="Files owned by this group.",
|
|
||||||
)
|
|
||||||
for group in groups
|
|
||||||
)
|
|
||||||
return FileSpacesResponse(spaces=spaces)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/folders", response_model=FileFoldersResponse)
|
|
||||||
def list_file_folders(
|
|
||||||
owner_type: Literal["user", "group"],
|
|
||||||
owner_id: str,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
folders = list_folders_for_user(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
owner_type=owner_type,
|
|
||||||
owner_id=owner_id,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
return FileFoldersResponse(folders=[_folder_response(folder) for folder in folders])
|
|
||||||
except FileStorageError as exc:
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/folders", response_model=FileFolderResponse)
|
|
||||||
def create_file_folder(
|
|
||||||
payload: FileFolderCreateRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
folder = create_folder(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
owner_type=payload.owner_type,
|
|
||||||
owner_id=payload.owner_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
path=payload.path,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return _folder_response(folder)
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
|
|
||||||
def delete_file_folder(
|
|
||||||
payload: FileFolderDeleteRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
deleted_folders, deleted_files = soft_delete_folder(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
owner_type=payload.owner_type,
|
|
||||||
owner_id=payload.owner_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
path=payload.path,
|
|
||||||
recursive=payload.recursive,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return FileFolderDeleteResponse(deleted_folders=deleted_folders, deleted_files=deleted_files)
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=FileListResponse)
|
|
||||||
def list_files(
|
|
||||||
owner_type: Literal["user", "group"] | None = None,
|
|
||||||
owner_id: str | None = None,
|
|
||||||
campaign_id: str | None = None,
|
|
||||||
path_prefix: str | None = None,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
|
||||||
):
|
|
||||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
|
||||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
|
||||||
assets = list_assets_for_user(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
owner_type=owner_type,
|
|
||||||
owner_id=owner_id,
|
|
||||||
campaign_id=campaign_id,
|
|
||||||
path_prefix=path_prefix,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
return FileListResponse(files=[_asset_response(session, asset, include_shares=True) for asset in assets])
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/upload", response_model=FileUploadResponse)
|
|
||||||
async def upload_files(
|
|
||||||
files: list[UploadFile] = FastAPIFile(...),
|
|
||||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
|
||||||
owner_id: str | None = Form(default=None),
|
|
||||||
path: str = Form(default=""),
|
|
||||||
campaign_id: str | None = Form(default=None),
|
|
||||||
unpack_zip: bool = Form(default=False),
|
|
||||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(default="reject"),
|
|
||||||
conflict_resolutions_json: str | None = Form(default=None),
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
|
||||||
):
|
|
||||||
target_owner = owner_id or principal.user.id
|
|
||||||
uploaded_assets: list[FileAsset] = []
|
|
||||||
try:
|
|
||||||
raw_resolutions = json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
|
||||||
upload_resolutions = _conflict_resolutions([ConflictResolutionRequest(**item) for item in raw_resolutions])
|
|
||||||
for upload in files:
|
|
||||||
filename = upload.filename or "file"
|
|
||||||
content_type = upload.content_type or None
|
|
||||||
upload_limit = settings.file_upload_zip_max_bytes if unpack_zip and filename.lower().endswith(".zip") else settings.file_upload_max_bytes
|
|
||||||
data = await _read_limited_upload(upload, max_bytes=upload_limit)
|
|
||||||
if unpack_zip and filename.lower().endswith(".zip"):
|
|
||||||
extracted = extract_zip_upload(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
owner_type=owner_type,
|
|
||||||
owner_id=target_owner,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
zip_data=data,
|
|
||||||
folder=path,
|
|
||||||
campaign_id=campaign_id,
|
|
||||||
conflict_strategy=conflict_strategy,
|
|
||||||
conflict_resolutions=upload_resolutions,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
max_file_bytes=settings.file_upload_max_bytes,
|
|
||||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
|
||||||
)
|
|
||||||
uploaded_assets.extend(item.asset for item in extracted)
|
|
||||||
continue
|
|
||||||
stored = create_file_asset(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
owner_type=owner_type,
|
|
||||||
owner_id=target_owner,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
filename=filename,
|
|
||||||
data=data,
|
|
||||||
folder=path,
|
|
||||||
content_type=content_type,
|
|
||||||
campaign_id=campaign_id,
|
|
||||||
conflict_strategy=conflict_strategy,
|
|
||||||
conflict_resolutions=upload_resolutions,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
uploaded_assets.append(stored.asset)
|
|
||||||
session.commit()
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
return FileUploadResponse(files=[_asset_response(session, asset, include_shares=True) for asset in uploaded_assets])
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
|
||||||
async def upload_zip(
|
|
||||||
file: UploadFile = FastAPIFile(...),
|
|
||||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
|
||||||
owner_id: str | None = Form(default=None),
|
|
||||||
path: str = Form(default=""),
|
|
||||||
campaign_id: str | None = Form(default=None),
|
|
||||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(default="reject"),
|
|
||||||
conflict_resolutions_json: str | None = Form(default=None),
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
|
||||||
):
|
|
||||||
data = await _read_limited_upload(file, max_bytes=settings.file_upload_zip_max_bytes)
|
|
||||||
target_owner = owner_id or principal.user.id
|
|
||||||
try:
|
|
||||||
raw_resolutions = json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
|
||||||
upload_resolutions = _conflict_resolutions([ConflictResolutionRequest(**item) for item in raw_resolutions])
|
|
||||||
extracted = extract_zip_upload(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
owner_type=owner_type,
|
|
||||||
owner_id=target_owner,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
zip_data=data,
|
|
||||||
folder=path,
|
|
||||||
campaign_id=campaign_id,
|
|
||||||
conflict_strategy=conflict_strategy,
|
|
||||||
conflict_resolutions=upload_resolutions,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
max_file_bytes=settings.file_upload_max_bytes,
|
|
||||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
return FileUploadResponse(files=[_asset_response(session, item.asset, include_shares=True) for item in extracted])
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
|
||||||
def get_file(
|
|
||||||
file_id: str,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
|
||||||
return _asset_response(session, asset, include_shares=True)
|
|
||||||
except FileStorageError as exc:
|
|
||||||
raise _http_error(exc, not_found=True) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{file_id}/download")
|
|
||||||
def download_file(
|
|
||||||
file_id: str,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
|
||||||
data, _, blob = read_asset_bytes(session, asset)
|
|
||||||
except FileStorageError as exc:
|
|
||||||
raise _http_error(exc, not_found=True) from exc
|
|
||||||
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
|
|
||||||
return StreamingResponse(BytesIO(data), media_type=blob.content_type or "application/octet-stream", headers=headers)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
|
||||||
def delete_file(
|
|
||||||
file_id: str,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
|
||||||
count = soft_delete_assets(session, [asset])
|
|
||||||
session.commit()
|
|
||||||
return BulkDeleteResponse(deleted_count=count)
|
|
||||||
except FileStorageError as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc, not_found=True) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
|
|
||||||
def bulk_delete_files(
|
|
||||||
payload: BulkDeleteRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
assets = [
|
|
||||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
|
||||||
for file_id in payload.file_ids
|
|
||||||
]
|
|
||||||
count = soft_delete_assets(session, assets)
|
|
||||||
session.commit()
|
|
||||||
return BulkDeleteResponse(deleted_count=count)
|
|
||||||
except FileStorageError as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/{file_id}/shares", response_model=FileShareResponse)
|
|
||||||
def create_share(
|
|
||||||
file_id: str,
|
|
||||||
payload: FileShareRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
|
||||||
share = share_file(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
asset=asset,
|
|
||||||
target_type=payload.target_type,
|
|
||||||
target_id=payload.target_id,
|
|
||||||
permission=payload.permission,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return FileShareResponse(
|
|
||||||
id=share.id,
|
|
||||||
target_type=share.target_type,
|
|
||||||
target_id=share.target_id,
|
|
||||||
permission=share.permission,
|
|
||||||
created_at=share.created_at.isoformat(),
|
|
||||||
revoked_at=share.revoked_at.isoformat() if share.revoked_at else None,
|
|
||||||
)
|
|
||||||
except FileStorageError as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/bulk-rename", response_model=RenameResponse)
|
|
||||||
def bulk_rename(
|
|
||||||
payload: RenameRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
plan = rename_selection(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
file_ids=payload.file_ids,
|
|
||||||
folder_paths=payload.folder_paths,
|
|
||||||
owner_type=payload.owner_type,
|
|
||||||
owner_id=payload.owner_id,
|
|
||||||
mode=payload.mode,
|
|
||||||
new_name=payload.new_name,
|
|
||||||
find=payload.find,
|
|
||||||
replacement=payload.replacement,
|
|
||||||
prefix=payload.prefix,
|
|
||||||
suffix=payload.suffix,
|
|
||||||
recursive=payload.recursive,
|
|
||||||
dry_run=payload.dry_run,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
if not payload.dry_run:
|
|
||||||
session.commit()
|
|
||||||
return RenameResponse(
|
|
||||||
dry_run=payload.dry_run,
|
|
||||||
items=[
|
|
||||||
RenamePreviewItem(
|
|
||||||
kind=item.kind,
|
|
||||||
id=item.id,
|
|
||||||
file_id=item.id if item.kind == "file" else None,
|
|
||||||
folder_path=item.old_path if item.kind == "folder" else None,
|
|
||||||
old_path=item.old_path,
|
|
||||||
new_path=item.new_path,
|
|
||||||
)
|
|
||||||
for item in plan
|
|
||||||
],
|
|
||||||
)
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/transfer", response_model=TransferResponse)
|
|
||||||
def transfer_files(
|
|
||||||
payload: TransferRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
files, folders = transfer_selection(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
operation=payload.operation,
|
|
||||||
file_ids=payload.file_ids,
|
|
||||||
folder_paths=payload.folder_paths,
|
|
||||||
source_owner_type=payload.source_owner_type,
|
|
||||||
source_owner_id=payload.source_owner_id,
|
|
||||||
target_owner_type=payload.target_owner_type,
|
|
||||||
target_owner_id=payload.target_owner_id,
|
|
||||||
target_folder=payload.target_folder,
|
|
||||||
conflict_strategy=payload.conflict_strategy,
|
|
||||||
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return TransferResponse(operation=payload.operation, files=files, folders=folders)
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
session.rollback()
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/archive.zip")
|
|
||||||
def download_archive(
|
|
||||||
payload: ArchiveRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
assets = [
|
|
||||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
|
||||||
for file_id in payload.file_ids
|
|
||||||
]
|
|
||||||
tmp = tempfile.NamedTemporaryFile(prefix="multimailer-files-", suffix=".zip", delete=False)
|
|
||||||
tmp_path = tmp.name
|
|
||||||
tmp.close()
|
|
||||||
try:
|
|
||||||
create_zip_file(session, assets, tmp_path)
|
|
||||||
except Exception:
|
|
||||||
_cleanup_temp_file(tmp_path)
|
|
||||||
raise
|
|
||||||
except FileStorageError as exc:
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
filename = filename_from_path(normalize_logical_path(payload.filename, fallback_filename="files.zip"))
|
|
||||||
headers = {"Content-Disposition": _attachment_disposition(filename)}
|
|
||||||
return FileResponse(tmp_path, media_type="application/zip", headers=headers, background=BackgroundTask(_cleanup_temp_file, tmp_path))
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
|
|
||||||
def resolve_file_patterns(
|
|
||||||
payload: PatternResolveRequest,
|
|
||||||
session: Session = Depends(get_session),
|
|
||||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
|
||||||
):
|
|
||||||
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
|
|
||||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
|
||||||
try:
|
|
||||||
assets = list_assets_for_user(
|
|
||||||
session,
|
|
||||||
tenant_id=principal.tenant_id,
|
|
||||||
user_id=principal.user.id,
|
|
||||||
owner_type=payload.owner_type,
|
|
||||||
owner_id=payload.owner_id,
|
|
||||||
campaign_id=payload.campaign_id,
|
|
||||||
path_prefix=payload.path_prefix,
|
|
||||||
is_admin=_is_admin(principal),
|
|
||||||
)
|
|
||||||
resolved, unmatched = resolve_patterns(assets, payload.patterns, base_path=payload.path_prefix, case_sensitive=payload.case_sensitive)
|
|
||||||
return PatternResolveResponse(
|
|
||||||
patterns=[PatternMatchResponse(pattern=item.pattern, matches=[_asset_response(session, asset) for asset in item.matches]) for item in resolved],
|
|
||||||
unmatched=[_asset_response(session, asset) for asset in unmatched] if payload.include_unmatched else [],
|
|
||||||
)
|
|
||||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
|
||||||
raise _http_error(exc) from exc
|
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Focused HTTP route modules for the Files API."""
|
||||||
@@ -0,0 +1,175 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from io import BytesIO
|
||||||
|
from fastapi import APIRouter, Depends
|
||||||
|
from fastapi.responses import StreamingResponse
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
BulkDeleteRequest,
|
||||||
|
BulkDeleteResponse,
|
||||||
|
FileAssetResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
get_asset_for_user,
|
||||||
|
read_asset_bytes,
|
||||||
|
read_asset_version_bytes,
|
||||||
|
soft_delete_assets,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_response,
|
||||||
|
_attachment_disposition,
|
||||||
|
_audit_connector_event,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||||
|
def get_file(
|
||||||
|
file_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
return _asset_response(session, asset, include_shares=True)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{file_id}/download")
|
||||||
|
def download_file(
|
||||||
|
file_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
data, version, blob = read_asset_bytes(session, asset)
|
||||||
|
_audit_connector_event(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.connector.accessed",
|
||||||
|
asset=asset,
|
||||||
|
version=version,
|
||||||
|
blob=blob,
|
||||||
|
operation="download",
|
||||||
|
commit=True,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
|
||||||
|
return StreamingResponse(
|
||||||
|
BytesIO(data),
|
||||||
|
media_type=blob.content_type or "application/octet-stream",
|
||||||
|
headers=headers,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{file_id}/versions/{version_id}/download")
|
||||||
|
def download_file_version(
|
||||||
|
file_id: str,
|
||||||
|
version_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
data, version, blob = read_asset_version_bytes(session, asset, version_id)
|
||||||
|
_audit_connector_event(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.connector.accessed",
|
||||||
|
asset=asset,
|
||||||
|
version=version,
|
||||||
|
blob=blob,
|
||||||
|
operation="download-version",
|
||||||
|
commit=True,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
headers = {
|
||||||
|
"Content-Disposition": _attachment_disposition(
|
||||||
|
version.filename_at_upload or asset.filename
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return StreamingResponse(
|
||||||
|
BytesIO(data),
|
||||||
|
media_type=blob.content_type or "application/octet-stream",
|
||||||
|
headers=headers,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
||||||
|
def delete_file(
|
||||||
|
file_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
require_write=True,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
count = soft_delete_assets(session, [asset])
|
||||||
|
session.commit()
|
||||||
|
return BulkDeleteResponse(deleted_count=count)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
|
||||||
|
def bulk_delete_files(
|
||||||
|
payload: BulkDeleteRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
assets = [
|
||||||
|
get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
require_write=True,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
for file_id in payload.file_ids
|
||||||
|
]
|
||||||
|
count = soft_delete_assets(session, assets)
|
||||||
|
session.commit()
|
||||||
|
return BulkDeleteResponse(deleted_count=count)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
@@ -0,0 +1,367 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_any_scope, require_scope
|
||||||
|
from govoplan_core.audit.logging import audit_from_principal
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileConnectorBrowseItem,
|
||||||
|
FileConnectorBrowseResponse,
|
||||||
|
FileConnectorImportRequest,
|
||||||
|
FileConnectorSyncResponse,
|
||||||
|
FileConnectorWriteRequest,
|
||||||
|
FileConnectorWriteResponse,
|
||||||
|
FileUploadResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.paths import UnsafeFilePathError
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_browse import (
|
||||||
|
ConnectorBrowseError,
|
||||||
|
ConnectorBrowseUnsupported,
|
||||||
|
browse_connector_profile,
|
||||||
|
normalize_connector_browse_path,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_imports import (
|
||||||
|
ConnectorImportError,
|
||||||
|
ConnectorImportUnsupported,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
connector_effective_endpoint_url,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorAccessRequest,
|
||||||
|
ConnectorPolicyDenied,
|
||||||
|
connector_policy_decision,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
create_file_asset,
|
||||||
|
get_asset_for_user,
|
||||||
|
read_asset_bytes,
|
||||||
|
sync_file_asset_from_source,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_spaces import (
|
||||||
|
get_connector_space_for_user,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_writes import write_connector_file
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_response,
|
||||||
|
_audit_connector_imports,
|
||||||
|
_audit_connector_sync,
|
||||||
|
_connector_browse_next_token,
|
||||||
|
_connector_policy_error,
|
||||||
|
_download_connector_payload,
|
||||||
|
_ensure_campaign_file_access,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
_visible_connector_profile,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connector-spaces/{space_id}/write-back",
|
||||||
|
response_model=FileConnectorWriteResponse,
|
||||||
|
)
|
||||||
|
def write_back_connector_file(
|
||||||
|
space_id: str,
|
||||||
|
payload: FileConnectorWriteRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:connector:write")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
space = get_connector_space_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
space_id=space_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
if space.read_only:
|
||||||
|
raise FileStorageError("This connector space is read-only")
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session, principal, space.connector_profile_id
|
||||||
|
)
|
||||||
|
requested_path = normalize_connector_browse_path(payload.remote_path)
|
||||||
|
remote_path = normalize_connector_browse_path(
|
||||||
|
"/".join(part for part in (space.remote_path, requested_path) if part)
|
||||||
|
)
|
||||||
|
decision = connector_policy_decision(
|
||||||
|
ConnectorAccessRequest(
|
||||||
|
connector_id=profile.id,
|
||||||
|
credential_id=profile.credential_profile_id,
|
||||||
|
provider=profile.provider,
|
||||||
|
external_path=remote_path,
|
||||||
|
external_url=connector_effective_endpoint_url(
|
||||||
|
provider=profile.provider,
|
||||||
|
endpoint_url=profile.endpoint_url,
|
||||||
|
metadata=profile.metadata,
|
||||||
|
),
|
||||||
|
operation="write",
|
||||||
|
),
|
||||||
|
profile.policy_sources,
|
||||||
|
)
|
||||||
|
if not decision.allowed:
|
||||||
|
raise ConnectorPolicyDenied(decision)
|
||||||
|
asset = get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=payload.file_id,
|
||||||
|
require_write=True,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
data, version, blob = read_asset_bytes(session, asset)
|
||||||
|
connector_library_id = space.library_id
|
||||||
|
content_type = blob.content_type
|
||||||
|
# Close the read snapshot before the independent recovery transaction
|
||||||
|
# records authority for the external effect. This avoids upgrading an
|
||||||
|
# older SQLite read snapshot after the ledger commit.
|
||||||
|
session.commit()
|
||||||
|
result = write_connector_file(
|
||||||
|
profile,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
library_id=connector_library_id,
|
||||||
|
remote_path=remote_path,
|
||||||
|
data=data,
|
||||||
|
content_type=content_type,
|
||||||
|
expected_revision=payload.expected_revision,
|
||||||
|
idempotency_key=payload.idempotency_key,
|
||||||
|
)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.connector.written",
|
||||||
|
object_type="file",
|
||||||
|
object_id=asset.id,
|
||||||
|
details={
|
||||||
|
"file_version_id": version.id,
|
||||||
|
"file_blob_id": blob.id,
|
||||||
|
"checksum_sha256": blob.checksum_sha256,
|
||||||
|
"connector_space_id": space.id,
|
||||||
|
"connector_profile_id": profile.id,
|
||||||
|
"remote_path": remote_path,
|
||||||
|
"recovery_operation_id": result.recovery_operation_id,
|
||||||
|
"recovery_status": result.status,
|
||||||
|
"revision": result.revision,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return FileConnectorWriteResponse(
|
||||||
|
recovery_operation_id=result.recovery_operation_id,
|
||||||
|
status=result.status,
|
||||||
|
replayed=result.replayed,
|
||||||
|
provider=result.provider,
|
||||||
|
remote_path=result.remote_path,
|
||||||
|
revision=result.revision,
|
||||||
|
checksum_sha256=result.checksum_sha256,
|
||||||
|
size_bytes=result.size_bytes,
|
||||||
|
)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ConnectorBrowseError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connectors/profiles/{profile_id}/import", response_model=FileUploadResponse
|
||||||
|
)
|
||||||
|
def import_connector_file(
|
||||||
|
profile_id: str,
|
||||||
|
payload: FileConnectorImportRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
if payload.campaign_id:
|
||||||
|
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session, principal, profile_id, campaign_id=payload.campaign_id
|
||||||
|
)
|
||||||
|
_source_path, downloaded, metadata = _download_connector_payload(
|
||||||
|
profile, payload, operation="import"
|
||||||
|
)
|
||||||
|
target_owner = payload.owner_id or principal.user.id
|
||||||
|
stored = create_file_asset(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
filename=downloaded.filename,
|
||||||
|
data=downloaded.data,
|
||||||
|
folder=payload.target_folder,
|
||||||
|
display_path=payload.target_path,
|
||||||
|
content_type=downloaded.content_type,
|
||||||
|
metadata=metadata,
|
||||||
|
campaign_id=payload.campaign_id,
|
||||||
|
conflict_strategy=payload.conflict_strategy,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
_audit_connector_imports(session, principal, [stored.asset])
|
||||||
|
session.commit()
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except ConnectorImportUnsupported as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||||
|
) from exc
|
||||||
|
except (
|
||||||
|
ConnectorImportError,
|
||||||
|
FileStorageError,
|
||||||
|
UnsafeFilePathError,
|
||||||
|
ValueError,
|
||||||
|
json.JSONDecodeError,
|
||||||
|
) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return FileUploadResponse(
|
||||||
|
files=[_asset_response(session, stored.asset, include_shares=True)]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connectors/profiles/{profile_id}/sync", response_model=FileConnectorSyncResponse
|
||||||
|
)
|
||||||
|
def sync_connector_file(
|
||||||
|
profile_id: str,
|
||||||
|
payload: FileConnectorImportRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
if payload.campaign_id:
|
||||||
|
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session, principal, profile_id, campaign_id=payload.campaign_id
|
||||||
|
)
|
||||||
|
_source_path, downloaded, metadata = _download_connector_payload(
|
||||||
|
profile, payload, operation="sync"
|
||||||
|
)
|
||||||
|
target_owner = payload.owner_id or principal.user.id
|
||||||
|
stored, sync_action, previous_version_id = sync_file_asset_from_source(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
filename=downloaded.filename,
|
||||||
|
data=downloaded.data,
|
||||||
|
folder=payload.target_folder,
|
||||||
|
display_path=payload.target_path,
|
||||||
|
content_type=downloaded.content_type,
|
||||||
|
metadata=metadata,
|
||||||
|
campaign_id=payload.campaign_id,
|
||||||
|
conflict_strategy=payload.conflict_strategy,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
_audit_connector_sync(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
stored.asset,
|
||||||
|
sync_action=sync_action,
|
||||||
|
previous_version_id=previous_version_id,
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except ConnectorImportUnsupported as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||||
|
) from exc
|
||||||
|
except (
|
||||||
|
ConnectorImportError,
|
||||||
|
FileStorageError,
|
||||||
|
UnsafeFilePathError,
|
||||||
|
ValueError,
|
||||||
|
json.JSONDecodeError,
|
||||||
|
) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return FileConnectorSyncResponse(
|
||||||
|
file=_asset_response(session, stored.asset, include_shares=True),
|
||||||
|
action=sync_action,
|
||||||
|
previous_version_id=previous_version_id,
|
||||||
|
current_version_id=stored.version.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/connectors/profiles/{profile_id}/browse",
|
||||||
|
response_model=FileConnectorBrowseResponse,
|
||||||
|
)
|
||||||
|
def browse_connector_profile_items(
|
||||||
|
profile_id: str,
|
||||||
|
path: str | None = None,
|
||||||
|
library_id: str | None = None,
|
||||||
|
continuation_token: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:upload",
|
||||||
|
"files:file:download",
|
||||||
|
"files:file:admin",
|
||||||
|
"system:settings:read",
|
||||||
|
"admin:settings:read",
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session, principal, profile_id, campaign_id=campaign_id
|
||||||
|
)
|
||||||
|
browse_path = normalize_connector_browse_path(path)
|
||||||
|
decision = connector_policy_decision(
|
||||||
|
ConnectorAccessRequest(
|
||||||
|
connector_id=profile.id,
|
||||||
|
credential_id=profile.credential_profile_id,
|
||||||
|
provider=profile.provider,
|
||||||
|
external_path=browse_path,
|
||||||
|
external_url=connector_effective_endpoint_url(
|
||||||
|
provider=profile.provider,
|
||||||
|
endpoint_url=profile.endpoint_url,
|
||||||
|
metadata=profile.metadata,
|
||||||
|
),
|
||||||
|
operation="browse",
|
||||||
|
),
|
||||||
|
profile.policy_sources,
|
||||||
|
)
|
||||||
|
if not decision.allowed:
|
||||||
|
raise ConnectorPolicyDenied(decision)
|
||||||
|
items = browse_connector_profile(
|
||||||
|
profile,
|
||||||
|
path=browse_path,
|
||||||
|
library_id=library_id,
|
||||||
|
continuation_token=continuation_token,
|
||||||
|
)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except ConnectorBrowseUnsupported as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||||
|
) from exc
|
||||||
|
except (ConnectorBrowseError, OSError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return FileConnectorBrowseResponse(
|
||||||
|
profile_id=profile.id,
|
||||||
|
provider=profile.provider,
|
||||||
|
path=browse_path,
|
||||||
|
library_id=library_id,
|
||||||
|
next_continuation_token=_connector_browse_next_token(items),
|
||||||
|
has_more=any(bool(item.metadata.get("listing_truncated")) for item in items),
|
||||||
|
decision=decision.to_dict(),
|
||||||
|
items=[FileConnectorBrowseItem(**item.to_response()) for item in items],
|
||||||
|
)
|
||||||
@@ -0,0 +1,261 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from fastapi import APIRouter, Depends
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||||
|
from govoplan_files.backend.change_tracking import (
|
||||||
|
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileConnectorProfileResponse,
|
||||||
|
FileConnectorProfileUpdateRequest,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||||
|
delete_connector_profile_row,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
connector_effective_endpoint_url,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profile_store import (
|
||||||
|
connector_profile_from_row,
|
||||||
|
get_connector_profile_row,
|
||||||
|
update_connector_profile_row,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorPolicyDenied,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
_can_read_disabled_connector_profiles,
|
||||||
|
_connector_policy_error,
|
||||||
|
_credential_row_for_profile,
|
||||||
|
_ensure_connector_configuration_allowed,
|
||||||
|
_ensure_connector_local_policy_allowed,
|
||||||
|
_http_error,
|
||||||
|
_record_connector_settings_change,
|
||||||
|
_require_connector_profile_write,
|
||||||
|
_visible_connector_profile,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||||
|
)
|
||||||
|
def get_connector_profile(
|
||||||
|
profile_id: str,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:upload",
|
||||||
|
"files:file:download",
|
||||||
|
"files:file:admin",
|
||||||
|
"system:settings:read",
|
||||||
|
"admin:settings:read",
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
profile_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled
|
||||||
|
and _can_read_disabled_connector_profiles(principal),
|
||||||
|
include_effective_policy=False,
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return FileConnectorProfileResponse(**profile.to_response())
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch(
|
||||||
|
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||||
|
)
|
||||||
|
def update_connector_profile(
|
||||||
|
profile_id: str,
|
||||||
|
payload: FileConnectorProfileUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
row = get_connector_profile_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
profile_id=profile_id,
|
||||||
|
include_disabled=True,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
_require_connector_profile_write(principal, row.scope_type)
|
||||||
|
credentials = payload.credentials
|
||||||
|
try:
|
||||||
|
credential_profile_id = (
|
||||||
|
payload.credential_profile_id
|
||||||
|
if payload.credential_profile_id is not None
|
||||||
|
else row.credential_profile_id
|
||||||
|
)
|
||||||
|
provider = payload.provider if payload.provider is not None else row.provider
|
||||||
|
credential_row = _credential_row_for_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
credential_profile_id=credential_profile_id,
|
||||||
|
provider=provider,
|
||||||
|
profile_id=row.id,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
include_disabled=True,
|
||||||
|
)
|
||||||
|
_ensure_connector_configuration_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
connector_id=row.id,
|
||||||
|
credential_id=credential_profile_id,
|
||||||
|
provider=provider,
|
||||||
|
endpoint_url=connector_effective_endpoint_url(
|
||||||
|
provider=provider,
|
||||||
|
endpoint_url=payload.endpoint_url
|
||||||
|
if payload.endpoint_url is not None
|
||||||
|
else row.endpoint_url,
|
||||||
|
metadata=payload.metadata
|
||||||
|
if payload.metadata is not None
|
||||||
|
else row.metadata_,
|
||||||
|
),
|
||||||
|
base_path=payload.base_path
|
||||||
|
if payload.base_path is not None
|
||||||
|
else row.base_path,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
operation="configure",
|
||||||
|
)
|
||||||
|
if payload.policy is not None:
|
||||||
|
_ensure_connector_local_policy_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
policy=payload.policy,
|
||||||
|
)
|
||||||
|
update_connector_profile_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
label=payload.label,
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=payload.endpoint_url,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
enabled=payload.enabled,
|
||||||
|
credential_profile_id=payload.credential_profile_id,
|
||||||
|
credential_mode=payload.credential_mode,
|
||||||
|
username=credentials.username if credentials else None,
|
||||||
|
password=credentials.password if credentials else None,
|
||||||
|
token=credentials.token if credentials else None,
|
||||||
|
password_env=credentials.password_env if credentials else None,
|
||||||
|
token_env=credentials.token_env if credentials else None,
|
||||||
|
secret_ref=credentials.secret_ref if credentials else None,
|
||||||
|
clear_password=payload.clear_password,
|
||||||
|
clear_token=payload.clear_token,
|
||||||
|
capabilities=payload.capabilities,
|
||||||
|
policy=payload.policy,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
resource_id=row.id,
|
||||||
|
operation="updated",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(row)
|
||||||
|
return FileConnectorProfileResponse(
|
||||||
|
**connector_profile_from_row(
|
||||||
|
row, credential_row=credential_row
|
||||||
|
).to_response()
|
||||||
|
)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete(
|
||||||
|
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||||
|
)
|
||||||
|
def deactivate_connector_profile(
|
||||||
|
profile_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
row = get_connector_profile_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
profile_id=profile_id,
|
||||||
|
include_disabled=True,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
_require_connector_profile_write(principal, row.scope_type)
|
||||||
|
try:
|
||||||
|
changed = delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason="api_delete",
|
||||||
|
user_id=principal.user.id,
|
||||||
|
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||||
|
)
|
||||||
|
if changed:
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
resource_id=row.id,
|
||||||
|
operation="deleted",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(row)
|
||||||
|
return FileConnectorProfileResponse(
|
||||||
|
**connector_profile_from_row(row).to_response()
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
except Exception:
|
||||||
|
session.rollback()
|
||||||
|
raise
|
||||||
@@ -0,0 +1,852 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Literal
|
||||||
|
from fastapi import APIRouter, Depends, Query, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||||
|
from govoplan_files.backend.change_tracking import (
|
||||||
|
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||||
|
FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||||
|
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileConnectorCredentialCreateRequest,
|
||||||
|
FileConnectorCredentialResponse,
|
||||||
|
FileConnectorCredentialsResponse,
|
||||||
|
FileConnectorCredentialUpdateRequest,
|
||||||
|
FileConnectorDiscoveryRequest,
|
||||||
|
FileConnectorDiscoveryResponse,
|
||||||
|
FileConnectorSettingsDeltaResponse,
|
||||||
|
FileConnectorPolicyEvaluateRequest,
|
||||||
|
FileConnectorPolicyEvaluateResponse,
|
||||||
|
FileConnectorPolicyResponse,
|
||||||
|
FileConnectorPolicyUpdateRequest,
|
||||||
|
FileConnectorProfileCreateRequest,
|
||||||
|
FileConnectorProfileResponse,
|
||||||
|
FileConnectorProfilesResponse,
|
||||||
|
FileConnectorProviderResponse,
|
||||||
|
FileConnectorProvidersResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_credential_store import (
|
||||||
|
create_connector_credential_row,
|
||||||
|
get_connector_credential_row,
|
||||||
|
list_database_connector_credentials,
|
||||||
|
update_connector_credential_row,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||||
|
delete_connector_credential_row,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_browse import (
|
||||||
|
ConnectorBrowseError,
|
||||||
|
ConnectorBrowseUnsupported,
|
||||||
|
browse_connector_profile,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
connector_effective_endpoint_url,
|
||||||
|
reject_api_controlled_deployment_references,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profile_store import (
|
||||||
|
connector_profile_from_row,
|
||||||
|
create_connector_profile_row,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_providers import (
|
||||||
|
connector_provider_descriptors,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorAccessRequest,
|
||||||
|
ConnectorPolicyDenied,
|
||||||
|
connector_policy_decision,
|
||||||
|
connector_policy_sources_from_payload,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy_store import (
|
||||||
|
connector_policy_response,
|
||||||
|
set_connector_policy,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||||
|
FILES_CONNECTOR_POLICY_RESOURCE,
|
||||||
|
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
_audit_connector_discovery_attempt,
|
||||||
|
_can_read_disabled_connector_profiles,
|
||||||
|
_connector_credential_response,
|
||||||
|
_connector_policy_error,
|
||||||
|
_credential_row_for_profile,
|
||||||
|
_discovery_profile_from_payload,
|
||||||
|
_ensure_campaign_file_access,
|
||||||
|
_ensure_connector_configuration_allowed,
|
||||||
|
_ensure_connector_credential_configuration_allowed,
|
||||||
|
_ensure_connector_local_policy_allowed,
|
||||||
|
_file_connector_policy_resource_id,
|
||||||
|
_file_connector_settings_entries,
|
||||||
|
_http_error,
|
||||||
|
_record_connector_settings_change,
|
||||||
|
_require_connector_credential_write,
|
||||||
|
_require_connector_policy_read,
|
||||||
|
_require_connector_profile_write,
|
||||||
|
_same_endpoint,
|
||||||
|
_visible_connector_profiles,
|
||||||
|
_webdav_discovery_candidates,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.services.connector_settings_delta import (
|
||||||
|
_full_file_connector_settings_delta_response,
|
||||||
|
_incremental_file_connector_settings_delta_response,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connector-policy/evaluate", response_model=FileConnectorPolicyEvaluateResponse
|
||||||
|
)
|
||||||
|
def evaluate_connector_policy(
|
||||||
|
payload: FileConnectorPolicyEvaluateRequest,
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope("files:file:read", "files:file:upload", "files:file:download")
|
||||||
|
),
|
||||||
|
):
|
||||||
|
del principal
|
||||||
|
sources = connector_policy_sources_from_payload(
|
||||||
|
[item.model_dump(mode="json") for item in payload.policy_sources]
|
||||||
|
)
|
||||||
|
request = ConnectorAccessRequest.from_provenance(
|
||||||
|
payload.source_provenance.model_dump(mode="json", exclude_none=True),
|
||||||
|
operation=payload.operation,
|
||||||
|
)
|
||||||
|
return FileConnectorPolicyEvaluateResponse(
|
||||||
|
decision=connector_policy_decision(request, sources).to_dict()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/connectors/settings/delta", response_model=FileConnectorSettingsDeltaResponse
|
||||||
|
)
|
||||||
|
def connector_settings_delta(
|
||||||
|
scope_type: str = Query(default="tenant"),
|
||||||
|
scope_id: str | None = Query(default=None),
|
||||||
|
provider: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
include_inactive: bool = False,
|
||||||
|
owner_type: Literal["user", "group"] | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
since: str | None = None,
|
||||||
|
limit: int = Query(default=100, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
scope_type = scope_type.strip().casefold()
|
||||||
|
_require_connector_policy_read(principal, scope_type)
|
||||||
|
try:
|
||||||
|
if since is None:
|
||||||
|
return _full_file_connector_settings_delta_response(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
include_inactive=include_inactive,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
)
|
||||||
|
entries, has_more = _file_connector_settings_entries(
|
||||||
|
session, tenant_id=principal.tenant_id, since=since, limit=limit
|
||||||
|
)
|
||||||
|
if entries is None:
|
||||||
|
return _full_file_connector_settings_delta_response(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
include_inactive=include_inactive,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
)
|
||||||
|
return _incremental_file_connector_settings_delta_response(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
entries=entries,
|
||||||
|
has_more=has_more,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
include_inactive=include_inactive,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
)
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/connectors/providers", response_model=FileConnectorProvidersResponse)
|
||||||
|
def list_connector_providers(
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope("files:file:read", "files:file:upload", "files:file:admin")
|
||||||
|
),
|
||||||
|
):
|
||||||
|
del principal
|
||||||
|
return FileConnectorProvidersResponse(
|
||||||
|
providers=[
|
||||||
|
FileConnectorProviderResponse(**item.to_response())
|
||||||
|
for item in connector_provider_descriptors()
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/connectors/discover", response_model=FileConnectorDiscoveryResponse)
|
||||||
|
def discover_connector_endpoint(
|
||||||
|
payload: FileConnectorDiscoveryRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
reject_api_controlled_deployment_references(
|
||||||
|
password_env=payload.credentials.password_env,
|
||||||
|
token_env=payload.credentials.token_env,
|
||||||
|
secret_ref=payload.credentials.secret_ref,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
if payload.provider not in {"webdav", "nextcloud"}:
|
||||||
|
return FileConnectorDiscoveryResponse(
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=None,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
status="unsupported",
|
||||||
|
message=f"Discovery is not implemented for {payload.provider} connectors yet",
|
||||||
|
)
|
||||||
|
candidates: list[dict[str, str]] = []
|
||||||
|
for endpoint_url in _webdav_discovery_candidates(payload):
|
||||||
|
profile = _discovery_profile_from_payload(
|
||||||
|
payload, endpoint_url, principal=principal
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_ensure_connector_configuration_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
connector_id=None,
|
||||||
|
credential_id=None,
|
||||||
|
provider=profile.provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=profile.base_path,
|
||||||
|
scope_type="tenant",
|
||||||
|
scope_id=principal.tenant_id,
|
||||||
|
operation="discover",
|
||||||
|
)
|
||||||
|
_audit_connector_discovery_attempt(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
provider=profile.provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=profile.base_path,
|
||||||
|
)
|
||||||
|
browse_connector_profile(profile, path=payload.base_path or "")
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (
|
||||||
|
ConnectorBrowseError,
|
||||||
|
ConnectorBrowseUnsupported,
|
||||||
|
OSError,
|
||||||
|
ValueError,
|
||||||
|
json.JSONDecodeError,
|
||||||
|
) as exc:
|
||||||
|
message = str(exc)
|
||||||
|
if "credentials were rejected" in message.casefold():
|
||||||
|
if payload.require_valid_credentials:
|
||||||
|
candidates.append(
|
||||||
|
{
|
||||||
|
"endpoint_url": endpoint_url,
|
||||||
|
"status": "credentials_rejected",
|
||||||
|
"message": "The endpoint exists, but the credentials were rejected.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return FileConnectorDiscoveryResponse(
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
status="credentials_rejected",
|
||||||
|
message="The endpoint was found, but login failed with these credentials.",
|
||||||
|
candidates=candidates,
|
||||||
|
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||||
|
)
|
||||||
|
candidates.append(
|
||||||
|
{
|
||||||
|
"endpoint_url": endpoint_url,
|
||||||
|
"status": "found",
|
||||||
|
"message": "The endpoint exists, but credentials are required or were rejected.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return FileConnectorDiscoveryResponse(
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
status="found",
|
||||||
|
message="The endpoint was found. Add working credentials before saving or testing the connection.",
|
||||||
|
candidates=candidates,
|
||||||
|
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||||
|
)
|
||||||
|
candidates.append(
|
||||||
|
{"endpoint_url": endpoint_url, "status": "failed", "message": message}
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
status_value = (
|
||||||
|
"usable" if _same_endpoint(endpoint_url, payload.endpoint_url) else "found"
|
||||||
|
)
|
||||||
|
message = (
|
||||||
|
"The supplied URL is directly usable."
|
||||||
|
if status_value == "usable"
|
||||||
|
else "A usable connector endpoint was discovered."
|
||||||
|
)
|
||||||
|
candidates.append(
|
||||||
|
{"endpoint_url": endpoint_url, "status": status_value, "message": message}
|
||||||
|
)
|
||||||
|
return FileConnectorDiscoveryResponse(
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
status=status_value,
|
||||||
|
message=message,
|
||||||
|
candidates=candidates,
|
||||||
|
metadata={"discovered_by": "webdav-propfind"},
|
||||||
|
)
|
||||||
|
return FileConnectorDiscoveryResponse(
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=None,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
status="not_found",
|
||||||
|
message="No usable WebDAV endpoint was found for this server URL.",
|
||||||
|
candidates=candidates,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/connectors/credentials", response_model=FileConnectorCredentialsResponse)
|
||||||
|
def list_connector_credentials(
|
||||||
|
provider: str | None = None,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
provider_norm = provider.strip().casefold() if provider else None
|
||||||
|
credentials = list_database_connector_credentials(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
include_disabled=include_disabled
|
||||||
|
and _can_read_disabled_connector_profiles(principal),
|
||||||
|
)
|
||||||
|
return FileConnectorCredentialsResponse(
|
||||||
|
credentials=[
|
||||||
|
FileConnectorCredentialResponse(**credential.to_response())
|
||||||
|
for credential in credentials
|
||||||
|
if provider_norm is None or credential.provider in {None, provider_norm}
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
|
||||||
|
)
|
||||||
|
def read_connector_policy(
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = Query(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
_require_connector_policy_read(principal, scope_type)
|
||||||
|
try:
|
||||||
|
return FileConnectorPolicyResponse(
|
||||||
|
**connector_policy_response(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.put(
|
||||||
|
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
|
||||||
|
)
|
||||||
|
def write_connector_policy(
|
||||||
|
scope_type: str,
|
||||||
|
payload: FileConnectorPolicyUpdateRequest,
|
||||||
|
scope_id: str | None = Query(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
_require_connector_profile_write(principal, scope_type)
|
||||||
|
try:
|
||||||
|
set_connector_policy(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
policy=payload.policy,
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_POLICY_RESOURCE,
|
||||||
|
resource_id=_file_connector_policy_resource_id(scope_type, scope_id),
|
||||||
|
operation="updated",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=None
|
||||||
|
if scope_type.strip().casefold() == "system"
|
||||||
|
else principal.tenant_id,
|
||||||
|
payload={"scope_type": scope_type.strip().casefold(), "scope_id": scope_id},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return FileConnectorPolicyResponse(
|
||||||
|
**connector_policy_response(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connectors/credentials",
|
||||||
|
response_model=FileConnectorCredentialResponse,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def create_connector_credential(
|
||||||
|
payload: FileConnectorCredentialCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
_require_connector_credential_write(principal, payload.scope_type)
|
||||||
|
credentials = payload.credentials
|
||||||
|
try:
|
||||||
|
_ensure_connector_credential_configuration_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
credential_id=payload.id,
|
||||||
|
provider=payload.provider,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
operation="configure_credentials",
|
||||||
|
)
|
||||||
|
_ensure_connector_local_policy_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
policy=payload.policy,
|
||||||
|
)
|
||||||
|
row = create_connector_credential_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
credential_id=payload.id,
|
||||||
|
label=payload.label,
|
||||||
|
provider=payload.provider,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
enabled=payload.enabled,
|
||||||
|
credential_mode=payload.credential_mode,
|
||||||
|
username=credentials.username,
|
||||||
|
password=credentials.password,
|
||||||
|
token=credentials.token,
|
||||||
|
password_env=credentials.password_env,
|
||||||
|
token_env=credentials.token_env,
|
||||||
|
secret_ref=credentials.secret_ref,
|
||||||
|
policy=payload.policy,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||||
|
resource_id=row.id,
|
||||||
|
operation="created",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(row)
|
||||||
|
return _connector_credential_response(row)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/connectors/credentials/{credential_id}",
|
||||||
|
response_model=FileConnectorCredentialResponse,
|
||||||
|
)
|
||||||
|
def get_connector_credential(
|
||||||
|
credential_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
row = get_connector_credential_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
credential_id=credential_id,
|
||||||
|
include_disabled=include_disabled
|
||||||
|
and _can_read_disabled_connector_profiles(principal),
|
||||||
|
)
|
||||||
|
return _connector_credential_response(row)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch(
|
||||||
|
"/connectors/credentials/{credential_id}",
|
||||||
|
response_model=FileConnectorCredentialResponse,
|
||||||
|
)
|
||||||
|
def update_connector_credential(
|
||||||
|
credential_id: str,
|
||||||
|
payload: FileConnectorCredentialUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
row = get_connector_credential_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
credential_id=credential_id,
|
||||||
|
include_disabled=True,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
_require_connector_credential_write(principal, row.scope_type)
|
||||||
|
credentials = payload.credentials
|
||||||
|
try:
|
||||||
|
provider = payload.provider if payload.provider is not None else row.provider
|
||||||
|
_ensure_connector_credential_configuration_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
credential_id=row.id,
|
||||||
|
provider=provider,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
operation="configure_credentials",
|
||||||
|
)
|
||||||
|
if payload.policy is not None:
|
||||||
|
_ensure_connector_local_policy_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
policy=payload.policy,
|
||||||
|
)
|
||||||
|
update_connector_credential_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
label=payload.label,
|
||||||
|
provider=payload.provider,
|
||||||
|
enabled=payload.enabled,
|
||||||
|
credential_mode=payload.credential_mode,
|
||||||
|
username=credentials.username if credentials else None,
|
||||||
|
password=credentials.password if credentials else None,
|
||||||
|
token=credentials.token if credentials else None,
|
||||||
|
password_env=credentials.password_env if credentials else None,
|
||||||
|
token_env=credentials.token_env if credentials else None,
|
||||||
|
secret_ref=credentials.secret_ref if credentials else None,
|
||||||
|
clear_password=payload.clear_password,
|
||||||
|
clear_token=payload.clear_token,
|
||||||
|
policy=payload.policy,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||||
|
resource_id=row.id,
|
||||||
|
operation="updated",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(row)
|
||||||
|
return _connector_credential_response(row)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete(
|
||||||
|
"/connectors/credentials/{credential_id}",
|
||||||
|
response_model=FileConnectorCredentialResponse,
|
||||||
|
)
|
||||||
|
def deactivate_connector_credential(
|
||||||
|
credential_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
row = get_connector_credential_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
credential_id=credential_id,
|
||||||
|
include_disabled=True,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
_require_connector_credential_write(principal, row.scope_type)
|
||||||
|
try:
|
||||||
|
deletion = delete_connector_credential_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason="api_delete",
|
||||||
|
user_id=principal.user.id,
|
||||||
|
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||||
|
)
|
||||||
|
if deletion.changed:
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||||
|
resource_id=row.id,
|
||||||
|
operation="deleted",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for profile in deletion.affected_profiles:
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
resource_id=profile.id,
|
||||||
|
operation="updated",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=profile.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": profile.scope_type,
|
||||||
|
"scope_id": profile.scope_id,
|
||||||
|
"provider": profile.provider,
|
||||||
|
"reason": "credential_deleted",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(row)
|
||||||
|
return _connector_credential_response(row)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
except Exception:
|
||||||
|
session.rollback()
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
|
||||||
|
def list_connector_profiles(
|
||||||
|
provider: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:read",
|
||||||
|
"files:file:upload",
|
||||||
|
"files:file:download",
|
||||||
|
"files:file:admin",
|
||||||
|
"system:settings:read",
|
||||||
|
"admin:settings:read",
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
if campaign_id:
|
||||||
|
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||||
|
profiles = _visible_connector_profiles(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled
|
||||||
|
and _can_read_disabled_connector_profiles(principal),
|
||||||
|
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||||
|
include_effective_policy=False,
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return FileConnectorProfilesResponse(
|
||||||
|
profiles=[
|
||||||
|
FileConnectorProfileResponse(**profile.to_response())
|
||||||
|
for profile in profiles
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connectors/profiles",
|
||||||
|
response_model=FileConnectorProfileResponse,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def create_connector_profile(
|
||||||
|
payload: FileConnectorProfileCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(
|
||||||
|
require_any_scope(
|
||||||
|
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
):
|
||||||
|
_require_connector_profile_write(principal, payload.scope_type)
|
||||||
|
credentials = payload.credentials
|
||||||
|
try:
|
||||||
|
credential_row = _credential_row_for_profile(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
credential_profile_id=payload.credential_profile_id,
|
||||||
|
provider=payload.provider,
|
||||||
|
profile_id=payload.id,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
)
|
||||||
|
_ensure_connector_configuration_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
connector_id=payload.id,
|
||||||
|
credential_id=payload.credential_profile_id,
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=connector_effective_endpoint_url(
|
||||||
|
provider=payload.provider,
|
||||||
|
endpoint_url=payload.endpoint_url,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
),
|
||||||
|
base_path=payload.base_path,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
operation="configure",
|
||||||
|
)
|
||||||
|
_ensure_connector_local_policy_allowed(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
policy=payload.policy,
|
||||||
|
)
|
||||||
|
row = create_connector_profile_row(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
profile_id=payload.id,
|
||||||
|
label=payload.label,
|
||||||
|
provider=payload.provider,
|
||||||
|
scope_type=payload.scope_type,
|
||||||
|
scope_id=payload.scope_id,
|
||||||
|
endpoint_url=payload.endpoint_url,
|
||||||
|
base_path=payload.base_path,
|
||||||
|
enabled=payload.enabled,
|
||||||
|
credential_profile_id=payload.credential_profile_id,
|
||||||
|
credential_mode=payload.credential_mode,
|
||||||
|
username=credentials.username,
|
||||||
|
password=credentials.password,
|
||||||
|
token=credentials.token,
|
||||||
|
password_env=credentials.password_env,
|
||||||
|
token_env=credentials.token_env,
|
||||||
|
secret_ref=credentials.secret_ref,
|
||||||
|
capabilities=payload.capabilities,
|
||||||
|
policy=payload.policy,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
resource_id=row.id,
|
||||||
|
operation="created",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
payload={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(row)
|
||||||
|
return FileConnectorProfileResponse(
|
||||||
|
**connector_profile_from_row(
|
||||||
|
row, credential_row=credential_row
|
||||||
|
).to_response()
|
||||||
|
)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
@@ -0,0 +1,151 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Literal
|
||||||
|
from fastapi import APIRouter, Depends, Query
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileFolderCreateRequest,
|
||||||
|
FileFolderDeleteRequest,
|
||||||
|
FileFolderDeleteResponse,
|
||||||
|
FileFolderResponse,
|
||||||
|
FileFoldersResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.paths import UnsafeFilePathError
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.folders import (
|
||||||
|
create_folder,
|
||||||
|
list_folders_for_user,
|
||||||
|
list_folders_for_user_window,
|
||||||
|
soft_delete_folder,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_folder_response,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.services.list_queries import (
|
||||||
|
FOLDERS_LIST_CURSOR_SCOPE,
|
||||||
|
_cursor_page_size,
|
||||||
|
_files_delta_watermark,
|
||||||
|
_folder_cursor_values,
|
||||||
|
_folders_list_fingerprint,
|
||||||
|
_next_folder_list_cursor,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/folders", response_model=FileFoldersResponse)
|
||||||
|
def list_file_folders(
|
||||||
|
owner_type: Literal["user", "group"],
|
||||||
|
owner_id: str,
|
||||||
|
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||||
|
cursor: str | None = None,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
watermark = _files_delta_watermark(session, principal.tenant_id)
|
||||||
|
effective_page_size = _cursor_page_size(
|
||||||
|
FOLDERS_LIST_CURSOR_SCOPE, cursor, page_size
|
||||||
|
)
|
||||||
|
if effective_page_size is None:
|
||||||
|
folders = list_folders_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
return FileFoldersResponse(
|
||||||
|
folders=[_folder_response(folder) for folder in folders],
|
||||||
|
watermark=watermark,
|
||||||
|
)
|
||||||
|
fingerprint = _folders_list_fingerprint(
|
||||||
|
principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
page_size=effective_page_size,
|
||||||
|
)
|
||||||
|
after_path, after_id = _folder_cursor_values(cursor, fingerprint=fingerprint)
|
||||||
|
folders, has_more = list_folders_for_user_window(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
page_size=effective_page_size,
|
||||||
|
after_path=after_path,
|
||||||
|
after_id=after_id,
|
||||||
|
)
|
||||||
|
return FileFoldersResponse(
|
||||||
|
folders=[_folder_response(folder) for folder in folders],
|
||||||
|
cursor=cursor,
|
||||||
|
next_cursor=_next_folder_list_cursor(
|
||||||
|
principal,
|
||||||
|
folders,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
page_size=effective_page_size,
|
||||||
|
has_more=has_more,
|
||||||
|
),
|
||||||
|
watermark=watermark,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/folders", response_model=FileFolderResponse)
|
||||||
|
def create_file_folder(
|
||||||
|
payload: FileFolderCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
folder = create_folder(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=payload.owner_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
path=payload.path,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _folder_response(folder)
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
|
||||||
|
def delete_file_folder(
|
||||||
|
payload: FileFolderDeleteRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
deleted_folders, deleted_files = soft_delete_folder(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=payload.owner_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
path=payload.path,
|
||||||
|
recursive=payload.recursive,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return FileFolderDeleteResponse(
|
||||||
|
deleted_folders=deleted_folders, deleted_files=deleted_files
|
||||||
|
)
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
from fastapi import (
|
||||||
|
APIRouter,
|
||||||
|
Depends,
|
||||||
|
File as FastAPIFile,
|
||||||
|
Header,
|
||||||
|
HTTPException,
|
||||||
|
UploadFile,
|
||||||
|
status,
|
||||||
|
)
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.events import (
|
||||||
|
EventActorRef,
|
||||||
|
EventObjectRef,
|
||||||
|
EventTenantRef,
|
||||||
|
PlatformEvent,
|
||||||
|
emit_platform_event,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.institutional import EvidenceReference
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.db.models import FileFormEvidenceGrant
|
||||||
|
from govoplan_files.backend.route_support import _read_limited_upload
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.files import create_file_asset
|
||||||
|
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/form-evidence/upload",
|
||||||
|
response_model=dict[str, object],
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def upload_form_evidence(
|
||||||
|
file: UploadFile = FastAPIFile(...),
|
||||||
|
x_form_evidence_token: str = Header(alias="X-Form-Evidence-Token"),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
) -> dict[str, object]:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
try:
|
||||||
|
grant = (
|
||||||
|
session.query(FileFormEvidenceGrant)
|
||||||
|
.filter(
|
||||||
|
FileFormEvidenceGrant.token_sha256
|
||||||
|
== _token_sha256(x_form_evidence_token)
|
||||||
|
)
|
||||||
|
.with_for_update()
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if grant is None or grant.status != "issued" or _aware(grant.expires_at) <= now:
|
||||||
|
raise _upload_unavailable()
|
||||||
|
content_type = (
|
||||||
|
str(file.content_type or "application/octet-stream")
|
||||||
|
.split(";", 1)[0]
|
||||||
|
.strip()
|
||||||
|
.casefold()
|
||||||
|
)
|
||||||
|
if grant.allowed_content_types and content_type not in set(
|
||||||
|
grant.allowed_content_types
|
||||||
|
):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
|
||||||
|
detail="This Form does not accept the uploaded document type.",
|
||||||
|
)
|
||||||
|
data = _read_limited_upload(file, max_bytes=grant.max_size_bytes)
|
||||||
|
stored = create_file_asset(
|
||||||
|
session,
|
||||||
|
tenant_id=grant.tenant_id,
|
||||||
|
owner_type="user",
|
||||||
|
owner_id=grant.custodian_user_id,
|
||||||
|
user_id=grant.custodian_user_id,
|
||||||
|
filename=file.filename or "form-attachment",
|
||||||
|
data=data,
|
||||||
|
folder=f"Form submissions/{grant.form_instance_id}",
|
||||||
|
content_type=content_type,
|
||||||
|
description="Managed attachment captured through Forms Runtime.",
|
||||||
|
metadata={
|
||||||
|
"form_evidence": {
|
||||||
|
"grant_id": grant.id,
|
||||||
|
"form_instance_id": grant.form_instance_id,
|
||||||
|
"form_definition_id": grant.form_definition_id,
|
||||||
|
"form_definition_revision": grant.form_definition_revision,
|
||||||
|
"purpose": grant.purpose,
|
||||||
|
},
|
||||||
|
"source_provenance": {
|
||||||
|
"source_type": "form_evidence",
|
||||||
|
"connector_id": "forms_runtime.evidence.files",
|
||||||
|
"provider": "forms_runtime",
|
||||||
|
"external_id": grant.id,
|
||||||
|
"revision": grant.form_definition_revision,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
conflict_strategy="rename",
|
||||||
|
is_admin=True,
|
||||||
|
)
|
||||||
|
grant.status = "uploaded"
|
||||||
|
grant.file_asset_id = stored.asset.id
|
||||||
|
grant.file_version_id = stored.version.id
|
||||||
|
grant.uploaded_at = now
|
||||||
|
session.add(grant)
|
||||||
|
evidence = EvidenceReference(
|
||||||
|
kind="document",
|
||||||
|
owner_module="files",
|
||||||
|
evidence_id=stored.asset.id,
|
||||||
|
tenant_id=grant.tenant_id,
|
||||||
|
version=stored.version.id,
|
||||||
|
checksum=stored.version.checksum_sha256,
|
||||||
|
source_ref=(f"form_submission:{grant.form_instance_id}:grant:{grant.id}"),
|
||||||
|
responsible_actor_ref=f"user:{grant.custodian_user_id}",
|
||||||
|
captured_at=now,
|
||||||
|
)
|
||||||
|
emit_platform_event(
|
||||||
|
session,
|
||||||
|
PlatformEvent(
|
||||||
|
type="files.form_evidence.uploaded",
|
||||||
|
module_id="files",
|
||||||
|
payload={
|
||||||
|
"grant_id": grant.id,
|
||||||
|
"form_instance_id": grant.form_instance_id,
|
||||||
|
"file_asset_id": stored.asset.id,
|
||||||
|
"file_version_id": stored.version.id,
|
||||||
|
"checksum_sha256": stored.version.checksum_sha256,
|
||||||
|
"size_bytes": stored.version.size_bytes,
|
||||||
|
},
|
||||||
|
occurred_at=now,
|
||||||
|
actor=EventActorRef(type="user", id=grant.custodian_user_id),
|
||||||
|
tenant=EventTenantRef(id=grant.tenant_id),
|
||||||
|
resource=EventObjectRef(type="file", id=stored.asset.id),
|
||||||
|
classification="confidential",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
except HTTPException:
|
||||||
|
session.rollback()
|
||||||
|
raise
|
||||||
|
except (FileStorageError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||||
|
detail=str(exc),
|
||||||
|
) from exc
|
||||||
|
return {
|
||||||
|
"grant_id": grant.id,
|
||||||
|
"evidence": evidence.to_dict(include_inspection=True),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _token_sha256(token: str) -> str:
|
||||||
|
clean = str(token or "").strip()
|
||||||
|
if len(clean) < 32:
|
||||||
|
raise _upload_unavailable()
|
||||||
|
return hashlib.sha256(clean.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _upload_unavailable() -> HTTPException:
|
||||||
|
return HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="The Form evidence upload is unavailable.",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _aware(value: datetime) -> datetime:
|
||||||
|
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["router", "upload_form_evidence"]
|
||||||
@@ -0,0 +1,387 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_core.audit.logging import audit_from_principal
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
FileIntegrityFinding,
|
||||||
|
FileIntegrityScan,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileIntegrityActionRequest,
|
||||||
|
FileIntegrityActionResponse,
|
||||||
|
FileIntegrityFindingResponse,
|
||||||
|
FileIntegrityFindingsResponse,
|
||||||
|
FileIntegrityScanCreateRequest,
|
||||||
|
FileIntegrityScanRunRequest,
|
||||||
|
FileIntegrityScanResponse,
|
||||||
|
FileIntegrityScansResponse,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.backends import StorageBackendError
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.integrity import (
|
||||||
|
cleanup_orphan_finding,
|
||||||
|
create_integrity_scan,
|
||||||
|
mark_integrity_scan_failed,
|
||||||
|
recheck_integrity_finding,
|
||||||
|
run_integrity_scan_batch,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files/integrity", tags=["files-integrity"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/scans", response_model=FileIntegrityScansResponse)
|
||||||
|
def list_integrity_scans(
|
||||||
|
limit: int = Query(default=50, ge=1, le=200),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||||
|
) -> FileIntegrityScansResponse:
|
||||||
|
rows = (
|
||||||
|
session.query(FileIntegrityScan)
|
||||||
|
.filter(FileIntegrityScan.tenant_id == principal.tenant_id)
|
||||||
|
.order_by(FileIntegrityScan.created_at.desc(), FileIntegrityScan.id.desc())
|
||||||
|
.limit(limit)
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
return FileIntegrityScansResponse(
|
||||||
|
scans=[_scan_response(row) for row in rows]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/scans",
|
||||||
|
response_model=FileIntegrityScanResponse,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def create_scan(
|
||||||
|
payload: FileIntegrityScanCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||||
|
) -> FileIntegrityScanResponse:
|
||||||
|
try:
|
||||||
|
scan = create_integrity_scan(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
verify_checksums=payload.verify_checksums,
|
||||||
|
batch_size=payload.batch_size,
|
||||||
|
)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.integrity.scan_created",
|
||||||
|
object_type="file_integrity_scan",
|
||||||
|
object_id=scan.id,
|
||||||
|
details={
|
||||||
|
"storage_backend": scan.storage_backend,
|
||||||
|
"verify_checksums": scan.verify_checksums,
|
||||||
|
"batch_size": scan.batch_size,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _scan_response(scan)
|
||||||
|
except (FileStorageError, StorageBackendError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _integrity_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/scans/{scan_id}/run", response_model=FileIntegrityScanResponse)
|
||||||
|
def run_scan_batch(
|
||||||
|
scan_id: str,
|
||||||
|
payload: FileIntegrityScanRunRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||||
|
) -> FileIntegrityScanResponse:
|
||||||
|
scan = _scan_for_tenant(
|
||||||
|
session,
|
||||||
|
scan_id,
|
||||||
|
principal.tenant_id,
|
||||||
|
for_update=True,
|
||||||
|
)
|
||||||
|
_assert_expected_revision(scan.revision, payload.expected_revision)
|
||||||
|
previous_status = scan.status
|
||||||
|
try:
|
||||||
|
run_integrity_scan_batch(session, scan)
|
||||||
|
if scan.status == "completed" and previous_status != "completed":
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.integrity.scan_completed",
|
||||||
|
object_type="file_integrity_scan",
|
||||||
|
object_id=scan.id,
|
||||||
|
details={
|
||||||
|
"verified_blob_count": scan.verified_blob_count,
|
||||||
|
"quarantined_blob_count": scan.quarantined_blob_count,
|
||||||
|
"orphan_object_count": scan.orphan_object_count,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _scan_response(scan)
|
||||||
|
except (FileStorageError, StorageBackendError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
scan = _scan_for_tenant(
|
||||||
|
session,
|
||||||
|
scan_id,
|
||||||
|
principal.tenant_id,
|
||||||
|
for_update=True,
|
||||||
|
)
|
||||||
|
mark_integrity_scan_failed(scan, error=exc)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.integrity.scan_failed",
|
||||||
|
object_type="file_integrity_scan",
|
||||||
|
object_id=scan.id,
|
||||||
|
details={"error_type": type(exc).__name__},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
raise _integrity_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/scans/{scan_id}/findings",
|
||||||
|
response_model=FileIntegrityFindingsResponse,
|
||||||
|
)
|
||||||
|
def list_integrity_findings(
|
||||||
|
scan_id: str,
|
||||||
|
state_filter: str | None = Query(default=None, alias="state"),
|
||||||
|
limit: int = Query(default=500, ge=1, le=1000),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||||
|
) -> FileIntegrityFindingsResponse:
|
||||||
|
scan = _scan_for_tenant(session, scan_id, principal.tenant_id)
|
||||||
|
query = session.query(FileIntegrityFinding).filter(
|
||||||
|
FileIntegrityFinding.scan_id == scan.id,
|
||||||
|
FileIntegrityFinding.tenant_id == principal.tenant_id,
|
||||||
|
)
|
||||||
|
if state_filter:
|
||||||
|
query = query.filter(FileIntegrityFinding.state == state_filter)
|
||||||
|
rows = (
|
||||||
|
query.order_by(
|
||||||
|
FileIntegrityFinding.created_at.asc(),
|
||||||
|
FileIntegrityFinding.id.asc(),
|
||||||
|
)
|
||||||
|
.limit(limit)
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
return FileIntegrityFindingsResponse(
|
||||||
|
findings=[_finding_response(row) for row in rows]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/findings/{finding_id}/recheck",
|
||||||
|
response_model=FileIntegrityActionResponse,
|
||||||
|
)
|
||||||
|
def recheck_finding(
|
||||||
|
finding_id: str,
|
||||||
|
payload: FileIntegrityActionRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||||
|
) -> FileIntegrityActionResponse:
|
||||||
|
finding = _finding_for_tenant(
|
||||||
|
session,
|
||||||
|
finding_id,
|
||||||
|
principal.tenant_id,
|
||||||
|
for_update=True,
|
||||||
|
)
|
||||||
|
_assert_expected_revision(finding.revision, payload.expected_revision)
|
||||||
|
try:
|
||||||
|
result = recheck_integrity_finding(
|
||||||
|
session,
|
||||||
|
finding,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
dry_run=payload.dry_run,
|
||||||
|
)
|
||||||
|
_audit_integrity_action(session, principal, result)
|
||||||
|
session.commit()
|
||||||
|
return _action_response(result)
|
||||||
|
except (FileStorageError, StorageBackendError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _integrity_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/findings/{finding_id}/cleanup",
|
||||||
|
response_model=FileIntegrityActionResponse,
|
||||||
|
)
|
||||||
|
def cleanup_finding(
|
||||||
|
finding_id: str,
|
||||||
|
payload: FileIntegrityActionRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||||
|
) -> FileIntegrityActionResponse:
|
||||||
|
finding = _finding_for_tenant(
|
||||||
|
session,
|
||||||
|
finding_id,
|
||||||
|
principal.tenant_id,
|
||||||
|
for_update=True,
|
||||||
|
)
|
||||||
|
_assert_expected_revision(finding.revision, payload.expected_revision)
|
||||||
|
try:
|
||||||
|
result = cleanup_orphan_finding(
|
||||||
|
session,
|
||||||
|
finding,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
dry_run=payload.dry_run,
|
||||||
|
)
|
||||||
|
_audit_integrity_action(session, principal, result)
|
||||||
|
session.commit()
|
||||||
|
return _action_response(result)
|
||||||
|
except (FileStorageError, StorageBackendError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _integrity_http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _scan_for_tenant(
|
||||||
|
session: Session,
|
||||||
|
scan_id: str,
|
||||||
|
tenant_id: str,
|
||||||
|
*,
|
||||||
|
for_update: bool = False,
|
||||||
|
) -> FileIntegrityScan:
|
||||||
|
query = session.query(FileIntegrityScan).filter(FileIntegrityScan.id == scan_id)
|
||||||
|
if for_update:
|
||||||
|
query = query.populate_existing().with_for_update()
|
||||||
|
scan = query.one_or_none()
|
||||||
|
if scan is None or scan.tenant_id != tenant_id:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="Integrity scan not found",
|
||||||
|
)
|
||||||
|
return scan
|
||||||
|
|
||||||
|
|
||||||
|
def _finding_for_tenant(
|
||||||
|
session: Session,
|
||||||
|
finding_id: str,
|
||||||
|
tenant_id: str,
|
||||||
|
*,
|
||||||
|
for_update: bool = False,
|
||||||
|
) -> FileIntegrityFinding:
|
||||||
|
query = session.query(FileIntegrityFinding).filter(
|
||||||
|
FileIntegrityFinding.id == finding_id
|
||||||
|
)
|
||||||
|
if for_update:
|
||||||
|
query = query.populate_existing().with_for_update()
|
||||||
|
finding = query.one_or_none()
|
||||||
|
if finding is None or finding.tenant_id != tenant_id:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="Integrity finding not found",
|
||||||
|
)
|
||||||
|
return finding
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_expected_revision(current: int, expected: int) -> None:
|
||||||
|
if current != expected:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_409_CONFLICT,
|
||||||
|
detail=(
|
||||||
|
"The integrity record changed after it was loaded; reload before "
|
||||||
|
"performing this action."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_integrity_action(session, principal, result) -> None:
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action=f"files.integrity.{result.action}",
|
||||||
|
object_type="file_integrity_finding",
|
||||||
|
object_id=result.finding.id,
|
||||||
|
details={
|
||||||
|
"scan_id": result.finding.scan_id,
|
||||||
|
"kind": result.finding.kind,
|
||||||
|
"dry_run": result.dry_run,
|
||||||
|
"changed": result.changed,
|
||||||
|
"storage_key_sha256": hashlib.sha256(
|
||||||
|
result.finding.storage_key.encode("utf-8")
|
||||||
|
).hexdigest(),
|
||||||
|
"inspection": result.inspection.kind
|
||||||
|
if result.inspection
|
||||||
|
else None,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _scan_response(scan: FileIntegrityScan) -> FileIntegrityScanResponse:
|
||||||
|
return FileIntegrityScanResponse(
|
||||||
|
id=scan.id,
|
||||||
|
tenant_id=scan.tenant_id,
|
||||||
|
storage_backend=scan.storage_backend,
|
||||||
|
storage_prefix=scan.storage_prefix,
|
||||||
|
status=scan.status,
|
||||||
|
revision=scan.revision,
|
||||||
|
phase=scan.phase,
|
||||||
|
verify_checksums=scan.verify_checksums,
|
||||||
|
batch_size=scan.batch_size,
|
||||||
|
scanned_blob_count=scan.scanned_blob_count,
|
||||||
|
verified_blob_count=scan.verified_blob_count,
|
||||||
|
quarantined_blob_count=scan.quarantined_blob_count,
|
||||||
|
scanned_object_count=scan.scanned_object_count,
|
||||||
|
orphan_object_count=scan.orphan_object_count,
|
||||||
|
created_by_user_id=scan.created_by_user_id,
|
||||||
|
started_at=scan.started_at.isoformat() if scan.started_at else None,
|
||||||
|
completed_at=scan.completed_at.isoformat()
|
||||||
|
if scan.completed_at
|
||||||
|
else None,
|
||||||
|
last_error=scan.last_error,
|
||||||
|
created_at=scan.created_at.isoformat(),
|
||||||
|
updated_at=scan.updated_at.isoformat(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _finding_response(
|
||||||
|
finding: FileIntegrityFinding,
|
||||||
|
) -> FileIntegrityFindingResponse:
|
||||||
|
return FileIntegrityFindingResponse(
|
||||||
|
id=finding.id,
|
||||||
|
scan_id=finding.scan_id,
|
||||||
|
tenant_id=finding.tenant_id,
|
||||||
|
kind=finding.kind,
|
||||||
|
state=finding.state,
|
||||||
|
revision=finding.revision,
|
||||||
|
blob_id=finding.blob_id,
|
||||||
|
storage_key=finding.storage_key,
|
||||||
|
expected_size_bytes=finding.expected_size_bytes,
|
||||||
|
observed_size_bytes=finding.observed_size_bytes,
|
||||||
|
expected_checksum_sha256=finding.expected_checksum_sha256,
|
||||||
|
observed_checksum_sha256=finding.observed_checksum_sha256,
|
||||||
|
resolved_at=finding.resolved_at.isoformat()
|
||||||
|
if finding.resolved_at
|
||||||
|
else None,
|
||||||
|
resolved_by_user_id=finding.resolved_by_user_id,
|
||||||
|
created_at=finding.created_at.isoformat(),
|
||||||
|
updated_at=finding.updated_at.isoformat(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _action_response(result) -> FileIntegrityActionResponse:
|
||||||
|
return FileIntegrityActionResponse(
|
||||||
|
action=result.action,
|
||||||
|
changed=result.changed,
|
||||||
|
dry_run=result.dry_run,
|
||||||
|
finding=_finding_response(result.finding),
|
||||||
|
inspection_kind=result.inspection.kind if result.inspection else None,
|
||||||
|
inspection_valid=result.inspection.valid if result.inspection else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _integrity_http_error(exc: Exception) -> HTTPException:
|
||||||
|
if isinstance(exc, FileStorageError):
|
||||||
|
return HTTPException(
|
||||||
|
status_code=status.HTTP_409_CONFLICT,
|
||||||
|
detail=str(exc),
|
||||||
|
)
|
||||||
|
return HTTPException(
|
||||||
|
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||||
|
detail="The configured file storage backend could not complete the integrity operation",
|
||||||
|
)
|
||||||
@@ -0,0 +1,297 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_core.audit.logging import audit_from_principal
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_response,
|
||||||
|
_connector_space_response,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileAssetResponse,
|
||||||
|
FileBlobGcRequest,
|
||||||
|
FileBlobGcResponse,
|
||||||
|
FileConnectorSpaceResponse,
|
||||||
|
FileFolderDeleteRequest,
|
||||||
|
FileFolderRestoreResponse,
|
||||||
|
FileLifecycleUpdateRequest,
|
||||||
|
FilePurgeExecuteRequest,
|
||||||
|
FilePurgePreviewItem,
|
||||||
|
FilePurgePreviewRequest,
|
||||||
|
FilePurgePreviewResponse,
|
||||||
|
FilePurgeResponse,
|
||||||
|
FileRestoreResponse,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.lifecycle import (
|
||||||
|
execute_asset_purge,
|
||||||
|
garbage_collect_unreferenced_blobs,
|
||||||
|
get_asset_for_lifecycle,
|
||||||
|
preview_asset_purge,
|
||||||
|
restore_asset,
|
||||||
|
restore_connector_space,
|
||||||
|
restore_folder,
|
||||||
|
set_asset_lifecycle,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files-lifecycle"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/{file_id}/lifecycle", response_model=FileAssetResponse)
|
||||||
|
def update_file_lifecycle(
|
||||||
|
file_id: str,
|
||||||
|
payload: FileLifecycleUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:retention")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_lifecycle(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
set_asset_lifecycle(
|
||||||
|
session,
|
||||||
|
asset,
|
||||||
|
retained_until=payload.retained_until,
|
||||||
|
legal_hold=payload.legal_hold,
|
||||||
|
reason=payload.reason,
|
||||||
|
expected_revision=payload.expected_revision,
|
||||||
|
)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.lifecycle.updated",
|
||||||
|
object_type="file",
|
||||||
|
object_id=asset.id,
|
||||||
|
details={
|
||||||
|
"retained_until": (
|
||||||
|
asset.retained_until.isoformat() if asset.retained_until else None
|
||||||
|
),
|
||||||
|
"legal_hold": asset.legal_hold,
|
||||||
|
"lifecycle_revision": asset.lifecycle_revision,
|
||||||
|
"reason": asset.lifecycle_reason,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _asset_response(session, asset, include_shares=True)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/assets/{file_id}/restore", response_model=FileRestoreResponse)
|
||||||
|
def restore_file(
|
||||||
|
file_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_lifecycle(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
changed = restore_asset(session, asset)
|
||||||
|
if changed:
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.file.restored",
|
||||||
|
object_type="file",
|
||||||
|
object_id=asset.id,
|
||||||
|
details={"display_path": asset.display_path},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return FileRestoreResponse(restored_count=1 if changed else 0)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/folders/restore", response_model=FileFolderRestoreResponse)
|
||||||
|
def restore_file_folder(
|
||||||
|
payload: FileFolderDeleteRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
restored_folders, restored_files = restore_folder(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=payload.owner_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
path=payload.path,
|
||||||
|
recursive=payload.recursive,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.folder.restored",
|
||||||
|
object_type="file_folder",
|
||||||
|
object_id=payload.path,
|
||||||
|
details={
|
||||||
|
"restored_folders": restored_folders,
|
||||||
|
"restored_files": restored_files,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return FileFolderRestoreResponse(
|
||||||
|
restored_folders=restored_folders, restored_files=restored_files
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connector-spaces/{space_id}/restore",
|
||||||
|
response_model=FileConnectorSpaceResponse,
|
||||||
|
)
|
||||||
|
def restore_file_connector_space(
|
||||||
|
space_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
space = restore_connector_space(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
space_id=space_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.connector_space.restored",
|
||||||
|
object_type="file_connector_space",
|
||||||
|
object_id=space.id,
|
||||||
|
details={"label": space.label},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _connector_space_response(space)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/purge/preview", response_model=FilePurgePreviewResponse)
|
||||||
|
def preview_file_purge(
|
||||||
|
payload: FilePurgePreviewRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
preview = preview_asset_purge(
|
||||||
|
session, tenant_id=principal.tenant_id, file_ids=payload.file_ids
|
||||||
|
)
|
||||||
|
return FilePurgePreviewResponse(
|
||||||
|
preview_sha256=preview.preview_sha256,
|
||||||
|
eligible=preview.eligible,
|
||||||
|
items=[
|
||||||
|
FilePurgePreviewItem(
|
||||||
|
file_id=item.file_id,
|
||||||
|
filename=item.filename,
|
||||||
|
lifecycle_revision=item.lifecycle_revision,
|
||||||
|
deleted_at=(item.deleted_at.isoformat() if item.deleted_at else None),
|
||||||
|
retained_until=(
|
||||||
|
item.retained_until.isoformat() if item.retained_until else None
|
||||||
|
),
|
||||||
|
legal_hold=item.legal_hold,
|
||||||
|
blockers=list(item.blockers),
|
||||||
|
blob_ids=list(item.blob_ids),
|
||||||
|
)
|
||||||
|
for item in preview.items
|
||||||
|
],
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/purge/execute", response_model=FilePurgeResponse)
|
||||||
|
def execute_file_purge(
|
||||||
|
payload: FilePurgeExecuteRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
result = execute_asset_purge(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
file_ids=payload.file_ids,
|
||||||
|
preview_sha256=payload.preview_sha256,
|
||||||
|
idempotency_key=payload.idempotency_key,
|
||||||
|
approval_reference=payload.approval_reference,
|
||||||
|
before_commit=lambda assets: audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.files.purged",
|
||||||
|
object_type="file_batch",
|
||||||
|
object_id=payload.preview_sha256,
|
||||||
|
details={
|
||||||
|
"file_ids": [asset.id for asset in assets],
|
||||||
|
"approval_reference": payload.approval_reference,
|
||||||
|
"preview_sha256": payload.preview_sha256,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return FilePurgeResponse(
|
||||||
|
recovery_operation_id=result.recovery_operation_id,
|
||||||
|
status=result.status,
|
||||||
|
replayed=result.replayed,
|
||||||
|
purged_files=result.purged_files,
|
||||||
|
released_blobs=result.released_blobs,
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/purge/blobs", response_model=FileBlobGcResponse)
|
||||||
|
def garbage_collect_file_blobs(
|
||||||
|
payload: FileBlobGcRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
result = garbage_collect_unreferenced_blobs(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
limit=payload.limit,
|
||||||
|
approval_reference=payload.approval_reference,
|
||||||
|
)
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="files.blobs.garbage_collected",
|
||||||
|
object_type="file_blob_batch",
|
||||||
|
object_id=principal.tenant_id,
|
||||||
|
details={
|
||||||
|
"inspected_blobs": result.inspected_blobs,
|
||||||
|
"deleted_blobs": result.deleted_blobs,
|
||||||
|
"unresolved_operation_ids": list(result.unresolved_operation_ids),
|
||||||
|
"approval_reference": payload.approval_reference,
|
||||||
|
},
|
||||||
|
commit=True,
|
||||||
|
)
|
||||||
|
return FileBlobGcResponse(
|
||||||
|
inspected_blobs=result.inspected_blobs,
|
||||||
|
deleted_blobs=result.deleted_blobs,
|
||||||
|
unresolved_operation_ids=list(result.unresolved_operation_ids),
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Literal
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileDeltaResponse,
|
||||||
|
FileListResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
count_assets_for_user,
|
||||||
|
list_assets_for_user,
|
||||||
|
list_assets_for_user_window,
|
||||||
|
list_recent_assets_for_user,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_list_response,
|
||||||
|
_ensure_campaign_file_access,
|
||||||
|
_ensure_list_owner_access,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.services.list_queries import (
|
||||||
|
FILES_LIST_CURSOR_SCOPE,
|
||||||
|
_cursor_page_size,
|
||||||
|
_file_cursor_values,
|
||||||
|
_files_delta_response,
|
||||||
|
_files_delta_watermark,
|
||||||
|
_files_list_fingerprint,
|
||||||
|
_full_file_delta_response,
|
||||||
|
_next_file_list_cursor,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/delta", response_model=FileDeltaResponse)
|
||||||
|
def files_delta(
|
||||||
|
owner_type: Literal["user", "group"] | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
path_prefix: str | None = None,
|
||||||
|
since: str | None = None,
|
||||||
|
limit: int = Query(default=500, ge=1, le=1000),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||||
|
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||||
|
if since is None:
|
||||||
|
return _full_file_delta_response(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
)
|
||||||
|
return _files_delta_response(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
since=since,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=FileListResponse)
|
||||||
|
def list_files(
|
||||||
|
owner_type: Literal["user", "group"] | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
path_prefix: str | None = None,
|
||||||
|
campaign_usage: Literal["linked", "unlinked"] | None = None,
|
||||||
|
audit_relevant: bool | None = None,
|
||||||
|
sort: Literal["path", "recent"] = "path",
|
||||||
|
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||||
|
cursor: str | None = None,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||||
|
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||||
|
watermark = _files_delta_watermark(session, principal.tenant_id)
|
||||||
|
total = count_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
if sort == "recent":
|
||||||
|
if cursor:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||||
|
detail="Recent file projections do not accept a path-order cursor.",
|
||||||
|
)
|
||||||
|
recent_limit = page_size or 25
|
||||||
|
assets = list_recent_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
limit=recent_limit,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
return FileListResponse(
|
||||||
|
files=_asset_list_response(session, assets, include_shares=True),
|
||||||
|
total=total,
|
||||||
|
watermark=watermark,
|
||||||
|
)
|
||||||
|
effective_page_size = _cursor_page_size(FILES_LIST_CURSOR_SCOPE, cursor, page_size)
|
||||||
|
if effective_page_size is not None:
|
||||||
|
fingerprint = _files_list_fingerprint(
|
||||||
|
principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
page_size=effective_page_size,
|
||||||
|
)
|
||||||
|
after_display_path, after_updated_at, after_id = _file_cursor_values(
|
||||||
|
cursor, fingerprint=fingerprint
|
||||||
|
)
|
||||||
|
assets, has_more = list_assets_for_user_window(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
page_size=effective_page_size,
|
||||||
|
after_display_path=after_display_path,
|
||||||
|
after_updated_at=after_updated_at,
|
||||||
|
after_id=after_id,
|
||||||
|
)
|
||||||
|
return FileListResponse(
|
||||||
|
files=_asset_list_response(session, assets, include_shares=True),
|
||||||
|
total=total,
|
||||||
|
cursor=cursor,
|
||||||
|
next_cursor=_next_file_list_cursor(
|
||||||
|
principal,
|
||||||
|
assets,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
page_size=effective_page_size,
|
||||||
|
has_more=has_more,
|
||||||
|
),
|
||||||
|
watermark=watermark,
|
||||||
|
)
|
||||||
|
assets = list_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
return FileListResponse(
|
||||||
|
files=_asset_list_response(session, assets, include_shares=True),
|
||||||
|
total=total,
|
||||||
|
watermark=watermark,
|
||||||
|
)
|
||||||
@@ -0,0 +1,330 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.api.v1.schemas import (
|
||||||
|
ReferenceOptionListResponse,
|
||||||
|
ReferenceOptionResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_core.audit.logging import audit_from_principal
|
||||||
|
from govoplan_core.core.references import (
|
||||||
|
access_scope_reference_page,
|
||||||
|
access_scope_reference_provider_available,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.runtime import get_registry
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
BulkFileShareRequest,
|
||||||
|
BulkFileShareResponse,
|
||||||
|
FileShareRequest,
|
||||||
|
FileShareResponse,
|
||||||
|
FileSharesResponse,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
current_file_share_for_target,
|
||||||
|
get_asset_for_share_management,
|
||||||
|
list_file_shares,
|
||||||
|
revoke_file_share,
|
||||||
|
share_file,
|
||||||
|
share_files,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_file_share_response,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/{file_id}/share-target-options",
|
||||||
|
response_model=ReferenceOptionListResponse,
|
||||||
|
)
|
||||||
|
def search_share_targets(
|
||||||
|
file_id: str,
|
||||||
|
target_type: str,
|
||||||
|
q: str = "",
|
||||||
|
selected: list[str] = Query(default=[]),
|
||||||
|
limit: int = Query(default=50, ge=1, le=200),
|
||||||
|
cursor: str | None = None,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||||
|
) -> ReferenceOptionListResponse:
|
||||||
|
if target_type not in {"user", "group"}:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||||
|
detail="Share target type must be user or group",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
get_asset_for_share_management(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
registry = get_registry()
|
||||||
|
page = access_scope_reference_page(
|
||||||
|
registry,
|
||||||
|
principal,
|
||||||
|
scope_type=target_type,
|
||||||
|
reference_kind="membership" if target_type == "user" else "group",
|
||||||
|
query=q,
|
||||||
|
selected_values=selected,
|
||||||
|
limit=limit,
|
||||||
|
cursor=cursor,
|
||||||
|
administrative=True,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||||
|
detail=str(exc),
|
||||||
|
) from exc
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return ReferenceOptionListResponse(
|
||||||
|
options=[
|
||||||
|
ReferenceOptionResponse(**option.to_dict()) for option in page.options
|
||||||
|
],
|
||||||
|
provider_available=access_scope_reference_provider_available(registry),
|
||||||
|
next_cursor=page.next_cursor,
|
||||||
|
has_more=page.has_more,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{file_id}/shares", response_model=FileSharesResponse)
|
||||||
|
def list_shares(
|
||||||
|
file_id: str,
|
||||||
|
include_inactive: bool = False,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_share_management(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
return FileSharesResponse(
|
||||||
|
shares=[
|
||||||
|
_file_share_response(share)
|
||||||
|
for share in list_file_shares(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
asset_id=asset.id,
|
||||||
|
include_inactive=include_inactive,
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{file_id}/shares", response_model=FileShareResponse)
|
||||||
|
def create_share(
|
||||||
|
file_id: str,
|
||||||
|
payload: FileShareRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_share_management(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
previous = current_file_share_for_target(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
asset_id=asset.id,
|
||||||
|
target_type=payload.target_type,
|
||||||
|
target_id=payload.target_id,
|
||||||
|
)
|
||||||
|
previous_permission = previous.permission if previous else None
|
||||||
|
previous_expiry = previous.expires_at if previous else None
|
||||||
|
share = share_file(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
asset=asset,
|
||||||
|
target_type=payload.target_type,
|
||||||
|
target_id=payload.target_id,
|
||||||
|
permission=payload.permission,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
expires_at=payload.expires_at,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
action = _share_audit_action(
|
||||||
|
existed=previous is not None,
|
||||||
|
previous_permission=previous_permission,
|
||||||
|
permission=share.permission,
|
||||||
|
previous_expiry=previous_expiry,
|
||||||
|
expiry=share.expires_at,
|
||||||
|
)
|
||||||
|
if action:
|
||||||
|
_audit_share_change(session, principal, share, action=action)
|
||||||
|
session.commit()
|
||||||
|
return _file_share_response(share)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/bulk-shares", response_model=BulkFileShareResponse)
|
||||||
|
def create_bulk_shares(
|
||||||
|
payload: BulkFileShareRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
file_ids = list(dict.fromkeys(payload.file_ids))
|
||||||
|
assets = [
|
||||||
|
get_asset_for_share_management(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
for file_id in file_ids
|
||||||
|
]
|
||||||
|
previous_by_asset = {
|
||||||
|
asset.id: current_file_share_for_target(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
asset_id=asset.id,
|
||||||
|
target_type=payload.target_type,
|
||||||
|
target_id=payload.target_id,
|
||||||
|
)
|
||||||
|
for asset in assets
|
||||||
|
}
|
||||||
|
previous_values = {
|
||||||
|
asset_id: (
|
||||||
|
share.permission if share else None,
|
||||||
|
share.expires_at if share else None,
|
||||||
|
)
|
||||||
|
for asset_id, share in previous_by_asset.items()
|
||||||
|
}
|
||||||
|
shares = share_files(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
assets=assets,
|
||||||
|
target_type=payload.target_type,
|
||||||
|
target_id=payload.target_id,
|
||||||
|
permission=payload.permission,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
expires_at=payload.expires_at,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
for share in shares:
|
||||||
|
previous_permission, previous_expiry = previous_values[share.file_asset_id]
|
||||||
|
action = _share_audit_action(
|
||||||
|
existed=previous_by_asset[share.file_asset_id] is not None,
|
||||||
|
previous_permission=previous_permission,
|
||||||
|
permission=share.permission,
|
||||||
|
previous_expiry=previous_expiry,
|
||||||
|
expiry=share.expires_at,
|
||||||
|
)
|
||||||
|
if action:
|
||||||
|
_audit_share_change(session, principal, share, action=action)
|
||||||
|
session.commit()
|
||||||
|
return BulkFileShareResponse(
|
||||||
|
shared_count=len(shares),
|
||||||
|
shares=[_file_share_response(share) for share in shares],
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/{file_id}/shares/{share_id}", response_model=FileShareResponse)
|
||||||
|
def revoke_share(
|
||||||
|
file_id: str,
|
||||||
|
share_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
asset = get_asset_for_share_management(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
share, changed = revoke_file_share(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
asset_id=asset.id,
|
||||||
|
share_id=share_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
)
|
||||||
|
if changed:
|
||||||
|
_audit_share_change(
|
||||||
|
session, principal, share, action="files.share.revoked"
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _file_share_response(share)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _share_audit_action(
|
||||||
|
*,
|
||||||
|
existed: bool,
|
||||||
|
previous_permission: str | None,
|
||||||
|
permission: str,
|
||||||
|
previous_expiry: datetime | None,
|
||||||
|
expiry: datetime | None,
|
||||||
|
) -> str | None:
|
||||||
|
if not existed:
|
||||||
|
return "files.share.granted"
|
||||||
|
permission_changed = previous_permission != permission
|
||||||
|
expiry_changed = _normalized_expiry(previous_expiry) != _normalized_expiry(expiry)
|
||||||
|
if permission_changed:
|
||||||
|
return "files.share.changed"
|
||||||
|
if expiry_changed:
|
||||||
|
return "files.share.expiry_changed"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_share_change(
|
||||||
|
session: Session,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
share,
|
||||||
|
*,
|
||||||
|
action: str,
|
||||||
|
) -> None:
|
||||||
|
audit_from_principal(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action=action,
|
||||||
|
object_type="file_share",
|
||||||
|
object_id=share.id,
|
||||||
|
details={
|
||||||
|
"file_asset_id": share.file_asset_id,
|
||||||
|
"target_type": share.target_type,
|
||||||
|
"target_id": share.target_id,
|
||||||
|
"permission": share.permission,
|
||||||
|
"expires_at": _normalized_expiry(share.expires_at),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalized_expiry(value: datetime | None) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
if value.tzinfo is None:
|
||||||
|
value = value.replace(tzinfo=timezone.utc)
|
||||||
|
return value.astimezone(timezone.utc).isoformat()
|
||||||
@@ -0,0 +1,308 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Literal
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_files.backend.change_tracking import (
|
||||||
|
FILES_CONNECTOR_SPACES_COLLECTION,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileConnectorSpaceCreateRequest,
|
||||||
|
FileConnectorSpaceResponse,
|
||||||
|
FileConnectorSpacesResponse,
|
||||||
|
FileConnectorSpaceUpdateRequest,
|
||||||
|
FileSpaceResponse,
|
||||||
|
FileSpacesResponse,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.access import group_refs_for_ids, user_group_ids
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_spaces import (
|
||||||
|
connector_space_owner_id,
|
||||||
|
create_connector_space,
|
||||||
|
get_connector_space_for_user,
|
||||||
|
list_connector_spaces_for_user,
|
||||||
|
soft_delete_connector_space,
|
||||||
|
update_connector_space,
|
||||||
|
validate_connector_space_write_mode,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorPolicyDenied,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
FILES_CONNECTOR_SPACE_RESOURCE,
|
||||||
|
_connector_policy_error,
|
||||||
|
_connector_space_file_space_response,
|
||||||
|
_connector_space_policy_decision,
|
||||||
|
_connector_space_response,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
_record_connector_settings_change,
|
||||||
|
_visible_connector_profile,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/spaces", response_model=FileSpacesResponse)
|
||||||
|
def list_file_spaces(
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
spaces = [
|
||||||
|
FileSpaceResponse(
|
||||||
|
id=f"user:{principal.user.id}",
|
||||||
|
label="My files",
|
||||||
|
owner_type="user",
|
||||||
|
owner_id=principal.user.id,
|
||||||
|
description="Files owned by your user account.",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
group_ids = user_group_ids(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
include_admin_groups=_is_admin(principal),
|
||||||
|
)
|
||||||
|
if group_ids:
|
||||||
|
groups = group_refs_for_ids(tenant_id=principal.tenant_id, group_ids=group_ids)
|
||||||
|
spaces.extend(
|
||||||
|
FileSpaceResponse(
|
||||||
|
id=f"group:{group.id}",
|
||||||
|
label=f"{group.name} files",
|
||||||
|
owner_type="group",
|
||||||
|
owner_id=group.id,
|
||||||
|
description="Files owned by this group.",
|
||||||
|
)
|
||||||
|
for group in groups
|
||||||
|
)
|
||||||
|
connector_spaces = list_connector_spaces_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
spaces.extend(
|
||||||
|
_connector_space_file_space_response(space) for space in connector_spaces
|
||||||
|
)
|
||||||
|
return FileSpacesResponse(spaces=spaces)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/connector-spaces", response_model=FileConnectorSpacesResponse)
|
||||||
|
def list_file_connector_spaces(
|
||||||
|
owner_type: Literal["user", "group"] | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
include_inactive: bool = False,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
spaces = list_connector_spaces_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
include_inactive=include_inactive and _is_admin(principal),
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
return FileConnectorSpacesResponse(
|
||||||
|
spaces=[_connector_space_response(space) for space in spaces]
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/connector-spaces",
|
||||||
|
response_model=FileConnectorSpaceResponse,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def create_file_connector_space(
|
||||||
|
payload: FileConnectorSpaceCreateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||||
|
):
|
||||||
|
if payload.owner_type == "group" and not payload.owner_id:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="owner_id is required for group connector spaces",
|
||||||
|
)
|
||||||
|
target_owner = payload.owner_id or principal.user.id
|
||||||
|
try:
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session, principal, payload.connector_profile_id
|
||||||
|
)
|
||||||
|
decision = _connector_space_policy_decision(
|
||||||
|
profile,
|
||||||
|
library_id=payload.library_id,
|
||||||
|
remote_path=payload.remote_path,
|
||||||
|
operation="link",
|
||||||
|
)
|
||||||
|
if not decision.allowed:
|
||||||
|
raise ConnectorPolicyDenied(decision)
|
||||||
|
validate_connector_space_write_mode(profile, read_only=payload.read_only)
|
||||||
|
space = create_connector_space(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
label=payload.label,
|
||||||
|
profile=profile,
|
||||||
|
library_id=payload.library_id,
|
||||||
|
remote_path=payload.remote_path,
|
||||||
|
sync_mode=payload.sync_mode,
|
||||||
|
read_only=payload.read_only,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||||
|
resource_id=space.id,
|
||||||
|
operation="created",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=space.tenant_id,
|
||||||
|
payload={
|
||||||
|
"owner_type": space.owner_type,
|
||||||
|
"owner_id": connector_space_owner_id(space),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _connector_space_response(space)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse)
|
||||||
|
def update_file_connector_space(
|
||||||
|
space_id: str,
|
||||||
|
payload: FileConnectorSpaceUpdateRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
space = get_connector_space_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
space_id=space_id,
|
||||||
|
include_inactive=_is_admin(principal),
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
|
try:
|
||||||
|
if (
|
||||||
|
payload.library_id is not None
|
||||||
|
or payload.remote_path is not None
|
||||||
|
or payload.read_only is not None
|
||||||
|
):
|
||||||
|
profile = _visible_connector_profile(
|
||||||
|
session, principal, space.connector_profile_id
|
||||||
|
)
|
||||||
|
decision = _connector_space_policy_decision(
|
||||||
|
profile,
|
||||||
|
library_id=payload.library_id
|
||||||
|
if payload.library_id is not None
|
||||||
|
else space.library_id,
|
||||||
|
remote_path=payload.remote_path
|
||||||
|
if payload.remote_path is not None
|
||||||
|
else space.remote_path,
|
||||||
|
operation="link",
|
||||||
|
)
|
||||||
|
if not decision.allowed:
|
||||||
|
raise ConnectorPolicyDenied(decision)
|
||||||
|
validate_connector_space_write_mode(
|
||||||
|
profile,
|
||||||
|
read_only=(
|
||||||
|
payload.read_only
|
||||||
|
if payload.read_only is not None
|
||||||
|
else space.read_only
|
||||||
|
),
|
||||||
|
)
|
||||||
|
update_connector_space(
|
||||||
|
session,
|
||||||
|
space,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
label=payload.label,
|
||||||
|
library_id=payload.library_id,
|
||||||
|
remote_path=payload.remote_path,
|
||||||
|
sync_mode=payload.sync_mode,
|
||||||
|
read_only=payload.read_only,
|
||||||
|
is_active=payload.is_active,
|
||||||
|
metadata=payload.metadata,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||||
|
resource_id=space.id,
|
||||||
|
operation="updated",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=space.tenant_id,
|
||||||
|
payload={
|
||||||
|
"owner_type": space.owner_type,
|
||||||
|
"owner_id": connector_space_owner_id(space),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _connector_space_response(space)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete(
|
||||||
|
"/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse
|
||||||
|
)
|
||||||
|
def delete_file_connector_space(
|
||||||
|
space_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
space = get_connector_space_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
space_id=space_id,
|
||||||
|
include_inactive=True,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
soft_delete_connector_space(
|
||||||
|
session, space, user_id=principal.user.id, is_admin=_is_admin(principal)
|
||||||
|
)
|
||||||
|
_record_connector_settings_change(
|
||||||
|
session,
|
||||||
|
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||||
|
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||||
|
resource_id=space.id,
|
||||||
|
operation="deleted",
|
||||||
|
principal=principal,
|
||||||
|
tenant_id=space.tenant_id,
|
||||||
|
payload={
|
||||||
|
"owner_type": space.owner_type,
|
||||||
|
"owner_id": connector_space_owner_id(space),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _connector_space_response(space)
|
||||||
|
except FileStorageError as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc, not_found=True) from exc
|
||||||
@@ -0,0 +1,213 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import tempfile
|
||||||
|
from fastapi import APIRouter, Depends
|
||||||
|
from fastapi.responses import FileResponse
|
||||||
|
from starlette.background import BackgroundTask
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
ArchiveRequest,
|
||||||
|
PatternMatchResponse,
|
||||||
|
PatternResolveRequest,
|
||||||
|
PatternResolveResponse,
|
||||||
|
RenamePreviewItem,
|
||||||
|
RenameRequest,
|
||||||
|
RenameResponse,
|
||||||
|
TransferRequest,
|
||||||
|
TransferResponse,
|
||||||
|
_conflict_resolutions,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.storage.paths import (
|
||||||
|
UnsafeFilePathError,
|
||||||
|
filename_from_path,
|
||||||
|
normalize_logical_path,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.archives import create_zip_file
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
get_asset_for_user,
|
||||||
|
list_assets_for_user,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.search import resolve_patterns
|
||||||
|
from govoplan_files.backend.storage.transfers import (
|
||||||
|
rename_selection,
|
||||||
|
transfer_selection,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_response,
|
||||||
|
_attachment_disposition,
|
||||||
|
_audit_connector_access,
|
||||||
|
_cleanup_temp_file,
|
||||||
|
_ensure_campaign_file_access,
|
||||||
|
_ensure_list_owner_access,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/bulk-rename", response_model=RenameResponse)
|
||||||
|
def bulk_rename(
|
||||||
|
payload: RenameRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
plan = rename_selection(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
file_ids=payload.file_ids,
|
||||||
|
folder_paths=payload.folder_paths,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=payload.owner_id,
|
||||||
|
mode=payload.mode,
|
||||||
|
new_name=payload.new_name,
|
||||||
|
find=payload.find,
|
||||||
|
replacement=payload.replacement,
|
||||||
|
prefix=payload.prefix,
|
||||||
|
suffix=payload.suffix,
|
||||||
|
recursive=payload.recursive,
|
||||||
|
dry_run=payload.dry_run,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
if not payload.dry_run:
|
||||||
|
session.commit()
|
||||||
|
return RenameResponse(
|
||||||
|
dry_run=payload.dry_run,
|
||||||
|
items=[
|
||||||
|
RenamePreviewItem(
|
||||||
|
kind=item.kind,
|
||||||
|
id=item.id,
|
||||||
|
file_id=item.id if item.kind == "file" else None,
|
||||||
|
folder_path=item.old_path if item.kind == "folder" else None,
|
||||||
|
old_path=item.old_path,
|
||||||
|
new_path=item.new_path,
|
||||||
|
)
|
||||||
|
for item in plan
|
||||||
|
],
|
||||||
|
)
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/transfer", response_model=TransferResponse)
|
||||||
|
def transfer_files(
|
||||||
|
payload: TransferRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
files, folders = transfer_selection(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
operation=payload.operation,
|
||||||
|
file_ids=payload.file_ids,
|
||||||
|
folder_paths=payload.folder_paths,
|
||||||
|
source_owner_type=payload.source_owner_type,
|
||||||
|
source_owner_id=payload.source_owner_id,
|
||||||
|
target_owner_type=payload.target_owner_type,
|
||||||
|
target_owner_id=payload.target_owner_id,
|
||||||
|
target_folder=payload.target_folder,
|
||||||
|
conflict_strategy=payload.conflict_strategy,
|
||||||
|
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return TransferResponse(
|
||||||
|
operation=payload.operation, files=files, folders=folders
|
||||||
|
)
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/archive.zip")
|
||||||
|
def download_archive(
|
||||||
|
payload: ArchiveRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
assets = [
|
||||||
|
get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
asset_id=file_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
for file_id in payload.file_ids
|
||||||
|
]
|
||||||
|
tmp = tempfile.NamedTemporaryFile(
|
||||||
|
prefix="govoplan-files-", suffix=".zip", delete=False
|
||||||
|
)
|
||||||
|
tmp_path = tmp.name
|
||||||
|
tmp.close()
|
||||||
|
try:
|
||||||
|
create_zip_file(session, assets, tmp_path)
|
||||||
|
except Exception:
|
||||||
|
_cleanup_temp_file(tmp_path)
|
||||||
|
raise
|
||||||
|
_audit_connector_access(session, principal, assets, operation="archive")
|
||||||
|
except FileStorageError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
filename = filename_from_path(
|
||||||
|
normalize_logical_path(payload.filename, fallback_filename="files.zip")
|
||||||
|
)
|
||||||
|
headers = {"Content-Disposition": _attachment_disposition(filename)}
|
||||||
|
return FileResponse(
|
||||||
|
tmp_path,
|
||||||
|
media_type="application/zip",
|
||||||
|
headers=headers,
|
||||||
|
background=BackgroundTask(_cleanup_temp_file, tmp_path),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
|
||||||
|
def resolve_file_patterns(
|
||||||
|
payload: PatternResolveRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||||
|
):
|
||||||
|
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
|
||||||
|
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||||
|
try:
|
||||||
|
assets = list_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=payload.owner_type,
|
||||||
|
owner_id=payload.owner_id,
|
||||||
|
campaign_id=payload.campaign_id,
|
||||||
|
path_prefix=payload.path_prefix,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
resolved, unmatched = resolve_patterns(
|
||||||
|
assets,
|
||||||
|
payload.patterns,
|
||||||
|
base_path=payload.path_prefix,
|
||||||
|
case_sensitive=payload.case_sensitive,
|
||||||
|
)
|
||||||
|
return PatternResolveResponse(
|
||||||
|
patterns=[
|
||||||
|
PatternMatchResponse(
|
||||||
|
pattern=item.pattern,
|
||||||
|
matches=[_asset_response(session, asset) for asset in item.matches],
|
||||||
|
)
|
||||||
|
for item in resolved
|
||||||
|
],
|
||||||
|
unmatched=[_asset_response(session, asset) for asset in unmatched]
|
||||||
|
if payload.include_unmatched
|
||||||
|
else [],
|
||||||
|
)
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
@@ -0,0 +1,487 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
import json
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
from typing import Literal
|
||||||
|
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||||
|
from govoplan_core.security.secrets import (
|
||||||
|
TransientPayloadError,
|
||||||
|
open_transient_payload,
|
||||||
|
seal_transient_payload,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
ArchiveEntryResponse,
|
||||||
|
ArchivePreviewResponse,
|
||||||
|
ConflictResolutionRequest,
|
||||||
|
FileUploadResponse,
|
||||||
|
_conflict_resolutions,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileAsset
|
||||||
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_files.backend.runtime import settings
|
||||||
|
from govoplan_files.backend.storage.paths import (
|
||||||
|
UnsafeFilePathError,
|
||||||
|
normalize_folder,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.archives import (
|
||||||
|
archive_format_for_filename,
|
||||||
|
extract_archive_upload,
|
||||||
|
extract_zip_upload,
|
||||||
|
inspect_archive,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorPolicyDenied,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
create_file_asset,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_response,
|
||||||
|
_audit_connector_imports,
|
||||||
|
_cleanup_temp_file,
|
||||||
|
_connector_policy_error,
|
||||||
|
_enforce_connector_policy,
|
||||||
|
_http_error,
|
||||||
|
_is_admin,
|
||||||
|
_read_limited_upload,
|
||||||
|
_source_metadata_from_form,
|
||||||
|
_spool_limited_upload_to_temp,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/files", tags=["files"])
|
||||||
|
_ARCHIVE_PREVIEW_PURPOSE = "files.archive-preview.v1"
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_suffix(filename: str) -> str:
|
||||||
|
lowered = filename.casefold()
|
||||||
|
for suffix in (".tar.bz2", ".tar.gz", ".tar.xz", ".tbz2", ".tgz", ".txz", ".tar", ".zip"):
|
||||||
|
if lowered.endswith(suffix):
|
||||||
|
return suffix
|
||||||
|
return ".archive"
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_sha256(path: str) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with open(path, "rb") as source:
|
||||||
|
while chunk := source.read(1024 * 1024):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_selected_paths(value: str) -> list[str]:
|
||||||
|
parsed = json.loads(value)
|
||||||
|
if not isinstance(parsed, list) or not parsed:
|
||||||
|
raise FileStorageError("Select at least one archive file or folder")
|
||||||
|
if len(parsed) > settings.file_archive_max_entries:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Archive selection exceeds the configured entry limit"
|
||||||
|
)
|
||||||
|
selected: list[str] = []
|
||||||
|
for item in parsed:
|
||||||
|
if not isinstance(item, str) or not item.strip():
|
||||||
|
raise FileStorageError("Archive selection contains an invalid path")
|
||||||
|
if len(item) > 4096:
|
||||||
|
raise FileStorageError("Archive selection path is too long")
|
||||||
|
selected.append(item)
|
||||||
|
return selected
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_archive_preview_token(
|
||||||
|
token: str,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
path: str,
|
||||||
|
campaign_id: str | None,
|
||||||
|
archive_format: str,
|
||||||
|
archive_sha256: str,
|
||||||
|
) -> None:
|
||||||
|
payload = open_transient_payload(
|
||||||
|
token,
|
||||||
|
ttl_seconds=settings.file_archive_preview_ttl_seconds,
|
||||||
|
)
|
||||||
|
expected = {
|
||||||
|
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"user_id": user_id,
|
||||||
|
"owner_type": owner_type,
|
||||||
|
"owner_id": owner_id,
|
||||||
|
"path": path,
|
||||||
|
"campaign_id": campaign_id or "",
|
||||||
|
"archive_format": archive_format,
|
||||||
|
}
|
||||||
|
if any(payload.get(key) != value for key, value in expected.items()):
|
||||||
|
raise FileStorageError(
|
||||||
|
"Archive preview does not match this upload destination"
|
||||||
|
)
|
||||||
|
token_digest = payload.get("archive_sha256")
|
||||||
|
if not isinstance(token_digest, str) or not hmac.compare_digest(
|
||||||
|
token_digest, archive_sha256
|
||||||
|
):
|
||||||
|
raise FileStorageError(
|
||||||
|
"Archive contents changed after preview; preview it again"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/archive-preview", response_model=ArchivePreviewResponse)
|
||||||
|
def preview_archive_upload(
|
||||||
|
file: UploadFile = FastAPIFile(...),
|
||||||
|
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||||
|
owner_id: str | None = Form(default=None),
|
||||||
|
path: str = Form(default=""),
|
||||||
|
campaign_id: str | None = Form(default=None),
|
||||||
|
password: str | None = Form(default=None),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||||
|
):
|
||||||
|
target_owner = owner_id or principal.user.id
|
||||||
|
archive_path: str | None = None
|
||||||
|
filename = file.filename or "archive"
|
||||||
|
try:
|
||||||
|
archive_format = archive_format_for_filename(filename)
|
||||||
|
archive_path = _spool_limited_upload_to_temp(
|
||||||
|
file,
|
||||||
|
max_bytes=settings.file_upload_zip_max_bytes,
|
||||||
|
suffix=_archive_suffix(filename),
|
||||||
|
)
|
||||||
|
inspection = inspect_archive(
|
||||||
|
archive_path,
|
||||||
|
filename=filename,
|
||||||
|
password=password,
|
||||||
|
max_entries=settings.file_archive_max_entries,
|
||||||
|
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||||
|
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||||
|
)
|
||||||
|
digest = _archive_sha256(archive_path)
|
||||||
|
normalized_path = normalize_folder(path)
|
||||||
|
preview_token = seal_transient_payload(
|
||||||
|
{
|
||||||
|
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||||
|
"tenant_id": principal.tenant_id,
|
||||||
|
"user_id": principal.user.id,
|
||||||
|
"owner_type": owner_type,
|
||||||
|
"owner_id": target_owner,
|
||||||
|
"path": normalized_path,
|
||||||
|
"campaign_id": campaign_id or "",
|
||||||
|
"archive_format": archive_format,
|
||||||
|
"archive_sha256": digest,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
expires_at = datetime.now(UTC) + timedelta(
|
||||||
|
seconds=settings.file_archive_preview_ttl_seconds
|
||||||
|
)
|
||||||
|
return ArchivePreviewResponse(
|
||||||
|
preview_token=preview_token,
|
||||||
|
archive_format=inspection.archive_format,
|
||||||
|
entries=[
|
||||||
|
ArchiveEntryResponse(
|
||||||
|
path=entry.path,
|
||||||
|
kind=entry.kind,
|
||||||
|
size_bytes=entry.size_bytes,
|
||||||
|
compressed_size_bytes=entry.compressed_size_bytes,
|
||||||
|
encrypted=entry.encrypted,
|
||||||
|
)
|
||||||
|
for entry in inspection.entries
|
||||||
|
],
|
||||||
|
file_count=inspection.file_count,
|
||||||
|
directory_count=inspection.directory_count,
|
||||||
|
expanded_size_bytes=inspection.expanded_size_bytes,
|
||||||
|
compressed_size_bytes=inspection.compressed_size_bytes,
|
||||||
|
requires_password=inspection.requires_password,
|
||||||
|
password_verified=inspection.password_verified,
|
||||||
|
expires_at=expires_at.isoformat(),
|
||||||
|
)
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
finally:
|
||||||
|
if archive_path:
|
||||||
|
_cleanup_temp_file(archive_path)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/archive-confirm", response_model=FileUploadResponse)
|
||||||
|
def confirm_archive_upload(
|
||||||
|
file: UploadFile = FastAPIFile(...),
|
||||||
|
preview_token: str = Form(...),
|
||||||
|
selected_paths_json: str = Form(...),
|
||||||
|
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||||
|
owner_id: str | None = Form(default=None),
|
||||||
|
path: str = Form(default=""),
|
||||||
|
campaign_id: str | None = Form(default=None),
|
||||||
|
password: str | None = Form(default=None),
|
||||||
|
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||||
|
default="reject"
|
||||||
|
),
|
||||||
|
conflict_resolutions_json: str | None = Form(default=None),
|
||||||
|
source_provenance_json: str | None = Form(default=None),
|
||||||
|
source_revision: str | None = Form(default=None),
|
||||||
|
connector_policy_json: str | None = Form(default=None),
|
||||||
|
encryption_vault_id: str | None = Form(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||||
|
):
|
||||||
|
target_owner = owner_id or principal.user.id
|
||||||
|
archive_path: str | None = None
|
||||||
|
filename = file.filename or "archive"
|
||||||
|
try:
|
||||||
|
raw_resolutions = (
|
||||||
|
json.loads(conflict_resolutions_json)
|
||||||
|
if conflict_resolutions_json
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
upload_resolutions = _conflict_resolutions(
|
||||||
|
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||||
|
)
|
||||||
|
selected_paths = _archive_selected_paths(selected_paths_json)
|
||||||
|
_enforce_connector_policy(
|
||||||
|
source_provenance_json,
|
||||||
|
connector_policy_json,
|
||||||
|
operation="import",
|
||||||
|
)
|
||||||
|
metadata = _source_metadata_from_form(
|
||||||
|
source_provenance_json, source_revision
|
||||||
|
)
|
||||||
|
archive_format = archive_format_for_filename(filename)
|
||||||
|
normalized_path = normalize_folder(path)
|
||||||
|
archive_path = _spool_limited_upload_to_temp(
|
||||||
|
file,
|
||||||
|
max_bytes=settings.file_upload_zip_max_bytes,
|
||||||
|
suffix=_archive_suffix(filename),
|
||||||
|
)
|
||||||
|
digest = _archive_sha256(archive_path)
|
||||||
|
_validate_archive_preview_token(
|
||||||
|
preview_token,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
path=normalized_path,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
archive_format=archive_format,
|
||||||
|
archive_sha256=digest,
|
||||||
|
)
|
||||||
|
extracted = extract_archive_upload(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
archive_data=archive_path,
|
||||||
|
filename=filename,
|
||||||
|
folder=normalized_path,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
selected_paths=selected_paths,
|
||||||
|
password=password,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=upload_resolutions,
|
||||||
|
metadata=metadata,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
max_entries=settings.file_archive_max_entries,
|
||||||
|
max_file_bytes=settings.file_upload_max_bytes,
|
||||||
|
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||||
|
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||||
|
)
|
||||||
|
uploaded_assets = [item.asset for item in extracted]
|
||||||
|
_audit_connector_imports(session, principal, uploaded_assets)
|
||||||
|
session.commit()
|
||||||
|
return FileUploadResponse(
|
||||||
|
files=[
|
||||||
|
_asset_response(session, asset, include_shares=True)
|
||||||
|
for asset in uploaded_assets
|
||||||
|
]
|
||||||
|
)
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (
|
||||||
|
FileStorageError,
|
||||||
|
TransientPayloadError,
|
||||||
|
UnsafeFilePathError,
|
||||||
|
ValueError,
|
||||||
|
json.JSONDecodeError,
|
||||||
|
) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
finally:
|
||||||
|
if archive_path:
|
||||||
|
_cleanup_temp_file(archive_path)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/upload", response_model=FileUploadResponse)
|
||||||
|
def upload_files(
|
||||||
|
files: list[UploadFile] = FastAPIFile(...),
|
||||||
|
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||||
|
owner_id: str | None = Form(default=None),
|
||||||
|
path: str = Form(default=""),
|
||||||
|
campaign_id: str | None = Form(default=None),
|
||||||
|
unpack_zip: bool = Form(default=False),
|
||||||
|
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||||
|
default="reject"
|
||||||
|
),
|
||||||
|
conflict_resolutions_json: str | None = Form(default=None),
|
||||||
|
source_provenance_json: str | None = Form(default=None),
|
||||||
|
source_revision: str | None = Form(default=None),
|
||||||
|
connector_policy_json: str | None = Form(default=None),
|
||||||
|
encryption_vault_id: str | None = Form(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||||
|
):
|
||||||
|
target_owner = owner_id or principal.user.id
|
||||||
|
uploaded_assets: list[FileAsset] = []
|
||||||
|
try:
|
||||||
|
raw_resolutions = (
|
||||||
|
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||||
|
)
|
||||||
|
upload_resolutions = _conflict_resolutions(
|
||||||
|
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||||
|
)
|
||||||
|
_enforce_connector_policy(
|
||||||
|
source_provenance_json, connector_policy_json, operation="import"
|
||||||
|
)
|
||||||
|
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
|
||||||
|
for upload in files:
|
||||||
|
filename = upload.filename or "file"
|
||||||
|
content_type = upload.content_type or None
|
||||||
|
upload_limit = (
|
||||||
|
settings.file_upload_zip_max_bytes
|
||||||
|
if unpack_zip and filename.lower().endswith(".zip")
|
||||||
|
else settings.file_upload_max_bytes
|
||||||
|
)
|
||||||
|
if unpack_zip and filename.lower().endswith(".zip"):
|
||||||
|
zip_path = _spool_limited_upload_to_temp(
|
||||||
|
upload, max_bytes=upload_limit, suffix=".zip"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
extracted = extract_zip_upload(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
zip_data=zip_path,
|
||||||
|
folder=path,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=upload_resolutions,
|
||||||
|
metadata=metadata,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
max_file_bytes=settings.file_upload_max_bytes,
|
||||||
|
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_temp_file(zip_path)
|
||||||
|
uploaded_assets.extend(item.asset for item in extracted)
|
||||||
|
continue
|
||||||
|
data = _read_limited_upload(upload, max_bytes=upload_limit)
|
||||||
|
stored = create_file_asset(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
folder=path,
|
||||||
|
content_type=content_type,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=upload_resolutions,
|
||||||
|
metadata=metadata,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
)
|
||||||
|
uploaded_assets.append(stored.asset)
|
||||||
|
_audit_connector_imports(session, principal, uploaded_assets)
|
||||||
|
session.commit()
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return FileUploadResponse(
|
||||||
|
files=[
|
||||||
|
_asset_response(session, asset, include_shares=True)
|
||||||
|
for asset in uploaded_assets
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||||
|
def upload_zip(
|
||||||
|
file: UploadFile = FastAPIFile(...),
|
||||||
|
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||||
|
owner_id: str | None = Form(default=None),
|
||||||
|
path: str = Form(default=""),
|
||||||
|
campaign_id: str | None = Form(default=None),
|
||||||
|
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||||
|
default="reject"
|
||||||
|
),
|
||||||
|
conflict_resolutions_json: str | None = Form(default=None),
|
||||||
|
source_provenance_json: str | None = Form(default=None),
|
||||||
|
source_revision: str | None = Form(default=None),
|
||||||
|
connector_policy_json: str | None = Form(default=None),
|
||||||
|
encryption_vault_id: str | None = Form(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||||
|
):
|
||||||
|
target_owner = owner_id or principal.user.id
|
||||||
|
zip_path: str | None = None
|
||||||
|
try:
|
||||||
|
raw_resolutions = (
|
||||||
|
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||||
|
)
|
||||||
|
upload_resolutions = _conflict_resolutions(
|
||||||
|
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||||
|
)
|
||||||
|
_enforce_connector_policy(
|
||||||
|
source_provenance_json, connector_policy_json, operation="import"
|
||||||
|
)
|
||||||
|
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
|
||||||
|
zip_path = _spool_limited_upload_to_temp(
|
||||||
|
file, max_bytes=settings.file_upload_zip_max_bytes, suffix=".zip"
|
||||||
|
)
|
||||||
|
extracted = extract_zip_upload(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=target_owner,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
zip_data=zip_path,
|
||||||
|
folder=path,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=upload_resolutions,
|
||||||
|
metadata=metadata,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
max_file_bytes=settings.file_upload_max_bytes,
|
||||||
|
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||||
|
)
|
||||||
|
_audit_connector_imports(session, principal, [item.asset for item in extracted])
|
||||||
|
session.commit()
|
||||||
|
except ConnectorPolicyDenied as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _connector_policy_error(exc) from exc
|
||||||
|
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||||
|
session.rollback()
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
finally:
|
||||||
|
if zip_path:
|
||||||
|
_cleanup_temp_file(zip_path)
|
||||||
|
return FileUploadResponse(
|
||||||
|
files=[
|
||||||
|
_asset_response(session, item.asset, include_shares=True)
|
||||||
|
for item in extracted
|
||||||
|
]
|
||||||
|
)
|
||||||
@@ -1,29 +1,10 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from typing import Any
|
from govoplan_core.core.runtime import ModuleRuntimeState
|
||||||
|
|
||||||
_runtime_settings: object | None = None
|
_runtime = ModuleRuntimeState("Files")
|
||||||
|
|
||||||
|
configure_runtime = _runtime.configure_runtime
|
||||||
def configure_runtime(*, settings: object | None = None) -> None:
|
get_registry = _runtime.get_registry
|
||||||
global _runtime_settings
|
get_settings = _runtime.get_settings
|
||||||
if settings is not None:
|
settings = _runtime.settings
|
||||||
_runtime_settings = settings
|
|
||||||
|
|
||||||
|
|
||||||
def get_settings() -> object:
|
|
||||||
if _runtime_settings is not None:
|
|
||||||
return _runtime_settings
|
|
||||||
try:
|
|
||||||
from govoplan_core.settings import settings as legacy_settings
|
|
||||||
except ModuleNotFoundError as exc:
|
|
||||||
raise RuntimeError("GovOPlaN Files runtime settings are not configured") from exc
|
|
||||||
return legacy_settings
|
|
||||||
|
|
||||||
|
|
||||||
class SettingsProxy:
|
|
||||||
def __getattr__(self, name: str) -> Any:
|
|
||||||
return getattr(get_settings(), name)
|
|
||||||
|
|
||||||
|
|
||||||
settings = SettingsProxy()
|
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||||
from govoplan_files.backend.storage.common import FileConflictResolution
|
from govoplan_files.backend.storage.common import FileConflictResolution
|
||||||
|
|
||||||
|
|
||||||
@@ -13,19 +15,168 @@ class FileSpaceResponse(BaseModel):
|
|||||||
owner_type: Literal["user", "group"]
|
owner_type: Literal["user", "group"]
|
||||||
owner_id: str
|
owner_id: str
|
||||||
description: str | None = None
|
description: str | None = None
|
||||||
|
space_type: Literal["managed", "connector"] = "managed"
|
||||||
|
connector_space_id: str | None = None
|
||||||
|
connector_profile_id: str | None = None
|
||||||
|
provider: str | None = None
|
||||||
|
library_id: str | None = None
|
||||||
|
remote_path: str | None = None
|
||||||
|
sync_mode: str | None = None
|
||||||
|
read_only: bool = False
|
||||||
|
|
||||||
|
|
||||||
class FileSpacesResponse(BaseModel):
|
class FileSpacesResponse(BaseModel):
|
||||||
spaces: list[FileSpaceResponse]
|
spaces: list[FileSpaceResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSpaceCreateRequest(BaseModel):
|
||||||
|
owner_type: Literal["user", "group"] = "user"
|
||||||
|
owner_id: str | None = None
|
||||||
|
label: str
|
||||||
|
connector_profile_id: str
|
||||||
|
library_id: str | None = None
|
||||||
|
remote_path: str = ""
|
||||||
|
sync_mode: Literal["manual"] = "manual"
|
||||||
|
read_only: bool = True
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSpaceUpdateRequest(BaseModel):
|
||||||
|
label: str | None = None
|
||||||
|
library_id: str | None = None
|
||||||
|
remote_path: str | None = None
|
||||||
|
sync_mode: Literal["manual"] | None = None
|
||||||
|
read_only: bool | None = None
|
||||||
|
is_active: bool | None = None
|
||||||
|
metadata: dict[str, Any] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSpaceResponse(BaseModel):
|
||||||
|
id: str
|
||||||
|
tenant_id: str
|
||||||
|
owner_type: Literal["user", "group"]
|
||||||
|
owner_id: str
|
||||||
|
label: str
|
||||||
|
connector_profile_id: str
|
||||||
|
provider: str
|
||||||
|
library_id: str | None = None
|
||||||
|
remote_path: str = ""
|
||||||
|
sync_mode: str
|
||||||
|
read_only: bool
|
||||||
|
is_active: bool
|
||||||
|
created_at: str
|
||||||
|
updated_at: str
|
||||||
|
deleted_at: str | None = None
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSpacesResponse(BaseModel):
|
||||||
|
spaces: list[FileConnectorSpaceResponse] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
class FileShareResponse(BaseModel):
|
class FileShareResponse(BaseModel):
|
||||||
id: str
|
id: str
|
||||||
|
file_asset_id: str
|
||||||
target_type: str
|
target_type: str
|
||||||
target_id: str
|
target_id: str
|
||||||
permission: str
|
permission: str
|
||||||
|
created_by_user_id: str | None = None
|
||||||
created_at: str
|
created_at: str
|
||||||
|
expires_at: str | None = None
|
||||||
revoked_at: str | None = None
|
revoked_at: str | None = None
|
||||||
|
revoked_by_user_id: str | None = None
|
||||||
|
active: bool
|
||||||
|
|
||||||
|
|
||||||
|
class FileSharesResponse(BaseModel):
|
||||||
|
shares: list[FileShareResponse] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityScanCreateRequest(BaseModel):
|
||||||
|
verify_checksums: bool = True
|
||||||
|
batch_size: int = Field(default=100, ge=1, le=1000)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityScanResponse(BaseModel):
|
||||||
|
id: str
|
||||||
|
tenant_id: str
|
||||||
|
storage_backend: str
|
||||||
|
storage_prefix: str
|
||||||
|
status: str
|
||||||
|
revision: int
|
||||||
|
phase: str
|
||||||
|
verify_checksums: bool
|
||||||
|
batch_size: int
|
||||||
|
scanned_blob_count: int
|
||||||
|
verified_blob_count: int
|
||||||
|
quarantined_blob_count: int
|
||||||
|
scanned_object_count: int
|
||||||
|
orphan_object_count: int
|
||||||
|
created_by_user_id: str | None = None
|
||||||
|
started_at: str | None = None
|
||||||
|
completed_at: str | None = None
|
||||||
|
last_error: str | None = None
|
||||||
|
created_at: str
|
||||||
|
updated_at: str
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityScansResponse(BaseModel):
|
||||||
|
scans: list[FileIntegrityScanResponse] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityFindingResponse(BaseModel):
|
||||||
|
id: str
|
||||||
|
scan_id: str
|
||||||
|
tenant_id: str
|
||||||
|
kind: str
|
||||||
|
state: str
|
||||||
|
revision: int
|
||||||
|
blob_id: str | None = None
|
||||||
|
storage_key: str
|
||||||
|
expected_size_bytes: int | None = None
|
||||||
|
observed_size_bytes: int | None = None
|
||||||
|
expected_checksum_sha256: str | None = None
|
||||||
|
observed_checksum_sha256: str | None = None
|
||||||
|
resolved_at: str | None = None
|
||||||
|
resolved_by_user_id: str | None = None
|
||||||
|
created_at: str
|
||||||
|
updated_at: str
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityFindingsResponse(BaseModel):
|
||||||
|
findings: list[FileIntegrityFindingResponse] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityActionRequest(BaseModel):
|
||||||
|
dry_run: bool = True
|
||||||
|
expected_revision: int = Field(ge=1)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityScanRunRequest(BaseModel):
|
||||||
|
expected_revision: int = Field(ge=1)
|
||||||
|
|
||||||
|
|
||||||
|
class FileIntegrityActionResponse(BaseModel):
|
||||||
|
action: str
|
||||||
|
changed: bool
|
||||||
|
dry_run: bool
|
||||||
|
finding: FileIntegrityFindingResponse
|
||||||
|
inspection_kind: str | None = None
|
||||||
|
inspection_valid: bool | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileSourceProvenance(BaseModel):
|
||||||
|
source_type: str | None = None
|
||||||
|
connector_id: str | None = None
|
||||||
|
provider: str | None = None
|
||||||
|
external_id: str | None = None
|
||||||
|
external_path: str | None = None
|
||||||
|
external_url: str | None = None
|
||||||
|
revision: str | None = None
|
||||||
|
revision_label: str | None = None
|
||||||
|
observed_at: str | None = None
|
||||||
|
imported_at: str | None = None
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
class FileAssetResponse(BaseModel):
|
class FileAssetResponse(BaseModel):
|
||||||
@@ -43,8 +194,14 @@ class FileAssetResponse(BaseModel):
|
|||||||
created_at: str
|
created_at: str
|
||||||
updated_at: str
|
updated_at: str
|
||||||
deleted_at: str | None = None
|
deleted_at: str | None = None
|
||||||
|
retained_until: str | None = None
|
||||||
|
legal_hold: bool = False
|
||||||
|
lifecycle_revision: int = 1
|
||||||
|
lifecycle_reason: str | None = None
|
||||||
audit_relevant: bool = False
|
audit_relevant: bool = False
|
||||||
metadata: dict[str, Any] | None = None
|
metadata: dict[str, Any] | None = None
|
||||||
|
source_provenance: FileSourceProvenance | None = None
|
||||||
|
source_revision: str | None = None
|
||||||
shares: list[FileShareResponse] = Field(default_factory=list)
|
shares: list[FileShareResponse] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
@@ -61,6 +218,9 @@ class FileFolderResponse(BaseModel):
|
|||||||
|
|
||||||
class FileFoldersResponse(BaseModel):
|
class FileFoldersResponse(BaseModel):
|
||||||
folders: list[FileFolderResponse]
|
folders: list[FileFolderResponse]
|
||||||
|
cursor: str | None = None
|
||||||
|
next_cursor: str | None = None
|
||||||
|
watermark: str | None = None
|
||||||
|
|
||||||
|
|
||||||
class FileFolderCreateRequest(BaseModel):
|
class FileFolderCreateRequest(BaseModel):
|
||||||
@@ -83,12 +243,323 @@ class FileFolderDeleteResponse(BaseModel):
|
|||||||
|
|
||||||
class FileListResponse(BaseModel):
|
class FileListResponse(BaseModel):
|
||||||
files: list[FileAssetResponse]
|
files: list[FileAssetResponse]
|
||||||
|
total: int
|
||||||
|
cursor: str | None = None
|
||||||
|
next_cursor: str | None = None
|
||||||
|
watermark: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileDeltaResponse(BaseModel):
|
||||||
|
files: list[FileAssetResponse] = Field(default_factory=list)
|
||||||
|
folders: list[FileFolderResponse] = Field(default_factory=list)
|
||||||
|
deleted: list[DeltaDeletedItem] = Field(default_factory=list)
|
||||||
|
watermark: str | None = None
|
||||||
|
has_more: bool = False
|
||||||
|
full: bool = False
|
||||||
|
|
||||||
|
|
||||||
class FileUploadResponse(BaseModel):
|
class FileUploadResponse(BaseModel):
|
||||||
files: list[FileAssetResponse]
|
files: list[FileAssetResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class ArchiveEntryResponse(BaseModel):
|
||||||
|
path: str
|
||||||
|
kind: Literal["file", "directory"]
|
||||||
|
size_bytes: int
|
||||||
|
compressed_size_bytes: int | None = None
|
||||||
|
encrypted: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class ArchivePreviewResponse(BaseModel):
|
||||||
|
preview_token: str
|
||||||
|
archive_format: str
|
||||||
|
entries: list[ArchiveEntryResponse] = Field(default_factory=list)
|
||||||
|
file_count: int
|
||||||
|
directory_count: int
|
||||||
|
expanded_size_bytes: int
|
||||||
|
compressed_size_bytes: int
|
||||||
|
requires_password: bool
|
||||||
|
password_verified: bool
|
||||||
|
expires_at: str
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicySource(BaseModel):
|
||||||
|
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||||
|
scope_id: str | None = None
|
||||||
|
label: str | None = None
|
||||||
|
policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicyEvaluateRequest(BaseModel):
|
||||||
|
source_provenance: FileSourceProvenance
|
||||||
|
operation: str = "access"
|
||||||
|
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicyEvaluateResponse(BaseModel):
|
||||||
|
decision: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicyUpdateRequest(BaseModel):
|
||||||
|
policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicyStepResponse(BaseModel):
|
||||||
|
scope_type: str
|
||||||
|
scope_id: str | None = None
|
||||||
|
path: str
|
||||||
|
label: str
|
||||||
|
applied_fields: list[str] = Field(default_factory=list)
|
||||||
|
policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorPolicyResponse(BaseModel):
|
||||||
|
scope_type: Literal["system", "tenant", "user", "group", "campaign"]
|
||||||
|
scope_id: str | None = None
|
||||||
|
policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
effective_policy: dict[str, Any] | None = None
|
||||||
|
parent_policy: dict[str, Any] | None = None
|
||||||
|
effective_policy_sources: list[FileConnectorPolicyStepResponse] = Field(default_factory=list)
|
||||||
|
parent_policy_sources: list[FileConnectorPolicyStepResponse] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProfileResponse(BaseModel):
|
||||||
|
id: str
|
||||||
|
label: str
|
||||||
|
provider: str
|
||||||
|
endpoint_url: str | None = None
|
||||||
|
base_path: str | None = None
|
||||||
|
enabled: bool = True
|
||||||
|
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||||
|
scope_id: str | None = None
|
||||||
|
source_path: str
|
||||||
|
credential_profile_id: str | None = None
|
||||||
|
credential_profile_label: str | None = None
|
||||||
|
credential_mode: str = "none"
|
||||||
|
credential_source: str | None = None
|
||||||
|
credentials_configured: bool = False
|
||||||
|
username: str | None = None
|
||||||
|
capabilities: list[str] = Field(default_factory=list)
|
||||||
|
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
source_kind: str = "settings"
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProfilesResponse(BaseModel):
|
||||||
|
profiles: list[FileConnectorProfileResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorCredentialResponse(BaseModel):
|
||||||
|
id: str
|
||||||
|
label: str
|
||||||
|
provider: str | None = None
|
||||||
|
enabled: bool = True
|
||||||
|
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||||
|
scope_id: str | None = None
|
||||||
|
source_path: str
|
||||||
|
credential_mode: str = "none"
|
||||||
|
credential_secret_source: str | None = None
|
||||||
|
credentials_configured: bool = False
|
||||||
|
username: str | None = None
|
||||||
|
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
source_kind: str = "database"
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorCredentialsResponse(BaseModel):
|
||||||
|
credentials: list[FileConnectorCredentialResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSettingsDeltaResponse(BaseModel):
|
||||||
|
profiles: list[FileConnectorProfileResponse] = Field(default_factory=list)
|
||||||
|
credentials: list[FileConnectorCredentialResponse] = Field(default_factory=list)
|
||||||
|
spaces: list[FileConnectorSpaceResponse] = Field(default_factory=list)
|
||||||
|
policy: FileConnectorPolicyResponse | None = None
|
||||||
|
changed_sections: list[str] = Field(default_factory=list)
|
||||||
|
deleted: list[DeltaDeletedItem] = Field(default_factory=list)
|
||||||
|
watermark: str | None = None
|
||||||
|
has_more: bool = False
|
||||||
|
full: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProfileCredentialsRequest(BaseModel):
|
||||||
|
username: str | None = None
|
||||||
|
password: str | None = None
|
||||||
|
token: str | None = None
|
||||||
|
password_env: str | None = None
|
||||||
|
token_env: str | None = None
|
||||||
|
secret_ref: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorDiscoveryCandidate(BaseModel):
|
||||||
|
endpoint_url: str
|
||||||
|
status: Literal["failed", "usable", "found", "credentials_rejected"]
|
||||||
|
message: str
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorDiscoveryRequest(BaseModel):
|
||||||
|
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||||
|
endpoint_url: str
|
||||||
|
base_path: str | None = None
|
||||||
|
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||||
|
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
require_valid_credentials: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorDiscoveryResponse(BaseModel):
|
||||||
|
provider: str
|
||||||
|
endpoint_url: str | None = None
|
||||||
|
base_path: str | None = None
|
||||||
|
status: Literal["usable", "found", "credentials_rejected", "not_found", "unsupported"]
|
||||||
|
message: str
|
||||||
|
candidates: list[FileConnectorDiscoveryCandidate] = Field(default_factory=list)
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProfileCreateRequest(BaseModel):
|
||||||
|
id: str
|
||||||
|
label: str
|
||||||
|
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||||
|
endpoint_url: str | None = None
|
||||||
|
base_path: str | None = None
|
||||||
|
enabled: bool = True
|
||||||
|
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||||
|
scope_id: str | None = None
|
||||||
|
credential_profile_id: str | None = None
|
||||||
|
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||||
|
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||||
|
capabilities: list[str] = Field(default_factory=lambda: ["browse", "import", "sync"])
|
||||||
|
policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProfileUpdateRequest(BaseModel):
|
||||||
|
label: str | None = None
|
||||||
|
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||||
|
endpoint_url: str | None = None
|
||||||
|
base_path: str | None = None
|
||||||
|
enabled: bool | None = None
|
||||||
|
credential_profile_id: str | None = None
|
||||||
|
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||||
|
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||||
|
clear_password: bool = False
|
||||||
|
clear_token: bool = False
|
||||||
|
capabilities: list[str] | None = None
|
||||||
|
policy: dict[str, Any] | None = None
|
||||||
|
metadata: dict[str, Any] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorCredentialCreateRequest(BaseModel):
|
||||||
|
id: str
|
||||||
|
label: str
|
||||||
|
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||||
|
enabled: bool = True
|
||||||
|
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||||
|
scope_id: str | None = None
|
||||||
|
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||||
|
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||||
|
policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorCredentialUpdateRequest(BaseModel):
|
||||||
|
label: str | None = None
|
||||||
|
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||||
|
enabled: bool | None = None
|
||||||
|
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||||
|
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||||
|
clear_password: bool = False
|
||||||
|
clear_token: bool = False
|
||||||
|
policy: dict[str, Any] | None = None
|
||||||
|
metadata: dict[str, Any] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProviderResponse(BaseModel):
|
||||||
|
provider: str
|
||||||
|
label: str
|
||||||
|
protocol: str
|
||||||
|
implemented: bool
|
||||||
|
installed: bool
|
||||||
|
browse_supported: bool
|
||||||
|
import_supported: bool
|
||||||
|
write_supported: bool = False
|
||||||
|
optional_dependency: str | None = None
|
||||||
|
permission_model: str
|
||||||
|
sync_strategy: str
|
||||||
|
conflict_strategy: str
|
||||||
|
preview_strategy: str
|
||||||
|
audit_events: list[str] = Field(default_factory=list)
|
||||||
|
notes: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorProvidersResponse(BaseModel):
|
||||||
|
providers: list[FileConnectorProviderResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorBrowseItem(BaseModel):
|
||||||
|
kind: Literal["library", "folder", "file"]
|
||||||
|
name: str
|
||||||
|
path: str
|
||||||
|
external_id: str | None = None
|
||||||
|
external_url: str | None = None
|
||||||
|
size_bytes: int | None = None
|
||||||
|
content_type: str | None = None
|
||||||
|
modified_at: str | None = None
|
||||||
|
etag: str | None = None
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorBrowseResponse(BaseModel):
|
||||||
|
profile_id: str
|
||||||
|
provider: str
|
||||||
|
path: str = ""
|
||||||
|
library_id: str | None = None
|
||||||
|
read_only: bool = True
|
||||||
|
next_continuation_token: str | None = None
|
||||||
|
has_more: bool = False
|
||||||
|
decision: dict[str, Any]
|
||||||
|
items: list[FileConnectorBrowseItem]
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorImportRequest(BaseModel):
|
||||||
|
library_id: str
|
||||||
|
path: str
|
||||||
|
owner_type: Literal["user", "group"] = "user"
|
||||||
|
owner_id: str | None = None
|
||||||
|
target_folder: str | None = None
|
||||||
|
target_path: str | None = None
|
||||||
|
campaign_id: str | None = None
|
||||||
|
conflict_strategy: Literal["reject", "overwrite", "rename"] = "reject"
|
||||||
|
source_revision: str | None = None
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorSyncResponse(BaseModel):
|
||||||
|
file: FileAssetResponse
|
||||||
|
action: Literal["created", "updated", "unchanged"]
|
||||||
|
previous_version_id: str | None = None
|
||||||
|
current_version_id: str
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorWriteRequest(BaseModel):
|
||||||
|
file_id: str
|
||||||
|
remote_path: str
|
||||||
|
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||||
|
expected_revision: str | None = Field(default=None, max_length=500)
|
||||||
|
|
||||||
|
|
||||||
|
class FileConnectorWriteResponse(BaseModel):
|
||||||
|
recovery_operation_id: str
|
||||||
|
status: str
|
||||||
|
replayed: bool = False
|
||||||
|
provider: str
|
||||||
|
remote_path: str
|
||||||
|
revision: str | None = None
|
||||||
|
checksum_sha256: str
|
||||||
|
size_bytes: int
|
||||||
|
|
||||||
|
|
||||||
class BulkDeleteRequest(BaseModel):
|
class BulkDeleteRequest(BaseModel):
|
||||||
file_ids: list[str]
|
file_ids: list[str]
|
||||||
|
|
||||||
@@ -97,6 +568,69 @@ class BulkDeleteResponse(BaseModel):
|
|||||||
deleted_count: int
|
deleted_count: int
|
||||||
|
|
||||||
|
|
||||||
|
class FileLifecycleUpdateRequest(BaseModel):
|
||||||
|
retained_until: datetime | None = None
|
||||||
|
legal_hold: bool = False
|
||||||
|
reason: str = Field(min_length=1, max_length=500)
|
||||||
|
expected_revision: int = Field(ge=1)
|
||||||
|
|
||||||
|
|
||||||
|
class FileRestoreResponse(BaseModel):
|
||||||
|
restored_count: int
|
||||||
|
|
||||||
|
|
||||||
|
class FileFolderRestoreResponse(BaseModel):
|
||||||
|
restored_folders: int
|
||||||
|
restored_files: int
|
||||||
|
|
||||||
|
|
||||||
|
class FilePurgePreviewRequest(BaseModel):
|
||||||
|
file_ids: list[str] = Field(min_length=1, max_length=100)
|
||||||
|
|
||||||
|
|
||||||
|
class FilePurgePreviewItem(BaseModel):
|
||||||
|
file_id: str
|
||||||
|
filename: str
|
||||||
|
lifecycle_revision: int
|
||||||
|
deleted_at: str | None = None
|
||||||
|
retained_until: str | None = None
|
||||||
|
legal_hold: bool = False
|
||||||
|
blockers: list[str] = Field(default_factory=list)
|
||||||
|
blob_ids: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class FilePurgePreviewResponse(BaseModel):
|
||||||
|
preview_sha256: str
|
||||||
|
eligible: bool
|
||||||
|
items: list[FilePurgePreviewItem]
|
||||||
|
|
||||||
|
|
||||||
|
class FilePurgeExecuteRequest(FilePurgePreviewRequest):
|
||||||
|
preview_sha256: str = Field(min_length=64, max_length=64)
|
||||||
|
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||||
|
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||||
|
confirmation: Literal["PURGE"]
|
||||||
|
|
||||||
|
|
||||||
|
class FilePurgeResponse(BaseModel):
|
||||||
|
recovery_operation_id: str
|
||||||
|
status: str
|
||||||
|
replayed: bool = False
|
||||||
|
purged_files: int = 0
|
||||||
|
released_blobs: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
class FileBlobGcRequest(BaseModel):
|
||||||
|
limit: int = Field(default=25, ge=1, le=100)
|
||||||
|
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||||
|
|
||||||
|
|
||||||
|
class FileBlobGcResponse(BaseModel):
|
||||||
|
inspected_blobs: int
|
||||||
|
deleted_blobs: int
|
||||||
|
unresolved_operation_ids: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
class ConflictResolutionRequest(BaseModel):
|
class ConflictResolutionRequest(BaseModel):
|
||||||
target_path: str
|
target_path: str
|
||||||
action: Literal["overwrite", "rename", "skip"]
|
action: Literal["overwrite", "rename", "skip"]
|
||||||
@@ -111,13 +645,27 @@ class FileShareRequest(BaseModel):
|
|||||||
target_type: Literal["user", "group", "campaign", "tenant"]
|
target_type: Literal["user", "group", "campaign", "tenant"]
|
||||||
target_id: str
|
target_id: str
|
||||||
permission: Literal["read", "write", "manage"] = "read"
|
permission: Literal["read", "write", "manage"] = "read"
|
||||||
|
expires_at: datetime | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class BulkFileShareRequest(BaseModel):
|
||||||
|
file_ids: list[str] = Field(default_factory=list, max_length=1000)
|
||||||
|
target_type: Literal["user", "group", "campaign", "tenant"]
|
||||||
|
target_id: str
|
||||||
|
permission: Literal["read", "write", "manage"] = "read"
|
||||||
|
expires_at: datetime | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class BulkFileShareResponse(BaseModel):
|
||||||
|
shares: list[FileShareResponse]
|
||||||
|
shared_count: int
|
||||||
|
|
||||||
|
|
||||||
class RenameRequest(BaseModel):
|
class RenameRequest(BaseModel):
|
||||||
file_ids: list[str] = Field(default_factory=list)
|
file_ids: list[str] = Field(default_factory=list)
|
||||||
folder_paths: list[str] = Field(default_factory=list)
|
folder_paths: list[str] = Field(default_factory=list)
|
||||||
owner_type: Literal["user", "group"] | None = None
|
owner_type: Literal["user", "group"]
|
||||||
owner_id: str | None = None
|
owner_id: str
|
||||||
mode: Literal["direct", "prefix", "suffix", "replace"]
|
mode: Literal["direct", "prefix", "suffix", "replace"]
|
||||||
new_name: str | None = None
|
new_name: str | None = None
|
||||||
find: str | None = None
|
find: str | None = None
|
||||||
|
|||||||
@@ -0,0 +1,329 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from sqlalchemy import func, or_, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.core.events import PlatformEvent
|
||||||
|
from govoplan_core.core.modules import ModuleContext
|
||||||
|
from govoplan_core.core.search import (
|
||||||
|
SearchAuthorizationRequest,
|
||||||
|
SearchBackfillPage,
|
||||||
|
SearchBackfillRequest,
|
||||||
|
SearchDocument,
|
||||||
|
SearchIndexChange,
|
||||||
|
SearchResourceReference,
|
||||||
|
SearchResourceType,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||||
|
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||||
|
|
||||||
|
|
||||||
|
PROVIDER_ID = "files.objects"
|
||||||
|
RESOURCE_MODELS = {
|
||||||
|
"file": FileAsset,
|
||||||
|
"folder": FileFolder,
|
||||||
|
}
|
||||||
|
READ_SCOPE = "files:file:read"
|
||||||
|
ADMIN_SCOPE = "files:file:admin"
|
||||||
|
|
||||||
|
|
||||||
|
class FilesSearchSource:
|
||||||
|
def resource_types(self) -> Sequence[SearchResourceType]:
|
||||||
|
return (
|
||||||
|
SearchResourceType(
|
||||||
|
provider_id=PROVIDER_ID,
|
||||||
|
module_id="files",
|
||||||
|
resource_type="file",
|
||||||
|
label="Files",
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
),
|
||||||
|
SearchResourceType(
|
||||||
|
provider_id=PROVIDER_ID,
|
||||||
|
module_id="files",
|
||||||
|
resource_type="folder",
|
||||||
|
label="File folders",
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def backfill(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
request: SearchBackfillRequest,
|
||||||
|
) -> SearchBackfillPage:
|
||||||
|
db = _session(session)
|
||||||
|
model = _model(request.provider_id, request.resource_type)
|
||||||
|
statement = select(model).where(
|
||||||
|
model.tenant_id == request.tenant_id,
|
||||||
|
model.deleted_at.is_(None),
|
||||||
|
)
|
||||||
|
if request.cursor:
|
||||||
|
statement = statement.where(model.id > request.cursor)
|
||||||
|
rows = list(
|
||||||
|
db.scalars(
|
||||||
|
statement.order_by(model.id).limit(request.limit + 1)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
has_more = len(rows) > request.limit
|
||||||
|
selected = rows[: request.limit]
|
||||||
|
shares = (
|
||||||
|
_shares_by_asset(db, selected)
|
||||||
|
if request.resource_type == "file"
|
||||||
|
else {}
|
||||||
|
)
|
||||||
|
high_watermark = db.scalar(
|
||||||
|
select(func.max(model.updated_at)).where(
|
||||||
|
model.tenant_id == request.tenant_id,
|
||||||
|
model.deleted_at.is_(None),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return SearchBackfillPage(
|
||||||
|
documents=tuple(
|
||||||
|
_document(
|
||||||
|
row,
|
||||||
|
resource_type=request.resource_type,
|
||||||
|
shares=shares.get(row.id, ()),
|
||||||
|
)
|
||||||
|
for row in selected
|
||||||
|
),
|
||||||
|
next_cursor=selected[-1].id if has_more and selected else None,
|
||||||
|
complete=not has_more,
|
||||||
|
high_watermark=(
|
||||||
|
high_watermark.isoformat()
|
||||||
|
if high_watermark is not None
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def authorize(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
requests: Sequence[SearchAuthorizationRequest],
|
||||||
|
) -> Mapping[str, bool]:
|
||||||
|
decisions = {item.reference.key: False for item in requests}
|
||||||
|
if not isinstance(principal, ApiPrincipal) or not (
|
||||||
|
principal.has(READ_SCOPE) or principal.has(ADMIN_SCOPE)
|
||||||
|
):
|
||||||
|
return decisions
|
||||||
|
db = _session(session)
|
||||||
|
for request in requests:
|
||||||
|
reference = request.reference
|
||||||
|
if (
|
||||||
|
reference.tenant_id != principal.tenant_id
|
||||||
|
or reference.module_id != "files"
|
||||||
|
or reference.resource_type not in RESOURCE_MODELS
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
decisions[reference.key] = _can_read(
|
||||||
|
db,
|
||||||
|
principal,
|
||||||
|
reference=reference,
|
||||||
|
)
|
||||||
|
return decisions
|
||||||
|
|
||||||
|
def index_changes_for_event(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
event: PlatformEvent,
|
||||||
|
delivery_key: str,
|
||||||
|
) -> Sequence[SearchIndexChange]:
|
||||||
|
if (
|
||||||
|
event.module_id != "files"
|
||||||
|
or event.tenant is None
|
||||||
|
or event.resource is None
|
||||||
|
or event.resource.id is None
|
||||||
|
or event.resource.type not in RESOURCE_MODELS
|
||||||
|
):
|
||||||
|
return ()
|
||||||
|
db = _session(session)
|
||||||
|
reference = SearchResourceReference(
|
||||||
|
tenant_id=event.tenant.id,
|
||||||
|
module_id="files",
|
||||||
|
resource_type=event.resource.type,
|
||||||
|
resource_id=event.resource.id,
|
||||||
|
)
|
||||||
|
model = RESOURCE_MODELS[event.resource.type]
|
||||||
|
row = db.get(model, event.resource.id)
|
||||||
|
deleted = row is None or row.tenant_id != event.tenant.id or row.deleted_at is not None
|
||||||
|
cursor = event.event_id
|
||||||
|
document = None
|
||||||
|
if not deleted:
|
||||||
|
shares = (
|
||||||
|
tuple(_active_shares(db, row.id))
|
||||||
|
if event.resource.type == "file"
|
||||||
|
else ()
|
||||||
|
)
|
||||||
|
document = _document(
|
||||||
|
row,
|
||||||
|
resource_type=event.resource.type,
|
||||||
|
shares=shares,
|
||||||
|
change_cursor=cursor,
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
SearchIndexChange(
|
||||||
|
change_id=f"{delivery_key}:{PROVIDER_ID}:{event.resource.type}",
|
||||||
|
provider_id=PROVIDER_ID,
|
||||||
|
kind="delete" if deleted else "upsert",
|
||||||
|
reference=reference,
|
||||||
|
source_revision=(
|
||||||
|
document.source_revision if document is not None else cursor
|
||||||
|
),
|
||||||
|
cursor=cursor,
|
||||||
|
document=document,
|
||||||
|
occurred_at=event.occurred_at,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def create_files_search_source(_context: ModuleContext) -> FilesSearchSource:
|
||||||
|
return FilesSearchSource()
|
||||||
|
|
||||||
|
|
||||||
|
def _model(provider_id: str, resource_type: str):
|
||||||
|
if provider_id != PROVIDER_ID or resource_type not in RESOURCE_MODELS:
|
||||||
|
raise ValueError("Unsupported Files search source.")
|
||||||
|
return RESOURCE_MODELS[resource_type]
|
||||||
|
|
||||||
|
|
||||||
|
def _document(
|
||||||
|
row: FileAsset | FileFolder,
|
||||||
|
*,
|
||||||
|
resource_type: str,
|
||||||
|
shares: Sequence[FileShare] = (),
|
||||||
|
change_cursor: str | None = None,
|
||||||
|
) -> SearchDocument:
|
||||||
|
is_file = isinstance(row, FileAsset)
|
||||||
|
title = row.filename if is_file else (row.path.rsplit("/", 1)[-1] or row.path)
|
||||||
|
path = row.display_path if is_file else row.path
|
||||||
|
owner_id = row.owner_user_id if row.owner_type == "user" else row.owner_group_id
|
||||||
|
tokens = [f"scope:{READ_SCOPE}", f"scope:{ADMIN_SCOPE}"]
|
||||||
|
if owner_id:
|
||||||
|
tokens.append(
|
||||||
|
f"membership:{owner_id}"
|
||||||
|
if row.owner_type == "user"
|
||||||
|
else f"group:{owner_id}"
|
||||||
|
)
|
||||||
|
for share in shares:
|
||||||
|
prefix = "membership" if share.target_type == "user" else share.target_type
|
||||||
|
if prefix in {"membership", "group", "tenant"}:
|
||||||
|
tokens.append(f"{prefix}:{share.target_id}")
|
||||||
|
updated_at = row.updated_at or row.created_at
|
||||||
|
revision = (
|
||||||
|
f"{row.current_version_id or 'none'}:{updated_at.isoformat()}"
|
||||||
|
if is_file
|
||||||
|
else updated_at.isoformat()
|
||||||
|
)
|
||||||
|
return SearchDocument(
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
module_id="files",
|
||||||
|
provider_id=PROVIDER_ID,
|
||||||
|
resource_type=resource_type,
|
||||||
|
resource_id=row.id,
|
||||||
|
title=title,
|
||||||
|
url=f"/files?{resource_type}Id={quote(row.id, safe='')}",
|
||||||
|
summary=((row.description or "") if is_file else path)[:4000] or None,
|
||||||
|
body=" ".join(
|
||||||
|
value for value in (path, row.description if is_file else None) if value
|
||||||
|
)[:200_000],
|
||||||
|
keywords=(path[:200], row.owner_type[:200]),
|
||||||
|
visibility="restricted",
|
||||||
|
acl_tokens=tuple(dict.fromkeys(tokens)),
|
||||||
|
metadata={
|
||||||
|
"path": path,
|
||||||
|
"owner_type": row.owner_type,
|
||||||
|
"current_version_id": row.current_version_id if is_file else None,
|
||||||
|
},
|
||||||
|
source_revision=revision,
|
||||||
|
change_cursor=change_cursor,
|
||||||
|
source_updated_at=updated_at,
|
||||||
|
requires_authorization_recheck=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _can_read(
|
||||||
|
session: Session,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
*,
|
||||||
|
reference: SearchResourceReference,
|
||||||
|
) -> bool:
|
||||||
|
model = RESOURCE_MODELS[reference.resource_type]
|
||||||
|
row = session.get(model, reference.resource_id)
|
||||||
|
if row is None or row.tenant_id != principal.tenant_id or row.deleted_at is not None:
|
||||||
|
return False
|
||||||
|
if principal.has(ADMIN_SCOPE):
|
||||||
|
return True
|
||||||
|
user_id = str(getattr(principal.user, "id", "") or principal.membership_id or "")
|
||||||
|
if row.owner_type == "user" and row.owner_user_id == user_id:
|
||||||
|
return True
|
||||||
|
if row.owner_type == "group" and row.owner_group_id in principal.group_ids:
|
||||||
|
return True
|
||||||
|
if reference.resource_type != "file":
|
||||||
|
return False
|
||||||
|
target_clauses = [
|
||||||
|
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||||
|
(FileShare.target_type == "tenant")
|
||||||
|
& (FileShare.target_id == principal.tenant_id),
|
||||||
|
]
|
||||||
|
if principal.group_ids:
|
||||||
|
target_clauses.append(
|
||||||
|
(FileShare.target_type == "group")
|
||||||
|
& (FileShare.target_id.in_(tuple(principal.group_ids)))
|
||||||
|
)
|
||||||
|
return session.scalar(
|
||||||
|
select(FileShare.id).where(
|
||||||
|
FileShare.tenant_id == principal.tenant_id,
|
||||||
|
FileShare.file_asset_id == row.id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
or_(*target_clauses),
|
||||||
|
).limit(1)
|
||||||
|
) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _active_shares(session: Session, asset_id: str) -> Sequence[FileShare]:
|
||||||
|
return tuple(
|
||||||
|
session.scalars(
|
||||||
|
select(FileShare).where(
|
||||||
|
FileShare.file_asset_id == asset_id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _shares_by_asset(
|
||||||
|
session: Session,
|
||||||
|
rows: Sequence[FileAsset | FileFolder],
|
||||||
|
) -> dict[str, tuple[FileShare, ...]]:
|
||||||
|
asset_ids = [row.id for row in rows if isinstance(row, FileAsset)]
|
||||||
|
grouped: dict[str, list[FileShare]] = {asset_id: [] for asset_id in asset_ids}
|
||||||
|
if not asset_ids:
|
||||||
|
return {}
|
||||||
|
for share in session.scalars(
|
||||||
|
select(FileShare).where(
|
||||||
|
FileShare.file_asset_id.in_(asset_ids),
|
||||||
|
effective_file_share_clause(),
|
||||||
|
)
|
||||||
|
):
|
||||||
|
grouped.setdefault(share.file_asset_id, []).append(share)
|
||||||
|
return {key: tuple(value) for key, value in grouped.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def _session(value: object) -> Session:
|
||||||
|
if not isinstance(value, Session):
|
||||||
|
raise TypeError("Files search requires a SQLAlchemy session.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"FilesSearchSource",
|
||||||
|
"PROVIDER_ID",
|
||||||
|
"create_files_search_source",
|
||||||
|
]
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Query and response assembly services used by Files routes."""
|
||||||
@@ -0,0 +1,292 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Literal
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||||
|
from govoplan_core.core.change_sequence import (
|
||||||
|
ChangeSequenceEntry,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.change_tracking import (
|
||||||
|
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||||
|
FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||||
|
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||||
|
FILES_CONNECTOR_SPACES_COLLECTION,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileConnectorCredentialResponse,
|
||||||
|
FileConnectorSettingsDeltaResponse,
|
||||||
|
FileConnectorPolicyResponse,
|
||||||
|
FileConnectorProfileResponse,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorSpace
|
||||||
|
from govoplan_files.backend.storage.connector_credential_store import (
|
||||||
|
ConnectorCredential,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
from govoplan_files.backend.storage.connector_policy_store import (
|
||||||
|
connector_policy_response,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||||
|
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||||
|
FILES_CONNECTOR_SPACE_RESOURCE,
|
||||||
|
_can_read_disabled_connector_profiles,
|
||||||
|
_connector_deleted_item,
|
||||||
|
_connector_space_response,
|
||||||
|
_ensure_campaign_file_access,
|
||||||
|
_file_connector_settings_response_watermark,
|
||||||
|
_file_connector_settings_watermark,
|
||||||
|
_visible_connector_credentials,
|
||||||
|
_visible_connector_profiles,
|
||||||
|
_visible_connector_spaces,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _full_file_connector_settings_delta_response(
|
||||||
|
session: Session,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
*,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
provider: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
include_disabled: bool,
|
||||||
|
include_inactive: bool,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
) -> FileConnectorSettingsDeltaResponse:
|
||||||
|
if campaign_id:
|
||||||
|
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||||
|
profiles = _visible_connector_profiles(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled
|
||||||
|
and _can_read_disabled_connector_profiles(principal),
|
||||||
|
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||||
|
include_effective_policy=False,
|
||||||
|
)
|
||||||
|
credentials = _visible_connector_credentials(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
provider=provider,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
)
|
||||||
|
spaces = _visible_connector_spaces(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
include_inactive=include_inactive,
|
||||||
|
)
|
||||||
|
return FileConnectorSettingsDeltaResponse(
|
||||||
|
profiles=[
|
||||||
|
FileConnectorProfileResponse(**profile.to_response())
|
||||||
|
for profile in profiles
|
||||||
|
],
|
||||||
|
credentials=[
|
||||||
|
FileConnectorCredentialResponse(**credential.to_response())
|
||||||
|
for credential in credentials
|
||||||
|
],
|
||||||
|
spaces=[_connector_space_response(space) for space in spaces],
|
||||||
|
policy=FileConnectorPolicyResponse(
|
||||||
|
**connector_policy_response(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
changed_sections=["profiles", "credentials", "spaces", "policy"],
|
||||||
|
deleted=[],
|
||||||
|
watermark=_file_connector_settings_watermark(
|
||||||
|
session, tenant_id=principal.tenant_id
|
||||||
|
),
|
||||||
|
has_more=False,
|
||||||
|
full=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _changed_file_connector_setting_ids(
|
||||||
|
entries: list[ChangeSequenceEntry],
|
||||||
|
) -> tuple[set[str], set[str], set[str], bool]:
|
||||||
|
changed_profile_ids = {
|
||||||
|
entry.resource_id
|
||||||
|
for entry in entries
|
||||||
|
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION
|
||||||
|
and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||||
|
}
|
||||||
|
changed_credential_ids = {
|
||||||
|
entry.resource_id
|
||||||
|
for entry in entries
|
||||||
|
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION
|
||||||
|
and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||||
|
}
|
||||||
|
changed_space_ids = {
|
||||||
|
entry.resource_id
|
||||||
|
for entry in entries
|
||||||
|
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION
|
||||||
|
and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||||
|
}
|
||||||
|
policy_changed = any(
|
||||||
|
entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
changed_profile_ids,
|
||||||
|
changed_credential_ids,
|
||||||
|
changed_space_ids,
|
||||||
|
policy_changed,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _file_connector_settings_changed_sections(
|
||||||
|
*,
|
||||||
|
changed_profile_ids: set[str],
|
||||||
|
changed_credential_ids: set[str],
|
||||||
|
changed_space_ids: set[str],
|
||||||
|
policy_changed: bool,
|
||||||
|
profiles: list[ConnectorProfile],
|
||||||
|
) -> list[str]:
|
||||||
|
changed_sections = []
|
||||||
|
if changed_profile_ids or any(
|
||||||
|
profile.credential_profile_id
|
||||||
|
and profile.credential_profile_id in changed_credential_ids
|
||||||
|
for profile in profiles
|
||||||
|
):
|
||||||
|
changed_sections.append("profiles")
|
||||||
|
if changed_credential_ids:
|
||||||
|
changed_sections.append("credentials")
|
||||||
|
if changed_space_ids:
|
||||||
|
changed_sections.append("spaces")
|
||||||
|
if policy_changed:
|
||||||
|
changed_sections.append("policy")
|
||||||
|
return changed_sections
|
||||||
|
|
||||||
|
|
||||||
|
def _file_connector_settings_deleted_items(
|
||||||
|
entries: list[ChangeSequenceEntry],
|
||||||
|
*,
|
||||||
|
visible_profiles: dict[str, ConnectorProfile],
|
||||||
|
visible_credentials: dict[str, ConnectorCredential],
|
||||||
|
visible_spaces: dict[str, FileConnectorSpace],
|
||||||
|
) -> list[DeltaDeletedItem]:
|
||||||
|
return [
|
||||||
|
_connector_deleted_item(entry)
|
||||||
|
for entry in entries
|
||||||
|
if (
|
||||||
|
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||||
|
and entry.resource_id not in visible_profiles
|
||||||
|
)
|
||||||
|
or (
|
||||||
|
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||||
|
and entry.resource_id not in visible_credentials
|
||||||
|
)
|
||||||
|
or (
|
||||||
|
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||||
|
and entry.resource_id not in visible_spaces
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _incremental_file_connector_settings_delta_response(
|
||||||
|
session: Session,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
*,
|
||||||
|
entries: list[ChangeSequenceEntry],
|
||||||
|
has_more: bool,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
provider: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
include_disabled: bool,
|
||||||
|
include_inactive: bool,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
) -> FileConnectorSettingsDeltaResponse:
|
||||||
|
changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed = (
|
||||||
|
_changed_file_connector_setting_ids(entries)
|
||||||
|
)
|
||||||
|
profiles = _visible_connector_profiles(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
provider=provider,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
include_disabled=include_disabled
|
||||||
|
and _can_read_disabled_connector_profiles(principal),
|
||||||
|
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||||
|
include_effective_policy=False,
|
||||||
|
)
|
||||||
|
visible_profiles = {
|
||||||
|
profile.id: profile
|
||||||
|
for profile in profiles
|
||||||
|
if profile.id in changed_profile_ids
|
||||||
|
or (
|
||||||
|
profile.credential_profile_id
|
||||||
|
and profile.credential_profile_id in changed_credential_ids
|
||||||
|
)
|
||||||
|
}
|
||||||
|
credentials = _visible_connector_credentials(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
provider=provider,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
)
|
||||||
|
visible_credentials = {
|
||||||
|
credential.id: credential
|
||||||
|
for credential in credentials
|
||||||
|
if credential.id in changed_credential_ids
|
||||||
|
}
|
||||||
|
spaces = _visible_connector_spaces(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
include_inactive=include_inactive,
|
||||||
|
)
|
||||||
|
visible_spaces = {
|
||||||
|
space.id: space for space in spaces if space.id in changed_space_ids
|
||||||
|
}
|
||||||
|
return FileConnectorSettingsDeltaResponse(
|
||||||
|
profiles=[
|
||||||
|
FileConnectorProfileResponse(**profile.to_response())
|
||||||
|
for profile in visible_profiles.values()
|
||||||
|
],
|
||||||
|
credentials=[
|
||||||
|
FileConnectorCredentialResponse(**credential.to_response())
|
||||||
|
for credential in visible_credentials.values()
|
||||||
|
],
|
||||||
|
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
|
||||||
|
policy=FileConnectorPolicyResponse(
|
||||||
|
**connector_policy_response(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if policy_changed
|
||||||
|
else None,
|
||||||
|
changed_sections=_file_connector_settings_changed_sections(
|
||||||
|
changed_profile_ids=changed_profile_ids,
|
||||||
|
changed_credential_ids=changed_credential_ids,
|
||||||
|
changed_space_ids=changed_space_ids,
|
||||||
|
policy_changed=policy_changed,
|
||||||
|
profiles=profiles,
|
||||||
|
),
|
||||||
|
deleted=_file_connector_settings_deleted_items(
|
||||||
|
entries,
|
||||||
|
visible_profiles=visible_profiles,
|
||||||
|
visible_credentials=visible_credentials,
|
||||||
|
visible_spaces=visible_spaces,
|
||||||
|
),
|
||||||
|
watermark=_file_connector_settings_response_watermark(
|
||||||
|
session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more
|
||||||
|
),
|
||||||
|
has_more=has_more,
|
||||||
|
full=False,
|
||||||
|
)
|
||||||
@@ -0,0 +1,620 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Literal
|
||||||
|
from fastapi import HTTPException, status
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||||
|
from govoplan_core.core.change_sequence import (
|
||||||
|
ChangeSequenceEntry,
|
||||||
|
decode_sequence_watermark,
|
||||||
|
encode_sequence_watermark,
|
||||||
|
max_sequence_id,
|
||||||
|
sequence_entries_since,
|
||||||
|
sequence_watermark_is_expired,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.pagination import (
|
||||||
|
KeysetCursorError,
|
||||||
|
decode_keyset_cursor,
|
||||||
|
encode_keyset_cursor,
|
||||||
|
keyset_query_fingerprint,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.change_tracking import (
|
||||||
|
FILES_ASSETS_COLLECTION,
|
||||||
|
FILES_FOLDERS_COLLECTION,
|
||||||
|
FILES_MODULE_ID,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.schemas import (
|
||||||
|
FileDeltaResponse,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||||
|
from govoplan_files.backend.storage.access import user_group_ids
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
list_assets_for_user,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.folders import list_folders_for_user
|
||||||
|
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||||
|
|
||||||
|
|
||||||
|
from govoplan_files.backend.route_support import (
|
||||||
|
_asset_list_response,
|
||||||
|
_folder_response,
|
||||||
|
_is_admin,
|
||||||
|
)
|
||||||
|
|
||||||
|
_FILES_DELTA_COLLECTIONS = (FILES_ASSETS_COLLECTION, FILES_FOLDERS_COLLECTION)
|
||||||
|
FILES_LIST_CURSOR_SCOPE = "files.list.v1"
|
||||||
|
FOLDERS_LIST_CURSOR_SCOPE = "files.folders.list.v1"
|
||||||
|
DEFAULT_FILE_LIST_PAGE_SIZE = 500
|
||||||
|
|
||||||
|
|
||||||
|
def _cursor_http_error(exc: Exception) -> HTTPException:
|
||||||
|
return HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc))
|
||||||
|
|
||||||
|
|
||||||
|
def _cursor_page_size(
|
||||||
|
scope: str, cursor: str | None, explicit_page_size: int | None
|
||||||
|
) -> int | None:
|
||||||
|
if explicit_page_size is not None:
|
||||||
|
return explicit_page_size
|
||||||
|
if not cursor:
|
||||||
|
return DEFAULT_FILE_LIST_PAGE_SIZE
|
||||||
|
try:
|
||||||
|
values = decode_keyset_cursor(scope, cursor)
|
||||||
|
except KeysetCursorError as exc:
|
||||||
|
raise _cursor_http_error(exc) from exc
|
||||||
|
raw_page_size = values.get("page_size")
|
||||||
|
if not isinstance(raw_page_size, int) or raw_page_size < 1 or raw_page_size > 1000:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||||
|
)
|
||||||
|
return raw_page_size
|
||||||
|
|
||||||
|
|
||||||
|
def _files_list_fingerprint(
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
*,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
campaign_usage: Literal["linked", "unlinked"] | None,
|
||||||
|
audit_relevant: bool | None,
|
||||||
|
page_size: int,
|
||||||
|
) -> str:
|
||||||
|
return keyset_query_fingerprint(
|
||||||
|
FILES_LIST_CURSOR_SCOPE,
|
||||||
|
{
|
||||||
|
"tenant_id": principal.tenant_id,
|
||||||
|
"actor": "admin" if _is_admin(principal) else principal.user.id,
|
||||||
|
"owner_type": owner_type,
|
||||||
|
"owner_id": owner_id,
|
||||||
|
"campaign_id": campaign_id,
|
||||||
|
"path_prefix": path_prefix or "",
|
||||||
|
"campaign_usage": campaign_usage or "",
|
||||||
|
"audit_relevant": audit_relevant,
|
||||||
|
"sort": "display_path.asc,updated_at.desc,id.asc",
|
||||||
|
"page_size": page_size,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _folders_list_fingerprint(
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
*,
|
||||||
|
owner_type: Literal["user", "group"],
|
||||||
|
owner_id: str,
|
||||||
|
page_size: int,
|
||||||
|
) -> str:
|
||||||
|
return keyset_query_fingerprint(
|
||||||
|
FOLDERS_LIST_CURSOR_SCOPE,
|
||||||
|
{
|
||||||
|
"tenant_id": principal.tenant_id,
|
||||||
|
"actor": "admin" if _is_admin(principal) else principal.user.id,
|
||||||
|
"owner_type": owner_type,
|
||||||
|
"owner_id": owner_id,
|
||||||
|
"sort": "path.asc,id.asc",
|
||||||
|
"page_size": page_size,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _file_cursor_values(
|
||||||
|
cursor: str | None, *, fingerprint: str
|
||||||
|
) -> tuple[str | None, datetime | None, str | None]:
|
||||||
|
try:
|
||||||
|
values = decode_keyset_cursor(
|
||||||
|
FILES_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
|
||||||
|
)
|
||||||
|
except KeysetCursorError as exc:
|
||||||
|
raise _cursor_http_error(exc) from exc
|
||||||
|
if values is None:
|
||||||
|
return None, None, None
|
||||||
|
display_path = values.get("display_path")
|
||||||
|
updated_at = values.get("updated_at")
|
||||||
|
asset_id = values.get("id")
|
||||||
|
if (
|
||||||
|
not isinstance(display_path, str)
|
||||||
|
or not isinstance(updated_at, str)
|
||||||
|
or not isinstance(asset_id, str)
|
||||||
|
):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
parsed_updated_at = datetime.fromisoformat(updated_at)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||||
|
) from exc
|
||||||
|
return display_path, parsed_updated_at, asset_id
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_cursor_values(
|
||||||
|
cursor: str | None, *, fingerprint: str
|
||||||
|
) -> tuple[str | None, str | None]:
|
||||||
|
try:
|
||||||
|
values = decode_keyset_cursor(
|
||||||
|
FOLDERS_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
|
||||||
|
)
|
||||||
|
except KeysetCursorError as exc:
|
||||||
|
raise _cursor_http_error(exc) from exc
|
||||||
|
if values is None:
|
||||||
|
return None, None
|
||||||
|
folder_path = values.get("path")
|
||||||
|
folder_id = values.get("id")
|
||||||
|
if not isinstance(folder_path, str) or not isinstance(folder_id, str):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||||
|
)
|
||||||
|
return folder_path, folder_id
|
||||||
|
|
||||||
|
|
||||||
|
def _next_file_list_cursor(
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
assets: list[FileAsset],
|
||||||
|
*,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
campaign_usage: Literal["linked", "unlinked"] | None,
|
||||||
|
audit_relevant: bool | None,
|
||||||
|
page_size: int,
|
||||||
|
has_more: bool,
|
||||||
|
) -> str | None:
|
||||||
|
if not has_more or not assets:
|
||||||
|
return None
|
||||||
|
last = assets[-1]
|
||||||
|
return encode_keyset_cursor(
|
||||||
|
FILES_LIST_CURSOR_SCOPE,
|
||||||
|
fingerprint=_files_list_fingerprint(
|
||||||
|
principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
page_size=page_size,
|
||||||
|
),
|
||||||
|
values={
|
||||||
|
"display_path": last.display_path,
|
||||||
|
"updated_at": last.updated_at,
|
||||||
|
"id": last.id,
|
||||||
|
"page_size": page_size,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _next_folder_list_cursor(
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
folders: list[FileFolder],
|
||||||
|
*,
|
||||||
|
owner_type: Literal["user", "group"],
|
||||||
|
owner_id: str,
|
||||||
|
page_size: int,
|
||||||
|
has_more: bool,
|
||||||
|
) -> str | None:
|
||||||
|
if not has_more or not folders:
|
||||||
|
return None
|
||||||
|
last = folders[-1]
|
||||||
|
return encode_keyset_cursor(
|
||||||
|
FOLDERS_LIST_CURSOR_SCOPE,
|
||||||
|
fingerprint=_folders_list_fingerprint(
|
||||||
|
principal, owner_type=owner_type, owner_id=owner_id, page_size=page_size
|
||||||
|
),
|
||||||
|
values={"path": last.path, "id": last.id, "page_size": page_size},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _files_delta_watermark(session: Session, tenant_id: str) -> str:
|
||||||
|
return encode_sequence_watermark(
|
||||||
|
max_sequence_id(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
module_id=FILES_MODULE_ID,
|
||||||
|
collections=_FILES_DELTA_COLLECTIONS,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _full_file_delta_response(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
) -> FileDeltaResponse:
|
||||||
|
assets = list_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
folders = _visible_folders_for_delta(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
)
|
||||||
|
return FileDeltaResponse(
|
||||||
|
files=_asset_list_response(session, assets, include_shares=True),
|
||||||
|
folders=[_folder_response(folder) for folder in folders],
|
||||||
|
deleted=[],
|
||||||
|
watermark=_files_delta_watermark(session, principal.tenant_id),
|
||||||
|
has_more=False,
|
||||||
|
full=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _visible_folders_for_delta(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
) -> list[FileFolder]:
|
||||||
|
if not owner_type or not owner_id:
|
||||||
|
return []
|
||||||
|
folders = list_folders_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
if not path_prefix:
|
||||||
|
return folders
|
||||||
|
normalized = path_prefix.strip().strip("/")
|
||||||
|
if not normalized:
|
||||||
|
return folders
|
||||||
|
return [
|
||||||
|
folder
|
||||||
|
for folder in folders
|
||||||
|
if folder.path == normalized or folder.path.startswith(f"{normalized}/")
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_path_matches(
|
||||||
|
entry_payload: dict[str, object], path_prefix: str | None
|
||||||
|
) -> bool:
|
||||||
|
if not path_prefix:
|
||||||
|
return True
|
||||||
|
normalized = path_prefix.strip().strip("/")
|
||||||
|
if not normalized:
|
||||||
|
return True
|
||||||
|
for key in ("path", "previous_path"):
|
||||||
|
value = entry_payload.get(key)
|
||||||
|
if isinstance(value, str) and (
|
||||||
|
value == normalized or value.startswith(f"{normalized}/")
|
||||||
|
):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_owner_matches(
|
||||||
|
entry_payload: dict[str, object],
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
) -> bool:
|
||||||
|
if not owner_type or not owner_id:
|
||||||
|
return True
|
||||||
|
return (
|
||||||
|
entry_payload.get("owner_type") == owner_type
|
||||||
|
and entry_payload.get("owner_id") == owner_id
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_campaign_matches(session: Session, entry, campaign_id: str | None) -> bool:
|
||||||
|
if not campaign_id:
|
||||||
|
return True
|
||||||
|
payload = entry.payload or {}
|
||||||
|
if (
|
||||||
|
payload.get("share_target_type") == "campaign"
|
||||||
|
and payload.get("share_target_id") == campaign_id
|
||||||
|
):
|
||||||
|
return True
|
||||||
|
if entry.resource_type != "file":
|
||||||
|
return False
|
||||||
|
return (
|
||||||
|
session.query(FileShare)
|
||||||
|
.filter(
|
||||||
|
FileShare.tenant_id == entry.tenant_id,
|
||||||
|
FileShare.file_asset_id == entry.resource_id,
|
||||||
|
FileShare.target_type == "campaign",
|
||||||
|
FileShare.target_id == campaign_id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _principal_group_ids_for_delta(
|
||||||
|
session: Session, principal: ApiPrincipal
|
||||||
|
) -> set[str]:
|
||||||
|
cache = session.info.setdefault("files_delta_group_ids", {})
|
||||||
|
key = (principal.tenant_id, principal.user.id)
|
||||||
|
if key not in cache:
|
||||||
|
cache[key] = set(
|
||||||
|
user_group_ids(
|
||||||
|
session, tenant_id=principal.tenant_id, user_id=principal.user.id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return cache[key]
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_subject_matches_principal(
|
||||||
|
session: Session, principal: ApiPrincipal, entry_payload: dict[str, object]
|
||||||
|
) -> bool:
|
||||||
|
if _is_admin(principal):
|
||||||
|
return True
|
||||||
|
owner_type = entry_payload.get("owner_type")
|
||||||
|
owner_id = entry_payload.get("owner_id")
|
||||||
|
if owner_type == "user" and owner_id == principal.user.id:
|
||||||
|
return True
|
||||||
|
if owner_type == "group" and isinstance(owner_id, str):
|
||||||
|
if owner_id in _principal_group_ids_for_delta(session, principal):
|
||||||
|
return True
|
||||||
|
share_target_type = entry_payload.get("share_target_type")
|
||||||
|
share_target_id = entry_payload.get("share_target_id")
|
||||||
|
if share_target_type == "user" and share_target_id == principal.user.id:
|
||||||
|
return True
|
||||||
|
if share_target_type == "tenant" and share_target_id == principal.tenant_id:
|
||||||
|
return True
|
||||||
|
if share_target_type == "group" and isinstance(share_target_id, str):
|
||||||
|
if share_target_id in _principal_group_ids_for_delta(session, principal):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_matches_delta_scope(
|
||||||
|
session: Session,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
entry,
|
||||||
|
*,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
) -> bool:
|
||||||
|
payload = entry.payload or {}
|
||||||
|
if (
|
||||||
|
not owner_type
|
||||||
|
and not campaign_id
|
||||||
|
and not _entry_subject_matches_principal(session, principal, payload)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
return (
|
||||||
|
_entry_owner_matches(payload, owner_type, owner_id)
|
||||||
|
and _entry_path_matches(payload, path_prefix)
|
||||||
|
and _entry_campaign_matches(session, entry, campaign_id)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_files_delta_watermark(since: str) -> int:
|
||||||
|
try:
|
||||||
|
return decode_sequence_watermark(since)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _changed_delta_resource_ids(
|
||||||
|
entries: list[ChangeSequenceEntry],
|
||||||
|
) -> tuple[list[str], list[str]]:
|
||||||
|
file_ids = list(
|
||||||
|
dict.fromkeys(
|
||||||
|
entry.resource_id for entry in entries if entry.resource_type == "file"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
folder_ids = list(
|
||||||
|
dict.fromkeys(
|
||||||
|
entry.resource_id for entry in entries if entry.resource_type == "folder"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return file_ids, folder_ids
|
||||||
|
|
||||||
|
|
||||||
|
def _visible_assets_for_delta(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
changed_file_ids: list[str],
|
||||||
|
) -> dict[str, FileAsset]:
|
||||||
|
return {
|
||||||
|
asset.id: asset
|
||||||
|
for asset in list_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
user_id=principal.user.id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
is_admin=_is_admin(principal),
|
||||||
|
)
|
||||||
|
if asset.id in changed_file_ids
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _changed_visible_folders_for_delta(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
changed_folder_ids: list[str],
|
||||||
|
) -> dict[str, FileFolder]:
|
||||||
|
return {
|
||||||
|
folder.id: folder
|
||||||
|
for folder in _visible_folders_for_delta(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
)
|
||||||
|
if folder.id in changed_folder_ids
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _deleted_delta_items(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
entries: list[ChangeSequenceEntry],
|
||||||
|
visible_assets: dict[str, FileAsset],
|
||||||
|
visible_folders: dict[str, FileFolder],
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
) -> list[DeltaDeletedItem]:
|
||||||
|
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
|
||||||
|
for entry in entries:
|
||||||
|
if entry.resource_type == "file" and entry.resource_id in visible_assets:
|
||||||
|
continue
|
||||||
|
if entry.resource_type == "folder" and entry.resource_id in visible_folders:
|
||||||
|
continue
|
||||||
|
if not _entry_matches_delta_scope(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
entry,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
|
||||||
|
id=entry.resource_id,
|
||||||
|
resource_type=entry.resource_type,
|
||||||
|
revision=encode_sequence_watermark(entry.id),
|
||||||
|
deleted_at=entry.created_at if entry.operation == "deleted" else None,
|
||||||
|
)
|
||||||
|
return list(deleted.values())
|
||||||
|
|
||||||
|
|
||||||
|
def _files_delta_response(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
principal: ApiPrincipal,
|
||||||
|
owner_type: Literal["user", "group"] | None,
|
||||||
|
owner_id: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
path_prefix: str | None,
|
||||||
|
since: str,
|
||||||
|
limit: int,
|
||||||
|
) -> FileDeltaResponse:
|
||||||
|
since_sequence = _decode_files_delta_watermark(since)
|
||||||
|
if sequence_watermark_is_expired(
|
||||||
|
session,
|
||||||
|
since=since_sequence,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
module_id=FILES_MODULE_ID,
|
||||||
|
collections=_FILES_DELTA_COLLECTIONS,
|
||||||
|
):
|
||||||
|
return _full_file_delta_response(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
)
|
||||||
|
|
||||||
|
entries_plus_one = sequence_entries_since(
|
||||||
|
session,
|
||||||
|
since=since_sequence,
|
||||||
|
tenant_id=principal.tenant_id,
|
||||||
|
module_id=FILES_MODULE_ID,
|
||||||
|
collections=_FILES_DELTA_COLLECTIONS,
|
||||||
|
limit=limit + 1,
|
||||||
|
)
|
||||||
|
has_more = len(entries_plus_one) > limit
|
||||||
|
entries = entries_plus_one[:limit]
|
||||||
|
|
||||||
|
changed_file_ids, changed_folder_ids = _changed_delta_resource_ids(entries)
|
||||||
|
visible_assets = _visible_assets_for_delta(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
changed_file_ids=changed_file_ids,
|
||||||
|
)
|
||||||
|
visible_folders = _changed_visible_folders_for_delta(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
changed_folder_ids=changed_folder_ids,
|
||||||
|
)
|
||||||
|
deleted = _deleted_delta_items(
|
||||||
|
session,
|
||||||
|
principal=principal,
|
||||||
|
entries=entries,
|
||||||
|
visible_assets=visible_assets,
|
||||||
|
visible_folders=visible_folders,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
)
|
||||||
|
|
||||||
|
watermark = (
|
||||||
|
encode_sequence_watermark(entries[-1].id)
|
||||||
|
if has_more and entries
|
||||||
|
else _files_delta_watermark(session, principal.tenant_id)
|
||||||
|
)
|
||||||
|
return FileDeltaResponse(
|
||||||
|
files=_asset_list_response(
|
||||||
|
session, list(visible_assets.values()), include_shares=True
|
||||||
|
),
|
||||||
|
folders=[_folder_response(folder) for folder in visible_folders.values()],
|
||||||
|
deleted=deleted,
|
||||||
|
watermark=watermark,
|
||||||
|
has_more=has_more,
|
||||||
|
full=False,
|
||||||
|
)
|
||||||
@@ -2,33 +2,50 @@ from __future__ import annotations
|
|||||||
|
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_core.db.models import Group, UserGroupMembership
|
from govoplan_core.core.access import (
|
||||||
|
CAPABILITY_ACCESS_DIRECTORY,
|
||||||
|
AccessDirectory,
|
||||||
|
GroupRef,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.runtime import get_registry
|
||||||
from govoplan_files.backend.storage.common import FileStorageError
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
|
||||||
|
|
||||||
|
def _access_directory() -> AccessDirectory:
|
||||||
|
registry = get_registry()
|
||||||
|
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||||
|
raise FileStorageError("Access directory capability is not configured")
|
||||||
|
capability = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||||
|
if not isinstance(capability, AccessDirectory):
|
||||||
|
raise FileStorageError("Access directory capability is invalid")
|
||||||
|
return capability
|
||||||
|
|
||||||
|
|
||||||
|
def group_refs_for_ids(*, tenant_id: str, group_ids: list[str]) -> list[GroupRef]:
|
||||||
|
if not group_ids:
|
||||||
|
return []
|
||||||
|
groups = _access_directory().get_groups(group_ids)
|
||||||
|
return sorted(
|
||||||
|
[group for group in groups.values() if group.tenant_id == tenant_id],
|
||||||
|
key=lambda group: group.name.casefold(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def user_group_ids(session: Session, *, tenant_id: str, user_id: str, include_admin_groups: bool = False) -> list[str]:
|
def user_group_ids(session: Session, *, tenant_id: str, user_id: str, include_admin_groups: bool = False) -> list[str]:
|
||||||
|
del session
|
||||||
|
directory = _access_directory()
|
||||||
if include_admin_groups:
|
if include_admin_groups:
|
||||||
return [row.id for row in session.query(Group).filter(Group.tenant_id == tenant_id).order_by(Group.name.asc()).all()]
|
return [group.id for group in directory.groups_for_tenant(tenant_id)]
|
||||||
return [
|
return [group.id for group in directory.groups_for_user(user_id, tenant_id=tenant_id)]
|
||||||
row.group_id
|
|
||||||
for row in session.query(UserGroupMembership)
|
|
||||||
.filter(UserGroupMembership.tenant_id == tenant_id, UserGroupMembership.user_id == user_id)
|
|
||||||
.all()
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def ensure_group_access(session: Session, *, tenant_id: str, group_id: str, user_id: str, is_admin: bool = False) -> None:
|
def ensure_group_access(session: Session, *, tenant_id: str, group_id: str, user_id: str, is_admin: bool = False) -> None:
|
||||||
group = session.get(Group, group_id)
|
group = _access_directory().get_group(group_id)
|
||||||
if not group or group.tenant_id != tenant_id:
|
if not group or group.tenant_id != tenant_id:
|
||||||
raise FileStorageError("Group not found")
|
raise FileStorageError("Group not found")
|
||||||
if is_admin:
|
if is_admin:
|
||||||
return
|
return
|
||||||
membership = (
|
if group_id not in user_group_ids(session, tenant_id=tenant_id, user_id=user_id):
|
||||||
session.query(UserGroupMembership)
|
|
||||||
.filter(UserGroupMembership.tenant_id == tenant_id, UserGroupMembership.user_id == user_id, UserGroupMembership.group_id == group_id)
|
|
||||||
.one_or_none()
|
|
||||||
)
|
|
||||||
if membership is None:
|
|
||||||
raise FileStorageError("No access to this group file space")
|
raise FileStorageError("No access to this group file space")
|
||||||
|
|
||||||
|
|
||||||
@@ -42,3 +59,21 @@ def ensure_owner_access(session: Session, *, tenant_id: str, owner_type: str, ow
|
|||||||
ensure_group_access(session, tenant_id=tenant_id, group_id=owner_id, user_id=user_id, is_admin=is_admin)
|
ensure_group_access(session, tenant_id=tenant_id, group_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||||
return
|
return
|
||||||
raise FileStorageError("Files must be owned by a user or group")
|
raise FileStorageError("Files must be owned by a user or group")
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_share_target_exists(*, tenant_id: str, target_type: str, target_id: str) -> None:
|
||||||
|
target_type = target_type.lower().strip()
|
||||||
|
if target_type == "user":
|
||||||
|
user = _access_directory().get_user(target_id)
|
||||||
|
if not user or user.tenant_id != tenant_id or user.status != "active":
|
||||||
|
raise FileStorageError("User not found")
|
||||||
|
return
|
||||||
|
if target_type == "group":
|
||||||
|
group = _access_directory().get_group(target_id)
|
||||||
|
if not group or group.tenant_id != tenant_id or group.status != "active":
|
||||||
|
raise FileStorageError("Group not found")
|
||||||
|
return
|
||||||
|
if target_type == "tenant":
|
||||||
|
if target_id != tenant_id:
|
||||||
|
raise FileStorageError("Tenant not found")
|
||||||
|
return
|
||||||
|
|||||||
@@ -1,53 +1,93 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import mimetypes
|
import mimetypes
|
||||||
|
import re
|
||||||
|
import stat
|
||||||
|
import tarfile
|
||||||
import zipfile
|
import zipfile
|
||||||
|
from dataclasses import dataclass
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from pathlib import Path
|
from os import PathLike
|
||||||
from typing import Iterable
|
from pathlib import Path, PurePosixPath
|
||||||
|
from typing import Any, BinaryIO, Iterable, Literal
|
||||||
|
|
||||||
|
import pyzipper
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_files.backend.db.models import FileAsset
|
from govoplan_files.backend.db.models import FileAsset
|
||||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile
|
from govoplan_files.backend.storage.backends import (
|
||||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
StorageBackendError,
|
||||||
from govoplan_files.backend.storage.files import create_file_asset, current_version_and_blob
|
get_storage_backend,
|
||||||
from govoplan_files.backend.storage.paths import filename_from_path, normalize_folder, normalize_logical_path
|
)
|
||||||
|
from govoplan_files.backend.storage.common import (
|
||||||
|
FileConflictResolution,
|
||||||
|
FileStorageError,
|
||||||
|
UploadedStoredFile,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.files import (
|
||||||
|
create_file_asset,
|
||||||
|
current_versions_and_blobs,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.paths import (
|
||||||
|
filename_from_path,
|
||||||
|
normalize_folder,
|
||||||
|
normalize_logical_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
_ZIP_READ_CHUNK_SIZE = 1024 * 1024
|
_ARCHIVE_READ_CHUNK_SIZE = 1024 * 1024
|
||||||
|
_WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:")
|
||||||
|
ARCHIVE_UPLOAD_MAX_ENTRIES = 10_000
|
||||||
|
# Kept for callers and documentation using the former ZIP-specific name.
|
||||||
|
ZIP_UPLOAD_MAX_FILES = ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||||
|
SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||||
|
".tar.bz2",
|
||||||
|
".tar.gz",
|
||||||
|
".tar.xz",
|
||||||
|
".tbz2",
|
||||||
|
".tgz",
|
||||||
|
".txz",
|
||||||
|
".tar",
|
||||||
|
".zip",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _read_zip_member(
|
class ArchivePasswordError(FileStorageError):
|
||||||
archive: zipfile.ZipFile,
|
pass
|
||||||
info: zipfile.ZipInfo,
|
|
||||||
*,
|
|
||||||
max_file_bytes: int,
|
|
||||||
max_total_bytes: int,
|
|
||||||
current_total: int,
|
|
||||||
) -> tuple[bytes, int]:
|
|
||||||
parts: list[bytes] = []
|
|
||||||
actual_size = 0
|
|
||||||
with archive.open(info) as source:
|
|
||||||
while True:
|
|
||||||
read_size = min(_ZIP_READ_CHUNK_SIZE, max_file_bytes + 1 - actual_size)
|
|
||||||
chunk = source.read(read_size)
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
actual_size += len(chunk)
|
|
||||||
if actual_size > max_file_bytes:
|
|
||||||
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
|
|
||||||
if current_total + actual_size > max_total_bytes:
|
|
||||||
raise FileStorageError("ZIP is too large after extraction")
|
|
||||||
parts.append(chunk)
|
|
||||||
return b"".join(parts), current_total + actual_size
|
|
||||||
|
|
||||||
|
|
||||||
def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path: str | Path) -> None:
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ArchiveEntry:
|
||||||
|
path: str
|
||||||
|
kind: Literal["file", "directory"]
|
||||||
|
size_bytes: int
|
||||||
|
compressed_size_bytes: int | None = None
|
||||||
|
encrypted: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ArchiveInspection:
|
||||||
|
archive_format: str
|
||||||
|
entries: tuple[ArchiveEntry, ...]
|
||||||
|
file_count: int
|
||||||
|
directory_count: int
|
||||||
|
expanded_size_bytes: int
|
||||||
|
compressed_size_bytes: int
|
||||||
|
requires_password: bool
|
||||||
|
password_verified: bool
|
||||||
|
|
||||||
|
|
||||||
|
def create_zip_file(
|
||||||
|
session: Session, assets: Iterable[FileAsset], output_path: str | Path
|
||||||
|
) -> None:
|
||||||
backend = get_storage_backend()
|
backend = get_storage_backend()
|
||||||
with zipfile.ZipFile(output_path, mode="w", compression=zipfile.ZIP_DEFLATED) as archive:
|
asset_list = list(assets)
|
||||||
for asset in assets:
|
version_blobs = current_versions_and_blobs(session, asset_list)
|
||||||
_version, blob = current_version_and_blob(session, asset)
|
with zipfile.ZipFile(
|
||||||
|
output_path, mode="w", compression=zipfile.ZIP_DEFLATED
|
||||||
|
) as archive:
|
||||||
|
for asset in asset_list:
|
||||||
|
_version, blob = version_blobs[asset.id]
|
||||||
info = zipfile.ZipInfo(asset.display_path)
|
info = zipfile.ZipInfo(asset.display_path)
|
||||||
info.compress_type = zipfile.ZIP_DEFLATED
|
info.compress_type = zipfile.ZIP_DEFLATED
|
||||||
with archive.open(info, "w") as member:
|
with archive.open(info, "w") as member:
|
||||||
@@ -59,6 +99,147 @@ def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path:
|
|||||||
raise FileStorageError(str(exc)) from exc
|
raise FileStorageError(str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def archive_format_for_filename(filename: str) -> str:
|
||||||
|
lowered = filename.strip().casefold()
|
||||||
|
if lowered.endswith(".zip"):
|
||||||
|
return "zip"
|
||||||
|
if lowered.endswith((".tar.gz", ".tgz")):
|
||||||
|
return "tar.gz"
|
||||||
|
if lowered.endswith((".tar.bz2", ".tbz2")):
|
||||||
|
return "tar.bz2"
|
||||||
|
if lowered.endswith((".tar.xz", ".txz")):
|
||||||
|
return "tar.xz"
|
||||||
|
if lowered.endswith(".tar"):
|
||||||
|
return "tar"
|
||||||
|
raise FileStorageError(
|
||||||
|
"Unsupported archive format. Use ZIP, TAR, TAR.GZ, TAR.BZ2, or TAR.XZ."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def is_supported_archive_filename(filename: str) -> bool:
|
||||||
|
try:
|
||||||
|
archive_format_for_filename(filename)
|
||||||
|
except FileStorageError:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def inspect_archive(
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
*,
|
||||||
|
filename: str,
|
||||||
|
password: str | None = None,
|
||||||
|
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||||
|
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||||
|
max_expansion_ratio: int = 100,
|
||||||
|
) -> ArchiveInspection:
|
||||||
|
archive_format = archive_format_for_filename(filename)
|
||||||
|
compressed_size = _archive_size(archive_data)
|
||||||
|
if archive_format == "zip":
|
||||||
|
entries, requires_password, password_verified = _inspect_zip(
|
||||||
|
archive_data,
|
||||||
|
password=password,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
entries = _inspect_tar(archive_data)
|
||||||
|
requires_password = False
|
||||||
|
password_verified = True
|
||||||
|
_validate_archive_limits(
|
||||||
|
entries,
|
||||||
|
compressed_size=compressed_size,
|
||||||
|
max_entries=max_entries,
|
||||||
|
max_expanded_bytes=max_expanded_bytes,
|
||||||
|
max_expansion_ratio=max_expansion_ratio,
|
||||||
|
)
|
||||||
|
complete_entries = _with_derived_directories(entries)
|
||||||
|
return ArchiveInspection(
|
||||||
|
archive_format=archive_format,
|
||||||
|
entries=tuple(complete_entries),
|
||||||
|
file_count=sum(entry.kind == "file" for entry in complete_entries),
|
||||||
|
directory_count=sum(
|
||||||
|
entry.kind == "directory" for entry in complete_entries
|
||||||
|
),
|
||||||
|
expanded_size_bytes=sum(
|
||||||
|
entry.size_bytes for entry in entries if entry.kind == "file"
|
||||||
|
),
|
||||||
|
compressed_size_bytes=compressed_size,
|
||||||
|
requires_password=requires_password,
|
||||||
|
password_verified=password_verified,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_archive_upload(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
user_id: str,
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
filename: str,
|
||||||
|
folder: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
selected_paths: Iterable[str] | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
conflict_strategy: str = "reject",
|
||||||
|
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||||
|
metadata: dict[str, Any] | None = None,
|
||||||
|
is_admin: bool = False,
|
||||||
|
encryption_vault_id: str | None = None,
|
||||||
|
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||||
|
max_file_bytes: int = 50 * 1024 * 1024,
|
||||||
|
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||||
|
max_expansion_ratio: int = 100,
|
||||||
|
) -> list[UploadedStoredFile]:
|
||||||
|
inspection = inspect_archive(
|
||||||
|
archive_data,
|
||||||
|
filename=filename,
|
||||||
|
password=password,
|
||||||
|
max_entries=max_entries,
|
||||||
|
max_expanded_bytes=max_expanded_bytes,
|
||||||
|
max_expansion_ratio=max_expansion_ratio,
|
||||||
|
)
|
||||||
|
if inspection.requires_password and not inspection.password_verified:
|
||||||
|
raise ArchivePasswordError("Archive password is required")
|
||||||
|
selected_files = _selected_file_paths(inspection.entries, selected_paths)
|
||||||
|
if not selected_files:
|
||||||
|
raise FileStorageError("Select at least one archive file to import")
|
||||||
|
actual_total_limit = min(
|
||||||
|
max_expanded_bytes,
|
||||||
|
inspection.compressed_size_bytes * max_expansion_ratio,
|
||||||
|
)
|
||||||
|
if inspection.archive_format == "zip":
|
||||||
|
members = _read_selected_zip_members(
|
||||||
|
archive_data,
|
||||||
|
selected_files=selected_files,
|
||||||
|
password=password,
|
||||||
|
max_file_bytes=max_file_bytes,
|
||||||
|
max_total_bytes=actual_total_limit,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
members = _read_selected_tar_members(
|
||||||
|
archive_data,
|
||||||
|
selected_files=selected_files,
|
||||||
|
max_file_bytes=max_file_bytes,
|
||||||
|
max_total_bytes=actual_total_limit,
|
||||||
|
)
|
||||||
|
return _store_archive_members(
|
||||||
|
session,
|
||||||
|
members=members,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
user_id=user_id,
|
||||||
|
folder=folder,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=conflict_resolutions,
|
||||||
|
metadata=metadata,
|
||||||
|
is_admin=is_admin,
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def extract_zip_upload(
|
def extract_zip_upload(
|
||||||
session: Session,
|
session: Session,
|
||||||
*,
|
*,
|
||||||
@@ -66,59 +247,416 @@ def extract_zip_upload(
|
|||||||
owner_type: str,
|
owner_type: str,
|
||||||
owner_id: str,
|
owner_id: str,
|
||||||
user_id: str,
|
user_id: str,
|
||||||
zip_data: bytes,
|
zip_data: bytes | str | PathLike[str],
|
||||||
folder: str | None,
|
folder: str | None,
|
||||||
campaign_id: str | None,
|
campaign_id: str | None,
|
||||||
conflict_strategy: str = "reject",
|
conflict_strategy: str = "reject",
|
||||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||||
|
metadata: dict[str, Any] | None = None,
|
||||||
is_admin: bool = False,
|
is_admin: bool = False,
|
||||||
max_files: int = 1000,
|
encryption_vault_id: str | None = None,
|
||||||
|
max_files: int = ZIP_UPLOAD_MAX_FILES,
|
||||||
max_file_bytes: int = 50 * 1024 * 1024,
|
max_file_bytes: int = 50 * 1024 * 1024,
|
||||||
max_total_bytes: int = 250 * 1024 * 1024,
|
max_total_bytes: int = 250 * 1024 * 1024,
|
||||||
) -> list[UploadedStoredFile]:
|
) -> list[UploadedStoredFile]:
|
||||||
uploaded: list[UploadedStoredFile] = []
|
"""Backward-compatible wrapper for the original immediate ZIP endpoint."""
|
||||||
total = 0
|
|
||||||
base_folder = normalize_folder(folder)
|
return extract_archive_upload(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
user_id=user_id,
|
||||||
|
archive_data=zip_data,
|
||||||
|
filename="archive.zip",
|
||||||
|
folder=folder,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=conflict_resolutions,
|
||||||
|
metadata=metadata,
|
||||||
|
is_admin=is_admin,
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
max_entries=max_files,
|
||||||
|
max_file_bytes=max_file_bytes,
|
||||||
|
max_expanded_bytes=max_total_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _inspect_zip(
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
*,
|
||||||
|
password: str | None,
|
||||||
|
) -> tuple[list[ArchiveEntry], bool, bool]:
|
||||||
try:
|
try:
|
||||||
with zipfile.ZipFile(BytesIO(zip_data)) as archive:
|
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||||
infos = [info for info in archive.infolist() if not info.is_dir()]
|
infos = archive.infolist()
|
||||||
if len(infos) > max_files:
|
entries = [_zip_entry(info) for info in infos]
|
||||||
raise FileStorageError(f"ZIP contains too many files (limit {max_files})")
|
encrypted_info = next(
|
||||||
for info in infos:
|
(
|
||||||
if info.flag_bits & 0x1:
|
info
|
||||||
raise FileStorageError("Encrypted ZIP uploads are not supported")
|
for info in infos
|
||||||
if info.file_size < 0:
|
if not info.is_dir() and bool(info.flag_bits & 0x1)
|
||||||
raise FileStorageError("Invalid ZIP member")
|
),
|
||||||
if info.file_size > max_file_bytes:
|
None,
|
||||||
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
|
)
|
||||||
if total + info.file_size > max_total_bytes:
|
if encrypted_info is None:
|
||||||
raise FileStorageError("ZIP is too large after extraction")
|
return entries, False, True
|
||||||
inner_path = normalize_logical_path(info.filename)
|
if not password:
|
||||||
target_path = f"{base_folder}/{inner_path}" if base_folder else inner_path
|
return entries, True, False
|
||||||
data, total = _read_zip_member(
|
try:
|
||||||
archive,
|
with archive.open(
|
||||||
info,
|
encrypted_info, pwd=password.encode("utf-8")
|
||||||
max_file_bytes=max_file_bytes,
|
) as member:
|
||||||
max_total_bytes=max_total_bytes,
|
member.read(1)
|
||||||
current_total=total,
|
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||||
)
|
raise ArchivePasswordError("Archive password is incorrect") from exc
|
||||||
uploaded.append(
|
return entries, True, True
|
||||||
create_file_asset(
|
except ArchivePasswordError:
|
||||||
session,
|
raise
|
||||||
tenant_id=tenant_id,
|
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||||
owner_type=owner_type,
|
raise FileStorageError("Invalid ZIP upload") from exc
|
||||||
owner_id=owner_id,
|
|
||||||
user_id=user_id,
|
|
||||||
filename=filename_from_path(inner_path),
|
def _inspect_tar(
|
||||||
data=data,
|
archive_data: bytes | str | PathLike[str],
|
||||||
display_path=target_path,
|
) -> list[ArchiveEntry]:
|
||||||
content_type=mimetypes.guess_type(inner_path)[0] or "application/octet-stream",
|
try:
|
||||||
campaign_id=campaign_id,
|
with _open_tar(archive_data) as archive:
|
||||||
conflict_strategy=conflict_strategy,
|
entries: list[ArchiveEntry] = []
|
||||||
conflict_resolutions=conflict_resolutions,
|
for member in archive.getmembers():
|
||||||
is_admin=is_admin,
|
path = _safe_member_path(member.name)
|
||||||
|
if member.isdir():
|
||||||
|
entries.append(
|
||||||
|
ArchiveEntry(path=path, kind="directory", size_bytes=0)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if not member.isfile():
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive member {member.name!r} is not a regular file or directory"
|
||||||
|
)
|
||||||
|
entries.append(
|
||||||
|
ArchiveEntry(
|
||||||
|
path=path,
|
||||||
|
kind="file",
|
||||||
|
size_bytes=max(0, int(member.size)),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
except zipfile.BadZipFile as exc:
|
return entries
|
||||||
raise FileStorageError("Invalid ZIP upload") from exc
|
except FileStorageError:
|
||||||
|
raise
|
||||||
|
except (OSError, tarfile.TarError) as exc:
|
||||||
|
raise FileStorageError("Invalid TAR upload") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _zip_entry(info: zipfile.ZipInfo) -> ArchiveEntry:
|
||||||
|
path = _safe_member_path(info.filename)
|
||||||
|
unix_mode = (info.external_attr >> 16) & 0xFFFF
|
||||||
|
file_type = stat.S_IFMT(unix_mode)
|
||||||
|
if file_type and not (
|
||||||
|
stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)
|
||||||
|
):
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive member {info.filename!r} is not a regular file or directory"
|
||||||
|
)
|
||||||
|
if info.file_size < 0 or info.compress_size < 0:
|
||||||
|
raise FileStorageError(f"Archive member {info.filename!r} has invalid size metadata")
|
||||||
|
return ArchiveEntry(
|
||||||
|
path=path,
|
||||||
|
kind="directory" if info.is_dir() else "file",
|
||||||
|
size_bytes=0 if info.is_dir() else int(info.file_size),
|
||||||
|
compressed_size_bytes=(
|
||||||
|
None if info.is_dir() else int(info.compress_size)
|
||||||
|
),
|
||||||
|
encrypted=bool(info.flag_bits & 0x1),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_archive_limits(
|
||||||
|
entries: list[ArchiveEntry],
|
||||||
|
*,
|
||||||
|
compressed_size: int,
|
||||||
|
max_entries: int,
|
||||||
|
max_expanded_bytes: int,
|
||||||
|
max_expansion_ratio: int,
|
||||||
|
) -> None:
|
||||||
|
if len(entries) > max_entries:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive contains too many entries (limit {max_entries})"
|
||||||
|
)
|
||||||
|
seen: dict[str, str] = {}
|
||||||
|
expanded_size = 0
|
||||||
|
for entry in entries:
|
||||||
|
previous_kind = seen.get(entry.path)
|
||||||
|
if previous_kind is not None:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive contains duplicate path {entry.path!r}"
|
||||||
|
)
|
||||||
|
seen[entry.path] = entry.kind
|
||||||
|
if entry.kind == "file":
|
||||||
|
expanded_size += entry.size_bytes
|
||||||
|
if expanded_size > max_expanded_bytes:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Archive is too large after extraction "
|
||||||
|
f"(limit {max_expanded_bytes} bytes)"
|
||||||
|
)
|
||||||
|
if expanded_size and (
|
||||||
|
compressed_size <= 0
|
||||||
|
or expanded_size > compressed_size * max_expansion_ratio
|
||||||
|
):
|
||||||
|
raise FileStorageError(
|
||||||
|
"Archive expansion ratio exceeds "
|
||||||
|
f"{max_expansion_ratio}:1"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _with_derived_directories(
|
||||||
|
entries: list[ArchiveEntry],
|
||||||
|
) -> list[ArchiveEntry]:
|
||||||
|
by_path = {entry.path: entry for entry in entries}
|
||||||
|
for entry in entries:
|
||||||
|
parent = PurePosixPath(entry.path).parent
|
||||||
|
while str(parent) not in {"", "."}:
|
||||||
|
path = str(parent)
|
||||||
|
existing = by_path.get(path)
|
||||||
|
if existing and existing.kind != "directory":
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive path {path!r} is both a file and a directory"
|
||||||
|
)
|
||||||
|
by_path.setdefault(
|
||||||
|
path,
|
||||||
|
ArchiveEntry(path=path, kind="directory", size_bytes=0),
|
||||||
|
)
|
||||||
|
parent = parent.parent
|
||||||
|
return sorted(
|
||||||
|
by_path.values(),
|
||||||
|
key=lambda entry: (
|
||||||
|
tuple(entry.path.casefold().split("/")),
|
||||||
|
entry.kind != "directory",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _selected_file_paths(
|
||||||
|
entries: tuple[ArchiveEntry, ...],
|
||||||
|
selected_paths: Iterable[str] | None,
|
||||||
|
) -> set[str]:
|
||||||
|
file_paths = {entry.path for entry in entries if entry.kind == "file"}
|
||||||
|
if selected_paths is None:
|
||||||
|
return file_paths
|
||||||
|
known = {entry.path: entry for entry in entries}
|
||||||
|
normalized = {_safe_member_path(path) for path in selected_paths}
|
||||||
|
unknown = normalized - known.keys()
|
||||||
|
if unknown:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive selection contains unknown path {sorted(unknown)[0]!r}"
|
||||||
|
)
|
||||||
|
selected_files: set[str] = set()
|
||||||
|
for path in normalized:
|
||||||
|
entry = known[path]
|
||||||
|
if entry.kind == "file":
|
||||||
|
selected_files.add(path)
|
||||||
|
continue
|
||||||
|
prefix = f"{path}/"
|
||||||
|
selected_files.update(
|
||||||
|
file_path
|
||||||
|
for file_path in file_paths
|
||||||
|
if file_path.startswith(prefix)
|
||||||
|
)
|
||||||
|
return selected_files
|
||||||
|
|
||||||
|
|
||||||
|
def _read_selected_zip_members(
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
*,
|
||||||
|
selected_files: set[str],
|
||||||
|
password: str | None,
|
||||||
|
max_file_bytes: int,
|
||||||
|
max_total_bytes: int,
|
||||||
|
) -> list[tuple[str, bytes]]:
|
||||||
|
result: list[tuple[str, bytes]] = []
|
||||||
|
total = 0
|
||||||
|
try:
|
||||||
|
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||||
|
for info in archive.infolist():
|
||||||
|
if info.is_dir():
|
||||||
|
continue
|
||||||
|
path = _safe_member_path(info.filename)
|
||||||
|
if path not in selected_files:
|
||||||
|
continue
|
||||||
|
pwd = password.encode("utf-8") if password else None
|
||||||
|
try:
|
||||||
|
with archive.open(info, pwd=pwd) as source:
|
||||||
|
data, total = _read_member(
|
||||||
|
source,
|
||||||
|
path=path,
|
||||||
|
max_file_bytes=max_file_bytes,
|
||||||
|
max_total_bytes=max_total_bytes,
|
||||||
|
current_total=total,
|
||||||
|
)
|
||||||
|
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||||
|
if info.flag_bits & 0x1:
|
||||||
|
raise ArchivePasswordError(
|
||||||
|
"Archive password is incorrect"
|
||||||
|
) from exc
|
||||||
|
raise
|
||||||
|
result.append((path, data))
|
||||||
|
except (FileStorageError, ArchivePasswordError):
|
||||||
|
raise
|
||||||
|
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||||
|
raise FileStorageError("ZIP extraction failed") from exc
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _read_selected_tar_members(
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
*,
|
||||||
|
selected_files: set[str],
|
||||||
|
max_file_bytes: int,
|
||||||
|
max_total_bytes: int,
|
||||||
|
) -> list[tuple[str, bytes]]:
|
||||||
|
result: list[tuple[str, bytes]] = []
|
||||||
|
total = 0
|
||||||
|
try:
|
||||||
|
with _open_tar(archive_data) as archive:
|
||||||
|
for member in archive.getmembers():
|
||||||
|
if not member.isfile():
|
||||||
|
continue
|
||||||
|
path = _safe_member_path(member.name)
|
||||||
|
if path not in selected_files:
|
||||||
|
continue
|
||||||
|
source = archive.extractfile(member)
|
||||||
|
if source is None:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive member {path!r} could not be read"
|
||||||
|
)
|
||||||
|
with source:
|
||||||
|
data, total = _read_member(
|
||||||
|
source,
|
||||||
|
path=path,
|
||||||
|
max_file_bytes=max_file_bytes,
|
||||||
|
max_total_bytes=max_total_bytes,
|
||||||
|
current_total=total,
|
||||||
|
)
|
||||||
|
result.append((path, data))
|
||||||
|
except FileStorageError:
|
||||||
|
raise
|
||||||
|
except (OSError, tarfile.TarError) as exc:
|
||||||
|
raise FileStorageError("TAR extraction failed") from exc
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _read_member(
|
||||||
|
source: BinaryIO,
|
||||||
|
*,
|
||||||
|
path: str,
|
||||||
|
max_file_bytes: int,
|
||||||
|
max_total_bytes: int,
|
||||||
|
current_total: int,
|
||||||
|
) -> tuple[bytes, int]:
|
||||||
|
parts: list[bytes] = []
|
||||||
|
actual_size = 0
|
||||||
|
while True:
|
||||||
|
read_size = min(
|
||||||
|
_ARCHIVE_READ_CHUNK_SIZE,
|
||||||
|
max_file_bytes + 1 - actual_size,
|
||||||
|
)
|
||||||
|
chunk = source.read(read_size)
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
actual_size += len(chunk)
|
||||||
|
if actual_size > max_file_bytes:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"Archive member {path!r} exceeds per-file limit"
|
||||||
|
)
|
||||||
|
if current_total + actual_size > max_total_bytes:
|
||||||
|
raise FileStorageError("Archive is too large after extraction")
|
||||||
|
parts.append(chunk)
|
||||||
|
return b"".join(parts), current_total + actual_size
|
||||||
|
|
||||||
|
|
||||||
|
def _store_archive_members(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
members: Iterable[tuple[str, bytes]],
|
||||||
|
tenant_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
user_id: str,
|
||||||
|
folder: str | None,
|
||||||
|
campaign_id: str | None,
|
||||||
|
conflict_strategy: str,
|
||||||
|
conflict_resolutions: Iterable[FileConflictResolution] | None,
|
||||||
|
metadata: dict[str, Any] | None,
|
||||||
|
is_admin: bool,
|
||||||
|
encryption_vault_id: str | None,
|
||||||
|
) -> list[UploadedStoredFile]:
|
||||||
|
uploaded: list[UploadedStoredFile] = []
|
||||||
|
base_folder = normalize_folder(folder)
|
||||||
|
for inner_path, data in members:
|
||||||
|
target_path = (
|
||||||
|
f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||||
|
)
|
||||||
|
uploaded.append(
|
||||||
|
create_file_asset(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
user_id=user_id,
|
||||||
|
filename=filename_from_path(inner_path),
|
||||||
|
data=data,
|
||||||
|
display_path=target_path,
|
||||||
|
content_type=mimetypes.guess_type(inner_path)[0]
|
||||||
|
or "application/octet-stream",
|
||||||
|
metadata=metadata,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
conflict_resolutions=conflict_resolutions,
|
||||||
|
is_admin=is_admin,
|
||||||
|
encryption_vault_id=encryption_vault_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
return uploaded
|
return uploaded
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_member_path(value: str) -> str:
|
||||||
|
raw = str(value or "").replace("\\", "/").strip()
|
||||||
|
if (
|
||||||
|
not raw
|
||||||
|
or "\x00" in raw
|
||||||
|
or raw.startswith("/")
|
||||||
|
or _WINDOWS_DRIVE_RE.match(raw)
|
||||||
|
):
|
||||||
|
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||||
|
if any(part == ".." for part in raw.split("/")):
|
||||||
|
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||||
|
try:
|
||||||
|
return normalize_logical_path(raw.rstrip("/"))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise FileStorageError(f"Unsafe archive member path {value!r}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_size(archive_data: bytes | str | PathLike[str]) -> int:
|
||||||
|
if isinstance(archive_data, bytes):
|
||||||
|
return len(archive_data)
|
||||||
|
try:
|
||||||
|
return Path(archive_data).stat().st_size
|
||||||
|
except OSError as exc:
|
||||||
|
raise FileStorageError("Archive upload could not be read") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_source(
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
) -> BytesIO | str | PathLike[str]:
|
||||||
|
return BytesIO(archive_data) if isinstance(archive_data, bytes) else archive_data
|
||||||
|
|
||||||
|
|
||||||
|
def _open_tar(
|
||||||
|
archive_data: bytes | str | PathLike[str],
|
||||||
|
) -> tarfile.TarFile:
|
||||||
|
if isinstance(archive_data, bytes):
|
||||||
|
return tarfile.open(fileobj=BytesIO(archive_data), mode="r:*")
|
||||||
|
try:
|
||||||
|
return tarfile.open(name=archive_data, mode="r:*")
|
||||||
|
except OSError as exc:
|
||||||
|
raise FileStorageError("Archive upload could not be read") from exc
|
||||||
|
|||||||
@@ -1,160 +1,32 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
from govoplan_core.core.object_storage import (
|
||||||
from pathlib import Path
|
LocalFilesystemStorageBackend,
|
||||||
from typing import Iterable, Protocol
|
S3StorageBackend,
|
||||||
|
StorageBackend,
|
||||||
|
StorageBackendError,
|
||||||
|
StorageObjectInfo,
|
||||||
|
StorageObjectMissing,
|
||||||
|
StorageObjectPage,
|
||||||
|
configured_storage_backend,
|
||||||
|
)
|
||||||
|
|
||||||
from govoplan_files.backend.runtime import settings
|
from govoplan_files.backend.runtime import settings
|
||||||
|
|
||||||
|
|
||||||
class StorageBackendError(RuntimeError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
class StorageBackend(Protocol):
|
|
||||||
name: str
|
|
||||||
|
|
||||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None: ...
|
|
||||||
def get_bytes(self, key: str) -> bytes: ...
|
|
||||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]: ...
|
|
||||||
def delete(self, key: str) -> None: ...
|
|
||||||
def exists(self, key: str) -> bool: ...
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
|
||||||
class LocalFilesystemStorageBackend:
|
|
||||||
root: Path
|
|
||||||
fallback_roots: tuple[Path, ...] = field(default_factory=tuple)
|
|
||||||
name: str = "local"
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
self.root = self.root.expanduser().resolve()
|
|
||||||
self.fallback_roots = tuple(root.expanduser().resolve() for root in self.fallback_roots if root)
|
|
||||||
self.root.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
def _path_for_root(self, root: Path, key: str) -> Path:
|
|
||||||
path = (root / key).resolve()
|
|
||||||
if not path.is_relative_to(root):
|
|
||||||
raise StorageBackendError("Storage key escapes local storage root")
|
|
||||||
return path
|
|
||||||
|
|
||||||
def _path(self, key: str) -> Path:
|
|
||||||
return self._path_for_root(self.root, key)
|
|
||||||
|
|
||||||
def _readable_path(self, key: str) -> Path:
|
|
||||||
primary = self._path(key)
|
|
||||||
if primary.exists() and primary.is_file():
|
|
||||||
return primary
|
|
||||||
for root in self.fallback_roots:
|
|
||||||
candidate = self._path_for_root(root, key)
|
|
||||||
if candidate.exists() and candidate.is_file():
|
|
||||||
return candidate
|
|
||||||
raise StorageBackendError("Stored object does not exist")
|
|
||||||
|
|
||||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
|
||||||
path = self._path(key)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
path.write_bytes(data)
|
|
||||||
|
|
||||||
def get_bytes(self, key: str) -> bytes:
|
|
||||||
return self._readable_path(key).read_bytes()
|
|
||||||
|
|
||||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
|
||||||
path = self._readable_path(key)
|
|
||||||
with path.open("rb") as handle:
|
|
||||||
while True:
|
|
||||||
chunk = handle.read(chunk_size)
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
yield chunk
|
|
||||||
|
|
||||||
def delete(self, key: str) -> None:
|
|
||||||
path = self._path(key)
|
|
||||||
if path.exists() and path.is_file():
|
|
||||||
path.unlink()
|
|
||||||
|
|
||||||
def exists(self, key: str) -> bool:
|
|
||||||
try:
|
|
||||||
self._readable_path(key)
|
|
||||||
except StorageBackendError:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
|
||||||
class S3StorageBackend:
|
|
||||||
bucket: str
|
|
||||||
endpoint_url: str
|
|
||||||
region_name: str
|
|
||||||
access_key_id: str
|
|
||||||
secret_access_key: str
|
|
||||||
name: str = "s3"
|
|
||||||
|
|
||||||
@property
|
|
||||||
def client(self):
|
|
||||||
try:
|
|
||||||
import boto3
|
|
||||||
except ModuleNotFoundError as exc:
|
|
||||||
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
|
|
||||||
return boto3.client(
|
|
||||||
"s3",
|
|
||||||
endpoint_url=self.endpoint_url,
|
|
||||||
region_name=self.region_name,
|
|
||||||
aws_access_key_id=self.access_key_id,
|
|
||||||
aws_secret_access_key=self.secret_access_key,
|
|
||||||
)
|
|
||||||
|
|
||||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
|
||||||
kwargs = {"Bucket": self.bucket, "Key": key, "Body": data}
|
|
||||||
if content_type:
|
|
||||||
kwargs["ContentType"] = content_type
|
|
||||||
self.client.put_object(**kwargs)
|
|
||||||
|
|
||||||
def get_bytes(self, key: str) -> bytes:
|
|
||||||
try:
|
|
||||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
|
||||||
return obj["Body"].read()
|
|
||||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
|
||||||
raise StorageBackendError(str(exc)) from exc
|
|
||||||
|
|
||||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
|
||||||
try:
|
|
||||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
|
||||||
body = obj["Body"]
|
|
||||||
while True:
|
|
||||||
chunk = body.read(chunk_size)
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
yield chunk
|
|
||||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
|
||||||
raise StorageBackendError(str(exc)) from exc
|
|
||||||
|
|
||||||
def delete(self, key: str) -> None:
|
|
||||||
self.client.delete_object(Bucket=self.bucket, Key=key)
|
|
||||||
|
|
||||||
def exists(self, key: str) -> bool:
|
|
||||||
try:
|
|
||||||
self.client.head_object(Bucket=self.bucket, Key=key)
|
|
||||||
return True
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _fallback_roots() -> tuple[Path, ...]:
|
|
||||||
raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
|
|
||||||
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())
|
|
||||||
|
|
||||||
|
|
||||||
def get_storage_backend() -> StorageBackend:
|
def get_storage_backend() -> StorageBackend:
|
||||||
backend = settings.file_storage_backend.lower().strip()
|
"""Return the deployment-owned backend for Files-managed objects."""
|
||||||
if backend in {"local", "filesystem", "fs"}:
|
|
||||||
return LocalFilesystemStorageBackend(Path(settings.file_storage_local_root), fallback_roots=_fallback_roots())
|
return configured_storage_backend(settings)
|
||||||
if backend in {"s3", "garage"}:
|
|
||||||
return S3StorageBackend(
|
|
||||||
bucket=settings.file_storage_s3_bucket or settings.s3_bucket,
|
__all__ = [
|
||||||
endpoint_url=settings.file_storage_s3_endpoint_url or settings.s3_endpoint_url,
|
"LocalFilesystemStorageBackend",
|
||||||
region_name=settings.file_storage_s3_region or settings.s3_region,
|
"S3StorageBackend",
|
||||||
access_key_id=settings.file_storage_s3_access_key_id or settings.s3_access_key_id,
|
"StorageBackend",
|
||||||
secret_access_key=settings.file_storage_s3_secret_access_key or settings.s3_secret_access_key,
|
"StorageBackendError",
|
||||||
)
|
"StorageObjectInfo",
|
||||||
raise StorageBackendError(f"Unsupported file storage backend: {settings.file_storage_backend}")
|
"StorageObjectMissing",
|
||||||
|
"StorageObjectPage",
|
||||||
|
"get_storage_backend",
|
||||||
|
]
|
||||||
|
|||||||
@@ -12,9 +12,17 @@ from typing import Any, Iterator
|
|||||||
|
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_files.backend.db.models import FileAsset
|
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
|
||||||
from govoplan_files.backend.storage.files import current_version_and_blob, list_assets_for_user, read_asset_bytes
|
from govoplan_files.backend.storage.backends import StorageBackend, get_storage_backend
|
||||||
|
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
|
||||||
|
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||||
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
|
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
|
||||||
|
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
|
||||||
|
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||||
|
from govoplan_files.backend.storage.integrity import (
|
||||||
|
ensure_blob_is_readable,
|
||||||
|
read_verified_blob_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
MANAGED_SOURCE_PREFIX = "managed:"
|
MANAGED_SOURCE_PREFIX = "managed:"
|
||||||
@@ -34,10 +42,17 @@ class ManagedAttachmentFile:
|
|||||||
checksum_sha256: str
|
checksum_sha256: str
|
||||||
size_bytes: int
|
size_bytes: int
|
||||||
content_type: str | None
|
content_type: str | None
|
||||||
|
linked_to_campaign: bool = True
|
||||||
|
source_provenance: dict[str, Any] | None = None
|
||||||
|
source_revision: str | None = None
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
def as_dict(self) -> dict[str, Any]:
|
||||||
payload = asdict(self)
|
payload = asdict(self)
|
||||||
payload.pop("local_path", None)
|
payload.pop("local_path", None)
|
||||||
|
if payload.get("source_provenance") is None:
|
||||||
|
payload.pop("source_provenance", None)
|
||||||
|
if payload.get("source_revision") is None:
|
||||||
|
payload.pop("source_revision", None)
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
|
||||||
@@ -47,6 +62,7 @@ class PreparedCampaignSnapshot:
|
|||||||
raw_json: dict[str, Any]
|
raw_json: dict[str, Any]
|
||||||
managed_files_by_local_path: dict[str, ManagedAttachmentFile]
|
managed_files_by_local_path: dict[str, ManagedAttachmentFile]
|
||||||
shared_assets: list[FileAsset]
|
shared_assets: list[FileAsset]
|
||||||
|
candidate_assets: list[FileAsset]
|
||||||
|
|
||||||
|
|
||||||
def parse_managed_source(value: object) -> tuple[str, str] | None:
|
def parse_managed_source(value: object) -> tuple[str, str] | None:
|
||||||
@@ -108,6 +124,99 @@ def _iter_rule_dicts(attachments: dict[str, Any], raw_json: dict[str, Any]):
|
|||||||
yield rule
|
yield rule
|
||||||
|
|
||||||
|
|
||||||
|
def _managed_base_sources(raw_json: dict[str, Any]) -> list[tuple[str, str, str]]:
|
||||||
|
attachments = raw_json.get("attachments")
|
||||||
|
if not isinstance(attachments, dict):
|
||||||
|
return []
|
||||||
|
base_paths = attachments.get("base_paths")
|
||||||
|
if not isinstance(base_paths, list):
|
||||||
|
return []
|
||||||
|
sources: list[tuple[str, str, str]] = []
|
||||||
|
seen: set[tuple[str, str, str]] = set()
|
||||||
|
for item in base_paths:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
parsed_source = parse_managed_source(item.get("source"))
|
||||||
|
if parsed_source is None:
|
||||||
|
continue
|
||||||
|
owner_type, owner_id = parsed_source
|
||||||
|
old_path = str(item.get("path") or ".").strip() or "."
|
||||||
|
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||||
|
key = (owner_type, owner_id, logical_root)
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
sources.append(key)
|
||||||
|
return sources
|
||||||
|
|
||||||
|
|
||||||
|
def _campaign_linked_asset_ids(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
campaign_id: str,
|
||||||
|
asset_ids: list[str],
|
||||||
|
) -> set[str]:
|
||||||
|
if not asset_ids:
|
||||||
|
return set()
|
||||||
|
linked: set[str] = set()
|
||||||
|
for offset in range(0, len(asset_ids), 900):
|
||||||
|
chunk = asset_ids[offset : offset + 900]
|
||||||
|
rows = (
|
||||||
|
session.query(FileShare.file_asset_id)
|
||||||
|
.filter(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id.in_(chunk),
|
||||||
|
FileShare.target_type == "campaign",
|
||||||
|
FileShare.target_id == campaign_id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
)
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
linked.update(row[0] for row in rows)
|
||||||
|
return linked
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate_assets_for_managed_sources(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
campaign_id: str,
|
||||||
|
raw_json: dict[str, Any],
|
||||||
|
user_id: str,
|
||||||
|
is_admin: bool,
|
||||||
|
shared_assets: list[FileAsset],
|
||||||
|
) -> tuple[list[FileAsset], set[str]]:
|
||||||
|
assets_by_id: dict[str, FileAsset] = {asset.id: asset for asset in shared_assets}
|
||||||
|
for owner_type, owner_id, logical_root in _managed_base_sources(raw_json):
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
for asset in list_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
path_prefix=logical_root or None,
|
||||||
|
is_admin=is_admin,
|
||||||
|
):
|
||||||
|
assets_by_id.setdefault(asset.id, asset)
|
||||||
|
candidate_assets = sorted(assets_by_id.values(), key=lambda asset: (asset.display_path, asset.updated_at, asset.id))
|
||||||
|
linked_ids = _campaign_linked_asset_ids(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
asset_ids=[asset.id for asset in candidate_assets],
|
||||||
|
)
|
||||||
|
return candidate_assets, linked_ids
|
||||||
|
|
||||||
|
|
||||||
def _selected_base_path(
|
def _selected_base_path(
|
||||||
rule: dict[str, Any],
|
rule: dict[str, Any],
|
||||||
prepared_by_id: dict[str, tuple[str, str]],
|
prepared_by_id: dict[str, tuple[str, str]],
|
||||||
@@ -127,6 +236,175 @@ def _selected_base_path(
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _prepared_attachment_config(raw_json: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any], list[Any]]:
|
||||||
|
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||||
|
attachments = prepared_json.get("attachments")
|
||||||
|
if not isinstance(attachments, dict):
|
||||||
|
attachments = {}
|
||||||
|
prepared_json["attachments"] = attachments
|
||||||
|
base_paths = attachments.get("base_paths")
|
||||||
|
if not isinstance(base_paths, list):
|
||||||
|
base_paths = []
|
||||||
|
return prepared_json, attachments, base_paths
|
||||||
|
|
||||||
|
|
||||||
|
def _campaign_snapshot_assets(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
campaign_id: str,
|
||||||
|
prepared_json: dict[str, Any],
|
||||||
|
include_unlinked_candidates: bool,
|
||||||
|
user_id: str,
|
||||||
|
is_admin: bool,
|
||||||
|
) -> tuple[list[FileAsset], list[FileAsset], set[str]]:
|
||||||
|
shared_assets = list_assets_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id="",
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
is_admin=True,
|
||||||
|
)
|
||||||
|
if not include_unlinked_candidates:
|
||||||
|
return shared_assets, shared_assets, {asset.id for asset in shared_assets}
|
||||||
|
candidate_assets, linked_asset_ids = _candidate_assets_for_managed_sources(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
raw_json=prepared_json,
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
shared_assets=shared_assets,
|
||||||
|
)
|
||||||
|
return shared_assets, candidate_assets, linked_asset_ids
|
||||||
|
|
||||||
|
|
||||||
|
def _assets_grouped_by_owner(assets: list[FileAsset]) -> dict[tuple[str, str], list[FileAsset]]:
|
||||||
|
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||||
|
for asset in assets:
|
||||||
|
owner_id = _asset_owner_id(asset)
|
||||||
|
if owner_id:
|
||||||
|
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||||
|
return assets_by_owner
|
||||||
|
|
||||||
|
|
||||||
|
def _materialize_managed_asset(
|
||||||
|
asset: FileAsset,
|
||||||
|
*,
|
||||||
|
owner_id: str,
|
||||||
|
logical_root: str,
|
||||||
|
local_root: Path,
|
||||||
|
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||||
|
backend: StorageBackend | None,
|
||||||
|
include_bytes: bool,
|
||||||
|
linked_asset_ids: set[str],
|
||||||
|
) -> tuple[str, ManagedAttachmentFile] | None:
|
||||||
|
relative_path = _relative_asset_path(asset, logical_root)
|
||||||
|
if not relative_path:
|
||||||
|
return None
|
||||||
|
target = _safe_local_target(local_root, relative_path)
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
version, blob = version_blobs[asset.id]
|
||||||
|
ensure_blob_is_readable(blob)
|
||||||
|
if include_bytes:
|
||||||
|
data = read_verified_blob_bytes(blob, backend=backend) if backend else b""
|
||||||
|
target.write_bytes(data)
|
||||||
|
else:
|
||||||
|
target.touch()
|
||||||
|
local_key = str(target.resolve())
|
||||||
|
return local_key, ManagedAttachmentFile(
|
||||||
|
local_path=local_key,
|
||||||
|
asset_id=asset.id,
|
||||||
|
version_id=version.id,
|
||||||
|
blob_id=blob.id,
|
||||||
|
display_path=asset.display_path,
|
||||||
|
relative_path=normalize_logical_path(relative_path),
|
||||||
|
filename=asset.filename,
|
||||||
|
owner_type=asset.owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
checksum_sha256=blob.checksum_sha256,
|
||||||
|
size_bytes=blob.size_bytes,
|
||||||
|
content_type=blob.content_type,
|
||||||
|
linked_to_campaign=asset.id in linked_asset_ids,
|
||||||
|
source_provenance=source_provenance_from_metadata(asset.metadata_),
|
||||||
|
source_revision=source_revision_from_metadata(asset.metadata_),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _prepare_managed_base_paths(
|
||||||
|
base_paths: list[Any],
|
||||||
|
*,
|
||||||
|
materialized_root: Path,
|
||||||
|
assets_by_owner: dict[tuple[str, str], list[FileAsset]],
|
||||||
|
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||||
|
backend: StorageBackend | None,
|
||||||
|
include_bytes: bool,
|
||||||
|
linked_asset_ids: set[str],
|
||||||
|
) -> tuple[
|
||||||
|
dict[str, ManagedAttachmentFile],
|
||||||
|
dict[str, tuple[str, str]],
|
||||||
|
dict[str, list[tuple[str, str]]],
|
||||||
|
tuple[str, str] | None,
|
||||||
|
]:
|
||||||
|
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||||
|
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||||
|
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||||
|
first_prepared: tuple[str, str] | None = None
|
||||||
|
for index, item in enumerate(base_paths):
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
parsed_source = parse_managed_source(item.get("source"))
|
||||||
|
if parsed_source is None:
|
||||||
|
continue
|
||||||
|
owner_type, owner_id = parsed_source
|
||||||
|
old_path = str(item.get("path") or ".").strip() or "."
|
||||||
|
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||||
|
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||||
|
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||||
|
local_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
local_root_string = str(local_root.resolve())
|
||||||
|
prepared = (base_path_id, local_root_string)
|
||||||
|
prepared_by_id[base_path_id] = prepared
|
||||||
|
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||||
|
if first_prepared is None:
|
||||||
|
first_prepared = prepared
|
||||||
|
item["path"] = local_root_string
|
||||||
|
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||||
|
materialized = _materialize_managed_asset(
|
||||||
|
asset,
|
||||||
|
owner_id=owner_id,
|
||||||
|
logical_root=logical_root,
|
||||||
|
local_root=local_root,
|
||||||
|
version_blobs=version_blobs,
|
||||||
|
backend=backend,
|
||||||
|
include_bytes=include_bytes,
|
||||||
|
linked_asset_ids=linked_asset_ids,
|
||||||
|
)
|
||||||
|
if materialized is not None:
|
||||||
|
local_key, managed_file = materialized
|
||||||
|
manifest[local_key] = managed_file
|
||||||
|
return manifest, prepared_by_id, prepared_by_old_path, first_prepared
|
||||||
|
|
||||||
|
|
||||||
|
def _rewrite_managed_attachment_rules(
|
||||||
|
attachments: dict[str, Any],
|
||||||
|
prepared_json: dict[str, Any],
|
||||||
|
*,
|
||||||
|
prepared_by_id: dict[str, tuple[str, str]],
|
||||||
|
prepared_by_old_path: dict[str, list[tuple[str, str]]],
|
||||||
|
first_prepared: tuple[str, str] | None,
|
||||||
|
) -> None:
|
||||||
|
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||||
|
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||||
|
if selected is None:
|
||||||
|
continue
|
||||||
|
base_path_id, local_root_string = selected
|
||||||
|
rule["base_path_id"] = base_path_id
|
||||||
|
rule["base_dir"] = local_root_string
|
||||||
|
if first_prepared is not None:
|
||||||
|
attachments["base_path"] = first_prepared[1]
|
||||||
|
|
||||||
|
|
||||||
def prepare_campaign_snapshot(
|
def prepare_campaign_snapshot(
|
||||||
session: Session,
|
session: Session,
|
||||||
*,
|
*,
|
||||||
@@ -135,6 +413,9 @@ def prepare_campaign_snapshot(
|
|||||||
raw_json: dict[str, Any],
|
raw_json: dict[str, Any],
|
||||||
destination: Path,
|
destination: Path,
|
||||||
include_bytes: bool,
|
include_bytes: bool,
|
||||||
|
include_unlinked_candidates: bool = False,
|
||||||
|
user_id: str = "",
|
||||||
|
is_admin: bool = False,
|
||||||
) -> PreparedCampaignSnapshot:
|
) -> PreparedCampaignSnapshot:
|
||||||
"""Create a temporary file-oriented campaign snapshot for managed attachments.
|
"""Create a temporary file-oriented campaign snapshot for managed attachments.
|
||||||
|
|
||||||
@@ -150,94 +431,35 @@ def prepare_campaign_snapshot(
|
|||||||
materialized_root = destination / "managed-attachments"
|
materialized_root = destination / "managed-attachments"
|
||||||
materialized_root.mkdir(parents=True, exist_ok=True)
|
materialized_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
prepared_json, attachments, base_paths = _prepared_attachment_config(raw_json)
|
||||||
attachments = prepared_json.get("attachments")
|
shared_assets, candidate_assets, linked_asset_ids = _campaign_snapshot_assets(
|
||||||
if not isinstance(attachments, dict):
|
|
||||||
attachments = {}
|
|
||||||
prepared_json["attachments"] = attachments
|
|
||||||
base_paths = attachments.get("base_paths")
|
|
||||||
if not isinstance(base_paths, list):
|
|
||||||
base_paths = []
|
|
||||||
|
|
||||||
shared_assets = list_assets_for_user(
|
|
||||||
session,
|
session,
|
||||||
tenant_id=tenant_id,
|
tenant_id=tenant_id,
|
||||||
user_id="",
|
|
||||||
campaign_id=campaign_id,
|
campaign_id=campaign_id,
|
||||||
is_admin=True,
|
prepared_json=prepared_json,
|
||||||
|
include_unlinked_candidates=include_unlinked_candidates,
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
assets_by_owner = _assets_grouped_by_owner(candidate_assets)
|
||||||
|
version_blobs = current_versions_and_blobs(session, candidate_assets)
|
||||||
|
backend = get_storage_backend() if include_bytes else None
|
||||||
|
manifest, prepared_by_id, prepared_by_old_path, first_prepared = _prepare_managed_base_paths(
|
||||||
|
base_paths,
|
||||||
|
materialized_root=materialized_root,
|
||||||
|
assets_by_owner=assets_by_owner,
|
||||||
|
version_blobs=version_blobs,
|
||||||
|
backend=backend,
|
||||||
|
include_bytes=include_bytes,
|
||||||
|
linked_asset_ids=linked_asset_ids,
|
||||||
|
)
|
||||||
|
_rewrite_managed_attachment_rules(
|
||||||
|
attachments,
|
||||||
|
prepared_json,
|
||||||
|
prepared_by_id=prepared_by_id,
|
||||||
|
prepared_by_old_path=prepared_by_old_path,
|
||||||
|
first_prepared=first_prepared,
|
||||||
)
|
)
|
||||||
|
|
||||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
|
||||||
for asset in shared_assets:
|
|
||||||
owner_id = _asset_owner_id(asset)
|
|
||||||
if owner_id:
|
|
||||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
|
||||||
|
|
||||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
|
||||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
|
||||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
|
||||||
first_prepared: tuple[str, str] | None = None
|
|
||||||
|
|
||||||
for index, item in enumerate(base_paths):
|
|
||||||
if not isinstance(item, dict):
|
|
||||||
continue
|
|
||||||
parsed_source = parse_managed_source(item.get("source"))
|
|
||||||
if parsed_source is None:
|
|
||||||
continue
|
|
||||||
owner_type, owner_id = parsed_source
|
|
||||||
old_path = str(item.get("path") or ".").strip() or "."
|
|
||||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
|
||||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
|
||||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
|
||||||
local_root.mkdir(parents=True, exist_ok=True)
|
|
||||||
local_root_string = str(local_root.resolve())
|
|
||||||
|
|
||||||
prepared = (base_path_id, local_root_string)
|
|
||||||
prepared_by_id[base_path_id] = prepared
|
|
||||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
|
||||||
if first_prepared is None:
|
|
||||||
first_prepared = prepared
|
|
||||||
|
|
||||||
item["path"] = local_root_string
|
|
||||||
|
|
||||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
|
||||||
relative_path = _relative_asset_path(asset, logical_root)
|
|
||||||
if not relative_path:
|
|
||||||
continue
|
|
||||||
target = _safe_local_target(local_root, relative_path)
|
|
||||||
target.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
if include_bytes:
|
|
||||||
data, version, blob = read_asset_bytes(session, asset)
|
|
||||||
target.write_bytes(data)
|
|
||||||
else:
|
|
||||||
version, blob = current_version_and_blob(session, asset)
|
|
||||||
target.touch()
|
|
||||||
local_key = str(target.resolve())
|
|
||||||
manifest[local_key] = ManagedAttachmentFile(
|
|
||||||
local_path=local_key,
|
|
||||||
asset_id=asset.id,
|
|
||||||
version_id=version.id,
|
|
||||||
blob_id=blob.id,
|
|
||||||
display_path=asset.display_path,
|
|
||||||
relative_path=normalize_logical_path(relative_path),
|
|
||||||
filename=asset.filename,
|
|
||||||
owner_type=asset.owner_type,
|
|
||||||
owner_id=owner_id,
|
|
||||||
checksum_sha256=blob.checksum_sha256,
|
|
||||||
size_bytes=blob.size_bytes,
|
|
||||||
content_type=blob.content_type,
|
|
||||||
)
|
|
||||||
|
|
||||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
|
||||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
|
||||||
if selected is None:
|
|
||||||
continue
|
|
||||||
base_path_id, local_root_string = selected
|
|
||||||
rule["base_path_id"] = base_path_id
|
|
||||||
rule["base_dir"] = local_root_string
|
|
||||||
|
|
||||||
if first_prepared is not None:
|
|
||||||
attachments["base_path"] = first_prepared[1]
|
|
||||||
|
|
||||||
snapshot_path = destination / "campaign.json"
|
snapshot_path = destination / "campaign.json"
|
||||||
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
|
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
@@ -246,6 +468,7 @@ def prepare_campaign_snapshot(
|
|||||||
raw_json=prepared_json,
|
raw_json=prepared_json,
|
||||||
managed_files_by_local_path=manifest,
|
managed_files_by_local_path=manifest,
|
||||||
shared_assets=shared_assets,
|
shared_assets=shared_assets,
|
||||||
|
candidate_assets=candidate_assets,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -257,7 +480,10 @@ def prepared_campaign_snapshot(
|
|||||||
campaign_id: str,
|
campaign_id: str,
|
||||||
raw_json: dict[str, Any],
|
raw_json: dict[str, Any],
|
||||||
include_bytes: bool,
|
include_bytes: bool,
|
||||||
prefix: str = "multimailer-managed-campaign-",
|
prefix: str = "govoplan-managed-campaign-",
|
||||||
|
include_unlinked_candidates: bool = False,
|
||||||
|
user_id: str = "",
|
||||||
|
is_admin: bool = False,
|
||||||
) -> Iterator[PreparedCampaignSnapshot]:
|
) -> Iterator[PreparedCampaignSnapshot]:
|
||||||
temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
|
temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
|
||||||
try:
|
try:
|
||||||
@@ -268,6 +494,9 @@ def prepared_campaign_snapshot(
|
|||||||
raw_json=raw_json,
|
raw_json=raw_json,
|
||||||
destination=temp_dir,
|
destination=temp_dir,
|
||||||
include_bytes=include_bytes,
|
include_bytes=include_bytes,
|
||||||
|
include_unlinked_candidates=include_unlinked_candidates,
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
)
|
)
|
||||||
finally:
|
finally:
|
||||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||||
@@ -285,6 +514,53 @@ def managed_match_payloads(
|
|||||||
return payloads
|
return payloads
|
||||||
|
|
||||||
|
|
||||||
|
def share_assets_with_campaign(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
campaign_id: str,
|
||||||
|
file_ids: list[str],
|
||||||
|
user_id: str,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
unique_ids = list(dict.fromkeys(file_id for file_id in file_ids if file_id))
|
||||||
|
if not unique_ids:
|
||||||
|
return []
|
||||||
|
assets = [
|
||||||
|
get_asset_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
asset_id=file_id,
|
||||||
|
require_write=True,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
for file_id in unique_ids
|
||||||
|
]
|
||||||
|
shares = share_files(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
assets=assets,
|
||||||
|
target_type="campaign",
|
||||||
|
target_id=campaign_id,
|
||||||
|
permission="read",
|
||||||
|
user_id=user_id,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"id": share.id,
|
||||||
|
"file_asset_id": share.file_asset_id,
|
||||||
|
"target_type": share.target_type,
|
||||||
|
"target_id": share.target_id,
|
||||||
|
"permission": share.permission,
|
||||||
|
"created_at": share.created_at.isoformat() if share.created_at else None,
|
||||||
|
"revoked_at": share.revoked_at.isoformat() if share.revoked_at else None,
|
||||||
|
}
|
||||||
|
for share in shares
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
|
def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
|
||||||
"""Return an attachment summary without temporary materialization paths.
|
"""Return an attachment summary without temporary materialization paths.
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +1,17 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
|
from dataclasses import dataclass, field
|
||||||
from pathlib import PurePosixPath
|
from pathlib import PurePosixPath
|
||||||
from typing import TYPE_CHECKING, Iterable
|
from typing import Any, Iterable
|
||||||
|
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileVersion
|
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileVersion
|
||||||
from govoplan_files.backend.storage.common import utcnow
|
from govoplan_files.backend.storage.common import utcnow
|
||||||
from govoplan_files.backend.storage.files import current_version_and_blob, list_assets_for_user
|
from govoplan_files.backend.storage.files import current_versions_and_blobs, list_assets_for_user
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
CampaignJobLike = Any
|
||||||
from govoplan_campaign.backend.db.models import CampaignJob
|
|
||||||
|
|
||||||
|
|
||||||
def _candidate_match_keys(raw_match: str) -> set[str]:
|
def _candidate_match_keys(raw_match: str) -> set[str]:
|
||||||
@@ -25,11 +25,20 @@ def _candidate_match_keys(raw_match: str) -> set[str]:
|
|||||||
AttachmentUseKey = tuple[str, str, str, str]
|
AttachmentUseKey = tuple[str, str, str, str]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class _AttachmentBatchRefs:
|
||||||
|
managed_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||||
|
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||||
|
asset_ids: set[str] = field(default_factory=set)
|
||||||
|
version_ids: set[str] = field(default_factory=set)
|
||||||
|
blob_ids: set[str] = field(default_factory=set)
|
||||||
|
|
||||||
|
|
||||||
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
|
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
|
||||||
return job_id, file_version_id, filename_used, stage
|
return job_id, file_version_id, filename_used, stage
|
||||||
|
|
||||||
|
|
||||||
def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJob], *, stage: str) -> set[AttachmentUseKey]:
|
def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJobLike], *, stage: str) -> set[AttachmentUseKey]:
|
||||||
job_ids = {job.id for job in jobs if job.id}
|
job_ids = {job.id for job in jobs if job.id}
|
||||||
if not job_ids:
|
if not job_ids:
|
||||||
return set()
|
return set()
|
||||||
@@ -72,13 +81,13 @@ def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJob], *, s
|
|||||||
return keys
|
return keys
|
||||||
|
|
||||||
|
|
||||||
def _known_use_keys(session: Session, job: CampaignJob, *, stage: str) -> set[AttachmentUseKey]:
|
def _known_use_keys(session: Session, job: CampaignJobLike, *, stage: str) -> set[AttachmentUseKey]:
|
||||||
return _known_use_keys_for_jobs(session, [job], stage=stage)
|
return _known_use_keys_for_jobs(session, [job], stage=stage)
|
||||||
|
|
||||||
|
|
||||||
def _add_use(
|
def _add_use(
|
||||||
session: Session,
|
session: Session,
|
||||||
job: CampaignJob,
|
job: CampaignJobLike,
|
||||||
*,
|
*,
|
||||||
asset: FileAsset,
|
asset: FileAsset,
|
||||||
version: FileVersion,
|
version: FileVersion,
|
||||||
@@ -118,7 +127,7 @@ def _add_use(
|
|||||||
|
|
||||||
def record_campaign_attachment_uses_for_jobs(
|
def record_campaign_attachment_uses_for_jobs(
|
||||||
session: Session,
|
session: Session,
|
||||||
jobs: Iterable[CampaignJob],
|
jobs: Iterable[CampaignJobLike],
|
||||||
*,
|
*,
|
||||||
stage: str = "built",
|
stage: str = "built",
|
||||||
) -> None:
|
) -> None:
|
||||||
@@ -135,62 +144,91 @@ def record_campaign_attachment_uses_for_jobs(
|
|||||||
if not job_list:
|
if not job_list:
|
||||||
return
|
return
|
||||||
|
|
||||||
managed_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
refs = _collect_attachment_batch_refs(job_list)
|
||||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
|
||||||
job_by_id = {job.id: job for job in job_list}
|
job_by_id = {job.id: job for job in job_list}
|
||||||
asset_ids: set[str] = set()
|
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||||
version_ids: set[str] = set()
|
assets_by_id, versions_by_id, blobs_by_id = _load_managed_attachment_entities(session, refs)
|
||||||
blob_ids: set[str] = set()
|
|
||||||
|
|
||||||
for job in job_list:
|
_record_managed_attachment_uses(
|
||||||
|
session,
|
||||||
|
job_by_id=job_by_id,
|
||||||
|
managed_by_job=refs.managed_by_job,
|
||||||
|
assets_by_id=assets_by_id,
|
||||||
|
versions_by_id=versions_by_id,
|
||||||
|
blobs_by_id=blobs_by_id,
|
||||||
|
stage=stage,
|
||||||
|
known_keys=known_keys,
|
||||||
|
)
|
||||||
|
_record_fallback_attachment_uses(
|
||||||
|
session,
|
||||||
|
job_by_id=job_by_id,
|
||||||
|
fallback_attachments_by_job=refs.fallback_attachments_by_job,
|
||||||
|
stage=stage,
|
||||||
|
known_keys=known_keys,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_attachment_batch_refs(jobs: list[CampaignJobLike]) -> _AttachmentBatchRefs:
|
||||||
|
refs = _AttachmentBatchRefs()
|
||||||
|
for job in jobs:
|
||||||
attachments = job.resolved_attachments or []
|
attachments = job.resolved_attachments or []
|
||||||
if not isinstance(attachments, list):
|
if not isinstance(attachments, list):
|
||||||
continue
|
continue
|
||||||
for attachment in attachments:
|
for attachment in attachments:
|
||||||
if not isinstance(attachment, dict):
|
if not isinstance(attachment, dict):
|
||||||
continue
|
continue
|
||||||
managed_matches = attachment.get("managed_matches")
|
if not _collect_managed_attachment_refs(refs, job.id, attachment):
|
||||||
if isinstance(managed_matches, list) and managed_matches:
|
refs.fallback_attachments_by_job[job.id].append(attachment)
|
||||||
for item in managed_matches:
|
return refs
|
||||||
if not isinstance(item, dict):
|
|
||||||
continue
|
|
||||||
asset_id = str(item.get("asset_id") or "")
|
|
||||||
version_id = str(item.get("version_id") or "")
|
|
||||||
blob_id = str(item.get("blob_id") or "")
|
|
||||||
if not asset_id or not version_id or not blob_id:
|
|
||||||
continue
|
|
||||||
managed_by_job[job.id].append(item)
|
|
||||||
asset_ids.add(asset_id)
|
|
||||||
version_ids.add(version_id)
|
|
||||||
blob_ids.add(blob_id)
|
|
||||||
else:
|
|
||||||
fallback_attachments_by_job[job.id].append(attachment)
|
|
||||||
|
|
||||||
assets_by_id = {
|
|
||||||
item.id: item
|
|
||||||
for item in session.query(FileAsset).filter(FileAsset.id.in_(asset_ids)).all()
|
|
||||||
} if asset_ids else {}
|
|
||||||
versions_by_id = {
|
|
||||||
item.id: item
|
|
||||||
for item in session.query(FileVersion).filter(FileVersion.id.in_(version_ids)).all()
|
|
||||||
} if version_ids else {}
|
|
||||||
blobs_by_id = {
|
|
||||||
item.id: item
|
|
||||||
for item in session.query(FileBlob).filter(FileBlob.id.in_(blob_ids)).all()
|
|
||||||
} if blob_ids else {}
|
|
||||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
|
||||||
|
|
||||||
|
def _collect_managed_attachment_refs(refs: _AttachmentBatchRefs, job_id: str, attachment: dict[str, object]) -> bool:
|
||||||
|
managed_matches = attachment.get("managed_matches")
|
||||||
|
if not isinstance(managed_matches, list) or not managed_matches:
|
||||||
|
return False
|
||||||
|
for item in managed_matches:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
asset_id = str(item.get("asset_id") or "")
|
||||||
|
version_id = str(item.get("version_id") or "")
|
||||||
|
blob_id = str(item.get("blob_id") or "")
|
||||||
|
if not asset_id or not version_id or not blob_id:
|
||||||
|
continue
|
||||||
|
refs.managed_by_job[job_id].append(item)
|
||||||
|
refs.asset_ids.add(asset_id)
|
||||||
|
refs.version_ids.add(version_id)
|
||||||
|
refs.blob_ids.add(blob_id)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _load_managed_attachment_entities(
|
||||||
|
session: Session,
|
||||||
|
refs: _AttachmentBatchRefs,
|
||||||
|
) -> tuple[dict[str, FileAsset], dict[str, FileVersion], dict[str, FileBlob]]:
|
||||||
|
assets_by_id = {item.id: item for item in session.query(FileAsset).filter(FileAsset.id.in_(refs.asset_ids)).all()} if refs.asset_ids else {}
|
||||||
|
versions_by_id = {item.id: item for item in session.query(FileVersion).filter(FileVersion.id.in_(refs.version_ids)).all()} if refs.version_ids else {}
|
||||||
|
blobs_by_id = {item.id: item for item in session.query(FileBlob).filter(FileBlob.id.in_(refs.blob_ids)).all()} if refs.blob_ids else {}
|
||||||
|
return assets_by_id, versions_by_id, blobs_by_id
|
||||||
|
|
||||||
|
|
||||||
|
def _record_managed_attachment_uses(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
job_by_id: dict[str, CampaignJobLike],
|
||||||
|
managed_by_job: dict[str, list[dict[str, object]]],
|
||||||
|
assets_by_id: dict[str, FileAsset],
|
||||||
|
versions_by_id: dict[str, FileVersion],
|
||||||
|
blobs_by_id: dict[str, FileBlob],
|
||||||
|
stage: str,
|
||||||
|
known_keys: set[AttachmentUseKey],
|
||||||
|
) -> None:
|
||||||
for job_id, items in managed_by_job.items():
|
for job_id, items in managed_by_job.items():
|
||||||
job = job_by_id[job_id]
|
job = job_by_id[job_id]
|
||||||
for item in items:
|
for item in items:
|
||||||
asset = assets_by_id.get(str(item.get("asset_id") or ""))
|
asset = assets_by_id.get(str(item.get("asset_id") or ""))
|
||||||
version = versions_by_id.get(str(item.get("version_id") or ""))
|
version = versions_by_id.get(str(item.get("version_id") or ""))
|
||||||
blob = blobs_by_id.get(str(item.get("blob_id") or ""))
|
blob = blobs_by_id.get(str(item.get("blob_id") or ""))
|
||||||
if not asset or not version or not blob:
|
if not _managed_attachment_entities_match_job(job, asset, version, blob):
|
||||||
continue
|
|
||||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
|
||||||
continue
|
|
||||||
if version.file_asset_id != asset.id or version.blob_id != blob.id:
|
|
||||||
continue
|
continue
|
||||||
_add_use(
|
_add_use(
|
||||||
session,
|
session,
|
||||||
@@ -203,53 +241,109 @@ def record_campaign_attachment_uses_for_jobs(
|
|||||||
known_keys=known_keys,
|
known_keys=known_keys,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _managed_attachment_entities_match_job(
|
||||||
|
job: CampaignJobLike,
|
||||||
|
asset: FileAsset | None,
|
||||||
|
version: FileVersion | None,
|
||||||
|
blob: FileBlob | None,
|
||||||
|
) -> bool:
|
||||||
|
if not asset or not version or not blob:
|
||||||
|
return False
|
||||||
|
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||||
|
return False
|
||||||
|
return version.file_asset_id == asset.id and version.blob_id == blob.id
|
||||||
|
|
||||||
|
|
||||||
|
def _record_fallback_attachment_uses(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
job_by_id: dict[str, CampaignJobLike],
|
||||||
|
fallback_attachments_by_job: dict[str, list[dict[str, object]]],
|
||||||
|
stage: str,
|
||||||
|
known_keys: set[AttachmentUseKey],
|
||||||
|
) -> None:
|
||||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
|
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
|
||||||
|
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {}
|
||||||
for job_id, attachments in fallback_attachments_by_job.items():
|
for job_id, attachments in fallback_attachments_by_job.items():
|
||||||
job = job_by_id[job_id]
|
job = job_by_id[job_id]
|
||||||
campaign_key = (job.tenant_id, job.campaign_id)
|
campaign_key = (job.tenant_id, job.campaign_id)
|
||||||
by_key = assets_by_campaign.get(campaign_key)
|
by_key, version_blobs = _fallback_campaign_assets(
|
||||||
if by_key is None:
|
session,
|
||||||
assets = list_assets_for_user(
|
job,
|
||||||
session,
|
campaign_key=campaign_key,
|
||||||
tenant_id=job.tenant_id,
|
assets_by_campaign=assets_by_campaign,
|
||||||
user_id="",
|
version_blobs_by_campaign=version_blobs_by_campaign,
|
||||||
campaign_id=job.campaign_id,
|
)
|
||||||
is_admin=True,
|
|
||||||
)
|
|
||||||
by_key = {}
|
|
||||||
for asset in assets:
|
|
||||||
by_key[asset.display_path.strip("/")] = asset
|
|
||||||
by_key.setdefault(asset.filename, asset)
|
|
||||||
assets_by_campaign[campaign_key] = by_key
|
|
||||||
|
|
||||||
for attachment in attachments:
|
for attachment in attachments:
|
||||||
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
_record_fallback_attachment(session, job, attachment, by_key=by_key, version_blobs=version_blobs, stage=stage, known_keys=known_keys)
|
||||||
for raw in matches:
|
|
||||||
if not isinstance(raw, str):
|
|
||||||
continue
|
|
||||||
asset = next((by_key[key] for key in _candidate_match_keys(raw) if key in by_key), None)
|
|
||||||
if not asset:
|
|
||||||
continue
|
|
||||||
version, blob = current_version_and_blob(session, asset)
|
|
||||||
_add_use(
|
|
||||||
session,
|
|
||||||
job,
|
|
||||||
asset=asset,
|
|
||||||
version=version,
|
|
||||||
blob=blob,
|
|
||||||
filename_used=asset.filename,
|
|
||||||
stage=stage,
|
|
||||||
known_keys=known_keys,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def record_campaign_attachment_uses_for_job(session: Session, job: CampaignJob, *, stage: str = "built") -> None:
|
def _fallback_campaign_assets(
|
||||||
|
session: Session,
|
||||||
|
job: CampaignJobLike,
|
||||||
|
*,
|
||||||
|
campaign_key: tuple[str, str],
|
||||||
|
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]],
|
||||||
|
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]],
|
||||||
|
) -> tuple[dict[str, FileAsset], dict[str, tuple[FileVersion, FileBlob]]]:
|
||||||
|
by_key = assets_by_campaign.get(campaign_key)
|
||||||
|
if by_key is None:
|
||||||
|
assets = list_assets_for_user(session, tenant_id=job.tenant_id, user_id="", campaign_id=job.campaign_id, is_admin=True)
|
||||||
|
by_key = _asset_lookup_by_path_and_filename(assets)
|
||||||
|
assets_by_campaign[campaign_key] = by_key
|
||||||
|
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
|
||||||
|
return by_key, version_blobs_by_campaign[campaign_key]
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_lookup_by_path_and_filename(assets: list[FileAsset]) -> dict[str, FileAsset]:
|
||||||
|
by_key: dict[str, FileAsset] = {}
|
||||||
|
for asset in assets:
|
||||||
|
by_key[asset.display_path.strip("/")] = asset
|
||||||
|
by_key.setdefault(asset.filename, asset)
|
||||||
|
return by_key
|
||||||
|
|
||||||
|
|
||||||
|
def _record_fallback_attachment(
|
||||||
|
session: Session,
|
||||||
|
job: CampaignJobLike,
|
||||||
|
attachment: dict[str, object],
|
||||||
|
*,
|
||||||
|
by_key: dict[str, FileAsset],
|
||||||
|
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||||
|
stage: str,
|
||||||
|
known_keys: set[AttachmentUseKey],
|
||||||
|
) -> None:
|
||||||
|
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
||||||
|
for raw in matches:
|
||||||
|
if not isinstance(raw, str):
|
||||||
|
continue
|
||||||
|
asset = next((by_key[key] for key in _candidate_match_keys(raw) if key in by_key), None)
|
||||||
|
if not asset:
|
||||||
|
continue
|
||||||
|
version_blob = version_blobs.get(asset.id)
|
||||||
|
if not version_blob:
|
||||||
|
continue
|
||||||
|
version, blob = version_blob
|
||||||
|
_add_use(
|
||||||
|
session,
|
||||||
|
job,
|
||||||
|
asset=asset,
|
||||||
|
version=version,
|
||||||
|
blob=blob,
|
||||||
|
filename_used=asset.filename,
|
||||||
|
stage=stage,
|
||||||
|
known_keys=known_keys,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def record_campaign_attachment_uses_for_job(session: Session, job: CampaignJobLike, *, stage: str = "built") -> None:
|
||||||
"""Record immutable managed file versions used by one built/sent job."""
|
"""Record immutable managed file versions used by one built/sent job."""
|
||||||
|
|
||||||
record_campaign_attachment_uses_for_jobs(session, [job], stage=stage)
|
record_campaign_attachment_uses_for_jobs(session, [job], stage=stage)
|
||||||
|
|
||||||
|
|
||||||
def mark_job_attachment_uses_sent(session: Session, job: CampaignJob) -> None:
|
def mark_job_attachment_uses_sent(session: Session, job: CampaignJobLike) -> None:
|
||||||
record_campaign_attachment_uses_for_job(session, job, stage="built")
|
record_campaign_attachment_uses_for_job(session, job, stage="built")
|
||||||
# Sessions use autoflush=False. Flush any compatibility-built evidence so
|
# Sessions use autoflush=False. Flush any compatibility-built evidence so
|
||||||
# the following query can copy it to the sent stage in the same call.
|
# the following query can copy it to the sent stage in the same call.
|
||||||
|
|||||||
@@ -0,0 +1,923 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from email.utils import parsedate_to_datetime
|
||||||
|
from importlib import import_module
|
||||||
|
import mimetypes
|
||||||
|
from typing import Any
|
||||||
|
from urllib.parse import quote, unquote, urljoin, urlsplit
|
||||||
|
import xml.etree.ElementTree as ET # nosec B405 - typing/element creation only; parsing uses defusedxml below.
|
||||||
|
|
||||||
|
from defusedxml import ElementTree as SafeElementTree
|
||||||
|
|
||||||
|
from govoplan_core.security.outbound_http import (
|
||||||
|
OutboundHttpError,
|
||||||
|
validate_outbound_host,
|
||||||
|
validate_outbound_http_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
ConnectorDeploymentConfigurationError,
|
||||||
|
connector_ca_bundle_path,
|
||||||
|
connector_secret_env_value,
|
||||||
|
validate_connector_tls_metadata,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
from govoplan_files.backend.storage.sdk_peer_pinning import (
|
||||||
|
SdkPeerPinningError,
|
||||||
|
create_pinned_s3_client,
|
||||||
|
install_pinned_smb_transport,
|
||||||
|
pinned_smb_connection_cache,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorBrowseError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorBrowseUnsupported(ConnectorBrowseError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorBrowseItem:
|
||||||
|
kind: str
|
||||||
|
name: str
|
||||||
|
path: str
|
||||||
|
external_id: str | None = None
|
||||||
|
external_url: str | None = None
|
||||||
|
size_bytes: int | None = None
|
||||||
|
content_type: str | None = None
|
||||||
|
modified_at: str | None = None
|
||||||
|
etag: str | None = None
|
||||||
|
metadata: Mapping[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def to_response(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"kind": self.kind,
|
||||||
|
"name": self.name,
|
||||||
|
"path": self.path,
|
||||||
|
"external_id": self.external_id,
|
||||||
|
"external_url": self.external_url,
|
||||||
|
"size_bytes": self.size_bytes,
|
||||||
|
"content_type": self.content_type,
|
||||||
|
"modified_at": self.modified_at,
|
||||||
|
"etag": self.etag,
|
||||||
|
"metadata": dict(self.metadata),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None, continuation_token: str | None = None) -> list[ConnectorBrowseItem]:
|
||||||
|
browse_path = normalize_connector_browse_path(path)
|
||||||
|
static_items = _static_listing(profile, path=browse_path, library_id=library_id)
|
||||||
|
if static_items is not None:
|
||||||
|
return static_items
|
||||||
|
if profile.provider == "seafile":
|
||||||
|
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||||
|
return _browse_webdav(profile, path=browse_path)
|
||||||
|
return _browse_seafile(profile, path=browse_path, library_id=library_id)
|
||||||
|
if profile.provider in {"webdav", "nextcloud"} or _metadata_string(profile, "browse_protocol") == "webdav":
|
||||||
|
return _browse_webdav(profile, path=browse_path)
|
||||||
|
if profile.provider == "smb":
|
||||||
|
return _browse_smb(profile, path=browse_path)
|
||||||
|
if profile.provider == "s3":
|
||||||
|
return _browse_s3(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
|
||||||
|
raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_webdav_multistatus(*, root_url: str, current_path: str, payload: str | bytes) -> list[ConnectorBrowseItem]:
|
||||||
|
try:
|
||||||
|
root = SafeElementTree.fromstring(payload)
|
||||||
|
except SafeElementTree.ParseError as exc:
|
||||||
|
raise ConnectorBrowseError("Connector returned invalid WebDAV XML") from exc
|
||||||
|
root_path = _url_path_root(root_url)
|
||||||
|
current = normalize_connector_browse_path(current_path)
|
||||||
|
items: list[ConnectorBrowseItem] = []
|
||||||
|
for response in root.findall("{DAV:}response"):
|
||||||
|
href = response.findtext("{DAV:}href")
|
||||||
|
if not href:
|
||||||
|
continue
|
||||||
|
relative_path = _relative_href_path(root_path, href)
|
||||||
|
if relative_path == current:
|
||||||
|
continue
|
||||||
|
if current and not relative_path.startswith(current.rstrip("/") + "/"):
|
||||||
|
continue
|
||||||
|
parent = relative_path.rsplit("/", 1)[0] if "/" in relative_path else ""
|
||||||
|
if parent != current:
|
||||||
|
continue
|
||||||
|
prop = _webdav_prop(response)
|
||||||
|
is_collection = prop.find("{DAV:}resourcetype/{DAV:}collection") is not None if prop is not None else False
|
||||||
|
name = _webdav_display_name(prop) or _path_name(relative_path)
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
items.append(
|
||||||
|
ConnectorBrowseItem(
|
||||||
|
kind="folder" if is_collection else "file",
|
||||||
|
name=name,
|
||||||
|
path=relative_path,
|
||||||
|
external_id=_text(prop, "{DAV:}getetag") or relative_path,
|
||||||
|
size_bytes=None if is_collection else _int(_text(prop, "{DAV:}getcontentlength")),
|
||||||
|
content_type=None if is_collection else _text(prop, "{DAV:}getcontenttype"),
|
||||||
|
modified_at=_http_date(_text(prop, "{DAV:}getlastmodified")),
|
||||||
|
etag=_text(prop, "{DAV:}getetag"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return sorted(items, key=lambda item: (item.kind != "library", item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||||
|
|
||||||
|
|
||||||
|
def seafile_libraries_from_payload(payload: object) -> list[ConnectorBrowseItem]:
|
||||||
|
if not isinstance(payload, list):
|
||||||
|
raise ConnectorBrowseError("Seafile library response must be a list")
|
||||||
|
libraries = [_seafile_library_item(item) for item in payload if isinstance(item, Mapping)]
|
||||||
|
return sorted(libraries, key=lambda item: item.name.casefold())
|
||||||
|
|
||||||
|
|
||||||
|
def seafile_directory_items_from_payload(payload: object, *, path: str | None = None) -> list[ConnectorBrowseItem]:
|
||||||
|
if payload == "uptodate":
|
||||||
|
return []
|
||||||
|
if not isinstance(payload, list):
|
||||||
|
raise ConnectorBrowseError("Seafile directory response must be a list")
|
||||||
|
browse_path = normalize_connector_browse_path(path)
|
||||||
|
items = [_seafile_directory_item(item, parent_path=browse_path) for item in payload if isinstance(item, Mapping)]
|
||||||
|
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||||
|
|
||||||
|
|
||||||
|
def _browse_seafile(profile: ConnectorProfile, *, path: str, library_id: str | None) -> list[ConnectorBrowseItem]:
|
||||||
|
repo_id, dir_path = _seafile_repo_and_path(path=path, library_id=library_id)
|
||||||
|
headers = _seafile_headers(profile)
|
||||||
|
if not repo_id:
|
||||||
|
payload = _request_json("GET", _seafile_url(profile, "api2/repos/"), headers=headers)
|
||||||
|
return seafile_libraries_from_payload(payload)
|
||||||
|
payload = _request_json(
|
||||||
|
"GET",
|
||||||
|
_seafile_url(profile, f"api2/repos/{quote(repo_id, safe='')}/dir/"),
|
||||||
|
headers=headers,
|
||||||
|
params={"p": "/" + dir_path if dir_path else "/"},
|
||||||
|
)
|
||||||
|
return seafile_directory_items_from_payload(payload, path=dir_path)
|
||||||
|
|
||||||
|
|
||||||
|
def _browse_webdav(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||||
|
root_url = _metadata_string(profile, "webdav_endpoint_url") or profile.endpoint_url
|
||||||
|
if not root_url:
|
||||||
|
raise ConnectorBrowseError("Connector profile does not define an endpoint URL")
|
||||||
|
url = _webdav_url(root_url, path)
|
||||||
|
headers = {"Depth": "1", "Content-Type": "application/xml; charset=utf-8"}
|
||||||
|
auth: tuple[str, str] | None = None
|
||||||
|
password = _profile_password(profile)
|
||||||
|
token = _profile_token(profile)
|
||||||
|
if profile.username and password:
|
||||||
|
auth = (profile.username, password)
|
||||||
|
elif token:
|
||||||
|
headers["Authorization"] = f"Bearer {token}"
|
||||||
|
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||||
|
raise ConnectorBrowseError("Secret-ref connector credentials need a runtime secret resolver before live browsing")
|
||||||
|
body = """<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<propfind xmlns="DAV:">
|
||||||
|
<prop>
|
||||||
|
<displayname />
|
||||||
|
<resourcetype />
|
||||||
|
<getcontentlength />
|
||||||
|
<getcontenttype />
|
||||||
|
<getlastmodified />
|
||||||
|
<getetag />
|
||||||
|
</prop>
|
||||||
|
</propfind>"""
|
||||||
|
try:
|
||||||
|
response = request_connector_bytes(
|
||||||
|
"PROPFIND",
|
||||||
|
url,
|
||||||
|
headers=headers,
|
||||||
|
content=body,
|
||||||
|
auth=auth,
|
||||||
|
timeout=15.0,
|
||||||
|
label="WebDAV browse",
|
||||||
|
)
|
||||||
|
except ConnectorHttpError as exc:
|
||||||
|
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||||
|
if response.status_code in {401, 403}:
|
||||||
|
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||||
|
if response.status_code not in {200, 207}:
|
||||||
|
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||||
|
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.content)
|
||||||
|
|
||||||
|
|
||||||
|
def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||||
|
location = _smb_location(profile)
|
||||||
|
unc_path = _smb_unc_path(location, path)
|
||||||
|
smbclient = _smbclient_module()
|
||||||
|
try:
|
||||||
|
entries = smbclient.scandir(unc_path, **_smb_client_kwargs(profile, location))
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorBrowseError(f"SMB connector browse failed: {exc}") from exc
|
||||||
|
items: list[ConnectorBrowseItem] = []
|
||||||
|
try:
|
||||||
|
with entries as iterator:
|
||||||
|
for entry in iterator:
|
||||||
|
name = _clean(getattr(entry, "name", None))
|
||||||
|
if not name or name in {".", ".."}:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
is_dir = bool(entry.is_dir())
|
||||||
|
except Exception:
|
||||||
|
is_dir = False
|
||||||
|
stat_result = _smb_entry_stat(entry)
|
||||||
|
item_path = _join_browse_path(path, name)
|
||||||
|
items.append(
|
||||||
|
ConnectorBrowseItem(
|
||||||
|
kind="folder" if is_dir else "file",
|
||||||
|
name=name,
|
||||||
|
path=item_path,
|
||||||
|
external_id=f"{location.share}:{item_path}",
|
||||||
|
size_bytes=None if is_dir else _smb_stat_size(stat_result),
|
||||||
|
content_type=None if is_dir else mimetypes.guess_type(name)[0],
|
||||||
|
modified_at=_smb_stat_modified_at(stat_result),
|
||||||
|
etag=_smb_stat_revision(stat_result),
|
||||||
|
metadata={
|
||||||
|
"share": location.share,
|
||||||
|
**({"server": location.server} if _metadata_bool(profile, "expose_server_metadata", default=False) else {}),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except ConnectorBrowseError:
|
||||||
|
raise
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorBrowseError(f"SMB connector browse failed: {exc}") from exc
|
||||||
|
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||||
|
|
||||||
|
|
||||||
|
def _browse_s3(profile: ConnectorProfile, *, path: str, library_id: str | None, continuation_token: str | None) -> list[ConnectorBrowseItem]:
|
||||||
|
client = _s3_client(profile)
|
||||||
|
try:
|
||||||
|
return _browse_s3_with_client(
|
||||||
|
client,
|
||||||
|
profile=profile,
|
||||||
|
path=path,
|
||||||
|
library_id=library_id,
|
||||||
|
continuation_token=continuation_token,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
close = getattr(client, "close", None)
|
||||||
|
if callable(close):
|
||||||
|
close()
|
||||||
|
|
||||||
|
|
||||||
|
def _browse_s3_with_client(
|
||||||
|
client: Any,
|
||||||
|
*,
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
path: str,
|
||||||
|
library_id: str | None,
|
||||||
|
continuation_token: str | None,
|
||||||
|
) -> list[ConnectorBrowseItem]:
|
||||||
|
bucket = _s3_bucket(profile, library_id)
|
||||||
|
if not bucket:
|
||||||
|
try:
|
||||||
|
payload = client.list_buckets()
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||||
|
buckets = payload.get("Buckets") if isinstance(payload, Mapping) else None
|
||||||
|
if not isinstance(buckets, list):
|
||||||
|
raise ConnectorBrowseError("S3 connector returned invalid bucket list")
|
||||||
|
return sorted(
|
||||||
|
(
|
||||||
|
ConnectorBrowseItem(
|
||||||
|
kind="library",
|
||||||
|
name=_clean(item.get("Name")) or "",
|
||||||
|
path=_clean(item.get("Name")) or "",
|
||||||
|
external_id=_clean(item.get("Name")),
|
||||||
|
modified_at=_timestamp(item.get("CreationDate")),
|
||||||
|
metadata={"bucket": _clean(item.get("Name"))},
|
||||||
|
)
|
||||||
|
for item in buckets
|
||||||
|
if isinstance(item, Mapping) and _clean(item.get("Name"))
|
||||||
|
),
|
||||||
|
key=lambda item: item.name.casefold(),
|
||||||
|
)
|
||||||
|
prefix = _s3_object_key(profile, path, directory=True)
|
||||||
|
params: dict[str, object] = {
|
||||||
|
"Bucket": bucket,
|
||||||
|
"Prefix": prefix,
|
||||||
|
"Delimiter": "/",
|
||||||
|
"MaxKeys": _s3_max_keys(profile),
|
||||||
|
}
|
||||||
|
next_page_request = _clean(continuation_token) or _metadata_string(profile, "continuation_token")
|
||||||
|
if next_page_request:
|
||||||
|
params["ContinuationToken"] = next_page_request
|
||||||
|
try:
|
||||||
|
payload = client.list_objects_v2(**params)
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||||
|
if not isinstance(payload, Mapping):
|
||||||
|
raise ConnectorBrowseError("S3 connector returned invalid object listing")
|
||||||
|
items = [
|
||||||
|
*_s3_prefix_items(bucket=bucket, browse_path=path, base_prefix=prefix, prefixes=payload.get("CommonPrefixes")),
|
||||||
|
*_s3_object_items(bucket=bucket, browse_path=path, base_prefix=prefix, objects=payload.get("Contents")),
|
||||||
|
]
|
||||||
|
next_token = _clean(payload.get("NextContinuationToken"))
|
||||||
|
if next_token and items:
|
||||||
|
last = items[-1]
|
||||||
|
items[-1] = ConnectorBrowseItem(
|
||||||
|
kind=last.kind,
|
||||||
|
name=last.name,
|
||||||
|
path=last.path,
|
||||||
|
external_id=last.external_id,
|
||||||
|
external_url=last.external_url,
|
||||||
|
size_bytes=last.size_bytes,
|
||||||
|
content_type=last.content_type,
|
||||||
|
modified_at=last.modified_at,
|
||||||
|
etag=last.etag,
|
||||||
|
metadata={**dict(last.metadata), "next_continuation_token": next_token, "listing_truncated": True},
|
||||||
|
)
|
||||||
|
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_client(profile: ConnectorProfile) -> Any:
|
||||||
|
if profile.secret_ref:
|
||||||
|
raise ConnectorBrowseError("Secret-ref S3 credentials need a runtime secret resolver before live browsing")
|
||||||
|
if profile.endpoint_url:
|
||||||
|
try:
|
||||||
|
endpoint_url = validate_outbound_http_url(
|
||||||
|
profile.endpoint_url,
|
||||||
|
label="S3 connector endpoint",
|
||||||
|
)
|
||||||
|
except OutboundHttpError as exc:
|
||||||
|
raise ConnectorBrowseError(str(exc)) from exc
|
||||||
|
else:
|
||||||
|
endpoint_url = None
|
||||||
|
try:
|
||||||
|
config_module = import_module("botocore.config")
|
||||||
|
unsigned = import_module("botocore").UNSIGNED
|
||||||
|
except ImportError as exc:
|
||||||
|
raise ConnectorBrowseUnsupported("S3 connector browsing requires the optional boto3 dependency") from exc
|
||||||
|
kwargs: dict[str, object] = {}
|
||||||
|
if endpoint_url:
|
||||||
|
kwargs["endpoint_url"] = endpoint_url
|
||||||
|
region = _metadata_string(profile, "region") or _metadata_string(profile, "aws_region")
|
||||||
|
if region:
|
||||||
|
kwargs["region_name"] = region
|
||||||
|
access_key = profile.username or _metadata_env(profile, "access_key_id_env") or _metadata_string(profile, "access_key_id")
|
||||||
|
secret_key = _profile_password(profile) or _metadata_env(profile, "secret_access_key_env")
|
||||||
|
session_token = _profile_token(profile) or _metadata_env(profile, "session_token_env")
|
||||||
|
if access_key:
|
||||||
|
kwargs["aws_access_key_id"] = access_key
|
||||||
|
if secret_key:
|
||||||
|
kwargs["aws_secret_access_key"] = secret_key
|
||||||
|
if session_token:
|
||||||
|
kwargs["aws_session_token"] = session_token
|
||||||
|
verify = _s3_verify(profile)
|
||||||
|
if verify is not None:
|
||||||
|
kwargs["verify"] = verify
|
||||||
|
addressing_style = _s3_addressing_style(profile)
|
||||||
|
config_values: dict[str, object] = {
|
||||||
|
"proxies": {},
|
||||||
|
"retries": {"mode": "standard", "max_attempts": 4},
|
||||||
|
}
|
||||||
|
if addressing_style:
|
||||||
|
config_values["s3"] = {"addressing_style": addressing_style}
|
||||||
|
if bool(access_key) != bool(secret_key):
|
||||||
|
raise ConnectorBrowseError("S3 connectors require both an access key and a secret key")
|
||||||
|
if not access_key:
|
||||||
|
config_values["signature_version"] = unsigned
|
||||||
|
kwargs["config"] = config_module.Config(**config_values)
|
||||||
|
try:
|
||||||
|
return create_pinned_s3_client(**kwargs)
|
||||||
|
except SdkPeerPinningError as exc:
|
||||||
|
raise ConnectorBrowseError(str(exc)) from exc
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorBrowseError(f"S3 connector could not be initialized: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_bucket(profile: ConnectorProfile, library_id: str | None = None) -> str | None:
|
||||||
|
return _metadata_string(profile, "bucket") or _metadata_string(profile, "bucket_name") or _clean(library_id)
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_object_key(profile: ConnectorProfile, path: str, *, directory: bool = False) -> str:
|
||||||
|
base_prefix = normalize_connector_browse_path(profile.base_path or _metadata_string(profile, "base_prefix"))
|
||||||
|
browse_path = normalize_connector_browse_path(path)
|
||||||
|
key = "/".join(part for part in (base_prefix, browse_path) if part)
|
||||||
|
if directory and key:
|
||||||
|
return key.rstrip("/") + "/"
|
||||||
|
return key
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_prefix_items(
|
||||||
|
*,
|
||||||
|
bucket: str,
|
||||||
|
browse_path: str,
|
||||||
|
base_prefix: str,
|
||||||
|
prefixes: object,
|
||||||
|
) -> list[ConnectorBrowseItem]:
|
||||||
|
if not isinstance(prefixes, list):
|
||||||
|
return []
|
||||||
|
items: list[ConnectorBrowseItem] = []
|
||||||
|
for item in prefixes:
|
||||||
|
if not isinstance(item, Mapping):
|
||||||
|
continue
|
||||||
|
key = _clean(item.get("Prefix"))
|
||||||
|
if not key:
|
||||||
|
continue
|
||||||
|
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||||
|
name = _path_name(relative)
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
path = _join_browse_path(browse_path, name)
|
||||||
|
items.append(
|
||||||
|
ConnectorBrowseItem(
|
||||||
|
kind="folder",
|
||||||
|
name=name,
|
||||||
|
path=path,
|
||||||
|
external_id=f"{bucket}:{key}",
|
||||||
|
metadata={"bucket": bucket, "key": key},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_object_items(
|
||||||
|
*,
|
||||||
|
bucket: str,
|
||||||
|
browse_path: str,
|
||||||
|
base_prefix: str,
|
||||||
|
objects: object,
|
||||||
|
) -> list[ConnectorBrowseItem]:
|
||||||
|
if not isinstance(objects, list):
|
||||||
|
return []
|
||||||
|
items: list[ConnectorBrowseItem] = []
|
||||||
|
for item in objects:
|
||||||
|
if not isinstance(item, Mapping):
|
||||||
|
continue
|
||||||
|
key = _clean(item.get("Key"))
|
||||||
|
if not key or key == base_prefix:
|
||||||
|
continue
|
||||||
|
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||||
|
name = _path_name(relative)
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
path = _join_browse_path(browse_path, name)
|
||||||
|
items.append(
|
||||||
|
ConnectorBrowseItem(
|
||||||
|
kind="file",
|
||||||
|
name=name,
|
||||||
|
path=path,
|
||||||
|
external_id=f"{bucket}:{key}",
|
||||||
|
size_bytes=_int(item.get("Size")),
|
||||||
|
content_type=mimetypes.guess_type(name)[0],
|
||||||
|
modified_at=_timestamp(item.get("LastModified")),
|
||||||
|
etag=_clean(item.get("ETag")),
|
||||||
|
metadata={
|
||||||
|
"bucket": bucket,
|
||||||
|
"key": key,
|
||||||
|
**({"storage_class": item["StorageClass"]} if "StorageClass" in item else {}),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_relative_key(key: str, *, base_prefix: str) -> str:
|
||||||
|
clean_key = key.rstrip("/")
|
||||||
|
clean_base = base_prefix.rstrip("/")
|
||||||
|
if clean_base and clean_key.startswith(f"{clean_base}/"):
|
||||||
|
return clean_key[len(clean_base) + 1 :]
|
||||||
|
return clean_key
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_max_keys(profile: ConnectorProfile) -> int:
|
||||||
|
configured = _int(profile.metadata.get("max_keys"))
|
||||||
|
if configured is None:
|
||||||
|
return 1000
|
||||||
|
return max(1, min(configured, 1000))
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_verify(profile: ConnectorProfile) -> bool | str | None:
|
||||||
|
try:
|
||||||
|
validate_connector_tls_metadata(profile.metadata)
|
||||||
|
except ConnectorDeploymentConfigurationError as exc:
|
||||||
|
raise ConnectorBrowseError(str(exc)) from exc
|
||||||
|
ca_bundle = _metadata_string(profile, "ca_bundle")
|
||||||
|
if ca_bundle:
|
||||||
|
try:
|
||||||
|
return connector_ca_bundle_path(ca_bundle)
|
||||||
|
except ConnectorDeploymentConfigurationError as exc:
|
||||||
|
raise ConnectorBrowseError(str(exc)) from exc
|
||||||
|
if "verify_tls" in profile.metadata:
|
||||||
|
return _metadata_bool(profile, "verify_tls", default=True)
|
||||||
|
if "tls_verify" in profile.metadata:
|
||||||
|
return _metadata_bool(profile, "tls_verify", default=True)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_addressing_style(profile: ConnectorProfile) -> str | None:
|
||||||
|
style = _metadata_string(profile, "addressing_style")
|
||||||
|
if style in {"path", "virtual", "auto"}:
|
||||||
|
return style
|
||||||
|
if _metadata_bool(profile, "path_style", default=False):
|
||||||
|
return "path"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]:
|
||||||
|
token = _seafile_token(profile)
|
||||||
|
return {"Authorization": f"Token {token}", "Accept": "application/json"}
|
||||||
|
|
||||||
|
|
||||||
|
def _seafile_token(profile: ConnectorProfile) -> str:
|
||||||
|
token = _profile_token(profile)
|
||||||
|
if token:
|
||||||
|
return token
|
||||||
|
password = _profile_password(profile)
|
||||||
|
if profile.username and password:
|
||||||
|
payload = _request_json(
|
||||||
|
"POST",
|
||||||
|
_seafile_url(profile, "api2/auth-token/"),
|
||||||
|
data={"username": profile.username, "password": password},
|
||||||
|
)
|
||||||
|
if not isinstance(payload, Mapping) or not _clean(payload.get("token")):
|
||||||
|
raise ConnectorBrowseError("Seafile did not return an account token")
|
||||||
|
return _clean(payload.get("token")) or ""
|
||||||
|
if profile.secret_ref:
|
||||||
|
raise ConnectorBrowseError("Secret-ref Seafile credentials need a runtime secret resolver before live browsing")
|
||||||
|
raise ConnectorBrowseError("Seafile connector profiles require token credentials or username plus password credentials")
|
||||||
|
|
||||||
|
|
||||||
|
def _seafile_url(profile: ConnectorProfile, suffix: str) -> str:
|
||||||
|
if not profile.endpoint_url:
|
||||||
|
raise ConnectorBrowseError("Seafile connector profile does not define endpoint_url")
|
||||||
|
return urljoin(profile.endpoint_url.rstrip("/") + "/", suffix.lstrip("/"))
|
||||||
|
|
||||||
|
|
||||||
|
def _request_json(
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
headers: Mapping[str, str] | None = None,
|
||||||
|
params: Mapping[str, str] | None = None,
|
||||||
|
data: Mapping[str, str] | None = None,
|
||||||
|
) -> object:
|
||||||
|
try:
|
||||||
|
response = request_connector_bytes(
|
||||||
|
method,
|
||||||
|
url,
|
||||||
|
headers=dict(headers or {}),
|
||||||
|
params=params,
|
||||||
|
data=data,
|
||||||
|
timeout=15.0,
|
||||||
|
label="Seafile API",
|
||||||
|
)
|
||||||
|
except ConnectorHttpError as exc:
|
||||||
|
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||||
|
if response.status_code in {401, 403}:
|
||||||
|
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||||
|
if response.status_code not in {200, 201}:
|
||||||
|
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||||
|
try:
|
||||||
|
return json.loads(response.content)
|
||||||
|
except (UnicodeDecodeError, ValueError) as exc:
|
||||||
|
raise ConnectorBrowseError("Connector returned invalid JSON") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _seafile_repo_and_path(*, path: str, library_id: str | None) -> tuple[str | None, str]:
|
||||||
|
repo_id = _clean(library_id)
|
||||||
|
if repo_id:
|
||||||
|
return repo_id, path
|
||||||
|
if not path:
|
||||||
|
return None, ""
|
||||||
|
first, _, rest = path.partition("/")
|
||||||
|
return first, rest
|
||||||
|
|
||||||
|
|
||||||
|
def _seafile_library_item(value: Mapping[str, Any]) -> ConnectorBrowseItem:
|
||||||
|
repo_id = _clean(value.get("id") or value.get("repo_id"))
|
||||||
|
name = _clean(value.get("name") or value.get("repo_name") or repo_id)
|
||||||
|
if not repo_id or not name:
|
||||||
|
raise ConnectorBrowseError("Seafile library entries require id and name")
|
||||||
|
return ConnectorBrowseItem(
|
||||||
|
kind="library",
|
||||||
|
name=name,
|
||||||
|
path=repo_id,
|
||||||
|
external_id=repo_id,
|
||||||
|
size_bytes=_int(value.get("size") or value.get("repo_size")),
|
||||||
|
modified_at=_timestamp(value.get("mtime")),
|
||||||
|
metadata={
|
||||||
|
key: value[key]
|
||||||
|
for key in ("type", "permission", "encrypted", "owner", "file_count")
|
||||||
|
if key in value
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _seafile_directory_item(value: Mapping[str, Any], *, parent_path: str) -> ConnectorBrowseItem:
|
||||||
|
name = _clean(value.get("name") or value.get("obj_name"))
|
||||||
|
if not name:
|
||||||
|
raise ConnectorBrowseError("Seafile directory entries require name")
|
||||||
|
item_type = str(value.get("type") or ("dir" if value.get("is_dir") else "file")).casefold()
|
||||||
|
kind = "folder" if item_type in {"dir", "folder"} else "file"
|
||||||
|
path = _join_browse_path(parent_path, name)
|
||||||
|
content_type = None if kind == "folder" else mimetypes.guess_type(name)[0]
|
||||||
|
return ConnectorBrowseItem(
|
||||||
|
kind=kind,
|
||||||
|
name=name,
|
||||||
|
path=path,
|
||||||
|
external_id=_clean(value.get("id") or value.get("obj_id")),
|
||||||
|
size_bytes=None if kind == "folder" else _int(value.get("size")),
|
||||||
|
content_type=content_type,
|
||||||
|
modified_at=_timestamp(value.get("mtime") or value.get("modified")),
|
||||||
|
etag=_clean(value.get("id") or value.get("obj_id")),
|
||||||
|
metadata={
|
||||||
|
key: value[key]
|
||||||
|
for key in ("permission", "modifier_email", "modifier_name")
|
||||||
|
if key in value
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _static_listing(profile: ConnectorProfile, *, path: str, library_id: str | None) -> list[ConnectorBrowseItem] | None:
|
||||||
|
listing = profile.metadata.get("static_listing")
|
||||||
|
if listing is None:
|
||||||
|
return None
|
||||||
|
if isinstance(listing, list):
|
||||||
|
raw_items = listing if path == "" else []
|
||||||
|
elif isinstance(listing, Mapping):
|
||||||
|
key = library_id or path
|
||||||
|
raw_items = listing.get(key)
|
||||||
|
if raw_items is None and key == "":
|
||||||
|
raw_items = listing.get("/")
|
||||||
|
else:
|
||||||
|
raise ConnectorBrowseError("Connector static_listing metadata must be a list or object")
|
||||||
|
if raw_items is None:
|
||||||
|
return []
|
||||||
|
if not isinstance(raw_items, list):
|
||||||
|
raise ConnectorBrowseError("Connector static_listing entries must be lists")
|
||||||
|
return sorted(
|
||||||
|
[_static_item(item, parent_path=path) for item in raw_items if isinstance(item, Mapping)],
|
||||||
|
key=lambda item: (item.kind != "library", item.kind != "folder", item.name.casefold(), item.path.casefold()),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _static_item(value: Mapping[str, Any], *, parent_path: str) -> ConnectorBrowseItem:
|
||||||
|
kind = str(value.get("kind") or "file").strip().casefold()
|
||||||
|
if kind not in {"library", "folder", "file"}:
|
||||||
|
raise ConnectorBrowseError("Connector browse items must be library, folder or file")
|
||||||
|
name = str(value.get("name") or value.get("path") or "").strip()
|
||||||
|
if not name:
|
||||||
|
raise ConnectorBrowseError("Connector browse items require name")
|
||||||
|
path = normalize_connector_browse_path(value.get("path"))
|
||||||
|
if not path:
|
||||||
|
path = _join_browse_path(parent_path, name)
|
||||||
|
return ConnectorBrowseItem(
|
||||||
|
kind=kind,
|
||||||
|
name=name,
|
||||||
|
path=path,
|
||||||
|
external_id=_clean(value.get("external_id")),
|
||||||
|
external_url=_clean(value.get("external_url")),
|
||||||
|
size_bytes=_int(value.get("size_bytes")),
|
||||||
|
content_type=_clean(value.get("content_type")),
|
||||||
|
modified_at=_clean(value.get("modified_at")),
|
||||||
|
etag=_clean(value.get("etag")),
|
||||||
|
metadata=value.get("metadata") if isinstance(value.get("metadata"), Mapping) else {},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _webdav_url(root_url: str, path: str) -> str:
|
||||||
|
base = root_url if root_url.endswith("/") else root_url + "/"
|
||||||
|
if not path:
|
||||||
|
return base
|
||||||
|
quoted = "/".join(quote(part, safe="") for part in path.split("/") if part)
|
||||||
|
return urljoin(base, quoted + "/")
|
||||||
|
|
||||||
|
|
||||||
|
def _url_path_root(root_url: str) -> str:
|
||||||
|
path = unquote(urlsplit(root_url).path or "/")
|
||||||
|
return path if path.endswith("/") else path + "/"
|
||||||
|
|
||||||
|
|
||||||
|
def _relative_href_path(root_path: str, href: str) -> str:
|
||||||
|
href_path = unquote(urlsplit(href).path or href).strip()
|
||||||
|
if href_path.startswith(root_path):
|
||||||
|
return normalize_connector_browse_path(href_path[len(root_path):])
|
||||||
|
return normalize_connector_browse_path(href_path.rsplit("/", 1)[-1])
|
||||||
|
|
||||||
|
|
||||||
|
def _webdav_prop(response: ET.Element) -> ET.Element:
|
||||||
|
prop = response.find("{DAV:}propstat/{DAV:}prop")
|
||||||
|
if prop is not None:
|
||||||
|
return prop
|
||||||
|
prop = response.find(".//{DAV:}prop")
|
||||||
|
return prop if prop is not None else ET.Element("prop")
|
||||||
|
|
||||||
|
|
||||||
|
def _webdav_display_name(prop: ET.Element | None) -> str | None:
|
||||||
|
return _clean(_text(prop, "{DAV:}displayname"))
|
||||||
|
|
||||||
|
|
||||||
|
def _text(prop: ET.Element | None, tag: str) -> str | None:
|
||||||
|
if prop is None:
|
||||||
|
return None
|
||||||
|
child = prop.find(tag)
|
||||||
|
return child.text.strip() if child is not None and child.text else None
|
||||||
|
|
||||||
|
|
||||||
|
def _http_date(value: str | None) -> str | None:
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return parsedate_to_datetime(value).isoformat()
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _timestamp(value: object) -> str | None:
|
||||||
|
number = _int(value)
|
||||||
|
if number is None:
|
||||||
|
return _clean(value)
|
||||||
|
return datetime.fromtimestamp(number, tz=timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata_string(profile: ConnectorProfile, key: str) -> str | None:
|
||||||
|
return _clean(profile.metadata.get(key))
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata_bool(profile: ConnectorProfile, key: str, *, default: bool = False) -> bool:
|
||||||
|
value = profile.metadata.get(key)
|
||||||
|
if value is None:
|
||||||
|
return default
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return value
|
||||||
|
return str(value).strip().casefold() in {"1", "true", "yes", "on"}
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata_env(profile: ConnectorProfile, key: str) -> str | None:
|
||||||
|
env_name = _metadata_string(profile, key)
|
||||||
|
if not env_name:
|
||||||
|
return None
|
||||||
|
return _env_required(env_name, profile)
|
||||||
|
|
||||||
|
|
||||||
|
def _env_required(name: str, profile: ConnectorProfile) -> str:
|
||||||
|
try:
|
||||||
|
return connector_secret_env_value(name, source_kind=profile.source_kind)
|
||||||
|
except ConnectorDeploymentConfigurationError as exc:
|
||||||
|
raise ConnectorBrowseError(f"Connector profile {profile.id}: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_connector_browse_path(value: object) -> str:
|
||||||
|
if value is None:
|
||||||
|
return ""
|
||||||
|
path = str(value).replace("\\", "/").strip().strip("/")
|
||||||
|
parts = [part for part in path.split("/") if part and part not in {"."}]
|
||||||
|
if any(part == ".." for part in parts):
|
||||||
|
raise ConnectorBrowseError("Connector browse paths cannot contain parent directory segments")
|
||||||
|
return "/".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _join_browse_path(parent: str, name: str) -> str:
|
||||||
|
child = normalize_connector_browse_path(name)
|
||||||
|
if not parent:
|
||||||
|
return child
|
||||||
|
return f"{parent.rstrip('/')}/{child}"
|
||||||
|
|
||||||
|
|
||||||
|
def _path_name(path: str) -> str:
|
||||||
|
return path.rstrip("/").rsplit("/", 1)[-1]
|
||||||
|
|
||||||
|
|
||||||
|
def _int(value: object) -> int | None:
|
||||||
|
if value is None or value == "":
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class _SmbLocation:
|
||||||
|
server: str
|
||||||
|
share: str
|
||||||
|
port: int
|
||||||
|
root_path: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
|
||||||
|
if not profile.endpoint_url:
|
||||||
|
raise ConnectorBrowseError("SMB connector profile does not define endpoint_url")
|
||||||
|
parsed = urlsplit(profile.endpoint_url)
|
||||||
|
if parsed.scheme.casefold() != "smb":
|
||||||
|
raise ConnectorBrowseError("SMB connector endpoint_url must use smb://")
|
||||||
|
server = _clean(parsed.hostname)
|
||||||
|
if not server:
|
||||||
|
raise ConnectorBrowseError("SMB connector endpoint_url must include a server")
|
||||||
|
port = parsed.port or _int(profile.metadata.get("port")) or 445
|
||||||
|
try:
|
||||||
|
validate_outbound_host(server, port=port, label="SMB connector endpoint")
|
||||||
|
except OutboundHttpError as exc:
|
||||||
|
raise ConnectorBrowseError(str(exc)) from exc
|
||||||
|
path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part]
|
||||||
|
if not path_parts:
|
||||||
|
raise ConnectorBrowseError("SMB connector endpoint_url must include a share name")
|
||||||
|
root_parts = path_parts[1:]
|
||||||
|
if profile.base_path:
|
||||||
|
root_parts.extend(normalize_connector_browse_path(profile.base_path).split("/"))
|
||||||
|
return _SmbLocation(
|
||||||
|
server=server,
|
||||||
|
share=path_parts[0],
|
||||||
|
port=port,
|
||||||
|
root_path=normalize_connector_browse_path("/".join(root_parts)),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_unc_path(location: _SmbLocation, path: str) -> str:
|
||||||
|
parts = [part for part in (location.root_path, normalize_connector_browse_path(path)) if part]
|
||||||
|
suffix = "\\".join(part.replace("/", "\\") for part in parts)
|
||||||
|
base = f"\\\\{location.server}\\{location.share}"
|
||||||
|
return f"{base}\\{suffix}" if suffix else base
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_client_kwargs(profile: ConnectorProfile, location: _SmbLocation) -> dict[str, object]:
|
||||||
|
kwargs: dict[str, object] = {
|
||||||
|
"port": location.port,
|
||||||
|
"connection_cache": pinned_smb_connection_cache(),
|
||||||
|
"require_signing": _metadata_bool(profile, "require_signing", default=True),
|
||||||
|
"auth_protocol": _metadata_string(profile, "auth_protocol") or "ntlm",
|
||||||
|
}
|
||||||
|
if "encrypt" in profile.metadata:
|
||||||
|
kwargs["encrypt"] = _metadata_bool(profile, "encrypt", default=False)
|
||||||
|
password = _profile_password(profile)
|
||||||
|
token = _profile_token(profile)
|
||||||
|
if profile.username and password:
|
||||||
|
kwargs["username"] = profile.username
|
||||||
|
kwargs["password"] = password
|
||||||
|
elif token:
|
||||||
|
raise ConnectorBrowseError("SMB connector profiles do not support bearer-token credentials")
|
||||||
|
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||||
|
raise ConnectorBrowseError("Secret-ref SMB credentials need a runtime secret resolver before live browsing")
|
||||||
|
return kwargs
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_password(profile: ConnectorProfile) -> str | None:
|
||||||
|
if profile.password_value:
|
||||||
|
return profile.password_value
|
||||||
|
if profile.password_env:
|
||||||
|
return _env_required(profile.password_env, profile)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_token(profile: ConnectorProfile) -> str | None:
|
||||||
|
if profile.token_value:
|
||||||
|
return profile.token_value
|
||||||
|
if profile.token_env:
|
||||||
|
return _env_required(profile.token_env, profile)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _smbclient_module() -> Any:
|
||||||
|
try:
|
||||||
|
return install_pinned_smb_transport(import_module("smbclient"))
|
||||||
|
except ImportError as exc:
|
||||||
|
raise ConnectorBrowseUnsupported("SMB connector browsing requires the optional smbprotocol dependency") from exc
|
||||||
|
except SdkPeerPinningError as exc:
|
||||||
|
raise ConnectorBrowseUnsupported(str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_entry_stat(entry: object) -> object | None:
|
||||||
|
try:
|
||||||
|
return entry.stat() # type: ignore[attr-defined]
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_stat_size(stat_result: object | None) -> int | None:
|
||||||
|
return _int(getattr(stat_result, "st_size", None))
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_stat_modified_at(stat_result: object | None) -> str | None:
|
||||||
|
value = getattr(stat_result, "st_mtime", None)
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return datetime.fromtimestamp(float(value), tz=timezone.utc).isoformat()
|
||||||
|
except (TypeError, ValueError, OSError, OverflowError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _smb_stat_revision(stat_result: object | None) -> str | None:
|
||||||
|
mtime_ns = getattr(stat_result, "st_mtime_ns", None)
|
||||||
|
size = getattr(stat_result, "st_size", None)
|
||||||
|
if mtime_ns is not None:
|
||||||
|
return f"{mtime_ns}:{size or 0}"
|
||||||
|
modified = getattr(stat_result, "st_mtime", None)
|
||||||
|
if modified is not None:
|
||||||
|
return f"{modified}:{size or 0}"
|
||||||
|
return None
|
||||||
@@ -0,0 +1,337 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from sqlalchemy import inspect
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.audit.logging import audit_event
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorCredentialDeletionResult:
|
||||||
|
changed: bool
|
||||||
|
affected_profiles: tuple[FileConnectorProfile, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
def delete_connector_credential_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorCredential,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
user_id: str | None = None,
|
||||||
|
api_key_id: str | None = None,
|
||||||
|
) -> ConnectorCredentialDeletionResult:
|
||||||
|
"""Scrub a credential tombstone and disable every profile that used it.
|
||||||
|
|
||||||
|
``secret_ref`` predates a Files-owned secret-provider contract. It may be
|
||||||
|
shared or deployment-owned, so it is detached locally and explicitly
|
||||||
|
audited without ever being passed to a provider delete operation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
dependent_profiles = (
|
||||||
|
tuple(
|
||||||
|
session.query(FileConnectorProfile)
|
||||||
|
.filter(FileConnectorProfile.credential_profile_id == row.id)
|
||||||
|
.order_by(FileConnectorProfile.id.asc())
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
if inspect(session.get_bind()).has_table(FileConnectorProfile.__tablename__)
|
||||||
|
else ()
|
||||||
|
)
|
||||||
|
affected_profiles: list[FileConnectorProfile] = []
|
||||||
|
for profile in dependent_profiles:
|
||||||
|
if _delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
profile,
|
||||||
|
deletion_reason="credential_deleted",
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
):
|
||||||
|
affected_profiles.append(profile)
|
||||||
|
|
||||||
|
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||||
|
removed_reference_kinds = _credential_reference_kinds(row)
|
||||||
|
removed_metadata = bool(row.metadata_)
|
||||||
|
changed = _credential_row_requires_deletion(row)
|
||||||
|
if changed:
|
||||||
|
_scrub_credential_row(row, user_id=user_id)
|
||||||
|
session.add(row)
|
||||||
|
_audit_credential_deletion(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason=deletion_reason,
|
||||||
|
deleted_secret_kinds=deleted_secret_kinds,
|
||||||
|
removed_reference_kinds=removed_reference_kinds,
|
||||||
|
removed_metadata=removed_metadata,
|
||||||
|
affected_profile_count=len(affected_profiles),
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
return ConnectorCredentialDeletionResult(
|
||||||
|
changed=changed or bool(affected_profiles),
|
||||||
|
affected_profiles=tuple(affected_profiles),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def delete_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
user_id: str | None = None,
|
||||||
|
api_key_id: str | None = None,
|
||||||
|
) -> bool:
|
||||||
|
changed = _delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason=deletion_reason,
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
)
|
||||||
|
session.flush()
|
||||||
|
return changed
|
||||||
|
|
||||||
|
|
||||||
|
def delete_connector_credentials_for_retirement(session: Session) -> int:
|
||||||
|
"""Scrub and audit stored connector material before destructive retirement.
|
||||||
|
|
||||||
|
Legacy external references are detached and audited as non-owned. Files
|
||||||
|
never sends those arbitrary references to a provider delete operation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
inspector = inspect(session.get_bind())
|
||||||
|
profiles = (
|
||||||
|
session.query(FileConnectorProfile).order_by(FileConnectorProfile.id.asc()).all()
|
||||||
|
if inspector.has_table(FileConnectorProfile.__tablename__)
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
credentials = (
|
||||||
|
session.query(FileConnectorCredential).order_by(FileConnectorCredential.id.asc()).all()
|
||||||
|
if inspector.has_table(FileConnectorCredential.__tablename__)
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
deleted = 0
|
||||||
|
for profile in profiles:
|
||||||
|
if not _profile_row_has_credential_material(profile):
|
||||||
|
continue
|
||||||
|
if _delete_connector_profile_row(
|
||||||
|
session,
|
||||||
|
profile,
|
||||||
|
deletion_reason="module_data_retired",
|
||||||
|
):
|
||||||
|
deleted += 1
|
||||||
|
for credential in credentials:
|
||||||
|
if not _credential_row_has_material(credential):
|
||||||
|
continue
|
||||||
|
result = delete_connector_credential_row(
|
||||||
|
session,
|
||||||
|
credential,
|
||||||
|
deletion_reason="module_data_retired",
|
||||||
|
)
|
||||||
|
if result.changed:
|
||||||
|
deleted += 1
|
||||||
|
session.flush()
|
||||||
|
return deleted
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
user_id: str | None = None,
|
||||||
|
api_key_id: str | None = None,
|
||||||
|
) -> bool:
|
||||||
|
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||||
|
removed_reference_kinds = _profile_reference_kinds(row)
|
||||||
|
removed_metadata = bool(row.metadata_)
|
||||||
|
changed = _profile_row_requires_deletion(row)
|
||||||
|
if not changed:
|
||||||
|
return False
|
||||||
|
_scrub_profile_row(row, user_id=user_id)
|
||||||
|
session.add(row)
|
||||||
|
_audit_profile_deletion(
|
||||||
|
session,
|
||||||
|
row,
|
||||||
|
deletion_reason=deletion_reason,
|
||||||
|
deleted_secret_kinds=deleted_secret_kinds,
|
||||||
|
removed_reference_kinds=removed_reference_kinds,
|
||||||
|
removed_metadata=removed_metadata,
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _scrub_credential_row(row: FileConnectorCredential, *, user_id: str | None) -> None:
|
||||||
|
row.enabled = False
|
||||||
|
row.credential_mode = "none"
|
||||||
|
row.username = None
|
||||||
|
row.password_encrypted = None
|
||||||
|
row.token_encrypted = None
|
||||||
|
row.password_env = None
|
||||||
|
row.token_env = None
|
||||||
|
row.secret_ref = None
|
||||||
|
row.metadata_ = {}
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
|
||||||
|
|
||||||
|
def _scrub_profile_row(row: FileConnectorProfile, *, user_id: str | None) -> None:
|
||||||
|
row.enabled = False
|
||||||
|
row.credential_profile_id = None
|
||||||
|
row.credential_mode = "none"
|
||||||
|
row.username = None
|
||||||
|
row.password_encrypted = None
|
||||||
|
row.token_encrypted = None
|
||||||
|
row.password_env = None
|
||||||
|
row.token_env = None
|
||||||
|
row.secret_ref = None
|
||||||
|
row.metadata_ = {}
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_row_requires_deletion(row: FileConnectorCredential) -> bool:
|
||||||
|
return bool(row.enabled or _credential_row_has_material(row))
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_row_requires_deletion(row: FileConnectorProfile) -> bool:
|
||||||
|
return bool(row.enabled or _profile_row_has_credential_material(row))
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_row_has_material(row: FileConnectorCredential) -> bool:
|
||||||
|
return bool(
|
||||||
|
row.username
|
||||||
|
or row.password_encrypted
|
||||||
|
or row.token_encrypted
|
||||||
|
or row.password_env
|
||||||
|
or row.token_env
|
||||||
|
or row.secret_ref
|
||||||
|
or row.metadata_
|
||||||
|
or row.credential_mode not in {"", "none", "anonymous"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_row_has_credential_material(row: FileConnectorProfile) -> bool:
|
||||||
|
return bool(
|
||||||
|
row.credential_profile_id
|
||||||
|
or row.username
|
||||||
|
or row.password_encrypted
|
||||||
|
or row.token_encrypted
|
||||||
|
or row.password_env
|
||||||
|
or row.token_env
|
||||||
|
or row.secret_ref
|
||||||
|
or row.metadata_
|
||||||
|
or row.credential_mode not in {"", "none", "anonymous"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _encrypted_secret_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||||
|
kinds: list[str] = []
|
||||||
|
if row.password_encrypted:
|
||||||
|
kinds.append("password")
|
||||||
|
if row.token_encrypted:
|
||||||
|
kinds.append("token")
|
||||||
|
return kinds
|
||||||
|
|
||||||
|
|
||||||
|
def _credential_reference_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||||
|
kinds: list[str] = []
|
||||||
|
if row.password_env:
|
||||||
|
kinds.append("password_env")
|
||||||
|
if row.token_env:
|
||||||
|
kinds.append("token_env")
|
||||||
|
if row.secret_ref:
|
||||||
|
kinds.append("unowned_external_secret_ref")
|
||||||
|
return kinds
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_reference_kinds(row: FileConnectorProfile) -> list[str]:
|
||||||
|
kinds = _credential_reference_kinds(row)
|
||||||
|
if row.credential_profile_id:
|
||||||
|
kinds.append("credential_profile")
|
||||||
|
return kinds
|
||||||
|
|
||||||
|
|
||||||
|
def _storage_backend(deleted_secret_kinds: list[str], removed_reference_kinds: list[str]) -> str:
|
||||||
|
if "unowned_external_secret_ref" in removed_reference_kinds:
|
||||||
|
return "unowned_external_reference_detached"
|
||||||
|
if deleted_secret_kinds:
|
||||||
|
return "encrypted_database"
|
||||||
|
if removed_reference_kinds:
|
||||||
|
return "reference_only"
|
||||||
|
return "none"
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_credential_deletion(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorCredential,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
deleted_secret_kinds: list[str],
|
||||||
|
removed_reference_kinds: list[str],
|
||||||
|
removed_metadata: bool,
|
||||||
|
affected_profile_count: int,
|
||||||
|
user_id: str | None,
|
||||||
|
api_key_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
audit_event(
|
||||||
|
session,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
scope=_audit_scope(row.scope_type),
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
action="files.connector_credential_deleted",
|
||||||
|
object_type="file_connector_credential",
|
||||||
|
object_id=row.id,
|
||||||
|
details={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||||
|
"deleted_secret_kinds": deleted_secret_kinds,
|
||||||
|
"removed_reference_kinds": removed_reference_kinds,
|
||||||
|
"removed_metadata": removed_metadata,
|
||||||
|
"affected_profile_count": affected_profile_count,
|
||||||
|
"deletion_reason": deletion_reason,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_profile_deletion(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
deletion_reason: str,
|
||||||
|
deleted_secret_kinds: list[str],
|
||||||
|
removed_reference_kinds: list[str],
|
||||||
|
removed_metadata: bool,
|
||||||
|
user_id: str | None,
|
||||||
|
api_key_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
audit_event(
|
||||||
|
session,
|
||||||
|
tenant_id=row.tenant_id,
|
||||||
|
scope=_audit_scope(row.scope_type),
|
||||||
|
user_id=user_id,
|
||||||
|
api_key_id=api_key_id,
|
||||||
|
action="files.connector_profile_deleted",
|
||||||
|
object_type="file_connector_profile",
|
||||||
|
object_id=row.id,
|
||||||
|
details={
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"provider": row.provider,
|
||||||
|
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||||
|
"deleted_secret_kinds": deleted_secret_kinds,
|
||||||
|
"removed_reference_kinds": removed_reference_kinds,
|
||||||
|
"removed_metadata": removed_metadata,
|
||||||
|
"deletion_reason": deletion_reason,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_scope(scope_type: str) -> str:
|
||||||
|
return "system" if scope_type == "system" else "tenant"
|
||||||
@@ -0,0 +1,540 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
from sqlalchemy import inspect
|
||||||
|
|
||||||
|
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||||
|
from govoplan_core.security.credential_envelopes import (
|
||||||
|
CredentialAccessContext,
|
||||||
|
CredentialEnvelope,
|
||||||
|
CredentialEnvelopeError,
|
||||||
|
ResolvedCredentialEnvelope,
|
||||||
|
list_credential_envelopes,
|
||||||
|
resolve_credential_envelope,
|
||||||
|
)
|
||||||
|
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorCredential
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||||
|
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
|
||||||
|
|
||||||
|
CORE_CREDENTIAL_ENVELOPE_PREFIX = "credential-envelope:"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorCredential:
|
||||||
|
id: str
|
||||||
|
label: str
|
||||||
|
scope_type: str
|
||||||
|
scope_id: str | None = None
|
||||||
|
provider: str | None = None
|
||||||
|
enabled: bool = True
|
||||||
|
credential_mode: str = "none"
|
||||||
|
username: str | None = None
|
||||||
|
password_env: str | None = None
|
||||||
|
token_env: str | None = None
|
||||||
|
secret_ref: str | None = None
|
||||||
|
password_value: str | None = None
|
||||||
|
token_value: str | None = None
|
||||||
|
policy_sources: tuple[ConnectorPolicySource, ...] = ()
|
||||||
|
metadata: Mapping[str, Any] | None = None
|
||||||
|
source_kind: str = "database"
|
||||||
|
has_secret: bool = False
|
||||||
|
|
||||||
|
@property
|
||||||
|
def source_path(self) -> str:
|
||||||
|
return policy_source_path(self.scope_type, self.scope_id)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def credential_secret_source(self) -> str | None:
|
||||||
|
if self.secret_ref:
|
||||||
|
return "secret_ref"
|
||||||
|
if self.password_value or self.token_value:
|
||||||
|
return "stored"
|
||||||
|
if self.token_env:
|
||||||
|
return "token_env"
|
||||||
|
if self.password_env:
|
||||||
|
return "password_env"
|
||||||
|
return None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def credentials_configured(self) -> bool:
|
||||||
|
if not self.enabled:
|
||||||
|
return False
|
||||||
|
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
|
||||||
|
return True
|
||||||
|
# Environment references are deliberately unavailable to API-managed
|
||||||
|
# credential rows. Legacy rows remain visible but fail closed.
|
||||||
|
return bool(self.has_secret or self.secret_ref or self.password_value or self.token_value)
|
||||||
|
|
||||||
|
def to_response(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": self.id,
|
||||||
|
"label": self.label,
|
||||||
|
"provider": self.provider,
|
||||||
|
"enabled": self.enabled,
|
||||||
|
"scope_type": self.scope_type,
|
||||||
|
"scope_id": self.scope_id,
|
||||||
|
"source_path": self.source_path,
|
||||||
|
"credential_mode": self.credential_mode,
|
||||||
|
"credential_secret_source": self.credential_secret_source,
|
||||||
|
"credentials_configured": self.credentials_configured,
|
||||||
|
"username": self.username,
|
||||||
|
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||||
|
"metadata": dict(self.metadata or {}),
|
||||||
|
"source_kind": self.source_kind,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def list_database_connector_credentials(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> list[ConnectorCredential]:
|
||||||
|
local = [
|
||||||
|
connector_credential_from_row(row)
|
||||||
|
for row in list_connector_credential_rows(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
)
|
||||||
|
]
|
||||||
|
return [
|
||||||
|
*local,
|
||||||
|
*list_reusable_connector_credentials(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def reusable_credential_reference(credential_id: str) -> str:
|
||||||
|
return f"{CORE_CREDENTIAL_ENVELOPE_PREFIX}{credential_id}"
|
||||||
|
|
||||||
|
|
||||||
|
def reusable_credential_id(credential_ref: str | None) -> str | None:
|
||||||
|
if not credential_ref or not credential_ref.startswith(CORE_CREDENTIAL_ENVELOPE_PREFIX):
|
||||||
|
return None
|
||||||
|
value = credential_ref.removeprefix(CORE_CREDENTIAL_ENVELOPE_PREFIX).strip()
|
||||||
|
return value or None
|
||||||
|
|
||||||
|
|
||||||
|
def file_credential_context(
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
profile_id: str | None = None,
|
||||||
|
scope_type: str = "tenant",
|
||||||
|
scope_id: str | None = None,
|
||||||
|
administrative: bool = False,
|
||||||
|
) -> CredentialAccessContext:
|
||||||
|
return CredentialAccessContext(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=scope_id if scope_type == "user" else None,
|
||||||
|
group_ids=frozenset({scope_id}) if scope_type == "group" and scope_id else frozenset(),
|
||||||
|
target_scope_type=scope_type,
|
||||||
|
target_scope_id=scope_id or tenant_id,
|
||||||
|
module_id="files",
|
||||||
|
server_ref=f"files:{profile_id}" if profile_id else None,
|
||||||
|
administrative=administrative,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_reusable_connector_credentials(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> list[ConnectorCredential]:
|
||||||
|
if not inspect(session.get_bind()).has_table(CredentialEnvelope.__tablename__):
|
||||||
|
return []
|
||||||
|
context = file_credential_context(tenant_id=tenant_id, administrative=True)
|
||||||
|
return [
|
||||||
|
connector_credential_from_envelope(row)
|
||||||
|
for row in list_credential_envelopes(
|
||||||
|
session,
|
||||||
|
context=context,
|
||||||
|
include_inactive=include_disabled,
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_reusable_connector_credential(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
credential_ref: str,
|
||||||
|
profile_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
) -> ConnectorCredential:
|
||||||
|
credential_id = reusable_credential_id(credential_ref)
|
||||||
|
if credential_id is None:
|
||||||
|
raise FileStorageError("Reusable credential reference is invalid")
|
||||||
|
try:
|
||||||
|
resolved = resolve_credential_envelope(
|
||||||
|
session,
|
||||||
|
credential_id=credential_id,
|
||||||
|
context=file_credential_context(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
profile_id=profile_id,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
except CredentialEnvelopeError as exc:
|
||||||
|
raise FileStorageError("Reusable credential is unavailable to this file connection") from exc
|
||||||
|
return connector_credential_from_resolved_envelope(
|
||||||
|
resolved,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def connector_credential_from_envelope(row: CredentialEnvelope) -> ConnectorCredential:
|
||||||
|
return ConnectorCredential(
|
||||||
|
id=reusable_credential_reference(row.id),
|
||||||
|
label=row.name,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
enabled=row.is_active,
|
||||||
|
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_keys),
|
||||||
|
username=_clean_public_value(row.public_data, "username"),
|
||||||
|
source_kind="credential_envelope",
|
||||||
|
has_secret=bool(row.secret_data_encrypted),
|
||||||
|
metadata={
|
||||||
|
"credential_envelope_id": row.id,
|
||||||
|
"credential_kind": row.credential_kind,
|
||||||
|
"allowed_modules": list(row.allowed_modules or []),
|
||||||
|
"allowed_server_refs": list(row.allowed_server_refs or []),
|
||||||
|
"inherit_to_lower_scopes": bool(row.inherit_to_lower_scopes),
|
||||||
|
"revision": row.revision,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def connector_credential_from_resolved_envelope(
|
||||||
|
row: ResolvedCredentialEnvelope,
|
||||||
|
*,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
) -> ConnectorCredential:
|
||||||
|
return ConnectorCredential(
|
||||||
|
id=reusable_credential_reference(row.id),
|
||||||
|
label=row.name,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
enabled=True,
|
||||||
|
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_data),
|
||||||
|
username=_clean_public_value(row.public_data, "username"),
|
||||||
|
password_value=_first_secret(row.secret_data, "password", "secret"),
|
||||||
|
token_value=_first_secret(
|
||||||
|
row.secret_data,
|
||||||
|
"access_token",
|
||||||
|
"bearer_token",
|
||||||
|
"token",
|
||||||
|
"api_key",
|
||||||
|
),
|
||||||
|
source_kind="credential_envelope",
|
||||||
|
has_secret=bool(row.secret_data),
|
||||||
|
metadata={
|
||||||
|
"credential_envelope_id": row.id,
|
||||||
|
"credential_kind": row.credential_kind,
|
||||||
|
"inherit_to_lower_scopes": True,
|
||||||
|
"revision": row.revision,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_connector_credential_rows(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> list[FileConnectorCredential]:
|
||||||
|
query = session.query(FileConnectorCredential).filter(
|
||||||
|
(FileConnectorCredential.scope_type == "system")
|
||||||
|
| (FileConnectorCredential.tenant_id == tenant_id)
|
||||||
|
)
|
||||||
|
if not include_disabled:
|
||||||
|
query = query.filter(FileConnectorCredential.enabled.is_(True))
|
||||||
|
return query.order_by(FileConnectorCredential.scope_type.asc(), FileConnectorCredential.label.asc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
def connector_credential_from_row(row: FileConnectorCredential) -> ConnectorCredential:
|
||||||
|
policy_sources = []
|
||||||
|
if row.policy:
|
||||||
|
policy_sources = connector_policy_sources_from_payload({
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"label": row.label,
|
||||||
|
"policy": row.policy,
|
||||||
|
})
|
||||||
|
return ConnectorCredential(
|
||||||
|
id=row.id,
|
||||||
|
label=row.label,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
provider=row.provider,
|
||||||
|
enabled=row.enabled,
|
||||||
|
credential_mode=row.credential_mode,
|
||||||
|
username=row.username,
|
||||||
|
password_env=row.password_env,
|
||||||
|
token_env=row.token_env,
|
||||||
|
secret_ref=row.secret_ref,
|
||||||
|
password_value=decrypt_secret(row.password_encrypted),
|
||||||
|
token_value=decrypt_secret(row.token_encrypted),
|
||||||
|
policy_sources=tuple(policy_sources),
|
||||||
|
metadata=dict(row.metadata_ or {}),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_connector_credential_row(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
credential_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> FileConnectorCredential:
|
||||||
|
row = session.get(FileConnectorCredential, credential_id)
|
||||||
|
if row is None or (row.scope_type != "system" and row.tenant_id != tenant_id):
|
||||||
|
raise FileStorageError("Connector credential not found")
|
||||||
|
if not include_disabled and not row.enabled:
|
||||||
|
raise FileStorageError("Connector credential not found")
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def credential_rows_by_id(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
credential_ids: set[str],
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> dict[str, FileConnectorCredential]:
|
||||||
|
if not credential_ids:
|
||||||
|
return {}
|
||||||
|
rows = session.query(FileConnectorCredential).filter(FileConnectorCredential.id.in_(credential_ids)).all()
|
||||||
|
result: dict[str, FileConnectorCredential] = {}
|
||||||
|
for row in rows:
|
||||||
|
if row.scope_type != "system" and row.tenant_id != tenant_id:
|
||||||
|
continue
|
||||||
|
if not include_disabled and not row.enabled:
|
||||||
|
continue
|
||||||
|
result[row.id] = row
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def create_connector_credential_row(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str | None,
|
||||||
|
credential_id: str,
|
||||||
|
label: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
provider: str | None = None,
|
||||||
|
enabled: bool = True,
|
||||||
|
credential_mode: str = "none",
|
||||||
|
username: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
token: str | None = None,
|
||||||
|
password_env: str | None = None,
|
||||||
|
token_env: str | None = None,
|
||||||
|
secret_ref: str | None = None,
|
||||||
|
policy: Mapping[str, Any] | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
) -> FileConnectorCredential:
|
||||||
|
reject_api_controlled_deployment_references(
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
clean_id = _normalize_id(credential_id)
|
||||||
|
if session.get(FileConnectorCredential, clean_id) is not None:
|
||||||
|
raise FileStorageError(f"Connector credential already exists: {clean_id}")
|
||||||
|
clean_scope_type, clean_scope_id, row_tenant_id = _normalize_scope(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
row = FileConnectorCredential(
|
||||||
|
id=clean_id,
|
||||||
|
tenant_id=row_tenant_id,
|
||||||
|
scope_type=clean_scope_type,
|
||||||
|
scope_id=clean_scope_id,
|
||||||
|
label=_normalize_label(label),
|
||||||
|
provider=_normalize_provider(provider),
|
||||||
|
enabled=bool(enabled),
|
||||||
|
credential_mode=_normalize_credential_mode(credential_mode),
|
||||||
|
username=_clean(username),
|
||||||
|
password_encrypted=encrypt_secret(_clean(password)),
|
||||||
|
token_encrypted=encrypt_secret(_clean(token)),
|
||||||
|
password_env=_clean(password_env),
|
||||||
|
token_env=_clean(token_env),
|
||||||
|
secret_ref=_clean(secret_ref),
|
||||||
|
policy=dict(policy or {}),
|
||||||
|
metadata_=dict(metadata or {}),
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
updated_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
session.add(row)
|
||||||
|
session.flush()
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def update_connector_credential_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorCredential,
|
||||||
|
*,
|
||||||
|
user_id: str | None,
|
||||||
|
label: str | None = None,
|
||||||
|
provider: str | None = None,
|
||||||
|
enabled: bool | None = None,
|
||||||
|
credential_mode: str | None = None,
|
||||||
|
username: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
token: str | None = None,
|
||||||
|
password_env: str | None = None,
|
||||||
|
token_env: str | None = None,
|
||||||
|
secret_ref: str | None = None,
|
||||||
|
policy: Mapping[str, Any] | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
clear_password: bool = False,
|
||||||
|
clear_token: bool = False,
|
||||||
|
) -> FileConnectorCredential:
|
||||||
|
reject_api_controlled_deployment_references(
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
|
||||||
|
if _clean(row.secret_ref):
|
||||||
|
raise FileStorageError(
|
||||||
|
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||||
|
"provider ownership and confirm provider-side deletion"
|
||||||
|
)
|
||||||
|
if label is not None:
|
||||||
|
row.label = _normalize_label(label)
|
||||||
|
if provider is not None:
|
||||||
|
row.provider = _normalize_provider(provider)
|
||||||
|
if enabled is not None:
|
||||||
|
row.enabled = bool(enabled)
|
||||||
|
if credential_mode is not None:
|
||||||
|
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||||
|
if username is not None:
|
||||||
|
row.username = _clean(username)
|
||||||
|
if password is not None:
|
||||||
|
row.password_encrypted = encrypt_secret(_clean(password))
|
||||||
|
elif clear_password:
|
||||||
|
row.password_encrypted = None
|
||||||
|
if token is not None:
|
||||||
|
row.token_encrypted = encrypt_secret(_clean(token))
|
||||||
|
elif clear_token:
|
||||||
|
row.token_encrypted = None
|
||||||
|
if password_env is not None:
|
||||||
|
row.password_env = _clean(password_env)
|
||||||
|
if token_env is not None:
|
||||||
|
row.token_env = _clean(token_env)
|
||||||
|
if secret_ref is not None:
|
||||||
|
row.secret_ref = _clean(secret_ref)
|
||||||
|
if policy is not None:
|
||||||
|
row.policy = dict(policy)
|
||||||
|
if metadata is not None:
|
||||||
|
row.metadata_ = dict(metadata)
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
session.add(row)
|
||||||
|
session.flush()
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||||
|
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||||
|
clean_scope_id = _clean(scope_id)
|
||||||
|
if clean_scope_type == "system":
|
||||||
|
return "system", None, None
|
||||||
|
if clean_scope_type == "tenant":
|
||||||
|
return "tenant", tenant_id, tenant_id
|
||||||
|
if clean_scope_type in {"user", "group", "campaign"}:
|
||||||
|
if not clean_scope_id:
|
||||||
|
raise FileStorageError(f"{clean_scope_type.capitalize()} connector credentials require scope_id")
|
||||||
|
return clean_scope_type, clean_scope_id, tenant_id
|
||||||
|
raise FileStorageError("Unsupported connector credential scope")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_id(value: str) -> str:
|
||||||
|
clean = _clean(value)
|
||||||
|
if not clean:
|
||||||
|
raise FileStorageError("Connector credential id is required")
|
||||||
|
if len(clean) > 255:
|
||||||
|
raise FileStorageError("Connector credential id is too long")
|
||||||
|
if any(char.isspace() for char in clean):
|
||||||
|
raise FileStorageError("Connector credential id cannot contain whitespace")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_label(value: str) -> str:
|
||||||
|
clean = value.strip()
|
||||||
|
if not clean:
|
||||||
|
raise FileStorageError("Connector credential label is required")
|
||||||
|
if len(clean) > 255:
|
||||||
|
raise FileStorageError("Connector credential label is too long")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_provider(value: str | None) -> str | None:
|
||||||
|
clean = _clean(value)
|
||||||
|
if clean is None:
|
||||||
|
return None
|
||||||
|
clean = clean.casefold()
|
||||||
|
if clean not in supported_connector_providers():
|
||||||
|
raise FileStorageError(f"Unsupported connector credential provider: {value}")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_credential_mode(value: str) -> str:
|
||||||
|
clean = value.strip().casefold() or "none"
|
||||||
|
if clean not in {"none", "anonymous", "basic", "token", "secret_ref"}:
|
||||||
|
raise FileStorageError(f"Unsupported connector credential mode: {value}")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"scope_type": source.scope_type,
|
||||||
|
"scope_id": source.scope_id,
|
||||||
|
"label": source.label,
|
||||||
|
"policy": dict(source.policy or {}),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _envelope_credential_mode(kind: str, secret_values: Mapping[str, Any] | list[str]) -> str:
|
||||||
|
keys = {str(key) for key in secret_values}
|
||||||
|
if kind in {"token", "oauth2", "api_key"} or keys.intersection(
|
||||||
|
{"access_token", "bearer_token", "token", "api_key"}
|
||||||
|
):
|
||||||
|
return "token"
|
||||||
|
return "basic"
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_public_value(values: Mapping[str, Any] | None, key: str) -> str | None:
|
||||||
|
return _clean((values or {}).get(key))
|
||||||
|
|
||||||
|
|
||||||
|
def _first_secret(values: Mapping[str, Any], *keys: str) -> str | None:
|
||||||
|
for key in keys:
|
||||||
|
value = _clean(values.get(key))
|
||||||
|
if value is not None:
|
||||||
|
return value
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
@@ -0,0 +1,241 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
_SECRET_ENV_ALLOWLIST = "GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST" # noqa: S105 # nosec B105 - configuration key.
|
||||||
|
_CA_BUNDLE_ALLOWLIST = "GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST"
|
||||||
|
_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||||
|
_SECRET_ENV_METADATA_KEYS = frozenset(
|
||||||
|
{
|
||||||
|
"access_key_id_env",
|
||||||
|
"secret_access_key_env",
|
||||||
|
"session_token_env",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_SECRET_VALUE_METADATA_KEYS = frozenset(
|
||||||
|
{
|
||||||
|
"password",
|
||||||
|
"token",
|
||||||
|
"access_token",
|
||||||
|
"auth_token",
|
||||||
|
"bearer_token",
|
||||||
|
"refresh_token",
|
||||||
|
"api_key",
|
||||||
|
"access_key",
|
||||||
|
"access_key_id",
|
||||||
|
"secret_key",
|
||||||
|
"secret_access_key",
|
||||||
|
"session_token",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorDeploymentConfigurationError(ValueError):
|
||||||
|
"""Raised when connector data crosses a deployment-owned trust boundary."""
|
||||||
|
|
||||||
|
|
||||||
|
def connector_secret_env_value(name: str, *, source_kind: str) -> str:
|
||||||
|
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||||
|
value = os.environ.get(clean_name)
|
||||||
|
if value is None or value == "":
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector credential environment variable is not configured")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def connector_secret_env_available(name: str | None, *, source_kind: str) -> bool:
|
||||||
|
if not name:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||||
|
except ConnectorDeploymentConfigurationError:
|
||||||
|
return False
|
||||||
|
return bool(os.environ.get(clean_name))
|
||||||
|
|
||||||
|
|
||||||
|
def validate_deployment_connector_references(
|
||||||
|
*,
|
||||||
|
source_kind: str,
|
||||||
|
password_env: str | None = None,
|
||||||
|
token_env: str | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Validate references in deployment-owned connector configuration."""
|
||||||
|
|
||||||
|
for name in (password_env, token_env):
|
||||||
|
if name:
|
||||||
|
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||||
|
for key in _SECRET_ENV_METADATA_KEYS:
|
||||||
|
name = _clean((metadata or {}).get(key))
|
||||||
|
if name:
|
||||||
|
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||||
|
validate_connector_tls_metadata(metadata)
|
||||||
|
|
||||||
|
|
||||||
|
def reject_api_controlled_deployment_references(
|
||||||
|
*,
|
||||||
|
password_env: str | None = None,
|
||||||
|
token_env: str | None = None,
|
||||||
|
secret_ref: str | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Reject process-secret selectors controlled through tenant-facing APIs."""
|
||||||
|
|
||||||
|
if _clean(password_env) or _clean(token_env):
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||||
|
)
|
||||||
|
if _clean(secret_ref):
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"External secret references cannot be managed through the Files API until Files can prove "
|
||||||
|
"provider ownership and confirm provider-side deletion"
|
||||||
|
)
|
||||||
|
for key, value in _metadata_entries(metadata or {}):
|
||||||
|
clean_key = str(key).strip().casefold()
|
||||||
|
if _clean(value) and (clean_key in _SECRET_ENV_METADATA_KEYS or clean_key.endswith("_env")):
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||||
|
)
|
||||||
|
if _clean(value) and (
|
||||||
|
clean_key in _SECRET_VALUE_METADATA_KEYS
|
||||||
|
or clean_key.endswith(("_password", "_secret", "_api_key"))
|
||||||
|
):
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"Connector credential values must use the dedicated encrypted credential fields, not metadata"
|
||||||
|
)
|
||||||
|
validate_connector_tls_metadata(metadata)
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata_entries(value: object) -> list[tuple[str, object]]:
|
||||||
|
entries: list[tuple[str, object]] = []
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
for key, item in value.items():
|
||||||
|
entries.append((str(key), item))
|
||||||
|
entries.extend(_metadata_entries(item))
|
||||||
|
elif isinstance(value, (list, tuple)):
|
||||||
|
for item in value:
|
||||||
|
entries.extend(_metadata_entries(item))
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def connector_ca_bundle_path(value: str) -> str:
|
||||||
|
clean_value = _clean(value)
|
||||||
|
if not clean_value:
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector CA bundle path is empty")
|
||||||
|
candidate = Path(clean_value)
|
||||||
|
if not candidate.is_absolute():
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector CA bundle paths must be absolute")
|
||||||
|
try:
|
||||||
|
resolved = candidate.resolve(strict=True)
|
||||||
|
except OSError as exc:
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector CA bundle path does not exist") from exc
|
||||||
|
allowed = _allowed_ca_bundle_paths()
|
||||||
|
if resolved not in allowed:
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
f"Connector CA bundle path is not listed in {_CA_BUNDLE_ALLOWLIST}"
|
||||||
|
)
|
||||||
|
if not resolved.is_file():
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector CA bundle path must be a regular file")
|
||||||
|
return str(resolved)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_connector_tls_metadata(metadata: Mapping[str, Any] | None) -> None:
|
||||||
|
values = metadata or {}
|
||||||
|
ca_bundle = _clean(values.get("ca_bundle"))
|
||||||
|
if ca_bundle:
|
||||||
|
connector_ca_bundle_path(ca_bundle)
|
||||||
|
for key in ("verify_tls", "tls_verify"):
|
||||||
|
if key in values and not _as_bool(values.get(key), default=True) and not _development_runtime():
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"Connector TLS certificate verification may only be disabled in dev/test environments"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def connector_effective_endpoint_url(
|
||||||
|
*,
|
||||||
|
provider: str | None,
|
||||||
|
endpoint_url: str | None,
|
||||||
|
metadata: Mapping[str, Any] | None,
|
||||||
|
) -> str | None:
|
||||||
|
"""Return the endpoint that connector I/O will actually use."""
|
||||||
|
|
||||||
|
clean_provider = (provider or "").strip().casefold()
|
||||||
|
values = metadata or {}
|
||||||
|
webdav_url = _clean(values.get("webdav_endpoint_url"))
|
||||||
|
browse_protocol = (_clean(values.get("browse_protocol")) or "").casefold()
|
||||||
|
if webdav_url and (clean_provider in {"seafile", "webdav", "nextcloud"} or browse_protocol == "webdav"):
|
||||||
|
return webdav_url
|
||||||
|
return _clean(endpoint_url)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_secret_env_reference(name: str, *, source_kind: str) -> str:
|
||||||
|
if source_kind.strip().casefold() != "settings":
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
"Environment-backed credentials may only be used by deployment-owned connector configuration"
|
||||||
|
)
|
||||||
|
clean_name = name.strip()
|
||||||
|
if not _ENV_NAME.fullmatch(clean_name):
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector credential environment variable name is invalid")
|
||||||
|
if clean_name not in _allowed_secret_env_names():
|
||||||
|
raise ConnectorDeploymentConfigurationError(
|
||||||
|
f"Connector credential environment variable is not listed in {_SECRET_ENV_ALLOWLIST}"
|
||||||
|
)
|
||||||
|
return clean_name
|
||||||
|
|
||||||
|
|
||||||
|
def _allowed_secret_env_names() -> frozenset[str]:
|
||||||
|
raw = os.environ.get(_SECRET_ENV_ALLOWLIST, "")
|
||||||
|
names = frozenset(item.strip() for item in raw.split(",") if item.strip())
|
||||||
|
invalid = sorted(name for name in names if not _ENV_NAME.fullmatch(name))
|
||||||
|
if invalid:
|
||||||
|
raise ConnectorDeploymentConfigurationError(f"{_SECRET_ENV_ALLOWLIST} contains an invalid variable name")
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def _allowed_ca_bundle_paths() -> frozenset[Path]:
|
||||||
|
raw = os.environ.get(_CA_BUNDLE_ALLOWLIST, "")
|
||||||
|
paths: set[Path] = set()
|
||||||
|
for item in (part.strip() for part in raw.split(",")):
|
||||||
|
if not item:
|
||||||
|
continue
|
||||||
|
path = Path(item)
|
||||||
|
if not path.is_absolute():
|
||||||
|
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} requires absolute paths")
|
||||||
|
try:
|
||||||
|
paths.add(path.resolve(strict=True))
|
||||||
|
except OSError as exc:
|
||||||
|
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} contains a missing path") from exc
|
||||||
|
return frozenset(paths)
|
||||||
|
|
||||||
|
|
||||||
|
def _development_runtime() -> bool:
|
||||||
|
app_env = os.environ.get("APP_ENV", "dev").strip().casefold()
|
||||||
|
install_profile = os.environ.get("GOVOPLAN_INSTALL_PROFILE", "").strip().casefold()
|
||||||
|
return app_env in _DEVELOPMENT_ENVIRONMENTS and (
|
||||||
|
not install_profile or install_profile in _DEVELOPMENT_ENVIRONMENTS
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _as_bool(value: object, *, default: bool) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return default
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return value
|
||||||
|
clean = str(value).strip().casefold()
|
||||||
|
if clean in {"1", "true", "yes", "on"}:
|
||||||
|
return True
|
||||||
|
if clean in {"0", "false", "no", "off"}:
|
||||||
|
return False
|
||||||
|
raise ConnectorDeploymentConfigurationError("Connector TLS verification setting must be true or false")
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
clean = str(value).strip()
|
||||||
|
return clean or None
|
||||||
@@ -0,0 +1,358 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
import mimetypes
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from govoplan_core.security.outbound_http import response_limit
|
||||||
|
|
||||||
|
from govoplan_files.backend.storage.connector_browse import (
|
||||||
|
ConnectorBrowseError,
|
||||||
|
ConnectorBrowseUnsupported,
|
||||||
|
_clean,
|
||||||
|
_metadata_string,
|
||||||
|
_profile_password,
|
||||||
|
_profile_token,
|
||||||
|
_request_json,
|
||||||
|
_smb_client_kwargs,
|
||||||
|
_smb_location,
|
||||||
|
_smb_stat_modified_at,
|
||||||
|
_smb_stat_revision,
|
||||||
|
_smb_stat_size,
|
||||||
|
_smb_unc_path,
|
||||||
|
_smbclient_module,
|
||||||
|
_s3_bucket,
|
||||||
|
_s3_client,
|
||||||
|
_s3_object_key,
|
||||||
|
_seafile_headers,
|
||||||
|
_seafile_url,
|
||||||
|
_webdav_url,
|
||||||
|
normalize_connector_browse_path,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||||
|
from govoplan_files.backend.storage.paths import filename_from_path
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorImportError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorImportUnsupported(ConnectorImportError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorDownloadedFile:
|
||||||
|
filename: str
|
||||||
|
data: bytes
|
||||||
|
content_type: str | None = None
|
||||||
|
revision: str | None = None
|
||||||
|
external_id: str | None = None
|
||||||
|
external_url: str | None = None
|
||||||
|
metadata: dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
def read_connector_file(
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
*,
|
||||||
|
library_id: str,
|
||||||
|
path: str,
|
||||||
|
max_bytes: int,
|
||||||
|
) -> ConnectorDownloadedFile:
|
||||||
|
max_bytes = min(max_bytes, response_limit("file"))
|
||||||
|
if profile.provider == "seafile":
|
||||||
|
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||||
|
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||||
|
return _read_seafile_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||||
|
if profile.provider in {"webdav", "nextcloud"}:
|
||||||
|
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||||
|
if profile.provider == "smb":
|
||||||
|
return _read_smb_file(profile, path=path, max_bytes=max_bytes)
|
||||||
|
if profile.provider == "s3":
|
||||||
|
return _read_s3_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||||
|
raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||||
|
repo_id = _clean(library_id)
|
||||||
|
if not repo_id:
|
||||||
|
raise ConnectorImportError("Seafile import requires library_id")
|
||||||
|
file_path = "/" + normalize_connector_browse_path(path)
|
||||||
|
if file_path == "/":
|
||||||
|
raise ConnectorImportError("Seafile import requires a file path")
|
||||||
|
try:
|
||||||
|
headers = _seafile_headers(profile)
|
||||||
|
detail = _request_json(
|
||||||
|
"GET",
|
||||||
|
_seafile_url(profile, f"api2/repos/{repo_id}/file/detail/"),
|
||||||
|
headers=headers,
|
||||||
|
params={"p": file_path},
|
||||||
|
)
|
||||||
|
if isinstance(detail, dict):
|
||||||
|
size = _int(detail.get("size"))
|
||||||
|
if size is not None and size > max_bytes:
|
||||||
|
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||||
|
download_url = _request_json(
|
||||||
|
"GET",
|
||||||
|
_seafile_url(profile, f"api2/repos/{repo_id}/file/"),
|
||||||
|
headers=headers,
|
||||||
|
params={"p": file_path, "reuse": "1"},
|
||||||
|
)
|
||||||
|
except ConnectorBrowseError as exc:
|
||||||
|
raise ConnectorImportError(str(exc)) from exc
|
||||||
|
if not isinstance(download_url, str) or not download_url.strip():
|
||||||
|
raise ConnectorImportError("Seafile did not return a file download URL")
|
||||||
|
try:
|
||||||
|
response = request_connector_bytes(
|
||||||
|
"GET",
|
||||||
|
download_url,
|
||||||
|
timeout=30.0,
|
||||||
|
kind="file",
|
||||||
|
max_bytes=max_bytes,
|
||||||
|
label="Seafile file download",
|
||||||
|
)
|
||||||
|
except ConnectorHttpError as exc:
|
||||||
|
raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}")
|
||||||
|
data = response.content
|
||||||
|
if len(data) > max_bytes:
|
||||||
|
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||||
|
detail = detail if isinstance(detail, dict) else {}
|
||||||
|
filename = filename_from_path(str(detail.get("name") or path))
|
||||||
|
content_type = response.headers.get("content-type") or mimetypes.guess_type(filename)[0]
|
||||||
|
external_id = f"{repo_id}:{normalize_connector_browse_path(path)}"
|
||||||
|
return ConnectorDownloadedFile(
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
content_type=content_type,
|
||||||
|
revision=_clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified")),
|
||||||
|
external_id=external_id,
|
||||||
|
external_url=download_url,
|
||||||
|
metadata={
|
||||||
|
"library_id": repo_id,
|
||||||
|
"library_path": normalize_connector_browse_path(path),
|
||||||
|
"size": len(data),
|
||||||
|
**({key: detail[key] for key in ("permission", "last_modified", "mtime", "uploader_email", "last_modifier_email") if key in detail}),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||||
|
root_url = _metadata_string(profile, "webdav_endpoint_url") or profile.endpoint_url
|
||||||
|
if not root_url:
|
||||||
|
raise ConnectorImportError("WebDAV connector profile does not define endpoint_url")
|
||||||
|
file_path = normalize_connector_browse_path(path)
|
||||||
|
if not file_path:
|
||||||
|
raise ConnectorImportError("WebDAV import requires a file path")
|
||||||
|
url = _webdav_url(root_url, file_path).rstrip("/")
|
||||||
|
headers: dict[str, str] = {}
|
||||||
|
auth: tuple[str, str] | None = None
|
||||||
|
password = _profile_password(profile)
|
||||||
|
token = _profile_token(profile)
|
||||||
|
if profile.username and password:
|
||||||
|
auth = (profile.username, password)
|
||||||
|
elif token:
|
||||||
|
headers["Authorization"] = f"Bearer {token}"
|
||||||
|
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||||
|
raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import")
|
||||||
|
try:
|
||||||
|
response = request_connector_bytes(
|
||||||
|
"GET",
|
||||||
|
url,
|
||||||
|
headers=headers,
|
||||||
|
auth=auth,
|
||||||
|
timeout=30.0,
|
||||||
|
kind="file",
|
||||||
|
max_bytes=max_bytes,
|
||||||
|
label="WebDAV file download",
|
||||||
|
)
|
||||||
|
except ConnectorHttpError as exc:
|
||||||
|
raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc
|
||||||
|
if response.status_code in {401, 403}:
|
||||||
|
raise ConnectorImportError("Connector credentials were rejected")
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise ConnectorImportError(f"WebDAV file download failed with HTTP {response.status_code}")
|
||||||
|
length = _int(response.headers.get("content-length"))
|
||||||
|
if length is not None and length > max_bytes:
|
||||||
|
raise ConnectorImportError(f"WebDAV file exceeds limit of {max_bytes} bytes")
|
||||||
|
data = response.content
|
||||||
|
if len(data) > max_bytes:
|
||||||
|
raise ConnectorImportError(f"WebDAV file exceeds limit of {max_bytes} bytes")
|
||||||
|
filename = filename_from_path(file_path)
|
||||||
|
revision = _clean(response.headers.get("etag") or response.headers.get("last-modified"))
|
||||||
|
return ConnectorDownloadedFile(
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
content_type=response.headers.get("content-type") or mimetypes.guess_type(filename)[0],
|
||||||
|
revision=revision,
|
||||||
|
external_id=file_path,
|
||||||
|
external_url=url,
|
||||||
|
metadata={"path": file_path, "size": len(data)},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||||
|
file_path = normalize_connector_browse_path(path)
|
||||||
|
if not file_path:
|
||||||
|
raise ConnectorImportError("SMB import requires a file path")
|
||||||
|
try:
|
||||||
|
location = _smb_location(profile)
|
||||||
|
unc_path = _smb_unc_path(location, file_path)
|
||||||
|
smbclient = _smbclient_module()
|
||||||
|
kwargs = _smb_client_kwargs(profile, location)
|
||||||
|
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||||
|
size = _smb_stat_size(stat_result)
|
||||||
|
if size is not None and size > max_bytes:
|
||||||
|
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||||
|
with smbclient.open_file(unc_path, mode="rb", **kwargs) as handle:
|
||||||
|
data = handle.read(max_bytes + 1)
|
||||||
|
except ConnectorBrowseUnsupported as exc:
|
||||||
|
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||||
|
except ConnectorBrowseError as exc:
|
||||||
|
raise ConnectorImportError(str(exc)) from exc
|
||||||
|
except ConnectorImportError:
|
||||||
|
raise
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorImportError(f"SMB file download failed: {exc}") from exc
|
||||||
|
if len(data) > max_bytes:
|
||||||
|
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||||
|
filename = filename_from_path(file_path)
|
||||||
|
revision = _smb_stat_revision(stat_result)
|
||||||
|
return ConnectorDownloadedFile(
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
content_type=mimetypes.guess_type(filename)[0],
|
||||||
|
revision=revision,
|
||||||
|
external_id=f"{location.share}:{file_path}",
|
||||||
|
external_url=f"smb://{location.server}:{location.port}/{location.share}/{file_path}",
|
||||||
|
metadata={
|
||||||
|
"share": location.share,
|
||||||
|
"path": file_path,
|
||||||
|
"size": len(data),
|
||||||
|
"modified_at": _smb_stat_modified_at(stat_result),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_import_client(profile: ConnectorProfile) -> Any:
|
||||||
|
try:
|
||||||
|
return _s3_client(profile)
|
||||||
|
except ConnectorBrowseUnsupported as exc:
|
||||||
|
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||||
|
except ConnectorBrowseError as exc:
|
||||||
|
raise ConnectorImportError(str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_object_detail(client: Any, *, bucket: str, key: str, max_bytes: int) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
detail = client.head_object(Bucket=bucket, Key=key)
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorImportError(f"S3 object metadata lookup failed: {exc}") from exc
|
||||||
|
if not isinstance(detail, dict):
|
||||||
|
raise ConnectorImportError("S3 connector returned invalid object metadata")
|
||||||
|
size = _int(detail.get("ContentLength"))
|
||||||
|
if size is not None and size > max_bytes:
|
||||||
|
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||||
|
return detail
|
||||||
|
|
||||||
|
|
||||||
|
def _download_s3_object(
|
||||||
|
client: Any,
|
||||||
|
*,
|
||||||
|
bucket: str,
|
||||||
|
key: str,
|
||||||
|
version_id: str | None,
|
||||||
|
max_bytes: int,
|
||||||
|
) -> tuple[Any, bytes]:
|
||||||
|
request: dict[str, object] = {"Bucket": bucket, "Key": key}
|
||||||
|
if version_id:
|
||||||
|
request["VersionId"] = version_id
|
||||||
|
try:
|
||||||
|
response = client.get_object(**request)
|
||||||
|
body = response.get("Body")
|
||||||
|
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||||
|
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||||
|
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
|
||||||
|
if len(data) > max_bytes:
|
||||||
|
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||||
|
return response, data
|
||||||
|
|
||||||
|
|
||||||
|
def _s3_download_metadata(
|
||||||
|
detail: dict[str, Any],
|
||||||
|
*,
|
||||||
|
bucket: str,
|
||||||
|
key: str,
|
||||||
|
version_id: str | None,
|
||||||
|
etag: str | None,
|
||||||
|
size: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
metadata: dict[str, Any] = {
|
||||||
|
"bucket": bucket,
|
||||||
|
"key": key,
|
||||||
|
"version_id": version_id,
|
||||||
|
"etag": etag,
|
||||||
|
"size": size,
|
||||||
|
}
|
||||||
|
for source_key, target_key in (
|
||||||
|
("ChecksumSHA256", "checksum_sha256"),
|
||||||
|
("ChecksumCRC32", "checksum_crc32"),
|
||||||
|
("StorageClass", "storage_class"),
|
||||||
|
):
|
||||||
|
if source_key in detail:
|
||||||
|
metadata[target_key] = detail[source_key]
|
||||||
|
return metadata
|
||||||
|
|
||||||
|
|
||||||
|
def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||||
|
bucket = _s3_bucket(profile, library_id)
|
||||||
|
if not bucket:
|
||||||
|
raise ConnectorImportError("S3 import requires a bucket in library_id or profile metadata")
|
||||||
|
key = _s3_object_key(profile, path)
|
||||||
|
if not key:
|
||||||
|
raise ConnectorImportError("S3 import requires an object key")
|
||||||
|
client = _s3_import_client(profile)
|
||||||
|
try:
|
||||||
|
detail = _s3_object_detail(client, bucket=bucket, key=key, max_bytes=max_bytes)
|
||||||
|
version_id = _clean(detail.get("VersionId"))
|
||||||
|
response, data = _download_s3_object(
|
||||||
|
client,
|
||||||
|
bucket=bucket,
|
||||||
|
key=key,
|
||||||
|
version_id=version_id,
|
||||||
|
max_bytes=max_bytes,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
close = getattr(client, "close", None)
|
||||||
|
if callable(close):
|
||||||
|
close()
|
||||||
|
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
|
||||||
|
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
|
||||||
|
filename = filename_from_path(key)
|
||||||
|
return ConnectorDownloadedFile(
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
content_type=content_type,
|
||||||
|
revision=version_id or etag or _clean(detail.get("LastModified")),
|
||||||
|
external_id=f"{bucket}:{key}",
|
||||||
|
external_url=f"s3://{bucket}/{key}",
|
||||||
|
metadata=_s3_download_metadata(
|
||||||
|
detail,
|
||||||
|
bucket=bucket,
|
||||||
|
key=key,
|
||||||
|
version_id=version_id,
|
||||||
|
etag=etag,
|
||||||
|
size=len(data),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _int(value: object) -> int | None:
|
||||||
|
if value is None or value == "":
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
@@ -0,0 +1,310 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterable, Mapping
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from fnmatch import fnmatchcase
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from govoplan_core.core.policy import PolicyDecision, PolicySourceStep, normalize_policy_scope_type
|
||||||
|
|
||||||
|
|
||||||
|
_ALLOW_KEYS = ("allow", "allowlist", "whitelist")
|
||||||
|
_DENY_KEYS = ("deny", "denylist", "blacklist")
|
||||||
|
|
||||||
|
_FIELD_ALIASES = {
|
||||||
|
"connectors": ("connectors", "connector_ids", "connector_id"),
|
||||||
|
"credentials": ("credentials", "credential_ids", "credential_id"),
|
||||||
|
"providers": ("providers", "provider"),
|
||||||
|
"external_ids": ("external_ids", "external_id"),
|
||||||
|
"external_paths": ("external_paths", "paths", "path_prefixes", "external_path"),
|
||||||
|
"external_urls": ("external_urls", "urls", "external_url"),
|
||||||
|
}
|
||||||
|
|
||||||
|
CONNECTOR_POLICY_FIELDS = tuple(_FIELD_ALIASES)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorPolicyDenied(RuntimeError):
|
||||||
|
def __init__(self, decision: PolicyDecision) -> None:
|
||||||
|
super().__init__(decision.reason or "Connector policy denied")
|
||||||
|
self.decision = decision
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorAccessRequest:
|
||||||
|
connector_id: str | None = None
|
||||||
|
credential_id: str | None = None
|
||||||
|
provider: str | None = None
|
||||||
|
external_id: str | None = None
|
||||||
|
external_path: str | None = None
|
||||||
|
external_url: str | None = None
|
||||||
|
operation: str = "access"
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_provenance(cls, value: Mapping[str, Any] | None, *, operation: str = "access") -> "ConnectorAccessRequest":
|
||||||
|
payload = value or {}
|
||||||
|
return cls(
|
||||||
|
connector_id=_clean(payload.get("connector_id")),
|
||||||
|
credential_id=_clean(payload.get("credential_id") or payload.get("connector_credential_id")),
|
||||||
|
provider=_clean(payload.get("provider") or payload.get("source_type")),
|
||||||
|
external_id=_clean(payload.get("external_id")),
|
||||||
|
external_path=_clean(payload.get("external_path")),
|
||||||
|
external_url=_clean(payload.get("external_url")),
|
||||||
|
operation=_clean(operation) or "access",
|
||||||
|
)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, str | None]:
|
||||||
|
return {
|
||||||
|
"connector_id": self.connector_id,
|
||||||
|
"credential_id": self.credential_id,
|
||||||
|
"provider": self.provider,
|
||||||
|
"external_id": self.external_id,
|
||||||
|
"external_path": self.external_path,
|
||||||
|
"external_url": self.external_url,
|
||||||
|
"operation": self.operation,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorPolicySource:
|
||||||
|
scope_type: str
|
||||||
|
label: str
|
||||||
|
scope_id: str | None = None
|
||||||
|
policy: Mapping[str, Any] | None = None
|
||||||
|
|
||||||
|
def source_step(self, applied_fields: Iterable[str]) -> PolicySourceStep:
|
||||||
|
return PolicySourceStep(
|
||||||
|
scope_type=normalize_policy_scope_type(self.scope_type),
|
||||||
|
scope_id=self.scope_id,
|
||||||
|
label=self.label,
|
||||||
|
applied_fields=tuple(dict.fromkeys(applied_fields)),
|
||||||
|
policy=dict(self.policy or {}),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def connector_policy_sources_from_payload(value: object) -> list[ConnectorPolicySource]:
|
||||||
|
if value is None or value == "":
|
||||||
|
return []
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
raw_sources = value.get("sources")
|
||||||
|
if raw_sources is None:
|
||||||
|
raw_sources = [value]
|
||||||
|
elif isinstance(value, list):
|
||||||
|
raw_sources = value
|
||||||
|
else:
|
||||||
|
raise ValueError("connector_policy must be a JSON object or list")
|
||||||
|
if not isinstance(raw_sources, list):
|
||||||
|
raise ValueError("connector_policy.sources must be a list")
|
||||||
|
return [_source_from_mapping(item) for item in raw_sources if isinstance(item, Mapping)]
|
||||||
|
|
||||||
|
|
||||||
|
def connector_policy_applied_fields(policy: Mapping[str, Any] | None) -> tuple[str, ...]:
|
||||||
|
return _applied_fields(_policy_mapping(policy))
|
||||||
|
|
||||||
|
|
||||||
|
def filtered_connector_policy_source(source: ConnectorPolicySource, fields: Iterable[str]) -> ConnectorPolicySource:
|
||||||
|
allowed_fields = {field for field in fields if field in _FIELD_ALIASES}
|
||||||
|
if not allowed_fields:
|
||||||
|
return ConnectorPolicySource(scope_type=source.scope_type, scope_id=source.scope_id, label=source.label, policy={})
|
||||||
|
policy = _policy_mapping(source.policy)
|
||||||
|
filtered: dict[str, Any] = {}
|
||||||
|
for prefix, keys in (("allow", _ALLOW_KEYS), ("deny", _DENY_KEYS)):
|
||||||
|
rules = _merged_rule(policy, keys)
|
||||||
|
selected = {field: _string_list(rules.get(field)) for field in allowed_fields}
|
||||||
|
selected = {field: values for field, values in selected.items() if values}
|
||||||
|
if selected:
|
||||||
|
filtered[prefix] = selected
|
||||||
|
return ConnectorPolicySource(scope_type=source.scope_type, scope_id=source.scope_id, label=source.label, policy=filtered)
|
||||||
|
|
||||||
|
|
||||||
|
def connector_policy_sources_for_fields(
|
||||||
|
sources: Iterable[ConnectorPolicySource],
|
||||||
|
fields: Iterable[str],
|
||||||
|
) -> list[ConnectorPolicySource]:
|
||||||
|
return [filtered_connector_policy_source(source, fields) for source in sources]
|
||||||
|
|
||||||
|
|
||||||
|
def connector_policy_decision(
|
||||||
|
request: ConnectorAccessRequest,
|
||||||
|
sources: Iterable[ConnectorPolicySource],
|
||||||
|
) -> PolicyDecision:
|
||||||
|
source_list = list(sources)
|
||||||
|
source_steps: list[PolicySourceStep] = []
|
||||||
|
deny_matches: list[dict[str, Any]] = []
|
||||||
|
allow_misses: list[dict[str, Any]] = []
|
||||||
|
|
||||||
|
for source in source_list:
|
||||||
|
policy = _policy_mapping(source.policy)
|
||||||
|
applied_fields = _applied_fields(policy)
|
||||||
|
if applied_fields:
|
||||||
|
source_steps.append(source.source_step(applied_fields))
|
||||||
|
|
||||||
|
deny_policy = _merged_rule(policy, _DENY_KEYS)
|
||||||
|
for field, patterns in _rules_by_field(deny_policy).items():
|
||||||
|
if _matches_field(request, field, patterns):
|
||||||
|
deny_matches.append({
|
||||||
|
"scope": source.source_step((f"deny.{field}",)).to_dict(),
|
||||||
|
"field": field,
|
||||||
|
"patterns": patterns,
|
||||||
|
})
|
||||||
|
|
||||||
|
allow_policy = _merged_rule(policy, _ALLOW_KEYS)
|
||||||
|
for field, patterns in _rules_by_field(allow_policy).items():
|
||||||
|
if not _matches_field(request, field, patterns):
|
||||||
|
allow_misses.append({
|
||||||
|
"scope": source.source_step((f"allow.{field}",)).to_dict(),
|
||||||
|
"field": field,
|
||||||
|
"patterns": patterns,
|
||||||
|
})
|
||||||
|
|
||||||
|
details = {
|
||||||
|
"request": request.to_dict(),
|
||||||
|
"deny_matches": deny_matches,
|
||||||
|
"allow_misses": allow_misses,
|
||||||
|
}
|
||||||
|
if deny_matches:
|
||||||
|
return PolicyDecision(
|
||||||
|
allowed=False,
|
||||||
|
reason="Connector access is denied by a blacklist rule.",
|
||||||
|
source_path=tuple(source_steps),
|
||||||
|
requirements=("connector_policy_denylist",),
|
||||||
|
details=details,
|
||||||
|
)
|
||||||
|
if allow_misses:
|
||||||
|
return PolicyDecision(
|
||||||
|
allowed=False,
|
||||||
|
reason="Connector access is not allowed by the configured whitelist.",
|
||||||
|
source_path=tuple(source_steps),
|
||||||
|
requirements=("connector_policy_allowlist",),
|
||||||
|
details=details,
|
||||||
|
)
|
||||||
|
return PolicyDecision(
|
||||||
|
allowed=True,
|
||||||
|
reason=None,
|
||||||
|
source_path=tuple(source_steps),
|
||||||
|
requirements=(),
|
||||||
|
details=details,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_connector_policy_allows(
|
||||||
|
request: ConnectorAccessRequest,
|
||||||
|
sources: Iterable[ConnectorPolicySource],
|
||||||
|
) -> PolicyDecision:
|
||||||
|
decision = connector_policy_decision(request, sources)
|
||||||
|
if not decision.allowed:
|
||||||
|
raise ConnectorPolicyDenied(decision)
|
||||||
|
return decision
|
||||||
|
|
||||||
|
|
||||||
|
def _source_from_mapping(value: Mapping[str, Any]) -> ConnectorPolicySource:
|
||||||
|
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||||
|
scope_id = _clean(value.get("scope_id"))
|
||||||
|
if scope_type == "system":
|
||||||
|
scope_id = None
|
||||||
|
elif not scope_id:
|
||||||
|
raise ValueError(f"{scope_type} connector policy sources require scope_id")
|
||||||
|
label = _clean(value.get("label")) or scope_type.capitalize()
|
||||||
|
policy = value.get("policy")
|
||||||
|
if policy is None:
|
||||||
|
policy = {key: value[key] for key in (*_ALLOW_KEYS, *_DENY_KEYS) if key in value}
|
||||||
|
if policy is not None and not isinstance(policy, Mapping):
|
||||||
|
raise ValueError("connector policy source policy must be a JSON object")
|
||||||
|
return ConnectorPolicySource(scope_type=scope_type, scope_id=scope_id, label=label, policy=policy or {})
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_mapping(value: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||||
|
return value if isinstance(value, Mapping) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _merged_rule(policy: Mapping[str, Any], keys: Iterable[str]) -> dict[str, Any]:
|
||||||
|
merged: dict[str, Any] = {}
|
||||||
|
for key in keys:
|
||||||
|
raw = policy.get(key)
|
||||||
|
if isinstance(raw, Mapping):
|
||||||
|
merged.update(raw)
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
def _rules_by_field(value: Mapping[str, Any]) -> dict[str, list[str]]:
|
||||||
|
result: dict[str, list[str]] = {}
|
||||||
|
for field, aliases in _FIELD_ALIASES.items():
|
||||||
|
items: list[str] = []
|
||||||
|
for alias in aliases:
|
||||||
|
items.extend(_string_list(value.get(alias)))
|
||||||
|
if items:
|
||||||
|
result[field] = list(dict.fromkeys(items))
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _applied_fields(policy: Mapping[str, Any]) -> tuple[str, ...]:
|
||||||
|
fields: list[str] = []
|
||||||
|
for prefix, keys in (("allow", _ALLOW_KEYS), ("deny", _DENY_KEYS)):
|
||||||
|
rules = _merged_rule(policy, keys)
|
||||||
|
fields.extend(f"{prefix}.{field}" for field in _rules_by_field(rules))
|
||||||
|
return tuple(dict.fromkeys(fields))
|
||||||
|
|
||||||
|
|
||||||
|
def _matches_field(request: ConnectorAccessRequest, field: str, patterns: list[str]) -> bool:
|
||||||
|
if field == "connectors":
|
||||||
|
return _matches_reference(request.connector_id, patterns)
|
||||||
|
if field == "credentials":
|
||||||
|
return _matches_reference(request.credential_id, patterns)
|
||||||
|
if field == "providers":
|
||||||
|
return _matches_reference(request.provider, patterns)
|
||||||
|
if field == "external_ids":
|
||||||
|
return _matches_glob(request.external_id, patterns)
|
||||||
|
if field == "external_paths":
|
||||||
|
return _matches_path(request.external_path, patterns)
|
||||||
|
if field == "external_urls":
|
||||||
|
return _matches_glob(request.external_url, patterns)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _matches_reference(value: str | None, patterns: list[str]) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return False
|
||||||
|
clean = value.casefold()
|
||||||
|
return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
|
||||||
|
|
||||||
|
|
||||||
|
def _matches_glob(value: str | None, patterns: list[str]) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return False
|
||||||
|
clean = value.casefold()
|
||||||
|
return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
|
||||||
|
|
||||||
|
|
||||||
|
def _matches_path(value: str | None, patterns: list[str]) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return False
|
||||||
|
clean = value.replace("\\", "/").strip()
|
||||||
|
for pattern in patterns:
|
||||||
|
normalized = pattern.replace("\\", "/").strip()
|
||||||
|
if normalized == "*":
|
||||||
|
return True
|
||||||
|
if any(token in normalized for token in "*?[]"):
|
||||||
|
if fnmatchcase(clean.casefold(), normalized.casefold()):
|
||||||
|
return True
|
||||||
|
continue
|
||||||
|
if clean.casefold() == normalized.casefold() or clean.casefold().startswith(normalized.rstrip("/").casefold() + "/"):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _string_list(value: object) -> list[str]:
|
||||||
|
if value is None:
|
||||||
|
return []
|
||||||
|
if isinstance(value, str):
|
||||||
|
values = [value]
|
||||||
|
elif isinstance(value, Iterable) and not isinstance(value, (bytes, bytearray, Mapping)):
|
||||||
|
values = [str(item) for item in value]
|
||||||
|
else:
|
||||||
|
values = [str(value)]
|
||||||
|
return [item.strip() for item in values if item.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
@@ -0,0 +1,334 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterable, Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.policy import normalize_policy_scope_type
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorPolicy
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
CONNECTOR_POLICY_FIELDS,
|
||||||
|
ConnectorPolicySource,
|
||||||
|
connector_policy_applied_fields,
|
||||||
|
)
|
||||||
|
|
||||||
|
_RULE_GROUPS = ("allow", "deny")
|
||||||
|
_FIELD_ALIASES = {
|
||||||
|
"connectors": ("connectors", "connector_ids", "connector_id"),
|
||||||
|
"credentials": ("credentials", "credential_ids", "credential_id"),
|
||||||
|
"providers": ("providers", "provider"),
|
||||||
|
"external_ids": ("external_ids", "external_id"),
|
||||||
|
"external_paths": ("external_paths", "paths", "path_prefixes", "external_path"),
|
||||||
|
"external_urls": ("external_urls", "urls", "external_url"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_connector_policy(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
return _normalize_connector_policy(row.policy if row is not None else None)
|
||||||
|
|
||||||
|
|
||||||
|
def set_connector_policy(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str | None,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
policy: Mapping[str, Any] | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
normalized = _normalize_connector_policy(policy)
|
||||||
|
parent = parent_connector_policy(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
if parent is not None:
|
||||||
|
_validate_lower_level_limits(parent, normalized)
|
||||||
|
row_tenant_id, row_scope_type, row_scope_id = _policy_scope_key(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=row_scope_type, scope_id=row_scope_id)
|
||||||
|
if row is None:
|
||||||
|
row = FileConnectorPolicy(
|
||||||
|
tenant_id=row_tenant_id,
|
||||||
|
scope_type=row_scope_type,
|
||||||
|
scope_id=row_scope_id,
|
||||||
|
policy=normalized,
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
updated_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
row.policy = normalized
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
session.add(row)
|
||||||
|
session.flush()
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def connector_policy_response(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
policy = get_connector_policy(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||||
|
effective_sources = effective_connector_policy_sources(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||||
|
parent_policy = parent_connector_policy(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||||
|
parent_sources = parent_connector_policy_sources(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||||
|
return {
|
||||||
|
"scope_type": clean_scope_type,
|
||||||
|
"scope_id": clean_scope_id,
|
||||||
|
"policy": policy,
|
||||||
|
"effective_policy": merge_connector_policies(source.policy for source in effective_sources),
|
||||||
|
"parent_policy": parent_policy,
|
||||||
|
"effective_policy_sources": [_source_step(source) for source in effective_sources],
|
||||||
|
"parent_policy_sources": [_source_step(source) for source in parent_sources],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def effective_connector_policy_sources(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
) -> list[ConnectorPolicySource]:
|
||||||
|
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
sources: list[ConnectorPolicySource] = []
|
||||||
|
for source_scope_type, source_scope_id in _policy_source_chain(tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id):
|
||||||
|
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=source_scope_type, scope_id=source_scope_id)
|
||||||
|
if row is None:
|
||||||
|
continue
|
||||||
|
policy = _normalize_connector_policy(row.policy)
|
||||||
|
if _policy_is_empty(policy):
|
||||||
|
continue
|
||||||
|
sources.append(
|
||||||
|
ConnectorPolicySource(
|
||||||
|
scope_type=source_scope_type,
|
||||||
|
scope_id=source_scope_id,
|
||||||
|
label=_policy_source_label(source_scope_type),
|
||||||
|
policy=policy,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return sources
|
||||||
|
|
||||||
|
|
||||||
|
def parent_connector_policy_sources(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
) -> list[ConnectorPolicySource]:
|
||||||
|
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
chain = _policy_source_chain(tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||||
|
parent_chain = chain[:-1] if chain else []
|
||||||
|
sources: list[ConnectorPolicySource] = []
|
||||||
|
for source_scope_type, source_scope_id in parent_chain:
|
||||||
|
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=source_scope_type, scope_id=source_scope_id)
|
||||||
|
if row is None:
|
||||||
|
continue
|
||||||
|
policy = _normalize_connector_policy(row.policy)
|
||||||
|
if _policy_is_empty(policy):
|
||||||
|
continue
|
||||||
|
sources.append(
|
||||||
|
ConnectorPolicySource(
|
||||||
|
scope_type=source_scope_type,
|
||||||
|
scope_id=source_scope_id,
|
||||||
|
label=_policy_source_label(source_scope_type),
|
||||||
|
policy=policy,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return sources
|
||||||
|
|
||||||
|
|
||||||
|
def parent_connector_policy(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
) -> dict[str, Any] | None:
|
||||||
|
sources = parent_connector_policy_sources(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
if not sources:
|
||||||
|
return None
|
||||||
|
return merge_connector_policies(source.policy for source in sources)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_connector_policy_allowed_by_parent(parent_policy: Mapping[str, Any] | None, local_policy: Mapping[str, Any] | None) -> None:
|
||||||
|
if parent_policy is None:
|
||||||
|
return
|
||||||
|
_validate_lower_level_limits(_normalize_connector_policy(parent_policy), _normalize_connector_policy(local_policy))
|
||||||
|
|
||||||
|
|
||||||
|
def merge_connector_policies(policies: Iterable[Mapping[str, Any] | None]) -> dict[str, Any]:
|
||||||
|
result = _empty_connector_policy()
|
||||||
|
for policy in policies:
|
||||||
|
normalized = _normalize_connector_policy(policy)
|
||||||
|
for group in _RULE_GROUPS:
|
||||||
|
rules = normalized.get(group) or {}
|
||||||
|
if not isinstance(rules, Mapping):
|
||||||
|
continue
|
||||||
|
target = result[group]
|
||||||
|
for field in CONNECTOR_POLICY_FIELDS:
|
||||||
|
values = _string_list(rules.get(field))
|
||||||
|
if values:
|
||||||
|
target[field] = _unique([*target.get(field, []), *values])
|
||||||
|
lower = normalized.get("allow_lower_level_limits") or {}
|
||||||
|
if isinstance(lower, Mapping):
|
||||||
|
for key, value in lower.items():
|
||||||
|
if isinstance(value, bool):
|
||||||
|
result["allow_lower_level_limits"][str(key)] = value
|
||||||
|
return _compact_connector_policy(result, keep_lower=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _connector_policy_row(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
) -> FileConnectorPolicy | None:
|
||||||
|
row_tenant_id, row_scope_type, row_scope_id = _policy_scope_key(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
query = session.query(FileConnectorPolicy).filter(FileConnectorPolicy.scope_type == row_scope_type)
|
||||||
|
query = query.filter(FileConnectorPolicy.tenant_id.is_(None) if row_tenant_id is None else FileConnectorPolicy.tenant_id == row_tenant_id)
|
||||||
|
query = query.filter(FileConnectorPolicy.scope_id.is_(None) if row_scope_id is None else FileConnectorPolicy.scope_id == row_scope_id)
|
||||||
|
return query.order_by(FileConnectorPolicy.created_at.asc()).first()
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_scope_key(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str | None, str, str | None]:
|
||||||
|
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
if clean_scope_type == "system":
|
||||||
|
return None, "system", None
|
||||||
|
return tenant_id, clean_scope_type, clean_scope_id
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_scope_ref(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None]:
|
||||||
|
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||||
|
clean_scope_id = _clean(scope_id)
|
||||||
|
if clean_scope_type == "system":
|
||||||
|
return "system", None
|
||||||
|
if clean_scope_type == "tenant":
|
||||||
|
if clean_scope_id and clean_scope_id != tenant_id:
|
||||||
|
raise FileStorageError("Tenant connector policy scope does not belong to the active tenant")
|
||||||
|
return "tenant", tenant_id
|
||||||
|
if clean_scope_type in {"user", "group", "campaign"}:
|
||||||
|
if not clean_scope_id:
|
||||||
|
raise FileStorageError(f"{clean_scope_type.capitalize()} connector policy requires scope_id")
|
||||||
|
return clean_scope_type, clean_scope_id
|
||||||
|
raise FileStorageError("Unsupported connector policy scope")
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_source_chain(*, tenant_id: str, scope_type: str, scope_id: str | None) -> list[tuple[str, str | None]]:
|
||||||
|
chain: list[tuple[str, str | None]] = [("system", None)]
|
||||||
|
if scope_type == "system":
|
||||||
|
return chain
|
||||||
|
chain.append(("tenant", tenant_id))
|
||||||
|
if scope_type == "tenant":
|
||||||
|
return chain
|
||||||
|
chain.append((scope_type, scope_id))
|
||||||
|
return chain
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_connector_policy(policy: Mapping[str, Any] | None) -> dict[str, Any]:
|
||||||
|
if policy is None:
|
||||||
|
return {}
|
||||||
|
if not isinstance(policy, Mapping):
|
||||||
|
raise FileStorageError("Connector policy must be a JSON object")
|
||||||
|
normalized = _empty_connector_policy()
|
||||||
|
for group in _RULE_GROUPS:
|
||||||
|
raw = policy.get(group)
|
||||||
|
if raw is None:
|
||||||
|
raw = policy.get(f"{group}list")
|
||||||
|
if raw is None:
|
||||||
|
raw = policy.get("whitelist" if group == "allow" else "blacklist")
|
||||||
|
if raw is not None and not isinstance(raw, Mapping):
|
||||||
|
raise FileStorageError(f"Connector policy {group} rules must be an object")
|
||||||
|
for field, aliases in _FIELD_ALIASES.items():
|
||||||
|
values: list[str] = []
|
||||||
|
if isinstance(raw, Mapping):
|
||||||
|
for alias in aliases:
|
||||||
|
values.extend(_string_list(raw.get(alias)))
|
||||||
|
if values:
|
||||||
|
normalized[group][field] = _unique(values)
|
||||||
|
lower = policy.get("allow_lower_level_limits")
|
||||||
|
if lower is not None:
|
||||||
|
if not isinstance(lower, Mapping):
|
||||||
|
raise FileStorageError("Connector policy allow_lower_level_limits must be an object")
|
||||||
|
for key, value in lower.items():
|
||||||
|
clean_key = str(key).strip()
|
||||||
|
if clean_key and isinstance(value, bool):
|
||||||
|
normalized["allow_lower_level_limits"][clean_key] = value
|
||||||
|
return _compact_connector_policy(normalized, keep_lower=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _empty_connector_policy() -> dict[str, Any]:
|
||||||
|
return {"allow": {}, "deny": {}, "allow_lower_level_limits": {}}
|
||||||
|
|
||||||
|
|
||||||
|
def _compact_connector_policy(policy: dict[str, Any], *, keep_lower: bool = False) -> dict[str, Any]:
|
||||||
|
compact: dict[str, Any] = {}
|
||||||
|
for group in _RULE_GROUPS:
|
||||||
|
rules = {field: _unique(_string_list(values)) for field, values in (policy.get(group) or {}).items()}
|
||||||
|
rules = {field: values for field, values in rules.items() if values}
|
||||||
|
if rules:
|
||||||
|
compact[group] = rules
|
||||||
|
lower = policy.get("allow_lower_level_limits") or {}
|
||||||
|
if isinstance(lower, Mapping):
|
||||||
|
lower_compact = {str(key): bool(value) for key, value in lower.items() if isinstance(value, bool)}
|
||||||
|
if lower_compact or keep_lower:
|
||||||
|
compact["allow_lower_level_limits"] = lower_compact
|
||||||
|
return compact
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_is_empty(policy: Mapping[str, Any]) -> bool:
|
||||||
|
return not connector_policy_applied_fields(policy) and not bool(policy.get("allow_lower_level_limits"))
|
||||||
|
|
||||||
|
|
||||||
|
def _source_step(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||||
|
return source.source_step(connector_policy_applied_fields(source.policy)).to_dict()
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_source_label(scope_type: str) -> str:
|
||||||
|
if scope_type == "system":
|
||||||
|
return "System connector policy"
|
||||||
|
if scope_type == "tenant":
|
||||||
|
return "Tenant connector policy"
|
||||||
|
return f"{scope_type.capitalize()} connector policy"
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_lower_level_limits(parent_policy: Mapping[str, Any], local_policy: Mapping[str, Any]) -> None:
|
||||||
|
limits = parent_policy.get("allow_lower_level_limits")
|
||||||
|
if not isinstance(limits, Mapping):
|
||||||
|
return
|
||||||
|
for field in connector_policy_applied_fields(local_policy):
|
||||||
|
allowed = limits.get(field)
|
||||||
|
if allowed is False:
|
||||||
|
raise FileStorageError(f"Parent connector policy does not allow lower-level {field} limits")
|
||||||
|
|
||||||
|
|
||||||
|
def _string_list(value: object) -> list[str]:
|
||||||
|
if value is None:
|
||||||
|
return []
|
||||||
|
if isinstance(value, str):
|
||||||
|
clean = value.strip()
|
||||||
|
return [clean] if clean else []
|
||||||
|
if isinstance(value, Iterable) and not isinstance(value, (str, bytes, bytearray, Mapping)):
|
||||||
|
return [str(item).strip() for item in value if str(item).strip()]
|
||||||
|
return [str(value).strip()] if str(value).strip() else []
|
||||||
|
|
||||||
|
|
||||||
|
def _unique(values: Iterable[str]) -> list[str]:
|
||||||
|
return list(dict.fromkeys(value for value in values if value))
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
@@ -0,0 +1,470 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable, Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.policy import normalize_policy_scope_type
|
||||||
|
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||||
|
from govoplan_files.backend.storage.connector_credential_store import (
|
||||||
|
ConnectorCredential,
|
||||||
|
connector_credential_from_row,
|
||||||
|
credential_rows_by_id,
|
||||||
|
resolve_reusable_connector_credential,
|
||||||
|
reusable_credential_id,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
|
||||||
|
|
||||||
|
|
||||||
|
def list_database_connector_profiles(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||||
|
) -> list[ConnectorProfile]:
|
||||||
|
profiles, _profile_ids = select_database_connector_profiles(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
row_visible=row_visible,
|
||||||
|
)
|
||||||
|
return profiles
|
||||||
|
|
||||||
|
|
||||||
|
def select_database_connector_profiles(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||||
|
) -> tuple[list[ConnectorProfile], set[str]]:
|
||||||
|
"""Return visible profiles and every database id that shadows settings."""
|
||||||
|
|
||||||
|
query = session.query(FileConnectorProfile).filter(
|
||||||
|
(FileConnectorProfile.scope_type == "system")
|
||||||
|
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||||
|
)
|
||||||
|
if not include_disabled:
|
||||||
|
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||||
|
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||||
|
profile_ids = {row.id for row in rows}
|
||||||
|
if row_visible is not None:
|
||||||
|
rows = [row for row in rows if row_visible(row)]
|
||||||
|
credential_ids = {
|
||||||
|
_clean(row.credential_profile_id)
|
||||||
|
for row in rows
|
||||||
|
if _clean(row.credential_profile_id) and not reusable_credential_id(row.credential_profile_id)
|
||||||
|
}
|
||||||
|
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
|
||||||
|
return (
|
||||||
|
[
|
||||||
|
connector_profile_from_row(
|
||||||
|
row,
|
||||||
|
credential_row=(
|
||||||
|
_resolve_profile_reusable_credential(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
row=row,
|
||||||
|
)
|
||||||
|
if reusable_credential_id(row.credential_profile_id)
|
||||||
|
else credentials.get(row.credential_profile_id or "")
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for row in rows
|
||||||
|
],
|
||||||
|
profile_ids,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_connector_profile_rows(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> list[FileConnectorProfile]:
|
||||||
|
query = session.query(FileConnectorProfile).filter(
|
||||||
|
(FileConnectorProfile.scope_type == "system")
|
||||||
|
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||||
|
)
|
||||||
|
if not include_disabled:
|
||||||
|
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||||
|
return query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
def connector_profile_from_row(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
credential_row: FileConnectorCredential | ConnectorCredential | None = None,
|
||||||
|
) -> ConnectorProfile:
|
||||||
|
policy_sources = []
|
||||||
|
if row.policy:
|
||||||
|
policy_sources = connector_policy_sources_from_payload({
|
||||||
|
"scope_type": row.scope_type,
|
||||||
|
"scope_id": row.scope_id,
|
||||||
|
"label": row.label,
|
||||||
|
"policy": row.policy,
|
||||||
|
})
|
||||||
|
credential = (
|
||||||
|
credential_row
|
||||||
|
if isinstance(credential_row, ConnectorCredential)
|
||||||
|
else connector_credential_from_row(credential_row)
|
||||||
|
if credential_row is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
if credential:
|
||||||
|
policy_sources.extend(credential.policy_sources)
|
||||||
|
metadata = dict(row.metadata_ or {})
|
||||||
|
if reusable_credential_id(row.credential_profile_id) and credential is None:
|
||||||
|
metadata["credential_unavailable"] = True
|
||||||
|
return ConnectorProfile(
|
||||||
|
id=row.id,
|
||||||
|
label=row.label,
|
||||||
|
provider=row.provider,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
endpoint_url=row.endpoint_url,
|
||||||
|
base_path=row.base_path,
|
||||||
|
enabled=row.enabled,
|
||||||
|
credential_profile_id=row.credential_profile_id,
|
||||||
|
credential_profile_label=credential.label if credential else None,
|
||||||
|
credential_mode=credential.credential_mode if credential else row.credential_mode,
|
||||||
|
username=credential.username if credential else row.username,
|
||||||
|
password_env=credential.password_env if credential else row.password_env,
|
||||||
|
token_env=credential.token_env if credential else row.token_env,
|
||||||
|
secret_ref=credential.secret_ref if credential else row.secret_ref,
|
||||||
|
password_value=credential.password_value if credential else decrypt_secret(row.password_encrypted),
|
||||||
|
token_value=credential.token_value if credential else decrypt_secret(row.token_encrypted),
|
||||||
|
capabilities=tuple(_string_list(row.capabilities)),
|
||||||
|
policy_sources=tuple(policy_sources),
|
||||||
|
metadata=metadata,
|
||||||
|
source_kind="database",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_profile_reusable_credential(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
) -> ConnectorCredential | None:
|
||||||
|
try:
|
||||||
|
return resolve_reusable_connector_credential(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
credential_ref=row.credential_profile_id or "",
|
||||||
|
profile_id=row.id,
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
)
|
||||||
|
except FileStorageError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def get_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
profile_id: str,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
) -> FileConnectorProfile:
|
||||||
|
row = session.get(FileConnectorProfile, profile_id)
|
||||||
|
if row is None or (row.scope_type != "system" and row.tenant_id != tenant_id):
|
||||||
|
raise FileStorageError("Connector profile not found")
|
||||||
|
if not include_disabled and not row.enabled:
|
||||||
|
raise FileStorageError("Connector profile not found")
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def create_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str | None,
|
||||||
|
profile_id: str,
|
||||||
|
label: str,
|
||||||
|
provider: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None = None,
|
||||||
|
endpoint_url: str | None = None,
|
||||||
|
base_path: str | None = None,
|
||||||
|
enabled: bool = True,
|
||||||
|
credential_profile_id: str | None = None,
|
||||||
|
credential_mode: str = "none",
|
||||||
|
username: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
token: str | None = None,
|
||||||
|
password_env: str | None = None,
|
||||||
|
token_env: str | None = None,
|
||||||
|
secret_ref: str | None = None,
|
||||||
|
capabilities: list[str] | None = None,
|
||||||
|
policy: Mapping[str, Any] | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
) -> FileConnectorProfile:
|
||||||
|
reject_api_controlled_deployment_references(
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
clean_id = _normalize_profile_id(profile_id)
|
||||||
|
if session.get(FileConnectorProfile, clean_id) is not None:
|
||||||
|
raise FileStorageError(f"Connector profile already exists: {clean_id}")
|
||||||
|
clean_scope_type, clean_scope_id, row_tenant_id = _normalize_scope(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||||
|
row = FileConnectorProfile(
|
||||||
|
id=clean_id,
|
||||||
|
tenant_id=row_tenant_id,
|
||||||
|
scope_type=clean_scope_type,
|
||||||
|
scope_id=clean_scope_id,
|
||||||
|
label=_normalize_label(label),
|
||||||
|
provider=_normalize_provider(provider),
|
||||||
|
endpoint_url=_clean(endpoint_url),
|
||||||
|
base_path=_clean(base_path),
|
||||||
|
enabled=bool(enabled),
|
||||||
|
credential_profile_id=_clean(credential_profile_id),
|
||||||
|
credential_mode=_normalize_credential_mode(credential_mode),
|
||||||
|
username=_clean(username),
|
||||||
|
password_encrypted=encrypt_secret(_clean(password)),
|
||||||
|
token_encrypted=encrypt_secret(_clean(token)),
|
||||||
|
password_env=_clean(password_env),
|
||||||
|
token_env=_clean(token_env),
|
||||||
|
secret_ref=_clean(secret_ref),
|
||||||
|
capabilities=_string_list(capabilities),
|
||||||
|
policy=dict(policy or {}),
|
||||||
|
metadata_=dict(metadata or {}),
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
updated_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
session.add(row)
|
||||||
|
session.flush()
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def update_connector_profile_row(
|
||||||
|
session: Session,
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
user_id: str | None,
|
||||||
|
label: str | None = None,
|
||||||
|
provider: str | None = None,
|
||||||
|
endpoint_url: str | None = None,
|
||||||
|
base_path: str | None = None,
|
||||||
|
enabled: bool | None = None,
|
||||||
|
credential_profile_id: str | None = None,
|
||||||
|
credential_mode: str | None = None,
|
||||||
|
username: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
token: str | None = None,
|
||||||
|
password_env: str | None = None,
|
||||||
|
token_env: str | None = None,
|
||||||
|
secret_ref: str | None = None,
|
||||||
|
capabilities: list[str] | None = None,
|
||||||
|
policy: Mapping[str, Any] | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
clear_password: bool = False,
|
||||||
|
clear_token: bool = False,
|
||||||
|
) -> FileConnectorProfile:
|
||||||
|
_validate_profile_update_references(
|
||||||
|
row,
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
_update_profile_connection_fields(
|
||||||
|
row,
|
||||||
|
label=label,
|
||||||
|
provider=provider,
|
||||||
|
endpoint_url=endpoint_url,
|
||||||
|
base_path=base_path,
|
||||||
|
enabled=enabled,
|
||||||
|
credential_profile_id=credential_profile_id,
|
||||||
|
credential_mode=credential_mode,
|
||||||
|
)
|
||||||
|
_update_profile_credential_fields(
|
||||||
|
row,
|
||||||
|
username=username,
|
||||||
|
password=password,
|
||||||
|
token=token,
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
clear_password=clear_password,
|
||||||
|
clear_token=clear_token,
|
||||||
|
)
|
||||||
|
_update_profile_governance_fields(
|
||||||
|
row,
|
||||||
|
capabilities=capabilities,
|
||||||
|
policy=policy,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
row.updated_by_user_id = user_id
|
||||||
|
session.add(row)
|
||||||
|
session.flush()
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_profile_update_references(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
password_env: str | None,
|
||||||
|
token_env: str | None,
|
||||||
|
secret_ref: str | None,
|
||||||
|
metadata: Mapping[str, Any] | None,
|
||||||
|
) -> None:
|
||||||
|
reject_api_controlled_deployment_references(
|
||||||
|
password_env=password_env,
|
||||||
|
token_env=token_env,
|
||||||
|
secret_ref=secret_ref,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
if secret_ref is None or _clean(secret_ref) == _clean(row.secret_ref):
|
||||||
|
return
|
||||||
|
if _clean(row.secret_ref):
|
||||||
|
raise FileStorageError(
|
||||||
|
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||||
|
"provider ownership and confirm provider-side deletion"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_profile_connection_fields(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
label: str | None,
|
||||||
|
provider: str | None,
|
||||||
|
endpoint_url: str | None,
|
||||||
|
base_path: str | None,
|
||||||
|
enabled: bool | None,
|
||||||
|
credential_profile_id: str | None,
|
||||||
|
credential_mode: str | None,
|
||||||
|
) -> None:
|
||||||
|
if label is not None:
|
||||||
|
row.label = _normalize_label(label)
|
||||||
|
if provider is not None:
|
||||||
|
row.provider = _normalize_provider(provider)
|
||||||
|
if endpoint_url is not None:
|
||||||
|
row.endpoint_url = _clean(endpoint_url)
|
||||||
|
if base_path is not None:
|
||||||
|
row.base_path = _clean(base_path)
|
||||||
|
if enabled is not None:
|
||||||
|
row.enabled = bool(enabled)
|
||||||
|
if credential_profile_id is not None:
|
||||||
|
row.credential_profile_id = _clean(credential_profile_id)
|
||||||
|
if credential_mode is not None:
|
||||||
|
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_profile_credential_fields(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
username: str | None,
|
||||||
|
password: str | None,
|
||||||
|
token: str | None,
|
||||||
|
password_env: str | None,
|
||||||
|
token_env: str | None,
|
||||||
|
secret_ref: str | None,
|
||||||
|
clear_password: bool,
|
||||||
|
clear_token: bool,
|
||||||
|
) -> None:
|
||||||
|
if username is not None:
|
||||||
|
row.username = _clean(username)
|
||||||
|
if password is not None:
|
||||||
|
row.password_encrypted = encrypt_secret(_clean(password))
|
||||||
|
elif clear_password:
|
||||||
|
row.password_encrypted = None
|
||||||
|
if token is not None:
|
||||||
|
row.token_encrypted = encrypt_secret(_clean(token))
|
||||||
|
elif clear_token:
|
||||||
|
row.token_encrypted = None
|
||||||
|
if password_env is not None:
|
||||||
|
row.password_env = _clean(password_env)
|
||||||
|
if token_env is not None:
|
||||||
|
row.token_env = _clean(token_env)
|
||||||
|
if secret_ref is not None:
|
||||||
|
row.secret_ref = _clean(secret_ref)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_profile_governance_fields(
|
||||||
|
row: FileConnectorProfile,
|
||||||
|
*,
|
||||||
|
capabilities: list[str] | None,
|
||||||
|
policy: Mapping[str, Any] | None,
|
||||||
|
metadata: Mapping[str, Any] | None,
|
||||||
|
) -> None:
|
||||||
|
if capabilities is not None:
|
||||||
|
row.capabilities = _string_list(capabilities)
|
||||||
|
if policy is not None:
|
||||||
|
row.policy = dict(policy)
|
||||||
|
if metadata is not None:
|
||||||
|
row.metadata_ = dict(metadata)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||||
|
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||||
|
clean_scope_id = _clean(scope_id)
|
||||||
|
if clean_scope_type == "system":
|
||||||
|
return "system", None, None
|
||||||
|
if clean_scope_type == "tenant":
|
||||||
|
return "tenant", tenant_id, tenant_id
|
||||||
|
if clean_scope_type in {"user", "group", "campaign"}:
|
||||||
|
if not clean_scope_id:
|
||||||
|
raise FileStorageError(f"{clean_scope_type.capitalize()} connector profiles require scope_id")
|
||||||
|
return clean_scope_type, clean_scope_id, tenant_id
|
||||||
|
raise FileStorageError("Unsupported connector profile scope")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_profile_id(value: str) -> str:
|
||||||
|
clean = _clean(value)
|
||||||
|
if not clean:
|
||||||
|
raise FileStorageError("Connector profile id is required")
|
||||||
|
if len(clean) > 255:
|
||||||
|
raise FileStorageError("Connector profile id is too long")
|
||||||
|
if any(char.isspace() for char in clean):
|
||||||
|
raise FileStorageError("Connector profile id cannot contain whitespace")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_label(value: str) -> str:
|
||||||
|
clean = value.strip()
|
||||||
|
if not clean:
|
||||||
|
raise FileStorageError("Connector profile label is required")
|
||||||
|
if len(clean) > 255:
|
||||||
|
raise FileStorageError("Connector profile label is too long")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_provider(value: str) -> str:
|
||||||
|
clean = value.strip().casefold()
|
||||||
|
if clean not in supported_connector_providers():
|
||||||
|
raise FileStorageError(f"Unsupported connector provider: {value}")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_credential_mode(value: str) -> str:
|
||||||
|
clean = value.strip().casefold() or "none"
|
||||||
|
if clean not in {"none", "anonymous", "basic", "token", "secret_ref"}:
|
||||||
|
raise FileStorageError(f"Unsupported connector credential mode: {value}")
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _string_list(value: object) -> list[str]:
|
||||||
|
if value is None:
|
||||||
|
return []
|
||||||
|
if isinstance(value, str):
|
||||||
|
values = value.split(",")
|
||||||
|
elif isinstance(value, (list, tuple, set)):
|
||||||
|
values = value
|
||||||
|
else:
|
||||||
|
values = [value]
|
||||||
|
return [str(item).strip() for item in values if str(item).strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
@@ -0,0 +1,343 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from collections.abc import Iterable, Mapping
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
connector_secret_env_available,
|
||||||
|
validate_deployment_connector_references,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||||
|
|
||||||
|
|
||||||
|
_ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON")
|
||||||
|
_ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE")
|
||||||
|
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"}
|
||||||
|
_INLINE_SECRET_FIELDS = (
|
||||||
|
"password",
|
||||||
|
"token",
|
||||||
|
"api_key",
|
||||||
|
"access_key",
|
||||||
|
"access_key_id",
|
||||||
|
"secret_key",
|
||||||
|
"secret_access_key",
|
||||||
|
"session_token",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def supported_connector_providers() -> set[str]:
|
||||||
|
return set(_SUPPORTED_PROVIDERS)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorProfile:
|
||||||
|
id: str
|
||||||
|
label: str
|
||||||
|
provider: str
|
||||||
|
scope_type: str = "system"
|
||||||
|
scope_id: str | None = None
|
||||||
|
endpoint_url: str | None = None
|
||||||
|
base_path: str | None = None
|
||||||
|
enabled: bool = True
|
||||||
|
credential_profile_id: str | None = None
|
||||||
|
credential_profile_label: str | None = None
|
||||||
|
credential_mode: str = "none"
|
||||||
|
username: str | None = None
|
||||||
|
password_env: str | None = None
|
||||||
|
token_env: str | None = None
|
||||||
|
secret_ref: str | None = None
|
||||||
|
password_value: str | None = field(default=None, repr=False)
|
||||||
|
token_value: str | None = field(default=None, repr=False)
|
||||||
|
has_inline_secret: bool = False
|
||||||
|
capabilities: tuple[str, ...] = ()
|
||||||
|
policy_sources: tuple[ConnectorPolicySource, ...] = ()
|
||||||
|
metadata: Mapping[str, Any] = field(default_factory=dict)
|
||||||
|
source_kind: str = "settings"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def source_path(self) -> str:
|
||||||
|
return policy_source_path(self.scope_type, self.scope_id)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def credential_source(self) -> str | None:
|
||||||
|
if self.credential_profile_id:
|
||||||
|
return "credential_profile"
|
||||||
|
if self.secret_ref:
|
||||||
|
return "secret_ref"
|
||||||
|
if self.token_value or self.password_value:
|
||||||
|
return "stored"
|
||||||
|
if self.token_env:
|
||||||
|
return "token_env"
|
||||||
|
if self.password_env:
|
||||||
|
return "password_env"
|
||||||
|
if self.has_inline_secret:
|
||||||
|
return "inline"
|
||||||
|
return None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def credentials_configured(self) -> bool:
|
||||||
|
if not self.enabled:
|
||||||
|
return False
|
||||||
|
mode = self.credential_mode.casefold()
|
||||||
|
if mode in {"", "none", "anonymous"}:
|
||||||
|
return True
|
||||||
|
if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value:
|
||||||
|
return True
|
||||||
|
if self.token_env:
|
||||||
|
return connector_secret_env_available(self.token_env, source_kind=self.source_kind)
|
||||||
|
if self.password_env:
|
||||||
|
return connector_secret_env_available(self.password_env, source_kind=self.source_kind)
|
||||||
|
return False
|
||||||
|
|
||||||
|
def to_response(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": self.id,
|
||||||
|
"label": self.label,
|
||||||
|
"provider": self.provider,
|
||||||
|
"endpoint_url": self.endpoint_url,
|
||||||
|
"base_path": self.base_path,
|
||||||
|
"enabled": self.enabled,
|
||||||
|
"scope_type": self.scope_type,
|
||||||
|
"scope_id": self.scope_id,
|
||||||
|
"source_path": self.source_path,
|
||||||
|
"credential_profile_id": self.credential_profile_id,
|
||||||
|
"credential_profile_label": self.credential_profile_label,
|
||||||
|
"credential_mode": self.credential_mode,
|
||||||
|
"credential_source": self.credential_source,
|
||||||
|
"credentials_configured": self.credentials_configured,
|
||||||
|
"username": self.username,
|
||||||
|
"capabilities": list(self.capabilities),
|
||||||
|
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||||
|
"metadata": _response_metadata(self.metadata),
|
||||||
|
"source_kind": self.source_kind,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def connector_profiles_from_settings(settings: object | None = None) -> list[ConnectorProfile]:
|
||||||
|
raw = _raw_profiles_payload(settings)
|
||||||
|
if raw in (None, ""):
|
||||||
|
return []
|
||||||
|
payload = json.loads(raw) if isinstance(raw, str) else raw
|
||||||
|
return connector_profiles_from_payload(payload)
|
||||||
|
|
||||||
|
|
||||||
|
def connector_profiles_from_payload(value: object) -> list[ConnectorProfile]:
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
raw_profiles = value.get("profiles", [])
|
||||||
|
elif isinstance(value, list):
|
||||||
|
raw_profiles = value
|
||||||
|
else:
|
||||||
|
raise ValueError("Connector profiles must be a JSON object with profiles or a list")
|
||||||
|
if not isinstance(raw_profiles, list):
|
||||||
|
raise ValueError("Connector profiles payload requires a profiles list")
|
||||||
|
return [_profile_from_mapping(item) for item in raw_profiles if isinstance(item, Mapping)]
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
|
||||||
|
profile_id = _profile_id_from_mapping(value)
|
||||||
|
provider = _provider_from_mapping(value)
|
||||||
|
scope_type, scope_id = _scope_from_mapping(value)
|
||||||
|
credentials = _credentials_from_mapping(value)
|
||||||
|
mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none"
|
||||||
|
profile = _profile_from_normalized_mapping(
|
||||||
|
value,
|
||||||
|
profile_id=profile_id,
|
||||||
|
provider=provider,
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
credential_mode=mode,
|
||||||
|
credentials=credentials,
|
||||||
|
)
|
||||||
|
result = ConnectorProfile(
|
||||||
|
id=profile.id,
|
||||||
|
label=profile.label,
|
||||||
|
provider=profile.provider,
|
||||||
|
scope_type=profile.scope_type,
|
||||||
|
scope_id=profile.scope_id,
|
||||||
|
endpoint_url=profile.endpoint_url,
|
||||||
|
base_path=profile.base_path,
|
||||||
|
enabled=profile.enabled,
|
||||||
|
credential_profile_id=profile.credential_profile_id,
|
||||||
|
credential_profile_label=profile.credential_profile_label,
|
||||||
|
credential_mode=profile.credential_mode,
|
||||||
|
username=profile.username,
|
||||||
|
password_env=profile.password_env,
|
||||||
|
token_env=profile.token_env,
|
||||||
|
secret_ref=profile.secret_ref,
|
||||||
|
password_value=profile.password_value,
|
||||||
|
token_value=profile.token_value,
|
||||||
|
has_inline_secret=profile.has_inline_secret,
|
||||||
|
capabilities=profile.capabilities,
|
||||||
|
policy_sources=tuple(_policy_sources_from_profile(value, profile)),
|
||||||
|
metadata=profile.metadata,
|
||||||
|
source_kind="settings",
|
||||||
|
)
|
||||||
|
validate_deployment_connector_references(
|
||||||
|
source_kind=result.source_kind,
|
||||||
|
password_env=result.password_env,
|
||||||
|
token_env=result.token_env,
|
||||||
|
metadata=result.metadata,
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_id_from_mapping(value: Mapping[str, Any]) -> str:
|
||||||
|
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
|
||||||
|
if not profile_id:
|
||||||
|
raise ValueError("Connector profiles require id")
|
||||||
|
return profile_id
|
||||||
|
|
||||||
|
|
||||||
|
def _provider_from_mapping(value: Mapping[str, Any]) -> str:
|
||||||
|
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
|
||||||
|
if provider not in _SUPPORTED_PROVIDERS:
|
||||||
|
raise ValueError(f"Unsupported connector provider: {provider}")
|
||||||
|
return provider
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_from_mapping(value: Mapping[str, Any]) -> tuple[str, str | None]:
|
||||||
|
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||||
|
scope_id = _clean(value.get("scope_id"))
|
||||||
|
if scope_type == "system":
|
||||||
|
return scope_type, None
|
||||||
|
if not scope_id:
|
||||||
|
raise ValueError(f"{scope_type} connector profiles require scope_id")
|
||||||
|
return scope_type, scope_id
|
||||||
|
|
||||||
|
|
||||||
|
def _credentials_from_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||||
|
credentials = value.get("credentials")
|
||||||
|
return credentials if isinstance(credentials, Mapping) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_from_normalized_mapping(
|
||||||
|
value: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
profile_id: str,
|
||||||
|
provider: str,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
credential_mode: str,
|
||||||
|
credentials: Mapping[str, Any],
|
||||||
|
) -> ConnectorProfile:
|
||||||
|
return ConnectorProfile(
|
||||||
|
id=profile_id,
|
||||||
|
label=_clean(value.get("label") or value.get("name")) or profile_id,
|
||||||
|
provider=provider,
|
||||||
|
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
|
||||||
|
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
|
||||||
|
enabled=_bool(value.get("enabled"), default=True),
|
||||||
|
scope_type=scope_type,
|
||||||
|
scope_id=scope_id,
|
||||||
|
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
|
||||||
|
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
|
||||||
|
credential_mode=credential_mode,
|
||||||
|
username=_clean(value.get("username") or credentials.get("username")),
|
||||||
|
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
|
||||||
|
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
|
||||||
|
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
|
||||||
|
password_value=_clean(value.get("password") or credentials.get("password")),
|
||||||
|
token_value=_clean(value.get("token") or credentials.get("token")),
|
||||||
|
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
|
||||||
|
capabilities=tuple(_string_list(value.get("capabilities"))),
|
||||||
|
metadata=_public_metadata(value.get("metadata")),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _raw_profiles_payload(settings: object | None) -> object:
|
||||||
|
for key in _ENV_JSON_KEYS:
|
||||||
|
value = os.environ.get(key)
|
||||||
|
if value:
|
||||||
|
return value
|
||||||
|
for key in _ENV_FILE_KEYS:
|
||||||
|
path = os.environ.get(key)
|
||||||
|
if path:
|
||||||
|
with open(path, encoding="utf-8") as handle:
|
||||||
|
return handle.read()
|
||||||
|
if settings is not None:
|
||||||
|
value = getattr(settings, "files_connector_profiles_json", None)
|
||||||
|
if value:
|
||||||
|
return value
|
||||||
|
value = getattr(settings, "files_connector_profiles", None)
|
||||||
|
if value:
|
||||||
|
return value
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_sources_from_profile(value: Mapping[str, Any], profile: ConnectorProfile) -> list[ConnectorPolicySource]:
|
||||||
|
raw_sources: list[Mapping[str, Any]] = []
|
||||||
|
policy = value.get("policy")
|
||||||
|
if isinstance(policy, Mapping):
|
||||||
|
raw_sources.append({
|
||||||
|
"scope_type": profile.scope_type,
|
||||||
|
"scope_id": profile.scope_id,
|
||||||
|
"label": profile.label,
|
||||||
|
"policy": policy,
|
||||||
|
})
|
||||||
|
configured_sources = value.get("policy_sources") or value.get("connector_policy_sources")
|
||||||
|
if configured_sources is not None:
|
||||||
|
raw = connector_policy_sources_from_payload(configured_sources)
|
||||||
|
raw_sources.extend(_policy_source_response(source) for source in raw)
|
||||||
|
return connector_policy_sources_from_payload(raw_sources)
|
||||||
|
|
||||||
|
|
||||||
|
def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"scope_type": source.scope_type,
|
||||||
|
"scope_id": source.scope_id,
|
||||||
|
"label": source.label,
|
||||||
|
"policy": dict(source.policy or {}),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _public_metadata(value: object) -> Mapping[str, Any]:
|
||||||
|
if not isinstance(value, Mapping):
|
||||||
|
return {}
|
||||||
|
return {
|
||||||
|
str(key): item
|
||||||
|
for key, item in value.items()
|
||||||
|
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _response_metadata(value: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
str(key): item
|
||||||
|
for key, item in value.items()
|
||||||
|
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||||
|
and not str(key).strip().casefold().endswith("_env")
|
||||||
|
and str(key).strip().casefold() != "ca_bundle"
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _string_list(value: object) -> list[str]:
|
||||||
|
if value is None:
|
||||||
|
return []
|
||||||
|
if isinstance(value, str):
|
||||||
|
values: Iterable[object] = value.split(",")
|
||||||
|
elif isinstance(value, Iterable) and not isinstance(value, (bytes, bytearray, Mapping)):
|
||||||
|
values = value
|
||||||
|
else:
|
||||||
|
values = (value,)
|
||||||
|
return [str(item).strip() for item in values if str(item).strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _bool(value: object, *, default: bool) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return default
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return value
|
||||||
|
if isinstance(value, str):
|
||||||
|
return value.strip().casefold() not in {"0", "false", "no", "off"}
|
||||||
|
return bool(value)
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
@@ -0,0 +1,191 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from importlib.util import find_spec
|
||||||
|
|
||||||
|
|
||||||
|
CONNECTOR_PROVIDER_CAPABILITY = "files.connector.providers"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorProviderDescriptor:
|
||||||
|
provider: str
|
||||||
|
label: str
|
||||||
|
protocol: str
|
||||||
|
implemented: bool
|
||||||
|
browse_supported: bool
|
||||||
|
import_supported: bool
|
||||||
|
optional_dependency: str | None
|
||||||
|
permission_model: str
|
||||||
|
sync_strategy: str
|
||||||
|
conflict_strategy: str
|
||||||
|
preview_strategy: str
|
||||||
|
audit_events: tuple[str, ...]
|
||||||
|
notes: str | None = None
|
||||||
|
|
||||||
|
def to_response(self) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"provider": self.provider,
|
||||||
|
"label": self.label,
|
||||||
|
"protocol": self.protocol,
|
||||||
|
"implemented": self.implemented,
|
||||||
|
"installed": self.installed,
|
||||||
|
"browse_supported": self.browse_supported,
|
||||||
|
"import_supported": self.import_supported,
|
||||||
|
"write_supported": self.provider == "s3",
|
||||||
|
"optional_dependency": self.optional_dependency,
|
||||||
|
"permission_model": self.permission_model,
|
||||||
|
"sync_strategy": self.sync_strategy,
|
||||||
|
"conflict_strategy": self.conflict_strategy,
|
||||||
|
"preview_strategy": self.preview_strategy,
|
||||||
|
"audit_events": list(self.audit_events),
|
||||||
|
"notes": self.notes,
|
||||||
|
}
|
||||||
|
|
||||||
|
@property
|
||||||
|
def installed(self) -> bool:
|
||||||
|
if self.optional_dependency is None:
|
||||||
|
return self.implemented
|
||||||
|
return find_spec(self.optional_dependency) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def connector_provider_descriptors() -> tuple[ConnectorProviderDescriptor, ...]:
|
||||||
|
freeze_model = "Imported or synced connector files become managed GovOPlaN files with frozen blob/version provenance."
|
||||||
|
governed_permissions = "GovOPlaN profile scope and connector policy are enforced before browse/import/sync; remote ACLs are still evaluated by the upstream service."
|
||||||
|
return (
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="seafile",
|
||||||
|
label="Seafile",
|
||||||
|
protocol="seafile-api",
|
||||||
|
implemented=True,
|
||||||
|
browse_supported=True,
|
||||||
|
import_supported=True,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="On-demand browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||||
|
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||||
|
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Native account-token API for libraries, directories, file detail, and download-link import; WebDAV opt-in remains available.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="nextcloud",
|
||||||
|
label="Nextcloud",
|
||||||
|
protocol="webdav",
|
||||||
|
implemented=True,
|
||||||
|
browse_supported=True,
|
||||||
|
import_supported=True,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="On-demand WebDAV browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||||
|
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||||
|
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Uses the configured Nextcloud WebDAV endpoint, basic auth, or bearer token profile credentials.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="webdav",
|
||||||
|
label="WebDAV",
|
||||||
|
protocol="webdav",
|
||||||
|
implemented=True,
|
||||||
|
browse_supported=True,
|
||||||
|
import_supported=True,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="On-demand WebDAV browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||||
|
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||||
|
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Generic WebDAV provider for standards-compatible stores.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="smb",
|
||||||
|
label="SMB",
|
||||||
|
protocol="smb",
|
||||||
|
implemented=True,
|
||||||
|
browse_supported=True,
|
||||||
|
import_supported=True,
|
||||||
|
optional_dependency="smbprotocol",
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="On-demand browse/import/sync over administrator-declared shares; sync updates managed versions and never mutates the remote share.",
|
||||||
|
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after download.",
|
||||||
|
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the share.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Initial sessions, reconnects, aliases, and DFS referral targets use the Files-owned pinned smbprotocol transport and the deployment-wide private-network policy.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="s3",
|
||||||
|
label="S3-compatible object storage",
|
||||||
|
protocol="s3-api",
|
||||||
|
implemented=True,
|
||||||
|
browse_supported=True,
|
||||||
|
import_supported=True,
|
||||||
|
optional_dependency="boto3",
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="On-demand bucket/prefix browse and object import/sync; sync updates managed versions and never mutates the remote bucket.",
|
||||||
|
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after object download.",
|
||||||
|
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the bucket.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Every botocore connection pool uses the Files-owned pinned transport, including retries, redirects, endpoint discovery, and virtual-host aliases; outbound proxies and ambient credential discovery are disabled.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="sharepoint",
|
||||||
|
label="SharePoint",
|
||||||
|
protocol="microsoft-graph/sharepoint",
|
||||||
|
implemented=False,
|
||||||
|
browse_supported=False,
|
||||||
|
import_supported=False,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph or SharePoint credentials.",
|
||||||
|
conflict_strategy=freeze_model,
|
||||||
|
preview_strategy="Will preview from frozen managed file after import, not directly from SharePoint.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Provider/profile key is reserved; live browsing/import still needs Graph/SharePoint authentication and paging implementation.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="onedrive",
|
||||||
|
label="OneDrive",
|
||||||
|
protocol="microsoft-graph/onedrive",
|
||||||
|
implemented=False,
|
||||||
|
browse_supported=False,
|
||||||
|
import_supported=False,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph credentials.",
|
||||||
|
conflict_strategy=freeze_model,
|
||||||
|
preview_strategy="Will preview from frozen managed file after import, not directly from OneDrive.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Provider/profile key is reserved; live browsing/import still needs Graph drive selection, OAuth/app registration, and paging implementation.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="nfs",
|
||||||
|
label="NFS",
|
||||||
|
protocol="nfs",
|
||||||
|
implemented=False,
|
||||||
|
browse_supported=False,
|
||||||
|
import_supported=False,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model=governed_permissions,
|
||||||
|
sync_strategy="Planned optional provider over administrator-mounted paths or a sidecar service.",
|
||||||
|
conflict_strategy="Will use managed import conflict_strategy handling after download.",
|
||||||
|
preview_strategy="Will preview from frozen managed file after import, not directly from the mount.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Kept optional because NFS availability is deployment and host-kernel dependent.",
|
||||||
|
),
|
||||||
|
ConnectorProviderDescriptor(
|
||||||
|
provider="local",
|
||||||
|
label="Local filesystem",
|
||||||
|
protocol="file",
|
||||||
|
implemented=False,
|
||||||
|
browse_supported=False,
|
||||||
|
import_supported=False,
|
||||||
|
optional_dependency=None,
|
||||||
|
permission_model="Local connector access should be restricted to administrator-declared roots plus GovOPlaN profile and policy checks.",
|
||||||
|
sync_strategy="Planned optional provider; managed storage local backend already exists separately.",
|
||||||
|
conflict_strategy=freeze_model,
|
||||||
|
preview_strategy="Will preview from frozen managed file after import or sync.",
|
||||||
|
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||||
|
notes="Separate from the existing managed-file local storage backend.",
|
||||||
|
),
|
||||||
|
)
|
||||||
@@ -0,0 +1,264 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from sqlalchemy import false, or_
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_files.backend.db.models import FileConnectorSpace
|
||||||
|
from govoplan_files.backend.storage.access import ensure_owner_access, user_group_ids
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||||
|
from govoplan_files.backend.storage.connector_browse import normalize_connector_browse_path
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
|
||||||
|
|
||||||
|
SYNC_MODES = {"manual"}
|
||||||
|
WRITABLE_PROVIDERS = {"s3"}
|
||||||
|
|
||||||
|
|
||||||
|
def connector_space_owner_id(space: FileConnectorSpace) -> str:
|
||||||
|
return space.owner_user_id if space.owner_type == "user" else space.owner_group_id # type: ignore[return-value]
|
||||||
|
|
||||||
|
|
||||||
|
def create_connector_space(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
user_id: str,
|
||||||
|
label: str,
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
library_id: str | None = None,
|
||||||
|
remote_path: str | None = None,
|
||||||
|
sync_mode: str = "manual",
|
||||||
|
read_only: bool = True,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> FileConnectorSpace:
|
||||||
|
owner_type = owner_type.lower().strip()
|
||||||
|
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||||
|
label = _normalize_label(label)
|
||||||
|
sync_mode = _normalize_sync_mode(sync_mode)
|
||||||
|
validate_connector_space_write_mode(profile, read_only=read_only)
|
||||||
|
remote_path = normalize_connector_browse_path(remote_path)
|
||||||
|
library_id = _clean_optional(library_id)
|
||||||
|
|
||||||
|
existing = _owned_query(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id).filter(
|
||||||
|
FileConnectorSpace.label == label,
|
||||||
|
FileConnectorSpace.deleted_at.is_(None),
|
||||||
|
).first()
|
||||||
|
if existing is not None:
|
||||||
|
raise FileStorageError(f"Connector space already exists: {label}")
|
||||||
|
|
||||||
|
space = FileConnectorSpace(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_user_id=owner_id if owner_type == "user" else None,
|
||||||
|
owner_group_id=owner_id if owner_type == "group" else None,
|
||||||
|
label=label,
|
||||||
|
connector_profile_id=profile.id,
|
||||||
|
provider=profile.provider,
|
||||||
|
library_id=library_id,
|
||||||
|
remote_path=remote_path,
|
||||||
|
sync_mode=sync_mode,
|
||||||
|
read_only=read_only,
|
||||||
|
is_active=True,
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
metadata_=dict(metadata or {}),
|
||||||
|
)
|
||||||
|
session.add(space)
|
||||||
|
session.flush()
|
||||||
|
return space
|
||||||
|
|
||||||
|
|
||||||
|
def list_connector_spaces_for_user(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
include_inactive: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> list[FileConnectorSpace]:
|
||||||
|
query = session.query(FileConnectorSpace).filter(
|
||||||
|
FileConnectorSpace.tenant_id == tenant_id,
|
||||||
|
FileConnectorSpace.deleted_at.is_(None),
|
||||||
|
)
|
||||||
|
if not include_inactive:
|
||||||
|
query = query.filter(FileConnectorSpace.is_active.is_(True))
|
||||||
|
|
||||||
|
if owner_type:
|
||||||
|
if not owner_id:
|
||||||
|
raise FileStorageError("owner_id is required when owner_type is set")
|
||||||
|
owner_type = owner_type.lower().strip()
|
||||||
|
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||||
|
query = _owner_filter(query, owner_type, owner_id)
|
||||||
|
elif not is_admin:
|
||||||
|
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||||
|
query = query.filter(
|
||||||
|
or_(
|
||||||
|
FileConnectorSpace.owner_user_id == user_id,
|
||||||
|
FileConnectorSpace.owner_group_id.in_(group_ids) if group_ids else false(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return query.order_by(FileConnectorSpace.owner_type.asc(), FileConnectorSpace.label.asc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
def get_connector_space_for_user(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
space_id: str,
|
||||||
|
include_inactive: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> FileConnectorSpace:
|
||||||
|
query = session.query(FileConnectorSpace).filter(
|
||||||
|
FileConnectorSpace.id == space_id,
|
||||||
|
FileConnectorSpace.tenant_id == tenant_id,
|
||||||
|
FileConnectorSpace.deleted_at.is_(None),
|
||||||
|
)
|
||||||
|
if not include_inactive:
|
||||||
|
query = query.filter(FileConnectorSpace.is_active.is_(True))
|
||||||
|
space = query.one_or_none()
|
||||||
|
if space is None:
|
||||||
|
raise FileStorageError("Connector space not found")
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=space.owner_type,
|
||||||
|
owner_id=connector_space_owner_id(space),
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return space
|
||||||
|
|
||||||
|
|
||||||
|
def update_connector_space(
|
||||||
|
session: Session,
|
||||||
|
space: FileConnectorSpace,
|
||||||
|
*,
|
||||||
|
user_id: str,
|
||||||
|
label: str | None = None,
|
||||||
|
library_id: str | None = None,
|
||||||
|
remote_path: str | None = None,
|
||||||
|
sync_mode: str | None = None,
|
||||||
|
read_only: bool | None = None,
|
||||||
|
is_active: bool | None = None,
|
||||||
|
metadata: Mapping[str, Any] | None = None,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> FileConnectorSpace:
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=space.tenant_id,
|
||||||
|
owner_type=space.owner_type,
|
||||||
|
owner_id=connector_space_owner_id(space),
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
if label is not None:
|
||||||
|
new_label = _normalize_label(label)
|
||||||
|
existing = _owned_query(
|
||||||
|
session,
|
||||||
|
tenant_id=space.tenant_id,
|
||||||
|
owner_type=space.owner_type,
|
||||||
|
owner_id=connector_space_owner_id(space),
|
||||||
|
).filter(
|
||||||
|
FileConnectorSpace.id != space.id,
|
||||||
|
FileConnectorSpace.label == new_label,
|
||||||
|
FileConnectorSpace.deleted_at.is_(None),
|
||||||
|
).first()
|
||||||
|
if existing is not None:
|
||||||
|
raise FileStorageError(f"Connector space already exists: {new_label}")
|
||||||
|
space.label = new_label
|
||||||
|
if library_id is not None:
|
||||||
|
space.library_id = _clean_optional(library_id)
|
||||||
|
if remote_path is not None:
|
||||||
|
space.remote_path = normalize_connector_browse_path(remote_path)
|
||||||
|
if sync_mode is not None:
|
||||||
|
space.sync_mode = _normalize_sync_mode(sync_mode)
|
||||||
|
if read_only is not None:
|
||||||
|
space.read_only = bool(read_only)
|
||||||
|
if is_active is not None:
|
||||||
|
space.is_active = bool(is_active)
|
||||||
|
if metadata is not None:
|
||||||
|
space.metadata_ = dict(metadata)
|
||||||
|
session.add(space)
|
||||||
|
session.flush()
|
||||||
|
return space
|
||||||
|
|
||||||
|
|
||||||
|
def validate_connector_space_write_mode(
|
||||||
|
profile: ConnectorProfile, *, read_only: bool
|
||||||
|
) -> None:
|
||||||
|
if read_only:
|
||||||
|
return
|
||||||
|
if profile.provider not in WRITABLE_PROVIDERS or "write" not in profile.capabilities:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Two-way mode requires an S3 connection with the write capability enabled"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def soft_delete_connector_space(
|
||||||
|
session: Session,
|
||||||
|
space: FileConnectorSpace,
|
||||||
|
*,
|
||||||
|
user_id: str,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> FileConnectorSpace:
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=space.tenant_id,
|
||||||
|
owner_type=space.owner_type,
|
||||||
|
owner_id=connector_space_owner_id(space),
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
space.deleted_at = utcnow()
|
||||||
|
space.is_active = False
|
||||||
|
session.add(space)
|
||||||
|
session.flush()
|
||||||
|
return space
|
||||||
|
|
||||||
|
|
||||||
|
def _owned_query(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
||||||
|
query = session.query(FileConnectorSpace).filter(
|
||||||
|
FileConnectorSpace.tenant_id == tenant_id,
|
||||||
|
FileConnectorSpace.owner_type == owner_type,
|
||||||
|
)
|
||||||
|
return _owner_filter(query, owner_type, owner_id)
|
||||||
|
|
||||||
|
|
||||||
|
def _owner_filter(query, owner_type: str, owner_id: str):
|
||||||
|
if owner_type == "user":
|
||||||
|
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||||
|
if owner_type == "group":
|
||||||
|
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||||
|
raise FileStorageError("Files must be owned by a user or group")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_label(value: str) -> str:
|
||||||
|
label = value.strip()
|
||||||
|
if not label:
|
||||||
|
raise FileStorageError("Connector space label is required")
|
||||||
|
if len(label) > 255:
|
||||||
|
raise FileStorageError("Connector space label is too long")
|
||||||
|
return label
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_sync_mode(value: str) -> str:
|
||||||
|
mode = value.strip().casefold()
|
||||||
|
if mode not in SYNC_MODES:
|
||||||
|
raise FileStorageError(f"Unsupported connector space sync mode: {value}")
|
||||||
|
return mode
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_optional(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
@@ -0,0 +1,213 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable, Iterable
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_files.backend.storage.connector_deployment import (
|
||||||
|
connector_effective_endpoint_url,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profile_store import (
|
||||||
|
select_database_connector_profiles,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import (
|
||||||
|
ConnectorProfile,
|
||||||
|
connector_profiles_from_settings,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy import (
|
||||||
|
ConnectorAccessRequest,
|
||||||
|
connector_policy_decision,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_policy_store import (
|
||||||
|
effective_connector_policy_sources,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_providers import (
|
||||||
|
connector_provider_descriptors,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
CampaignVisibility = Callable[[str], bool]
|
||||||
|
|
||||||
|
|
||||||
|
def visible_connector_profiles_for_actor(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: Iterable[str],
|
||||||
|
settings: object | None,
|
||||||
|
provider: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
campaign_visible: CampaignVisibility | None = None,
|
||||||
|
include_disabled: bool = False,
|
||||||
|
include_admin_scopes: bool = False,
|
||||||
|
include_effective_policy: bool = True,
|
||||||
|
) -> list[ConnectorProfile]:
|
||||||
|
"""Return the same actor-filtered connector set used by API and docs surfaces."""
|
||||||
|
|
||||||
|
provider_norm = provider.strip().casefold() if provider else None
|
||||||
|
actor_group_ids = {str(group_id) for group_id in group_ids if str(group_id)}
|
||||||
|
database_profiles, database_profile_ids = select_database_connector_profiles(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
include_disabled=include_disabled,
|
||||||
|
row_visible=lambda row: (
|
||||||
|
include_admin_scopes
|
||||||
|
and row.scope_type in {"user", "group", "campaign"}
|
||||||
|
) or _scope_visible_to_actor(
|
||||||
|
scope_type=row.scope_type,
|
||||||
|
scope_id=row.scope_id,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=actor_group_ids,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
configured_profiles = connector_profiles_from_settings(settings)
|
||||||
|
profiles_by_id: dict[str, ConnectorProfile] = {}
|
||||||
|
for profile in database_profiles:
|
||||||
|
if profile.id not in profiles_by_id:
|
||||||
|
profiles_by_id[profile.id] = profile
|
||||||
|
for profile in configured_profiles:
|
||||||
|
if profile.id not in database_profile_ids and profile.id not in profiles_by_id:
|
||||||
|
profiles_by_id[profile.id] = profile
|
||||||
|
|
||||||
|
visible: list[ConnectorProfile] = []
|
||||||
|
for profile in profiles_by_id.values():
|
||||||
|
if not (profile.enabled or include_disabled):
|
||||||
|
continue
|
||||||
|
if provider_norm is not None and profile.provider != provider_norm:
|
||||||
|
continue
|
||||||
|
admin_scope_visible = include_admin_scopes and profile.scope_type in {
|
||||||
|
"user",
|
||||||
|
"group",
|
||||||
|
"campaign",
|
||||||
|
}
|
||||||
|
if not admin_scope_visible and not _profile_visible_to_actor(
|
||||||
|
profile,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=actor_group_ids,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
visible.append(
|
||||||
|
_with_effective_connector_policy(
|
||||||
|
session, tenant_id=tenant_id, profile=profile
|
||||||
|
)
|
||||||
|
if include_effective_policy
|
||||||
|
else profile
|
||||||
|
)
|
||||||
|
return visible
|
||||||
|
|
||||||
|
|
||||||
|
def connector_profile_usable_for_import(profile: ConnectorProfile) -> bool:
|
||||||
|
"""Whether a visible profile can safely offer the current live browse/import task."""
|
||||||
|
|
||||||
|
effective_endpoint_url = connector_effective_endpoint_url(
|
||||||
|
provider=profile.provider,
|
||||||
|
endpoint_url=profile.endpoint_url,
|
||||||
|
metadata=profile.metadata,
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
not profile.enabled
|
||||||
|
or not effective_endpoint_url
|
||||||
|
or not profile.credentials_configured
|
||||||
|
or bool(profile.secret_ref)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
descriptor = next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in connector_provider_descriptors()
|
||||||
|
if item.provider == profile.provider
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if descriptor is None:
|
||||||
|
return False
|
||||||
|
if not (
|
||||||
|
descriptor.implemented
|
||||||
|
and descriptor.installed
|
||||||
|
and descriptor.browse_supported
|
||||||
|
and descriptor.import_supported
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
# Prove that the initial root browse performed by the current Files UI is
|
||||||
|
# policy-allowed. A later selected remote path/item is checked again.
|
||||||
|
return connector_policy_decision(
|
||||||
|
ConnectorAccessRequest(
|
||||||
|
connector_id=profile.id,
|
||||||
|
credential_id=profile.credential_profile_id,
|
||||||
|
provider=profile.provider,
|
||||||
|
external_path="",
|
||||||
|
external_url=effective_endpoint_url,
|
||||||
|
operation="import",
|
||||||
|
),
|
||||||
|
profile.policy_sources,
|
||||||
|
).allowed
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_visible_to_actor(
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: set[str],
|
||||||
|
campaign_id: str | None,
|
||||||
|
campaign_visible: CampaignVisibility | None,
|
||||||
|
) -> bool:
|
||||||
|
return _scope_visible_to_actor(
|
||||||
|
scope_type=profile.scope_type,
|
||||||
|
scope_id=profile.scope_id,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
group_ids=group_ids,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
campaign_visible=campaign_visible,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _scope_visible_to_actor(
|
||||||
|
*,
|
||||||
|
scope_type: str,
|
||||||
|
scope_id: str | None,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
group_ids: set[str],
|
||||||
|
campaign_id: str | None,
|
||||||
|
campaign_visible: CampaignVisibility | None,
|
||||||
|
) -> bool:
|
||||||
|
if scope_type == "system":
|
||||||
|
return True
|
||||||
|
if scope_type == "tenant":
|
||||||
|
return scope_id == tenant_id
|
||||||
|
if scope_type == "user":
|
||||||
|
return scope_id == user_id
|
||||||
|
if scope_type == "group":
|
||||||
|
return bool(scope_id and scope_id in group_ids)
|
||||||
|
if scope_type != "campaign" or not scope_id:
|
||||||
|
return False
|
||||||
|
if campaign_id and scope_id != campaign_id:
|
||||||
|
return False
|
||||||
|
return bool(campaign_visible and campaign_visible(scope_id))
|
||||||
|
|
||||||
|
|
||||||
|
def _with_effective_connector_policy(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
) -> ConnectorProfile:
|
||||||
|
sources = effective_connector_policy_sources(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
scope_type=profile.scope_type,
|
||||||
|
scope_id=profile.scope_id,
|
||||||
|
)
|
||||||
|
if not sources:
|
||||||
|
return profile
|
||||||
|
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))
|
||||||
@@ -0,0 +1,438 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import hashlib
|
||||||
|
from typing import Any, Mapping
|
||||||
|
|
||||||
|
from govoplan_core.core.recovery import (
|
||||||
|
RecoveryGuaranteeError,
|
||||||
|
RecoveryMode,
|
||||||
|
RecoveryPlan,
|
||||||
|
RecoveryStatus,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.recovery_runtime import (
|
||||||
|
RecoveryOperationBusy,
|
||||||
|
RecoveryOperationStateConflict,
|
||||||
|
begin_durable_recovery_operation,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||||
|
from govoplan_core.db.session import get_database
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
from govoplan_files.backend.storage.connector_browse import (
|
||||||
|
ConnectorBrowseError,
|
||||||
|
ConnectorBrowseUnsupported,
|
||||||
|
_clean,
|
||||||
|
_s3_bucket,
|
||||||
|
_s3_client,
|
||||||
|
_s3_object_key,
|
||||||
|
normalize_connector_browse_path,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorWriteResult:
|
||||||
|
recovery_operation_id: str
|
||||||
|
status: str
|
||||||
|
replayed: bool
|
||||||
|
provider: str
|
||||||
|
remote_path: str
|
||||||
|
revision: str | None
|
||||||
|
checksum_sha256: str
|
||||||
|
size_bytes: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class _HeadProbe:
|
||||||
|
observed: Mapping[str, Any] | None
|
||||||
|
verified: bool
|
||||||
|
exception_type: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def write_connector_file(
|
||||||
|
profile: ConnectorProfile,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
library_id: str | None,
|
||||||
|
remote_path: str,
|
||||||
|
data: bytes,
|
||||||
|
content_type: str | None,
|
||||||
|
expected_revision: str | None,
|
||||||
|
idempotency_key: str,
|
||||||
|
) -> ConnectorWriteResult:
|
||||||
|
if profile.provider != "s3" or "write" not in profile.capabilities:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Remote writes require an S3 connection with the write capability enabled"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
normalized_path = normalize_connector_browse_path(remote_path)
|
||||||
|
except ConnectorBrowseError as exc:
|
||||||
|
raise FileStorageError(str(exc)) from exc
|
||||||
|
if not normalized_path:
|
||||||
|
raise FileStorageError("Remote write requires an object path")
|
||||||
|
try:
|
||||||
|
bucket = _s3_bucket(profile, library_id)
|
||||||
|
key = _s3_object_key(profile, normalized_path)
|
||||||
|
except ConnectorBrowseError as exc:
|
||||||
|
raise FileStorageError(str(exc)) from exc
|
||||||
|
if not bucket or not key:
|
||||||
|
raise FileStorageError("S3 remote write requires a bucket and object path")
|
||||||
|
checksum = hashlib.sha256(data).hexdigest()
|
||||||
|
target_digest = hashlib.sha256(f"{bucket}:{key}".encode("utf-8")).hexdigest()
|
||||||
|
request = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"connector_profile_id": profile.id,
|
||||||
|
"provider": profile.provider,
|
||||||
|
"remote_target_sha256": target_digest,
|
||||||
|
"content_sha256": checksum,
|
||||||
|
"size_bytes": len(data),
|
||||||
|
"expected_revision": expected_revision,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
started = begin_durable_recovery_operation(
|
||||||
|
get_database().SessionLocal,
|
||||||
|
identity=process_runtime_identity(),
|
||||||
|
module_id="files",
|
||||||
|
operation_type="connector-s3-write",
|
||||||
|
idempotency_key=f"files-connector-write:{idempotency_key}",
|
||||||
|
request=request,
|
||||||
|
recovery_plan=RecoveryPlan(
|
||||||
|
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||||
|
preconditions=(
|
||||||
|
"the connector space is explicitly configured for two-way writes",
|
||||||
|
"the connection and inherited policy allow the exact remote path",
|
||||||
|
"a conditional create or expected remote revision prevents blind overwrite",
|
||||||
|
),
|
||||||
|
forward_recovery_steps=(
|
||||||
|
"inspect provider metadata for the request and content digests",
|
||||||
|
"resolve success only for an exact digest match or retry after verified absence",
|
||||||
|
),
|
||||||
|
verification_steps=(
|
||||||
|
"read S3 object metadata after the write",
|
||||||
|
"match the recorded request and content digests without downloading content",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
precondition_evidence={
|
||||||
|
"remote_target_sha256": target_digest,
|
||||||
|
"content_sha256": checksum,
|
||||||
|
"size_bytes": len(data),
|
||||||
|
"conditional_write": True,
|
||||||
|
},
|
||||||
|
lease_resource_key=f"files:connector:{tenant_id}:{profile.id}:{target_digest[:40]}",
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_connector_object",
|
||||||
|
resource_id=target_digest,
|
||||||
|
metadata={
|
||||||
|
"resources": ["postgresql", "s3-connector"],
|
||||||
|
"provider": "s3",
|
||||||
|
"connector_profile_id": profile.id,
|
||||||
|
},
|
||||||
|
block_unresolved_resource=True,
|
||||||
|
)
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This remote object is owned by another write or unresolved recovery operation"
|
||||||
|
) from exc
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; the remote object was not changed"
|
||||||
|
) from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
return ConnectorWriteResult(
|
||||||
|
recovery_operation_id=started.operation_id,
|
||||||
|
status=started.status,
|
||||||
|
replayed=True,
|
||||||
|
provider="s3",
|
||||||
|
remote_path=normalized_path,
|
||||||
|
revision=None,
|
||||||
|
checksum_sha256=checksum,
|
||||||
|
size_bytes=len(data),
|
||||||
|
)
|
||||||
|
operation = started.operation
|
||||||
|
try:
|
||||||
|
client = _s3_client(profile)
|
||||||
|
except (ConnectorBrowseError, ConnectorBrowseUnsupported) as exc:
|
||||||
|
operation.reject(
|
||||||
|
summary="The S3 client was unavailable before the remote effect",
|
||||||
|
evidence=_verified(
|
||||||
|
{"external_effect_started": False}, exception_type=type(exc).__name__
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError(str(exc)) from exc
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
before = _head_object(client, bucket=bucket, key=key)
|
||||||
|
except Exception as exc:
|
||||||
|
operation.reject(
|
||||||
|
summary="Remote preconditions could not be inspected before the effect",
|
||||||
|
evidence=_verified(
|
||||||
|
{"external_effect_started": False},
|
||||||
|
exception_type=type(exc).__name__,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError("S3 remote revision lookup failed") from exc
|
||||||
|
expected = _normalize_revision(expected_revision)
|
||||||
|
if before is not None and expected is None:
|
||||||
|
operation.reject(
|
||||||
|
summary="A blind remote overwrite was rejected",
|
||||||
|
evidence=_verified(
|
||||||
|
{"remote_object_exists": True, "expected_revision_supplied": False}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError(
|
||||||
|
"The remote object already exists; reload it and supply its expected revision"
|
||||||
|
)
|
||||||
|
if expected is not None and (
|
||||||
|
before is None or expected not in _observed_revisions(before)
|
||||||
|
):
|
||||||
|
operation.reject(
|
||||||
|
summary="The expected remote revision did not match",
|
||||||
|
evidence=_verified(
|
||||||
|
{
|
||||||
|
"remote_object_exists": before is not None,
|
||||||
|
"expected_revision_matches": False,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError("The remote object changed; reload before writing")
|
||||||
|
params: dict[str, object] = {
|
||||||
|
"Bucket": bucket,
|
||||||
|
"Key": key,
|
||||||
|
"Body": data,
|
||||||
|
"Metadata": {
|
||||||
|
"govoplan-sha256": checksum,
|
||||||
|
"govoplan-request-id": started.operation_id,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if content_type:
|
||||||
|
params["ContentType"] = content_type
|
||||||
|
if before is None:
|
||||||
|
params["IfNoneMatch"] = "*"
|
||||||
|
else:
|
||||||
|
params["IfMatch"] = str(before.get("ETag") or expected_revision or "")
|
||||||
|
try:
|
||||||
|
client.put_object(**params)
|
||||||
|
except Exception as exc:
|
||||||
|
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||||
|
if probe.verified and _matches_effect(
|
||||||
|
probe.observed,
|
||||||
|
checksum=checksum,
|
||||||
|
operation_id=started.operation_id,
|
||||||
|
):
|
||||||
|
operation.succeed(
|
||||||
|
evidence=_success_evidence(
|
||||||
|
probe.observed,
|
||||||
|
checksum=checksum,
|
||||||
|
operation_id=started.operation_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return _result(
|
||||||
|
started.operation_id,
|
||||||
|
normalized_path,
|
||||||
|
checksum,
|
||||||
|
len(data),
|
||||||
|
probe.observed,
|
||||||
|
)
|
||||||
|
operation.unresolved(
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||||
|
summary="The S3 write outcome requires reconciliation",
|
||||||
|
evidence={
|
||||||
|
"remote_target_sha256": target_digest,
|
||||||
|
"observed": _public_observation(probe),
|
||||||
|
"exception_type": type(exc).__name__,
|
||||||
|
},
|
||||||
|
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||||
|
)
|
||||||
|
return ConnectorWriteResult(
|
||||||
|
recovery_operation_id=started.operation_id,
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||||
|
replayed=False,
|
||||||
|
provider="s3",
|
||||||
|
remote_path=normalized_path,
|
||||||
|
revision=_revision(probe.observed),
|
||||||
|
checksum_sha256=checksum,
|
||||||
|
size_bytes=len(data),
|
||||||
|
)
|
||||||
|
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||||
|
if not probe.verified:
|
||||||
|
operation.unresolved(
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||||
|
summary="The S3 write returned but provider evidence could not be queried",
|
||||||
|
evidence={
|
||||||
|
"remote_target_sha256": target_digest,
|
||||||
|
"observed": _public_observation(probe),
|
||||||
|
},
|
||||||
|
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||||
|
)
|
||||||
|
return ConnectorWriteResult(
|
||||||
|
recovery_operation_id=started.operation_id,
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||||
|
replayed=False,
|
||||||
|
provider="s3",
|
||||||
|
remote_path=normalized_path,
|
||||||
|
revision=None,
|
||||||
|
checksum_sha256=checksum,
|
||||||
|
size_bytes=len(data),
|
||||||
|
)
|
||||||
|
observed = probe.observed
|
||||||
|
if not _matches_effect(
|
||||||
|
observed, checksum=checksum, operation_id=started.operation_id
|
||||||
|
):
|
||||||
|
operation.unresolved(
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||||
|
summary="The S3 write returned but exact provider evidence did not match",
|
||||||
|
evidence={
|
||||||
|
"remote_target_sha256": target_digest,
|
||||||
|
"observed": _public_observation(probe),
|
||||||
|
},
|
||||||
|
failure_summary="Reconcile the remote object before another write",
|
||||||
|
)
|
||||||
|
return ConnectorWriteResult(
|
||||||
|
recovery_operation_id=started.operation_id,
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED.value,
|
||||||
|
replayed=False,
|
||||||
|
provider="s3",
|
||||||
|
remote_path=normalized_path,
|
||||||
|
revision=_revision(observed),
|
||||||
|
checksum_sha256=checksum,
|
||||||
|
size_bytes=len(data),
|
||||||
|
)
|
||||||
|
operation.succeed(
|
||||||
|
evidence=_success_evidence(
|
||||||
|
observed, checksum=checksum, operation_id=started.operation_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return _result(
|
||||||
|
started.operation_id,
|
||||||
|
normalized_path,
|
||||||
|
checksum,
|
||||||
|
len(data),
|
||||||
|
observed,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
close = getattr(client, "close", None)
|
||||||
|
if callable(close):
|
||||||
|
close()
|
||||||
|
|
||||||
|
|
||||||
|
def _head_object(client: Any, *, bucket: str, key: str) -> Mapping[str, Any] | None:
|
||||||
|
try:
|
||||||
|
response = client.head_object(Bucket=bucket, Key=key)
|
||||||
|
except Exception as exc:
|
||||||
|
if _is_not_found(exc):
|
||||||
|
return None
|
||||||
|
raise
|
||||||
|
if not isinstance(response, Mapping):
|
||||||
|
raise FileStorageError("S3 connector returned invalid object metadata")
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_head_object(client: Any, *, bucket: str, key: str) -> _HeadProbe:
|
||||||
|
try:
|
||||||
|
return _HeadProbe(
|
||||||
|
observed=_head_object(client, bucket=bucket, key=key), verified=True
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
return _HeadProbe(
|
||||||
|
observed=None,
|
||||||
|
verified=False,
|
||||||
|
exception_type=type(exc).__name__,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_not_found(exc: Exception) -> bool:
|
||||||
|
response = getattr(exc, "response", None)
|
||||||
|
if not isinstance(response, Mapping):
|
||||||
|
return False
|
||||||
|
error = response.get("Error")
|
||||||
|
code = error.get("Code") if isinstance(error, Mapping) else None
|
||||||
|
status = response.get("ResponseMetadata")
|
||||||
|
http_status = status.get("HTTPStatusCode") if isinstance(status, Mapping) else None
|
||||||
|
return str(code).casefold() in {"404", "nosuchkey", "notfound"} or http_status == 404
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata(observed: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||||
|
value = observed.get("Metadata") if observed else None
|
||||||
|
return value if isinstance(value, Mapping) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _matches_effect(
|
||||||
|
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||||
|
) -> bool:
|
||||||
|
metadata = _metadata(observed)
|
||||||
|
return bool(
|
||||||
|
observed is not None
|
||||||
|
and _clean(metadata.get("govoplan-sha256")) == checksum
|
||||||
|
and _clean(metadata.get("govoplan-request-id")) == operation_id
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _revision(observed: Mapping[str, Any] | None) -> str | None:
|
||||||
|
if observed is None:
|
||||||
|
return None
|
||||||
|
return _clean(observed.get("VersionId") or observed.get("ETag"))
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_revision(value: str | None) -> str | None:
|
||||||
|
cleaned = _clean(value)
|
||||||
|
return cleaned.strip('"') if cleaned else None
|
||||||
|
|
||||||
|
|
||||||
|
def _observed_revisions(observed: Mapping[str, Any]) -> set[str]:
|
||||||
|
return {
|
||||||
|
normalized
|
||||||
|
for value in (observed.get("VersionId"), observed.get("ETag"))
|
||||||
|
if (normalized := _normalize_revision(_clean(value))) is not None
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _success_evidence(
|
||||||
|
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||||
|
) -> dict[str, object]:
|
||||||
|
return _verified(
|
||||||
|
{
|
||||||
|
"remote_object_present": observed is not None,
|
||||||
|
"content_digest_matches": _clean(_metadata(observed).get("govoplan-sha256")) == checksum,
|
||||||
|
"request_marker_matches": _clean(_metadata(observed).get("govoplan-request-id")) == operation_id,
|
||||||
|
},
|
||||||
|
revision=_revision(observed),
|
||||||
|
content_sha256=checksum,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _public_observation(probe: _HeadProbe) -> dict[str, object]:
|
||||||
|
observed = probe.observed
|
||||||
|
return {
|
||||||
|
"probe_verified": probe.verified,
|
||||||
|
"probe_exception_type": probe.exception_type,
|
||||||
|
"present": observed is not None,
|
||||||
|
"revision": _revision(observed),
|
||||||
|
"has_content_digest": bool(_clean(_metadata(observed).get("govoplan-sha256"))),
|
||||||
|
"has_request_marker": bool(_clean(_metadata(observed).get("govoplan-request-id"))),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _verified(checks: dict[str, object], **details: object) -> dict[str, object]:
|
||||||
|
return {"verified": True, "checks": checks, **details}
|
||||||
|
|
||||||
|
|
||||||
|
def _result(
|
||||||
|
operation_id: str,
|
||||||
|
remote_path: str,
|
||||||
|
checksum: str,
|
||||||
|
size_bytes: int,
|
||||||
|
observed: Mapping[str, Any] | None,
|
||||||
|
) -> ConnectorWriteResult:
|
||||||
|
return ConnectorWriteResult(
|
||||||
|
recovery_operation_id=operation_id,
|
||||||
|
status=RecoveryStatus.SUCCEEDED.value,
|
||||||
|
replayed=False,
|
||||||
|
provider="s3",
|
||||||
|
remote_path=remote_path,
|
||||||
|
revision=_revision(observed),
|
||||||
|
checksum_sha256=checksum,
|
||||||
|
size_bytes=size_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["ConnectorWriteResult", "write_connector_file"]
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from govoplan_core.core.encryption import (
|
||||||
|
ContentProtectionRequest,
|
||||||
|
ContentUnprotectionRequest,
|
||||||
|
ProtectedContent,
|
||||||
|
encryption_content_cipher,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.runtime import get_registry
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
|
||||||
|
|
||||||
|
FILES_PROTECTION_PROFILE = "files-server-envelope-v1"
|
||||||
|
|
||||||
|
|
||||||
|
def protect_blob_content(
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
blob_id: str,
|
||||||
|
vault_id: str,
|
||||||
|
ciphertext_ref: str,
|
||||||
|
plaintext: bytes,
|
||||||
|
actor_id: str,
|
||||||
|
content_type: str | None,
|
||||||
|
) -> ProtectedContent:
|
||||||
|
capability = encryption_content_cipher(get_registry())
|
||||||
|
if capability is None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"File encryption was requested, but the Encryption module is unavailable."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
return capability.protect_content(
|
||||||
|
session,
|
||||||
|
request=ContentProtectionRequest(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_module="files",
|
||||||
|
resource_type="file_blob",
|
||||||
|
resource_id=blob_id,
|
||||||
|
profile_id=FILES_PROTECTION_PROFILE,
|
||||||
|
vault_id=vault_id,
|
||||||
|
ciphertext_ref=ciphertext_ref,
|
||||||
|
plaintext=plaintext,
|
||||||
|
policy_decision_ref="files:explicit-vault-selection:v1",
|
||||||
|
idempotency_key=f"file-blob:{blob_id}:content:v1",
|
||||||
|
actor_id=actor_id,
|
||||||
|
metadata={
|
||||||
|
"content_type": content_type or "application/octet-stream",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Managed file content could not be protected by the configured vault."
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def unprotect_blob_content(
|
||||||
|
session: object,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
blob_id: str,
|
||||||
|
envelope_id: str,
|
||||||
|
ciphertext: bytes,
|
||||||
|
) -> bytes:
|
||||||
|
capability = encryption_content_cipher(get_registry())
|
||||||
|
if capability is None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This file is encrypted and cannot be read while Encryption is unavailable."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
return capability.unprotect_content(
|
||||||
|
session,
|
||||||
|
request=ContentUnprotectionRequest(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_module="files",
|
||||||
|
resource_type="file_blob",
|
||||||
|
resource_id=blob_id,
|
||||||
|
envelope_id=envelope_id,
|
||||||
|
ciphertext=ciphertext,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Managed file content could not be opened with its protection envelope."
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"FILES_PROTECTION_PROFILE",
|
||||||
|
"protect_blob_content",
|
||||||
|
"unprotect_blob_content",
|
||||||
|
]
|
||||||
@@ -2,30 +2,47 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import mimetypes
|
import mimetypes
|
||||||
|
from datetime import datetime
|
||||||
from pathlib import PurePosixPath
|
from pathlib import PurePosixPath
|
||||||
from typing import Any, Iterable
|
from typing import Any, Iterable
|
||||||
from uuid import uuid4
|
from uuid import uuid4
|
||||||
|
|
||||||
from sqlalchemy import or_
|
from sqlalchemy import and_, exists, func, or_
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_core.db.models import Group, Tenant, User
|
from govoplan_core.core.campaigns import CAPABILITY_CAMPAIGNS_ACCESS, CampaignAccessProvider
|
||||||
from govoplan_core.core.optional import reraise_unless_missing_package
|
|
||||||
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileShare, FileVersion
|
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileShare, FileVersion
|
||||||
from govoplan_files.backend.runtime import settings
|
from govoplan_files.backend.runtime import get_registry, settings
|
||||||
from govoplan_files.backend.storage.access import ensure_owner_access, user_group_ids
|
from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids
|
||||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
StorageBackendError,
|
||||||
|
StorageObjectMissing,
|
||||||
|
get_storage_backend,
|
||||||
|
)
|
||||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
||||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component
|
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
|
||||||
|
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||||
|
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||||
|
from govoplan_files.backend.storage.integrity import (
|
||||||
|
QUARANTINED_BLOB_STATUSES,
|
||||||
|
read_verified_blob_bytes,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||||
|
|
||||||
|
|
||||||
def _campaign_model():
|
def _campaign_access_provider() -> CampaignAccessProvider:
|
||||||
try:
|
registry = get_registry()
|
||||||
from govoplan_campaign.backend.db.models import Campaign
|
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||||
except ModuleNotFoundError as exc:
|
raise FileStorageError("Campaign module is not installed")
|
||||||
reraise_unless_missing_package(exc, "govoplan_campaign")
|
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||||
raise FileStorageError("Campaign module is not installed") from exc
|
if not isinstance(capability, CampaignAccessProvider):
|
||||||
return Campaign
|
raise FileStorageError("Campaign access capability is invalid")
|
||||||
|
return capability
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_campaign_exists(session: Session, *, tenant_id: str, campaign_id: str) -> None:
|
||||||
|
if not _campaign_access_provider().campaign_exists(session, tenant_id=tenant_id, campaign_id=campaign_id):
|
||||||
|
raise FileStorageError("Campaign not found")
|
||||||
|
|
||||||
|
|
||||||
def _asset_query_for_owner(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
def _asset_query_for_owner(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
||||||
@@ -45,8 +62,9 @@ def _storage_backend_name() -> str:
|
|||||||
return settings.file_storage_backend.lower().strip()
|
return settings.file_storage_backend.lower().strip()
|
||||||
|
|
||||||
|
|
||||||
def _storage_key(*, tenant_id: str, checksum: str, filename: str) -> str:
|
def _storage_key(*, tenant_id: str, checksum: str) -> str:
|
||||||
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}-{safe_storage_component(filename)}"
|
# Object locators remain opaque so recovery evidence never persists names.
|
||||||
|
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}.blob"
|
||||||
|
|
||||||
|
|
||||||
def _get_or_create_blob(
|
def _get_or_create_blob(
|
||||||
@@ -56,40 +74,144 @@ def _get_or_create_blob(
|
|||||||
data: bytes,
|
data: bytes,
|
||||||
filename: str,
|
filename: str,
|
||||||
content_type: str | None,
|
content_type: str | None,
|
||||||
|
actor_id: str,
|
||||||
|
encryption_vault_id: str | None = None,
|
||||||
) -> FileBlob:
|
) -> FileBlob:
|
||||||
checksum = hashlib.sha256(data).hexdigest()
|
checksum = hashlib.sha256(data).hexdigest()
|
||||||
size = len(data)
|
size = len(data)
|
||||||
|
vault_id = str(encryption_vault_id or "").strip() or None
|
||||||
|
protection_discriminator = f"vault:{vault_id}" if vault_id else "plaintext"
|
||||||
blob = (
|
blob = (
|
||||||
session.query(FileBlob)
|
session.query(FileBlob)
|
||||||
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size)
|
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size, FileBlob.protection_discriminator == protection_discriminator)
|
||||||
|
.with_for_update()
|
||||||
.one_or_none()
|
.one_or_none()
|
||||||
)
|
)
|
||||||
if blob:
|
if blob:
|
||||||
backend = get_storage_backend()
|
backend = get_storage_backend()
|
||||||
if not backend.exists(blob.storage_key):
|
repair_required = (
|
||||||
|
blob.integrity_status in QUARANTINED_BLOB_STATUSES
|
||||||
|
or blob.quarantined_at is not None
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
backend.stat(blob.storage_key)
|
||||||
|
except StorageObjectMissing:
|
||||||
|
repair_required = True
|
||||||
|
except StorageBackendError as exc:
|
||||||
|
raise FileStorageError(str(exc)) from exc
|
||||||
|
if repair_required:
|
||||||
|
repair_token = hashlib.sha256(
|
||||||
|
repr(
|
||||||
|
(
|
||||||
|
blob.integrity_status,
|
||||||
|
blob.integrity_checked_at,
|
||||||
|
blob.quarantined_at,
|
||||||
|
blob.storage_checksum_sha256,
|
||||||
|
)
|
||||||
|
).encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
recovery = begin_blob_write_recovery(
|
||||||
|
session,
|
||||||
|
backend=backend,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob.id,
|
||||||
|
storage_key=blob.storage_key,
|
||||||
|
semantic_checksum_sha256=checksum,
|
||||||
|
semantic_size_bytes=size,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
created_new=False,
|
||||||
|
repair_token=repair_token,
|
||||||
|
)
|
||||||
|
stored_data = data
|
||||||
|
expected_envelope_id = blob.encryption_envelope_id
|
||||||
|
if vault_id:
|
||||||
|
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||||
|
|
||||||
|
protected = protect_blob_content(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob.id,
|
||||||
|
vault_id=vault_id,
|
||||||
|
ciphertext_ref=blob.storage_key,
|
||||||
|
plaintext=data,
|
||||||
|
actor_id=actor_id,
|
||||||
|
content_type=content_type,
|
||||||
|
)
|
||||||
|
if blob.encryption_envelope_id not in {None, protected.envelope.envelope_id}:
|
||||||
|
raise FileStorageError("The existing encrypted blob has another protection envelope.")
|
||||||
|
stored_data = protected.ciphertext
|
||||||
|
blob.encryption_envelope_id = protected.envelope.envelope_id
|
||||||
|
expected_envelope_id = protected.envelope.envelope_id
|
||||||
|
blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest()
|
||||||
|
blob.storage_size_bytes = len(stored_data)
|
||||||
|
recovery.prepare_stored_bytes(
|
||||||
|
stored_data,
|
||||||
|
envelope_id=expected_envelope_id,
|
||||||
|
)
|
||||||
try:
|
try:
|
||||||
backend.put_bytes(blob.storage_key, data, content_type=content_type)
|
backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||||
except StorageBackendError as exc:
|
except StorageBackendError as exc:
|
||||||
raise FileStorageError(str(exc)) from exc
|
raise FileStorageError(str(exc)) from exc
|
||||||
|
blob.integrity_status = "verified"
|
||||||
|
blob.integrity_checked_at = utcnow()
|
||||||
|
blob.integrity_failure = None
|
||||||
|
blob.quarantined_at = None
|
||||||
blob.ref_count += 1
|
blob.ref_count += 1
|
||||||
session.add(blob)
|
session.add(blob)
|
||||||
return blob
|
return blob
|
||||||
|
|
||||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum, filename=filename)
|
blob_id = str(uuid4())
|
||||||
|
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
|
||||||
backend = get_storage_backend()
|
backend = get_storage_backend()
|
||||||
|
recovery = begin_blob_write_recovery(
|
||||||
|
session,
|
||||||
|
backend=backend,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob_id,
|
||||||
|
storage_key=storage_key,
|
||||||
|
semantic_checksum_sha256=checksum,
|
||||||
|
semantic_size_bytes=size,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
created_new=True,
|
||||||
|
)
|
||||||
|
stored_data = data
|
||||||
|
envelope_id = None
|
||||||
|
if vault_id:
|
||||||
|
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||||
|
|
||||||
|
protected = protect_blob_content(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob_id,
|
||||||
|
vault_id=vault_id,
|
||||||
|
ciphertext_ref=storage_key,
|
||||||
|
plaintext=data,
|
||||||
|
actor_id=actor_id,
|
||||||
|
content_type=content_type,
|
||||||
|
)
|
||||||
|
stored_data = protected.ciphertext
|
||||||
|
envelope_id = protected.envelope.envelope_id
|
||||||
|
recovery.prepare_stored_bytes(stored_data, envelope_id=envelope_id)
|
||||||
try:
|
try:
|
||||||
backend.put_bytes(storage_key, data, content_type=content_type)
|
backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||||
except StorageBackendError as exc:
|
except StorageBackendError as exc:
|
||||||
raise FileStorageError(str(exc)) from exc
|
raise FileStorageError(str(exc)) from exc
|
||||||
blob = FileBlob(
|
blob = FileBlob(
|
||||||
|
id=blob_id,
|
||||||
tenant_id=tenant_id,
|
tenant_id=tenant_id,
|
||||||
storage_backend=_storage_backend_name(),
|
storage_backend=_storage_backend_name(),
|
||||||
storage_bucket=_storage_bucket_name(),
|
storage_bucket=_storage_bucket_name(),
|
||||||
storage_key=storage_key,
|
storage_key=storage_key,
|
||||||
checksum_sha256=checksum,
|
checksum_sha256=checksum,
|
||||||
size_bytes=size,
|
size_bytes=size,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
encryption_envelope_id=envelope_id,
|
||||||
|
storage_checksum_sha256=hashlib.sha256(stored_data).hexdigest() if vault_id else None,
|
||||||
|
storage_size_bytes=len(stored_data) if vault_id else None,
|
||||||
content_type=content_type,
|
content_type=content_type,
|
||||||
ref_count=1,
|
ref_count=1,
|
||||||
|
integrity_status="verified",
|
||||||
|
integrity_checked_at=utcnow(),
|
||||||
)
|
)
|
||||||
session.add(blob)
|
session.add(blob)
|
||||||
session.flush()
|
session.flush()
|
||||||
@@ -114,6 +236,7 @@ def create_file_asset(
|
|||||||
conflict_strategy: str = "reject",
|
conflict_strategy: str = "reject",
|
||||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||||
is_admin: bool = False,
|
is_admin: bool = False,
|
||||||
|
encryption_vault_id: str | None = None,
|
||||||
) -> UploadedStoredFile:
|
) -> UploadedStoredFile:
|
||||||
owner_type = owner_type.lower().strip()
|
owner_type = owner_type.lower().strip()
|
||||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||||
@@ -141,7 +264,7 @@ def create_file_asset(
|
|||||||
elif action == "rename":
|
elif action == "rename":
|
||||||
logical_path = _next_available_logical_path(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, desired_path=logical_path)
|
logical_path = _next_available_logical_path(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, desired_path=logical_path)
|
||||||
|
|
||||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type)
|
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=encryption_vault_id)
|
||||||
asset = FileAsset(
|
asset = FileAsset(
|
||||||
tenant_id=tenant_id,
|
tenant_id=tenant_id,
|
||||||
owner_type=owner_type,
|
owner_type=owner_type,
|
||||||
@@ -176,6 +299,150 @@ def create_file_asset(
|
|||||||
return UploadedStoredFile(asset=asset, version=version, blob=blob)
|
return UploadedStoredFile(asset=asset, version=version, blob=blob)
|
||||||
|
|
||||||
|
|
||||||
|
def sync_file_asset_from_source(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
user_id: str,
|
||||||
|
filename: str,
|
||||||
|
data: bytes,
|
||||||
|
metadata: dict[str, Any],
|
||||||
|
folder: str | None = None,
|
||||||
|
display_path: str | None = None,
|
||||||
|
content_type: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
conflict_strategy: str = "rename",
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> tuple[UploadedStoredFile, str, str | None]:
|
||||||
|
owner_type = owner_type.lower().strip()
|
||||||
|
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||||
|
provenance = source_provenance_from_metadata(metadata)
|
||||||
|
if not provenance:
|
||||||
|
raise FileStorageError("Connector sync requires source provenance")
|
||||||
|
existing = find_asset_by_source(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
source_provenance=provenance,
|
||||||
|
)
|
||||||
|
if existing is None:
|
||||||
|
stored = create_file_asset(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
user_id=user_id,
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
folder=folder,
|
||||||
|
display_path=display_path,
|
||||||
|
content_type=content_type,
|
||||||
|
metadata=metadata,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
conflict_strategy=conflict_strategy,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return stored, "created", None
|
||||||
|
|
||||||
|
previous_version_id = existing.current_version_id
|
||||||
|
stored, action = update_file_asset_content(
|
||||||
|
session,
|
||||||
|
existing,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
filename=filename,
|
||||||
|
data=data,
|
||||||
|
content_type=content_type,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
if campaign_id:
|
||||||
|
share_file(session, tenant_id=tenant_id, asset=existing, target_type="campaign", target_id=campaign_id, permission="read", user_id=user_id)
|
||||||
|
return stored, action, previous_version_id
|
||||||
|
|
||||||
|
|
||||||
|
def find_asset_by_source(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
source_provenance: dict[str, Any],
|
||||||
|
) -> FileAsset | None:
|
||||||
|
wanted = _source_identity(source_provenance)
|
||||||
|
if wanted is None:
|
||||||
|
return None
|
||||||
|
assets = (
|
||||||
|
_asset_query_for_owner(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id)
|
||||||
|
.filter(FileAsset.deleted_at.is_(None))
|
||||||
|
.order_by(FileAsset.updated_at.desc())
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
for asset in assets:
|
||||||
|
if _source_identity(source_provenance_from_metadata(asset.metadata_ or {})) == wanted:
|
||||||
|
return asset
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def update_file_asset_content(
|
||||||
|
session: Session,
|
||||||
|
asset: FileAsset,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
filename: str,
|
||||||
|
data: bytes,
|
||||||
|
content_type: str | None,
|
||||||
|
metadata: dict[str, Any],
|
||||||
|
encryption_vault_id: str | None = None,
|
||||||
|
) -> tuple[UploadedStoredFile, str]:
|
||||||
|
if asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||||
|
raise FileStorageError("File not found")
|
||||||
|
safe_filename = filename_from_path(normalize_logical_path(filename, fallback_filename="file"))
|
||||||
|
if not content_type:
|
||||||
|
content_type = mimetypes.guess_type(safe_filename)[0] or "application/octet-stream"
|
||||||
|
current_version, current_blob = current_version_and_blob(session, asset)
|
||||||
|
checksum = hashlib.sha256(data).hexdigest()
|
||||||
|
asset.metadata_ = metadata
|
||||||
|
session.add(asset)
|
||||||
|
inherited_vault_id = encryption_vault_id
|
||||||
|
if inherited_vault_id is None and current_blob.encryption_envelope_id:
|
||||||
|
prefix = "vault:"
|
||||||
|
if current_blob.protection_discriminator.startswith(prefix):
|
||||||
|
inherited_vault_id = current_blob.protection_discriminator[len(prefix) :]
|
||||||
|
inherited_vault_id = str(inherited_vault_id or "").strip() or None
|
||||||
|
target_protection = (
|
||||||
|
f"vault:{inherited_vault_id}" if inherited_vault_id else "plaintext"
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
current_blob.checksum_sha256 == checksum
|
||||||
|
and current_blob.size_bytes == len(data)
|
||||||
|
and current_blob.protection_discriminator == target_protection
|
||||||
|
):
|
||||||
|
return UploadedStoredFile(asset=asset, version=current_version, blob=current_blob), "unchanged"
|
||||||
|
|
||||||
|
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=inherited_vault_id)
|
||||||
|
version = FileVersion(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
file_asset_id=asset.id,
|
||||||
|
blob_id=blob.id,
|
||||||
|
version_number=_next_version_number(session, asset.id),
|
||||||
|
filename_at_upload=safe_filename,
|
||||||
|
display_path_at_upload=asset.display_path,
|
||||||
|
content_type=content_type,
|
||||||
|
size_bytes=blob.size_bytes,
|
||||||
|
checksum_sha256=blob.checksum_sha256,
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
session.add(version)
|
||||||
|
session.flush()
|
||||||
|
asset.current_version_id = version.id
|
||||||
|
session.add(asset)
|
||||||
|
return UploadedStoredFile(asset=asset, version=version, blob=blob), "updated"
|
||||||
|
|
||||||
|
|
||||||
def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_id: str, require_write: bool = False, is_admin: bool = False) -> FileAsset:
|
def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_id: str, require_write: bool = False, is_admin: bool = False) -> FileAsset:
|
||||||
asset = session.get(FileAsset, asset_id)
|
asset = session.get(FileAsset, asset_id)
|
||||||
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||||
@@ -192,7 +459,7 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
|
|||||||
.filter(
|
.filter(
|
||||||
FileShare.tenant_id == tenant_id,
|
FileShare.tenant_id == tenant_id,
|
||||||
FileShare.file_asset_id == asset.id,
|
FileShare.file_asset_id == asset.id,
|
||||||
FileShare.revoked_at.is_(None),
|
effective_file_share_clause(),
|
||||||
FileShare.permission.in_(permission_values),
|
FileShare.permission.in_(permission_values),
|
||||||
or_(
|
or_(
|
||||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||||
@@ -207,6 +474,32 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
|
|||||||
return asset
|
return asset
|
||||||
|
|
||||||
|
|
||||||
|
def get_asset_for_share_management(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
asset_id: str,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> FileAsset:
|
||||||
|
asset = session.get(FileAsset, asset_id)
|
||||||
|
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||||
|
raise FileStorageError("File not found")
|
||||||
|
if is_admin:
|
||||||
|
return asset
|
||||||
|
owns_asset = (
|
||||||
|
asset.owner_type == "user"
|
||||||
|
and asset.owner_user_id == user_id
|
||||||
|
) or (
|
||||||
|
asset.owner_type == "group"
|
||||||
|
and asset.owner_group_id
|
||||||
|
in user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||||
|
)
|
||||||
|
if not owns_asset:
|
||||||
|
raise FileStorageError("Only file owners and administrators can manage shares")
|
||||||
|
return asset
|
||||||
|
|
||||||
|
|
||||||
def list_assets_for_user(
|
def list_assets_for_user(
|
||||||
session: Session,
|
session: Session,
|
||||||
*,
|
*,
|
||||||
@@ -216,9 +509,119 @@ def list_assets_for_user(
|
|||||||
owner_id: str | None = None,
|
owner_id: str | None = None,
|
||||||
campaign_id: str | None = None,
|
campaign_id: str | None = None,
|
||||||
path_prefix: str | None = None,
|
path_prefix: str | None = None,
|
||||||
|
campaign_usage: str | None = None,
|
||||||
|
audit_relevant: bool | None = None,
|
||||||
include_deleted: bool = False,
|
include_deleted: bool = False,
|
||||||
is_admin: bool = False,
|
is_admin: bool = False,
|
||||||
) -> list[FileAsset]:
|
) -> list[FileAsset]:
|
||||||
|
query = _asset_visibility_query_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
include_deleted=include_deleted,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
def list_recent_assets_for_user(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
limit: int,
|
||||||
|
owner_type: str | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
path_prefix: str | None = None,
|
||||||
|
campaign_usage: str | None = None,
|
||||||
|
audit_relevant: bool | None = None,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> list[FileAsset]:
|
||||||
|
"""Return a bounded recent projection through the normal access query."""
|
||||||
|
|
||||||
|
query = _asset_visibility_query_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
query.order_by(FileAsset.updated_at.desc(), FileAsset.id.asc())
|
||||||
|
.limit(max(1, limit))
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_assets_for_user_window(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
path_prefix: str | None = None,
|
||||||
|
campaign_usage: str | None = None,
|
||||||
|
audit_relevant: bool | None = None,
|
||||||
|
include_deleted: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
page_size: int,
|
||||||
|
after_display_path: str | None = None,
|
||||||
|
after_updated_at=None,
|
||||||
|
after_id: str | None = None,
|
||||||
|
) -> tuple[list[FileAsset], bool]:
|
||||||
|
query = _asset_visibility_query_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
include_deleted=include_deleted,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
if after_display_path is not None and after_updated_at is not None and after_id:
|
||||||
|
query = query.filter(
|
||||||
|
or_(
|
||||||
|
FileAsset.display_path > after_display_path,
|
||||||
|
and_(FileAsset.display_path == after_display_path, FileAsset.updated_at < after_updated_at),
|
||||||
|
and_(FileAsset.display_path == after_display_path, FileAsset.updated_at == after_updated_at, FileAsset.id > after_id),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
rows = query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).limit(page_size + 1).all()
|
||||||
|
return rows[:page_size], len(rows) > page_size
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_visibility_query_for_user(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
path_prefix: str | None = None,
|
||||||
|
campaign_usage: str | None = None,
|
||||||
|
audit_relevant: bool | None = None,
|
||||||
|
include_deleted: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
):
|
||||||
query = session.query(FileAsset).filter(FileAsset.tenant_id == tenant_id)
|
query = session.query(FileAsset).filter(FileAsset.tenant_id == tenant_id)
|
||||||
if not include_deleted:
|
if not include_deleted:
|
||||||
query = query.filter(FileAsset.deleted_at.is_(None))
|
query = query.filter(FileAsset.deleted_at.is_(None))
|
||||||
@@ -229,28 +632,95 @@ def list_assets_for_user(
|
|||||||
if owner_type == "group" and owner_id:
|
if owner_type == "group" and owner_id:
|
||||||
query = query.filter(FileAsset.owner_group_id == owner_id)
|
query = query.filter(FileAsset.owner_group_id == owner_id)
|
||||||
if campaign_id:
|
if campaign_id:
|
||||||
query = query.join(FileShare, FileShare.file_asset_id == FileAsset.id).filter(
|
campaign_share = exists().where(
|
||||||
FileShare.tenant_id == tenant_id,
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id == FileAsset.id,
|
||||||
FileShare.target_type == "campaign",
|
FileShare.target_type == "campaign",
|
||||||
FileShare.target_id == campaign_id,
|
FileShare.target_id == campaign_id,
|
||||||
FileShare.revoked_at.is_(None),
|
effective_file_share_clause(),
|
||||||
)
|
)
|
||||||
|
query = query.filter(campaign_share)
|
||||||
elif not is_admin and not owner_type:
|
elif not is_admin and not owner_type:
|
||||||
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||||
query = query.outerjoin(FileShare, FileShare.file_asset_id == FileAsset.id).filter(
|
active_share = exists().where(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id == FileAsset.id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
or_(
|
||||||
|
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||||
|
(FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
|
||||||
|
(FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
query = query.filter(
|
||||||
or_(
|
or_(
|
||||||
(FileAsset.owner_type == "user") & (FileAsset.owner_user_id == user_id),
|
(FileAsset.owner_type == "user") & (FileAsset.owner_user_id == user_id),
|
||||||
(FileAsset.owner_type == "group") & (FileAsset.owner_group_id.in_(group_ids)),
|
(FileAsset.owner_type == "group") & (FileAsset.owner_group_id.in_(group_ids)),
|
||||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
active_share,
|
||||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
|
|
||||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
if path_prefix:
|
if path_prefix:
|
||||||
prefix = normalize_folder(path_prefix)
|
prefix = normalize_folder(path_prefix)
|
||||||
if prefix:
|
if prefix:
|
||||||
query = query.filter(FileAsset.display_path.like(f"{prefix}/%"))
|
query = query.filter(FileAsset.display_path.like(f"{prefix}/%"))
|
||||||
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc()).all()
|
campaign_share_exists = exists().where(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id == FileAsset.id,
|
||||||
|
FileShare.target_type == "campaign",
|
||||||
|
effective_file_share_clause(),
|
||||||
|
)
|
||||||
|
campaign_use_exists = exists().where(
|
||||||
|
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||||
|
CampaignAttachmentUse.file_asset_id == FileAsset.id,
|
||||||
|
)
|
||||||
|
if campaign_usage == "linked":
|
||||||
|
query = query.filter(or_(campaign_share_exists, campaign_use_exists))
|
||||||
|
elif campaign_usage == "unlinked":
|
||||||
|
query = query.filter(~or_(campaign_share_exists, campaign_use_exists))
|
||||||
|
|
||||||
|
if audit_relevant is not None:
|
||||||
|
sent_use_exists = exists().where(
|
||||||
|
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||||
|
CampaignAttachmentUse.file_asset_id == FileAsset.id,
|
||||||
|
CampaignAttachmentUse.use_stage == "sent",
|
||||||
|
)
|
||||||
|
query = query.filter(sent_use_exists if audit_relevant else ~sent_use_exists)
|
||||||
|
return query
|
||||||
|
|
||||||
|
|
||||||
|
def count_assets_for_user(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str | None = None,
|
||||||
|
owner_id: str | None = None,
|
||||||
|
campaign_id: str | None = None,
|
||||||
|
path_prefix: str | None = None,
|
||||||
|
campaign_usage: str | None = None,
|
||||||
|
audit_relevant: bool | None = None,
|
||||||
|
include_deleted: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
) -> int:
|
||||||
|
query = _asset_visibility_query_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
campaign_id=campaign_id,
|
||||||
|
path_prefix=path_prefix,
|
||||||
|
campaign_usage=campaign_usage,
|
||||||
|
audit_relevant=audit_relevant,
|
||||||
|
include_deleted=include_deleted,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return int(
|
||||||
|
query.order_by(None)
|
||||||
|
.with_entities(func.count(func.distinct(FileAsset.id)))
|
||||||
|
.scalar()
|
||||||
|
or 0
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVersion, FileBlob]:
|
def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVersion, FileBlob]:
|
||||||
@@ -265,13 +735,51 @@ def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVe
|
|||||||
return version, blob
|
return version, blob
|
||||||
|
|
||||||
|
|
||||||
|
def current_versions_and_blobs(session: Session, assets: Iterable[FileAsset]) -> dict[str, tuple[FileVersion, FileBlob]]:
|
||||||
|
asset_list = list(assets)
|
||||||
|
if not asset_list:
|
||||||
|
return {}
|
||||||
|
version_ids = [asset.current_version_id for asset in asset_list if asset.current_version_id]
|
||||||
|
if len(version_ids) != len(asset_list):
|
||||||
|
raise FileStorageError("File has no current version")
|
||||||
|
|
||||||
|
rows: list[tuple[FileVersion, FileBlob]] = []
|
||||||
|
for chunk in _chunks(version_ids):
|
||||||
|
rows.extend(
|
||||||
|
session.query(FileVersion, FileBlob)
|
||||||
|
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
|
||||||
|
.filter(FileVersion.id.in_(chunk))
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
by_version_id = {version.id: (version, blob) for version, blob in rows}
|
||||||
|
result: dict[str, tuple[FileVersion, FileBlob]] = {}
|
||||||
|
for asset in asset_list:
|
||||||
|
version_blob = by_version_id.get(asset.current_version_id or "")
|
||||||
|
if not version_blob:
|
||||||
|
raise FileStorageError("File version not found")
|
||||||
|
result[asset.id] = version_blob
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVersion, FileBlob]:
|
def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVersion, FileBlob]:
|
||||||
version, blob = current_version_and_blob(session, asset)
|
version, blob = current_version_and_blob(session, asset)
|
||||||
backend = get_storage_backend()
|
backend = get_storage_backend()
|
||||||
try:
|
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||||
return backend.get_bytes(blob.storage_key), version, blob
|
|
||||||
except StorageBackendError as exc:
|
|
||||||
raise FileStorageError(str(exc)) from exc
|
def read_asset_version_bytes(
|
||||||
|
session: Session,
|
||||||
|
asset: FileAsset,
|
||||||
|
version_id: str,
|
||||||
|
) -> tuple[bytes, FileVersion, FileBlob]:
|
||||||
|
version = session.get(FileVersion, version_id)
|
||||||
|
if version is None or version.file_asset_id != asset.id:
|
||||||
|
raise FileStorageError("File version not found")
|
||||||
|
blob = session.get(FileBlob, version.blob_id)
|
||||||
|
if blob is None or blob.tenant_id != asset.tenant_id:
|
||||||
|
raise FileStorageError("File blob not found")
|
||||||
|
backend = get_storage_backend()
|
||||||
|
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||||
|
|
||||||
|
|
||||||
def share_file(
|
def share_file(
|
||||||
@@ -283,6 +791,7 @@ def share_file(
|
|||||||
target_id: str,
|
target_id: str,
|
||||||
permission: str,
|
permission: str,
|
||||||
user_id: str,
|
user_id: str,
|
||||||
|
expires_at: datetime | None = None,
|
||||||
) -> FileShare:
|
) -> FileShare:
|
||||||
target_type = target_type.lower().strip()
|
target_type = target_type.lower().strip()
|
||||||
permission = permission.lower().strip()
|
permission = permission.lower().strip()
|
||||||
@@ -292,23 +801,11 @@ def share_file(
|
|||||||
raise FileStorageError("Unsupported share target")
|
raise FileStorageError("Unsupported share target")
|
||||||
if permission not in {"read", "write", "manage"}:
|
if permission not in {"read", "write", "manage"}:
|
||||||
raise FileStorageError("Unsupported file permission")
|
raise FileStorageError("Unsupported file permission")
|
||||||
if target_type == "user":
|
_validate_share_expiry(expires_at)
|
||||||
target_user = session.get(User, target_id)
|
if target_type in {"user", "group", "tenant"}:
|
||||||
if not target_user or target_user.tenant_id != tenant_id or not target_user.is_active:
|
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
|
||||||
raise FileStorageError("User not found")
|
|
||||||
if target_type == "group":
|
|
||||||
group = session.get(Group, target_id)
|
|
||||||
if not group or group.tenant_id != tenant_id or not group.is_active:
|
|
||||||
raise FileStorageError("Group not found")
|
|
||||||
if target_type == "tenant":
|
|
||||||
tenant = session.get(Tenant, target_id)
|
|
||||||
if target_id != tenant_id or not tenant or not tenant.is_active:
|
|
||||||
raise FileStorageError("Tenant not found")
|
|
||||||
if target_type == "campaign":
|
if target_type == "campaign":
|
||||||
Campaign = _campaign_model()
|
_ensure_campaign_exists(session, tenant_id=tenant_id, campaign_id=target_id)
|
||||||
campaign = session.get(Campaign, target_id)
|
|
||||||
if not campaign or campaign.tenant_id != tenant_id:
|
|
||||||
raise FileStorageError("Campaign not found")
|
|
||||||
existing = (
|
existing = (
|
||||||
session.query(FileShare)
|
session.query(FileShare)
|
||||||
.filter(
|
.filter(
|
||||||
@@ -322,6 +819,7 @@ def share_file(
|
|||||||
)
|
)
|
||||||
if existing:
|
if existing:
|
||||||
existing.permission = permission
|
existing.permission = permission
|
||||||
|
existing.expires_at = expires_at
|
||||||
session.add(existing)
|
session.add(existing)
|
||||||
return existing
|
return existing
|
||||||
share = FileShare(
|
share = FileShare(
|
||||||
@@ -331,11 +829,151 @@ def share_file(
|
|||||||
target_id=target_id,
|
target_id=target_id,
|
||||||
permission=permission,
|
permission=permission,
|
||||||
created_by_user_id=user_id,
|
created_by_user_id=user_id,
|
||||||
|
expires_at=expires_at,
|
||||||
)
|
)
|
||||||
session.add(share)
|
session.add(share)
|
||||||
return share
|
return share
|
||||||
|
|
||||||
|
|
||||||
|
def share_files(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
assets: Iterable[FileAsset],
|
||||||
|
target_type: str,
|
||||||
|
target_id: str,
|
||||||
|
permission: str,
|
||||||
|
user_id: str,
|
||||||
|
expires_at: datetime | None = None,
|
||||||
|
) -> list[FileShare]:
|
||||||
|
target_type = target_type.lower().strip()
|
||||||
|
permission = permission.lower().strip()
|
||||||
|
if target_type not in {"user", "group", "campaign", "tenant"}:
|
||||||
|
raise FileStorageError("Unsupported share target")
|
||||||
|
if permission not in {"read", "write", "manage"}:
|
||||||
|
raise FileStorageError("Unsupported file permission")
|
||||||
|
_validate_share_expiry(expires_at)
|
||||||
|
if target_type in {"user", "group", "tenant"}:
|
||||||
|
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
|
||||||
|
if target_type == "campaign":
|
||||||
|
_ensure_campaign_exists(session, tenant_id=tenant_id, campaign_id=target_id)
|
||||||
|
|
||||||
|
asset_list = list(assets)
|
||||||
|
if not asset_list:
|
||||||
|
return []
|
||||||
|
for asset in asset_list:
|
||||||
|
if asset.tenant_id != tenant_id:
|
||||||
|
raise FileStorageError("File not found")
|
||||||
|
|
||||||
|
existing_by_asset: dict[str, FileShare] = {}
|
||||||
|
for share in session.query(FileShare).filter(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id.in_([asset.id for asset in asset_list]),
|
||||||
|
FileShare.target_type == target_type,
|
||||||
|
FileShare.target_id == target_id,
|
||||||
|
FileShare.revoked_at.is_(None),
|
||||||
|
).all():
|
||||||
|
existing_by_asset.setdefault(share.file_asset_id, share)
|
||||||
|
|
||||||
|
shares: list[FileShare] = []
|
||||||
|
for asset in asset_list:
|
||||||
|
existing = existing_by_asset.get(asset.id)
|
||||||
|
if existing:
|
||||||
|
existing.permission = permission
|
||||||
|
existing.expires_at = expires_at
|
||||||
|
session.add(existing)
|
||||||
|
shares.append(existing)
|
||||||
|
continue
|
||||||
|
share = FileShare(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
file_asset_id=asset.id,
|
||||||
|
target_type=target_type,
|
||||||
|
target_id=target_id,
|
||||||
|
permission=permission,
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
expires_at=expires_at,
|
||||||
|
)
|
||||||
|
session.add(share)
|
||||||
|
shares.append(share)
|
||||||
|
return shares
|
||||||
|
|
||||||
|
|
||||||
|
def list_file_shares(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
asset_id: str,
|
||||||
|
include_inactive: bool = False,
|
||||||
|
) -> list[FileShare]:
|
||||||
|
query = session.query(FileShare).filter(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id == asset_id,
|
||||||
|
)
|
||||||
|
if not include_inactive:
|
||||||
|
query = query.filter(effective_file_share_clause())
|
||||||
|
return query.order_by(FileShare.created_at.desc(), FileShare.id.desc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
def revoke_file_share(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
asset_id: str,
|
||||||
|
share_id: str,
|
||||||
|
user_id: str,
|
||||||
|
) -> tuple[FileShare, bool]:
|
||||||
|
share = (
|
||||||
|
session.query(FileShare)
|
||||||
|
.filter(
|
||||||
|
FileShare.id == share_id,
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id == asset_id,
|
||||||
|
)
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if share is None:
|
||||||
|
raise FileStorageError("File share not found")
|
||||||
|
if share.revoked_at is not None:
|
||||||
|
return share, False
|
||||||
|
share.revoked_at = utcnow()
|
||||||
|
share.revoked_by_user_id = user_id
|
||||||
|
session.add(share)
|
||||||
|
return share, True
|
||||||
|
|
||||||
|
|
||||||
|
def current_file_share_for_target(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
asset_id: str,
|
||||||
|
target_type: str,
|
||||||
|
target_id: str,
|
||||||
|
) -> FileShare | None:
|
||||||
|
return (
|
||||||
|
session.query(FileShare)
|
||||||
|
.filter(
|
||||||
|
FileShare.tenant_id == tenant_id,
|
||||||
|
FileShare.file_asset_id == asset_id,
|
||||||
|
FileShare.target_type == target_type.lower().strip(),
|
||||||
|
FileShare.target_id == target_id,
|
||||||
|
FileShare.revoked_at.is_(None),
|
||||||
|
)
|
||||||
|
.order_by(FileShare.created_at.desc(), FileShare.id.desc())
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_share_expiry(expires_at: datetime | None) -> None:
|
||||||
|
if expires_at is None:
|
||||||
|
return
|
||||||
|
from govoplan_files.backend.storage.share_state import file_share_is_active
|
||||||
|
|
||||||
|
candidate = FileShare(expires_at=expires_at)
|
||||||
|
if not file_share_is_active(candidate):
|
||||||
|
raise FileStorageError("File share expiry must be in the future")
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def soft_delete_assets(session: Session, assets: Iterable[FileAsset]) -> int:
|
def soft_delete_assets(session: Session, assets: Iterable[FileAsset]) -> int:
|
||||||
count = 0
|
count = 0
|
||||||
now = utcnow()
|
now = utcnow()
|
||||||
@@ -364,6 +1002,11 @@ def _asset_owner_id(asset: FileAsset) -> str:
|
|||||||
raise FileStorageError("File has no valid owner")
|
raise FileStorageError("File has no valid owner")
|
||||||
|
|
||||||
|
|
||||||
|
def _chunks(values: list[str], size: int = 900):
|
||||||
|
for index in range(0, len(values), size):
|
||||||
|
yield values[index:index + size]
|
||||||
|
|
||||||
|
|
||||||
def _active_asset_exists(session: Session, *, tenant_id: str, owner_type: str, owner_id: str, path: str, exclude_asset_id: str | None = None) -> bool:
|
def _active_asset_exists(session: Session, *, tenant_id: str, owner_type: str, owner_id: str, path: str, exclude_asset_id: str | None = None) -> bool:
|
||||||
return _active_asset_at_path(
|
return _active_asset_at_path(
|
||||||
session,
|
session,
|
||||||
@@ -470,6 +1113,39 @@ def _normalize_conflict_strategy(strategy: str | None) -> str:
|
|||||||
return normalized
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def _next_version_number(session: Session, asset_id: str) -> int:
|
||||||
|
row = (
|
||||||
|
session.query(FileVersion.version_number)
|
||||||
|
.filter(FileVersion.file_asset_id == asset_id)
|
||||||
|
.order_by(FileVersion.version_number.desc())
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
return (int(row[0]) if row else 0) + 1
|
||||||
|
|
||||||
|
|
||||||
|
def _source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||||
|
if not provenance:
|
||||||
|
return None
|
||||||
|
connector_id = _clean_identity(provenance.get("connector_id"))
|
||||||
|
provider = _clean_identity(provenance.get("provider"))
|
||||||
|
external_id = _clean_identity(provenance.get("external_id"))
|
||||||
|
if connector_id and external_id:
|
||||||
|
return ("external_id", connector_id, provider, external_id)
|
||||||
|
external_path = _clean_identity(provenance.get("external_path"))
|
||||||
|
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||||
|
library_id = _clean_identity(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||||
|
if connector_id and external_path:
|
||||||
|
return ("external_path", connector_id, provider, library_id, external_path)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_identity(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
|
|
||||||
|
|
||||||
def _copy_asset_to_path(
|
def _copy_asset_to_path(
|
||||||
session: Session,
|
session: Session,
|
||||||
asset: FileAsset,
|
asset: FileAsset,
|
||||||
|
|||||||
@@ -68,13 +68,63 @@ def list_folders_for_user(
|
|||||||
include_deleted: bool = False,
|
include_deleted: bool = False,
|
||||||
is_admin: bool = False,
|
is_admin: bool = False,
|
||||||
) -> list[FileFolder]:
|
) -> list[FileFolder]:
|
||||||
|
query = _folder_visibility_query_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
include_deleted=include_deleted,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return query.order_by(FileFolder.path.asc(), FileFolder.id.asc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
def list_folders_for_user_window(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
include_deleted: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
page_size: int,
|
||||||
|
after_path: str | None = None,
|
||||||
|
after_id: str | None = None,
|
||||||
|
) -> tuple[list[FileFolder], bool]:
|
||||||
|
query = _folder_visibility_query_for_user(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
user_id=user_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
include_deleted=include_deleted,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
if after_path is not None and after_id:
|
||||||
|
query = query.filter(or_(FileFolder.path > after_path, (FileFolder.path == after_path) & (FileFolder.id > after_id)))
|
||||||
|
rows = query.order_by(FileFolder.path.asc(), FileFolder.id.asc()).limit(page_size + 1).all()
|
||||||
|
return rows[:page_size], len(rows) > page_size
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_visibility_query_for_user(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
include_deleted: bool = False,
|
||||||
|
is_admin: bool = False,
|
||||||
|
):
|
||||||
owner_type = owner_type.lower().strip()
|
owner_type = owner_type.lower().strip()
|
||||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||||
query = session.query(FileFolder).filter(FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type)
|
query = session.query(FileFolder).filter(FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type)
|
||||||
query = _owner_filter(query, owner_type, owner_id)
|
query = _owner_filter(query, owner_type, owner_id)
|
||||||
if not include_deleted:
|
if not include_deleted:
|
||||||
query = query.filter(FileFolder.deleted_at.is_(None))
|
query = query.filter(FileFolder.deleted_at.is_(None))
|
||||||
return query.order_by(FileFolder.path.asc()).all()
|
return query
|
||||||
|
|
||||||
|
|
||||||
def soft_delete_folder(
|
def soft_delete_folder(
|
||||||
|
|||||||
@@ -0,0 +1,247 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import socket
|
||||||
|
import ssl
|
||||||
|
import select
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any, Iterator
|
||||||
|
|
||||||
|
import httpcore
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from govoplan_core.security.outbound_http import (
|
||||||
|
OutboundHttpError,
|
||||||
|
bounded_chunks_bytes,
|
||||||
|
create_outbound_connection,
|
||||||
|
response_limit,
|
||||||
|
validate_outbound_http_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorHttpError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConnectorHttpResponse:
|
||||||
|
status_code: int
|
||||||
|
headers: Mapping[str, str]
|
||||||
|
content: bytes
|
||||||
|
|
||||||
|
|
||||||
|
_HTTPCORE_TRANSPORT_ERRORS = (
|
||||||
|
httpcore.TimeoutException,
|
||||||
|
httpcore.NetworkError,
|
||||||
|
httpcore.ProtocolError,
|
||||||
|
httpcore.ProxyError,
|
||||||
|
httpcore.UnsupportedProtocol,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _SocketNetworkStream(httpcore.NetworkStream):
|
||||||
|
"""Public httpcore NetworkStream adapter around an approved socket."""
|
||||||
|
|
||||||
|
def __init__(self, sock: socket.socket) -> None:
|
||||||
|
self._socket = sock
|
||||||
|
|
||||||
|
def read(self, max_bytes: int, timeout: float | None = None) -> bytes:
|
||||||
|
try:
|
||||||
|
self._socket.settimeout(timeout)
|
||||||
|
return self._socket.recv(max_bytes)
|
||||||
|
except socket.timeout as exc:
|
||||||
|
raise httpcore.ReadTimeout(str(exc)) from exc
|
||||||
|
except OSError as exc:
|
||||||
|
raise httpcore.ReadError(str(exc)) from exc
|
||||||
|
|
||||||
|
def write(self, buffer: bytes, timeout: float | None = None) -> None:
|
||||||
|
try:
|
||||||
|
self._socket.settimeout(timeout)
|
||||||
|
self._socket.sendall(buffer)
|
||||||
|
except socket.timeout as exc:
|
||||||
|
raise httpcore.WriteTimeout(str(exc)) from exc
|
||||||
|
except OSError as exc:
|
||||||
|
raise httpcore.WriteError(str(exc)) from exc
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self._socket.close()
|
||||||
|
|
||||||
|
def start_tls(
|
||||||
|
self,
|
||||||
|
ssl_context: ssl.SSLContext,
|
||||||
|
server_hostname: str | None = None,
|
||||||
|
timeout: float | None = None,
|
||||||
|
) -> httpcore.NetworkStream:
|
||||||
|
try:
|
||||||
|
self._socket.settimeout(timeout)
|
||||||
|
tls_socket = ssl_context.wrap_socket(self._socket, server_hostname=server_hostname)
|
||||||
|
except socket.timeout as exc:
|
||||||
|
self.close()
|
||||||
|
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||||
|
except OSError as exc:
|
||||||
|
self.close()
|
||||||
|
raise httpcore.ConnectError(str(exc)) from exc
|
||||||
|
return _SocketNetworkStream(tls_socket)
|
||||||
|
|
||||||
|
def get_extra_info(self, info: str) -> Any:
|
||||||
|
if info == "ssl_object" and isinstance(self._socket, ssl.SSLSocket):
|
||||||
|
return self._socket
|
||||||
|
if info == "client_addr":
|
||||||
|
return self._socket.getsockname()
|
||||||
|
if info == "server_addr":
|
||||||
|
return self._socket.getpeername()
|
||||||
|
if info == "socket":
|
||||||
|
return self._socket
|
||||||
|
if info == "is_readable":
|
||||||
|
try:
|
||||||
|
return bool(select.select([self._socket], [], [], 0)[0])
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return True
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class _OutboundPolicyNetworkBackend(httpcore.NetworkBackend):
|
||||||
|
def connect_tcp(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int,
|
||||||
|
timeout: float | None = None,
|
||||||
|
local_address: str | None = None,
|
||||||
|
socket_options: Any = None,
|
||||||
|
) -> httpcore.NetworkStream:
|
||||||
|
source_address = None if local_address is None else (local_address, 0)
|
||||||
|
try:
|
||||||
|
sock = create_outbound_connection(
|
||||||
|
host,
|
||||||
|
port,
|
||||||
|
timeout=timeout,
|
||||||
|
source_address=source_address,
|
||||||
|
socket_options=socket_options,
|
||||||
|
label="File connector HTTP endpoint",
|
||||||
|
)
|
||||||
|
except socket.timeout as exc:
|
||||||
|
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||||
|
except (OSError, OutboundHttpError) as exc:
|
||||||
|
raise httpcore.ConnectError(str(exc)) from exc
|
||||||
|
return _SocketNetworkStream(sock)
|
||||||
|
|
||||||
|
def connect_unix_socket(self, path: str, timeout: float | None = None, socket_options: Any = None): # type: ignore[no-untyped-def]
|
||||||
|
del path, timeout, socket_options
|
||||||
|
raise httpcore.ConnectError("Unix sockets are not supported for file connectors")
|
||||||
|
|
||||||
|
|
||||||
|
class _HttpcoreResponseStream(httpx.SyncByteStream):
|
||||||
|
def __init__(self, stream: Any, *, request: httpx.Request) -> None:
|
||||||
|
self._stream = stream
|
||||||
|
self._request = request
|
||||||
|
|
||||||
|
def __iter__(self): # type: ignore[no-untyped-def]
|
||||||
|
try:
|
||||||
|
yield from self._stream
|
||||||
|
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||||
|
raise httpx.TransportError(str(exc), request=self._request) from exc
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
if hasattr(self._stream, "close"):
|
||||||
|
self._stream.close()
|
||||||
|
|
||||||
|
|
||||||
|
class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._connection_pool = httpcore.ConnectionPool(
|
||||||
|
ssl_context=httpx.create_ssl_context(verify=True, trust_env=False),
|
||||||
|
max_connections=100,
|
||||||
|
max_keepalive_connections=20,
|
||||||
|
keepalive_expiry=5.0,
|
||||||
|
network_backend=_OutboundPolicyNetworkBackend(),
|
||||||
|
)
|
||||||
|
|
||||||
|
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||||
|
core_request = httpcore.Request(
|
||||||
|
method=request.method,
|
||||||
|
url=httpcore.URL(
|
||||||
|
scheme=request.url.raw_scheme,
|
||||||
|
host=request.url.raw_host,
|
||||||
|
port=request.url.port,
|
||||||
|
target=request.url.raw_path,
|
||||||
|
),
|
||||||
|
headers=request.headers.raw,
|
||||||
|
content=request.stream,
|
||||||
|
extensions=request.extensions,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
response = self._connection_pool.handle_request(core_request)
|
||||||
|
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||||
|
raise httpx.TransportError(str(exc), request=request) from exc
|
||||||
|
return httpx.Response(
|
||||||
|
status_code=response.status,
|
||||||
|
headers=response.headers,
|
||||||
|
stream=_HttpcoreResponseStream(response.stream, request=request),
|
||||||
|
extensions=response.extensions,
|
||||||
|
)
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self._connection_pool.close()
|
||||||
|
|
||||||
|
|
||||||
|
_CONNECTOR_HTTP_CLIENT = httpx.Client(
|
||||||
|
transport=_OutboundPolicyHTTPTransport(),
|
||||||
|
follow_redirects=False,
|
||||||
|
timeout=15.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
|
||||||
|
timeout = kwargs.pop("timeout", 15.0)
|
||||||
|
with _CONNECTOR_HTTP_CLIENT.stream(
|
||||||
|
method,
|
||||||
|
url,
|
||||||
|
timeout=timeout,
|
||||||
|
**kwargs,
|
||||||
|
) as response:
|
||||||
|
yield response
|
||||||
|
|
||||||
|
|
||||||
|
def request_connector_bytes(
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
headers: Mapping[str, str] | None = None,
|
||||||
|
params: Mapping[str, str] | None = None,
|
||||||
|
data: Mapping[str, str] | bytes | str | None = None,
|
||||||
|
content: bytes | str | None = None,
|
||||||
|
auth: Any = None,
|
||||||
|
timeout: float = 15.0,
|
||||||
|
kind: str = "structured",
|
||||||
|
max_bytes: int | None = None,
|
||||||
|
label: str = "File connector",
|
||||||
|
) -> ConnectorHttpResponse:
|
||||||
|
try:
|
||||||
|
validated_url = validate_outbound_http_url(url, label=f"{label} URL")
|
||||||
|
effective_limit = response_limit(kind) if max_bytes is None else min(int(max_bytes), response_limit(kind))
|
||||||
|
with _stream_connector_request(
|
||||||
|
method,
|
||||||
|
validated_url,
|
||||||
|
headers=dict(headers or {}),
|
||||||
|
params=params,
|
||||||
|
data=data,
|
||||||
|
content=content,
|
||||||
|
auth=auth,
|
||||||
|
timeout=timeout,
|
||||||
|
) as response:
|
||||||
|
body = bounded_chunks_bytes(
|
||||||
|
response.iter_bytes(),
|
||||||
|
headers=response.headers,
|
||||||
|
max_bytes=effective_limit,
|
||||||
|
kind=kind,
|
||||||
|
label=f"{label} response",
|
||||||
|
)
|
||||||
|
return ConnectorHttpResponse(
|
||||||
|
status_code=response.status_code,
|
||||||
|
headers=dict(response.headers),
|
||||||
|
content=body,
|
||||||
|
)
|
||||||
|
except (httpx.HTTPError, OutboundHttpError, ValueError) as exc:
|
||||||
|
raise ConnectorHttpError(str(exc)) from exc
|
||||||
@@ -0,0 +1,550 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session, object_session
|
||||||
|
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
FileBlob,
|
||||||
|
FileIntegrityFinding,
|
||||||
|
FileIntegrityScan,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
StorageBackend,
|
||||||
|
StorageBackendError,
|
||||||
|
StorageObjectMissing,
|
||||||
|
get_storage_backend,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.recovery import (
|
||||||
|
begin_orphan_cleanup_recovery,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||||
|
|
||||||
|
|
||||||
|
TERMINAL_SCAN_STATUSES = {"completed", "cancelled"}
|
||||||
|
QUARANTINED_BLOB_STATUSES = {
|
||||||
|
"missing",
|
||||||
|
"size_mismatch",
|
||||||
|
"checksum_mismatch",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class BlobInspection:
|
||||||
|
valid: bool
|
||||||
|
kind: str
|
||||||
|
observed_size_bytes: int | None = None
|
||||||
|
observed_checksum_sha256: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class IntegrityActionResult:
|
||||||
|
action: str
|
||||||
|
changed: bool
|
||||||
|
dry_run: bool
|
||||||
|
finding: FileIntegrityFinding
|
||||||
|
inspection: BlobInspection | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def storage_prefix_for_tenant(tenant_id: str) -> str:
|
||||||
|
return f"tenants/{tenant_id}/files/"
|
||||||
|
|
||||||
|
|
||||||
|
def create_integrity_scan(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
verify_checksums: bool = True,
|
||||||
|
batch_size: int = 100,
|
||||||
|
backend: StorageBackend | None = None,
|
||||||
|
) -> FileIntegrityScan:
|
||||||
|
active_backend = backend or get_storage_backend()
|
||||||
|
scan = FileIntegrityScan(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
storage_backend=active_backend.name,
|
||||||
|
storage_prefix=storage_prefix_for_tenant(tenant_id),
|
||||||
|
verify_checksums=verify_checksums,
|
||||||
|
batch_size=max(1, min(int(batch_size), 1000)),
|
||||||
|
created_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
session.add(scan)
|
||||||
|
session.flush()
|
||||||
|
return scan
|
||||||
|
|
||||||
|
|
||||||
|
def run_integrity_scan_batch(
|
||||||
|
session: Session,
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend | None = None,
|
||||||
|
) -> FileIntegrityScan:
|
||||||
|
if scan.status in TERMINAL_SCAN_STATUSES:
|
||||||
|
return scan
|
||||||
|
active_backend = backend or get_storage_backend()
|
||||||
|
if active_backend.name != scan.storage_backend:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The configured storage backend changed after this integrity scan started"
|
||||||
|
)
|
||||||
|
if scan.started_at is None:
|
||||||
|
scan.started_at = utcnow()
|
||||||
|
scan.status = "running"
|
||||||
|
scan.last_error = None
|
||||||
|
if scan.phase == "blobs":
|
||||||
|
_scan_blob_batch(session, scan, active_backend)
|
||||||
|
elif scan.phase == "objects":
|
||||||
|
_scan_object_batch(session, scan, active_backend)
|
||||||
|
else:
|
||||||
|
scan.phase = "completed"
|
||||||
|
scan.status = "completed"
|
||||||
|
scan.completed_at = utcnow()
|
||||||
|
scan.revision += 1
|
||||||
|
session.add(scan)
|
||||||
|
return scan
|
||||||
|
|
||||||
|
|
||||||
|
def mark_integrity_scan_failed(
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
*,
|
||||||
|
error: Exception,
|
||||||
|
) -> None:
|
||||||
|
scan.status = "failed"
|
||||||
|
scan.last_error = type(error).__name__[:255]
|
||||||
|
scan.revision += 1
|
||||||
|
|
||||||
|
|
||||||
|
def inspect_blob(
|
||||||
|
blob: FileBlob,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend,
|
||||||
|
verify_checksum: bool = True,
|
||||||
|
) -> BlobInspection:
|
||||||
|
expected_size = blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes
|
||||||
|
expected_checksum = blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256
|
||||||
|
try:
|
||||||
|
info = backend.stat(blob.storage_key)
|
||||||
|
except StorageObjectMissing:
|
||||||
|
return BlobInspection(valid=False, kind="missing")
|
||||||
|
if info.size_bytes != expected_size:
|
||||||
|
return BlobInspection(
|
||||||
|
valid=False,
|
||||||
|
kind="size_mismatch",
|
||||||
|
observed_size_bytes=info.size_bytes,
|
||||||
|
)
|
||||||
|
if not verify_checksum:
|
||||||
|
return BlobInspection(
|
||||||
|
valid=True,
|
||||||
|
kind="verified",
|
||||||
|
observed_size_bytes=info.size_bytes,
|
||||||
|
)
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
observed_size = 0
|
||||||
|
for chunk in backend.iter_bytes(blob.storage_key):
|
||||||
|
observed_size += len(chunk)
|
||||||
|
digest.update(chunk)
|
||||||
|
observed_checksum = digest.hexdigest()
|
||||||
|
if observed_size != expected_size:
|
||||||
|
return BlobInspection(
|
||||||
|
valid=False,
|
||||||
|
kind="size_mismatch",
|
||||||
|
observed_size_bytes=observed_size,
|
||||||
|
observed_checksum_sha256=observed_checksum,
|
||||||
|
)
|
||||||
|
if observed_checksum != expected_checksum:
|
||||||
|
return BlobInspection(
|
||||||
|
valid=False,
|
||||||
|
kind="checksum_mismatch",
|
||||||
|
observed_size_bytes=observed_size,
|
||||||
|
observed_checksum_sha256=observed_checksum,
|
||||||
|
)
|
||||||
|
return BlobInspection(
|
||||||
|
valid=True,
|
||||||
|
kind="verified",
|
||||||
|
observed_size_bytes=observed_size,
|
||||||
|
observed_checksum_sha256=observed_checksum,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_blob_inspection(
|
||||||
|
session: Session,
|
||||||
|
blob: FileBlob,
|
||||||
|
inspection: BlobInspection,
|
||||||
|
*,
|
||||||
|
checked_at=None,
|
||||||
|
) -> None:
|
||||||
|
timestamp = checked_at or utcnow()
|
||||||
|
blob.integrity_checked_at = timestamp
|
||||||
|
if inspection.valid:
|
||||||
|
blob.integrity_status = "verified"
|
||||||
|
blob.integrity_failure = None
|
||||||
|
blob.quarantined_at = None
|
||||||
|
else:
|
||||||
|
blob.integrity_status = inspection.kind
|
||||||
|
blob.integrity_failure = inspection.kind
|
||||||
|
blob.quarantined_at = blob.quarantined_at or timestamp
|
||||||
|
session.add(blob)
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_blob_is_readable(blob: FileBlob) -> None:
|
||||||
|
if blob.integrity_status in QUARANTINED_BLOB_STATUSES or blob.quarantined_at:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Managed file content is quarantined because integrity verification failed"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def read_verified_blob_bytes(
|
||||||
|
blob: FileBlob,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend,
|
||||||
|
) -> bytes:
|
||||||
|
ensure_blob_is_readable(blob)
|
||||||
|
try:
|
||||||
|
data = backend.get_bytes(blob.storage_key)
|
||||||
|
except StorageBackendError as exc:
|
||||||
|
raise FileStorageError("Managed file content is not available") from exc
|
||||||
|
expected_storage_size = blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes
|
||||||
|
expected_storage_checksum = blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256
|
||||||
|
if len(data) != expected_storage_size:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Managed file content failed its recorded size verification"
|
||||||
|
)
|
||||||
|
if hashlib.sha256(data).hexdigest() != expected_storage_checksum:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Managed file content failed its recorded checksum verification"
|
||||||
|
)
|
||||||
|
if blob.encryption_envelope_id:
|
||||||
|
session = object_session(blob)
|
||||||
|
if session is None:
|
||||||
|
raise FileStorageError("Encrypted managed content requires an attached database session")
|
||||||
|
from govoplan_files.backend.storage.content_protection import unprotect_blob_content
|
||||||
|
|
||||||
|
data = unprotect_blob_content(
|
||||||
|
session,
|
||||||
|
tenant_id=blob.tenant_id,
|
||||||
|
blob_id=blob.id,
|
||||||
|
envelope_id=blob.encryption_envelope_id,
|
||||||
|
ciphertext=data,
|
||||||
|
)
|
||||||
|
if len(data) != blob.size_bytes:
|
||||||
|
raise FileStorageError("Decrypted managed file content failed its recorded size verification")
|
||||||
|
if hashlib.sha256(data).hexdigest() != blob.checksum_sha256:
|
||||||
|
raise FileStorageError("Decrypted managed file content failed its recorded checksum verification")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def recheck_integrity_finding(
|
||||||
|
session: Session,
|
||||||
|
finding: FileIntegrityFinding,
|
||||||
|
*,
|
||||||
|
user_id: str,
|
||||||
|
dry_run: bool,
|
||||||
|
backend: StorageBackend | None = None,
|
||||||
|
) -> IntegrityActionResult:
|
||||||
|
if finding.kind == "orphan_object" or not finding.blob_id:
|
||||||
|
raise FileStorageError("Only blob integrity findings can be rechecked")
|
||||||
|
blob = session.get(FileBlob, finding.blob_id)
|
||||||
|
if blob is None or blob.tenant_id != finding.tenant_id:
|
||||||
|
raise FileStorageError("Integrity finding blob no longer exists")
|
||||||
|
active_backend = backend or get_storage_backend()
|
||||||
|
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||||
|
if scan is None or scan.storage_backend != active_backend.name:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The configured storage backend does not match the integrity finding"
|
||||||
|
)
|
||||||
|
inspection = inspect_blob(
|
||||||
|
blob,
|
||||||
|
backend=active_backend,
|
||||||
|
verify_checksum=True,
|
||||||
|
)
|
||||||
|
changed = False
|
||||||
|
if not dry_run:
|
||||||
|
previous = (
|
||||||
|
blob.integrity_status,
|
||||||
|
blob.quarantined_at,
|
||||||
|
finding.state,
|
||||||
|
)
|
||||||
|
apply_blob_inspection(session, blob, inspection)
|
||||||
|
_update_finding_from_inspection(finding, inspection)
|
||||||
|
if inspection.valid:
|
||||||
|
finding.state = "resolved"
|
||||||
|
finding.resolved_at = utcnow()
|
||||||
|
finding.resolved_by_user_id = user_id
|
||||||
|
session.add(finding)
|
||||||
|
finding.revision += 1
|
||||||
|
changed = previous != (
|
||||||
|
blob.integrity_status,
|
||||||
|
blob.quarantined_at,
|
||||||
|
finding.state,
|
||||||
|
)
|
||||||
|
return IntegrityActionResult(
|
||||||
|
action="recheck",
|
||||||
|
changed=changed,
|
||||||
|
dry_run=dry_run,
|
||||||
|
finding=finding,
|
||||||
|
inspection=inspection,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def cleanup_orphan_finding(
|
||||||
|
session: Session,
|
||||||
|
finding: FileIntegrityFinding,
|
||||||
|
*,
|
||||||
|
user_id: str,
|
||||||
|
dry_run: bool,
|
||||||
|
backend: StorageBackend | None = None,
|
||||||
|
) -> IntegrityActionResult:
|
||||||
|
if finding.kind != "orphan_object":
|
||||||
|
raise FileStorageError("Only orphan-object findings can be cleaned up")
|
||||||
|
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||||
|
if scan is None or scan.tenant_id != finding.tenant_id:
|
||||||
|
raise FileStorageError("Integrity scan not found")
|
||||||
|
if not finding.storage_key.startswith(scan.storage_prefix):
|
||||||
|
raise FileStorageError("Orphan cleanup is outside the scan storage scope")
|
||||||
|
referenced = (
|
||||||
|
session.query(FileBlob.id)
|
||||||
|
.filter(
|
||||||
|
FileBlob.tenant_id == finding.tenant_id,
|
||||||
|
FileBlob.storage_backend == scan.storage_backend,
|
||||||
|
FileBlob.storage_key == finding.storage_key,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
if referenced:
|
||||||
|
if not dry_run and finding.state != "resolved":
|
||||||
|
finding.state = "resolved"
|
||||||
|
finding.resolved_at = utcnow()
|
||||||
|
finding.resolved_by_user_id = user_id
|
||||||
|
finding.revision += 1
|
||||||
|
session.add(finding)
|
||||||
|
return IntegrityActionResult(
|
||||||
|
action="retained_referenced",
|
||||||
|
changed=True,
|
||||||
|
dry_run=False,
|
||||||
|
finding=finding,
|
||||||
|
)
|
||||||
|
return IntegrityActionResult(
|
||||||
|
action="retained_referenced",
|
||||||
|
changed=False,
|
||||||
|
dry_run=dry_run,
|
||||||
|
finding=finding,
|
||||||
|
)
|
||||||
|
if finding.state == "deleted":
|
||||||
|
return IntegrityActionResult(
|
||||||
|
action="already_deleted",
|
||||||
|
changed=False,
|
||||||
|
dry_run=dry_run,
|
||||||
|
finding=finding,
|
||||||
|
)
|
||||||
|
if dry_run:
|
||||||
|
return IntegrityActionResult(
|
||||||
|
action="would_delete",
|
||||||
|
changed=False,
|
||||||
|
dry_run=True,
|
||||||
|
finding=finding,
|
||||||
|
)
|
||||||
|
active_backend = backend or get_storage_backend()
|
||||||
|
if active_backend.name != scan.storage_backend:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The configured storage backend does not match the integrity finding"
|
||||||
|
)
|
||||||
|
begin_orphan_cleanup_recovery(
|
||||||
|
session,
|
||||||
|
finding,
|
||||||
|
backend=active_backend,
|
||||||
|
user_id=user_id,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
active_backend.stat(finding.storage_key)
|
||||||
|
except StorageObjectMissing:
|
||||||
|
action = "already_absent"
|
||||||
|
else:
|
||||||
|
active_backend.delete(finding.storage_key)
|
||||||
|
action = "deleted"
|
||||||
|
finding.state = "deleted"
|
||||||
|
finding.resolved_at = utcnow()
|
||||||
|
finding.resolved_by_user_id = user_id
|
||||||
|
finding.revision += 1
|
||||||
|
session.add(finding)
|
||||||
|
return IntegrityActionResult(
|
||||||
|
action=action,
|
||||||
|
changed=True,
|
||||||
|
dry_run=False,
|
||||||
|
finding=finding,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _scan_blob_batch(
|
||||||
|
session: Session,
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
backend: StorageBackend,
|
||||||
|
) -> None:
|
||||||
|
query = session.query(FileBlob).filter(
|
||||||
|
FileBlob.tenant_id == scan.tenant_id,
|
||||||
|
FileBlob.storage_backend == scan.storage_backend,
|
||||||
|
)
|
||||||
|
if scan.blob_cursor:
|
||||||
|
query = query.filter(FileBlob.id > scan.blob_cursor)
|
||||||
|
blobs = query.order_by(FileBlob.id.asc()).limit(scan.batch_size).all()
|
||||||
|
for blob in blobs:
|
||||||
|
inspection = inspect_blob(
|
||||||
|
blob,
|
||||||
|
backend=backend,
|
||||||
|
verify_checksum=scan.verify_checksums,
|
||||||
|
)
|
||||||
|
apply_blob_inspection(session, blob, inspection)
|
||||||
|
scan.scanned_blob_count += 1
|
||||||
|
if inspection.valid:
|
||||||
|
scan.verified_blob_count += 1
|
||||||
|
_resolve_scan_blob_findings(session, scan, blob)
|
||||||
|
else:
|
||||||
|
scan.quarantined_blob_count += 1
|
||||||
|
_record_blob_finding(session, scan, blob, inspection)
|
||||||
|
scan.blob_cursor = blob.id
|
||||||
|
if len(blobs) < scan.batch_size:
|
||||||
|
scan.phase = "objects"
|
||||||
|
scan.object_cursor = None
|
||||||
|
|
||||||
|
|
||||||
|
def _scan_object_batch(
|
||||||
|
session: Session,
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
backend: StorageBackend,
|
||||||
|
) -> None:
|
||||||
|
page = backend.list_objects(
|
||||||
|
prefix=scan.storage_prefix,
|
||||||
|
after=scan.object_cursor,
|
||||||
|
limit=scan.batch_size,
|
||||||
|
)
|
||||||
|
keys = [item.key for item in page.objects]
|
||||||
|
referenced_keys = {
|
||||||
|
row[0]
|
||||||
|
for row in session.query(FileBlob.storage_key)
|
||||||
|
.filter(
|
||||||
|
FileBlob.tenant_id == scan.tenant_id,
|
||||||
|
FileBlob.storage_backend == scan.storage_backend,
|
||||||
|
FileBlob.storage_key.in_(keys),
|
||||||
|
)
|
||||||
|
.all()
|
||||||
|
} if keys else set()
|
||||||
|
for item in page.objects:
|
||||||
|
scan.scanned_object_count += 1
|
||||||
|
if item.key in referenced_keys:
|
||||||
|
continue
|
||||||
|
scan.orphan_object_count += 1
|
||||||
|
_record_orphan_finding(
|
||||||
|
session,
|
||||||
|
scan,
|
||||||
|
storage_key=item.key,
|
||||||
|
size_bytes=item.size_bytes,
|
||||||
|
)
|
||||||
|
scan.object_cursor = page.next_cursor
|
||||||
|
if page.next_cursor is None:
|
||||||
|
scan.phase = "completed"
|
||||||
|
scan.status = "completed"
|
||||||
|
scan.completed_at = utcnow()
|
||||||
|
|
||||||
|
|
||||||
|
def _record_blob_finding(
|
||||||
|
session: Session,
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
blob: FileBlob,
|
||||||
|
inspection: BlobInspection,
|
||||||
|
) -> FileIntegrityFinding:
|
||||||
|
finding = (
|
||||||
|
session.query(FileIntegrityFinding)
|
||||||
|
.filter(
|
||||||
|
FileIntegrityFinding.scan_id == scan.id,
|
||||||
|
FileIntegrityFinding.blob_id == blob.id,
|
||||||
|
FileIntegrityFinding.kind == inspection.kind,
|
||||||
|
)
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if finding is None:
|
||||||
|
finding = FileIntegrityFinding(
|
||||||
|
scan_id=scan.id,
|
||||||
|
tenant_id=scan.tenant_id,
|
||||||
|
kind=inspection.kind,
|
||||||
|
blob_id=blob.id,
|
||||||
|
storage_key=blob.storage_key,
|
||||||
|
expected_size_bytes=blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes,
|
||||||
|
expected_checksum_sha256=blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
finding.revision += 1
|
||||||
|
_update_finding_from_inspection(finding, inspection)
|
||||||
|
session.add(finding)
|
||||||
|
return finding
|
||||||
|
|
||||||
|
|
||||||
|
def _record_orphan_finding(
|
||||||
|
session: Session,
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
*,
|
||||||
|
storage_key: str,
|
||||||
|
size_bytes: int,
|
||||||
|
) -> FileIntegrityFinding:
|
||||||
|
finding = (
|
||||||
|
session.query(FileIntegrityFinding)
|
||||||
|
.filter(
|
||||||
|
FileIntegrityFinding.scan_id == scan.id,
|
||||||
|
FileIntegrityFinding.kind == "orphan_object",
|
||||||
|
FileIntegrityFinding.storage_key == storage_key,
|
||||||
|
)
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
if finding is None:
|
||||||
|
finding = FileIntegrityFinding(
|
||||||
|
scan_id=scan.id,
|
||||||
|
tenant_id=scan.tenant_id,
|
||||||
|
kind="orphan_object",
|
||||||
|
storage_key=storage_key,
|
||||||
|
observed_size_bytes=size_bytes,
|
||||||
|
)
|
||||||
|
session.add(finding)
|
||||||
|
return finding
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_scan_blob_findings(
|
||||||
|
session: Session,
|
||||||
|
scan: FileIntegrityScan,
|
||||||
|
blob: FileBlob,
|
||||||
|
) -> None:
|
||||||
|
for finding in (
|
||||||
|
session.query(FileIntegrityFinding)
|
||||||
|
.filter(
|
||||||
|
FileIntegrityFinding.scan_id == scan.id,
|
||||||
|
FileIntegrityFinding.blob_id == blob.id,
|
||||||
|
FileIntegrityFinding.state == "open",
|
||||||
|
)
|
||||||
|
.all()
|
||||||
|
):
|
||||||
|
finding.state = "resolved"
|
||||||
|
finding.resolved_at = utcnow()
|
||||||
|
finding.revision += 1
|
||||||
|
session.add(finding)
|
||||||
|
|
||||||
|
|
||||||
|
def _update_finding_from_inspection(
|
||||||
|
finding: FileIntegrityFinding,
|
||||||
|
inspection: BlobInspection,
|
||||||
|
) -> None:
|
||||||
|
finding.observed_size_bytes = inspection.observed_size_bytes
|
||||||
|
finding.observed_checksum_sha256 = inspection.observed_checksum_sha256
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"BlobInspection",
|
||||||
|
"IntegrityActionResult",
|
||||||
|
"QUARANTINED_BLOB_STATUSES",
|
||||||
|
"apply_blob_inspection",
|
||||||
|
"cleanup_orphan_finding",
|
||||||
|
"create_integrity_scan",
|
||||||
|
"ensure_blob_is_readable",
|
||||||
|
"inspect_blob",
|
||||||
|
"mark_integrity_scan_failed",
|
||||||
|
"read_verified_blob_bytes",
|
||||||
|
"recheck_integrity_finding",
|
||||||
|
"run_integrity_scan_batch",
|
||||||
|
"storage_prefix_for_tenant",
|
||||||
|
]
|
||||||
@@ -0,0 +1,796 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from typing import Callable, Iterable
|
||||||
|
|
||||||
|
from sqlalchemy import func, inspect, or_
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.recovery import (
|
||||||
|
RecoveryGuaranteeError,
|
||||||
|
RecoveryMode,
|
||||||
|
RecoveryPlan,
|
||||||
|
RecoveryStatus,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.recovery_runtime import (
|
||||||
|
RecoveryOperationBusy,
|
||||||
|
RecoveryOperationStateConflict,
|
||||||
|
begin_durable_recovery_operation,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||||
|
from govoplan_core.db.session import get_database
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
CampaignAttachmentUse,
|
||||||
|
FileAsset,
|
||||||
|
FileBlob,
|
||||||
|
FileConnectorSpace,
|
||||||
|
FileFolder,
|
||||||
|
FileFormEvidenceGrant,
|
||||||
|
FileShare,
|
||||||
|
FileVersion,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||||
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
StorageBackendError,
|
||||||
|
get_storage_backend,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||||
|
from govoplan_files.backend.storage.connector_spaces import connector_space_owner_id
|
||||||
|
from govoplan_files.backend.storage.paths import normalize_folder
|
||||||
|
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PurgePreviewItem:
|
||||||
|
file_id: str
|
||||||
|
filename: str
|
||||||
|
lifecycle_revision: int
|
||||||
|
deleted_at: datetime | None
|
||||||
|
retained_until: datetime | None
|
||||||
|
legal_hold: bool
|
||||||
|
blockers: tuple[str, ...]
|
||||||
|
blob_ids: tuple[str, ...]
|
||||||
|
|
||||||
|
def digest_payload(self) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"file_id": self.file_id,
|
||||||
|
"filename": self.filename,
|
||||||
|
"lifecycle_revision": self.lifecycle_revision,
|
||||||
|
"deleted_at": _iso(self.deleted_at),
|
||||||
|
"retained_until": _iso(self.retained_until),
|
||||||
|
"legal_hold": self.legal_hold,
|
||||||
|
"blockers": list(self.blockers),
|
||||||
|
"blob_ids": list(self.blob_ids),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PurgePreview:
|
||||||
|
preview_sha256: str
|
||||||
|
items: tuple[PurgePreviewItem, ...]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def eligible(self) -> bool:
|
||||||
|
return all(not item.blockers for item in self.items)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PurgeResult:
|
||||||
|
recovery_operation_id: str
|
||||||
|
status: str
|
||||||
|
replayed: bool
|
||||||
|
purged_files: int
|
||||||
|
released_blobs: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class BlobGcResult:
|
||||||
|
inspected_blobs: int
|
||||||
|
deleted_blobs: int
|
||||||
|
unresolved_operation_ids: tuple[str, ...]
|
||||||
|
|
||||||
|
|
||||||
|
def set_asset_lifecycle(
|
||||||
|
session: Session,
|
||||||
|
asset: FileAsset,
|
||||||
|
*,
|
||||||
|
retained_until: datetime | None,
|
||||||
|
legal_hold: bool,
|
||||||
|
reason: str,
|
||||||
|
expected_revision: int,
|
||||||
|
) -> FileAsset:
|
||||||
|
if asset.lifecycle_revision != expected_revision:
|
||||||
|
raise FileStorageError("File lifecycle settings changed; reload before saving")
|
||||||
|
asset.retained_until = retained_until
|
||||||
|
asset.legal_hold = bool(legal_hold)
|
||||||
|
asset.lifecycle_reason = reason.strip()
|
||||||
|
asset.lifecycle_revision += 1
|
||||||
|
session.add(asset)
|
||||||
|
session.flush()
|
||||||
|
return asset
|
||||||
|
|
||||||
|
|
||||||
|
def get_asset_for_lifecycle(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
asset_id: str,
|
||||||
|
is_admin: bool,
|
||||||
|
) -> FileAsset:
|
||||||
|
asset = session.get(FileAsset, asset_id)
|
||||||
|
if asset is None or asset.tenant_id != tenant_id:
|
||||||
|
raise FileStorageError("File not found")
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=asset.owner_type,
|
||||||
|
owner_id=_asset_owner_id(asset),
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
return asset
|
||||||
|
|
||||||
|
|
||||||
|
def restore_asset(session: Session, asset: FileAsset) -> bool:
|
||||||
|
if asset.deleted_at is None:
|
||||||
|
return False
|
||||||
|
collision = _asset_owner_query(session, asset).filter(
|
||||||
|
FileAsset.id != asset.id,
|
||||||
|
FileAsset.display_path == asset.display_path,
|
||||||
|
FileAsset.deleted_at.is_(None),
|
||||||
|
).first()
|
||||||
|
if collision is not None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The file cannot be restored because its path is already in use"
|
||||||
|
)
|
||||||
|
asset.deleted_at = None
|
||||||
|
asset.lifecycle_revision += 1
|
||||||
|
session.add(asset)
|
||||||
|
session.flush()
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def restore_folder(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
owner_type: str,
|
||||||
|
owner_id: str,
|
||||||
|
user_id: str,
|
||||||
|
path: str,
|
||||||
|
recursive: bool,
|
||||||
|
is_admin: bool,
|
||||||
|
) -> tuple[int, int]:
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
normalized = normalize_folder(path)
|
||||||
|
if not normalized:
|
||||||
|
raise FileStorageError("Folder path is required")
|
||||||
|
prefix = f"{normalized}/"
|
||||||
|
folders = _folder_owner_query(
|
||||||
|
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||||
|
).filter(FileFolder.deleted_at.is_not(None))
|
||||||
|
assets = _asset_owner_query_by_values(
|
||||||
|
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||||
|
).filter(FileAsset.deleted_at.is_not(None))
|
||||||
|
if recursive:
|
||||||
|
folders = folders.filter(
|
||||||
|
(FileFolder.path == normalized) | FileFolder.path.like(f"{prefix}%")
|
||||||
|
)
|
||||||
|
assets = assets.filter(FileAsset.display_path.like(f"{prefix}%"))
|
||||||
|
else:
|
||||||
|
folders = folders.filter(FileFolder.path == normalized)
|
||||||
|
assets = assets.filter(False)
|
||||||
|
folder_rows = folders.order_by(FileFolder.path.asc()).all()
|
||||||
|
asset_rows = assets.order_by(FileAsset.display_path.asc()).all()
|
||||||
|
if not folder_rows:
|
||||||
|
raise FileStorageError("Deleted folder not found")
|
||||||
|
for folder in folder_rows:
|
||||||
|
collision = _folder_owner_query(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=owner_type,
|
||||||
|
owner_id=owner_id,
|
||||||
|
).filter(
|
||||||
|
FileFolder.id != folder.id,
|
||||||
|
FileFolder.path == folder.path,
|
||||||
|
FileFolder.deleted_at.is_(None),
|
||||||
|
).first()
|
||||||
|
if collision is not None:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"The folder cannot be restored because its path is in use: {folder.path}"
|
||||||
|
)
|
||||||
|
for asset in asset_rows:
|
||||||
|
collision = _asset_owner_query(session, asset).filter(
|
||||||
|
FileAsset.id != asset.id,
|
||||||
|
FileAsset.display_path == asset.display_path,
|
||||||
|
FileAsset.deleted_at.is_(None),
|
||||||
|
).first()
|
||||||
|
if collision is not None:
|
||||||
|
raise FileStorageError(
|
||||||
|
f"A file cannot be restored because its path is in use: {asset.display_path}"
|
||||||
|
)
|
||||||
|
for folder in folder_rows:
|
||||||
|
folder.deleted_at = None
|
||||||
|
session.add(folder)
|
||||||
|
for asset in asset_rows:
|
||||||
|
asset.deleted_at = None
|
||||||
|
asset.lifecycle_revision += 1
|
||||||
|
session.add(asset)
|
||||||
|
session.flush()
|
||||||
|
return len(folder_rows), len(asset_rows)
|
||||||
|
|
||||||
|
|
||||||
|
def restore_connector_space(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
user_id: str,
|
||||||
|
space_id: str,
|
||||||
|
is_admin: bool,
|
||||||
|
) -> FileConnectorSpace:
|
||||||
|
space = session.get(FileConnectorSpace, space_id)
|
||||||
|
if space is None or space.tenant_id != tenant_id or space.deleted_at is None:
|
||||||
|
raise FileStorageError("Deleted connector space not found")
|
||||||
|
ensure_owner_access(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
owner_type=space.owner_type,
|
||||||
|
owner_id=connector_space_owner_id(space),
|
||||||
|
user_id=user_id,
|
||||||
|
is_admin=is_admin,
|
||||||
|
)
|
||||||
|
collision = _connector_space_owner_query(session, space).filter(
|
||||||
|
FileConnectorSpace.id != space.id,
|
||||||
|
FileConnectorSpace.label == space.label,
|
||||||
|
FileConnectorSpace.deleted_at.is_(None),
|
||||||
|
).first()
|
||||||
|
if collision is not None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The connector space cannot be restored because its label is in use"
|
||||||
|
)
|
||||||
|
space.deleted_at = None
|
||||||
|
space.is_active = True
|
||||||
|
session.add(space)
|
||||||
|
session.flush()
|
||||||
|
return space
|
||||||
|
|
||||||
|
|
||||||
|
def preview_asset_purge(
|
||||||
|
session: Session, *, tenant_id: str, file_ids: Iterable[str]
|
||||||
|
) -> PurgePreview:
|
||||||
|
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||||
|
if not normalized_ids or len(normalized_ids) > 100:
|
||||||
|
raise FileStorageError("Purge preview must contain between 1 and 100 files")
|
||||||
|
assets = (
|
||||||
|
session.query(FileAsset)
|
||||||
|
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||||
|
.order_by(FileAsset.id.asc())
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
by_id = {asset.id: asset for asset in assets}
|
||||||
|
items: list[PurgePreviewItem] = []
|
||||||
|
for file_id in normalized_ids:
|
||||||
|
asset = by_id.get(file_id)
|
||||||
|
if asset is None:
|
||||||
|
items.append(
|
||||||
|
PurgePreviewItem(
|
||||||
|
file_id=file_id,
|
||||||
|
filename="",
|
||||||
|
lifecycle_revision=0,
|
||||||
|
deleted_at=None,
|
||||||
|
retained_until=None,
|
||||||
|
legal_hold=False,
|
||||||
|
blockers=("not_found",),
|
||||||
|
blob_ids=(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
blockers = _asset_purge_blockers(session, asset)
|
||||||
|
blob_ids = tuple(
|
||||||
|
sorted(
|
||||||
|
row[0]
|
||||||
|
for row in session.query(FileVersion.blob_id)
|
||||||
|
.filter(FileVersion.file_asset_id == asset.id)
|
||||||
|
.distinct()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
PurgePreviewItem(
|
||||||
|
file_id=asset.id,
|
||||||
|
filename=asset.filename,
|
||||||
|
lifecycle_revision=asset.lifecycle_revision,
|
||||||
|
deleted_at=asset.deleted_at,
|
||||||
|
retained_until=asset.retained_until,
|
||||||
|
legal_hold=asset.legal_hold,
|
||||||
|
blockers=tuple(blockers),
|
||||||
|
blob_ids=blob_ids,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
payload = [item.digest_payload() for item in items]
|
||||||
|
digest = hashlib.sha256(
|
||||||
|
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
return PurgePreview(preview_sha256=digest, items=tuple(items))
|
||||||
|
|
||||||
|
|
||||||
|
def execute_asset_purge(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
file_ids: Iterable[str],
|
||||||
|
preview_sha256: str,
|
||||||
|
idempotency_key: str,
|
||||||
|
approval_reference: str,
|
||||||
|
before_commit: Callable[[list[FileAsset]], None] | None = None,
|
||||||
|
) -> PurgeResult:
|
||||||
|
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||||
|
if not normalized_ids or len(normalized_ids) > 100:
|
||||||
|
raise FileStorageError("Purge must contain between 1 and 100 files")
|
||||||
|
request = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"file_ids": normalized_ids,
|
||||||
|
"preview_sha256": preview_sha256,
|
||||||
|
"approval_reference": approval_reference,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
started = begin_durable_recovery_operation(
|
||||||
|
get_database().SessionLocal,
|
||||||
|
identity=process_runtime_identity(),
|
||||||
|
module_id="files",
|
||||||
|
operation_type="asset-hard-purge",
|
||||||
|
idempotency_key=f"files-asset-purge:{idempotency_key}",
|
||||||
|
request=request,
|
||||||
|
recovery_plan=RecoveryPlan(
|
||||||
|
mode=RecoveryMode.IRREVERSIBLE,
|
||||||
|
preconditions=(
|
||||||
|
"the actor holds the dedicated Files purge permission",
|
||||||
|
"the signed-off preview still matches current lifecycle state",
|
||||||
|
"every target is soft-deleted and free of retention or legal-hold blockers",
|
||||||
|
),
|
||||||
|
verification_steps=(
|
||||||
|
"verify every target FileAsset and FileVersion row is absent",
|
||||||
|
"recalculate retained FileBlob reference counts before later garbage collection",
|
||||||
|
),
|
||||||
|
approval_reference=approval_reference,
|
||||||
|
),
|
||||||
|
precondition_evidence={
|
||||||
|
"preview_sha256": preview_sha256,
|
||||||
|
"target_count": len(normalized_ids),
|
||||||
|
},
|
||||||
|
lease_resource_key=f"files:purge:{tenant_id}",
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_asset_batch",
|
||||||
|
resource_id=preview_sha256,
|
||||||
|
metadata={"resources": ["postgresql"], "bounded_target_count": len(normalized_ids)},
|
||||||
|
block_unresolved_resource=True,
|
||||||
|
)
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"Another purge or unresolved retry owns the tenant purge fence"
|
||||||
|
) from exc
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||||
|
raise FileStorageError("The Files recovery ledger is unavailable; nothing was purged") from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
return PurgeResult(
|
||||||
|
recovery_operation_id=started.operation_id,
|
||||||
|
status=started.status,
|
||||||
|
replayed=True,
|
||||||
|
purged_files=0,
|
||||||
|
released_blobs=0,
|
||||||
|
)
|
||||||
|
operation = started.operation
|
||||||
|
try:
|
||||||
|
preview = preview_asset_purge(session, tenant_id=tenant_id, file_ids=normalized_ids)
|
||||||
|
if preview.preview_sha256 != preview_sha256:
|
||||||
|
operation.reject(
|
||||||
|
summary="The purge preview became stale before execution",
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{"preview_matches": False},
|
||||||
|
expected_preview_sha256=preview_sha256,
|
||||||
|
observed_preview_sha256=preview.preview_sha256,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError("The purge preview is stale; preview again")
|
||||||
|
if not preview.eligible:
|
||||||
|
operation.reject(
|
||||||
|
summary="The purge was blocked by current lifecycle policy",
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{"lifecycle_policy_allows": False},
|
||||||
|
blocked_files=[
|
||||||
|
{"file_id": item.file_id, "blockers": list(item.blockers)}
|
||||||
|
for item in preview.items
|
||||||
|
if item.blockers
|
||||||
|
]
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError("One or more files are not eligible for purge")
|
||||||
|
assets = (
|
||||||
|
session.query(FileAsset)
|
||||||
|
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||||
|
.with_for_update()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
blob_ids = sorted({blob_id for item in preview.items for blob_id in item.blob_ids})
|
||||||
|
if before_commit is not None:
|
||||||
|
before_commit(assets)
|
||||||
|
session.query(FileShare).filter(
|
||||||
|
FileShare.file_asset_id.in_(normalized_ids)
|
||||||
|
).delete(synchronize_session=False)
|
||||||
|
session.query(FileVersion).filter(
|
||||||
|
FileVersion.file_asset_id.in_(normalized_ids)
|
||||||
|
).delete(synchronize_session=False)
|
||||||
|
for asset in assets:
|
||||||
|
session.delete(asset)
|
||||||
|
session.flush()
|
||||||
|
released_blobs = 0
|
||||||
|
for blob_id in blob_ids:
|
||||||
|
references = int(
|
||||||
|
session.query(func.count(FileVersion.id))
|
||||||
|
.filter(FileVersion.blob_id == blob_id)
|
||||||
|
.scalar()
|
||||||
|
or 0
|
||||||
|
)
|
||||||
|
blob = session.get(FileBlob, blob_id)
|
||||||
|
if blob is not None:
|
||||||
|
blob.ref_count = references
|
||||||
|
session.add(blob)
|
||||||
|
if references == 0:
|
||||||
|
released_blobs += 1
|
||||||
|
operation.commit_verified_success(
|
||||||
|
session,
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{
|
||||||
|
"assets_absent": True,
|
||||||
|
"versions_absent": True,
|
||||||
|
"blob_reference_counts_recalculated": True,
|
||||||
|
},
|
||||||
|
purged_file_ids=normalized_ids,
|
||||||
|
purged_files=len(assets),
|
||||||
|
released_blobs=released_blobs,
|
||||||
|
preview_sha256=preview_sha256,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return PurgeResult(
|
||||||
|
recovery_operation_id=started.operation_id,
|
||||||
|
status=RecoveryStatus.SUCCEEDED.value,
|
||||||
|
replayed=False,
|
||||||
|
purged_files=len(assets),
|
||||||
|
released_blobs=released_blobs,
|
||||||
|
)
|
||||||
|
except FileStorageError:
|
||||||
|
session.rollback()
|
||||||
|
if not operation.closed:
|
||||||
|
operation.release_unresolved()
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
session.rollback()
|
||||||
|
if not operation.closed:
|
||||||
|
operation.reject(
|
||||||
|
summary="The purge failed before its database transaction committed",
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{"database_transaction_committed": False},
|
||||||
|
exception_type=type(exc).__name__,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def garbage_collect_unreferenced_blobs(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
limit: int,
|
||||||
|
approval_reference: str,
|
||||||
|
) -> BlobGcResult:
|
||||||
|
candidate_rows = (
|
||||||
|
session.query(FileBlob)
|
||||||
|
.filter(FileBlob.tenant_id == tenant_id)
|
||||||
|
.filter(
|
||||||
|
or_(FileBlob.retained_until.is_(None), FileBlob.retained_until <= utcnow())
|
||||||
|
)
|
||||||
|
.filter(
|
||||||
|
~session.query(FileVersion.id)
|
||||||
|
.filter(FileVersion.blob_id == FileBlob.id)
|
||||||
|
.exists()
|
||||||
|
)
|
||||||
|
.order_by(FileBlob.created_at.asc(), FileBlob.id.asc())
|
||||||
|
.limit(limit)
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
candidates = tuple(
|
||||||
|
(row.id, row.storage_key) for row in candidate_rows
|
||||||
|
)
|
||||||
|
# Close the read snapshot before the independent recovery transaction takes
|
||||||
|
# its durable SQLite/PostgreSQL write lock.
|
||||||
|
session.commit()
|
||||||
|
deleted = 0
|
||||||
|
unresolved: list[str] = []
|
||||||
|
backend = get_storage_backend()
|
||||||
|
for candidate_id, storage_key in candidates:
|
||||||
|
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||||
|
request = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"blob_id": candidate_id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"approval_reference": approval_reference,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
started = begin_durable_recovery_operation(
|
||||||
|
get_database().SessionLocal,
|
||||||
|
identity=process_runtime_identity(),
|
||||||
|
module_id="files",
|
||||||
|
operation_type="blob-garbage-collection",
|
||||||
|
idempotency_key=(
|
||||||
|
f"files-blob-gc:{candidate_id}:{key_digest[:12]}:"
|
||||||
|
f"{hashlib.sha256(approval_reference.encode('utf-8')).hexdigest()[:12]}"
|
||||||
|
),
|
||||||
|
request=request,
|
||||||
|
recovery_plan=RecoveryPlan(
|
||||||
|
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||||
|
preconditions=(
|
||||||
|
"the caller holds dedicated Files purge authority",
|
||||||
|
"a fresh database check shows no FileVersion reference",
|
||||||
|
),
|
||||||
|
forward_recovery_steps=(
|
||||||
|
"verify the object is absent",
|
||||||
|
"delete the unreferenced FileBlob row after absence is proven",
|
||||||
|
),
|
||||||
|
verification_steps=(
|
||||||
|
"recheck FileVersion references while holding the blob fence",
|
||||||
|
"probe the exact managed storage key after deletion",
|
||||||
|
),
|
||||||
|
approval_reference=approval_reference,
|
||||||
|
),
|
||||||
|
precondition_evidence={
|
||||||
|
"blob_id": candidate_id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"observed_reference_count": 0,
|
||||||
|
},
|
||||||
|
lease_resource_key=f"files:blob:{tenant_id}:{candidate_id}",
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_blob",
|
||||||
|
resource_id=candidate_id,
|
||||||
|
metadata={"resources": ["postgresql", "object-storage"], "storage_backend": backend.name},
|
||||||
|
block_unresolved_resource=True,
|
||||||
|
)
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict):
|
||||||
|
continue
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; no blob bytes were deleted"
|
||||||
|
) from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
continue
|
||||||
|
operation = started.operation
|
||||||
|
try:
|
||||||
|
blob = (
|
||||||
|
session.query(FileBlob)
|
||||||
|
.filter(FileBlob.id == candidate_id, FileBlob.tenant_id == tenant_id)
|
||||||
|
.with_for_update()
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
references = int(
|
||||||
|
session.query(func.count(FileVersion.id))
|
||||||
|
.filter(FileVersion.blob_id == candidate_id)
|
||||||
|
.scalar()
|
||||||
|
or 0
|
||||||
|
)
|
||||||
|
blob_retained = bool(
|
||||||
|
blob is not None
|
||||||
|
and blob.retained_until is not None
|
||||||
|
and _as_utc(blob.retained_until) > utcnow()
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
session.rollback()
|
||||||
|
if not operation.closed:
|
||||||
|
operation.reject(
|
||||||
|
summary="Blob eligibility could not be rechecked before deletion",
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{"external_effect_started": False},
|
||||||
|
exception_type=type(exc).__name__,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
raise FileStorageError(
|
||||||
|
"Blob eligibility could not be rechecked; no bytes were deleted"
|
||||||
|
) from exc
|
||||||
|
if blob is None or references or blob_retained:
|
||||||
|
operation.reject(
|
||||||
|
summary="Blob garbage collection was no longer eligible",
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{
|
||||||
|
"blob_present": blob is not None,
|
||||||
|
"reference_count_zero": references == 0,
|
||||||
|
"blob_retention_expired": not blob_retained,
|
||||||
|
},
|
||||||
|
reference_count=references,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
session.rollback()
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
backend.delete(blob.storage_key)
|
||||||
|
object_present = backend.exists(blob.storage_key)
|
||||||
|
except (StorageBackendError, OSError) as exc:
|
||||||
|
try:
|
||||||
|
object_present = backend.exists(blob.storage_key)
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
object_present = None
|
||||||
|
session.rollback()
|
||||||
|
if object_present is True:
|
||||||
|
operation.fail(
|
||||||
|
summary="The unreferenced blob object was retained",
|
||||||
|
evidence={"object_present": True, "exception_type": type(exc).__name__},
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
operation.unresolved(
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||||
|
summary="The blob deletion outcome could not be verified",
|
||||||
|
evidence={"object_present": object_present, "exception_type": type(exc).__name__},
|
||||||
|
failure_summary="Reconcile the exact blob key before retrying garbage collection",
|
||||||
|
)
|
||||||
|
unresolved.append(started.operation_id)
|
||||||
|
continue
|
||||||
|
if object_present:
|
||||||
|
session.rollback()
|
||||||
|
operation.fail(
|
||||||
|
summary="The object store did not delete the unreferenced blob",
|
||||||
|
evidence={"object_present": True},
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
session.delete(blob)
|
||||||
|
session.flush()
|
||||||
|
operation.commit_verified_success(
|
||||||
|
session,
|
||||||
|
evidence=_verified_evidence(
|
||||||
|
{
|
||||||
|
"object_absent": True,
|
||||||
|
"database_blob_absent": True,
|
||||||
|
"reference_count_zero": True,
|
||||||
|
},
|
||||||
|
blob_id=candidate_id,
|
||||||
|
storage_key_sha256=key_digest,
|
||||||
|
object_present=False,
|
||||||
|
database_blob_present=False,
|
||||||
|
reference_count=0,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
deleted += 1
|
||||||
|
except Exception as exc:
|
||||||
|
session.rollback()
|
||||||
|
if not operation.closed:
|
||||||
|
operation.unresolved(
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||||
|
summary="The object is absent but blob metadata still requires forward recovery",
|
||||||
|
evidence={"object_present": False, "exception_type": type(exc).__name__},
|
||||||
|
failure_summary="Delete the FileBlob row only after rechecking all FileVersion references",
|
||||||
|
)
|
||||||
|
unresolved.append(started.operation_id)
|
||||||
|
return BlobGcResult(
|
||||||
|
inspected_blobs=len(candidates),
|
||||||
|
deleted_blobs=deleted,
|
||||||
|
unresolved_operation_ids=tuple(unresolved),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_purge_blockers(session: Session, asset: FileAsset) -> list[str]:
|
||||||
|
blockers: list[str] = []
|
||||||
|
if asset.deleted_at is None:
|
||||||
|
blockers.append("not_soft_deleted")
|
||||||
|
if asset.legal_hold:
|
||||||
|
blockers.append("legal_hold")
|
||||||
|
if asset.retained_until is not None and _as_utc(asset.retained_until) > utcnow():
|
||||||
|
blockers.append("retention_active")
|
||||||
|
if _table_exists(session, CampaignAttachmentUse.__tablename__) and session.query(
|
||||||
|
CampaignAttachmentUse.id
|
||||||
|
).filter(CampaignAttachmentUse.file_asset_id == asset.id).first() is not None:
|
||||||
|
blockers.append("campaign_evidence")
|
||||||
|
if _table_exists(session, FileFormEvidenceGrant.__tablename__) and session.query(
|
||||||
|
FileFormEvidenceGrant.id
|
||||||
|
).filter(FileFormEvidenceGrant.file_asset_id == asset.id).first() is not None:
|
||||||
|
blockers.append("form_evidence")
|
||||||
|
if session.query(FileShare.id).filter(
|
||||||
|
FileShare.file_asset_id == asset.id,
|
||||||
|
effective_file_share_clause(),
|
||||||
|
).first() is not None:
|
||||||
|
blockers.append("active_share")
|
||||||
|
return blockers
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_owner_id(asset: FileAsset) -> str:
|
||||||
|
owner_id = asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id
|
||||||
|
if not owner_id:
|
||||||
|
raise FileStorageError("File has no valid owner")
|
||||||
|
return owner_id
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_owner_query(session: Session, asset: FileAsset):
|
||||||
|
return _asset_owner_query_by_values(
|
||||||
|
session,
|
||||||
|
tenant_id=asset.tenant_id,
|
||||||
|
owner_type=asset.owner_type,
|
||||||
|
owner_id=_asset_owner_id(asset),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _asset_owner_query_by_values(
|
||||||
|
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||||
|
):
|
||||||
|
query = session.query(FileAsset).filter(
|
||||||
|
FileAsset.tenant_id == tenant_id, FileAsset.owner_type == owner_type
|
||||||
|
)
|
||||||
|
if owner_type == "user":
|
||||||
|
return query.filter(FileAsset.owner_user_id == owner_id)
|
||||||
|
if owner_type == "group":
|
||||||
|
return query.filter(FileAsset.owner_group_id == owner_id)
|
||||||
|
raise FileStorageError("Files must be owned by a user or group")
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_owner_query(
|
||||||
|
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||||
|
):
|
||||||
|
query = session.query(FileFolder).filter(
|
||||||
|
FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type
|
||||||
|
)
|
||||||
|
if owner_type == "user":
|
||||||
|
return query.filter(FileFolder.owner_user_id == owner_id)
|
||||||
|
if owner_type == "group":
|
||||||
|
return query.filter(FileFolder.owner_group_id == owner_id)
|
||||||
|
raise FileStorageError("Folders must be owned by a user or group")
|
||||||
|
|
||||||
|
|
||||||
|
def _connector_space_owner_query(session: Session, space: FileConnectorSpace):
|
||||||
|
query = session.query(FileConnectorSpace).filter(
|
||||||
|
FileConnectorSpace.tenant_id == space.tenant_id,
|
||||||
|
FileConnectorSpace.owner_type == space.owner_type,
|
||||||
|
)
|
||||||
|
owner_id = connector_space_owner_id(space)
|
||||||
|
if space.owner_type == "user":
|
||||||
|
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||||
|
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||||
|
|
||||||
|
|
||||||
|
def _as_utc(value: datetime) -> datetime:
|
||||||
|
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _iso(value: datetime | None) -> str | None:
|
||||||
|
return _as_utc(value).isoformat() if value is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def _table_exists(session: Session, table_name: str) -> bool:
|
||||||
|
return bool(session.bind is not None and inspect(session.bind).has_table(table_name))
|
||||||
|
|
||||||
|
|
||||||
|
def _verified_evidence(
|
||||||
|
checks: dict[str, object], **details: object
|
||||||
|
) -> dict[str, object]:
|
||||||
|
return {"verified": True, "checks": checks, **details}
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"BlobGcResult",
|
||||||
|
"PurgePreview",
|
||||||
|
"PurgePreviewItem",
|
||||||
|
"PurgeResult",
|
||||||
|
"execute_asset_purge",
|
||||||
|
"garbage_collect_unreferenced_blobs",
|
||||||
|
"get_asset_for_lifecycle",
|
||||||
|
"preview_asset_purge",
|
||||||
|
"restore_asset",
|
||||||
|
"restore_connector_space",
|
||||||
|
"restore_folder",
|
||||||
|
"set_asset_lifecycle",
|
||||||
|
]
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
SOURCE_PROVENANCE_METADATA_KEY = "source_provenance"
|
||||||
|
SOURCE_REVISION_METADATA_KEY = "source_revision"
|
||||||
|
|
||||||
|
_PROVENANCE_STRING_FIELDS = {
|
||||||
|
"source_type",
|
||||||
|
"connector_id",
|
||||||
|
"provider",
|
||||||
|
"external_id",
|
||||||
|
"external_path",
|
||||||
|
"external_url",
|
||||||
|
"revision",
|
||||||
|
"revision_label",
|
||||||
|
"observed_at",
|
||||||
|
"imported_at",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_source_revision(value: object) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
text = str(value).strip()
|
||||||
|
return text or None
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_source_provenance(value: object, *, source_revision: object = None) -> dict[str, Any] | None:
|
||||||
|
if value is None:
|
||||||
|
revision = normalize_source_revision(source_revision)
|
||||||
|
return {"revision": revision} if revision else None
|
||||||
|
if not isinstance(value, Mapping):
|
||||||
|
raise ValueError("source_provenance must be a JSON object")
|
||||||
|
|
||||||
|
payload: dict[str, Any] = {}
|
||||||
|
for field in _PROVENANCE_STRING_FIELDS:
|
||||||
|
normalized = normalize_source_revision(value.get(field))
|
||||||
|
if normalized:
|
||||||
|
payload[field] = normalized
|
||||||
|
|
||||||
|
extra = value.get("metadata")
|
||||||
|
if isinstance(extra, Mapping) and extra:
|
||||||
|
payload["metadata"] = dict(extra)
|
||||||
|
elif extra is not None and extra != "":
|
||||||
|
raise ValueError("source_provenance.metadata must be a JSON object")
|
||||||
|
|
||||||
|
revision = normalize_source_revision(source_revision)
|
||||||
|
if revision:
|
||||||
|
payload["revision"] = revision
|
||||||
|
return payload or None
|
||||||
|
|
||||||
|
|
||||||
|
def source_metadata(
|
||||||
|
*,
|
||||||
|
existing: Mapping[str, Any] | None = None,
|
||||||
|
source_provenance: object = None,
|
||||||
|
source_revision: object = None,
|
||||||
|
) -> dict[str, Any] | None:
|
||||||
|
payload = dict(existing or {})
|
||||||
|
normalized_provenance = normalize_source_provenance(source_provenance, source_revision=source_revision)
|
||||||
|
normalized_revision = normalize_source_revision(source_revision)
|
||||||
|
if normalized_provenance:
|
||||||
|
payload[SOURCE_PROVENANCE_METADATA_KEY] = normalized_provenance
|
||||||
|
if normalized_revision:
|
||||||
|
payload[SOURCE_REVISION_METADATA_KEY] = normalized_revision
|
||||||
|
elif normalized_provenance and normalized_provenance.get("revision"):
|
||||||
|
payload[SOURCE_REVISION_METADATA_KEY] = str(normalized_provenance["revision"])
|
||||||
|
return payload or None
|
||||||
|
|
||||||
|
|
||||||
|
def source_provenance_from_metadata(metadata: Mapping[str, Any] | None) -> dict[str, Any] | None:
|
||||||
|
if not isinstance(metadata, Mapping):
|
||||||
|
return None
|
||||||
|
value = metadata.get(SOURCE_PROVENANCE_METADATA_KEY)
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
return dict(value)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def source_revision_from_metadata(metadata: Mapping[str, Any] | None) -> str | None:
|
||||||
|
if not isinstance(metadata, Mapping):
|
||||||
|
return None
|
||||||
|
revision = normalize_source_revision(metadata.get(SOURCE_REVISION_METADATA_KEY))
|
||||||
|
if revision:
|
||||||
|
return revision
|
||||||
|
provenance = source_provenance_from_metadata(metadata)
|
||||||
|
return normalize_source_revision(provenance.get("revision") if provenance else None)
|
||||||
@@ -0,0 +1,931 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
import hashlib
|
||||||
|
from threading import Lock
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from sqlalchemy import event, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.recovery import (
|
||||||
|
RecoveryGuaranteeError,
|
||||||
|
RecoveryMode,
|
||||||
|
RecoveryOperation,
|
||||||
|
RecoveryPlan,
|
||||||
|
RecoveryStatus,
|
||||||
|
plan_recovery_operation,
|
||||||
|
prepare_recovery_operation,
|
||||||
|
start_recovery_operation,
|
||||||
|
verify_recovery_evidence_chain,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.recovery_runtime import (
|
||||||
|
DurableRecoveryOperation,
|
||||||
|
DurableRecoveryStart,
|
||||||
|
RecoveryOperationBusy,
|
||||||
|
RecoveryOperationStateConflict,
|
||||||
|
begin_durable_recovery_operation,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.runtime_coordination import (
|
||||||
|
RuntimeIdentity,
|
||||||
|
acquire_lease,
|
||||||
|
process_runtime_identity,
|
||||||
|
release_lease,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.session import get_database
|
||||||
|
from govoplan_files.backend.db.models import (
|
||||||
|
FileBlob,
|
||||||
|
FileIntegrityFinding,
|
||||||
|
FileIntegrityScan,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.backends import (
|
||||||
|
StorageBackend,
|
||||||
|
StorageBackendError,
|
||||||
|
StorageObjectMissing,
|
||||||
|
)
|
||||||
|
from govoplan_files.backend.storage.common import FileStorageError
|
||||||
|
|
||||||
|
|
||||||
|
_PENDING_EFFECTS_KEY = "govoplan_files_pending_recovery_effects"
|
||||||
|
_HOOKS_INSTALLED_KEY = "govoplan_files_recovery_hooks_installed"
|
||||||
|
_ROLLBACK_ERRORS_KEY = "govoplan_files_recovery_rollback_errors"
|
||||||
|
_SQLITE_FENCES: set[str] = set()
|
||||||
|
_SQLITE_FENCES_LOCK = Lock()
|
||||||
|
|
||||||
|
|
||||||
|
class _PendingEffect(Protocol):
|
||||||
|
def settle(self, *, committed: bool) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class PendingBlobWrite:
|
||||||
|
operation: DurableRecoveryOperation
|
||||||
|
backend: StorageBackend
|
||||||
|
tenant_id: str
|
||||||
|
blob_id: str
|
||||||
|
storage_key: str
|
||||||
|
semantic_checksum_sha256: str
|
||||||
|
semantic_size_bytes: int
|
||||||
|
protection_discriminator: str
|
||||||
|
created_new: bool
|
||||||
|
expected_storage_checksum_sha256: str | None = None
|
||||||
|
expected_storage_size_bytes: int | None = None
|
||||||
|
expected_envelope_id: str | None = None
|
||||||
|
sqlite_fence_key: str | None = None
|
||||||
|
restart_after_rollback: Callable[[], DurableRecoveryOperation] | None = None
|
||||||
|
|
||||||
|
def prepare_stored_bytes(
|
||||||
|
self,
|
||||||
|
data: bytes,
|
||||||
|
*,
|
||||||
|
envelope_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
"""Retain process-local verification evidence without recording content."""
|
||||||
|
|
||||||
|
self.expected_storage_checksum_sha256 = hashlib.sha256(data).hexdigest()
|
||||||
|
self.expected_storage_size_bytes = len(data)
|
||||||
|
self.expected_envelope_id = envelope_id
|
||||||
|
|
||||||
|
def settle(self, *, committed: bool) -> None:
|
||||||
|
if not committed and self.restart_after_rollback is not None:
|
||||||
|
try:
|
||||||
|
self.operation = self.restart_after_rollback()
|
||||||
|
except Exception:
|
||||||
|
# SQLite cannot make the pre-effect ledger row independent of
|
||||||
|
# the caller transaction. If reconstructing the evidence row
|
||||||
|
# also fails, still avoid leaving a known new orphan behind.
|
||||||
|
if self.created_new:
|
||||||
|
try:
|
||||||
|
self.backend.delete(self.storage_key)
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
pass
|
||||||
|
raise
|
||||||
|
evidence = _blob_write_evidence(self)
|
||||||
|
if _blob_write_complete(evidence):
|
||||||
|
self.operation.succeed(evidence=evidence)
|
||||||
|
return
|
||||||
|
|
||||||
|
if self.created_new and evidence.get("database_blob_present") is False:
|
||||||
|
self._settle_unreferenced_new_object(evidence)
|
||||||
|
return
|
||||||
|
|
||||||
|
if not self.created_new and _object_matches(evidence):
|
||||||
|
if _forward_complete_blob_repair(self):
|
||||||
|
completed = _blob_write_evidence(self)
|
||||||
|
if _blob_write_complete(completed):
|
||||||
|
self.operation.succeed(evidence=completed)
|
||||||
|
return
|
||||||
|
|
||||||
|
if evidence.get("database_blob_present") is True and not _object_matches(
|
||||||
|
evidence
|
||||||
|
):
|
||||||
|
_quarantine_blob_after_failed_verification(self, evidence)
|
||||||
|
evidence = _blob_write_evidence(self)
|
||||||
|
|
||||||
|
status = (
|
||||||
|
RecoveryStatus.OUTCOME_UNKNOWN
|
||||||
|
if not evidence.get("verified")
|
||||||
|
else RecoveryStatus.RECOVERY_REQUIRED
|
||||||
|
)
|
||||||
|
transaction = "committed" if committed else "rolled back"
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=status,
|
||||||
|
summary=f"Managed Files blob write remained unresolved after the database transaction {transaction}",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary=(
|
||||||
|
"The managed object and Files blob metadata require reconciliation"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _settle_unreferenced_new_object(self, evidence: dict[str, object]) -> None:
|
||||||
|
object_present = evidence.get("object_present")
|
||||||
|
if object_present is False:
|
||||||
|
self.operation.reject(
|
||||||
|
summary="The Files blob write left no durable object or database row",
|
||||||
|
evidence=evidence,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if object_present is not True:
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||||
|
summary="The unreferenced Files object could not be probed",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary="Object storage availability prevented upload compensation",
|
||||||
|
)
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self.backend.delete(self.storage_key)
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||||
|
summary="An unreferenced Files object could not be compensated",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary="Delete the unreferenced managed object after verifying that no FileBlob references it",
|
||||||
|
)
|
||||||
|
return
|
||||||
|
recovered = _blob_write_evidence(self)
|
||||||
|
if (
|
||||||
|
recovered.get("verified") is True
|
||||||
|
and recovered.get("database_blob_present") is False
|
||||||
|
and recovered.get("object_present") is False
|
||||||
|
):
|
||||||
|
self.operation.compensate(
|
||||||
|
failure_summary="The Files database transaction did not retain the new blob",
|
||||||
|
failure_evidence=evidence,
|
||||||
|
recovery_evidence=recovered,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||||
|
summary="Files upload compensation could not be verified",
|
||||||
|
evidence=recovered,
|
||||||
|
failure_summary="The unreferenced managed object requires operator reconciliation",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class PendingOrphanCleanup:
|
||||||
|
operation: DurableRecoveryOperation
|
||||||
|
backend: StorageBackend
|
||||||
|
finding_id: str
|
||||||
|
tenant_id: str
|
||||||
|
storage_key: str
|
||||||
|
resolved_by_user_id: str
|
||||||
|
|
||||||
|
def settle(self, *, committed: bool) -> None:
|
||||||
|
del committed
|
||||||
|
evidence = _orphan_cleanup_evidence(self)
|
||||||
|
if _orphan_cleanup_complete(evidence):
|
||||||
|
self.operation.succeed(evidence=evidence)
|
||||||
|
return
|
||||||
|
if (
|
||||||
|
evidence.get("verified") is True
|
||||||
|
and evidence.get("object_present") is True
|
||||||
|
and evidence.get("finding_deleted") is False
|
||||||
|
):
|
||||||
|
self.operation.reject(
|
||||||
|
summary="The orphan object was retained and the cleanup finding stayed open",
|
||||||
|
evidence=evidence,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if evidence.get("object_present") is False:
|
||||||
|
if _forward_complete_orphan_finding(self):
|
||||||
|
completed = _orphan_cleanup_evidence(self)
|
||||||
|
if _orphan_cleanup_complete(completed):
|
||||||
|
self.operation.succeed(evidence=completed)
|
||||||
|
return
|
||||||
|
status = (
|
||||||
|
RecoveryStatus.OUTCOME_UNKNOWN
|
||||||
|
if not evidence.get("verified")
|
||||||
|
else RecoveryStatus.RECOVERY_REQUIRED
|
||||||
|
)
|
||||||
|
self.operation.unresolved(
|
||||||
|
status=status,
|
||||||
|
summary="Files orphan cleanup requires reconciliation",
|
||||||
|
evidence=evidence,
|
||||||
|
failure_summary="Recheck the object and integrity-finding state before another cleanup attempt",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def begin_blob_write_recovery(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend,
|
||||||
|
tenant_id: str,
|
||||||
|
blob_id: str,
|
||||||
|
storage_key: str,
|
||||||
|
semantic_checksum_sha256: str,
|
||||||
|
semantic_size_bytes: int,
|
||||||
|
protection_discriminator: str,
|
||||||
|
created_new: bool,
|
||||||
|
repair_token: str | None = None,
|
||||||
|
) -> PendingBlobWrite:
|
||||||
|
disposition = "create" if created_new else "repair"
|
||||||
|
state_token = repair_token or blob_id
|
||||||
|
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||||
|
idempotency_key = (
|
||||||
|
f"files-blob-{disposition}:{blob_id}:{state_token[:48]}"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
identity = process_runtime_identity()
|
||||||
|
except RuntimeError as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; no object was written"
|
||||||
|
) from exc
|
||||||
|
lease_resource_key = f"files:blob:{tenant_id}:{blob_id}"
|
||||||
|
request = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"blob_id": blob_id,
|
||||||
|
"storage_key": storage_key if created_new else None,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||||
|
"semantic_size_bytes": semantic_size_bytes,
|
||||||
|
"protection_discriminator": protection_discriminator,
|
||||||
|
"disposition": disposition,
|
||||||
|
}
|
||||||
|
recovery_plan = RecoveryPlan(
|
||||||
|
mode=(
|
||||||
|
RecoveryMode.COMPENSATION
|
||||||
|
if created_new
|
||||||
|
else RecoveryMode.FORWARD_RECOVERY
|
||||||
|
),
|
||||||
|
preconditions=(
|
||||||
|
"the caller has Files write authority for the target owner",
|
||||||
|
"the storage key belongs to the tenant Files namespace",
|
||||||
|
"the request records digests rather than file contents",
|
||||||
|
),
|
||||||
|
compensation_steps=(
|
||||||
|
"verify that no FileBlob references the newly reserved key",
|
||||||
|
"delete only that unreferenced key and verify absence",
|
||||||
|
)
|
||||||
|
if created_new
|
||||||
|
else (),
|
||||||
|
forward_recovery_steps=(
|
||||||
|
"verify the expected bytes at the existing blob key",
|
||||||
|
"forward-complete matching integrity metadata or quarantine the blob",
|
||||||
|
)
|
||||||
|
if not created_new
|
||||||
|
else (),
|
||||||
|
verification_steps=(
|
||||||
|
"reload FileBlob metadata through an independent session",
|
||||||
|
"stream and hash the managed object independently",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
precondition_evidence = {
|
||||||
|
"blob_id": blob_id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||||
|
"semantic_size_bytes": semantic_size_bytes,
|
||||||
|
"created_new": created_new,
|
||||||
|
}
|
||||||
|
metadata = {
|
||||||
|
"resources": ["postgresql", "object-storage"],
|
||||||
|
"storage_backend": backend.name,
|
||||||
|
"durability_mode": "independent_transaction",
|
||||||
|
}
|
||||||
|
session_factory = get_database().SessionLocal
|
||||||
|
sqlite_mode = session.get_bind().dialect.name == "sqlite"
|
||||||
|
sqlite_fence_key: str | None = None
|
||||||
|
|
||||||
|
def start_independent(*, reconstructed_after_rollback: bool = False) -> DurableRecoveryStart:
|
||||||
|
return begin_durable_recovery_operation(
|
||||||
|
session_factory,
|
||||||
|
identity=identity,
|
||||||
|
module_id="files",
|
||||||
|
operation_type=f"blob-{disposition}",
|
||||||
|
idempotency_key=idempotency_key,
|
||||||
|
request=request,
|
||||||
|
recovery_plan=recovery_plan,
|
||||||
|
precondition_evidence={
|
||||||
|
**precondition_evidence,
|
||||||
|
**(
|
||||||
|
{
|
||||||
|
"sqlite_caller_transaction_rolled_back": True,
|
||||||
|
"effect_may_have_preceded_durable_intent": True,
|
||||||
|
}
|
||||||
|
if reconstructed_after_rollback
|
||||||
|
else {}
|
||||||
|
),
|
||||||
|
},
|
||||||
|
lease_resource_key=lease_resource_key,
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_blob",
|
||||||
|
resource_id=blob_id,
|
||||||
|
metadata={
|
||||||
|
**metadata,
|
||||||
|
"durability_mode": (
|
||||||
|
"sqlite_post_rollback_reconstruction"
|
||||||
|
if reconstructed_after_rollback
|
||||||
|
else "independent_transaction"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
if sqlite_mode:
|
||||||
|
_reserve_sqlite_fence(lease_resource_key)
|
||||||
|
sqlite_fence_key = lease_resource_key
|
||||||
|
started = _begin_caller_transaction_recovery_operation(
|
||||||
|
session,
|
||||||
|
session_factory=session_factory,
|
||||||
|
identity=identity,
|
||||||
|
module_id="files",
|
||||||
|
operation_type=f"blob-{disposition}",
|
||||||
|
idempotency_key=idempotency_key,
|
||||||
|
request=request,
|
||||||
|
recovery_plan=recovery_plan,
|
||||||
|
precondition_evidence={
|
||||||
|
**precondition_evidence,
|
||||||
|
"sqlite_caller_transaction": True,
|
||||||
|
"reduced_crash_durability": True,
|
||||||
|
},
|
||||||
|
lease_resource_key=lease_resource_key,
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_blob",
|
||||||
|
resource_id=blob_id,
|
||||||
|
metadata={
|
||||||
|
**metadata,
|
||||||
|
"resources": ["sqlite", "object-storage"],
|
||||||
|
"durability_mode": "sqlite_caller_transaction",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
started = start_independent()
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||||
|
if sqlite_fence_key is not None:
|
||||||
|
_release_sqlite_fence(sqlite_fence_key)
|
||||||
|
raise FileStorageError(
|
||||||
|
"This managed blob is already owned by another recovery operation"
|
||||||
|
) from exc
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||||
|
if sqlite_fence_key is not None:
|
||||||
|
_release_sqlite_fence(sqlite_fence_key)
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; no object was written"
|
||||||
|
) from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
if sqlite_fence_key is not None:
|
||||||
|
_release_sqlite_fence(sqlite_fence_key)
|
||||||
|
raise FileStorageError(
|
||||||
|
"The matching Files blob operation was already completed; reload before retrying"
|
||||||
|
)
|
||||||
|
pending = PendingBlobWrite(
|
||||||
|
operation=started.operation,
|
||||||
|
backend=backend,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
blob_id=blob_id,
|
||||||
|
storage_key=storage_key,
|
||||||
|
semantic_checksum_sha256=semantic_checksum_sha256,
|
||||||
|
semantic_size_bytes=semantic_size_bytes,
|
||||||
|
protection_discriminator=protection_discriminator,
|
||||||
|
created_new=created_new,
|
||||||
|
sqlite_fence_key=sqlite_fence_key,
|
||||||
|
restart_after_rollback=(
|
||||||
|
lambda: _require_started_operation(
|
||||||
|
start_independent(reconstructed_after_rollback=True)
|
||||||
|
)
|
||||||
|
if sqlite_mode
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
)
|
||||||
|
_register_pending_effect(session, pending)
|
||||||
|
return pending
|
||||||
|
|
||||||
|
|
||||||
|
def begin_orphan_cleanup_recovery(
|
||||||
|
session: Session,
|
||||||
|
finding: FileIntegrityFinding,
|
||||||
|
*,
|
||||||
|
backend: StorageBackend,
|
||||||
|
user_id: str,
|
||||||
|
) -> PendingOrphanCleanup:
|
||||||
|
key_digest = hashlib.sha256(finding.storage_key.encode("utf-8")).hexdigest()
|
||||||
|
try:
|
||||||
|
started = begin_durable_recovery_operation(
|
||||||
|
get_database().SessionLocal,
|
||||||
|
identity=process_runtime_identity(),
|
||||||
|
module_id="files",
|
||||||
|
operation_type="integrity-orphan-cleanup",
|
||||||
|
idempotency_key=f"files-orphan-cleanup:{finding.id}",
|
||||||
|
request={
|
||||||
|
"tenant_id": finding.tenant_id,
|
||||||
|
"finding_id": finding.id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
},
|
||||||
|
recovery_plan=RecoveryPlan(
|
||||||
|
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||||
|
preconditions=(
|
||||||
|
"the integrity finding identifies an unreferenced object",
|
||||||
|
"the object key remains inside the completed scan scope",
|
||||||
|
"a fresh database reference check found no FileBlob",
|
||||||
|
),
|
||||||
|
forward_recovery_steps=(
|
||||||
|
"verify object absence independently",
|
||||||
|
"mark the durable finding deleted only after absence is proven",
|
||||||
|
),
|
||||||
|
verification_steps=(
|
||||||
|
"reload the finding and its scan through an independent session",
|
||||||
|
"probe the original storage key through the configured backend",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
precondition_evidence={
|
||||||
|
"finding_id": finding.id,
|
||||||
|
"scan_id": finding.scan_id,
|
||||||
|
"storage_key_sha256": key_digest,
|
||||||
|
"finding_state": finding.state,
|
||||||
|
},
|
||||||
|
lease_resource_key=(
|
||||||
|
f"files:orphan-cleanup:{finding.tenant_id}:{key_digest[:40]}"
|
||||||
|
),
|
||||||
|
lease_ttl_seconds=15 * 60,
|
||||||
|
resource_type="file_integrity_finding",
|
||||||
|
resource_id=finding.id,
|
||||||
|
metadata={
|
||||||
|
"resources": ["postgresql", "object-storage"],
|
||||||
|
"storage_backend": backend.name,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This orphan object is already owned by another recovery operation"
|
||||||
|
) from exc
|
||||||
|
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||||
|
raise FileStorageError(
|
||||||
|
"The Files recovery ledger is unavailable; no object was deleted"
|
||||||
|
) from exc
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
raise FileStorageError(
|
||||||
|
"This orphan cleanup was already completed; reload the finding"
|
||||||
|
)
|
||||||
|
pending = PendingOrphanCleanup(
|
||||||
|
operation=started.operation,
|
||||||
|
backend=backend,
|
||||||
|
finding_id=finding.id,
|
||||||
|
tenant_id=finding.tenant_id,
|
||||||
|
storage_key=finding.storage_key,
|
||||||
|
resolved_by_user_id=user_id,
|
||||||
|
)
|
||||||
|
_register_pending_effect(session, pending)
|
||||||
|
return pending
|
||||||
|
|
||||||
|
|
||||||
|
def _register_pending_effect(session: Session, effect: _PendingEffect) -> None:
|
||||||
|
if not session.in_transaction():
|
||||||
|
session.begin()
|
||||||
|
pending = session.info.setdefault(_PENDING_EFFECTS_KEY, [])
|
||||||
|
pending.append(effect)
|
||||||
|
if session.info.get(_HOOKS_INSTALLED_KEY):
|
||||||
|
return
|
||||||
|
event.listen(session, "after_commit", _after_session_commit)
|
||||||
|
event.listen(session, "after_rollback", _after_session_rollback)
|
||||||
|
session.info[_HOOKS_INSTALLED_KEY] = True
|
||||||
|
|
||||||
|
|
||||||
|
def _after_session_commit(session: Session) -> None:
|
||||||
|
# SQLAlchemy also emits after_commit for a released SAVEPOINT. A Files
|
||||||
|
# batch may open one while creating a later recovery row; settle only when
|
||||||
|
# the outer business transaction has actually become visible.
|
||||||
|
if session.in_nested_transaction():
|
||||||
|
return
|
||||||
|
_settle_pending_effects(session, committed=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _after_session_rollback(session: Session) -> None:
|
||||||
|
if session.in_nested_transaction():
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
_settle_pending_effects(session, committed=False)
|
||||||
|
except RecoveryGuaranteeError as exc:
|
||||||
|
session.info.setdefault(_ROLLBACK_ERRORS_KEY, []).append(str(exc))
|
||||||
|
|
||||||
|
|
||||||
|
def _settle_pending_effects(session: Session, *, committed: bool) -> None:
|
||||||
|
pending = list(session.info.pop(_PENDING_EFFECTS_KEY, []))
|
||||||
|
failures: list[Exception] = []
|
||||||
|
for effect in pending:
|
||||||
|
try:
|
||||||
|
effect.settle(committed=committed)
|
||||||
|
except Exception as exc: # preserve every effect's chance to settle
|
||||||
|
failures.append(exc)
|
||||||
|
operation = getattr(effect, "operation", None)
|
||||||
|
if operation is not None and not operation.closed:
|
||||||
|
try:
|
||||||
|
operation.release_unresolved()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
sqlite_fence_key = getattr(effect, "sqlite_fence_key", None)
|
||||||
|
if sqlite_fence_key:
|
||||||
|
_release_sqlite_fence(sqlite_fence_key)
|
||||||
|
if failures:
|
||||||
|
raise RecoveryGuaranteeError(
|
||||||
|
f"{len(failures)} Files recovery operation(s) could not be finalized"
|
||||||
|
) from failures[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _reserve_sqlite_fence(resource_key: str) -> None:
|
||||||
|
with _SQLITE_FENCES_LOCK:
|
||||||
|
if resource_key in _SQLITE_FENCES:
|
||||||
|
raise RecoveryOperationBusy(
|
||||||
|
f"Another local SQLite transaction owns {resource_key}"
|
||||||
|
)
|
||||||
|
_SQLITE_FENCES.add(resource_key)
|
||||||
|
|
||||||
|
|
||||||
|
def _release_sqlite_fence(resource_key: str) -> None:
|
||||||
|
with _SQLITE_FENCES_LOCK:
|
||||||
|
_SQLITE_FENCES.discard(resource_key)
|
||||||
|
|
||||||
|
|
||||||
|
def _require_started_operation(started: DurableRecoveryStart) -> DurableRecoveryOperation:
|
||||||
|
if started.replayed or started.operation is None:
|
||||||
|
raise RecoveryOperationStateConflict(started.operation_id, started.status)
|
||||||
|
return started.operation
|
||||||
|
|
||||||
|
|
||||||
|
def _begin_caller_transaction_recovery_operation(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
session_factory: Callable[[], Session],
|
||||||
|
identity: RuntimeIdentity,
|
||||||
|
module_id: str,
|
||||||
|
operation_type: str,
|
||||||
|
idempotency_key: str,
|
||||||
|
request: dict[str, object],
|
||||||
|
recovery_plan: RecoveryPlan,
|
||||||
|
precondition_evidence: dict[str, object],
|
||||||
|
lease_resource_key: str,
|
||||||
|
lease_ttl_seconds: int,
|
||||||
|
resource_type: str,
|
||||||
|
resource_id: str,
|
||||||
|
metadata: dict[str, object],
|
||||||
|
) -> DurableRecoveryStart:
|
||||||
|
"""Start SQLite recovery evidence inside the caller transaction.
|
||||||
|
|
||||||
|
SQLite permits only one writer, so opening the normal independent ledger
|
||||||
|
transaction after an earlier member of a batch has written will deadlock
|
||||||
|
until the busy timeout. The caller-transaction mode preserves fencing,
|
||||||
|
request hashing, checkpoints, commit verification, and rollback
|
||||||
|
compensation, but cannot make the intent survive a hard process loss
|
||||||
|
before the caller commits. PostgreSQL never uses this reduced mode.
|
||||||
|
"""
|
||||||
|
|
||||||
|
claim = acquire_lease(
|
||||||
|
session,
|
||||||
|
installation_id=identity.installation_id,
|
||||||
|
resource_key=lease_resource_key,
|
||||||
|
holder_node_id=identity.node_id,
|
||||||
|
holder_incarnation=identity.incarnation,
|
||||||
|
ttl_seconds=lease_ttl_seconds,
|
||||||
|
metadata={
|
||||||
|
"module_id": module_id,
|
||||||
|
"operation_type": operation_type,
|
||||||
|
"durability_mode": "sqlite_caller_transaction",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
if claim is None:
|
||||||
|
raise RecoveryOperationBusy(
|
||||||
|
f"Another runtime owns the recovery fence for {lease_resource_key}"
|
||||||
|
)
|
||||||
|
existing = session.execute(
|
||||||
|
select(RecoveryOperation).where(
|
||||||
|
RecoveryOperation.installation_id == identity.installation_id,
|
||||||
|
RecoveryOperation.module_id == module_id,
|
||||||
|
RecoveryOperation.idempotency_key == idempotency_key,
|
||||||
|
)
|
||||||
|
).scalar_one_or_none()
|
||||||
|
operation = plan_recovery_operation(
|
||||||
|
session,
|
||||||
|
installation_id=identity.installation_id,
|
||||||
|
module_id=module_id,
|
||||||
|
operation_type=operation_type,
|
||||||
|
idempotency_key=idempotency_key,
|
||||||
|
request=request,
|
||||||
|
recovery_plan=recovery_plan,
|
||||||
|
resource_type=resource_type,
|
||||||
|
resource_id=resource_id,
|
||||||
|
lease_claim=claim,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
if existing is not None:
|
||||||
|
if operation.status == RecoveryStatus.SUCCEEDED.value:
|
||||||
|
release_lease(session, claim)
|
||||||
|
return DurableRecoveryStart(
|
||||||
|
operation_id=operation.id,
|
||||||
|
status=operation.status,
|
||||||
|
replayed=True,
|
||||||
|
operation=None,
|
||||||
|
)
|
||||||
|
raise RecoveryOperationStateConflict(operation.id, operation.status)
|
||||||
|
prepare_recovery_operation(
|
||||||
|
session,
|
||||||
|
operation,
|
||||||
|
evidence=precondition_evidence,
|
||||||
|
lease_claim=claim,
|
||||||
|
)
|
||||||
|
start_recovery_operation(
|
||||||
|
session,
|
||||||
|
operation,
|
||||||
|
evidence={"lease_resource_key": lease_resource_key},
|
||||||
|
lease_claim=claim,
|
||||||
|
)
|
||||||
|
if not verify_recovery_evidence_chain(session, operation.id):
|
||||||
|
raise RecoveryGuaranteeError(
|
||||||
|
"Recovery checkpoint chain verification failed before side effects"
|
||||||
|
)
|
||||||
|
return DurableRecoveryStart(
|
||||||
|
operation_id=operation.id,
|
||||||
|
status=operation.status,
|
||||||
|
replayed=False,
|
||||||
|
operation=DurableRecoveryOperation(
|
||||||
|
session_factory=session_factory,
|
||||||
|
operation_id=operation.id,
|
||||||
|
lease_claim=claim,
|
||||||
|
lease_ttl_seconds=lease_ttl_seconds,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _blob_write_evidence(effect: PendingBlobWrite) -> dict[str, object]:
|
||||||
|
database_blob_present: bool | None
|
||||||
|
database_matches: bool | None
|
||||||
|
database_integrity_verified: bool | None
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
blob = session.get(FileBlob, effect.blob_id)
|
||||||
|
database_blob_present = blob is not None
|
||||||
|
database_matches = bool(
|
||||||
|
blob is not None
|
||||||
|
and blob.tenant_id == effect.tenant_id
|
||||||
|
and blob.storage_key == effect.storage_key
|
||||||
|
and blob.checksum_sha256 == effect.semantic_checksum_sha256
|
||||||
|
and blob.size_bytes == effect.semantic_size_bytes
|
||||||
|
and blob.protection_discriminator
|
||||||
|
== effect.protection_discriminator
|
||||||
|
and blob.encryption_envelope_id == effect.expected_envelope_id
|
||||||
|
and (
|
||||||
|
effect.expected_storage_checksum_sha256
|
||||||
|
== (
|
||||||
|
blob.storage_checksum_sha256
|
||||||
|
or blob.checksum_sha256
|
||||||
|
)
|
||||||
|
)
|
||||||
|
) if blob is not None else False
|
||||||
|
database_integrity_verified = bool(
|
||||||
|
blob is not None
|
||||||
|
and blob.integrity_status == "verified"
|
||||||
|
and blob.quarantined_at is None
|
||||||
|
) if blob is not None else False
|
||||||
|
except Exception:
|
||||||
|
database_blob_present = None
|
||||||
|
database_matches = None
|
||||||
|
database_integrity_verified = None
|
||||||
|
|
||||||
|
object_present: bool | None
|
||||||
|
observed_size: int | None = None
|
||||||
|
observed_checksum: str | None = None
|
||||||
|
try:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
observed_size = 0
|
||||||
|
for chunk in effect.backend.iter_bytes(effect.storage_key):
|
||||||
|
observed_size += len(chunk)
|
||||||
|
digest.update(chunk)
|
||||||
|
observed_checksum = digest.hexdigest()
|
||||||
|
object_present = True
|
||||||
|
except StorageObjectMissing:
|
||||||
|
object_present = False
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
object_present = None
|
||||||
|
|
||||||
|
object_size_matches = (
|
||||||
|
observed_size == effect.expected_storage_size_bytes
|
||||||
|
if object_present is True and effect.expected_storage_size_bytes is not None
|
||||||
|
else False if object_present is False else None
|
||||||
|
)
|
||||||
|
object_checksum_matches = (
|
||||||
|
observed_checksum == effect.expected_storage_checksum_sha256
|
||||||
|
if object_present is True
|
||||||
|
and effect.expected_storage_checksum_sha256 is not None
|
||||||
|
else False if object_present is False else None
|
||||||
|
)
|
||||||
|
verified = database_blob_present is not None and object_present is not None
|
||||||
|
return {
|
||||||
|
"verified": verified,
|
||||||
|
"checks": {
|
||||||
|
"database_reloaded": database_blob_present is not None,
|
||||||
|
"object_probed": object_present is not None,
|
||||||
|
"database_matches_request": database_matches,
|
||||||
|
"database_integrity_verified": database_integrity_verified,
|
||||||
|
"object_size_matches": object_size_matches,
|
||||||
|
"object_checksum_matches": object_checksum_matches,
|
||||||
|
},
|
||||||
|
"blob_id": effect.blob_id,
|
||||||
|
"database_blob_present": database_blob_present,
|
||||||
|
"database_matches_request": database_matches,
|
||||||
|
"database_integrity_verified": database_integrity_verified,
|
||||||
|
"object_present": object_present,
|
||||||
|
"observed_size_bytes": observed_size,
|
||||||
|
"observed_checksum_sha256": observed_checksum,
|
||||||
|
"expected_storage_size_bytes": effect.expected_storage_size_bytes,
|
||||||
|
"expected_storage_checksum_sha256": effect.expected_storage_checksum_sha256,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _blob_write_complete(evidence: dict[str, object]) -> bool:
|
||||||
|
return bool(
|
||||||
|
evidence.get("verified") is True
|
||||||
|
and evidence.get("database_matches_request") is True
|
||||||
|
and evidence.get("database_integrity_verified") is True
|
||||||
|
and _object_matches(evidence)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _object_matches(evidence: dict[str, object]) -> bool:
|
||||||
|
checks = evidence.get("checks")
|
||||||
|
return bool(
|
||||||
|
isinstance(checks, dict)
|
||||||
|
and evidence.get("object_present") is True
|
||||||
|
and checks.get("object_size_matches") is True
|
||||||
|
and checks.get("object_checksum_matches") is True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _forward_complete_blob_repair(effect: PendingBlobWrite) -> bool:
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
blob = session.get(FileBlob, effect.blob_id)
|
||||||
|
if (
|
||||||
|
blob is None
|
||||||
|
or blob.tenant_id != effect.tenant_id
|
||||||
|
or blob.storage_key != effect.storage_key
|
||||||
|
or blob.checksum_sha256 != effect.semantic_checksum_sha256
|
||||||
|
or blob.size_bytes != effect.semantic_size_bytes
|
||||||
|
or blob.protection_discriminator
|
||||||
|
!= effect.protection_discriminator
|
||||||
|
or blob.encryption_envelope_id != effect.expected_envelope_id
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
blob.storage_checksum_sha256 = (
|
||||||
|
effect.expected_storage_checksum_sha256
|
||||||
|
if effect.expected_envelope_id
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
blob.storage_size_bytes = (
|
||||||
|
effect.expected_storage_size_bytes
|
||||||
|
if effect.expected_envelope_id
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
blob.integrity_status = "verified"
|
||||||
|
blob.integrity_checked_at = datetime.now(UTC)
|
||||||
|
blob.integrity_failure = None
|
||||||
|
blob.quarantined_at = None
|
||||||
|
session.add(blob)
|
||||||
|
session.commit()
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _quarantine_blob_after_failed_verification(
|
||||||
|
effect: PendingBlobWrite,
|
||||||
|
evidence: dict[str, object],
|
||||||
|
) -> None:
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
blob = session.get(FileBlob, effect.blob_id)
|
||||||
|
if blob is None or blob.storage_key != effect.storage_key:
|
||||||
|
return
|
||||||
|
blob.integrity_status = (
|
||||||
|
"missing"
|
||||||
|
if evidence.get("object_present") is False
|
||||||
|
else "checksum_mismatch"
|
||||||
|
)
|
||||||
|
blob.integrity_checked_at = datetime.now(UTC)
|
||||||
|
blob.integrity_failure = "recovery_verification_failed"
|
||||||
|
blob.quarantined_at = datetime.now(UTC)
|
||||||
|
session.add(blob)
|
||||||
|
session.commit()
|
||||||
|
except Exception:
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def _orphan_cleanup_evidence(effect: PendingOrphanCleanup) -> dict[str, object]:
|
||||||
|
try:
|
||||||
|
object_present: bool | None = effect.backend.exists(effect.storage_key)
|
||||||
|
except (StorageBackendError, OSError):
|
||||||
|
object_present = None
|
||||||
|
finding_present: bool | None
|
||||||
|
finding_deleted: bool | None
|
||||||
|
still_unreferenced: bool | None
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
finding = session.get(FileIntegrityFinding, effect.finding_id)
|
||||||
|
finding_present = finding is not None
|
||||||
|
finding_deleted = bool(
|
||||||
|
finding is not None and finding.state == "deleted"
|
||||||
|
) if finding is not None else False
|
||||||
|
referenced = (
|
||||||
|
session.query(FileBlob.id)
|
||||||
|
.filter(
|
||||||
|
FileBlob.tenant_id == effect.tenant_id,
|
||||||
|
FileBlob.storage_key == effect.storage_key,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
still_unreferenced = referenced is None
|
||||||
|
except Exception:
|
||||||
|
finding_present = None
|
||||||
|
finding_deleted = None
|
||||||
|
still_unreferenced = None
|
||||||
|
verified = object_present is not None and finding_present is not None
|
||||||
|
return {
|
||||||
|
"verified": verified,
|
||||||
|
"checks": {
|
||||||
|
"database_reloaded": finding_present is not None,
|
||||||
|
"object_probed": object_present is not None,
|
||||||
|
"finding_marked_deleted": finding_deleted,
|
||||||
|
"object_absent": (
|
||||||
|
not object_present if object_present is not None else None
|
||||||
|
),
|
||||||
|
"still_unreferenced": still_unreferenced,
|
||||||
|
},
|
||||||
|
"finding_id": effect.finding_id,
|
||||||
|
"finding_present": finding_present,
|
||||||
|
"finding_deleted": finding_deleted,
|
||||||
|
"object_present": object_present,
|
||||||
|
"still_unreferenced": still_unreferenced,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _orphan_cleanup_complete(evidence: dict[str, object]) -> bool:
|
||||||
|
return bool(
|
||||||
|
evidence.get("verified") is True
|
||||||
|
and evidence.get("finding_deleted") is True
|
||||||
|
and evidence.get("object_present") is False
|
||||||
|
and evidence.get("still_unreferenced") is True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _forward_complete_orphan_finding(effect: PendingOrphanCleanup) -> bool:
|
||||||
|
try:
|
||||||
|
with effect.operation.session_factory() as session:
|
||||||
|
finding = session.get(FileIntegrityFinding, effect.finding_id)
|
||||||
|
if (
|
||||||
|
finding is None
|
||||||
|
or finding.tenant_id != effect.tenant_id
|
||||||
|
or finding.storage_key != effect.storage_key
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||||
|
if scan is None or not finding.storage_key.startswith(
|
||||||
|
scan.storage_prefix
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
referenced = (
|
||||||
|
session.query(FileBlob.id)
|
||||||
|
.filter(
|
||||||
|
FileBlob.tenant_id == effect.tenant_id,
|
||||||
|
FileBlob.storage_key == effect.storage_key,
|
||||||
|
)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
if referenced:
|
||||||
|
return False
|
||||||
|
finding.state = "deleted"
|
||||||
|
finding.resolved_at = datetime.now(UTC)
|
||||||
|
finding.resolved_by_user_id = effect.resolved_by_user_id
|
||||||
|
session.add(finding)
|
||||||
|
session.commit()
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"PendingBlobWrite",
|
||||||
|
"PendingOrphanCleanup",
|
||||||
|
"begin_blob_write_recovery",
|
||||||
|
"begin_orphan_cleanup_recovery",
|
||||||
|
]
|
||||||
@@ -0,0 +1,331 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import inspect
|
||||||
|
import socket
|
||||||
|
import threading
|
||||||
|
from functools import lru_cache
|
||||||
|
from importlib import import_module
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from govoplan_core.security.outbound_http import (
|
||||||
|
OutboundHttpError,
|
||||||
|
create_outbound_connection,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SdkPeerPinningError(RuntimeError):
|
||||||
|
"""Raised when an optional SDK cannot be bound to the pinned transport."""
|
||||||
|
|
||||||
|
|
||||||
|
_BOTOCORE_CLIENT_CREATION_LOCK = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def create_pinned_s3_client(**kwargs: Any) -> Any:
|
||||||
|
"""Construct an S3 client whose first and every later socket is pinned.
|
||||||
|
|
||||||
|
Botocore does not expose the HTTP-session class through boto3's public
|
||||||
|
client API. Its endpoint creator does expose that seam, so this function
|
||||||
|
replaces the creator only for the bounded client-construction operation.
|
||||||
|
The lock avoids an unsafe interleaving with another GovOPlaN client
|
||||||
|
construction. Any unrelated botocore client created during the short
|
||||||
|
replacement window also receives the stricter transport.
|
||||||
|
"""
|
||||||
|
|
||||||
|
try:
|
||||||
|
boto3 = import_module("boto3")
|
||||||
|
botocore_args = import_module("botocore.args")
|
||||||
|
except ImportError as exc:
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"S3 connector browsing requires the optional boto3 dependency"
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
pinned_session_cls = _botocore_transport_types()[0]
|
||||||
|
original_creator = getattr(botocore_args, "EndpointCreator", None)
|
||||||
|
if original_creator is None or not hasattr(original_creator, "create_endpoint"):
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed botocore release does not expose the endpoint transport seam required for peer pinning"
|
||||||
|
)
|
||||||
|
|
||||||
|
class PinnedEndpointCreator(original_creator): # type: ignore[misc, valid-type]
|
||||||
|
def create_endpoint(self, *args: Any, **endpoint_kwargs: Any) -> Any:
|
||||||
|
endpoint_kwargs["http_session_cls"] = pinned_session_cls
|
||||||
|
return super().create_endpoint(*args, **endpoint_kwargs)
|
||||||
|
|
||||||
|
with _BOTOCORE_CLIENT_CREATION_LOCK:
|
||||||
|
current_creator = getattr(botocore_args, "EndpointCreator", None)
|
||||||
|
if current_creator is not original_creator:
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"Botocore endpoint construction changed concurrently; refusing to create an unproven S3 client"
|
||||||
|
)
|
||||||
|
botocore_args.EndpointCreator = PinnedEndpointCreator
|
||||||
|
try:
|
||||||
|
session_type = getattr(getattr(boto3, "session", None), "Session", None)
|
||||||
|
if session_type is None:
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed boto3 release does not expose its isolated session constructor"
|
||||||
|
)
|
||||||
|
client = session_type().client("s3", **kwargs)
|
||||||
|
finally:
|
||||||
|
if getattr(botocore_args, "EndpointCreator", None) is PinnedEndpointCreator:
|
||||||
|
botocore_args.EndpointCreator = original_creator
|
||||||
|
|
||||||
|
http_session = getattr(getattr(client, "_endpoint", None), "http_session", None)
|
||||||
|
if http_session is None or not isinstance(http_session, pinned_session_cls):
|
||||||
|
close = getattr(client, "close", None)
|
||||||
|
if callable(close):
|
||||||
|
close()
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"Botocore did not install the required pinned HTTP transport; the S3 client was discarded"
|
||||||
|
)
|
||||||
|
return client
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def _botocore_transport_types() -> tuple[type[Any], type[Any], type[Any]]:
|
||||||
|
try:
|
||||||
|
awsrequest = import_module("botocore.awsrequest")
|
||||||
|
httpsession = import_module("botocore.httpsession")
|
||||||
|
urllib3_exceptions = import_module("urllib3.exceptions")
|
||||||
|
except ImportError as exc:
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"S3 connector browsing requires a compatible botocore HTTP transport"
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
required = (
|
||||||
|
"AWSHTTPConnection",
|
||||||
|
"AWSHTTPSConnection",
|
||||||
|
"AWSHTTPConnectionPool",
|
||||||
|
"AWSHTTPSConnectionPool",
|
||||||
|
)
|
||||||
|
if any(not hasattr(awsrequest, name) for name in required) or not hasattr(
|
||||||
|
httpsession, "URLLib3Session"
|
||||||
|
):
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed botocore release is missing the connection classes required for peer pinning"
|
||||||
|
)
|
||||||
|
|
||||||
|
def pinned_new_connection(connection: Any) -> socket.socket:
|
||||||
|
hostname = str(getattr(connection, "_dns_host", "") or "").strip()
|
||||||
|
port = int(getattr(connection, "port", 0) or 0)
|
||||||
|
if not hostname or not port:
|
||||||
|
raise urllib3_exceptions.NewConnectionError(
|
||||||
|
connection, "Pinned S3 connection is missing its target authority"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
return create_outbound_connection(
|
||||||
|
hostname,
|
||||||
|
port,
|
||||||
|
timeout=getattr(connection, "timeout", None),
|
||||||
|
source_address=getattr(connection, "source_address", None),
|
||||||
|
socket_options=getattr(connection, "socket_options", None),
|
||||||
|
label="S3 connector peer",
|
||||||
|
)
|
||||||
|
except socket.timeout as exc:
|
||||||
|
raise urllib3_exceptions.ConnectTimeoutError(
|
||||||
|
connection,
|
||||||
|
f"Connection to {hostname} timed out while selecting an approved peer",
|
||||||
|
) from exc
|
||||||
|
except (OSError, OutboundHttpError, ValueError) as exc:
|
||||||
|
raise urllib3_exceptions.NewConnectionError(
|
||||||
|
connection,
|
||||||
|
f"S3 connector peer was rejected: {exc}",
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
class PinnedAWSHTTPConnection(awsrequest.AWSHTTPConnection):
|
||||||
|
_new_conn = pinned_new_connection
|
||||||
|
|
||||||
|
class PinnedAWSHTTPSConnection(awsrequest.AWSHTTPSConnection):
|
||||||
|
_new_conn = pinned_new_connection
|
||||||
|
|
||||||
|
class PinnedAWSHTTPConnectionPool(awsrequest.AWSHTTPConnectionPool):
|
||||||
|
ConnectionCls = PinnedAWSHTTPConnection
|
||||||
|
|
||||||
|
class PinnedAWSHTTPSConnectionPool(awsrequest.AWSHTTPSConnectionPool):
|
||||||
|
ConnectionCls = PinnedAWSHTTPSConnection
|
||||||
|
|
||||||
|
class PinnedURLLib3Session(httpsession.URLLib3Session):
|
||||||
|
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||||
|
# A proxy would select the final target outside this process. Until
|
||||||
|
# proxy peer delegation is modeled, connector traffic is direct.
|
||||||
|
kwargs["proxies"] = {}
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
self._pool_classes_by_scheme = {
|
||||||
|
"http": PinnedAWSHTTPConnectionPool,
|
||||||
|
"https": PinnedAWSHTTPSConnectionPool,
|
||||||
|
}
|
||||||
|
manager = getattr(self, "_manager", None)
|
||||||
|
if manager is None or not hasattr(manager, "pool_classes_by_scheme"):
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed botocore pool manager cannot enforce pinned connection classes"
|
||||||
|
)
|
||||||
|
manager.pool_classes_by_scheme = self._pool_classes_by_scheme
|
||||||
|
|
||||||
|
return PinnedURLLib3Session, PinnedAWSHTTPConnection, PinnedAWSHTTPSConnection
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def pinned_smb_connection_cache() -> dict[str, Any]:
|
||||||
|
"""Return the Files-owned cache; unproven process-global sessions are never reused."""
|
||||||
|
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def install_pinned_smb_transport(smbclient: Any) -> Any:
|
||||||
|
"""Install a process-wide fail-closed smbclient session factory.
|
||||||
|
|
||||||
|
smbclient routes initial connections, reconnects, and DFS targets through
|
||||||
|
``smbclient._pool.register_session``. Replacing that single factory with a
|
||||||
|
behavior-compatible implementation ensures every target uses a socket
|
||||||
|
selected by Core's deployment-wide outbound policy.
|
||||||
|
"""
|
||||||
|
|
||||||
|
try:
|
||||||
|
pool = import_module("smbclient._pool")
|
||||||
|
connection_module = import_module("smbprotocol.connection")
|
||||||
|
session_module = import_module("smbprotocol.session")
|
||||||
|
transport_module = import_module("smbprotocol.transport")
|
||||||
|
except ImportError as exc:
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"SMB connector browsing requires the optional smbprotocol dependency"
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
current = getattr(pool, "register_session", None)
|
||||||
|
if getattr(current, "__govoplan_peer_pinned__", False):
|
||||||
|
expected_tcp = getattr(current, "__govoplan_pinned_tcp__", None)
|
||||||
|
if expected_tcp is None or getattr(connection_module, "Tcp", None) is not expected_tcp:
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed SMB transport changed after peer pinning; refusing to reuse the session factory"
|
||||||
|
)
|
||||||
|
return smbclient
|
||||||
|
required_parameters = {
|
||||||
|
"server",
|
||||||
|
"username",
|
||||||
|
"password",
|
||||||
|
"port",
|
||||||
|
"encrypt",
|
||||||
|
"connection_timeout",
|
||||||
|
"connection_cache",
|
||||||
|
"auth_protocol",
|
||||||
|
"require_signing",
|
||||||
|
}
|
||||||
|
if current is None or not required_parameters.issubset(
|
||||||
|
inspect.signature(current).parameters
|
||||||
|
):
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed smbprotocol release does not expose the session seam required for peer pinning"
|
||||||
|
)
|
||||||
|
|
||||||
|
class PinnedTcp(transport_module.Tcp):
|
||||||
|
def connect(self) -> None:
|
||||||
|
with self._sock_lock:
|
||||||
|
if self.connected:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self._sock = create_outbound_connection(
|
||||||
|
self.server,
|
||||||
|
int(self.port),
|
||||||
|
timeout=self.timeout,
|
||||||
|
label="SMB connector peer",
|
||||||
|
)
|
||||||
|
except (OSError, OutboundHttpError, ValueError) as exc:
|
||||||
|
raise ValueError(
|
||||||
|
f"SMB connector peer '{self.server}:{self.port}' was rejected: {exc}"
|
||||||
|
) from exc
|
||||||
|
self._sock.settimeout(None)
|
||||||
|
self.connected = True
|
||||||
|
|
||||||
|
# Connection.connect() instantiates the module-level Tcp symbol on every
|
||||||
|
# reconnect. Replacing it process-wide is deliberate: an SMB connection
|
||||||
|
# created by another code path must become stricter, never bypass Files'
|
||||||
|
# peer boundary.
|
||||||
|
if not hasattr(connection_module, "Tcp"):
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"The installed smbprotocol release cannot install the pinned TCP transport"
|
||||||
|
)
|
||||||
|
connection_module.Tcp = PinnedTcp
|
||||||
|
|
||||||
|
def pinned_register_session(
|
||||||
|
server: str,
|
||||||
|
username: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
port: int = 445,
|
||||||
|
encrypt: bool | None = None,
|
||||||
|
connection_timeout: float = 60,
|
||||||
|
connection_cache: dict[str, Any] | None = None,
|
||||||
|
auth_protocol: str = "negotiate",
|
||||||
|
require_signing: bool = True,
|
||||||
|
) -> Any:
|
||||||
|
cache = pinned_smb_connection_cache() if connection_cache is None else connection_cache
|
||||||
|
connection_key = f"{server.lower()}:{port}"
|
||||||
|
connection = cache.get(connection_key)
|
||||||
|
transport = getattr(connection, "transport", None)
|
||||||
|
if connection is not None and not isinstance(transport, PinnedTcp):
|
||||||
|
disconnect = getattr(connection, "disconnect", None)
|
||||||
|
if callable(disconnect):
|
||||||
|
try:
|
||||||
|
disconnect(close=True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
cache.pop(connection_key, None)
|
||||||
|
connection = None
|
||||||
|
if connection is None or not getattr(connection.transport, "connected", False):
|
||||||
|
connection = connection_module.Connection(
|
||||||
|
pool.ClientConfig().client_guid,
|
||||||
|
server,
|
||||||
|
port,
|
||||||
|
require_signing=require_signing,
|
||||||
|
)
|
||||||
|
connection.transport = PinnedTcp(server, port)
|
||||||
|
connection.connect(timeout=connection_timeout)
|
||||||
|
if not isinstance(connection.transport, PinnedTcp):
|
||||||
|
disconnect = getattr(connection, "disconnect", None)
|
||||||
|
if callable(disconnect):
|
||||||
|
try:
|
||||||
|
disconnect(close=True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
raise SdkPeerPinningError(
|
||||||
|
"smbprotocol replaced the required pinned TCP transport during connection setup"
|
||||||
|
)
|
||||||
|
cache[connection_key] = connection
|
||||||
|
|
||||||
|
session = next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in connection.session_table.values()
|
||||||
|
if username is None or item.username == username
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if session is None:
|
||||||
|
session = session_module.Session(
|
||||||
|
connection,
|
||||||
|
username=username,
|
||||||
|
password=password,
|
||||||
|
require_encryption=(encrypt is True),
|
||||||
|
auth_protocol=auth_protocol,
|
||||||
|
)
|
||||||
|
session.connect()
|
||||||
|
elif encrypt is not None:
|
||||||
|
if session.encrypt_data and not encrypt:
|
||||||
|
raise ValueError(
|
||||||
|
"Cannot disable encryption on an already negotiated session."
|
||||||
|
)
|
||||||
|
if not session.encrypt_data and encrypt:
|
||||||
|
session.encrypt = True
|
||||||
|
return session
|
||||||
|
|
||||||
|
pinned_register_session.__govoplan_peer_pinned__ = True # type: ignore[attr-defined]
|
||||||
|
pinned_register_session.__govoplan_pinned_tcp__ = PinnedTcp # type: ignore[attr-defined]
|
||||||
|
pool.register_session = pinned_register_session
|
||||||
|
if hasattr(smbclient, "register_session"):
|
||||||
|
smbclient.register_session = pinned_register_session
|
||||||
|
return smbclient
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"SdkPeerPinningError",
|
||||||
|
"create_pinned_s3_client",
|
||||||
|
"install_pinned_smb_transport",
|
||||||
|
"pinned_smb_connection_cache",
|
||||||
|
]
|
||||||
@@ -15,24 +15,10 @@ from govoplan_files.backend.storage.common import (
|
|||||||
utcnow,
|
utcnow,
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.files import (
|
from govoplan_files.backend.storage.files import (
|
||||||
_active_asset_at_path,
|
|
||||||
_active_asset_exists,
|
|
||||||
_asset_owner_id,
|
|
||||||
_asset_query_for_owner,
|
|
||||||
_candidate_renamed_path,
|
|
||||||
_copy_asset_to_path,
|
|
||||||
_get_or_create_blob,
|
|
||||||
_next_available_logical_path,
|
|
||||||
_normalize_conflict_strategy,
|
|
||||||
_resolution_by_path,
|
|
||||||
_soft_delete_conflicting_asset,
|
|
||||||
_split_logical_path,
|
|
||||||
_storage_backend_name,
|
|
||||||
_storage_bucket_name,
|
|
||||||
_storage_key,
|
|
||||||
asset_is_audit_relevant,
|
asset_is_audit_relevant,
|
||||||
create_file_asset,
|
create_file_asset,
|
||||||
current_version_and_blob,
|
current_version_and_blob,
|
||||||
|
current_versions_and_blobs,
|
||||||
get_asset_for_user,
|
get_asset_for_user,
|
||||||
list_assets_for_user,
|
list_assets_for_user,
|
||||||
read_asset_bytes,
|
read_asset_bytes,
|
||||||
@@ -41,16 +27,12 @@ from govoplan_files.backend.storage.files import (
|
|||||||
soft_delete_assets,
|
soft_delete_assets,
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.folders import (
|
from govoplan_files.backend.storage.folders import (
|
||||||
_active_folder_exists,
|
|
||||||
_ensure_target_folder_hierarchy,
|
|
||||||
_folder_query_for_owner,
|
|
||||||
_owner_filter,
|
|
||||||
create_folder,
|
create_folder,
|
||||||
list_folders_for_user,
|
list_folders_for_user,
|
||||||
soft_delete_folder,
|
soft_delete_folder,
|
||||||
)
|
)
|
||||||
from govoplan_files.backend.storage.search import match_assets, resolve_patterns
|
from govoplan_files.backend.storage.search import match_assets, resolve_patterns
|
||||||
from govoplan_files.backend.storage.transfers import build_rename_preview, rename_selection, transfer_selection
|
from govoplan_files.backend.storage.transfers import build_rename_preview, legacy_rename_selection_without_owner_context, rename_selection, transfer_selection
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"FileConflictResolution",
|
"FileConflictResolution",
|
||||||
@@ -64,6 +46,7 @@ __all__ = [
|
|||||||
"create_folder",
|
"create_folder",
|
||||||
"create_zip_file",
|
"create_zip_file",
|
||||||
"current_version_and_blob",
|
"current_version_and_blob",
|
||||||
|
"current_versions_and_blobs",
|
||||||
"ensure_group_access",
|
"ensure_group_access",
|
||||||
"ensure_owner_access",
|
"ensure_owner_access",
|
||||||
"extract_zip_upload",
|
"extract_zip_upload",
|
||||||
@@ -74,6 +57,7 @@ __all__ = [
|
|||||||
"match_assets",
|
"match_assets",
|
||||||
"read_asset_bytes",
|
"read_asset_bytes",
|
||||||
"record_campaign_attachment_uses_for_job",
|
"record_campaign_attachment_uses_for_job",
|
||||||
|
"legacy_rename_selection_without_owner_context",
|
||||||
"rename_asset",
|
"rename_asset",
|
||||||
"rename_selection",
|
"rename_selection",
|
||||||
"resolve_patterns",
|
"resolve_patterns",
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user