Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
378f4d6ac5 | ||
|
|
a8c3192c3a | ||
|
|
ce4eaefbfd | ||
|
|
6c3cf1c55e | ||
|
|
95aef18955 | ||
|
|
a9c7a7a40c | ||
|
|
e1361427b8 | ||
|
|
ad55d47645 | ||
|
|
21122e058e | ||
|
|
25140fb0b6 | ||
|
|
e9561a87e1 | ||
|
|
9d7d3f96ec | ||
|
|
44ba0ec9c1 | ||
|
|
1e511e4c15 | ||
|
|
cb37079465 | ||
|
|
04e2bae190 | ||
|
|
aa9f90c71e | ||
|
|
60c3d14f08 | ||
|
|
3a3360650f | ||
|
|
c9f8e9262e | ||
|
|
edee29af51 | ||
|
|
0293e49def | ||
|
|
aaa364b8d4 | ||
|
|
4bb17c0f4c | ||
|
|
92e649477f | ||
|
|
0c68e904cf | ||
|
|
8ec31b16ee | ||
|
|
7e6be4b017 | ||
|
|
04882f1628 | ||
|
|
d8ae506ff8 | ||
|
|
6baf2a421b | ||
|
|
b993d8e31a | ||
|
|
359c4e9570 | ||
|
|
233ce40983 | ||
|
|
f084a0f3a9 | ||
|
|
b752dea610 | ||
|
|
159a012833 | ||
|
|
835eacfc5d | ||
|
|
85606d5580 | ||
|
|
632cc6cf7d | ||
|
|
86a905a3a7 | ||
|
|
5b868272b9 | ||
|
|
93eea839c8 | ||
|
|
b667e7ff0e | ||
|
|
0fd8f6e02a | ||
|
|
2b34f6e305 | ||
|
|
58af1a20a7 | ||
|
|
4722161592 | ||
|
|
1444ba80a0 | ||
|
|
02ef83ecee | ||
|
|
1069f85796 | ||
|
|
1401c78c8a | ||
|
|
b6109245a7 | ||
|
|
f5d40b23c2 | ||
|
|
94ea629635 | ||
|
|
6c8a8c655d | ||
|
|
062ad5ddfb | ||
|
|
9adfa91e74 | ||
|
|
65d8ed80b5 | ||
|
|
cffe161f29 | ||
|
|
5248e7de4a | ||
|
|
d5d0df792b | ||
|
|
15ade8df75 | ||
|
|
f3c485ef61 | ||
|
|
0e36b20a14 | ||
|
|
06e6e7191b | ||
|
|
3449cbc8a5 | ||
|
|
92950af6f4 | ||
|
|
8826cf2890 | ||
|
|
f2dfb6c90e | ||
|
|
3bc1d3489e | ||
|
|
d1051293b2 | ||
|
|
8c5f149f07 | ||
|
|
f3210234d3 | ||
|
|
b8b395e8b5 | ||
|
|
8bf7296bf5 | ||
|
|
620fdda2fc | ||
|
|
58c0441763 | ||
|
|
ba7fb2def7 | ||
|
|
4e7c77135c | ||
|
|
13fd7fc3bd | ||
|
|
18e6c3eb9b | ||
|
|
a5c24fe73e | ||
|
|
ee5b06b1e7 |
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: "Bug"
|
||||
about: "Report a reproducible defect, regression, or incorrect behavior"
|
||||
title: "[Bug] "
|
||||
labels:
|
||||
- type/bug
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
- Repository:
|
||||
- Area/module:
|
||||
- Affected version or commit:
|
||||
|
||||
## Behavior
|
||||
|
||||
Expected:
|
||||
|
||||
Actual:
|
||||
|
||||
## Reproduction
|
||||
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
|
||||
## Evidence
|
||||
|
||||
Logs, screenshots, traces, or failing test output:
|
||||
|
||||
## Verification Target
|
||||
|
||||
Command or workflow that should pass when fixed:
|
||||
@@ -0,0 +1 @@
|
||||
blank_issues_enabled: false
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: "Docs / workflow"
|
||||
about: "Request documentation, process, or developer workflow changes"
|
||||
title: "[Docs] "
|
||||
labels:
|
||||
- type/docs
|
||||
- status/triage
|
||||
- module/files
|
||||
- area/docs
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
- Repository:
|
||||
- Document or workflow:
|
||||
|
||||
## Current State
|
||||
|
||||
What is missing, unclear, duplicated, or stale?
|
||||
|
||||
## Desired State
|
||||
|
||||
What should the docs or workflow make clear?
|
||||
|
||||
## Verification Target
|
||||
|
||||
How should this be checked?
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: "Feature"
|
||||
about: "Propose new user-visible behavior or platform capability"
|
||||
title: "[Feature] "
|
||||
labels:
|
||||
- type/feature
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
What user, operator, or developer problem should this solve?
|
||||
|
||||
## Proposed Capability
|
||||
|
||||
What should exist when this is done?
|
||||
|
||||
## Ownership
|
||||
|
||||
- Owning repository:
|
||||
- Related module repositories:
|
||||
- Extension point or integration boundary:
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ]
|
||||
- [ ]
|
||||
|
||||
## Verification Target
|
||||
|
||||
Command, scenario, or UI flow that should prove completion:
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: "Task"
|
||||
about: "Track implementation, maintenance, or migration work"
|
||||
title: "[Task] "
|
||||
labels:
|
||||
- type/task
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
What needs to be completed?
|
||||
|
||||
## Scope
|
||||
|
||||
- Owning repository:
|
||||
- In-scope:
|
||||
- Out-of-scope:
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ]
|
||||
- [ ]
|
||||
|
||||
## Verification Target
|
||||
|
||||
Command or manual check:
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: "Tech debt"
|
||||
about: "Track cleanup, refactoring, risk reduction, or deferred engineering work"
|
||||
title: "[Debt] "
|
||||
labels:
|
||||
- type/debt
|
||||
- status/triage
|
||||
- module/files
|
||||
---
|
||||
|
||||
## Current Cost
|
||||
|
||||
What does this make harder, riskier, slower, or more fragile?
|
||||
|
||||
## Desired Shape
|
||||
|
||||
What should the code, tests, or architecture look like afterwards?
|
||||
|
||||
## Constraints
|
||||
|
||||
What behavior, compatibility, or module boundary must be preserved?
|
||||
|
||||
## Verification Target
|
||||
|
||||
Focused checks that should pass:
|
||||
@@ -0,0 +1,15 @@
|
||||
## Issue
|
||||
|
||||
Closes #
|
||||
|
||||
## Summary
|
||||
|
||||
-
|
||||
|
||||
## Verification
|
||||
|
||||
-
|
||||
|
||||
## Notes
|
||||
|
||||
Follow-up issues:
|
||||
@@ -0,0 +1,270 @@
|
||||
name: Module Package Release
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_tag:
|
||||
description: Existing protected version tag to publish
|
||||
required: true
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
publish-packages:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
GITEA_REPOSITORY: ${{ gitea.repository }}
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
|
||||
with:
|
||||
node-version: "22"
|
||||
- name: Select and validate protected release tag
|
||||
shell: bash
|
||||
env:
|
||||
REQUESTED_TAG: ${{ inputs.release_tag }}
|
||||
TRIGGER_TAG: ${{ gitea.ref_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tag="${REQUESTED_TAG:-$TRIGGER_TAG}"
|
||||
case "$tag" in
|
||||
v[0-9]*.[0-9]*.[0-9]*) ;;
|
||||
*) echo "Release tag must start with a SemVer-shaped vX.Y.Z value" >&2; exit 1 ;;
|
||||
esac
|
||||
git fetch --force origin "refs/tags/$tag:refs/tags/$tag" refs/heads/main:refs/remotes/origin/main
|
||||
tag_commit="$(git rev-list -n 1 "$tag")"
|
||||
git merge-base --is-ancestor "$tag_commit" refs/remotes/origin/main || {
|
||||
echo "Release tag is not contained in main" >&2
|
||||
exit 1
|
||||
}
|
||||
git checkout --detach "$tag"
|
||||
printf 'RELEASE_TAG=%s\n' "$tag" >> "$GITEA_ENV"
|
||||
printf 'SOURCE_DATE_EPOCH=%s\n' "$(git show -s --format=%ct HEAD)" >> "$GITEA_ENV"
|
||||
- name: Validate package versions
|
||||
run: |
|
||||
python - <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
import os
|
||||
import re
|
||||
import tomllib
|
||||
|
||||
tag = os.environ["RELEASE_TAG"]
|
||||
expected = tag.removeprefix("v")
|
||||
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
|
||||
if project.get("version") != expected:
|
||||
raise SystemExit(f"pyproject version {project.get('version')!r} does not match {tag}")
|
||||
if re.fullmatch(r"govoplan-[a-z0-9-]+", str(project.get("name", ""))) is None:
|
||||
raise SystemExit("Python distribution name must use the govoplan-* namespace")
|
||||
webui = Path("webui/package.json")
|
||||
if webui.is_file():
|
||||
package = json.loads(webui.read_text(encoding="utf-8"))
|
||||
if package.get("version") != expected:
|
||||
raise SystemExit(f"WebUI version {package.get('version')!r} does not match {tag}")
|
||||
if re.fullmatch(r"@govoplan/[a-z0-9-]+-webui", str(package.get("name", ""))) is None:
|
||||
raise SystemExit("WebUI package name must use the @govoplan/*-webui namespace")
|
||||
release = Path("webui/package.release.json")
|
||||
if release.is_file():
|
||||
release_package = json.loads(release.read_text(encoding="utf-8"))
|
||||
if (
|
||||
release_package.get("name") != package.get("name")
|
||||
or release_package.get("version") != expected
|
||||
):
|
||||
raise SystemExit("WebUI release package identity does not match package.json and the release tag")
|
||||
PY
|
||||
- name: Build immutable package artifacts
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m pip install --disable-pip-version-check build==1.5.0 twine==7.0.0
|
||||
rm -rf dist .package-webui
|
||||
python -m build --wheel --outdir dist
|
||||
python -m twine check dist/*.whl
|
||||
if [[ -f webui/package.json ]]; then
|
||||
mkdir .package-webui
|
||||
cp -a webui/. .package-webui/
|
||||
rm -rf .package-webui/node_modules .package-webui/dist
|
||||
if [[ -f .package-webui/package.release.json ]]; then
|
||||
cp .package-webui/package.release.json .package-webui/package.json
|
||||
fi
|
||||
node <<'NODE'
|
||||
const fs = require("node:fs");
|
||||
const path = ".package-webui/package.json";
|
||||
const packageJson = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||
const groups = ["dependencies", "optionalDependencies", "peerDependencies"];
|
||||
for (const group of groups) {
|
||||
for (const [name, specifier] of Object.entries(packageJson[group] || {})) {
|
||||
if (!name.startsWith("@govoplan/")) continue;
|
||||
if (typeof specifier !== "string") {
|
||||
throw new Error(`${group}.${name} must use a string version`);
|
||||
}
|
||||
const packageSlug = name.slice("@govoplan/".length);
|
||||
if (!packageSlug.endsWith("-webui")) {
|
||||
throw new Error(`${group}.${name} is outside the WebUI package namespace`);
|
||||
}
|
||||
const repository = `govoplan-${packageSlug.slice(0, -"-webui".length)}`;
|
||||
const escapedRepository = repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const gitTag = specifier.match(
|
||||
new RegExp(
|
||||
`^git\\+(?:ssh://git@|https://)git\\.add-ideas\\.de/(?:GovOPlaN|add-ideas)/${escapedRepository}\\.git#v([0-9]+\\.[0-9]+\\.[0-9]+)$`,
|
||||
),
|
||||
);
|
||||
if (gitTag) {
|
||||
packageJson[group][name] = gitTag[1];
|
||||
continue;
|
||||
}
|
||||
if (specifier.startsWith("file:") || specifier.startsWith("git+")) {
|
||||
throw new Error(
|
||||
`${group}.${name} must resolve to an exact registry version for publication`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
delete packageJson.private;
|
||||
fs.writeFileSync(path, `${JSON.stringify(packageJson, null, 2)}\n`);
|
||||
NODE
|
||||
npm pkg delete private --prefix .package-webui
|
||||
(cd .package-webui && npm pack --ignore-scripts --pack-destination ../dist)
|
||||
fi
|
||||
python - <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
artifacts = []
|
||||
for path in sorted(Path("dist").iterdir()):
|
||||
if path.suffix not in {".whl", ".tgz"}:
|
||||
continue
|
||||
digest = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
artifacts.append({"filename": path.name, "sha256": digest, "size": path.stat().st_size})
|
||||
payload = {
|
||||
"schema_version": "1",
|
||||
"repository": os.environ["GITEA_REPOSITORY"],
|
||||
"tag": os.environ["RELEASE_TAG"],
|
||||
"commit": subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip(),
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
Path("dist/package-artifacts.json").write_text(
|
||||
json.dumps(payload, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
PY
|
||||
- name: Retain package hash evidence
|
||||
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
|
||||
with:
|
||||
name: module-packages-${{ gitea.ref_name }}
|
||||
path: dist/package-artifacts.json
|
||||
- name: Check immutable registry state
|
||||
shell: bash
|
||||
env:
|
||||
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test -n "$PACKAGE_TOKEN"
|
||||
python - <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tomllib
|
||||
from urllib.error import HTTPError
|
||||
from urllib.parse import quote
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
api_root = "https://git.add-ideas.de/api/v1/packages/GovOPlaN"
|
||||
token = os.environ["PACKAGE_TOKEN"]
|
||||
|
||||
def should_publish(kind, name, version, path):
|
||||
package_url = "/".join(
|
||||
(api_root, kind, quote(name, safe=""), quote(version, safe=""), "files")
|
||||
)
|
||||
request = Request(
|
||||
package_url,
|
||||
headers={"Accept": "application/json", "Authorization": f"token {token}"},
|
||||
)
|
||||
try:
|
||||
with urlopen(request, timeout=30) as response:
|
||||
files = json.load(response)
|
||||
except HTTPError as exc:
|
||||
if exc.code == 404:
|
||||
print(f"{kind} package {name}=={version} is not published yet")
|
||||
return True
|
||||
raise
|
||||
if not isinstance(files, list) or len(files) != 1:
|
||||
raise SystemExit(
|
||||
f"immutable {kind} package {name}=={version} has an unexpected file set"
|
||||
)
|
||||
expected_sha256 = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
if files[0].get("sha256") != expected_sha256:
|
||||
raise SystemExit(
|
||||
f"immutable {kind} package {name}=={version} already exists with a different SHA-256"
|
||||
)
|
||||
print(f"verified existing {kind} package {name}=={version} ({expected_sha256})")
|
||||
return False
|
||||
|
||||
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
|
||||
wheels = tuple(Path("dist").glob("*.whl"))
|
||||
if len(wheels) != 1:
|
||||
raise SystemExit("release build must contain exactly one wheel")
|
||||
publish_pypi = should_publish(
|
||||
"pypi", str(project["name"]), str(project["version"]), wheels[0]
|
||||
)
|
||||
|
||||
tarballs = tuple(Path("dist").glob("*.tgz"))
|
||||
if len(tarballs) > 1:
|
||||
raise SystemExit("release build must contain at most one npm package")
|
||||
publish_npm = False
|
||||
if tarballs:
|
||||
webui = json.loads(
|
||||
Path(".package-webui/package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
publish_npm = should_publish(
|
||||
"npm", str(webui["name"]), str(webui["version"]), tarballs[0]
|
||||
)
|
||||
|
||||
with Path(os.environ["GITEA_ENV"]).open("a", encoding="utf-8") as env_file:
|
||||
env_file.write(f"PUBLISH_PYPI={int(publish_pypi)}\n")
|
||||
env_file.write(f"PUBLISH_NPM={int(publish_npm)}\n")
|
||||
PY
|
||||
- name: Publish wheel and WebUI package
|
||||
shell: bash
|
||||
env:
|
||||
PACKAGE_USERNAME: ${{ secrets.GOVOPLAN_PACKAGE_USERNAME }}
|
||||
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test -n "$PACKAGE_USERNAME"
|
||||
test -n "$PACKAGE_TOKEN"
|
||||
if [[ "$PUBLISH_PYPI" == 1 ]]; then
|
||||
TWINE_USERNAME="$PACKAGE_USERNAME" TWINE_PASSWORD="$PACKAGE_TOKEN" \
|
||||
python -m twine upload --non-interactive \
|
||||
--repository-url https://git.add-ideas.de/api/packages/GovOPlaN/pypi \
|
||||
dist/*.whl
|
||||
else
|
||||
echo "Exact wheel is already present; skipping immutable retry."
|
||||
fi
|
||||
shopt -s nullglob
|
||||
webui_packages=(dist/*.tgz)
|
||||
if (( ${#webui_packages[@]} )) && [[ "$PUBLISH_NPM" == 1 ]]; then
|
||||
npmrc="$(mktemp)"
|
||||
trap 'rm -f "$npmrc"' EXIT
|
||||
chmod 600 "$npmrc"
|
||||
printf '%s\n' \
|
||||
'@govoplan:registry=https://git.add-ideas.de/api/packages/GovOPlaN/npm/' \
|
||||
"//git.add-ideas.de/api/packages/GovOPlaN/npm/:_authToken=$PACKAGE_TOKEN" \
|
||||
> "$npmrc"
|
||||
NPM_CONFIG_USERCONFIG="$npmrc" npm publish "./${webui_packages[0]}" \
|
||||
--ignore-scripts --access public \
|
||||
--registry https://git.add-ideas.de/api/packages/GovOPlaN/npm/
|
||||
elif (( ${#webui_packages[@]} )); then
|
||||
echo "Exact WebUI package is already present; skipping immutable retry."
|
||||
fi
|
||||
+18
@@ -327,3 +327,21 @@ dist
|
||||
# Built Visual Studio Code Extensions
|
||||
*.vsix
|
||||
|
||||
# GovOPlaN shared ignore rules from govoplan-core
|
||||
# Local WebUI test/build scratch directories
|
||||
.component-test-build/
|
||||
.module-test-build/
|
||||
.policy-test-build/
|
||||
.template-preview-test-build/
|
||||
.import-test-build/
|
||||
webui/.component-test-build/
|
||||
webui/.module-test-build/
|
||||
webui/.policy-test-build/
|
||||
webui/.template-preview-test-build/
|
||||
webui/.import-test-build/
|
||||
*.db
|
||||
# GovOPlaN local runtime state
|
||||
runtime/
|
||||
# GovOPlaN WebUI test output
|
||||
webui/.module-test-build/
|
||||
webui/.component-test-build/
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# GovOPlaN Files Codex Guide
|
||||
|
||||
## Documentation Contract
|
||||
|
||||
- Treat documentation as part of every behavior change. Update this module's manifest-driven `DocumentationTopic` contributions for affected user and administrator behavior.
|
||||
- Keep feature content here; `govoplan-docs` projects it without importing Files internals.
|
||||
- Maintain a static user/admin baseline and run `/mnt/DATA/git/govoplan/tools/checks/check-manifest-shapes.py` after behavior or manifest changes.
|
||||
|
||||
## Scope
|
||||
|
||||
This repository owns the `files` module: managed file storage APIs, file metadata, shares, uploads/downloads, folder and pattern helpers, backend module manifest, and `@govoplan/files-webui`.
|
||||
@@ -18,8 +24,8 @@ cd /mnt/DATA/git/govoplan-core
|
||||
For combined checks, run:
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-core
|
||||
./scripts/check-focused.sh
|
||||
cd /mnt/DATA/git/govoplan
|
||||
tools/checks/check-focused.sh
|
||||
```
|
||||
|
||||
For WebUI permutation checks that include files:
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# govoplan-files
|
||||
|
||||
<!-- govoplan-repository-type:start -->
|
||||
**Repository type:** module (domain).
|
||||
<!-- govoplan-repository-type:end -->
|
||||
|
||||
GovOPlaN Files is the managed file module. It bundles backend storage APIs and the Files WebUI package so file features can be installed as one module.
|
||||
|
||||
## Ownership
|
||||
@@ -46,10 +50,166 @@ Frontend package:
|
||||
@govoplan/files-webui
|
||||
```
|
||||
|
||||
The campaign module can integrate with files when both modules are installed, for example for managed attachment selection and campaign file sharing.
|
||||
The campaign module can integrate with files when both modules are installed,
|
||||
for example for managed attachment selection and campaign file sharing. Files
|
||||
does not import campaign internals; campaign share/existence checks use the core
|
||||
`campaigns.access` capability registered by the campaign module.
|
||||
|
||||
Files also publishes the optional `privacy.dsar.files` capability. The Core
|
||||
data-subject workflow can use it to collect bounded, tenant-scoped file,
|
||||
version, share, folder, evidence, and non-secret configuration metadata for a
|
||||
direct membership subject. The provider may revoke a subject-targeted share or
|
||||
detach mutable actor references idempotently, but it never exports credential
|
||||
material or raw bytes and never bypasses Files retention, legal hold, evidence,
|
||||
purge approval, audit, or recovery controls. See the handbook's data-subject
|
||||
request coverage section for the review and erasure boundary.
|
||||
|
||||
Platform RBAC and governance rules are documented in `govoplan-core/docs/`.
|
||||
|
||||
Managed files can carry source provenance for connector and import workflows.
|
||||
Upload callers may provide `source_provenance_json` and `source_revision`; the
|
||||
module stores those values under file metadata, returns normalized
|
||||
`source_provenance` and `source_revision` fields in file responses, and carries
|
||||
them into managed campaign attachment matches for frozen execution evidence.
|
||||
|
||||
Connector policy preflight is available through
|
||||
`POST /api/v1/files/connector-policy/evaluate`, and provenance-bearing uploads
|
||||
can pass `connector_policy_json` to enforce the same policy before file content
|
||||
is read. Policy payloads contain ordered `sources`; each source has
|
||||
`scope_type`, optional `scope_id`, optional `label`, and a `policy` object. The
|
||||
policy object supports `allow`/`allowlist`/`whitelist` and
|
||||
`deny`/`denylist`/`blacklist` rules for `connectors`, `providers`,
|
||||
`external_ids`, `external_paths`, and `external_urls`. Deny rules win across the
|
||||
hierarchy; allow rules narrow access at each source that defines them.
|
||||
|
||||
Connector endpoint settings are exposed through governed connector profiles.
|
||||
Profiles can be supplied as JSON through
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`,
|
||||
`endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and
|
||||
deployment-owned credential references such as `password_env`, `token_env`, or
|
||||
`secret_ref`. Environment references require an exact name in the deployment-wide
|
||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
|
||||
process environment variables and may use only Files-owned encrypted password or
|
||||
token values. API-created `secret_ref` values fail closed until Files has an
|
||||
ownership contract that can confirm provider-side deletion. Legacy external
|
||||
references are treated as non-owned: deleting a profile or credential detaches
|
||||
and audits the reference but never passes it to an arbitrary secret provider.
|
||||
Profile and credential deletion immediately clears encrypted values, credential
|
||||
identities, deployment references, and private metadata in the same transaction
|
||||
as the non-secret audit record; an audit failure rolls the deletion back.
|
||||
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
|
||||
current principal (system, tenant, user, group, or accessible campaign scope) and
|
||||
redacts secret values and environment variable names. Use the returned
|
||||
`policy_sources` with connector policy preflight before importing files.
|
||||
`GET /api/v1/files/connectors/providers` exposes provider descriptors for
|
||||
Seafile, Nextcloud, WebDAV, SMB, NFS, and local filesystem connectors. The
|
||||
descriptor declares implementation status, optional dependencies, permission
|
||||
mapping, sync/index strategy, conflict handling, preview behavior, and audit
|
||||
events so provider coverage remains visible without forcing every optional
|
||||
protocol dependency to be installed.
|
||||
`GET /api/v1/files/connectors/profiles/{profile_id}/browse` provides read-only
|
||||
connector browsing. Browse entries use a shared `library`/`folder`/`file` shape,
|
||||
run profile policy with the `browse` operation, and support Seafile,
|
||||
WebDAV/Nextcloud, and SMB when the optional `smb` extra is installed. Seafile
|
||||
profiles browse libraries and directories via Seafile's read-only API; profiles
|
||||
can still opt into WebDAV browsing by setting `metadata.webdav_endpoint_url` or
|
||||
`metadata.browse_protocol` to `webdav`. S3-compatible profiles support bucket
|
||||
and prefix browsing when the optional `s3` extra is installed.
|
||||
`POST /api/v1/files/connectors/profiles/{profile_id}/import` imports a Seafile
|
||||
or WebDAV/Nextcloud/SMB file into managed storage through the same governance,
|
||||
conflict handling, source provenance, and connector audit path as direct
|
||||
uploads. The Seafile provider uses account-token auth and the native file
|
||||
download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET`
|
||||
requests against the configured WebDAV endpoint. SMB profiles use
|
||||
`smb://server[:port]/share[/path]` endpoints and deployment-owned or encrypted
|
||||
stored credentials through `smbprotocol`. Files installs a pinned transport for
|
||||
every initial session, reconnect, alias, and DFS referral target. The deployment
|
||||
private-network policy is re-evaluated immediately before each socket opens.
|
||||
|
||||
S3 connector browse/import binds botocore HTTP and HTTPS pools to the same pinned
|
||||
socket policy. Retries, redirects, endpoint discovery, bucket aliases, and new
|
||||
connections are therefore revalidated while TLS keeps the configured hostname
|
||||
for SNI and certificate checks. Connector clients do not use outbound proxies or
|
||||
ambient AWS credential discovery; configure credentials on the governed profile,
|
||||
or explicitly use an anonymous profile for public objects. Keep connector spaces
|
||||
read-only unless a remote write is explicitly needed.
|
||||
An S3 profile with the `write` capability can back a two-way connector space.
|
||||
`POST /api/v1/files/connector-spaces/{space_id}/write-back` then conditionally
|
||||
creates or replaces one remote object from a managed file, records durable
|
||||
recovery intent before the provider effect, and verifies request/content markers
|
||||
afterward. Automatic remote delete, rename, move, and ACL propagation remain
|
||||
disabled; other connector providers remain read-only.
|
||||
An incompatible SDK upgrade fails closed before a usable client/session is
|
||||
returned.
|
||||
|
||||
Durable platform storage has a separate deployment boundary: installer-owned
|
||||
Garage is accepted only at its exact generated endpoint, while an
|
||||
operator-selected external backend requires a clean HTTPS origin and
|
||||
`FILE_STORAGE_S3_ENDPOINT_TRUSTED=true`. That flag is deployment configuration,
|
||||
cannot be supplied through a Files connector profile, and does not replace
|
||||
operator responsibility for DNS, certificates, egress, bucket policy,
|
||||
versioning, and recovery.
|
||||
|
||||
The actual local/S3 backend implementation is owned by Core so Files, Campaign,
|
||||
and workers resolve the same object namespace. Files owns file metadata and key
|
||||
layout. Node-local storage is supported only for `local` or one-host
|
||||
`host-shared` profiles; multi-host `shared` deployments require S3.
|
||||
|
||||
Destructive Files-module retirement applies the same credential lifecycle before
|
||||
dropping tables. Every remaining Files-owned encrypted connector secret is
|
||||
scrubbed and audited first, while legacy non-owned external references are
|
||||
detached and identified as such in the audit record. Retirement does not claim
|
||||
or attempt provider-side deletion for references Files cannot prove it owns.
|
||||
|
||||
Local connector development assets live in `dev/connectors/`. The compose stack
|
||||
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
|
||||
manual interoperability testing.
|
||||
|
||||
Connector and collaboration ownership boundaries are documented in
|
||||
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
|
||||
The role-adaptive user, administration, integration, and operator guide is the
|
||||
[Files handbook](docs/FILES_HANDBOOK.md).
|
||||
The Files route, connector surfaces, consequence classes, and pattern-language
|
||||
verification are recorded in
|
||||
[Files interface pattern migration](docs/INTERFACE_PATTERN_MIGRATION.md).
|
||||
|
||||
Archive imports use a two-phase preview and confirmation flow for ZIP, TAR,
|
||||
TAR.GZ, TAR.BZ2, and TAR.XZ. Requests are spooled to bounded temporary files;
|
||||
the server validates paths, entry count, expanded size, and expansion ratio,
|
||||
then returns a 30-minute tenant/user-bound preview token. Confirmation reuploads
|
||||
the original archive and stores only the selected members. Password-protected
|
||||
ZIP passwords remain request-only and are never included in the preview token.
|
||||
|
||||
Managed blob writes and applied orphan cleanup use Core's durable recovery
|
||||
ledger. On PostgreSQL, intent, request digests, recovery mode, and a distributed
|
||||
lease are committed before physical storage effects; the Files session commit
|
||||
verifies database and streamed object evidence, while rollback compensates only
|
||||
a newly reserved unreferenced key. Development SQLite records blob intent in
|
||||
the caller transaction to avoid its second-writer deadlock, then verifies on
|
||||
commit or reconstructs compensation evidence after handled rollback. Because a
|
||||
hard loss before that commit can leave an unrecorded object, SQLite requires a
|
||||
complete integrity scan after a crash and is not a production recovery profile.
|
||||
New object keys are opaque and do not retain the uploaded filename. Uncertain
|
||||
or mismatched effects remain visible through Ops.
|
||||
|
||||
Operators with `files:file:admin` can run bounded, resumable integrity scans in
|
||||
Administration. Scan batches and finding actions carry monotonic revisions;
|
||||
stale resume, recheck, or cleanup requests fail before touching object storage.
|
||||
Orphan cleanup always requires a dry-run preview followed by separate
|
||||
confirmation and records recovery-ledger evidence.
|
||||
|
||||
Bulk rename and transfer APIs are owner-scoped: callers must provide the active
|
||||
user or group file space with `owner_type` and `owner_id`. The storage layer
|
||||
keeps a named legacy file-only helper for historical callers that lack owner
|
||||
context, and regression tests cover its write-access checks.
|
||||
|
||||
Optional producer modules can persist generated output through the provider-
|
||||
neutral `files.artifact_store` capability. Files remains responsible for upload
|
||||
authorization, ownership, path normalization, versioning, and blob storage;
|
||||
producers receive stable file/version references without importing Files
|
||||
internals. See [Generated Artifact Store](docs/GENERATED_ARTIFACT_STORE.md).
|
||||
|
||||
## Release packaging
|
||||
|
||||
The repository root includes a `package.json` for git-based WebUI installs. It exports the package `@govoplan/files-webui` from `webui/src` so release builds can depend on tagged git refs instead of local `file:` paths.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
NEXTCLOUD_HOST_PORT=9081
|
||||
NEXTCLOUD_DB_ROOT_PASSWORD=govoplan-nextcloud-root
|
||||
NEXTCLOUD_DB_PASSWORD=govoplan-nextcloud
|
||||
NEXTCLOUD_ADMIN_USER=admin
|
||||
NEXTCLOUD_ADMIN_PASSWORD=govoplan-nextcloud-admin
|
||||
|
||||
SEAFILE_HOST_PORT=9082
|
||||
SEAFILE_MYSQL_ROOT_PASSWORD=govoplan-seafile-root
|
||||
SEAFILE_ADMIN_EMAIL=admin@example.local
|
||||
SEAFILE_ADMIN_PASSWORD=govoplan-seafile-admin
|
||||
|
||||
WEBDAV_HOST_PORT=9083
|
||||
WEBDAV_USER=govoplan
|
||||
WEBDAV_PASSWORD=govoplan-webdav
|
||||
|
||||
SMB_HOST_PORT=1445
|
||||
SMB_SHARE_NAME=files
|
||||
SMB_USER=govoplan
|
||||
SMB_PASSWORD=govoplan-smb
|
||||
|
||||
MINIO_API_HOST_PORT=9000
|
||||
MINIO_CONSOLE_HOST_PORT=9001
|
||||
MINIO_ROOT_USER=govoplan
|
||||
MINIO_ROOT_PASSWORD=govoplan-minio
|
||||
MINIO_BUCKET=govoplan
|
||||
@@ -0,0 +1,172 @@
|
||||
# Connector Dev Stack
|
||||
|
||||
This directory keeps local Docker Compose assets for GovOPlaN file connector
|
||||
development. The stack is intentionally separate from production deployment and
|
||||
binds services to localhost high ports.
|
||||
|
||||
## Start
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
cp .env.example .env
|
||||
mkdir -p data/smb data/webdav
|
||||
docker compose up -d nextcloud nextcloud-db webdav smb minio
|
||||
docker compose up -d seafile-db seafile-memcached seafile
|
||||
```
|
||||
|
||||
If the SMB service was already running before a compose change, recreate it so
|
||||
the image and share configuration are applied:
|
||||
|
||||
```bash
|
||||
docker compose rm -sf smb
|
||||
docker compose up -d --build smb
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
|
||||
- Nextcloud: `http://127.0.0.1:9081`, WebDAV root
|
||||
`http://127.0.0.1:9081/remote.php/dav/files/admin/`
|
||||
- Seafile: `http://127.0.0.1:9082`, WebDAV root
|
||||
`http://127.0.0.1:9082/seafdav/`
|
||||
- WebDAV: `http://127.0.0.1:9083`
|
||||
- SMB: `smb://127.0.0.1:1445/files`
|
||||
- MinIO/S3 API: `http://127.0.0.1:9000`, console
|
||||
`http://127.0.0.1:9001`
|
||||
|
||||
The local fixture data under `data/` is ignored by git.
|
||||
|
||||
## GovOPlaN Profile Config
|
||||
|
||||
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. This deployment-owned configuration
|
||||
may reference environment variables only when their exact names are listed in
|
||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; profile API responses expose only the
|
||||
credential source and configured state.
|
||||
|
||||
Example local profile file:
|
||||
|
||||
```json
|
||||
{
|
||||
"profiles": [
|
||||
{
|
||||
"id": "dev-seafile",
|
||||
"label": "Dev Seafile",
|
||||
"provider": "seafile",
|
||||
"endpoint_url": "http://127.0.0.1:9082",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "admin@example.local",
|
||||
"password_env": "SEAFILE_ADMIN_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"metadata": { "webdav_endpoint_url": "http://127.0.0.1:9082/seafdav/" },
|
||||
"policy": { "allow": { "providers": ["seafile"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-nextcloud",
|
||||
"label": "Dev Nextcloud",
|
||||
"provider": "nextcloud",
|
||||
"endpoint_url": "http://127.0.0.1:9081/remote.php/dav/files/admin/",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "admin",
|
||||
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["nextcloud", "webdav"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-webdav",
|
||||
"label": "Dev WebDAV",
|
||||
"provider": "webdav",
|
||||
"endpoint_url": "http://127.0.0.1:9083",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "WEBDAV_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["webdav"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-smb",
|
||||
"label": "Dev SMB",
|
||||
"provider": "smb",
|
||||
"endpoint_url": "smb://127.0.0.1:1445/files",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "SMB_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["smb"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"label": "Dev MinIO",
|
||||
"provider": "s3",
|
||||
"endpoint_url": "http://127.0.0.1:9000",
|
||||
"base_path": "",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "MINIO_ROOT_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"metadata": {
|
||||
"bucket": "govoplan",
|
||||
"region": "us-east-1",
|
||||
"path_style": true,
|
||||
"verify_tls": false
|
||||
},
|
||||
"policy": { "allow": { "providers": ["s3"] } }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Start GovOPlaN with:
|
||||
|
||||
```bash
|
||||
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
|
||||
export GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST=SEAFILE_ADMIN_PASSWORD,NEXTCLOUD_ADMIN_PASSWORD,WEBDAV_PASSWORD,SMB_PASSWORD,MINIO_ROOT_PASSWORD
|
||||
export GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true
|
||||
```
|
||||
|
||||
## Smoke Test
|
||||
|
||||
Run the connector helper smoke checks from an environment where
|
||||
`govoplan-files` and `httpx` are importable:
|
||||
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py
|
||||
```
|
||||
|
||||
The script reads `.env` from this directory when present and seeds tiny WebDAV,
|
||||
Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud are browsed and imported
|
||||
through the pinned connector transport. SMB and S3 product access deliberately
|
||||
fails closed until their SDK transports support peer pinning, so their fixtures
|
||||
are retained for transport development and reported as expected optional skips.
|
||||
The `--require-smb` and `--require-s3` switches are useful only while developing
|
||||
that transport support and currently make the smoke check fail by design.
|
||||
|
||||
SMB smoke checks need the optional Python dependency in the environment running
|
||||
the script:
|
||||
|
||||
```bash
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
|
||||
```
|
||||
|
||||
S3 smoke checks need the optional boto3 dependency in the environment running
|
||||
the script:
|
||||
|
||||
```bash
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[s3]
|
||||
```
|
||||
|
||||
The SMB service is built from `dev/connectors/smb/` so the development share is
|
||||
deterministic: one `files` share backed by `data/smb`, with the credentials from
|
||||
`.env`.
|
||||
|
||||
The SMB image runs as the non-root `govoplan` user with UID/GID `1000` and
|
||||
listens on unprivileged container port `1445`. The default host endpoint remains
|
||||
`smb://127.0.0.1:1445/files`. `SMB_USER` must stay `govoplan` unless the image is
|
||||
rebuilt with a matching user; `SMB_PORT` must be `1024` or higher. `SMB_PASSWORD`
|
||||
is applied when the image is built, so rebuild the `smb` service after changing
|
||||
it in `.env`.
|
||||
@@ -0,0 +1,2 @@
|
||||
*
|
||||
!.gitignore
|
||||
@@ -0,0 +1,120 @@
|
||||
name: govoplan-files-connectors
|
||||
|
||||
services:
|
||||
nextcloud-db:
|
||||
image: mariadb:10.11
|
||||
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${NEXTCLOUD_DB_ROOT_PASSWORD:-govoplan-nextcloud-root}
|
||||
MYSQL_DATABASE: nextcloud
|
||||
MYSQL_USER: nextcloud
|
||||
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
|
||||
volumes:
|
||||
- nextcloud-db:/var/lib/mysql
|
||||
|
||||
nextcloud:
|
||||
image: nextcloud:apache
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- nextcloud-db
|
||||
ports:
|
||||
- "127.0.0.1:${NEXTCLOUD_HOST_PORT:-9081}:80"
|
||||
environment:
|
||||
MYSQL_HOST: nextcloud-db
|
||||
MYSQL_DATABASE: nextcloud
|
||||
MYSQL_USER: nextcloud
|
||||
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
|
||||
NEXTCLOUD_ADMIN_USER: ${NEXTCLOUD_ADMIN_USER:-admin}
|
||||
NEXTCLOUD_ADMIN_PASSWORD: ${NEXTCLOUD_ADMIN_PASSWORD:-govoplan-nextcloud-admin}
|
||||
NEXTCLOUD_TRUSTED_DOMAINS: "localhost 127.0.0.1"
|
||||
volumes:
|
||||
- nextcloud:/var/www/html
|
||||
|
||||
seafile-db:
|
||||
image: mariadb:10.11
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
|
||||
MARIADB_AUTO_UPGRADE: "1"
|
||||
MYSQL_LOG_CONSOLE: "true"
|
||||
volumes:
|
||||
- seafile-db:/var/lib/mysql
|
||||
|
||||
seafile-memcached:
|
||||
image: memcached:1.6
|
||||
restart: unless-stopped
|
||||
command: memcached -m 256
|
||||
|
||||
seafile:
|
||||
image: seafileltd/seafile-mc:11.0-latest
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- seafile-db
|
||||
- seafile-memcached
|
||||
ports:
|
||||
- "127.0.0.1:${SEAFILE_HOST_PORT:-9082}:80"
|
||||
environment:
|
||||
DB_HOST: seafile-db
|
||||
DB_ROOT_PASSWD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
|
||||
TIME_ZONE: Etc/UTC
|
||||
SEAFILE_ADMIN_EMAIL: ${SEAFILE_ADMIN_EMAIL:-admin@example.local}
|
||||
SEAFILE_ADMIN_PASSWORD: ${SEAFILE_ADMIN_PASSWORD:-govoplan-seafile-admin}
|
||||
SEAFILE_SERVER_LETSENCRYPT: "false"
|
||||
SEAFILE_SERVER_HOSTNAME: "127.0.0.1:${SEAFILE_HOST_PORT:-9082}"
|
||||
volumes:
|
||||
- seafile-data:/shared
|
||||
|
||||
webdav:
|
||||
image: rclone/rclone:latest
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- serve
|
||||
- webdav
|
||||
- /data
|
||||
- --addr
|
||||
- :8080
|
||||
- --user
|
||||
- ${WEBDAV_USER:-govoplan}
|
||||
- --pass
|
||||
- ${WEBDAV_PASSWORD:-govoplan-webdav}
|
||||
ports:
|
||||
- "127.0.0.1:${WEBDAV_HOST_PORT:-9083}:8080"
|
||||
volumes:
|
||||
- ./data/webdav:/data
|
||||
|
||||
smb:
|
||||
build:
|
||||
context: ./smb
|
||||
args:
|
||||
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
|
||||
image: govoplan-files-samba-dev:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
|
||||
SMB_USER: ${SMB_USER:-govoplan}
|
||||
SMB_PORT: ${SMB_PORT:-1445}
|
||||
ports:
|
||||
- "127.0.0.1:${SMB_HOST_PORT:-1445}:${SMB_PORT:-1445}"
|
||||
volumes:
|
||||
- ./data/smb:/storage
|
||||
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-govoplan}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-govoplan-minio}
|
||||
ports:
|
||||
- "127.0.0.1:${MINIO_API_HOST_PORT:-9000}:9000"
|
||||
- "127.0.0.1:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"
|
||||
volumes:
|
||||
- minio:/data
|
||||
|
||||
volumes:
|
||||
nextcloud-db:
|
||||
nextcloud:
|
||||
seafile-db:
|
||||
seafile-data:
|
||||
minio:
|
||||
@@ -0,0 +1,29 @@
|
||||
FROM alpine:3.20
|
||||
|
||||
ARG SMB_PASSWORD=govoplan-smb
|
||||
|
||||
RUN apk add --no-cache samba-server samba-common-tools \
|
||||
&& addgroup -S -g 1000 govoplan \
|
||||
&& adduser -S -D -H -h /nonexistent -s /sbin/nologin -u 1000 -G govoplan govoplan \
|
||||
&& mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba /var/log/samba/cores \
|
||||
&& printf '%s\n' \
|
||||
'[global]' \
|
||||
' passdb backend = tdbsam' \
|
||||
' private dir = /var/lib/samba/private' \
|
||||
' lock directory = /run/samba' \
|
||||
' state directory = /var/lib/samba' \
|
||||
' cache directory = /var/cache/samba' \
|
||||
' pid directory = /run/samba' \
|
||||
> /etc/samba/smb.conf \
|
||||
&& printf '%s\n%s\n' "$SMB_PASSWORD" "$SMB_PASSWORD" | smbpasswd -s -a govoplan \
|
||||
&& smbpasswd -e govoplan \
|
||||
&& chown -R govoplan:govoplan /storage /var/lib/samba /var/cache/samba /run/samba /etc/samba /var/log/samba
|
||||
|
||||
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
|
||||
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
|
||||
|
||||
EXPOSE 1445
|
||||
|
||||
USER govoplan:govoplan
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
share_name="${SMB_SHARE_NAME:-files}"
|
||||
user_name="${SMB_USER:-govoplan}"
|
||||
smb_port="${SMB_PORT:-1445}"
|
||||
runtime_user="$(id -un)"
|
||||
runtime_group="$(id -gn)"
|
||||
|
||||
case "$share_name" in
|
||||
"" | *[!A-Za-z0-9_.-]*)
|
||||
printf 'SMB_SHARE_NAME must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
case "$user_name" in
|
||||
"" | *[!A-Za-z0-9_.-]*)
|
||||
printf 'SMB_USER must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
case "$smb_port" in
|
||||
"" | *[!0-9]*)
|
||||
printf 'SMB_PORT must be numeric.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if [ "$smb_port" -lt 1024 ]; then
|
||||
printf 'SMB_PORT must be >= 1024 because the dev Samba container runs as a non-root user.\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$user_name" != "$runtime_user" ]; then
|
||||
printf 'SMB_USER=%s is not supported by the non-root dev image; use %s or rebuild the image with a matching user.\n' "$user_name" "$runtime_user" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba
|
||||
|
||||
cat > /etc/samba/smb.conf <<EOF
|
||||
[global]
|
||||
server role = standalone server
|
||||
workgroup = WORKGROUP
|
||||
security = user
|
||||
map to guest = Never
|
||||
guest account = $runtime_user
|
||||
server min protocol = SMB2
|
||||
server signing = mandatory
|
||||
smb ports = $smb_port
|
||||
load printers = no
|
||||
printing = bsd
|
||||
disable spoolss = yes
|
||||
log level = 1
|
||||
passdb backend = tdbsam
|
||||
private dir = /var/lib/samba/private
|
||||
lock directory = /run/samba
|
||||
state directory = /var/lib/samba
|
||||
cache directory = /var/cache/samba
|
||||
pid directory = /run/samba
|
||||
|
||||
[$share_name]
|
||||
path = /storage
|
||||
browseable = yes
|
||||
read only = no
|
||||
guest ok = no
|
||||
valid users = $user_name
|
||||
force user = $runtime_user
|
||||
force group = $runtime_group
|
||||
create mask = 0664
|
||||
directory mask = 0775
|
||||
EOF
|
||||
|
||||
if ! pdbedit -L -u "$user_name" >/dev/null 2>&1; then
|
||||
printf 'Samba user %s is missing from passdb. Rebuild the smb image so the non-root passdb is seeded.\n' "$user_name" >&2
|
||||
printf 'Run: docker compose build --no-cache smb && docker compose up -d smb\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec smbd --foreground --no-process-group --debug-stdout -p "$smb_port"
|
||||
@@ -0,0 +1,282 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import socket
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import browse_connector_profile
|
||||
from govoplan_files.backend.storage.connector_imports import read_connector_file
|
||||
from govoplan_files.backend.storage.connector_profiles import connector_profiles_from_payload
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
|
||||
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
|
||||
parser.add_argument("--require-s3", action="store_true", help="Fail if the S3 connector cannot browse/import.")
|
||||
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
|
||||
args = parser.parse_args()
|
||||
|
||||
_load_dotenv()
|
||||
_default_env()
|
||||
preflight_failures = _preflight_services(require_smb=args.require_smb, require_s3=args.require_s3)
|
||||
if preflight_failures:
|
||||
for failure in preflight_failures:
|
||||
print(f"FAIL {failure}")
|
||||
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb minio")
|
||||
return 1
|
||||
_seed_webdav_fixture()
|
||||
_seed_smb_fixture()
|
||||
try:
|
||||
_seed_nextcloud_fixture()
|
||||
except httpx.HTTPError as exc:
|
||||
print(f"FAIL dev-nextcloud seed failed: {exc}")
|
||||
return 1
|
||||
try:
|
||||
_seed_s3_fixture()
|
||||
except Exception as exc:
|
||||
if args.require_s3:
|
||||
print(f"FAIL dev-s3 seed failed: {exc}")
|
||||
return 1
|
||||
print(f"SKIP dev-s3 seed failed: {exc}")
|
||||
|
||||
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
|
||||
if args.debug_smb:
|
||||
_debug_smb(profiles["dev-smb"])
|
||||
failures: list[str] = []
|
||||
for profile_id, folder, file_path, expected in (
|
||||
("dev-webdav", "GovOPlaN", "GovOPlaN/webdav-live.txt", "webdav live fixture"),
|
||||
("dev-nextcloud", "GovOPlaN", "GovOPlaN/nextcloud-live.txt", "nextcloud live fixture"),
|
||||
):
|
||||
try:
|
||||
_exercise_profile(profiles[profile_id], folder=folder, file_path=file_path, expected=expected)
|
||||
except Exception as exc:
|
||||
failures.append(f"{profile_id}: {exc}")
|
||||
|
||||
try:
|
||||
_exercise_profile(profiles["dev-smb"], folder="GovOPlaN", file_path="GovOPlaN/smb-live.txt", expected="smb live fixture")
|
||||
except Exception as exc:
|
||||
message = f"dev-smb: {exc}"
|
||||
if args.require_smb:
|
||||
failures.append(message)
|
||||
else:
|
||||
print(f"SKIP {message}")
|
||||
|
||||
try:
|
||||
_exercise_profile(profiles["dev-s3"], folder="GovOPlaN", file_path="GovOPlaN/s3-live.txt", expected="s3 live fixture")
|
||||
except Exception as exc:
|
||||
message = f"dev-s3: {exc}"
|
||||
if args.require_s3:
|
||||
failures.append(message)
|
||||
else:
|
||||
print(f"SKIP {message}")
|
||||
|
||||
if failures:
|
||||
for failure in failures:
|
||||
print(f"FAIL {failure}")
|
||||
return 1
|
||||
print("OK connector dev stack smoke checks passed")
|
||||
return 0
|
||||
|
||||
|
||||
def _default_env() -> None:
|
||||
defaults = {
|
||||
"NEXTCLOUD_ADMIN_USER": "admin",
|
||||
"NEXTCLOUD_ADMIN_PASSWORD": "govoplan-nextcloud-admin",
|
||||
"WEBDAV_USER": "govoplan",
|
||||
"WEBDAV_PASSWORD": "govoplan-webdav",
|
||||
"SMB_SHARE_NAME": "files",
|
||||
"SMB_USER": "govoplan",
|
||||
"SMB_PASSWORD": "govoplan-smb",
|
||||
"MINIO_ROOT_USER": "govoplan",
|
||||
"MINIO_ROOT_PASSWORD": "govoplan-minio",
|
||||
"MINIO_BUCKET": "govoplan",
|
||||
}
|
||||
for key, value in defaults.items():
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
|
||||
def _load_dotenv() -> None:
|
||||
path = ROOT / ".env"
|
||||
if not path.exists():
|
||||
return
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
text = line.strip()
|
||||
if not text or text.startswith("#") or "=" not in text:
|
||||
continue
|
||||
key, value = text.split("=", 1)
|
||||
key = key.strip()
|
||||
if not key or key in os.environ:
|
||||
continue
|
||||
os.environ[key] = value.strip().strip("\"'")
|
||||
|
||||
|
||||
def _preflight_services(*, require_smb: bool, require_s3: bool) -> list[str]:
|
||||
failures: list[str] = []
|
||||
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
|
||||
try:
|
||||
response = httpx.get(nextcloud_url, timeout=3.0)
|
||||
if response.status_code != 200:
|
||||
failures.append(f"dev-nextcloud expected HTTP 200 at {nextcloud_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-nextcloud is not reachable at {nextcloud_url}: {exc}")
|
||||
|
||||
webdav_url = f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/"
|
||||
try:
|
||||
response = httpx.get(webdav_url, timeout=3.0)
|
||||
if response.status_code not in {200, 401}:
|
||||
failures.append(f"dev-webdav expected HTTP 200/401 at {webdav_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-webdav is not reachable at {webdav_url}: {exc}")
|
||||
|
||||
if require_smb:
|
||||
smb_port = int(os.getenv("SMB_HOST_PORT", "1445"))
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", smb_port), timeout=3.0):
|
||||
pass
|
||||
except OSError as exc:
|
||||
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
|
||||
if require_s3:
|
||||
minio_url = f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}/minio/health/live"
|
||||
try:
|
||||
response = httpx.get(minio_url, timeout=3.0)
|
||||
if response.status_code != 200:
|
||||
failures.append(f"dev-s3 expected HTTP 200 at {minio_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-s3 is not reachable at {minio_url}: {exc}")
|
||||
return failures
|
||||
|
||||
|
||||
def _profile_payloads() -> list[dict[str, object]]:
|
||||
return [
|
||||
{
|
||||
"id": "dev-webdav",
|
||||
"provider": "webdav",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("WEBDAV_USER", "govoplan"),
|
||||
"password_env": "WEBDAV_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-nextcloud",
|
||||
"provider": "nextcloud",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("NEXTCLOUD_ADMIN_USER", "admin"),
|
||||
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-smb",
|
||||
"provider": "smb",
|
||||
"endpoint_url": f"smb://127.0.0.1:{os.getenv('SMB_HOST_PORT', '1445')}/{os.getenv('SMB_SHARE_NAME', 'files')}",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("SMB_USER", "govoplan"),
|
||||
"password_env": "SMB_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"provider": "s3",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||
"password_env": "MINIO_ROOT_PASSWORD",
|
||||
"metadata": {
|
||||
"bucket": os.getenv("MINIO_BUCKET", "govoplan"),
|
||||
"region": "us-east-1",
|
||||
"path_style": True,
|
||||
"verify_tls": False,
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _exercise_profile(profile, *, folder: str, file_path: str, expected: str) -> None:
|
||||
items = browse_connector_profile(profile, path=folder)
|
||||
paths = {item.path for item in items}
|
||||
if file_path not in paths:
|
||||
raise RuntimeError(f"{file_path!r} not found; saw {sorted(paths)!r}")
|
||||
downloaded = read_connector_file(profile, library_id="", path=file_path, max_bytes=1024 * 1024)
|
||||
text = downloaded.data.decode("utf-8").strip()
|
||||
if text != expected:
|
||||
raise RuntimeError(f"{file_path!r} content mismatch: {text!r}")
|
||||
print(f"OK {profile.id} browse/import {file_path}")
|
||||
|
||||
|
||||
def _debug_smb(profile) -> None:
|
||||
try:
|
||||
import smbclient
|
||||
except ImportError as exc:
|
||||
print(f"DEBUG smbclient import failed: {exc}")
|
||||
return
|
||||
from govoplan_files.backend.storage.connector_browse import _smb_client_kwargs, _smb_location, _smb_unc_path
|
||||
|
||||
location = _smb_location(profile)
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
print(f"DEBUG SMB endpoint=//{location.server}:{location.port}/{location.share} root_path={location.root_path!r}")
|
||||
print(f"DEBUG SMB user={kwargs.get('username')!r} require_signing={kwargs.get('require_signing')!r} auth_protocol={kwargs.get('auth_protocol')!r}")
|
||||
try:
|
||||
smbclient.reset_connection_cache()
|
||||
except Exception as exc:
|
||||
print(f"DEBUG SMB reset cache failed: {exc}")
|
||||
for path in (_smb_unc_path(location, ""), _smb_unc_path(location, "GovOPlaN")):
|
||||
try:
|
||||
print(f"DEBUG SMB list {path}: {smbclient.listdir(path, **kwargs)}")
|
||||
except Exception as exc:
|
||||
print(f"DEBUG SMB list {path} failed: {type(exc).__name__}: {exc}")
|
||||
|
||||
|
||||
def _seed_webdav_fixture() -> None:
|
||||
path = ROOT / "data" / "webdav" / "GovOPlaN"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
(path / "webdav-live.txt").write_text("webdav live fixture\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _seed_smb_fixture() -> None:
|
||||
path = ROOT / "data" / "smb" / "GovOPlaN"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
(path / "smb-live.txt").write_text("smb live fixture\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _seed_nextcloud_fixture() -> None:
|
||||
base_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/GovOPlaN"
|
||||
auth = (os.getenv("NEXTCLOUD_ADMIN_USER", "admin"), os.getenv("NEXTCLOUD_ADMIN_PASSWORD", "govoplan-nextcloud-admin"))
|
||||
response = httpx.request("MKCOL", base_url, auth=auth, timeout=15.0)
|
||||
if response.status_code not in {201, 405}:
|
||||
raise RuntimeError(f"Nextcloud MKCOL failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||
response = httpx.put(f"{base_url}/nextcloud-live.txt", content=b"nextcloud live fixture\n", auth=auth, timeout=15.0)
|
||||
if response.status_code not in {200, 201, 204}:
|
||||
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||
|
||||
|
||||
def _seed_s3_fixture() -> None:
|
||||
try:
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("boto3 is not installed; install govoplan-files[s3]") from exc
|
||||
bucket = os.getenv("MINIO_BUCKET", "govoplan")
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||
aws_access_key_id=os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||
aws_secret_access_key=os.getenv("MINIO_ROOT_PASSWORD", "govoplan-minio"),
|
||||
region_name="us-east-1",
|
||||
config=Config(s3={"addressing_style": "path"}),
|
||||
)
|
||||
try:
|
||||
client.create_bucket(Bucket=bucket)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code")
|
||||
if code not in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
|
||||
raise
|
||||
client.put_object(Bucket=bucket, Key="GovOPlaN/s3-live.txt", Body=b"s3 live fixture\n", ContentType="text/plain")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,78 @@
|
||||
# File Connector Boundary
|
||||
|
||||
GovOPlaN Files owns the managed-file boundary: frozen blobs, versions, shares,
|
||||
campaign attachment evidence, provenance, connector policy, connector profiles,
|
||||
and read-only browse/import APIs that turn external files into managed
|
||||
GovOPlaN files.
|
||||
|
||||
The built-in connector layer is intentionally on-demand. It does not run remote
|
||||
indexing, background sync, remote mutation, or upstream permission management.
|
||||
Connectors may browse an external source, import one selected file, freeze it as
|
||||
a managed file, record provenance, and audit the access.
|
||||
|
||||
## Built-In Baseline
|
||||
|
||||
The files module may keep small baseline providers when they are needed for
|
||||
normal product workflows and can share the same governance model:
|
||||
|
||||
- Seafile through the native read/download API
|
||||
- Nextcloud through WebDAV
|
||||
- generic WebDAV
|
||||
- SMB through the optional `smb` extra
|
||||
- S3-compatible stores through the optional `s3` extra
|
||||
|
||||
These providers are surfaced through connector descriptors at
|
||||
`GET /api/v1/files/connectors/providers`. Provider descriptors declare whether
|
||||
the provider is implemented, whether its optional dependency is installed, and
|
||||
which browse/import behaviors are available.
|
||||
|
||||
## Separate Connector Module Candidates
|
||||
|
||||
A separate connector module becomes appropriate when integration needs exceed
|
||||
on-demand browse/import:
|
||||
|
||||
- background indexing or synchronization
|
||||
- bidirectional remote mutation
|
||||
- long-running transfer workers
|
||||
- provider-specific credential lifecycle or OAuth flows
|
||||
- DMS/eAkte metadata models, registers, retention, or filing plans
|
||||
- S3-compatible object store administration
|
||||
- NFS host-mount lifecycle or sidecar coordination
|
||||
- provider-specific WebUI administration beyond profile fields
|
||||
|
||||
In that model, `govoplan-files` should keep the managed-file import contract,
|
||||
policy checks, provenance model, and audit events. A connector module should own
|
||||
provider-specific discovery, synchronization, credentials, health checks, and
|
||||
any remote write behavior.
|
||||
|
||||
## Provider Responsibilities
|
||||
|
||||
Every provider must:
|
||||
|
||||
- enforce GovOPlaN profile visibility and connector policy before browse/import
|
||||
- keep credentials as encrypted values or scoped secret references, never API
|
||||
response values; process-environment references are allowed only in
|
||||
deployment-owned profiles with an exact deployment allowlist
|
||||
- import external files into managed storage before they are used in campaigns
|
||||
or workflows
|
||||
- preserve source provenance and revision metadata
|
||||
- emit connector audit events for imported or accessed files
|
||||
- treat remote ACLs as upstream checks, not as a replacement for GovOPlaN policy
|
||||
- use a transport that pins every connection to a policy-validated DNS/IP answer
|
||||
and revalidates redirects; SDK transports without that guarantee fail closed
|
||||
|
||||
SMB initial connections, reconnects, aliases, and DFS referral targets are
|
||||
created through a Files-owned pinned `smbprotocol` transport. S3 HTTP/HTTPS pools
|
||||
use the equivalent botocore adapter for every connection selected by retries,
|
||||
redirects, endpoint discovery, and virtual-host addressing. Both adapters apply
|
||||
the deployment-wide private-network policy at socket creation. S3 keeps the
|
||||
configured hostname for TLS SNI and certificate verification, but does not use
|
||||
outbound proxies or ambient AWS credential discovery. An incompatible optional
|
||||
SDK release fails closed before a usable session or client is returned.
|
||||
|
||||
## Non-Goals For Files
|
||||
|
||||
Files does not own collaborative editing, comments, document review workflows,
|
||||
remote lock orchestration, DMS records management, or external system data
|
||||
models. Those belong to future document, workflow, DMS, or connector modules and
|
||||
should integrate through capabilities and managed-file imports.
|
||||
@@ -0,0 +1,155 @@
|
||||
# Connector Spaces
|
||||
|
||||
GovOPlaN Files should treat external file shares as two separate product
|
||||
objects:
|
||||
|
||||
1. A governed connection profile defines a reusable remote endpoint such as a
|
||||
WebDAV server, Nextcloud account, Seafile server, or SMB share. Profiles are
|
||||
administered in settings at system or tenant scope, with the same
|
||||
inheritance and limit semantics used by mail-server profiles.
|
||||
2. A governed credential profile defines reusable authentication material for
|
||||
one provider or for any compatible provider. Credentials are administered
|
||||
separately from connection profiles and can carry their own allow/deny
|
||||
policy.
|
||||
3. A linked file space binds one concrete remote folder or library path from an
|
||||
allowed profile to a user or group. Linked spaces appear beside "My files"
|
||||
and group file spaces in the files module.
|
||||
|
||||
This keeps secrets, endpoint governance, and tenant limits in settings, while
|
||||
keeping user/group workspaces and concrete folder choices in the files module.
|
||||
|
||||
## Model
|
||||
|
||||
Connection profile:
|
||||
|
||||
- provider: `seafile`, `nextcloud`, `webdav`, `smb`, or a future provider
|
||||
- endpoint URL and optional base path
|
||||
- scope: `system` or `tenant` for administered profiles; user/group profiles
|
||||
may be allowed later only when policy explicitly permits them
|
||||
- optional credential profile id
|
||||
- capabilities: browse, sync, import, optional write when a provider supports it
|
||||
- profile-local policy for allow/deny rules
|
||||
|
||||
Credential profile:
|
||||
|
||||
- provider: a specific provider or any provider
|
||||
- scope: `system` or `tenant` for administered credentials
|
||||
- credential mode: anonymous, scoped secret reference, or encrypted stored
|
||||
password/token; environment references are reserved for deployment-owned JSON
|
||||
profiles and exact deployment allowlisting
|
||||
- username and redacted secret configuration
|
||||
- credential-local policy for allow/deny rules
|
||||
|
||||
Connector policy:
|
||||
|
||||
- system policy is the baseline
|
||||
- tenant policy inherits system policy and may narrow it unless system allows
|
||||
lower-level relaxation
|
||||
- future user/group policy may narrow tenant policy for self-service links
|
||||
- policies can allow or block providers, profile ids, endpoint URLs, external
|
||||
path prefixes, credential ids, credential inheritance, and local linked-space
|
||||
creation
|
||||
|
||||
Linked connector space:
|
||||
|
||||
- owner type: user or group
|
||||
- display name
|
||||
- connector profile id
|
||||
- remote library id or share id
|
||||
- remote root path
|
||||
- sync mode: manual initially; background sync is future work
|
||||
- read-only flag from provider/policy
|
||||
- active/deleted state
|
||||
|
||||
The linked space is addressable as a normal file space in the Files UI. It can
|
||||
browse the remote folder, synchronize one selected file, or run a bounded manual
|
||||
folder synchronization into managed storage. Background sync jobs remain a
|
||||
separate scheduling concern.
|
||||
|
||||
## Policy Semantics
|
||||
|
||||
Use mail-profile terminology because administrators already see it there:
|
||||
|
||||
- must use: lower scopes are restricted to selected profile ids or providers
|
||||
- can use: lower scopes may choose from inherited allowed profiles
|
||||
- shall not use anything outside: endpoint URLs and path prefixes are enforced
|
||||
by deny/allow rules before browse, import, or sync
|
||||
- may create local links: controls whether users/groups can add linked spaces
|
||||
from inherited profiles
|
||||
|
||||
Connector policy should provide an explainable effective policy response with
|
||||
source path entries: system, tenant, user, group, campaign when applicable.
|
||||
|
||||
## Current State
|
||||
|
||||
Implemented:
|
||||
|
||||
- provider descriptors for Seafile, Nextcloud, WebDAV, and SMB
|
||||
- database-backed connector profiles with system and tenant scope
|
||||
- database-backed connector credentials with system and tenant scope
|
||||
- encrypted stored password/token support for database credentials
|
||||
- JSON/environment-defined connector profiles with system, tenant, user, group,
|
||||
and campaign visibility
|
||||
- connector allow/deny policy enforcement before browse/import/sync, including
|
||||
provider, connection profile, credential profile, and path checks
|
||||
- read-only browse endpoints
|
||||
- import and sync into managed files with provenance and revision metadata
|
||||
- audit events for connector import, sync, and access
|
||||
- settings/admin UI sections for system and tenant file connections and
|
||||
credentials
|
||||
- linked connector-space rows owned by users or groups
|
||||
- file-space API responses that include linked connector spaces
|
||||
- Files UI entry point to create linked connector spaces from a browsed remote
|
||||
profile/folder
|
||||
- Files UI sync dialog for choosing a profile, browsing a remote folder, and
|
||||
syncing a selected file into a managed destination folder
|
||||
- Files UI connector-space view with provider/read-only/manual-sync state
|
||||
- bounded folder-level manual sync with recursive traversal, pagination,
|
||||
per-item review outcomes, source-provenance matching, and one summary audit
|
||||
event
|
||||
- Docker dev stack smoke checks for WebDAV, Nextcloud, and SMB
|
||||
|
||||
Missing:
|
||||
|
||||
- explicit connector profile policy rows equivalent to mail-profile policies
|
||||
- effective connector policy/explain UI equivalent to mail-profile policies
|
||||
- edit/manage actions for existing linked connector spaces
|
||||
- optional background sync or remote-write semantics
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
1. Persist governed connector profiles and policies.
|
||||
- `file_connector_profiles` exists for system and tenant profiles.
|
||||
- `file_connector_credentials` exists for system and tenant credentials.
|
||||
- Environment JSON profiles remain bootstrap/compatibility profiles.
|
||||
- Profile and credential CRUD endpoints exist for database-backed records.
|
||||
- Remaining: `file_connector_policies` plus effective policy/explain output.
|
||||
|
||||
2. Add linked connector spaces.
|
||||
- `file_connector_spaces` exists with owner user/group, profile id, library
|
||||
id, remote path, display label, sync mode, and active state.
|
||||
- Connector policy is enforced before creating or using a link.
|
||||
- Linked connector spaces are returned from `/api/v1/files/spaces`.
|
||||
|
||||
3. Surface linked spaces in the Files UI.
|
||||
- Linked spaces appear beside managed user/group spaces.
|
||||
- The add-space dialog browses an allowed profile and links the current
|
||||
remote folder/library root.
|
||||
- The connector browser/sync flow is reused when a linked space is open.
|
||||
- Remaining: edit/manage actions for existing linked spaces.
|
||||
|
||||
4. Add sync orchestration.
|
||||
- Manual selected-file and folder-level synchronization are available.
|
||||
- Folder sync preserves remote relative paths, bounds file count and depth,
|
||||
and reports created, updated, unchanged, skipped, conflict, policy-denied,
|
||||
and failed items without rolling back successful siblings.
|
||||
- The default conflict strategy is `skip`, so unrelated managed files are
|
||||
left untouched. `rename`, `reject`, and `overwrite` remain explicit choices.
|
||||
- Add optional scheduled/background sync separately; it must reuse the same
|
||||
policy, provenance, result, and audit contracts.
|
||||
|
||||
5. Add provider-specific expansion.
|
||||
- OAuth/secret-store credentials.
|
||||
- Remote write and delete only where policy and provider support it.
|
||||
- DMS/eAkte metadata integrations in a separate connector or documents
|
||||
module.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Document Collaboration Boundary
|
||||
|
||||
GovOPlaN Files is the governed managed-file module. It stores uploaded/imported
|
||||
blobs, versions, shares, provenance, ZIP imports/exports, connector imports, and
|
||||
campaign attachment evidence. It should stay reliable, auditable, and simple.
|
||||
|
||||
Collaborative document behavior should be owned by a future documents module or
|
||||
by provider-specific connector modules when the behavior is delegated to systems
|
||||
such as Nextcloud, Seafile, Collabora, OnlyOffice, OpenDesk, or DMS/eAkte
|
||||
platforms.
|
||||
|
||||
## Files Owns
|
||||
|
||||
- managed blobs and immutable version evidence
|
||||
- owner-scoped user/group file spaces
|
||||
- shares and access checks for managed files
|
||||
- connector browse/import into managed storage
|
||||
- source provenance, source revision, and audit events
|
||||
- freeze-before-send campaign attachment behavior
|
||||
- previews generated from managed file versions
|
||||
|
||||
## Documents Or Connectors Should Own
|
||||
|
||||
- co-editing sessions and editor launch URLs
|
||||
- comments, suggestions, document tasks, and review state
|
||||
- check-in/check-out, remote locks, and lock timeouts
|
||||
- semantic document versions beyond blob versions
|
||||
- template merge flows and generated-document lifecycle
|
||||
- external DMS metadata, filing plans, record categories, and retention classes
|
||||
- provider-specific sync state and conflict resolution
|
||||
- collaborative presence and notification behavior
|
||||
|
||||
## Integration Shape
|
||||
|
||||
The documents layer should integrate with files through stable contracts:
|
||||
|
||||
- create or import a managed file version for evidence points
|
||||
- attach source provenance when a file came from an external editor or DMS
|
||||
- ask files for read/download access rather than importing files internals
|
||||
- emit workflow/audit events when a collaborative document reaches a governed
|
||||
state such as reviewed, approved, frozen, sent, or archived
|
||||
- use connector providers for provider-specific browse/import/write behavior
|
||||
|
||||
Files may expose links to a document or connector module, but it should not own
|
||||
the collaboration state machine. A collaborative document can produce many
|
||||
working states; GovOPlaN Files should persist the governed snapshots that other
|
||||
modules can safely use as evidence.
|
||||
|
||||
## Practical Rule
|
||||
|
||||
If a feature is about storing, sharing, importing, downloading, or freezing a
|
||||
file, it belongs in `govoplan-files`. If a feature is about people jointly
|
||||
editing, reviewing, locking, commenting on, or routing a document through a
|
||||
semantic process, it belongs in a documents/workflow/DMS integration module.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
||||
# Generated Artifact Store
|
||||
|
||||
Files implements Core's optional `files.artifact_store` capability for modules
|
||||
that generate deterministic output without owning file storage.
|
||||
|
||||
The producer supplies bytes, filename, content type, destination folder,
|
||||
optional idempotency key, and bounded non-secret provenance. Files applies the
|
||||
actor's `files:file:upload` permission, tenant/user ownership, path rules,
|
||||
versioning, configured blob backend, and conflict behavior. An idempotency key
|
||||
is represented as source provenance so an unchanged retry does not create an
|
||||
unrelated file version.
|
||||
|
||||
The shared Files session owns finalization. Before a new managed object is
|
||||
written, Files commits a lease-fenced Core recovery operation containing only
|
||||
identifiers and digests. The caller's eventual session commit independently
|
||||
verifies both `FileBlob` metadata and stored bytes; rollback compensates only an
|
||||
unreferenced key. Producers must therefore complete the supplied transaction
|
||||
normally and must not bypass or replace Files session lifecycle handling.
|
||||
|
||||
The response contains only file/version identifiers, display path, media type,
|
||||
size, digest, and storage provenance. Producers must not put credentials,
|
||||
tokens, or rendered plaintext into metadata. Storing an artifact proves Files
|
||||
accepted it; it does not prove printing, mailing, or any other external effect.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Files Interface Pattern Migration
|
||||
|
||||
This inventory records the Files-owned part of the GovOPlaN interface pattern
|
||||
language. Core owns the shell and shared components; Files owns the composition
|
||||
and consequences described here.
|
||||
|
||||
## Surface inventory
|
||||
|
||||
| Surface | Primary task | Archetype | Consequence | Pattern evidence |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `/files` space and folder panes | Browse managed and connected content without losing location | Directory/explorer | Low for navigation; medium for exposing filenames and provenance | Full-height two-pane workspace, bounded panes, stable selection and contextual Help Center link |
|
||||
| `/files` toolbar and property filters | Find and act on the current selection | Explorer actions and local filtering | Medium for upload, move, copy, share and synchronization; high for delete | Actions remain beside the affected list, disabled controls explain permission/state/selection blockers, destructive work uses `ConfirmDialog` |
|
||||
| Upload/archive, transfer, rename and connector-import dialogs | Supply and review one bounded change | Adaptive create/edit or guided import | Medium to high because files, paths and external bytes change | Shared `Dialog`, `FileDropZone`, validation, conflict review, unsaved inputs and explicit confirmation |
|
||||
| File share dialog | Inspect and change access | Review/decision | High because another actor gains access | Shared dialog, access explanation, stable row actions and destructive confirmation |
|
||||
| System/tenant/group/user connector surfaces | Compare connections, credentials and effective policy | Administration/configuration | High because endpoints, secrets and inherited policy control external access | Shared `ConnectionTree`, adaptive forms, `ActionBlockerHint`, policy provenance and contextual admin help |
|
||||
| Connection and credential editors | Create or edit one governed endpoint or secret | Adaptive create/edit | High because a saved change may enable remote access | Relevant fields only, typed credential controls, discovery/test, advanced compatibility section, unsaved-change guard and disabled-save reasons |
|
||||
| Connector policy card | Narrow inherited connector access | Effective-policy editor | High because deny/allow changes affect lower scopes | Typed reference selectors, deny precedence warning, effective source evidence and permission blocker |
|
||||
| `files.widget.spaces` | See available managed/connected spaces and open Files | Dashboard widget | Low; names and provider state may still be sensitive | Shared loading/alert/status components, bounded item count, permission-filtered contribution |
|
||||
| Files chooser capability used by another module | Select a managed snapshot without importing Files internals | Directory chooser | Medium because the exact selected version becomes another module's input | Shared dialog/confirmation, capability boundary and exact file/version evidence |
|
||||
|
||||
## State and consequence contract
|
||||
|
||||
- Loading, errors, success, empty results, access explanations and confirmation
|
||||
use Core components. Files does not reproduce the application shell.
|
||||
- A connector that comes from deployment settings remains visible but read-only;
|
||||
its action explains that bootstrap configuration and a restart are required.
|
||||
- Missing permission, target, selection, endpoint, or compatible provider is an
|
||||
explained disabled state. It is not represented only by color or absence.
|
||||
- Provider metadata JSON is an expert compatibility escape hatch inside the
|
||||
collapsed shared advanced-options component. Ordinary connector setup uses
|
||||
typed provider, endpoint, credential, capability and policy controls.
|
||||
- Endpoint discovery and credential tests are explicit and report their result;
|
||||
they do not save the draft. Save remains the only committing action.
|
||||
- Connector/profile disable and managed-file delete remain confirmed actions and
|
||||
state their immediate effect. Soft deletion must not be described as purge.
|
||||
- External connector data, paths and credential references are rendered only in
|
||||
already-authorized administration or explorer contexts. Secret values are
|
||||
never returned for rendering.
|
||||
|
||||
## Accessibility and responsive evidence
|
||||
|
||||
Shared `Dialog` owns focus entry, Escape handling and focus return. Form and
|
||||
toolbar DOM order is the keyboard order; disabled-action tooltips are themselves
|
||||
focusable and expose the reason. The connector form uses semantic sections and
|
||||
labels, status is textual as well as colored, and result alerts are announced by
|
||||
the shared alert component. The explorer collapses to one column below 1050 px;
|
||||
connector forms and action rows collapse below 760 px while preserving source
|
||||
order. Long provider choices scroll inside the segmented control rather than
|
||||
expanding the page.
|
||||
|
||||
The focused structural test guards these contracts, optional-module boundaries,
|
||||
confirmation, contextual help, advanced-only JSON and responsive rules. Core's
|
||||
TypeScript build, structural localization audit, module-permutation suite and
|
||||
full-product bundle check provide the integration gates.
|
||||
|
||||
+8
-8
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.1",
|
||||
"version": "0.1.20",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "webui/src/index.ts",
|
||||
@@ -19,14 +19,14 @@
|
||||
"LICENSE"
|
||||
],
|
||||
"peerDependencies": {
|
||||
"@govoplan/core-webui": "^0.1.1",
|
||||
"lucide-react": "^0.555.0",
|
||||
"react": "^19.0.0",
|
||||
"react-dom": "^19.0.0",
|
||||
"react-router-dom": "^7.1.1",
|
||||
"@vitejs/plugin-react": "^4.3.4",
|
||||
"@vitejs/plugin-react": "^5.2.0",
|
||||
"vite": "^7.3.6",
|
||||
"typescript": "^5.7.2",
|
||||
"vite": "^6.0.6"
|
||||
"react": ">=19.2.7 <20",
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.18"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
+13
-2
@@ -4,14 +4,25 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-files"
|
||||
version = "0.1.3"
|
||||
version = "0.1.20"
|
||||
description = "GovOPlaN files module with backend and WebUI integration."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = { file = "LICENSE" }
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.3",
|
||||
"govoplan-core>=0.1.20",
|
||||
"defusedxml>=0.7,<1",
|
||||
"pyzipper>=0.3.6,<1",
|
||||
"python-multipart>=0.0.31,<1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
s3 = [
|
||||
"boto3>=1.34,<2",
|
||||
]
|
||||
smb = [
|
||||
"smbprotocol>=1.13",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
|
||||
@@ -1,17 +1,68 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.auth import ApiPrincipal, has_scope
|
||||
from govoplan_core.core.access import AccessDecisionProvenance, PrincipalRef
|
||||
from govoplan_core.core.files import FileAccessProvider
|
||||
from govoplan_core.core.files import (
|
||||
ManagedTabularFile,
|
||||
ManagedTabularFileAccessError,
|
||||
ManagedTabularFileContent,
|
||||
ManagedTabularFileNotFoundError,
|
||||
ManagedTabularFileProvider,
|
||||
ManagedTabularFileUnavailableError,
|
||||
ManagedTabularFileValidationError,
|
||||
PostboxFileReferenceRef,
|
||||
PostboxFileReferenceRequest,
|
||||
PostboxFileReferenceProvider,
|
||||
ManagedArtifactRef,
|
||||
ManagedArtifactStore,
|
||||
ManagedArtifactWriteRequest,
|
||||
)
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_core.security.module_permissions import scopes_grant_compatible
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileFolder, FileShare, FileVersion
|
||||
from govoplan_files.backend.runtime import configure_runtime
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.campaign_attachments import (
|
||||
annotate_built_messages_with_managed_files,
|
||||
managed_match_payloads,
|
||||
prepared_campaign_snapshot,
|
||||
public_attachment_summary_payload,
|
||||
share_assets_with_campaign,
|
||||
)
|
||||
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
list_recent_assets_for_user,
|
||||
read_asset_version_bytes,
|
||||
sync_file_asset_from_source,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
VIRTUAL_FOLDER_RESOURCE_PREFIX = "virtual-folder:v1"
|
||||
|
||||
WRITE_ACTIONS = {
|
||||
"files:file:upload",
|
||||
"files:file:organize",
|
||||
"files:file:share",
|
||||
"files:file:delete",
|
||||
"files:upload",
|
||||
"files:organize",
|
||||
"files:share",
|
||||
"files:delete",
|
||||
}
|
||||
|
||||
|
||||
class FilesCampaignCapability:
|
||||
@@ -21,6 +72,7 @@ class FilesCampaignCapability:
|
||||
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
|
||||
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
|
||||
current_version_and_blob = staticmethod(current_version_and_blob)
|
||||
share_assets_with_campaign = staticmethod(share_assets_with_campaign)
|
||||
|
||||
@staticmethod
|
||||
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:
|
||||
@@ -30,5 +82,679 @@ class FilesCampaignCapability:
|
||||
|
||||
|
||||
def campaign_capability(context: ModuleContext) -> FilesCampaignCapability:
|
||||
configure_runtime(settings=context.settings)
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesCampaignCapability()
|
||||
|
||||
|
||||
class FilesArtifactStore(ManagedArtifactStore):
|
||||
def store_artifact(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: ManagedArtifactWriteRequest,
|
||||
) -> ManagedArtifactRef:
|
||||
if not hasattr(session, "query") or not hasattr(session, "flush"):
|
||||
raise TypeError("Files artifact storage requires a SQLAlchemy session.")
|
||||
if not hasattr(principal, "has") or not principal.has("files:file:upload"):
|
||||
raise PermissionError("Managed artifact storage requires files:file:upload.")
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(getattr(user, "id", "") or "")
|
||||
tenant_id = str(getattr(principal, "tenant_id", "") or "")
|
||||
if not user_id or not tenant_id:
|
||||
raise PermissionError("Managed artifact storage requires a tenant user principal.")
|
||||
metadata = dict(request.metadata)
|
||||
if request.idempotency_key:
|
||||
metadata["source_provenance"] = {
|
||||
"source_type": "generated_artifact",
|
||||
"connector_id": "files.artifact_store",
|
||||
"provider": str(metadata.get("producer_module") or "platform"),
|
||||
"external_id": request.idempotency_key,
|
||||
"revision": str(metadata.get("output_sha256") or "") or None,
|
||||
}
|
||||
stored, _action, _previous_version_id = sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=user_id,
|
||||
user_id=user_id,
|
||||
filename=request.filename,
|
||||
data=request.payload,
|
||||
metadata=metadata,
|
||||
folder=request.folder,
|
||||
content_type=request.content_type,
|
||||
conflict_strategy="rename",
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
else:
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=user_id,
|
||||
user_id=user_id,
|
||||
filename=request.filename,
|
||||
data=request.payload,
|
||||
folder=request.folder,
|
||||
content_type=request.content_type,
|
||||
description=request.description,
|
||||
metadata=metadata,
|
||||
conflict_strategy="rename",
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
return ManagedArtifactRef(
|
||||
file_asset_id=stored.asset.id,
|
||||
file_version_id=stored.version.id,
|
||||
filename=stored.version.filename_at_upload,
|
||||
display_path=stored.asset.display_path,
|
||||
content_type=stored.version.content_type or request.content_type,
|
||||
size_bytes=stored.version.size_bytes,
|
||||
sha256=stored.version.checksum_sha256,
|
||||
provenance={
|
||||
"module": "files",
|
||||
"owner_type": stored.asset.owner_type,
|
||||
"managed": True,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def artifact_store_capability(context: ModuleContext) -> FilesArtifactStore:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesArtifactStore()
|
||||
|
||||
|
||||
class FilesPostboxReferenceService(PostboxFileReferenceProvider):
|
||||
"""Resolve Postbox evidence without turning message access into file access."""
|
||||
|
||||
def resolve_postbox_references(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
requests: tuple[PostboxFileReferenceRequest, ...],
|
||||
) -> tuple[PostboxFileReferenceRef, ...]:
|
||||
if not hasattr(session, "get"):
|
||||
raise TypeError("Postbox file resolution requires a SQLAlchemy session.")
|
||||
principal_tenant = str(getattr(principal, "tenant_id", "") or "")
|
||||
can_download = bool(
|
||||
hasattr(principal, "has") and principal.has("files:file:download")
|
||||
)
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(getattr(user, "id", "") or "")
|
||||
is_admin = bool(
|
||||
hasattr(principal, "has") and principal.has("files:file:admin")
|
||||
)
|
||||
results: list[PostboxFileReferenceRef] = []
|
||||
for request in requests:
|
||||
if principal_tenant != tenant_id:
|
||||
results.append(_unavailable_postbox_reference(request, "tenant_mismatch"))
|
||||
continue
|
||||
if not can_download or not user_id:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(
|
||||
request,
|
||||
"download_permission_missing",
|
||||
)
|
||||
)
|
||||
continue
|
||||
reference_type = request.reference_type.strip().casefold()
|
||||
version: FileVersion | None = None
|
||||
if reference_type in {"file", "file_asset", "files:file"}:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "exact_version_required")
|
||||
)
|
||||
continue
|
||||
elif reference_type in {"file_version", "files:file_version"}:
|
||||
version = session.get(FileVersion, request.reference_id) # type: ignore[attr-defined]
|
||||
asset = (
|
||||
session.get(FileAsset, version.file_asset_id) # type: ignore[attr-defined]
|
||||
if version is not None
|
||||
else None
|
||||
)
|
||||
else:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(
|
||||
request,
|
||||
"unsupported_reference_type",
|
||||
)
|
||||
)
|
||||
continue
|
||||
if (
|
||||
asset is None
|
||||
or version is None
|
||||
or asset.tenant_id != tenant_id
|
||||
or version.tenant_id != tenant_id
|
||||
or asset.deleted_at is not None
|
||||
):
|
||||
results.append(_unavailable_postbox_reference(request, "file_not_found"))
|
||||
continue
|
||||
try:
|
||||
get_asset_for_user(
|
||||
session, # type: ignore[arg-type]
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=asset.id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
except FileStorageError:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "file_access_denied")
|
||||
)
|
||||
continue
|
||||
blob = session.get(FileBlob, version.blob_id) # type: ignore[attr-defined]
|
||||
if blob is None or blob.tenant_id != tenant_id:
|
||||
results.append(
|
||||
_unavailable_postbox_reference(request, "file_payload_missing")
|
||||
)
|
||||
continue
|
||||
results.append(
|
||||
PostboxFileReferenceRef(
|
||||
reference_type=request.reference_type,
|
||||
reference_id=request.reference_id,
|
||||
available=True,
|
||||
reason_code="available",
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=version.id,
|
||||
filename=version.filename_at_upload or asset.filename,
|
||||
content_type=version.content_type or blob.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
sha256=version.checksum_sha256,
|
||||
download_path=(
|
||||
f"/api/v1/files/{asset.id}/versions/{version.id}/download"
|
||||
),
|
||||
provenance={
|
||||
"module": "files",
|
||||
"postbox_id": request.postbox_id,
|
||||
"message_id": request.message_id,
|
||||
"display_path": asset.display_path,
|
||||
"exact_version": True,
|
||||
},
|
||||
)
|
||||
)
|
||||
return tuple(results)
|
||||
|
||||
|
||||
def _unavailable_postbox_reference(
|
||||
request: PostboxFileReferenceRequest,
|
||||
reason_code: str,
|
||||
) -> PostboxFileReferenceRef:
|
||||
return PostboxFileReferenceRef(
|
||||
reference_type=request.reference_type,
|
||||
reference_id=request.reference_id,
|
||||
available=False,
|
||||
reason_code=reason_code,
|
||||
)
|
||||
|
||||
|
||||
def postbox_reference_capability(
|
||||
context: ModuleContext,
|
||||
) -> FilesPostboxReferenceService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesPostboxReferenceService()
|
||||
|
||||
|
||||
class FilesAccessService(FileAccessProvider):
|
||||
def explain_resource_provenance(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
action: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
normalized_type = resource_type.lower().strip()
|
||||
if normalized_type in {"file", "file_asset", "files:file"}:
|
||||
return self._explain_file(session, principal, resource_id=resource_id, action=action)
|
||||
if normalized_type in {"folder", "file_folder", "files:folder"}:
|
||||
return self._explain_folder(session, principal, resource_id=resource_id, action=action)
|
||||
return ()
|
||||
|
||||
def _explain_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_id: str,
|
||||
action: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
asset = session.get(FileAsset, resource_id) # type: ignore[attr-defined]
|
||||
if asset is None or (principal.tenant_id and asset.tenant_id != principal.tenant_id):
|
||||
return (_missing_resource("file", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
items = [_file_resource(asset)]
|
||||
items.extend(_owner_provenance(asset.owner_type, _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id), principal, source="files.owner"))
|
||||
items.extend(_admin_provenance(principal, "files:file:admin", source="files.admin_scope"))
|
||||
permission_values = {"write", "manage"} if _requires_write_share(action) else {"read", "write", "manage"}
|
||||
shares = (
|
||||
session.query(FileShare) # type: ignore[attr-defined]
|
||||
.filter(
|
||||
FileShare.tenant_id == asset.tenant_id,
|
||||
FileShare.file_asset_id == asset.id,
|
||||
effective_file_share_clause(),
|
||||
FileShare.permission.in_(sorted(permission_values)),
|
||||
or_(
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == principal.membership_id),
|
||||
(FileShare.target_type == "group") & (FileShare.target_id.in_(sorted(principal.group_ids))),
|
||||
(FileShare.target_type == "tenant") & (FileShare.target_id == asset.tenant_id),
|
||||
),
|
||||
)
|
||||
.order_by(FileShare.target_type.asc(), FileShare.target_id.asc())
|
||||
.all()
|
||||
)
|
||||
for share in shares:
|
||||
items.append(_share_provenance(share, source="files.share"))
|
||||
return tuple(items)
|
||||
|
||||
def _explain_folder(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_id: str,
|
||||
action: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
del action
|
||||
folder = session.get(FileFolder, resource_id) # type: ignore[attr-defined]
|
||||
if folder is None:
|
||||
return self._explain_virtual_folder(session, principal, resource_id=resource_id)
|
||||
if principal.tenant_id and folder.tenant_id != principal.tenant_id:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
return tuple([
|
||||
AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=folder.id,
|
||||
label=folder.path,
|
||||
tenant_id=folder.tenant_id,
|
||||
source="files.folder",
|
||||
details={
|
||||
"resource_type": "folder",
|
||||
"path": folder.path,
|
||||
"owner_type": folder.owner_type,
|
||||
"deleted": folder.deleted_at is not None,
|
||||
},
|
||||
),
|
||||
*_owner_provenance(folder.owner_type, _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id), principal, source="files.owner"),
|
||||
*_admin_provenance(principal, "files:file:admin", source="files.admin_scope"),
|
||||
])
|
||||
|
||||
def _explain_virtual_folder(
|
||||
self,
|
||||
session: object,
|
||||
principal: PrincipalRef,
|
||||
*,
|
||||
resource_id: str,
|
||||
) -> tuple[AccessDecisionProvenance, ...]:
|
||||
folder_ref = parse_virtual_folder_resource_id(resource_id)
|
||||
if folder_ref is None:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
tenant_id, owner_type, owner_id, path = folder_ref
|
||||
if principal.tenant_id and tenant_id != principal.tenant_id:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
child_prefix = f"{path}/"
|
||||
owner_filter = FileAsset.owner_user_id == owner_id if owner_type == "user" else FileAsset.owner_group_id == owner_id
|
||||
child = (
|
||||
session.query(FileAsset.id) # type: ignore[attr-defined]
|
||||
.filter(
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
FileAsset.owner_type == owner_type,
|
||||
owner_filter,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
FileAsset.display_path.like(f"{child_prefix}%"),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if child is None:
|
||||
return (_missing_resource("folder", resource_id, principal.tenant_id, source="files.not_found"),)
|
||||
return tuple([
|
||||
AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=resource_id,
|
||||
label=path,
|
||||
tenant_id=tenant_id,
|
||||
source="files.virtual_folder",
|
||||
details={
|
||||
"resource_type": "folder",
|
||||
"path": path,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"virtual": True,
|
||||
"deleted": False,
|
||||
},
|
||||
),
|
||||
*_owner_provenance(owner_type, owner_id, principal, source="files.owner"),
|
||||
*_admin_provenance(principal, "files:file:admin", source="files.admin_scope"),
|
||||
])
|
||||
|
||||
|
||||
def access_capability(context: ModuleContext) -> FilesAccessService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesAccessService()
|
||||
|
||||
|
||||
class FilesManagedTabularFileService(ManagedTabularFileProvider):
|
||||
"""Expose authorized immutable CSV/XLSX versions to optional consumers."""
|
||||
|
||||
def list_tabular_files(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
query: str = "",
|
||||
limit: int = 100,
|
||||
) -> tuple[ManagedTabularFile, ...]:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:read",
|
||||
)
|
||||
normalized_query = str(query or "").strip().casefold()
|
||||
requested_limit = max(1, min(int(limit), 100))
|
||||
assets = list_recent_assets_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
limit=min(500, requested_limit * 5),
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
results: list[ManagedTabularFile] = []
|
||||
for asset in assets:
|
||||
if normalized_query and normalized_query not in (
|
||||
f"{asset.filename} {asset.display_path} {asset.description or ''}"
|
||||
).casefold():
|
||||
continue
|
||||
try:
|
||||
version, blob = current_version_and_blob(db, asset)
|
||||
except FileStorageError:
|
||||
continue
|
||||
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||
continue
|
||||
results.append(_managed_tabular_file(asset, version, blob))
|
||||
if len(results) >= requested_limit:
|
||||
break
|
||||
return tuple(results)
|
||||
|
||||
def get_tabular_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str | None = None,
|
||||
) -> ManagedTabularFile | None:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:read",
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_asset_id,
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
except FileStorageError:
|
||||
return None
|
||||
if file_version_id:
|
||||
version = db.get(FileVersion, file_version_id)
|
||||
if (
|
||||
version is None
|
||||
or version.file_asset_id != asset.id
|
||||
or version.tenant_id != api_principal.tenant_id
|
||||
):
|
||||
return None
|
||||
blob = db.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != api_principal.tenant_id:
|
||||
return None
|
||||
else:
|
||||
try:
|
||||
version, blob = current_version_and_blob(db, asset)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileUnavailableError(
|
||||
"Managed file version metadata is unavailable."
|
||||
) from exc
|
||||
if not _is_tabular_file(version.filename_at_upload, version.content_type):
|
||||
return None
|
||||
return _managed_tabular_file(asset, version, blob)
|
||||
|
||||
def read_tabular_file(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str,
|
||||
max_bytes: int,
|
||||
) -> ManagedTabularFileContent:
|
||||
db, api_principal, user_id = _tabular_context(
|
||||
session,
|
||||
principal,
|
||||
required_scope="files:file:download",
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_asset_id,
|
||||
is_admin=has_scope(api_principal, "files:file:admin"),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileNotFoundError(
|
||||
"Managed tabular file not found."
|
||||
) from exc
|
||||
metadata = self.get_tabular_file(
|
||||
db,
|
||||
api_principal,
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=file_version_id,
|
||||
)
|
||||
if metadata is None:
|
||||
raise ManagedTabularFileNotFoundError(
|
||||
"Managed tabular file version not found."
|
||||
)
|
||||
effective_max = max(1, int(max_bytes))
|
||||
if metadata.size_bytes > effective_max:
|
||||
raise ManagedTabularFileValidationError(
|
||||
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||
)
|
||||
try:
|
||||
payload, version, blob = read_asset_version_bytes(
|
||||
db,
|
||||
asset,
|
||||
file_version_id,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise ManagedTabularFileUnavailableError(
|
||||
"Managed tabular file content is unavailable or failed integrity verification."
|
||||
) from exc
|
||||
if len(payload) > effective_max:
|
||||
raise ManagedTabularFileValidationError(
|
||||
f"Managed tabular files are limited to {effective_max:,} bytes for this operation."
|
||||
)
|
||||
result = _managed_tabular_file(asset, version, blob)
|
||||
audit_from_principal(
|
||||
db,
|
||||
api_principal,
|
||||
action="files.tabular_content.read",
|
||||
object_type="file_version",
|
||||
object_id=version.id,
|
||||
details={
|
||||
"file_asset_id": asset.id,
|
||||
"size_bytes": version.size_bytes,
|
||||
"checksum_sha256": version.checksum_sha256,
|
||||
"consumer": "tabular_content",
|
||||
},
|
||||
)
|
||||
return ManagedTabularFileContent(file=result, payload=payload)
|
||||
|
||||
|
||||
def managed_tabular_file_capability(
|
||||
context: ModuleContext,
|
||||
) -> FilesManagedTabularFileService:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesManagedTabularFileService()
|
||||
|
||||
|
||||
def _tabular_context(
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
required_scope: str,
|
||||
) -> tuple[Session, ApiPrincipal, str]:
|
||||
if not isinstance(session, Session):
|
||||
raise TypeError("Managed tabular file access requires a SQLAlchemy session.")
|
||||
if not isinstance(principal, ApiPrincipal):
|
||||
raise ManagedTabularFileAccessError(
|
||||
"Managed tabular file access requires a tenant API principal."
|
||||
)
|
||||
if not (
|
||||
has_scope(principal, required_scope)
|
||||
or has_scope(principal, "files:file:admin")
|
||||
):
|
||||
raise ManagedTabularFileAccessError(
|
||||
f"Managed tabular file access requires {required_scope}."
|
||||
)
|
||||
user_id = str(getattr(principal.user, "id", "") or "").strip()
|
||||
if not principal.tenant_id or not user_id:
|
||||
raise ManagedTabularFileAccessError(
|
||||
"Managed tabular file access requires a tenant user principal."
|
||||
)
|
||||
return session, principal, user_id
|
||||
|
||||
|
||||
def _managed_tabular_file(
|
||||
asset: FileAsset,
|
||||
version: FileVersion,
|
||||
blob: FileBlob,
|
||||
) -> ManagedTabularFile:
|
||||
return ManagedTabularFile(
|
||||
file_asset_id=asset.id,
|
||||
file_version_id=version.id,
|
||||
filename=version.filename_at_upload or asset.filename,
|
||||
display_path=version.display_path_at_upload or asset.display_path,
|
||||
content_type=version.content_type or blob.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
sha256=version.checksum_sha256,
|
||||
updated_at=version.created_at,
|
||||
current_version=asset.current_version_id == version.id,
|
||||
)
|
||||
|
||||
|
||||
def _is_tabular_file(filename: str, content_type: str | None) -> bool:
|
||||
normalized_name = str(filename or "").strip().casefold()
|
||||
normalized_type = str(content_type or "").split(";", 1)[0].strip().casefold()
|
||||
return normalized_name.endswith((".csv", ".xlsx")) or normalized_type in {
|
||||
"text/csv",
|
||||
"application/csv",
|
||||
"application/vnd.ms-excel",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
}
|
||||
|
||||
|
||||
def virtual_folder_resource_id(*, tenant_id: str, owner_type: str, owner_id: str, path: str) -> str:
|
||||
normalized_path = normalize_folder(path)
|
||||
encoded_path = base64.urlsafe_b64encode(normalized_path.encode("utf-8")).decode("ascii").rstrip("=")
|
||||
return f"{VIRTUAL_FOLDER_RESOURCE_PREFIX}:{tenant_id}:{owner_type}:{owner_id}:{encoded_path}"
|
||||
|
||||
|
||||
def parse_virtual_folder_resource_id(resource_id: str) -> tuple[str, str, str, str] | None:
|
||||
parts = resource_id.split(":", 5)
|
||||
if len(parts) != 6 or f"{parts[0]}:{parts[1]}" != VIRTUAL_FOLDER_RESOURCE_PREFIX:
|
||||
return None
|
||||
_, _, tenant_id, owner_type, owner_id, encoded_path = parts
|
||||
if owner_type not in {"user", "group"} or not tenant_id or not owner_id or not encoded_path:
|
||||
return None
|
||||
padding = "=" * (-len(encoded_path) % 4)
|
||||
try:
|
||||
decoded_path = base64.urlsafe_b64decode(f"{encoded_path}{padding}").decode("utf-8")
|
||||
path = normalize_folder(decoded_path)
|
||||
except (binascii.Error, UnicodeDecodeError, ValueError):
|
||||
return None
|
||||
if not path:
|
||||
return None
|
||||
return tenant_id, owner_type, owner_id, path
|
||||
|
||||
|
||||
def _file_resource(asset: FileAsset) -> AccessDecisionProvenance:
|
||||
return AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=asset.id,
|
||||
label=asset.filename,
|
||||
tenant_id=asset.tenant_id,
|
||||
source="files.file",
|
||||
details={
|
||||
"resource_type": "file",
|
||||
"display_path": asset.display_path,
|
||||
"owner_type": asset.owner_type,
|
||||
"deleted": asset.deleted_at is not None,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _missing_resource(resource_type: str, resource_id: str, tenant_id: str | None, *, source: str) -> AccessDecisionProvenance:
|
||||
return AccessDecisionProvenance(
|
||||
kind="resource",
|
||||
id=resource_id,
|
||||
tenant_id=tenant_id,
|
||||
source=source,
|
||||
details={"resource_type": resource_type, "found": False},
|
||||
)
|
||||
|
||||
|
||||
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||
return owner_user_id if owner_type == "user" else owner_group_id
|
||||
|
||||
|
||||
def _owner_provenance(owner_type: str, owner_id: str | None, principal: PrincipalRef, *, source: str) -> tuple[AccessDecisionProvenance, ...]:
|
||||
if owner_id is None:
|
||||
return ()
|
||||
matches_user = owner_type == "user" and owner_id == principal.membership_id
|
||||
matches_group = owner_type == "group" and owner_id in principal.group_ids
|
||||
if not (matches_user or matches_group):
|
||||
return ()
|
||||
return (
|
||||
AccessDecisionProvenance(
|
||||
kind="owner",
|
||||
id=owner_id,
|
||||
tenant_id=principal.tenant_id,
|
||||
source=source,
|
||||
details={"owner_type": owner_type},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _admin_provenance(principal: PrincipalRef, required_scope: str, *, source: str) -> tuple[AccessDecisionProvenance, ...]:
|
||||
if not scopes_grant_compatible(principal.scopes, required_scope):
|
||||
return ()
|
||||
return (
|
||||
AccessDecisionProvenance(
|
||||
kind="policy",
|
||||
id=required_scope,
|
||||
label=required_scope,
|
||||
tenant_id=principal.tenant_id,
|
||||
source=source,
|
||||
details={"grant": "tenant_admin"},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _share_provenance(share: FileShare, *, source: str) -> AccessDecisionProvenance:
|
||||
return AccessDecisionProvenance(
|
||||
kind="share",
|
||||
id=share.id,
|
||||
label=share.permission,
|
||||
tenant_id=share.tenant_id,
|
||||
source=source,
|
||||
details={
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _requires_write_share(action: str) -> bool:
|
||||
return action in WRITE_ACTIONS
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import event
|
||||
from sqlalchemy.orm import Session as OrmSession
|
||||
|
||||
from govoplan_core.core.change_sequence import record_change
|
||||
from govoplan_core.core.sqlalchemy_change_tracking import (
|
||||
ensure_object_id,
|
||||
has_attr_changes,
|
||||
object_state,
|
||||
operation_for_soft_deletable,
|
||||
previous_value,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid
|
||||
|
||||
FILES_MODULE_ID = "files"
|
||||
FILES_ASSETS_COLLECTION = "files.assets"
|
||||
FILES_FOLDERS_COLLECTION = "files.folders"
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION = "files.connector_profiles"
|
||||
FILES_CONNECTOR_CREDENTIALS_COLLECTION = "files.connector_credentials"
|
||||
FILES_CONNECTOR_POLICIES_COLLECTION = "files.connector_policies"
|
||||
FILES_CONNECTOR_SPACES_COLLECTION = "files.connector_spaces"
|
||||
|
||||
_REGISTERED = False
|
||||
|
||||
|
||||
def register_files_change_tracking() -> None:
|
||||
global _REGISTERED
|
||||
if _REGISTERED:
|
||||
return
|
||||
event.listen(OrmSession, "before_flush", _record_files_changes)
|
||||
_REGISTERED = True
|
||||
|
||||
|
||||
def _record_files_changes(session: OrmSession, _flush_context: object, _instances: object) -> None:
|
||||
for obj in tuple(session.new) + tuple(session.dirty):
|
||||
if isinstance(obj, FileAsset):
|
||||
_record_asset_change(session, obj)
|
||||
elif isinstance(obj, FileFolder):
|
||||
_record_folder_change(session, obj)
|
||||
elif isinstance(obj, FileShare):
|
||||
_record_share_visibility_change(session, obj)
|
||||
|
||||
|
||||
def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
operation = operation_for_soft_deletable(
|
||||
asset,
|
||||
changed_attrs=(
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
"current_version_id",
|
||||
"display_path",
|
||||
"filename",
|
||||
"description",
|
||||
"deleted_at",
|
||||
"retained_until",
|
||||
"legal_hold",
|
||||
"lifecycle_revision",
|
||||
"lifecycle_reason",
|
||||
"metadata_",
|
||||
),
|
||||
)
|
||||
if operation is None:
|
||||
return
|
||||
resource_id = _ensure_id(asset)
|
||||
record_change(
|
||||
session,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collection=FILES_ASSETS_COLLECTION,
|
||||
resource_type="file",
|
||||
resource_id=resource_id,
|
||||
operation=operation,
|
||||
tenant_id=asset.tenant_id,
|
||||
actor_type="user" if asset.created_by_user_id else None,
|
||||
actor_id=asset.created_by_user_id,
|
||||
payload={
|
||||
"owner_type": asset.owner_type,
|
||||
"owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id),
|
||||
"path": asset.display_path,
|
||||
"previous_path": previous_value(asset, "display_path"),
|
||||
"filename": asset.filename,
|
||||
"deleted_at": _isoformat(asset.deleted_at),
|
||||
"retained_until": _isoformat(asset.retained_until),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
|
||||
operation = operation_for_soft_deletable(
|
||||
folder,
|
||||
changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"),
|
||||
)
|
||||
if operation is None:
|
||||
return
|
||||
resource_id = _ensure_id(folder)
|
||||
record_change(
|
||||
session,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collection=FILES_FOLDERS_COLLECTION,
|
||||
resource_type="folder",
|
||||
resource_id=resource_id,
|
||||
operation=operation,
|
||||
tenant_id=folder.tenant_id,
|
||||
actor_type="user" if folder.created_by_user_id else None,
|
||||
actor_id=folder.created_by_user_id,
|
||||
payload={
|
||||
"owner_type": folder.owner_type,
|
||||
"owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id),
|
||||
"path": folder.path,
|
||||
"previous_path": previous_value(folder, "path"),
|
||||
"deleted_at": _isoformat(folder.deleted_at),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None:
|
||||
state = object_state(share)
|
||||
if not share.file_asset_id:
|
||||
return
|
||||
if not state.pending and not has_attr_changes(
|
||||
state,
|
||||
(
|
||||
"file_asset_id",
|
||||
"target_type",
|
||||
"target_id",
|
||||
"permission",
|
||||
"expires_at",
|
||||
"revoked_at",
|
||||
"revoked_by_user_id",
|
||||
),
|
||||
):
|
||||
return
|
||||
_ensure_id(share)
|
||||
record_change(
|
||||
session,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collection=FILES_ASSETS_COLLECTION,
|
||||
resource_type="file",
|
||||
resource_id=share.file_asset_id,
|
||||
operation="updated",
|
||||
tenant_id=share.tenant_id,
|
||||
actor_type="user" if share.created_by_user_id else None,
|
||||
actor_id=share.created_by_user_id,
|
||||
payload={
|
||||
"share_id": share.id,
|
||||
"share_target_type": share.target_type,
|
||||
"share_target_id": share.target_id,
|
||||
"share_permission": share.permission,
|
||||
"share_expires_at": _isoformat(share.expires_at),
|
||||
"share_revoked_at": _isoformat(share.revoked_at),
|
||||
"share_revoked_by_user_id": share.revoked_by_user_id,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _ensure_id(obj: object) -> str:
|
||||
return ensure_object_id(obj, new_uuid)
|
||||
|
||||
|
||||
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||
if owner_type == "user":
|
||||
return owner_user_id
|
||||
if owner_type == "group":
|
||||
return owner_group_id
|
||||
return None
|
||||
|
||||
|
||||
def _isoformat(value: datetime | None) -> str | None:
|
||||
return value.isoformat() if value else None
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FILES_ASSETS_COLLECTION",
|
||||
"FILES_CONNECTOR_CREDENTIALS_COLLECTION",
|
||||
"FILES_CONNECTOR_POLICIES_COLLECTION",
|
||||
"FILES_CONNECTOR_PROFILES_COLLECTION",
|
||||
"FILES_CONNECTOR_SPACES_COLLECTION",
|
||||
"FILES_FOLDERS_COLLECTION",
|
||||
"FILES_MODULE_ID",
|
||||
"register_files_change_tracking",
|
||||
]
|
||||
@@ -0,0 +1,600 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from govoplan_core.core.configuration_packages import (
|
||||
ConfigurationApplyResult,
|
||||
ConfigurationDiagnostic,
|
||||
ConfigurationExportResult,
|
||||
ConfigurationExportSelection,
|
||||
ConfigurationPackageFragment,
|
||||
ConfigurationPlanItem,
|
||||
ConfigurationPreflightContext,
|
||||
ConfigurationPreflightResult,
|
||||
ConfigurationProvider,
|
||||
ConfigurationProviderDescription,
|
||||
)
|
||||
from govoplan_core.core.infrastructure_capabilities import (
|
||||
InfrastructureCapability,
|
||||
InfrastructureCapabilityReceipt,
|
||||
)
|
||||
from govoplan_files.backend.runtime import settings as runtime_settings
|
||||
|
||||
|
||||
FILES_CONFIGURATION_CAPABILITY = "files.configuration"
|
||||
MANAGED_STORAGE_FRAGMENT = "managed_storage"
|
||||
_PAYLOAD_KEYS = frozenset(
|
||||
{"capability_id", "expected_backend", "expected_source"}
|
||||
)
|
||||
|
||||
|
||||
class FilesConfigurationProvider(ConfigurationProvider):
|
||||
module_id = "files"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
settings: object | None = None,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> None:
|
||||
self._settings = runtime_settings if settings is None else settings
|
||||
self._environment = os.environ if environment is None else environment
|
||||
|
||||
def describe(self) -> ConfigurationProviderDescription:
|
||||
return ConfigurationProviderDescription(
|
||||
module_id=self.module_id,
|
||||
fragment_types=(MANAGED_STORAGE_FRAGMENT,),
|
||||
schema_refs={
|
||||
MANAGED_STORAGE_FRAGMENT: "govoplan/files/configuration/managed-storage.v1"
|
||||
},
|
||||
exported_scopes=("system",),
|
||||
)
|
||||
|
||||
def preflight(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationPreflightResult:
|
||||
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||
return ConfigurationPreflightResult(
|
||||
diagnostics=(_unsupported(fragment),),
|
||||
plan=(_blocked_plan(fragment, "Fragment type is unsupported."),),
|
||||
)
|
||||
diagnostics, binding_ref = self._binding_diagnostics(fragment, context)
|
||||
blocked = any(item.severity == "blocker" for item in diagnostics)
|
||||
return ConfigurationPreflightResult(
|
||||
diagnostics=tuple(diagnostics),
|
||||
plan=(
|
||||
ConfigurationPlanItem(
|
||||
action="blocked" if blocked else "skip",
|
||||
module_id="files",
|
||||
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||
fragment_id=fragment.fragment_id or binding_ref,
|
||||
summary=(
|
||||
"Managed storage does not match the deployment receipt."
|
||||
if blocked
|
||||
else "Deployment-owned managed storage already matches the receipt; no module state is rewritten."
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
def apply(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
supplied_data: Mapping[str, Any],
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationApplyResult:
|
||||
del supplied_data
|
||||
if fragment.fragment_type != MANAGED_STORAGE_FRAGMENT:
|
||||
return ConfigurationApplyResult(diagnostics=(_unsupported(fragment),))
|
||||
diagnostics, _binding_ref = self._binding_diagnostics(fragment, context)
|
||||
return ConfigurationApplyResult(
|
||||
diagnostics=tuple(
|
||||
item for item in diagnostics if item.severity == "blocker"
|
||||
)
|
||||
)
|
||||
|
||||
def export(
|
||||
self,
|
||||
selection: ConfigurationExportSelection,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> ConfigurationExportResult:
|
||||
del selection
|
||||
receipt = context.infrastructure_receipt
|
||||
if context.infrastructure_receipt_error:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
_receipt_error(
|
||||
context.infrastructure_receipt_error,
|
||||
object_ref="files.storage",
|
||||
),
|
||||
)
|
||||
)
|
||||
if receipt is None:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_missing",
|
||||
message="Files managed-storage export requires the deployment capability receipt.",
|
||||
module_id="files",
|
||||
object_ref="files.storage",
|
||||
resolution="Mount the installer-generated receipt before exporting deployment configuration.",
|
||||
),
|
||||
)
|
||||
)
|
||||
capability = receipt.capability("files.storage")
|
||||
if capability is None:
|
||||
return ConfigurationExportResult(
|
||||
diagnostics=(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_missing",
|
||||
message="The deployment receipt does not declare managed file storage.",
|
||||
module_id="files",
|
||||
object_ref="files.storage",
|
||||
),
|
||||
)
|
||||
)
|
||||
return ConfigurationExportResult(
|
||||
fragments=(
|
||||
ConfigurationPackageFragment(
|
||||
module_id="files",
|
||||
fragment_type=MANAGED_STORAGE_FRAGMENT,
|
||||
fragment_id=f"{receipt.installation_id}:files.storage",
|
||||
payload={
|
||||
"capability_id": "files.storage",
|
||||
"expected_backend": _expected_backend(capability.source),
|
||||
"expected_source": capability.source,
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def health(
|
||||
self,
|
||||
import_result: ConfigurationApplyResult,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> tuple[ConfigurationDiagnostic, ...]:
|
||||
del context
|
||||
return tuple(
|
||||
item for item in import_result.diagnostics if item.severity == "blocker"
|
||||
)
|
||||
|
||||
def _binding_diagnostics(
|
||||
self,
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
) -> tuple[list[ConfigurationDiagnostic], str]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
payload = _files_fragment_payload(fragment, diagnostics)
|
||||
if payload is None:
|
||||
return diagnostics, fragment.fragment_id or "files.storage"
|
||||
receipt_state = _files_receipt_capability(
|
||||
fragment,
|
||||
context,
|
||||
payload,
|
||||
diagnostics,
|
||||
)
|
||||
if receipt_state is None:
|
||||
return diagnostics, fragment.fragment_id or "files.storage"
|
||||
_receipt, capability, binding_ref = receipt_state
|
||||
expected_backend = _expected_backend(capability.source)
|
||||
diagnostics.extend(
|
||||
_package_expectation_diagnostics(
|
||||
payload,
|
||||
capability,
|
||||
expected_backend,
|
||||
binding_ref,
|
||||
)
|
||||
)
|
||||
active_backend = _normalized_backend(
|
||||
getattr(self._settings, "file_storage_backend", "local")
|
||||
)
|
||||
if active_backend != expected_backend:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_runtime_mismatch",
|
||||
message=(
|
||||
f"Files runtime uses {active_backend!r}, while the deployment receipt declares {expected_backend!r}."
|
||||
),
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Reconcile deployment environment and receipt before starting or importing Files configuration.",
|
||||
)
|
||||
)
|
||||
return diagnostics, binding_ref
|
||||
diagnostics.extend(
|
||||
self._runtime_storage_diagnostics(
|
||||
expected_backend,
|
||||
capability,
|
||||
binding_ref,
|
||||
)
|
||||
)
|
||||
if "files" not in capability.dependent_modules:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="info",
|
||||
code="infrastructure_consumer_not_declared",
|
||||
message="Files is active but was not selected as a receipt consumer when the deployment plan was generated.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Regenerate the deployment plan so removal-impact inventory includes Files.",
|
||||
)
|
||||
)
|
||||
return diagnostics, binding_ref
|
||||
|
||||
def _runtime_storage_diagnostics(
|
||||
self,
|
||||
expected_backend: str,
|
||||
capability: InfrastructureCapability,
|
||||
binding_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
if expected_backend == "s3":
|
||||
return self._s3_diagnostics(
|
||||
capability.endpoint,
|
||||
capability.secret_refs,
|
||||
capability.source,
|
||||
binding_ref,
|
||||
)
|
||||
root = Path(
|
||||
str(getattr(self._settings, "file_storage_local_root", "") or "")
|
||||
)
|
||||
if root.is_absolute():
|
||||
return []
|
||||
return [
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_local_root_not_durable",
|
||||
message="Receipt-bound local file storage requires an absolute deployment-managed path.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Set FILE_STORAGE_LOCAL_ROOT to the mounted persistent-volume path.",
|
||||
)
|
||||
]
|
||||
|
||||
def _s3_diagnostics(
|
||||
self,
|
||||
endpoint: Mapping[str, object],
|
||||
secret_refs: tuple[str, ...],
|
||||
source: str,
|
||||
object_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
endpoint_url = str(
|
||||
getattr(self._settings, "file_storage_s3_endpoint_url", "")
|
||||
or getattr(self._settings, "s3_endpoint_url", "")
|
||||
or ""
|
||||
).strip()
|
||||
runtime_endpoint = _redacted_endpoint(
|
||||
endpoint_url,
|
||||
default_port=int(endpoint.get("port") or 443),
|
||||
)
|
||||
receipt_endpoint = {
|
||||
key: endpoint.get(key) for key in ("scheme", "host", "port") if endpoint.get(key) is not None
|
||||
}
|
||||
if receipt_endpoint and runtime_endpoint != receipt_endpoint:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_endpoint_mismatch",
|
||||
message="Files S3 endpoint does not match the sanitized deployment receipt endpoint.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Reconcile FILE_STORAGE_S3_ENDPOINT_URL and the deployment plan before apply.",
|
||||
)
|
||||
)
|
||||
bucket = str(
|
||||
getattr(self._settings, "file_storage_s3_bucket", "")
|
||||
or getattr(self._settings, "s3_bucket", "")
|
||||
or ""
|
||||
).strip()
|
||||
if not bucket:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_bucket_missing",
|
||||
message="Files S3 storage has no configured bucket.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Set FILE_STORAGE_S3_BUCKET in the deployment environment.",
|
||||
)
|
||||
)
|
||||
for reference in secret_refs:
|
||||
variable = reference.removeprefix("env:")
|
||||
if not variable.startswith("FILE_STORAGE_"):
|
||||
continue
|
||||
if not str(self._environment.get(variable, "")).strip():
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_secret_reference_unresolved",
|
||||
message=f"Required storage secret reference {reference} is not available to the runtime.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Provide the referenced environment secret without copying its value into the package.",
|
||||
)
|
||||
)
|
||||
deployment_managed = bool(
|
||||
getattr(
|
||||
self._settings,
|
||||
"file_storage_s3_deployment_managed",
|
||||
False,
|
||||
)
|
||||
)
|
||||
endpoint_trusted = bool(
|
||||
getattr(self._settings, "file_storage_s3_endpoint_trusted", False)
|
||||
)
|
||||
if source == "installer-managed-garage" and not deployment_managed:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_management_boundary_mismatch",
|
||||
message="Garage is installer-managed in the receipt but not marked deployment-managed in Files runtime.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Set FILE_STORAGE_S3_DEPLOYMENT_MANAGED=true for the installer-managed Garage endpoint.",
|
||||
)
|
||||
)
|
||||
if source == "operator-supplied-s3" and not endpoint_trusted:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_storage_trust_boundary_missing",
|
||||
message="External S3 storage must be explicitly marked as a trusted deployment endpoint.",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Review the endpoint and set FILE_STORAGE_S3_ENDPOINT_TRUSTED=true in deployment configuration.",
|
||||
)
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
|
||||
def _files_fragment_payload(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
diagnostics: list[ConfigurationDiagnostic],
|
||||
) -> Mapping[str, Any] | None:
|
||||
payload = fragment.payload
|
||||
if not isinstance(payload, Mapping):
|
||||
diagnostics.append(
|
||||
_invalid(fragment, "Managed-storage payload must be an object.")
|
||||
)
|
||||
return None
|
||||
unknown = sorted(set(payload) - _PAYLOAD_KEYS)
|
||||
if not unknown:
|
||||
return payload
|
||||
secret_like = any(
|
||||
marker in key.casefold()
|
||||
for key in unknown
|
||||
for marker in ("password", "secret", "token", "credential", "access_key")
|
||||
)
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code=(
|
||||
"files_configuration_secret_forbidden"
|
||||
if secret_like
|
||||
else "files_configuration_payload_invalid"
|
||||
),
|
||||
message=(
|
||||
"Managed-storage fragments accept receipt references and non-secret expectations only."
|
||||
if secret_like
|
||||
else f"Managed-storage payload contains unsupported fields: {', '.join(unknown)}."
|
||||
),
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or MANAGED_STORAGE_FRAGMENT,
|
||||
resolution="Keep storage credentials in deployment environment references, never in a configuration package.",
|
||||
)
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def _files_receipt_capability(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
context: ConfigurationPreflightContext,
|
||||
payload: Mapping[str, Any],
|
||||
diagnostics: list[ConfigurationDiagnostic],
|
||||
) -> tuple[
|
||||
InfrastructureCapabilityReceipt,
|
||||
InfrastructureCapability,
|
||||
str,
|
||||
] | None:
|
||||
if context.infrastructure_receipt_error:
|
||||
diagnostics.append(
|
||||
_receipt_error(
|
||||
context.infrastructure_receipt_error,
|
||||
object_ref=fragment.fragment_id or "files.storage",
|
||||
)
|
||||
)
|
||||
return None
|
||||
receipt = context.infrastructure_receipt
|
||||
if receipt is None:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_missing",
|
||||
message="Files managed-storage configuration requires the deployment capability receipt.",
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or "files.storage",
|
||||
resolution="Mount the installer-generated receipt and rerun preflight.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
binding_ref = fragment.fragment_id or f"{receipt.installation_id}:files.storage"
|
||||
capability_id = _text(payload.get("capability_id")) or "files.storage"
|
||||
if capability_id != "files.storage":
|
||||
diagnostics.append(
|
||||
_invalid(
|
||||
fragment,
|
||||
"Files managed-storage fragments must reference capability 'files.storage'.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
capability = receipt.capability(capability_id)
|
||||
if capability is None:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_missing",
|
||||
message="The deployment receipt does not declare managed file storage.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Regenerate the receipt from a deployment profile that declares managed storage.",
|
||||
)
|
||||
)
|
||||
return None
|
||||
state_diagnostic = _storage_capability_state_diagnostic(capability, binding_ref)
|
||||
if state_diagnostic is not None:
|
||||
diagnostics.append(state_diagnostic)
|
||||
return None
|
||||
return receipt, capability, binding_ref
|
||||
|
||||
|
||||
def _storage_capability_state_diagnostic(
|
||||
capability: InfrastructureCapability,
|
||||
binding_ref: str,
|
||||
) -> ConfigurationDiagnostic | None:
|
||||
if capability.state == "unavailable":
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_unavailable",
|
||||
message="The deployment receipt states that managed file storage is unavailable.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Select local, managed Garage, or external S3 storage in the deployment profile.",
|
||||
)
|
||||
if capability.state == "available_unconfigured":
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_capability_unconfigured",
|
||||
message="Managed file storage is available but has not been bound by the deployment runtime.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Complete the deployment-owned storage configuration before importing Files configuration.",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _package_expectation_diagnostics(
|
||||
payload: Mapping[str, Any],
|
||||
capability: InfrastructureCapability,
|
||||
expected_backend: str,
|
||||
binding_ref: str,
|
||||
) -> list[ConfigurationDiagnostic]:
|
||||
diagnostics: list[ConfigurationDiagnostic] = []
|
||||
package_backend = (
|
||||
_text(payload.get("expected_backend")) or expected_backend
|
||||
).casefold()
|
||||
if package_backend != expected_backend:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_backend_mismatch",
|
||||
message="The package storage backend expectation conflicts with the deployment receipt.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Use the receipt backend or regenerate the deployment package after review.",
|
||||
)
|
||||
)
|
||||
package_source = _text(payload.get("expected_source"))
|
||||
if package_source and package_source != capability.source:
|
||||
diagnostics.append(
|
||||
ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_source_mismatch",
|
||||
message="The package storage source expectation conflicts with the deployment receipt.",
|
||||
module_id="files",
|
||||
object_ref=binding_ref,
|
||||
resolution="Review the provider replacement and regenerate the package from the active receipt.",
|
||||
)
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
|
||||
def _expected_backend(source: str) -> str:
|
||||
return "local" if source == "host-local" else "s3"
|
||||
|
||||
|
||||
def _normalized_backend(value: object) -> str:
|
||||
clean = str(value or "local").strip().casefold()
|
||||
if clean in {"local", "filesystem", "fs"}:
|
||||
return "local"
|
||||
if clean in {"s3", "garage"}:
|
||||
return "s3"
|
||||
return clean
|
||||
|
||||
|
||||
def _redacted_endpoint(value: str, *, default_port: int) -> dict[str, object]:
|
||||
try:
|
||||
parsed = urlsplit(value)
|
||||
host = parsed.hostname
|
||||
if not parsed.scheme or not host:
|
||||
return {"reference": "unresolved"}
|
||||
port = parsed.port or default_port
|
||||
except ValueError:
|
||||
return {"reference": "unresolved"}
|
||||
return {"scheme": parsed.scheme, "host": host, "port": port}
|
||||
|
||||
|
||||
def _text(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _receipt_error(error: str, *, object_ref: str) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="infrastructure_receipt_invalid",
|
||||
message=f"The deployment capability receipt is invalid: {error}",
|
||||
module_id="files",
|
||||
object_ref=object_ref,
|
||||
resolution="Repair or regenerate the deployment receipt before importing Files configuration.",
|
||||
)
|
||||
|
||||
|
||||
def _invalid(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
message: str,
|
||||
) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="files_configuration_payload_invalid",
|
||||
message=message,
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||
resolution="Review the Files configuration-package fragment and rerun preflight.",
|
||||
)
|
||||
|
||||
|
||||
def _unsupported(fragment: ConfigurationPackageFragment) -> ConfigurationDiagnostic:
|
||||
return ConfigurationDiagnostic(
|
||||
severity="blocker",
|
||||
code="fragment_type_unsupported",
|
||||
message=f"Files configuration does not support fragment type {fragment.fragment_type!r}.",
|
||||
module_id="files",
|
||||
object_ref=fragment.fragment_id or fragment.fragment_type,
|
||||
)
|
||||
|
||||
|
||||
def _blocked_plan(
|
||||
fragment: ConfigurationPackageFragment,
|
||||
summary: str,
|
||||
) -> ConfigurationPlanItem:
|
||||
return ConfigurationPlanItem(
|
||||
action="blocked",
|
||||
module_id="files",
|
||||
fragment_type=fragment.fragment_type,
|
||||
fragment_id=fragment.fragment_id,
|
||||
summary=summary,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["FILES_CONFIGURATION_CAPABILITY", "FilesConfigurationProvider"]
|
||||
@@ -1 +1,25 @@
|
||||
from govoplan_files.backend.db.models import *
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"CampaignAttachmentUse",
|
||||
"FileAsset",
|
||||
"FileBlob",
|
||||
"FileConnectorCredential",
|
||||
"FileConnectorPolicy",
|
||||
"FileConnectorProfile",
|
||||
"FileConnectorSpace",
|
||||
"FileFolder",
|
||||
"FileShare",
|
||||
"FileVersion",
|
||||
]
|
||||
|
||||
@@ -4,7 +4,18 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, Index, Integer, JSON, String, Text, UniqueConstraint, text
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
JSON,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from govoplan_core.db.base import Base, TimestampMixin
|
||||
@@ -16,18 +27,123 @@ def new_uuid() -> str:
|
||||
|
||||
class FileBlob(Base, TimestampMixin):
|
||||
__tablename__ = "file_blobs"
|
||||
__table_args__ = (UniqueConstraint("tenant_id", "checksum_sha256", "size_bytes", name="uq_file_blobs_tenant_checksum_size"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"checksum_sha256",
|
||||
"size_bytes",
|
||||
"protection_discriminator",
|
||||
name="uq_file_blobs_tenant_checksum_size_protection",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||
storage_bucket: Mapped[str | None] = mapped_column(String(255))
|
||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
protection_discriminator: Mapped[str] = mapped_column(
|
||||
String(320), default="plaintext", nullable=False, index=True
|
||||
)
|
||||
encryption_envelope_id: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True, index=True
|
||||
)
|
||||
storage_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
storage_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
ref_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
retained_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
integrity_status: Mapped[str] = mapped_column(
|
||||
String(30), default="unchecked", nullable=False, index=True
|
||||
)
|
||||
integrity_checked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
integrity_failure: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
quarantined_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileIntegrityScan(Base, TimestampMixin):
|
||||
__tablename__ = "file_integrity_scans"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
storage_backend: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||
storage_prefix: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30), default="pending", nullable=False, index=True
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
phase: Mapped[str] = mapped_column(String(30), default="blobs", nullable=False)
|
||||
verify_checksums: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False
|
||||
)
|
||||
batch_size: Mapped[int] = mapped_column(Integer, default=100, nullable=False)
|
||||
blob_cursor: Mapped[str | None] = mapped_column(String(36), nullable=True)
|
||||
object_cursor: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
scanned_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
verified_blob_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
quarantined_blob_count: Mapped[int] = mapped_column(
|
||||
Integer, default=0, nullable=False
|
||||
)
|
||||
scanned_object_count: Mapped[int] = mapped_column(
|
||||
Integer, default=0, nullable=False
|
||||
)
|
||||
orphan_object_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
started_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
completed_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
last_error: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
|
||||
class FileIntegrityFinding(Base, TimestampMixin):
|
||||
__tablename__ = "file_integrity_findings"
|
||||
__table_args__ = (
|
||||
Index("ix_file_integrity_findings_scan_state", "scan_id", "state"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
scan_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_integrity_scans.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
kind: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
|
||||
state: Mapped[str] = mapped_column(
|
||||
String(30), default="open", nullable=False, index=True
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
blob_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
storage_key: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
expected_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
observed_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
expected_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
observed_checksum_sha256: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True
|
||||
)
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
resolved_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileFolder(Base, TimestampMixin):
|
||||
@@ -35,14 +151,18 @@ class FileFolder(Base, TimestampMixin):
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"uq_file_folders_active_user_path",
|
||||
"tenant_id", "owner_user_id", "path",
|
||||
"tenant_id",
|
||||
"owner_user_id",
|
||||
"path",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
),
|
||||
Index(
|
||||
"uq_file_folders_active_group_path",
|
||||
"tenant_id", "owner_group_id", "path",
|
||||
"tenant_id",
|
||||
"owner_group_id",
|
||||
"path",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
@@ -50,91 +170,414 @@ class FileFolder(Base, TimestampMixin):
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class FileAsset(Base, TimestampMixin):
|
||||
__tablename__ = "file_assets"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(ForeignKey("groups.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
current_version_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
current_version_id: Mapped[str | None] = mapped_column(
|
||||
String(36), nullable=True, index=True
|
||||
)
|
||||
display_path: Mapped[str] = mapped_column(String(1000), nullable=False, index=True)
|
||||
filename: Mapped[str] = mapped_column(String(500), nullable=False, index=True)
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column("metadata", JSON, nullable=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
retained_until: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
legal_hold: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, nullable=False, index=True
|
||||
)
|
||||
lifecycle_revision: Mapped[int] = mapped_column(
|
||||
Integer, default=1, nullable=False
|
||||
)
|
||||
lifecycle_reason: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class FileVersion(Base, TimestampMixin):
|
||||
__tablename__ = "file_versions"
|
||||
__table_args__ = (UniqueConstraint("file_asset_id", "version_number", name="uq_file_versions_asset_number"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"file_asset_id", "version_number", name="uq_file_versions_asset_number"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
blob_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
version_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
filename_at_upload: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
display_path_at_upload: Mapped[str] = mapped_column(String(1000), nullable=False)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileFormEvidenceGrant(Base, TimestampMixin):
|
||||
__tablename__ = "file_form_evidence_grants"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_file_form_evidence_grants_idempotency",
|
||||
),
|
||||
Index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
"tenant_id",
|
||||
"form_instance_id",
|
||||
"form_definition_id",
|
||||
"form_definition_revision",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
form_instance_id: Mapped[str] = mapped_column(
|
||||
String(36), nullable=False, index=True
|
||||
)
|
||||
form_definition_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
form_definition_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
token_sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
request_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
custodian_user_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="RESTRICT"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
evidence_kind: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
purpose: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30), default="issued", nullable=False, index=True
|
||||
)
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
max_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
allowed_content_types: Mapped[list[str]] = mapped_column(
|
||||
JSON, default=list, nullable=False
|
||||
)
|
||||
file_asset_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||
)
|
||||
file_version_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=True, index=True
|
||||
)
|
||||
uploaded_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any]] = mapped_column(
|
||||
"metadata", JSON, default=dict, nullable=False
|
||||
)
|
||||
|
||||
|
||||
class FileShare(Base, TimestampMixin):
|
||||
__tablename__ = "file_shares"
|
||||
__table_args__ = (UniqueConstraint("file_asset_id", "target_type", "target_id", "revoked_at", name="uq_file_shares_active_target"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"file_asset_id",
|
||||
"target_type",
|
||||
"target_id",
|
||||
"revoked_at",
|
||||
name="uq_file_shares_active_target",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
target_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
target_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
permission: Mapped[str] = mapped_column(String(20), default="read", nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
revoked_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorProfile(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_profiles"
|
||||
__table_args__ = (
|
||||
Index("ix_file_connector_profiles_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(
|
||||
String(20), default="tenant", nullable=False, index=True
|
||||
)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||
endpoint_url: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
base_path: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
credential_profile_id: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True, index=True
|
||||
)
|
||||
credential_mode: Mapped[str] = mapped_column(
|
||||
String(30), default="none", nullable=False
|
||||
)
|
||||
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
password_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
capabilities: Mapped[list[str] | None] = mapped_column(JSON, nullable=True)
|
||||
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorCredential(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_credentials"
|
||||
__table_args__ = (
|
||||
Index("ix_file_connector_credentials_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(
|
||||
String(20), default="tenant", nullable=False, index=True
|
||||
)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
provider: Mapped[str | None] = mapped_column(String(50), nullable=True, index=True)
|
||||
enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
credential_mode: Mapped[str] = mapped_column(
|
||||
String(30), default="none", nullable=False
|
||||
)
|
||||
username: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||
password_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
token_encrypted: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
password_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
token_env: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
secret_ref: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
policy: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorPolicy(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_policies"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
name="uq_file_connector_policies_scope",
|
||||
),
|
||||
Index("ix_file_connector_policies_scope", "scope_type", "scope_id"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
scope_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
scope_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
policy: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
updated_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
class FileConnectorSpace(Base, TimestampMixin):
|
||||
__tablename__ = "file_connector_spaces"
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"ix_file_connector_spaces_owner",
|
||||
"tenant_id",
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
),
|
||||
Index(
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"tenant_id",
|
||||
"owner_user_id",
|
||||
"label",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
),
|
||||
Index(
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"tenant_id",
|
||||
"owner_group_id",
|
||||
"label",
|
||||
unique=True,
|
||||
sqlite_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
owner_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
|
||||
owner_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
owner_group_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_groups.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
label: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
connector_profile_id: Mapped[str] = mapped_column(
|
||||
String(255), nullable=False, index=True
|
||||
)
|
||||
provider: Mapped[str] = mapped_column(String(50), nullable=False, index=True)
|
||||
library_id: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
remote_path: Mapped[str] = mapped_column(String(1000), default="", nullable=False)
|
||||
sync_mode: Mapped[str] = mapped_column(String(30), default="manual", nullable=False)
|
||||
read_only: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
is_active: Mapped[bool] = mapped_column(
|
||||
Boolean, default=True, nullable=False, index=True
|
||||
)
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("access_users.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
deleted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
metadata_: Mapped[dict[str, Any] | None] = mapped_column(
|
||||
"metadata", JSON, nullable=True
|
||||
)
|
||||
|
||||
|
||||
class CampaignAttachmentUse(Base, TimestampMixin):
|
||||
__tablename__ = "campaign_attachment_uses"
|
||||
__table_args__ = (UniqueConstraint("campaign_job_id", "file_version_id", "filename_used", "use_stage", name="uq_campaign_attachment_uses_job_file_stage"),)
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"campaign_job_id",
|
||||
"file_version_id",
|
||||
"filename_used",
|
||||
"use_stage",
|
||||
name="uq_campaign_attachment_uses_job_file_stage",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_id: Mapped[str] = mapped_column(ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_version_id: Mapped[str] = mapped_column(ForeignKey("campaign_versions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
campaign_job_id: Mapped[str | None] = mapped_column(ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
campaign_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("campaigns.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
campaign_version_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("campaign_versions.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
campaign_job_id: Mapped[str | None] = mapped_column(
|
||||
ForeignKey("campaign_jobs.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
)
|
||||
entry_index: Mapped[int | None] = mapped_column(Integer)
|
||||
entry_id: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_version_id: Mapped[str] = mapped_column(ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_blob_id: Mapped[str] = mapped_column(ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True)
|
||||
file_asset_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_assets.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
file_version_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_versions.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
file_blob_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("file_blobs.id", ondelete="RESTRICT"), nullable=False, index=True
|
||||
)
|
||||
filename_used: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
content_type: Mapped[str | None] = mapped_column(String(255))
|
||||
use_stage: Mapped[str] = mapped_column(String(20), default="built", nullable=False, index=True)
|
||||
used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
|
||||
use_stage: Mapped[str] = mapped_column(
|
||||
String(20), default="built", nullable=False, index=True
|
||||
)
|
||||
used_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CampaignAttachmentUse",
|
||||
"FileAsset",
|
||||
"FileBlob",
|
||||
"FileConnectorCredential",
|
||||
"FileConnectorPolicy",
|
||||
"FileConnectorProfile",
|
||||
"FileConnectorSpace",
|
||||
"FileFolder",
|
||||
"FileFormEvidenceGrant",
|
||||
"FileShare",
|
||||
"FileVersion",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,517 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.campaigns import (
|
||||
CAPABILITY_CAMPAIGNS_ACCESS,
|
||||
CampaignAccessProvider,
|
||||
)
|
||||
from govoplan_core.core.modules import (
|
||||
DocumentationCondition,
|
||||
DocumentationContext,
|
||||
DocumentationLink,
|
||||
DocumentationTopic,
|
||||
)
|
||||
from govoplan_files.backend.storage.archives import ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||
from govoplan_files.backend.storage.access import user_group_ids
|
||||
from govoplan_files.backend.storage.connector_visibility import (
|
||||
connector_profile_usable_for_import,
|
||||
visible_connector_profiles_for_actor,
|
||||
)
|
||||
|
||||
|
||||
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
|
||||
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
|
||||
_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES = 2 * 1024 * 1024 * 1024
|
||||
_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO = 100
|
||||
_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS = 30 * 60
|
||||
_FILES_READ_SCOPE = "files:file:read"
|
||||
_FILES_UPLOAD_SCOPE = "files:file:upload"
|
||||
|
||||
|
||||
def documentation_topics(
|
||||
context: DocumentationContext,
|
||||
) -> tuple[DocumentationTopic, ...]:
|
||||
if context.documentation_type != "user":
|
||||
return ()
|
||||
|
||||
upload_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_upload_max_bytes",
|
||||
default=_DEFAULT_UPLOAD_MAX_BYTES,
|
||||
)
|
||||
zip_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_upload_zip_max_bytes",
|
||||
default=_DEFAULT_ZIP_MAX_BYTES,
|
||||
)
|
||||
archive_expanded_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_max_expanded_bytes",
|
||||
default=_DEFAULT_ARCHIVE_MAX_EXPANDED_BYTES,
|
||||
)
|
||||
archive_entry_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_max_entries",
|
||||
default=ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
)
|
||||
archive_ratio_limit = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_max_expansion_ratio",
|
||||
default=_DEFAULT_ARCHIVE_MAX_EXPANSION_RATIO,
|
||||
)
|
||||
archive_preview_ttl = _configured_positive_int(
|
||||
context.settings,
|
||||
"file_archive_preview_ttl_seconds",
|
||||
default=_DEFAULT_ARCHIVE_PREVIEW_TTL_SECONDS,
|
||||
)
|
||||
topics: list[DocumentationTopic] = []
|
||||
if upload_limit is not None:
|
||||
topics.append(_upload_topic(upload_limit))
|
||||
if (
|
||||
upload_limit is not None
|
||||
and zip_limit is not None
|
||||
and archive_expanded_limit is not None
|
||||
and archive_entry_limit is not None
|
||||
and archive_ratio_limit is not None
|
||||
and archive_preview_ttl is not None
|
||||
):
|
||||
topics.append(
|
||||
_archive_topic(
|
||||
upload_limit,
|
||||
zip_limit,
|
||||
archive_expanded_limit,
|
||||
archive_entry_limit,
|
||||
archive_ratio_limit,
|
||||
archive_preview_ttl,
|
||||
)
|
||||
)
|
||||
topics.append(_connector_import_topic(context))
|
||||
return tuple(topics)
|
||||
|
||||
|
||||
def _upload_topic(max_bytes: int) -> DocumentationTopic:
|
||||
limit = _format_byte_limit(max_bytes)
|
||||
return DocumentationTopic(
|
||||
id="files.workflow.upload-managed-files",
|
||||
title="Upload managed files",
|
||||
summary=f"Upload files to a personal or accessible group space; each uploaded file may contain at most {limit}.",
|
||||
body=(
|
||||
f"The current deployment accepts at most {limit} for each ordinary upload. "
|
||||
"A name conflict is never resolved silently: reject stops the upload, rename chooses a copy name, and overwrite retires the old asset before creating a new one."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "file_manager", "process_participant"),
|
||||
order=39,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(
|
||||
label="Files handbook",
|
||||
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||
kind="repository",
|
||||
),
|
||||
),
|
||||
related_modules=("campaigns",),
|
||||
unlocks=(
|
||||
"Users and connected processes can place governed content in managed storage.",
|
||||
),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/files",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list"],
|
||||
"prerequisites": [
|
||||
"You may view and upload managed files.",
|
||||
"The destination personal or group space grants this account write access; upload permission alone does not grant access to every space.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and choose My files or an accessible group space.",
|
||||
"Open the intended destination folder and choose Upload, or drag files into the file list.",
|
||||
"For every name conflict, explicitly reject, rename, overwrite, or skip the affected item.",
|
||||
"Wait for the upload to finish, then open the resulting file details.",
|
||||
],
|
||||
"outcome": "Each accepted file is stored as a governed managed asset in the selected space.",
|
||||
"verification": "Confirm the owner, logical path, size, checksum, and current version in Files.",
|
||||
"constraints": [
|
||||
{
|
||||
"id": "ordinary-upload-size",
|
||||
"label": "Maximum size per file",
|
||||
"description": f"The current safe upload limit is {limit} per file.",
|
||||
"values": [limit],
|
||||
},
|
||||
],
|
||||
"related_topic_ids": [
|
||||
"files.workflow.upload-and-unpack-zip",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.find-and-download-files",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _archive_topic(
|
||||
max_file_bytes: int,
|
||||
max_archive_request_bytes: int,
|
||||
max_expanded_bytes: int,
|
||||
max_entries: int,
|
||||
max_expansion_ratio: int,
|
||||
preview_ttl_seconds: int,
|
||||
) -> DocumentationTopic:
|
||||
member_limit = _format_byte_limit(max_file_bytes)
|
||||
request_limit = _format_byte_limit(max_archive_request_bytes)
|
||||
expanded_limit = _format_byte_limit(max_expanded_bytes)
|
||||
preview_minutes = max(1, preview_ttl_seconds // 60)
|
||||
return DocumentationTopic(
|
||||
id="files.workflow.upload-and-unpack-zip",
|
||||
title="Preview and unpack an archive",
|
||||
summary=(
|
||||
f"Review and selectively unpack up to {max_entries:,} entries from ZIP or TAR archives before any managed file is created."
|
||||
),
|
||||
body=(
|
||||
f"The deployment accepts ZIP, TAR, TAR.GZ, TAR.BZ2, and TAR.XZ requests up to {request_limit}, limits actual expanded data to {expanded_limit}, "
|
||||
f"each member to {member_limit}, the archive to {max_entries:,} entries, and expansion to {max_expansion_ratio}:1. "
|
||||
f"The server-issued preview expires after {preview_minutes} minutes. Password-protected ZIP archives are supported; passwords remain request-only. "
|
||||
"Unsafe paths and special filesystem entries are rejected. Actual extracted bytes are counted instead of trusting archive headers."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "file_manager", "process_participant"),
|
||||
order=40,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(
|
||||
label="Files handbook",
|
||||
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||
kind="repository",
|
||||
),
|
||||
),
|
||||
unlocks=(
|
||||
"A bounded archive can become governed managed files without trusting archive paths or size declarations.",
|
||||
),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/files",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list"],
|
||||
"prerequisites": [
|
||||
"You may view and upload managed files.",
|
||||
"The archive fits the configured request, expansion, size, and entry limits.",
|
||||
"The destination personal or group space grants this account write access.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and choose the managed destination space and folder.",
|
||||
"Enable Preview and unpack archive, then choose or drag one supported archive.",
|
||||
"Review the discovered entries, supply a ZIP password when required, and select the files or folders to import.",
|
||||
"Resolve every destination conflict explicitly.",
|
||||
"Confirm the selection, then wait for extraction and finalization to finish before leaving the page.",
|
||||
],
|
||||
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
|
||||
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
|
||||
"constraints": [
|
||||
{
|
||||
"id": "archive-request-and-total",
|
||||
"label": "Maximum archive request and expanded total",
|
||||
"description": f"The compressed request is limited to {request_limit}; actual expanded data is limited to {expanded_limit}.",
|
||||
"values": [request_limit, expanded_limit],
|
||||
},
|
||||
{
|
||||
"id": "archive-member-size",
|
||||
"label": "Maximum extracted member size",
|
||||
"description": f"Each extracted file is limited to {member_limit}.",
|
||||
"values": [member_limit],
|
||||
},
|
||||
{
|
||||
"id": "archive-entry-count",
|
||||
"label": "Maximum entry count",
|
||||
"description": f"An archive may contain at most {max_entries:,} declared entries.",
|
||||
"values": [f"{max_entries:,} entries"],
|
||||
},
|
||||
{
|
||||
"id": "archive-expansion-ratio",
|
||||
"label": "Maximum expansion ratio",
|
||||
"description": f"Declared and actual output may not exceed {max_expansion_ratio} times the compressed request size.",
|
||||
"values": [f"{max_expansion_ratio}:1"],
|
||||
},
|
||||
],
|
||||
"related_topic_ids": [
|
||||
"files.workflow.upload-managed-files",
|
||||
"files.workflow.organize-managed-files",
|
||||
"files.workflow.find-and-download-files",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _connector_import_topic(context: DocumentationContext) -> DocumentationTopic:
|
||||
principal = context.principal
|
||||
if not _has_all_scopes(principal, (_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE)):
|
||||
return _connector_import_limitation(
|
||||
"Connector import is not available to this account because both permission to view Files and permission to upload managed files are required."
|
||||
)
|
||||
|
||||
tenant_id = _safe_text_attribute(principal, "tenant_id")
|
||||
user_id = _safe_text_attribute(getattr(principal, "user", None), "id")
|
||||
session = context.session
|
||||
if not tenant_id or not user_id or not isinstance(session, Session):
|
||||
return _connector_import_limitation(
|
||||
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||
)
|
||||
|
||||
try:
|
||||
member_group_ids = _actor_group_ids(
|
||||
session,
|
||||
principal=principal,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
include_admin_groups=False,
|
||||
)
|
||||
connector_group_ids = (
|
||||
_actor_group_ids(
|
||||
session,
|
||||
principal=principal,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
include_admin_groups=True,
|
||||
)
|
||||
if _has_all_scopes(principal, ("files:file:admin",))
|
||||
else member_group_ids
|
||||
)
|
||||
profiles = visible_connector_profiles_for_actor(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=connector_group_ids,
|
||||
settings=context.settings,
|
||||
campaign_visible=_campaign_visibility(
|
||||
context,
|
||||
session=session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=member_group_ids,
|
||||
),
|
||||
include_effective_policy=True,
|
||||
)
|
||||
usable_profiles = tuple(
|
||||
profile
|
||||
for profile in profiles
|
||||
if connector_profile_usable_for_import(profile)
|
||||
)
|
||||
except Exception:
|
||||
return _connector_import_limitation(
|
||||
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
|
||||
)
|
||||
if not usable_profiles:
|
||||
return _connector_import_limitation(
|
||||
"No connection visible to this account is currently eligible to offer an enabled, credential-ready, policy-allowed browse/import path through a pinning-safe provider. Ask a Files administrator to configure or authorize one."
|
||||
)
|
||||
|
||||
return DocumentationTopic(
|
||||
id="files.workflow.import-managed-snapshot",
|
||||
title="Import an external file as a governed snapshot",
|
||||
summary="Browse an authorized connection read-only and import one selected file into managed storage as a frozen, traceable snapshot.",
|
||||
body=(
|
||||
"At least one connection visible to this account is currently eligible to offer the governed browse/import path. "
|
||||
"The selected remote path and item are re-authorized when used, and browse, import, and sync never mutate the remote source. "
|
||||
"The managed snapshot stays unchanged until an explicit manual sync."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("user",),
|
||||
audience=("file_user", "campaign_manager", "report_author"),
|
||||
order=41,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(
|
||||
label="Files handbook",
|
||||
href="govoplan-files/docs/FILES_HANDBOOK.md",
|
||||
kind="repository",
|
||||
),
|
||||
),
|
||||
related_modules=("campaigns",),
|
||||
unlocks=(
|
||||
"Campaigns, reports, and workflows can consume a managed snapshot with stable source evidence.",
|
||||
),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/files",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list", "files.connector-import"],
|
||||
"prerequisites": [
|
||||
"You may view and upload managed files.",
|
||||
"At least one enabled, credential-ready, policy-allowed connection using a pinning-safe provider is visible to this account.",
|
||||
"The selected remote path and item must pass their operation-time policy checks.",
|
||||
"The managed destination space grants this account write access.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Files and choose a managed destination space.",
|
||||
"Choose Sync from connection, select an available connection, and browse to the permitted remote file.",
|
||||
"Import the selected file and resolve any destination conflict explicitly.",
|
||||
"Review the managed file's source and current-version details before using it in another task.",
|
||||
],
|
||||
"current_configuration": [
|
||||
"At least one actor-visible connection is eligible to offer a safe browse/import operation; the selected endpoint, path, and item are still checked when used.",
|
||||
"Every selected remote path and item is re-authorized at operation time.",
|
||||
],
|
||||
"outcome": "The external content is a tenant-managed snapshot with a checksum, exact version, and recorded source context.",
|
||||
"verification": "Reopen the managed file and confirm its recorded source context, source revision when available, checksum, and current version.",
|
||||
"related_topic_ids": [
|
||||
"files.governed-connectors-and-provenance",
|
||||
"files.reference.integrity-recovery-and-fail-closed-transports",
|
||||
"files.reference.snapshot-provenance-and-capabilities",
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _connector_import_limitation(message: str) -> DocumentationTopic:
|
||||
return DocumentationTopic(
|
||||
id="files.connector-import-unavailable",
|
||||
title="External file import is not currently available",
|
||||
summary=message,
|
||||
body=(
|
||||
f"{message} Files never exposes connection endpoints, storage paths, credential references, or raw connector policies in user documentation."
|
||||
),
|
||||
layer="available",
|
||||
documentation_types=("user",),
|
||||
order=41,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
any_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
|
||||
),
|
||||
),
|
||||
links=(DocumentationLink(label="Files", href="/files", kind="runtime"),),
|
||||
source_module_id="files",
|
||||
metadata={
|
||||
"kind": "reference",
|
||||
"screen": "Files",
|
||||
"help_contexts": ["files.list", "files.connector-import"],
|
||||
"limitations": [message],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _configured_positive_int(
|
||||
settings: object | None, name: str, *, default: int
|
||||
) -> int | None:
|
||||
raw_value = getattr(settings, name, default) if settings is not None else default
|
||||
try:
|
||||
value = int(raw_value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return value if value > 0 else None
|
||||
|
||||
|
||||
def _format_byte_limit(value: int) -> str:
|
||||
units = ((1024**3, "GiB"), (1024**2, "MiB"), (1024, "KiB"))
|
||||
for divisor, label in units:
|
||||
if value % divisor == 0:
|
||||
return f"{value // divisor:,} {label} ({value:,} bytes)"
|
||||
return f"{value:,} bytes"
|
||||
|
||||
|
||||
def _has_all_scopes(principal: object | None, scopes: tuple[str, ...]) -> bool:
|
||||
checker = getattr(principal, "has", None)
|
||||
if callable(checker):
|
||||
try:
|
||||
return all(bool(checker(scope)) for scope in scopes)
|
||||
except Exception:
|
||||
return False
|
||||
granted = {str(scope) for scope in getattr(principal, "scopes", ())}
|
||||
return all(scope in granted for scope in scopes)
|
||||
|
||||
|
||||
def _safe_text_attribute(value: object | None, name: str) -> str:
|
||||
try:
|
||||
result = getattr(value, name, "")
|
||||
except Exception:
|
||||
return ""
|
||||
return str(result or "")
|
||||
|
||||
|
||||
def _principal_group_ids(principal: object) -> tuple[str, ...]:
|
||||
try:
|
||||
values = getattr(principal, "group_ids", ())
|
||||
except Exception:
|
||||
return ()
|
||||
return tuple(str(value) for value in values if str(value))
|
||||
|
||||
|
||||
def _actor_group_ids(
|
||||
session: Session,
|
||||
*,
|
||||
principal: object,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
include_admin_groups: bool,
|
||||
) -> tuple[str, ...]:
|
||||
try:
|
||||
values = user_group_ids(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
include_admin_groups=include_admin_groups,
|
||||
)
|
||||
except Exception:
|
||||
return _principal_group_ids(principal)
|
||||
return tuple(str(value) for value in values if str(value))
|
||||
|
||||
|
||||
def _campaign_visibility(
|
||||
context: DocumentationContext,
|
||||
*,
|
||||
session: Session,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: tuple[str, ...],
|
||||
):
|
||||
principal = context.principal
|
||||
registry = context.registry
|
||||
if not _has_all_scopes(principal, ("campaigns:campaign:read",)):
|
||||
return None
|
||||
try:
|
||||
if not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||
return None
|
||||
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(capability, CampaignAccessProvider):
|
||||
return None
|
||||
|
||||
def campaign_visible(campaign_id: str) -> bool:
|
||||
try:
|
||||
return capability.campaign_exists(
|
||||
session, tenant_id=tenant_id, campaign_id=campaign_id
|
||||
) and capability.can_read_campaign(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
user_id=user_id,
|
||||
group_ids=group_ids,
|
||||
tenant_admin=_has_all_scopes(principal, ("tenant:*",)),
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
return campaign_visible
|
||||
@@ -0,0 +1,879 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import inspect, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import (
|
||||
DsarErasureActionRef,
|
||||
DsarExecutionResultRef,
|
||||
DsarRecordRef,
|
||||
DsarSubjectRef,
|
||||
dsar_capability_name,
|
||||
)
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
|
||||
|
||||
FILES_DSAR_CAPABILITY = dsar_capability_name("files")
|
||||
_MAX_RECORDS = 5_000
|
||||
|
||||
|
||||
class FilesDsarProvider:
|
||||
provider_id = "files"
|
||||
module_id = "files"
|
||||
|
||||
def search_subject(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
) -> Sequence[DsarRecordRef]:
|
||||
db = _session(session)
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return ()
|
||||
|
||||
records: list[DsarRecordRef] = []
|
||||
form_evidence_available = _has_table(db, FileFormEvidenceGrant)
|
||||
campaign_evidence_available = _has_table(db, CampaignAttachmentUse)
|
||||
retention_reasons: dict[str, str | None] = {}
|
||||
|
||||
def retention_reason(asset: FileAsset) -> str | None:
|
||||
if asset.id not in retention_reasons:
|
||||
retention_reasons[asset.id] = _asset_retention_reason(
|
||||
db,
|
||||
asset,
|
||||
form_evidence_available=form_evidence_available,
|
||||
campaign_evidence_available=campaign_evidence_available,
|
||||
)
|
||||
return retention_reasons[asset.id]
|
||||
|
||||
def append(record: DsarRecordRef) -> None:
|
||||
if len(records) >= _MAX_RECORDS:
|
||||
raise ValueError(
|
||||
"Files DSAR match limit exceeded; narrow the subject selectors."
|
||||
)
|
||||
records.append(record)
|
||||
|
||||
assets = _bounded_rows(
|
||||
db.query(FileAsset)
|
||||
.filter(
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
FileAsset.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileAsset.id)
|
||||
)
|
||||
for asset in assets:
|
||||
match_fields = _matching_fields(
|
||||
asset,
|
||||
subject_user_id,
|
||||
("owner_user_id", "created_by_user_id"),
|
||||
)
|
||||
asset_retention_reason = retention_reason(asset)
|
||||
append(
|
||||
_record(
|
||||
"file_asset",
|
||||
asset.id,
|
||||
"managed_file",
|
||||
asset.filename,
|
||||
{
|
||||
"match_fields": match_fields,
|
||||
"owner_type": asset.owner_type,
|
||||
"display_path": asset.display_path,
|
||||
"filename": asset.filename,
|
||||
"description": asset.description,
|
||||
"deleted_at": _iso(asset.deleted_at),
|
||||
"retained_until": _iso(asset.retained_until),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_reason": asset.lifecycle_reason,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
},
|
||||
observed_at=asset.updated_at,
|
||||
immutable=asset_retention_reason is not None,
|
||||
retention_reason=asset_retention_reason,
|
||||
source_path=f"/files?file={asset.id}",
|
||||
)
|
||||
)
|
||||
|
||||
version_rows = _bounded_rows(
|
||||
db.query(FileVersion, FileAsset)
|
||||
.join(FileAsset, FileAsset.id == FileVersion.file_asset_id)
|
||||
.filter(
|
||||
FileVersion.tenant_id == tenant_id,
|
||||
FileAsset.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
FileVersion.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileVersion.id)
|
||||
)
|
||||
for version, asset in version_rows:
|
||||
asset_retention_reason = retention_reason(asset)
|
||||
append(
|
||||
_record(
|
||||
"file_version",
|
||||
version.id,
|
||||
"managed_file_version",
|
||||
version.filename_at_upload,
|
||||
{
|
||||
"match_fields": (
|
||||
["asset.owner_user_id"]
|
||||
if asset.owner_user_id == subject_user_id
|
||||
else []
|
||||
)
|
||||
+ (
|
||||
["created_by_user_id"]
|
||||
if version.created_by_user_id == subject_user_id
|
||||
else []
|
||||
),
|
||||
"file_asset_id": version.file_asset_id,
|
||||
"version_number": version.version_number,
|
||||
"filename_at_upload": version.filename_at_upload,
|
||||
"display_path_at_upload": version.display_path_at_upload,
|
||||
"content_type": version.content_type,
|
||||
"size_bytes": version.size_bytes,
|
||||
"checksum_sha256": version.checksum_sha256,
|
||||
"created_at": _iso(version.created_at),
|
||||
},
|
||||
observed_at=version.updated_at,
|
||||
immutable=asset_retention_reason is not None,
|
||||
retention_reason=asset_retention_reason,
|
||||
source_path=(
|
||||
f"/api/v1/files/{version.file_asset_id}/versions/"
|
||||
f"{version.id}/download"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for folder in _bounded_rows(
|
||||
db.query(FileFolder)
|
||||
.filter(
|
||||
FileFolder.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileFolder.owner_user_id == subject_user_id,
|
||||
FileFolder.created_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileFolder.id)
|
||||
):
|
||||
append(
|
||||
_record(
|
||||
"file_folder",
|
||||
folder.id,
|
||||
"managed_folder",
|
||||
folder.path,
|
||||
{
|
||||
"match_fields": _matching_fields(
|
||||
folder,
|
||||
subject_user_id,
|
||||
("owner_user_id", "created_by_user_id"),
|
||||
),
|
||||
"owner_type": folder.owner_type,
|
||||
"path": folder.path,
|
||||
"deleted_at": _iso(folder.deleted_at),
|
||||
},
|
||||
observed_at=folder.updated_at,
|
||||
source_path="/files",
|
||||
)
|
||||
)
|
||||
|
||||
for share in _bounded_rows(
|
||||
db.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
or_(
|
||||
(FileShare.target_type == "user")
|
||||
& (FileShare.target_id == subject_user_id),
|
||||
FileShare.created_by_user_id == subject_user_id,
|
||||
FileShare.revoked_by_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileShare.id)
|
||||
):
|
||||
target_matches = (
|
||||
share.target_type == "user" and share.target_id == subject_user_id
|
||||
)
|
||||
match_fields = _matching_fields(
|
||||
share,
|
||||
subject_user_id,
|
||||
("created_by_user_id", "revoked_by_user_id"),
|
||||
)
|
||||
if target_matches:
|
||||
match_fields.insert(0, "target_id")
|
||||
append(
|
||||
_record(
|
||||
"file_share",
|
||||
share.id,
|
||||
"file_access_evidence",
|
||||
f"Share for file {share.file_asset_id}",
|
||||
{
|
||||
"match_fields": match_fields,
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_is_subject": target_matches,
|
||||
"permission": share.permission,
|
||||
"expires_at": _iso(share.expires_at),
|
||||
"revoked_at": _iso(share.revoked_at),
|
||||
},
|
||||
observed_at=share.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"File-sharing history is institutional access evidence."
|
||||
),
|
||||
source_path=f"/files?file={share.file_asset_id}",
|
||||
)
|
||||
)
|
||||
|
||||
if form_evidence_available:
|
||||
evidence_rows = _bounded_rows(
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.outerjoin(
|
||||
FileAsset,
|
||||
FileAsset.id == FileFormEvidenceGrant.file_asset_id,
|
||||
)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == tenant_id,
|
||||
or_(
|
||||
FileFormEvidenceGrant.custodian_user_id == subject_user_id,
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
),
|
||||
)
|
||||
.order_by(FileFormEvidenceGrant.id)
|
||||
)
|
||||
for grant in evidence_rows:
|
||||
append(
|
||||
_record(
|
||||
"file_form_evidence",
|
||||
grant.id,
|
||||
"form_evidence",
|
||||
f"Form evidence {grant.form_definition_id}",
|
||||
{
|
||||
"match_fields": (
|
||||
["custodian_user_id"]
|
||||
if grant.custodian_user_id == subject_user_id
|
||||
else ["asset.owner_user_id"]
|
||||
),
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"form_definition_id": grant.form_definition_id,
|
||||
"form_definition_revision": grant.form_definition_revision,
|
||||
"evidence_kind": grant.evidence_kind,
|
||||
"purpose": grant.purpose,
|
||||
"status": grant.status,
|
||||
"expires_at": _iso(grant.expires_at),
|
||||
"file_asset_id": grant.file_asset_id,
|
||||
"file_version_id": grant.file_version_id,
|
||||
},
|
||||
observed_at=grant.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Submitted Form attachment evidence follows the owning "
|
||||
"process retention and cannot be erased through Files alone."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
if campaign_evidence_available:
|
||||
campaign_rows = _bounded_rows(
|
||||
db.query(CampaignAttachmentUse)
|
||||
.join(FileAsset, FileAsset.id == CampaignAttachmentUse.file_asset_id)
|
||||
.filter(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
FileAsset.owner_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(CampaignAttachmentUse.id)
|
||||
)
|
||||
for use in campaign_rows:
|
||||
append(
|
||||
_record(
|
||||
"campaign_attachment_use",
|
||||
use.id,
|
||||
"delivery_evidence",
|
||||
use.filename_used,
|
||||
{
|
||||
"match_fields": ["asset.owner_user_id"],
|
||||
"campaign_id": use.campaign_id,
|
||||
"campaign_version_id": use.campaign_version_id,
|
||||
"campaign_job_id": use.campaign_job_id,
|
||||
"file_asset_id": use.file_asset_id,
|
||||
"file_version_id": use.file_version_id,
|
||||
"filename_used": use.filename_used,
|
||||
"checksum_sha256": use.checksum_sha256,
|
||||
"size_bytes": use.size_bytes,
|
||||
"use_stage": use.use_stage,
|
||||
"used_at": _iso(use.used_at),
|
||||
},
|
||||
observed_at=use.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Campaign attachment use is immutable delivery evidence."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
self._append_configuration_references(
|
||||
db,
|
||||
append=append,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
)
|
||||
self._append_integrity_references(
|
||||
db,
|
||||
append=append,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
)
|
||||
return tuple(records)
|
||||
|
||||
def plan_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
records: Sequence[DsarRecordRef],
|
||||
) -> Sequence[DsarErasureActionRef]:
|
||||
del session
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return ()
|
||||
actions: list[DsarErasureActionRef] = []
|
||||
for record in records:
|
||||
if record.provider_id != self.provider_id or record.module_id != self.module_id:
|
||||
raise ValueError("Files DSAR received a foreign provider record.")
|
||||
match_fields = {
|
||||
str(value) for value in record.data.get("match_fields", ())
|
||||
}
|
||||
if record.immutable_evidence:
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:retain:{record.resource_type}:{record.resource_id}",
|
||||
"retain",
|
||||
record,
|
||||
f"Retain {record.title}",
|
||||
record.retention_reason
|
||||
or "Institutional evidence must be retained.",
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
elif record.resource_type in {
|
||||
"file_asset",
|
||||
"file_version",
|
||||
"file_folder",
|
||||
}:
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:review:{record.resource_type}:{record.resource_id}",
|
||||
"manual_review",
|
||||
record,
|
||||
f"Review {record.title}",
|
||||
(
|
||||
"Managed file content, names, paths, ownership, and shared "
|
||||
"references require a case decision. Approved byte erasure "
|
||||
"must use the separate Files purge workflow."
|
||||
),
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
if (
|
||||
record.resource_type == "file_share"
|
||||
and "target_id" in match_fields
|
||||
and record.data.get("revoked_at") is None
|
||||
):
|
||||
actions.append(
|
||||
_action(
|
||||
f"files:revoke:file_share:{record.resource_id}",
|
||||
"revoke",
|
||||
record,
|
||||
"Revoke active file share",
|
||||
"The active user-targeted share can be revoked without deleting file evidence.",
|
||||
executable=True,
|
||||
metadata={"subject_user_id": subject_user_id},
|
||||
)
|
||||
)
|
||||
for field_name in sorted(
|
||||
match_fields.intersection(_detachable_fields(record.resource_type))
|
||||
):
|
||||
actions.append(
|
||||
_action(
|
||||
(
|
||||
f"files:detach:{record.resource_type}:"
|
||||
f"{field_name}:{record.resource_id}"
|
||||
),
|
||||
"detach",
|
||||
record,
|
||||
f"Detach {field_name.replace('_', ' ')}",
|
||||
(
|
||||
"Remove the mutable subject reference while preserving the "
|
||||
"governed resource and DSAR evidence."
|
||||
),
|
||||
executable=True,
|
||||
metadata={
|
||||
"field": field_name,
|
||||
"subject_user_id": subject_user_id,
|
||||
},
|
||||
)
|
||||
)
|
||||
action_ids = [action.action_id for action in actions]
|
||||
if len(action_ids) != len(set(action_ids)):
|
||||
raise ValueError("Files DSAR produced duplicate action ids.")
|
||||
return tuple(actions)
|
||||
|
||||
def execute_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
actions: Sequence[DsarErasureActionRef],
|
||||
request_id: str,
|
||||
) -> Sequence[DsarExecutionResultRef]:
|
||||
db = _session(session)
|
||||
subject_user_id = _subject_user_id(subject)
|
||||
if subject_user_id is None:
|
||||
return tuple(
|
||||
_blocked(action, "Files requires a direct membership subject reference.")
|
||||
for action in actions
|
||||
)
|
||||
results: list[DsarExecutionResultRef] = []
|
||||
for action in actions:
|
||||
if (
|
||||
action.provider_id != self.provider_id
|
||||
or action.module_id != self.module_id
|
||||
or action.metadata.get("subject_user_id") != subject_user_id
|
||||
):
|
||||
results.append(_blocked(action, "The Files DSAR action is stale or invalid."))
|
||||
continue
|
||||
if action.action_id.startswith("files:revoke:file_share:"):
|
||||
results.append(
|
||||
_revoke_share(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
action=action,
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
elif action.action_id.startswith("files:detach:"):
|
||||
results.append(
|
||||
_detach_reference(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
subject_user_id=subject_user_id,
|
||||
action=action,
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
else:
|
||||
results.append(_blocked(action, "Files does not execute this action kind."))
|
||||
db.flush()
|
||||
return tuple(results)
|
||||
|
||||
def _append_configuration_references(
|
||||
self,
|
||||
db: Session,
|
||||
*,
|
||||
append: object,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
) -> None:
|
||||
configurations = (
|
||||
(FileConnectorProfile, "connector_profile", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorCredential, "connector_credential", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorPolicy, "connector_policy", ("created_by_user_id", "updated_by_user_id")),
|
||||
(FileConnectorSpace, "connector_space", ("owner_user_id", "created_by_user_id")),
|
||||
)
|
||||
for model, resource_type, fields in configurations:
|
||||
if not _has_table(db, model):
|
||||
continue
|
||||
conditions = [getattr(model, field) == subject_user_id for field in fields]
|
||||
query = db.query(model).filter(or_(*conditions))
|
||||
if hasattr(model, "tenant_id"):
|
||||
query = query.filter(model.tenant_id == tenant_id)
|
||||
for row in _bounded_rows(query.order_by(model.id)):
|
||||
match_fields = _matching_fields(row, subject_user_id, fields)
|
||||
data: dict[str, object] = {
|
||||
"match_fields": match_fields,
|
||||
"label": getattr(row, "label", None),
|
||||
"provider": getattr(row, "provider", None),
|
||||
}
|
||||
if isinstance(row, FileConnectorCredential):
|
||||
data["credential_mode"] = row.credential_mode
|
||||
elif isinstance(row, FileConnectorSpace):
|
||||
data.update(
|
||||
{
|
||||
"remote_path": row.remote_path,
|
||||
"sync_mode": row.sync_mode,
|
||||
"read_only": row.read_only,
|
||||
"deleted_at": _iso(row.deleted_at),
|
||||
}
|
||||
)
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
resource_type,
|
||||
row.id,
|
||||
"connector_configuration_evidence",
|
||||
getattr(row, "label", None) or resource_type.replace("_", " "),
|
||||
data,
|
||||
observed_at=row.updated_at,
|
||||
immutable=True,
|
||||
retention_reason=(
|
||||
"Connector configuration history is institutional evidence; "
|
||||
"credential secrets are excluded from the DSAR export."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def _append_integrity_references(
|
||||
self,
|
||||
db: Session,
|
||||
*,
|
||||
append: object,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
) -> None:
|
||||
if _has_table(db, FileIntegrityScan):
|
||||
for scan in _bounded_rows(
|
||||
db.query(FileIntegrityScan)
|
||||
.filter(
|
||||
FileIntegrityScan.tenant_id == tenant_id,
|
||||
FileIntegrityScan.created_by_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(FileIntegrityScan.id)
|
||||
):
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
"file_integrity_scan",
|
||||
scan.id,
|
||||
"storage_integrity_evidence",
|
||||
f"Integrity scan {scan.id}",
|
||||
{
|
||||
"match_fields": ["created_by_user_id"],
|
||||
"storage_backend": scan.storage_backend,
|
||||
"status": scan.status,
|
||||
"started_at": _iso(scan.started_at),
|
||||
"completed_at": _iso(scan.completed_at),
|
||||
},
|
||||
observed_at=scan.updated_at,
|
||||
immutable=True,
|
||||
retention_reason="Storage integrity scans are operator evidence.",
|
||||
)
|
||||
)
|
||||
if _has_table(db, FileIntegrityFinding):
|
||||
for finding in _bounded_rows(
|
||||
db.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.tenant_id == tenant_id,
|
||||
FileIntegrityFinding.resolved_by_user_id == subject_user_id,
|
||||
)
|
||||
.order_by(FileIntegrityFinding.id)
|
||||
):
|
||||
append( # type: ignore[operator]
|
||||
_record(
|
||||
"file_integrity_finding",
|
||||
finding.id,
|
||||
"storage_integrity_evidence",
|
||||
f"Integrity finding {finding.kind}",
|
||||
{
|
||||
"match_fields": ["resolved_by_user_id"],
|
||||
"kind": finding.kind,
|
||||
"state": finding.state,
|
||||
"resolved_at": _iso(finding.resolved_at),
|
||||
},
|
||||
observed_at=finding.updated_at,
|
||||
immutable=True,
|
||||
retention_reason="Integrity resolution is operator evidence.",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _asset_retention_reason(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
form_evidence_available: bool,
|
||||
campaign_evidence_available: bool,
|
||||
) -> str | None:
|
||||
reasons: list[str] = []
|
||||
if asset.legal_hold:
|
||||
reasons.append("The file is under legal hold.")
|
||||
retained_until = _aware(asset.retained_until)
|
||||
if retained_until is not None and retained_until > datetime.now(timezone.utc):
|
||||
reasons.append(f"The file is retained until {retained_until.isoformat()}.")
|
||||
if form_evidence_available:
|
||||
has_form_evidence = (
|
||||
session.query(FileFormEvidenceGrant.id)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == asset.tenant_id,
|
||||
FileFormEvidenceGrant.file_asset_id == asset.id,
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
if has_form_evidence:
|
||||
reasons.append("The file is referenced by submitted Form evidence.")
|
||||
if campaign_evidence_available:
|
||||
has_campaign_evidence = (
|
||||
session.query(CampaignAttachmentUse.id)
|
||||
.filter(
|
||||
CampaignAttachmentUse.tenant_id == asset.tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == asset.id,
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
if has_campaign_evidence:
|
||||
reasons.append("The file is referenced by Campaign delivery evidence.")
|
||||
return " ".join(reasons) or None
|
||||
|
||||
|
||||
_DETACHABLE_MODELS: dict[str, tuple[type[object], frozenset[str]]] = {
|
||||
"file_asset": (FileAsset, frozenset({"created_by_user_id"})),
|
||||
"file_version": (FileVersion, frozenset({"created_by_user_id"})),
|
||||
"file_folder": (FileFolder, frozenset({"created_by_user_id"})),
|
||||
"file_share": (
|
||||
FileShare,
|
||||
frozenset({"created_by_user_id", "revoked_by_user_id"}),
|
||||
),
|
||||
"connector_profile": (
|
||||
FileConnectorProfile,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_credential": (
|
||||
FileConnectorCredential,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_policy": (
|
||||
FileConnectorPolicy,
|
||||
frozenset({"created_by_user_id", "updated_by_user_id"}),
|
||||
),
|
||||
"connector_space": (FileConnectorSpace, frozenset({"created_by_user_id"})),
|
||||
}
|
||||
|
||||
|
||||
def _detachable_fields(resource_type: str) -> frozenset[str]:
|
||||
entry = _DETACHABLE_MODELS.get(resource_type)
|
||||
return entry[1] if entry is not None else frozenset()
|
||||
|
||||
|
||||
def _revoke_share(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
action: DsarErasureActionRef,
|
||||
request_id: str,
|
||||
) -> DsarExecutionResultRef:
|
||||
row = (
|
||||
session.query(FileShare)
|
||||
.filter(FileShare.id == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if (
|
||||
row is None
|
||||
or row.tenant_id != tenant_id
|
||||
or row.target_type != "user"
|
||||
or row.target_id != subject_user_id
|
||||
):
|
||||
return _blocked(action, "The subject-targeted file share is no longer available.")
|
||||
if row.revoked_at is not None:
|
||||
return _result(
|
||||
action,
|
||||
"unchanged",
|
||||
"The file share was already revoked.",
|
||||
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||
)
|
||||
row.revoked_at = datetime.now(timezone.utc)
|
||||
row.revoked_by_user_id = None
|
||||
return _result(
|
||||
action,
|
||||
"executed",
|
||||
"The subject-targeted file share was revoked.",
|
||||
{"request_id": request_id, "revoked_at": _iso(row.revoked_at)},
|
||||
)
|
||||
|
||||
|
||||
def _detach_reference(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject_user_id: str,
|
||||
action: DsarErasureActionRef,
|
||||
request_id: str,
|
||||
) -> DsarExecutionResultRef:
|
||||
entry = _DETACHABLE_MODELS.get(action.resource_type)
|
||||
field_name = str(action.metadata.get("field") or "")
|
||||
if entry is None or field_name not in entry[1]:
|
||||
return _blocked(action, "The requested Files subject reference is not detachable.")
|
||||
model = entry[0]
|
||||
row = (
|
||||
session.query(model)
|
||||
.filter(getattr(model, "id") == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if row is None or getattr(row, "tenant_id", None) != tenant_id:
|
||||
return _blocked(action, "The Files resource is no longer available.")
|
||||
current = getattr(row, field_name)
|
||||
if current is None:
|
||||
return _result(
|
||||
action,
|
||||
"unchanged",
|
||||
"The subject reference was already detached.",
|
||||
{"request_id": request_id, "field": field_name},
|
||||
)
|
||||
if current != subject_user_id:
|
||||
return _blocked(action, "The Files subject reference changed after planning.")
|
||||
setattr(row, field_name, None)
|
||||
return _result(
|
||||
action,
|
||||
"executed",
|
||||
"The mutable subject reference was detached.",
|
||||
{"request_id": request_id, "field": field_name},
|
||||
)
|
||||
|
||||
|
||||
def _subject_user_id(subject: DsarSubjectRef) -> str | None:
|
||||
candidates: list[str] = []
|
||||
if subject.membership_id:
|
||||
candidates.append(subject.membership_id)
|
||||
for key, value in subject.external_references.items():
|
||||
if key in {
|
||||
"files.user",
|
||||
"files.membership",
|
||||
"access.membership",
|
||||
"membership_id",
|
||||
}:
|
||||
candidates.append(value)
|
||||
normalized = {value.strip() for value in candidates if value.strip()}
|
||||
if len(normalized) != 1:
|
||||
return None
|
||||
return normalized.pop()
|
||||
|
||||
|
||||
def _matching_fields(
|
||||
row: object,
|
||||
subject_user_id: str,
|
||||
fields: Sequence[str],
|
||||
) -> list[str]:
|
||||
return [field for field in fields if getattr(row, field) == subject_user_id]
|
||||
|
||||
|
||||
def _record(
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
category: str,
|
||||
title: str,
|
||||
data: dict[str, object],
|
||||
*,
|
||||
observed_at: datetime | None = None,
|
||||
immutable: bool = False,
|
||||
retention_reason: str | None = None,
|
||||
source_path: str | None = None,
|
||||
) -> DsarRecordRef:
|
||||
return DsarRecordRef(
|
||||
provider_id="files",
|
||||
module_id="files",
|
||||
resource_type=resource_type,
|
||||
resource_id=resource_id,
|
||||
category=category,
|
||||
title=title,
|
||||
data=data,
|
||||
observed_at=observed_at,
|
||||
immutable_evidence=immutable,
|
||||
retention_reason=retention_reason,
|
||||
source_path=source_path,
|
||||
)
|
||||
|
||||
|
||||
def _action(
|
||||
action_id: str,
|
||||
kind: str,
|
||||
record: DsarRecordRef,
|
||||
title: str,
|
||||
rationale: str,
|
||||
*,
|
||||
executable: bool,
|
||||
metadata: dict[str, object] | None = None,
|
||||
) -> DsarErasureActionRef:
|
||||
return DsarErasureActionRef(
|
||||
action_id=action_id,
|
||||
provider_id="files",
|
||||
module_id="files",
|
||||
kind=kind, # type: ignore[arg-type]
|
||||
resource_type=record.resource_type,
|
||||
resource_id=record.resource_id,
|
||||
title=title,
|
||||
rationale=rationale,
|
||||
executable=executable,
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
|
||||
def _result(
|
||||
action: DsarErasureActionRef,
|
||||
status: str,
|
||||
summary: str,
|
||||
evidence: dict[str, object] | None = None,
|
||||
) -> DsarExecutionResultRef:
|
||||
return DsarExecutionResultRef(
|
||||
action_id=action.action_id,
|
||||
status=status, # type: ignore[arg-type]
|
||||
summary=summary,
|
||||
evidence=evidence or {},
|
||||
)
|
||||
|
||||
|
||||
def _blocked(action: DsarErasureActionRef, summary: str) -> DsarExecutionResultRef:
|
||||
return _result(action, "blocked", summary)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files DSAR provider requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
def _bounded_rows(query: object) -> list[object]:
|
||||
rows = query.limit(_MAX_RECORDS + 1).all() # type: ignore[attr-defined]
|
||||
if len(rows) > _MAX_RECORDS:
|
||||
raise ValueError("Files DSAR match limit exceeded; narrow the subject selectors.")
|
||||
return rows
|
||||
|
||||
|
||||
def _has_table(session: Session, model: type[object]) -> bool:
|
||||
return inspect(session.connection()).has_table(model.__tablename__)
|
||||
|
||||
|
||||
def _aware(value: datetime | None) -> datetime | None:
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
aware = _aware(value)
|
||||
return aware.isoformat() if aware else None
|
||||
|
||||
|
||||
__all__ = ["FILES_DSAR_CAPABILITY", "FilesDsarProvider"]
|
||||
@@ -0,0 +1,449 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.access import CAPABILITY_ACCESS_DIRECTORY, AccessDirectory
|
||||
from govoplan_core.core.form_evidence import (
|
||||
FormEvidenceContractError,
|
||||
FormEvidenceGrant,
|
||||
FormEvidenceGrantRequest,
|
||||
FormEvidenceInspection,
|
||||
FormEvidenceInspectionRequest,
|
||||
FormEvidenceState,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileFormEvidenceGrant,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.runtime import configure_runtime
|
||||
|
||||
|
||||
PROVIDER_ID = "files"
|
||||
CAPABILITY_FORM_EVIDENCE_FILES = "forms_runtime.evidence.files"
|
||||
MAX_GRANT_TTL = timedelta(minutes=15)
|
||||
|
||||
|
||||
class FilesFormEvidenceProvider:
|
||||
provider_id = PROVIDER_ID
|
||||
|
||||
def __init__(self, registry: object | None, settings: object) -> None:
|
||||
self._registry = registry
|
||||
self._settings = settings
|
||||
|
||||
def supported_kinds(self) -> Sequence[str]:
|
||||
return ("document",)
|
||||
|
||||
def create_upload_grant(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: FormEvidenceGrantRequest,
|
||||
) -> FormEvidenceGrant:
|
||||
db = _session(session)
|
||||
_assert_tenant(principal, request.tenant_id)
|
||||
if request.evidence_kind != "document":
|
||||
raise FormEvidenceContractError(
|
||||
"Files can accept only document evidence for Forms Runtime."
|
||||
)
|
||||
custodian_user_id = _custodian_user_id(request.custodian_ref)
|
||||
self._assert_active_custodian(
|
||||
tenant_id=request.tenant_id,
|
||||
user_id=custodian_user_id,
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
if request.expires_at <= now:
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence upload grant expiry must be in the future."
|
||||
)
|
||||
allowed_content_types = _content_types(request.allowed_content_types)
|
||||
configured_max = int(getattr(self._settings, "file_upload_max_bytes"))
|
||||
max_size_bytes = min(request.max_size_bytes or configured_max, configured_max)
|
||||
request_sha256 = _request_sha256(
|
||||
request,
|
||||
custodian_user_id=custodian_user_id,
|
||||
max_size_bytes=max_size_bytes,
|
||||
allowed_content_types=allowed_content_types,
|
||||
)
|
||||
existing = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.idempotency_key == request.idempotency_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is not None:
|
||||
if not secrets.compare_digest(existing.request_sha256, request_sha256):
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence grant idempotency conflict."
|
||||
)
|
||||
if (
|
||||
existing.status in {"expired", "revoked"}
|
||||
or _aware(existing.expires_at) <= now
|
||||
):
|
||||
raise FormEvidenceContractError(
|
||||
"The existing Form evidence upload grant is no longer usable; "
|
||||
"request a new grant with a new idempotency key."
|
||||
)
|
||||
return _grant_response(existing, upload_token=None, replayed=True)
|
||||
|
||||
metadata = _bounded_metadata(request.metadata)
|
||||
remaining_attachments = metadata.get("remaining_attachments")
|
||||
if isinstance(remaining_attachments, int):
|
||||
existing_attachment_ids = set(metadata.get("existing_attachment_ids", ()))
|
||||
active_grants = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||
FileFormEvidenceGrant.form_definition_id
|
||||
== request.definition_ref.object_id,
|
||||
FileFormEvidenceGrant.form_definition_revision
|
||||
== str(request.definition_ref.version),
|
||||
FileFormEvidenceGrant.status.in_(("issued", "uploaded")),
|
||||
FileFormEvidenceGrant.expires_at > now,
|
||||
)
|
||||
.with_for_update()
|
||||
.all()
|
||||
)
|
||||
outstanding = sum(
|
||||
1
|
||||
for item in active_grants
|
||||
if not item.file_asset_id
|
||||
or item.file_asset_id not in existing_attachment_ids
|
||||
)
|
||||
if outstanding >= remaining_attachments:
|
||||
raise FormEvidenceContractError(
|
||||
"This Form already has the maximum number of active attachment uploads."
|
||||
)
|
||||
|
||||
upload_token = secrets.token_urlsafe(32)
|
||||
grant = FileFormEvidenceGrant(
|
||||
tenant_id=request.tenant_id,
|
||||
form_instance_id=request.instance_id,
|
||||
form_definition_id=request.definition_ref.object_id,
|
||||
form_definition_revision=str(request.definition_ref.version),
|
||||
token_sha256=_token_sha256(upload_token),
|
||||
idempotency_key=request.idempotency_key,
|
||||
request_sha256=request_sha256,
|
||||
custodian_user_id=custodian_user_id,
|
||||
evidence_kind=request.evidence_kind,
|
||||
purpose=request.purpose,
|
||||
status="issued",
|
||||
expires_at=min(request.expires_at, now + MAX_GRANT_TTL),
|
||||
max_size_bytes=max_size_bytes,
|
||||
allowed_content_types=list(allowed_content_types),
|
||||
metadata_=metadata,
|
||||
)
|
||||
db.add(grant)
|
||||
db.flush()
|
||||
return _grant_response(grant, upload_token=upload_token, replayed=False)
|
||||
|
||||
def inspect_evidence(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
request: FormEvidenceInspectionRequest,
|
||||
) -> FormEvidenceInspection:
|
||||
db = _session(session)
|
||||
_assert_tenant(principal, request.tenant_id)
|
||||
now = datetime.now(UTC)
|
||||
if request.evidence.owner_module != PROVIDER_ID:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The evidence owner does not match the Files provider.",
|
||||
)
|
||||
if request.evidence.kind != "document" or not request.evidence.version:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="Files evidence requires an exact document version.",
|
||||
)
|
||||
grant = (
|
||||
db.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.tenant_id == request.tenant_id,
|
||||
FileFormEvidenceGrant.form_instance_id == request.instance_id,
|
||||
FileFormEvidenceGrant.form_definition_id
|
||||
== request.definition_ref.object_id,
|
||||
FileFormEvidenceGrant.form_definition_revision
|
||||
== str(request.definition_ref.version),
|
||||
FileFormEvidenceGrant.file_asset_id == request.evidence.evidence_id,
|
||||
FileFormEvidenceGrant.file_version_id == request.evidence.version,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if grant is None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The document was not captured for this exact Form submission.",
|
||||
)
|
||||
if grant.status == "revoked":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="revoked",
|
||||
observed_at=now,
|
||||
reason="The Form evidence grant was revoked.",
|
||||
)
|
||||
if grant.status != "uploaded":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="pending",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The Form evidence upload has not completed.",
|
||||
)
|
||||
asset = db.get(FileAsset, grant.file_asset_id)
|
||||
version = db.get(FileVersion, grant.file_version_id)
|
||||
blob = db.get(FileBlob, version.blob_id) if version is not None else None
|
||||
if asset is None or version is None or blob is None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="unavailable",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The managed document cannot currently be reconstructed.",
|
||||
)
|
||||
if (
|
||||
asset.tenant_id != request.tenant_id
|
||||
or version.tenant_id != request.tenant_id
|
||||
or blob.tenant_id != request.tenant_id
|
||||
or version.file_asset_id != asset.id
|
||||
):
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document crosses an evidence ownership boundary.",
|
||||
)
|
||||
if asset.deleted_at is not None:
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="revoked",
|
||||
observed_at=now,
|
||||
reason="The managed document is no longer active.",
|
||||
)
|
||||
if blob.quarantined_at is not None or blob.integrity_status == "quarantined":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document failed its integrity gate.",
|
||||
)
|
||||
if blob.integrity_status != "verified":
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="pending",
|
||||
observed_at=now,
|
||||
retryable=True,
|
||||
reason="The managed document is awaiting integrity verification.",
|
||||
)
|
||||
if (
|
||||
not request.evidence.checksum
|
||||
or not secrets.compare_digest(
|
||||
request.evidence.checksum,
|
||||
version.checksum_sha256,
|
||||
)
|
||||
or not secrets.compare_digest(
|
||||
version.checksum_sha256,
|
||||
blob.checksum_sha256,
|
||||
)
|
||||
):
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="rejected",
|
||||
observed_at=now,
|
||||
reason="The managed document checksum does not match the evidence.",
|
||||
)
|
||||
return _inspection(
|
||||
request.evidence,
|
||||
state="accepted",
|
||||
observed_at=now,
|
||||
metadata={
|
||||
"content_type": version.content_type,
|
||||
"size_bytes": version.size_bytes,
|
||||
"integrity_status": blob.integrity_status,
|
||||
"grant_id": grant.id,
|
||||
},
|
||||
)
|
||||
|
||||
def _assert_active_custodian(self, *, tenant_id: str, user_id: str) -> None:
|
||||
registry = self._registry
|
||||
if registry is None or not hasattr(registry, "has_capability"):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory is unavailable for Form evidence custody."
|
||||
)
|
||||
if not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory is unavailable for Form evidence custody."
|
||||
)
|
||||
directory = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||
if not isinstance(directory, AccessDirectory):
|
||||
raise FormEvidenceContractError(
|
||||
"The Access directory capability is invalid."
|
||||
)
|
||||
user = directory.get_user(user_id)
|
||||
if user is None or user.tenant_id != tenant_id or user.status != "active":
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence requires an active same-tenant custodian."
|
||||
)
|
||||
|
||||
|
||||
def create_files_form_evidence_provider(
|
||||
context: ModuleContext,
|
||||
) -> FilesFormEvidenceProvider:
|
||||
configure_runtime(registry=context.registry, settings=context.settings)
|
||||
return FilesFormEvidenceProvider(context.registry, context.settings)
|
||||
|
||||
|
||||
def _grant_response(
|
||||
grant: FileFormEvidenceGrant,
|
||||
*,
|
||||
upload_token: str | None,
|
||||
replayed: bool,
|
||||
) -> FormEvidenceGrant:
|
||||
return FormEvidenceGrant(
|
||||
provider_id=PROVIDER_ID,
|
||||
grant_id=grant.id,
|
||||
upload_token=upload_token,
|
||||
upload_url="/api/v1/files/form-evidence/upload",
|
||||
expires_at=_aware(grant.expires_at),
|
||||
max_size_bytes=grant.max_size_bytes,
|
||||
allowed_content_types=tuple(grant.allowed_content_types),
|
||||
replayed=replayed,
|
||||
)
|
||||
|
||||
|
||||
def _inspection(
|
||||
reference: EvidenceReference,
|
||||
*,
|
||||
state: FormEvidenceState,
|
||||
observed_at: datetime,
|
||||
retryable: bool = False,
|
||||
reason: str | None = None,
|
||||
metadata: Mapping[str, object] | None = None,
|
||||
) -> FormEvidenceInspection:
|
||||
return FormEvidenceInspection(
|
||||
provider_id=PROVIDER_ID,
|
||||
reference=reference,
|
||||
state=state,
|
||||
observed_at=observed_at,
|
||||
retryable=retryable,
|
||||
reason=reason,
|
||||
metadata=dict(metadata or {}),
|
||||
)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files Form evidence requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
def _assert_tenant(principal: object, tenant_id: str) -> None:
|
||||
principal_tenant = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not principal_tenant or principal_tenant != tenant_id:
|
||||
raise PermissionError("Form evidence cannot cross tenants.")
|
||||
|
||||
|
||||
def _custodian_user_id(value: str | None) -> str:
|
||||
clean = str(value or "").strip()
|
||||
if not clean.startswith("user:") or len(clean) <= len("user:"):
|
||||
raise FormEvidenceContractError(
|
||||
"Files Form evidence requires a user custodian."
|
||||
)
|
||||
return clean.removeprefix("user:")
|
||||
|
||||
|
||||
def _content_types(values: Sequence[str]) -> tuple[str, ...]:
|
||||
cleaned = tuple(
|
||||
dict.fromkeys(
|
||||
value.split(";", 1)[0].strip().casefold()
|
||||
for value in values
|
||||
if value.strip()
|
||||
)
|
||||
)
|
||||
if len(cleaned) > 50 or any(len(value) > 255 for value in cleaned):
|
||||
raise FormEvidenceContractError(
|
||||
"Form evidence content-type restrictions are too large."
|
||||
)
|
||||
return cleaned
|
||||
|
||||
|
||||
def _bounded_metadata(value: Mapping[str, object]) -> dict[str, object]:
|
||||
remaining = value.get("remaining_attachments")
|
||||
result: dict[str, object] = {}
|
||||
if isinstance(remaining, int) and remaining >= 0:
|
||||
result["remaining_attachments"] = remaining
|
||||
raw_ids = value.get("existing_attachment_ids")
|
||||
if isinstance(raw_ids, (list, tuple)):
|
||||
clean_ids = tuple(
|
||||
dict.fromkeys(
|
||||
item.strip()
|
||||
for item in raw_ids
|
||||
if isinstance(item, str) and item.strip() and len(item.strip()) <= 255
|
||||
)
|
||||
)
|
||||
if len(clean_ids) <= 1000:
|
||||
result["existing_attachment_ids"] = clean_ids
|
||||
return result
|
||||
|
||||
|
||||
def _request_sha256(
|
||||
request: FormEvidenceGrantRequest,
|
||||
*,
|
||||
custodian_user_id: str,
|
||||
max_size_bytes: int,
|
||||
allowed_content_types: Sequence[str],
|
||||
) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
{
|
||||
"tenant_id": request.tenant_id,
|
||||
"instance_id": request.instance_id,
|
||||
"definition_ref": request.definition_ref.to_dict(),
|
||||
"evidence_kind": request.evidence_kind,
|
||||
"purpose": request.purpose,
|
||||
"expires_at": request.expires_at.isoformat(),
|
||||
"custodian_user_id": custodian_user_id,
|
||||
"max_size_bytes": max_size_bytes,
|
||||
"allowed_content_types": list(allowed_content_types),
|
||||
"metadata": _bounded_metadata(request.metadata),
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _token_sha256(token: str) -> str:
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CAPABILITY_FORM_EVIDENCE_FILES",
|
||||
"FilesFormEvidenceProvider",
|
||||
"PROVIDER_ID",
|
||||
"create_files_form_evidence_provider",
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
+119
@@ -0,0 +1,119 @@
|
||||
"""file connector spaces
|
||||
|
||||
Revision ID: 4f5a6b7c8d9e
|
||||
Revises: 2e3f4a5b6c7d
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "4f5a6b7c8d9e"
|
||||
down_revision = "2e3f4a5b6c7d"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(tables: set[str]) -> str:
|
||||
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
tables = set(inspector.get_table_names())
|
||||
scope_fk_target = _scope_fk_target(tables)
|
||||
if "file_connector_spaces" not in tables:
|
||||
op.create_table(
|
||||
"file_connector_spaces",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("owner_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("owner_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("owner_group_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("label", sa.String(length=255), nullable=False),
|
||||
sa.Column("connector_profile_id", sa.String(length=255), nullable=False),
|
||||
sa.Column("provider", sa.String(length=50), nullable=False),
|
||||
sa.Column("library_id", sa.String(length=255), nullable=True),
|
||||
sa.Column("remote_path", sa.String(length=1000), nullable=False),
|
||||
sa.Column("sync_mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("read_only", sa.Boolean(), nullable=False),
|
||||
sa.Column("is_active", sa.Boolean(), nullable=False),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_spaces_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["owner_group_id"], ["access_groups.id"], name=op.f("fk_file_connector_spaces_owner_group_id_groups"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["owner_user_id"], ["access_users.id"], name=op.f("fk_file_connector_spaces_owner_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_spaces_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_spaces")),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_spaces")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"owner_type",
|
||||
"owner_user_id",
|
||||
"owner_group_id",
|
||||
"connector_profile_id",
|
||||
"provider",
|
||||
"is_active",
|
||||
"created_by_user_id",
|
||||
"deleted_at",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_spaces_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_spaces", [column], unique=False)
|
||||
if "ix_file_connector_spaces_owner" not in indexes:
|
||||
op.create_index(
|
||||
"ix_file_connector_spaces_owner",
|
||||
"file_connector_spaces",
|
||||
["tenant_id", "owner_type", "owner_user_id", "owner_group_id"],
|
||||
unique=False,
|
||||
)
|
||||
if "uq_file_connector_spaces_active_user_label" not in indexes:
|
||||
op.create_index(
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"file_connector_spaces",
|
||||
["tenant_id", "owner_user_id", "label"],
|
||||
unique=True,
|
||||
sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"),
|
||||
)
|
||||
if "uq_file_connector_spaces_active_group_label" not in indexes:
|
||||
op.create_index(
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"file_connector_spaces",
|
||||
["tenant_id", "owner_group_id", "label"],
|
||||
unique=True,
|
||||
sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_spaces" not in inspector.get_table_names():
|
||||
return
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_spaces")}
|
||||
for name in (
|
||||
"uq_file_connector_spaces_active_group_label",
|
||||
"uq_file_connector_spaces_active_user_label",
|
||||
"ix_file_connector_spaces_owner",
|
||||
op.f("ix_file_connector_spaces_deleted_at"),
|
||||
op.f("ix_file_connector_spaces_created_by_user_id"),
|
||||
op.f("ix_file_connector_spaces_is_active"),
|
||||
op.f("ix_file_connector_spaces_provider"),
|
||||
op.f("ix_file_connector_spaces_connector_profile_id"),
|
||||
op.f("ix_file_connector_spaces_owner_group_id"),
|
||||
op.f("ix_file_connector_spaces_owner_user_id"),
|
||||
op.f("ix_file_connector_spaces_owner_type"),
|
||||
op.f("ix_file_connector_spaces_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_spaces")
|
||||
op.drop_table("file_connector_spaces")
|
||||
+94
@@ -0,0 +1,94 @@
|
||||
"""file connector profiles
|
||||
|
||||
Revision ID: 5a6b7c8d9e0f
|
||||
Revises: 4f5a6b7c8d9e
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "5a6b7c8d9e0f"
|
||||
down_revision = "4f5a6b7c8d9e"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(inspector) -> str:
|
||||
tables = set(inspector.get_table_names())
|
||||
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
scope_fk_target = _scope_fk_target(inspector)
|
||||
if "file_connector_profiles" not in inspector.get_table_names():
|
||||
op.create_table(
|
||||
"file_connector_profiles",
|
||||
sa.Column("id", sa.String(length=255), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("label", sa.String(length=255), nullable=False),
|
||||
sa.Column("provider", sa.String(length=50), nullable=False),
|
||||
sa.Column("endpoint_url", sa.String(length=1000), nullable=True),
|
||||
sa.Column("base_path", sa.String(length=1000), nullable=True),
|
||||
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||
sa.Column("credential_mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("username", sa.String(length=320), nullable=True),
|
||||
sa.Column("password_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("token_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("password_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("token_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("secret_ref", sa.String(length=1000), nullable=True),
|
||||
sa.Column("capabilities", sa.JSON(), nullable=True),
|
||||
sa.Column("policy", sa.JSON(), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_profiles_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_profiles_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_profiles_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_profiles")),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
"provider",
|
||||
"enabled",
|
||||
"created_by_user_id",
|
||||
"updated_by_user_id",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_profiles_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_profiles", [column], unique=False)
|
||||
if "ix_file_connector_profiles_scope" not in indexes:
|
||||
op.create_index("ix_file_connector_profiles_scope", "file_connector_profiles", ["scope_type", "scope_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_profiles" not in inspector.get_table_names():
|
||||
return
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
for name in (
|
||||
"ix_file_connector_profiles_scope",
|
||||
op.f("ix_file_connector_profiles_updated_by_user_id"),
|
||||
op.f("ix_file_connector_profiles_created_by_user_id"),
|
||||
op.f("ix_file_connector_profiles_enabled"),
|
||||
op.f("ix_file_connector_profiles_provider"),
|
||||
op.f("ix_file_connector_profiles_scope_id"),
|
||||
op.f("ix_file_connector_profiles_scope_type"),
|
||||
op.f("ix_file_connector_profiles_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_profiles")
|
||||
op.drop_table("file_connector_profiles")
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
"""file connector credentials
|
||||
|
||||
Revision ID: 6b7c8d9e0f1a
|
||||
Revises: 5a6b7c8d9e0f
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "6b7c8d9e0f1a"
|
||||
down_revision = "5a6b7c8d9e0f"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(table_names: set[str]) -> str:
|
||||
return "core_scopes.id" if "core_scopes" in table_names else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
table_names = set(inspector.get_table_names())
|
||||
scope_fk_target = _scope_fk_target(table_names)
|
||||
|
||||
if "file_connector_profiles" in table_names:
|
||||
columns = {item["name"] for item in inspector.get_columns("file_connector_profiles")}
|
||||
if "credential_profile_id" not in columns:
|
||||
op.add_column("file_connector_profiles", sa.Column("credential_profile_id", sa.String(length=255), nullable=True))
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
index_name = op.f("ix_file_connector_profiles_credential_profile_id")
|
||||
if index_name not in indexes:
|
||||
op.create_index(index_name, "file_connector_profiles", ["credential_profile_id"], unique=False)
|
||||
|
||||
if "file_connector_credentials" not in table_names:
|
||||
op.create_table(
|
||||
"file_connector_credentials",
|
||||
sa.Column("id", sa.String(length=255), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("label", sa.String(length=255), nullable=False),
|
||||
sa.Column("provider", sa.String(length=50), nullable=True),
|
||||
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||
sa.Column("credential_mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("username", sa.String(length=320), nullable=True),
|
||||
sa.Column("password_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("token_encrypted", sa.Text(), nullable=True),
|
||||
sa.Column("password_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("token_env", sa.String(length=255), nullable=True),
|
||||
sa.Column("secret_ref", sa.String(length=1000), nullable=True),
|
||||
sa.Column("policy", sa.JSON(), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=True),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_credentials_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_credentials_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_credentials_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_credentials")),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_credentials")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
"provider",
|
||||
"enabled",
|
||||
"created_by_user_id",
|
||||
"updated_by_user_id",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_credentials_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_credentials", [column], unique=False)
|
||||
if "ix_file_connector_credentials_scope" not in indexes:
|
||||
op.create_index("ix_file_connector_credentials_scope", "file_connector_credentials", ["scope_type", "scope_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_credentials" in inspector.get_table_names():
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_credentials")}
|
||||
for name in (
|
||||
"ix_file_connector_credentials_scope",
|
||||
op.f("ix_file_connector_credentials_updated_by_user_id"),
|
||||
op.f("ix_file_connector_credentials_created_by_user_id"),
|
||||
op.f("ix_file_connector_credentials_enabled"),
|
||||
op.f("ix_file_connector_credentials_provider"),
|
||||
op.f("ix_file_connector_credentials_scope_id"),
|
||||
op.f("ix_file_connector_credentials_scope_type"),
|
||||
op.f("ix_file_connector_credentials_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_credentials")
|
||||
op.drop_table("file_connector_credentials")
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_profiles" not in inspector.get_table_names():
|
||||
return
|
||||
profile_indexes = {item["name"] for item in inspector.get_indexes("file_connector_profiles")}
|
||||
profile_index_name = op.f("ix_file_connector_profiles_credential_profile_id")
|
||||
if profile_index_name in profile_indexes:
|
||||
op.drop_index(profile_index_name, table_name="file_connector_profiles")
|
||||
profile_columns = {item["name"] for item in inspector.get_columns("file_connector_profiles")}
|
||||
if "credential_profile_id" in profile_columns:
|
||||
op.drop_column("file_connector_profiles", "credential_profile_id")
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
"""file connector policies
|
||||
|
||||
Revision ID: a7b8c9d0e1f3
|
||||
Revises: 6b7c8d9e0f1a
|
||||
Create Date: 2026-07-08 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "a7b8c9d0e1f3"
|
||||
down_revision = "6b7c8d9e0f1a"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _scope_fk_target(inspector) -> str:
|
||||
tables = set(inspector.get_table_names())
|
||||
return "core_scopes.id" if "core_scopes" in tables else "tenancy_tenants.id"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
scope_fk_target = _scope_fk_target(inspector)
|
||||
if "file_connector_policies" not in inspector.get_table_names():
|
||||
op.create_table(
|
||||
"file_connector_policies",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("scope_type", sa.String(length=20), nullable=False),
|
||||
sa.Column("scope_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("policy", sa.JSON(), nullable=False),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("updated_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["created_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_policies_created_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["tenant_id"], [scope_fk_target], name=op.f("fk_file_connector_policies_tenant_id_scopes"), ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["updated_by_user_id"], ["access_users.id"], name=op.f("fk_file_connector_policies_updated_by_user_id_users"), ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_connector_policies")),
|
||||
sa.UniqueConstraint("tenant_id", "scope_type", "scope_id", name="uq_file_connector_policies_scope"),
|
||||
)
|
||||
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_policies")}
|
||||
for column in (
|
||||
"tenant_id",
|
||||
"scope_type",
|
||||
"scope_id",
|
||||
"created_by_user_id",
|
||||
"updated_by_user_id",
|
||||
):
|
||||
name = op.f(f"ix_file_connector_policies_{column}")
|
||||
if name not in indexes:
|
||||
op.create_index(name, "file_connector_policies", [column], unique=False)
|
||||
if "ix_file_connector_policies_scope" not in indexes:
|
||||
op.create_index("ix_file_connector_policies_scope", "file_connector_policies", ["scope_type", "scope_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if "file_connector_policies" not in inspector.get_table_names():
|
||||
return
|
||||
indexes = {item["name"] for item in inspector.get_indexes("file_connector_policies")}
|
||||
for name in (
|
||||
"ix_file_connector_policies_scope",
|
||||
op.f("ix_file_connector_policies_updated_by_user_id"),
|
||||
op.f("ix_file_connector_policies_created_by_user_id"),
|
||||
op.f("ix_file_connector_policies_scope_id"),
|
||||
op.f("ix_file_connector_policies_scope_type"),
|
||||
op.f("ix_file_connector_policies_tenant_id"),
|
||||
):
|
||||
if name in indexes:
|
||||
op.drop_index(name, table_name="file_connector_policies")
|
||||
op.drop_table("file_connector_policies")
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
"""add purpose-bound Form evidence upload grants
|
||||
|
||||
Revision ID: a2b3c4d5e6f8
|
||||
Revises: f1a2b3c4d5e7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "a2b3c4d5e6f8"
|
||||
down_revision = "f1a2b3c4d5e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"file_form_evidence_grants",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("form_instance_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("form_definition_id", sa.String(length=255), nullable=False),
|
||||
sa.Column("form_definition_revision", sa.String(length=255), nullable=False),
|
||||
sa.Column("token_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||
sa.Column("request_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("custodian_user_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("evidence_kind", sa.String(length=30), nullable=False),
|
||||
sa.Column("purpose", sa.String(length=500), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("max_size_bytes", sa.Integer(), nullable=False),
|
||||
sa.Column("allowed_content_types", sa.JSON(), nullable=False),
|
||||
sa.Column("file_asset_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("file_version_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("uploaded_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("metadata", sa.JSON(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["custodian_user_id"],
|
||||
["access_users.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["file_asset_id"],
|
||||
["file_assets.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["file_version_id"],
|
||||
["file_versions.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_file_form_evidence_grants_idempotency",
|
||||
),
|
||||
sa.UniqueConstraint("token_sha256"),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_tenant_id",
|
||||
"file_form_evidence_grants",
|
||||
["tenant_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_form_instance_id",
|
||||
"file_form_evidence_grants",
|
||||
["form_instance_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_custodian_user_id",
|
||||
"file_form_evidence_grants",
|
||||
["custodian_user_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_status",
|
||||
"file_form_evidence_grants",
|
||||
["status"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_expires_at",
|
||||
"file_form_evidence_grants",
|
||||
["expires_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_file_asset_id",
|
||||
"file_form_evidence_grants",
|
||||
["file_asset_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_file_version_id",
|
||||
"file_form_evidence_grants",
|
||||
["file_version_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
"file_form_evidence_grants",
|
||||
[
|
||||
"tenant_id",
|
||||
"form_instance_id",
|
||||
"form_definition_id",
|
||||
"form_definition_revision",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_form",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_file_version_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_file_asset_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_expires_at",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_status",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_custodian_user_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_form_instance_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_file_form_evidence_grants_tenant_id",
|
||||
table_name="file_form_evidence_grants",
|
||||
)
|
||||
op.drop_table("file_form_evidence_grants")
|
||||
@@ -0,0 +1,52 @@
|
||||
"""govern managed file retention and legal holds
|
||||
|
||||
Revision ID: a2b3c4d5e6f9
|
||||
Revises: f1a2b3c4d5e7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "a2b3c4d5e6f9"
|
||||
down_revision = "f1a2b3c4d5e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_assets") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("retained_until", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"legal_hold", sa.Boolean(), nullable=False, server_default=sa.false()
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"lifecycle_revision", sa.Integer(), nullable=False, server_default="1"
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("lifecycle_reason", sa.String(length=500), nullable=True)
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_assets_retained_until"), ["retained_until"], unique=False
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_assets_legal_hold"), ["legal_hold"], unique=False
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_assets") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_assets_legal_hold"))
|
||||
batch_op.drop_index(op.f("ix_file_assets_retained_until"))
|
||||
batch_op.drop_column("lifecycle_reason")
|
||||
batch_op.drop_column("lifecycle_revision")
|
||||
batch_op.drop_column("legal_hold")
|
||||
batch_op.drop_column("retained_until")
|
||||
@@ -0,0 +1,158 @@
|
||||
"""v0.1.7 files baseline
|
||||
|
||||
Revision ID: a7b8c9d0e1f3
|
||||
Revises: None
|
||||
Create Date: 2026-07-11 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = 'a7b8c9d0e1f3'
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = '4f2a9c8e7b6d'
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table('file_connector_credentials',
|
||||
sa.Column('id', sa.String(length=255), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('label', sa.String(length=255), nullable=False),
|
||||
sa.Column('provider', sa.String(length=50), nullable=True),
|
||||
sa.Column('enabled', sa.Boolean(), nullable=False),
|
||||
sa.Column('credential_mode', sa.String(length=30), nullable=False),
|
||||
sa.Column('username', sa.String(length=320), nullable=True),
|
||||
sa.Column('password_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('token_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('password_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('token_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
|
||||
sa.Column('policy', sa.JSON(), nullable=True),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_credentials_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_credentials'))
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_credentials_created_by_user_id'), 'file_connector_credentials', ['created_by_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_enabled'), 'file_connector_credentials', ['enabled'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_provider'), 'file_connector_credentials', ['provider'], unique=False)
|
||||
op.create_index('ix_file_connector_credentials_scope', 'file_connector_credentials', ['scope_type', 'scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_scope_id'), 'file_connector_credentials', ['scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_scope_type'), 'file_connector_credentials', ['scope_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_tenant_id'), 'file_connector_credentials', ['tenant_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_credentials_updated_by_user_id'), 'file_connector_credentials', ['updated_by_user_id'], unique=False)
|
||||
op.create_table('file_connector_policies',
|
||||
sa.Column('id', sa.String(length=36), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('policy', sa.JSON(), nullable=False),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_policies_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_policies')),
|
||||
sa.UniqueConstraint('tenant_id', 'scope_type', 'scope_id', name='uq_file_connector_policies_scope')
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_policies_created_by_user_id'), 'file_connector_policies', ['created_by_user_id'], unique=False)
|
||||
op.create_index('ix_file_connector_policies_scope', 'file_connector_policies', ['scope_type', 'scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_scope_id'), 'file_connector_policies', ['scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_scope_type'), 'file_connector_policies', ['scope_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_tenant_id'), 'file_connector_policies', ['tenant_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_policies_updated_by_user_id'), 'file_connector_policies', ['updated_by_user_id'], unique=False)
|
||||
op.create_table('file_connector_profiles',
|
||||
sa.Column('id', sa.String(length=255), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('scope_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('scope_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('label', sa.String(length=255), nullable=False),
|
||||
sa.Column('provider', sa.String(length=50), nullable=False),
|
||||
sa.Column('endpoint_url', sa.String(length=1000), nullable=True),
|
||||
sa.Column('base_path', sa.String(length=1000), nullable=True),
|
||||
sa.Column('enabled', sa.Boolean(), nullable=False),
|
||||
sa.Column('credential_profile_id', sa.String(length=255), nullable=True),
|
||||
sa.Column('credential_mode', sa.String(length=30), nullable=False),
|
||||
sa.Column('username', sa.String(length=320), nullable=True),
|
||||
sa.Column('password_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('token_encrypted', sa.Text(), nullable=True),
|
||||
sa.Column('password_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('token_env', sa.String(length=255), nullable=True),
|
||||
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
|
||||
sa.Column('capabilities', sa.JSON(), nullable=True),
|
||||
sa.Column('policy', sa.JSON(), nullable=True),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_profiles_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_updated_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_profiles'))
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_profiles_created_by_user_id'), 'file_connector_profiles', ['created_by_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_credential_profile_id'), 'file_connector_profiles', ['credential_profile_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_enabled'), 'file_connector_profiles', ['enabled'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_provider'), 'file_connector_profiles', ['provider'], unique=False)
|
||||
op.create_index('ix_file_connector_profiles_scope', 'file_connector_profiles', ['scope_type', 'scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_scope_id'), 'file_connector_profiles', ['scope_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_scope_type'), 'file_connector_profiles', ['scope_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_tenant_id'), 'file_connector_profiles', ['tenant_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_profiles_updated_by_user_id'), 'file_connector_profiles', ['updated_by_user_id'], unique=False)
|
||||
op.create_table('file_connector_spaces',
|
||||
sa.Column('id', sa.String(length=36), nullable=False),
|
||||
sa.Column('tenant_id', sa.String(length=36), nullable=False),
|
||||
sa.Column('owner_type', sa.String(length=20), nullable=False),
|
||||
sa.Column('owner_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('owner_group_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('label', sa.String(length=255), nullable=False),
|
||||
sa.Column('connector_profile_id', sa.String(length=255), nullable=False),
|
||||
sa.Column('provider', sa.String(length=50), nullable=False),
|
||||
sa.Column('library_id', sa.String(length=255), nullable=True),
|
||||
sa.Column('remote_path', sa.String(length=1000), nullable=False),
|
||||
sa.Column('sync_mode', sa.String(length=30), nullable=False),
|
||||
sa.Column('read_only', sa.Boolean(), nullable=False),
|
||||
sa.Column('is_active', sa.Boolean(), nullable=False),
|
||||
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
|
||||
sa.Column('deleted_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_created_by_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['owner_group_id'], ['access_groups.id'], name=op.f('fk_file_connector_spaces_owner_group_id_access_groups'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_owner_user_id_access_users'), ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_spaces_tenant_id_scopes'), ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_spaces'))
|
||||
)
|
||||
op.create_index(op.f('ix_file_connector_spaces_connector_profile_id'), 'file_connector_spaces', ['connector_profile_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_created_by_user_id'), 'file_connector_spaces', ['created_by_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_deleted_at'), 'file_connector_spaces', ['deleted_at'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_is_active'), 'file_connector_spaces', ['is_active'], unique=False)
|
||||
op.create_index('ix_file_connector_spaces_owner', 'file_connector_spaces', ['tenant_id', 'owner_type', 'owner_user_id', 'owner_group_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_owner_group_id'), 'file_connector_spaces', ['owner_group_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_owner_type'), 'file_connector_spaces', ['owner_type'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_owner_user_id'), 'file_connector_spaces', ['owner_user_id'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_provider'), 'file_connector_spaces', ['provider'], unique=False)
|
||||
op.create_index(op.f('ix_file_connector_spaces_tenant_id'), 'file_connector_spaces', ['tenant_id'], unique=False)
|
||||
op.create_index('uq_file_connector_spaces_active_group_label', 'file_connector_spaces', ['tenant_id', 'owner_group_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"))
|
||||
op.create_index('uq_file_connector_spaces_active_user_label', 'file_connector_spaces', ['tenant_id', 'owner_user_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table('file_connector_spaces')
|
||||
op.drop_table('file_connector_profiles')
|
||||
op.drop_table('file_connector_policies')
|
||||
op.drop_table('file_connector_credentials')
|
||||
@@ -0,0 +1,53 @@
|
||||
"""file share lifecycle
|
||||
|
||||
Revision ID: b8c9d0e1f2a4
|
||||
Revises: a7b8c9d0e1f3
|
||||
Create Date: 2026-07-30 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "b8c9d0e1f2a4"
|
||||
down_revision = "a7b8c9d0e1f3"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_shares") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("revoked_by_user_id", sa.String(length=36), nullable=True)
|
||||
)
|
||||
batch_op.create_foreign_key(
|
||||
op.f("fk_file_shares_revoked_by_user_id_access_users"),
|
||||
"access_users",
|
||||
["revoked_by_user_id"],
|
||||
["id"],
|
||||
ondelete="SET NULL",
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_shares_expires_at"), ["expires_at"], unique=False
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_shares_revoked_by_user_id"),
|
||||
["revoked_by_user_id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_shares") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_shares_revoked_by_user_id"))
|
||||
batch_op.drop_index(op.f("ix_file_shares_expires_at"))
|
||||
batch_op.drop_constraint(
|
||||
op.f("fk_file_shares_revoked_by_user_id_access_users"),
|
||||
type_="foreignkey",
|
||||
)
|
||||
batch_op.drop_column("revoked_by_user_id")
|
||||
batch_op.drop_column("expires_at")
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
"""file integrity reconciliation
|
||||
|
||||
Revision ID: c9d0e1f2a3b5
|
||||
Revises: b8c9d0e1f2a4
|
||||
Create Date: 2026-07-30 00:00:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "c9d0e1f2a3b5"
|
||||
down_revision = "b8c9d0e1f2a4"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"integrity_status",
|
||||
sa.String(length=30),
|
||||
nullable=False,
|
||||
server_default="unchecked",
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("integrity_checked_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("integrity_failure", sa.String(length=100), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_integrity_status"),
|
||||
["integrity_status"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_integrity_checked_at"),
|
||||
["integrity_checked_at"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_quarantined_at"),
|
||||
["quarantined_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"file_integrity_scans",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("storage_backend", sa.String(length=50), nullable=False),
|
||||
sa.Column("storage_prefix", sa.String(length=1000), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("phase", sa.String(length=30), nullable=False),
|
||||
sa.Column("verify_checksums", sa.Boolean(), nullable=False),
|
||||
sa.Column("batch_size", sa.Integer(), nullable=False),
|
||||
sa.Column("blob_cursor", sa.String(length=36), nullable=True),
|
||||
sa.Column("object_cursor", sa.String(length=1000), nullable=True),
|
||||
sa.Column("scanned_blob_count", sa.Integer(), nullable=False),
|
||||
sa.Column("verified_blob_count", sa.Integer(), nullable=False),
|
||||
sa.Column("quarantined_blob_count", sa.Integer(), nullable=False),
|
||||
sa.Column("scanned_object_count", sa.Integer(), nullable=False),
|
||||
sa.Column("orphan_object_count", sa.Integer(), nullable=False),
|
||||
sa.Column("created_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_error", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["created_by_user_id"],
|
||||
["access_users.id"],
|
||||
name=op.f(
|
||||
"fk_file_integrity_scans_created_by_user_id_access_users"
|
||||
),
|
||||
ondelete="SET NULL",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_scans")),
|
||||
)
|
||||
for column in ("tenant_id", "status", "created_by_user_id"):
|
||||
op.create_index(
|
||||
op.f(f"ix_file_integrity_scans_{column}"),
|
||||
"file_integrity_scans",
|
||||
[column],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"file_integrity_findings",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("scan_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("kind", sa.String(length=40), nullable=False),
|
||||
sa.Column("state", sa.String(length=30), nullable=False),
|
||||
sa.Column("blob_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("storage_key", sa.String(length=1000), nullable=False),
|
||||
sa.Column("expected_size_bytes", sa.Integer(), nullable=True),
|
||||
sa.Column("observed_size_bytes", sa.Integer(), nullable=True),
|
||||
sa.Column("expected_checksum_sha256", sa.String(length=64), nullable=True),
|
||||
sa.Column("observed_checksum_sha256", sa.String(length=64), nullable=True),
|
||||
sa.Column("resolved_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("resolved_by_user_id", sa.String(length=36), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["blob_id"],
|
||||
["file_blobs.id"],
|
||||
name=op.f("fk_file_integrity_findings_blob_id_file_blobs"),
|
||||
ondelete="SET NULL",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["resolved_by_user_id"],
|
||||
["access_users.id"],
|
||||
name=op.f(
|
||||
"fk_file_integrity_findings_resolved_by_user_id_access_users"
|
||||
),
|
||||
ondelete="SET NULL",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["scan_id"],
|
||||
["file_integrity_scans.id"],
|
||||
name=op.f(
|
||||
"fk_file_integrity_findings_scan_id_file_integrity_scans"
|
||||
),
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_integrity_findings")),
|
||||
)
|
||||
for column in (
|
||||
"scan_id",
|
||||
"tenant_id",
|
||||
"kind",
|
||||
"state",
|
||||
"blob_id",
|
||||
"resolved_by_user_id",
|
||||
):
|
||||
op.create_index(
|
||||
op.f(f"ix_file_integrity_findings_{column}"),
|
||||
"file_integrity_findings",
|
||||
[column],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_file_integrity_findings_scan_state",
|
||||
"file_integrity_findings",
|
||||
["scan_id", "state"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("file_integrity_findings")
|
||||
op.drop_table("file_integrity_scans")
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_blobs_quarantined_at"))
|
||||
batch_op.drop_index(op.f("ix_file_blobs_integrity_checked_at"))
|
||||
batch_op.drop_index(op.f("ix_file_blobs_integrity_status"))
|
||||
batch_op.drop_column("quarantined_at")
|
||||
batch_op.drop_column("integrity_failure")
|
||||
batch_op.drop_column("integrity_checked_at")
|
||||
batch_op.drop_column("integrity_status")
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
"""file content-protection metadata
|
||||
|
||||
Revision ID: d0e1f2a3b4c6
|
||||
Revises: c9d0e1f2a3b5
|
||||
Create Date: 2026-08-02 00:00:00.000000
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "d0e1f2a3b4c6"
|
||||
down_revision = "c9d0e1f2a3b5"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column(
|
||||
"protection_discriminator",
|
||||
sa.String(length=320),
|
||||
nullable=False,
|
||||
server_default="plaintext",
|
||||
)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("encryption_envelope_id", sa.String(length=255), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("storage_checksum_sha256", sa.String(length=64), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("storage_size_bytes", sa.Integer(), nullable=True)
|
||||
)
|
||||
batch_op.drop_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size",
|
||||
type_="unique",
|
||||
)
|
||||
batch_op.create_unique_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size_protection",
|
||||
[
|
||||
"tenant_id",
|
||||
"checksum_sha256",
|
||||
"size_bytes",
|
||||
"protection_discriminator",
|
||||
],
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_protection_discriminator"),
|
||||
["protection_discriminator"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
op.f("ix_file_blobs_encryption_envelope_id"),
|
||||
["encryption_envelope_id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_blobs") as batch_op:
|
||||
batch_op.drop_index(op.f("ix_file_blobs_encryption_envelope_id"))
|
||||
batch_op.drop_index(op.f("ix_file_blobs_protection_discriminator"))
|
||||
batch_op.drop_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size_protection",
|
||||
type_="unique",
|
||||
)
|
||||
batch_op.create_unique_constraint(
|
||||
"uq_file_blobs_tenant_checksum_size",
|
||||
["tenant_id", "checksum_sha256", "size_bytes"],
|
||||
)
|
||||
batch_op.drop_column("storage_size_bytes")
|
||||
batch_op.drop_column("storage_checksum_sha256")
|
||||
batch_op.drop_column("encryption_envelope_id")
|
||||
batch_op.drop_column("protection_discriminator")
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
"""add stale-action revisions to Files integrity operations
|
||||
|
||||
Revision ID: f1a2b3c4d5e7
|
||||
Revises: d0e1f2a3b4c6
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "f1a2b3c4d5e7"
|
||||
down_revision = "d0e1f2a3b4c6"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("file_integrity_scans") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("revision", sa.Integer(), nullable=False, server_default="1")
|
||||
)
|
||||
with op.batch_alter_table("file_integrity_findings") as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("revision", sa.Integer(), nullable=False, server_default="1")
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("file_integrity_findings") as batch_op:
|
||||
batch_op.drop_column("revision")
|
||||
with op.batch_alter_table("file_integrity_scans") as batch_op:
|
||||
batch_op.drop_column("revision")
|
||||
@@ -0,0 +1,67 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import secrets
|
||||
from uuid import uuid4
|
||||
|
||||
from govoplan_core.core.operations import OperationalCheck
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
|
||||
|
||||
def managed_storage_roundtrip_check() -> OperationalCheck:
|
||||
"""Exercise the configured managed store without retaining probe data."""
|
||||
|
||||
backend = get_storage_backend()
|
||||
key = f".govoplan-health/probes/{uuid4().hex}.bin"
|
||||
payload = secrets.token_bytes(64)
|
||||
expected_digest = hashlib.sha256(payload).hexdigest()
|
||||
delete_error: Exception | None = None
|
||||
try:
|
||||
backend.put_bytes(key, payload, content_type="application/octet-stream")
|
||||
stored = backend.get_bytes(key)
|
||||
info = backend.stat(key)
|
||||
if info.size_bytes != len(payload):
|
||||
raise StorageBackendError("Managed storage returned an unexpected object size")
|
||||
if hashlib.sha256(stored).hexdigest() != expected_digest:
|
||||
raise StorageBackendError("Managed storage returned different bytes than were written")
|
||||
except Exception as exc: # noqa: BLE001 - operational boundary reports provider failures.
|
||||
return OperationalCheck(
|
||||
id="files.managed_storage_roundtrip",
|
||||
label="Managed file storage",
|
||||
state="error",
|
||||
detail=(
|
||||
"The configured managed file store failed a bounded write/read/stat/delete "
|
||||
f"probe ({type(exc).__name__})."
|
||||
),
|
||||
readiness_critical=True,
|
||||
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
backend.delete(key)
|
||||
except Exception as exc: # noqa: BLE001 - reported below when the data probe passed.
|
||||
delete_error = exc
|
||||
|
||||
if delete_error is not None:
|
||||
return OperationalCheck(
|
||||
id="files.managed_storage_roundtrip",
|
||||
label="Managed file storage",
|
||||
state="error",
|
||||
detail=(
|
||||
"Managed file bytes round-tripped, but probe cleanup failed "
|
||||
f"({type(delete_error).__name__})."
|
||||
),
|
||||
readiness_critical=True,
|
||||
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||
)
|
||||
return OperationalCheck(
|
||||
id="files.managed_storage_roundtrip",
|
||||
label="Managed file storage",
|
||||
state="ok",
|
||||
detail="The configured managed file store passed write, read, stat, integrity, and delete checks.",
|
||||
metrics={"backend": backend.name, "probe_bytes": len(payload)},
|
||||
)
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.provider_governance import (
|
||||
ExternalProviderRuntimeState,
|
||||
ExternalProviderStateContext,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileConnectorProfile, FileConnectorSpace
|
||||
from govoplan_files.backend.storage.connector_providers import (
|
||||
ConnectorProviderDescriptor,
|
||||
connector_provider_descriptors,
|
||||
)
|
||||
|
||||
|
||||
REMOTE_STORAGE_PROVIDER_ID = "files.remote_storage"
|
||||
|
||||
|
||||
def remote_storage_provider_states(
|
||||
context: ExternalProviderStateContext,
|
||||
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||
if not isinstance(context.session, Session):
|
||||
raise RuntimeError("Files provider state requires a database session.")
|
||||
statement = select(FileConnectorProfile)
|
||||
if context.tenant_id is not None:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
FileConnectorProfile.tenant_id.is_(None),
|
||||
FileConnectorProfile.tenant_id == context.tenant_id,
|
||||
)
|
||||
)
|
||||
profiles = tuple(
|
||||
context.session.scalars(
|
||||
statement.order_by(
|
||||
FileConnectorProfile.tenant_id,
|
||||
FileConnectorProfile.id,
|
||||
).limit(context.max_items + 1)
|
||||
)
|
||||
)
|
||||
if not profiles:
|
||||
return ()
|
||||
|
||||
profile_ids = tuple(item.id for item in profiles)
|
||||
space_statement = select(FileConnectorSpace).where(
|
||||
FileConnectorSpace.connector_profile_id.in_(profile_ids),
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
)
|
||||
if context.tenant_id is not None:
|
||||
space_statement = space_statement.where(
|
||||
FileConnectorSpace.tenant_id == context.tenant_id
|
||||
)
|
||||
spaces_by_profile: dict[str, list[FileConnectorSpace]] = defaultdict(list)
|
||||
for space in context.session.scalars(space_statement):
|
||||
spaces_by_profile[space.connector_profile_id].append(space)
|
||||
|
||||
descriptors = {
|
||||
item.provider: item for item in connector_provider_descriptors()
|
||||
}
|
||||
observed_at = datetime.now(UTC)
|
||||
return tuple(
|
||||
_profile_state(
|
||||
profile,
|
||||
spaces=spaces_by_profile.get(profile.id, []),
|
||||
descriptor=descriptors.get(profile.provider),
|
||||
observed_at=observed_at,
|
||||
)
|
||||
for profile in profiles
|
||||
)
|
||||
|
||||
|
||||
def _profile_state(
|
||||
profile: FileConnectorProfile,
|
||||
*,
|
||||
spaces: list[FileConnectorSpace],
|
||||
descriptor: ConnectorProviderDescriptor | None,
|
||||
observed_at: datetime,
|
||||
) -> ExternalProviderRuntimeState:
|
||||
active_spaces = tuple(item for item in spaces if item.is_active)
|
||||
active = bool(profile.enabled)
|
||||
implementation_ready = bool(
|
||||
descriptor is not None and descriptor.implemented and descriptor.installed
|
||||
)
|
||||
health = (
|
||||
"inactive"
|
||||
if not active
|
||||
else "error"
|
||||
if not implementation_ready
|
||||
else "unknown"
|
||||
)
|
||||
recovery = (
|
||||
"not_applicable"
|
||||
if not active
|
||||
else "unsupported"
|
||||
if not implementation_ready
|
||||
else "attention"
|
||||
)
|
||||
detail = (
|
||||
"Remote-storage connector profile is disabled."
|
||||
if not active
|
||||
else "The configured provider is not available in this runtime."
|
||||
if not implementation_ready
|
||||
else "Software support is available; no live remote health observation is retained."
|
||||
)
|
||||
return ExternalProviderRuntimeState(
|
||||
provider_id=REMOTE_STORAGE_PROVIDER_ID,
|
||||
binding_ref=f"files:connector-profile:{profile.id}",
|
||||
authority_mode="external_mirror",
|
||||
observed_at=observed_at,
|
||||
configured=True,
|
||||
active=active,
|
||||
health=health,
|
||||
freshness="unknown" if active else "not_applicable",
|
||||
conflict="not_applicable",
|
||||
recovery=recovery,
|
||||
detail=detail,
|
||||
metrics={
|
||||
"provider": profile.provider,
|
||||
"configured_spaces": len(spaces),
|
||||
"active_spaces": len(active_spaces),
|
||||
"write_requested_spaces": sum(
|
||||
1 for item in active_spaces if not item.read_only
|
||||
),
|
||||
"software_implemented": bool(descriptor and descriptor.implemented),
|
||||
"optional_dependency_available": bool(descriptor and descriptor.installed),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["REMOTE_STORAGE_PROVIDER_ID", "remote_storage_provider_states"]
|
||||
@@ -0,0 +1,113 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from urllib.parse import quote
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.records import (
|
||||
RecordContractError,
|
||||
RecordSourceLocator,
|
||||
RecordSourceReference,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileBlob, FileVersion
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import get_asset_for_user
|
||||
|
||||
|
||||
CAPABILITY_RECORD_SOURCE_FILES = "records.source.files"
|
||||
|
||||
|
||||
class FilesRecordSource:
|
||||
provider_id = "files"
|
||||
|
||||
def resource_types(self) -> Sequence[str]:
|
||||
return ("file_version",)
|
||||
|
||||
def resolve(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
locator: RecordSourceLocator,
|
||||
purpose: str,
|
||||
) -> RecordSourceReference:
|
||||
if not isinstance(session, Session):
|
||||
raise RecordContractError(
|
||||
"Files record references require a database session."
|
||||
)
|
||||
tenant_id = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not tenant_id or locator.tenant_id != tenant_id:
|
||||
raise RecordContractError("Files record references cannot cross tenants.")
|
||||
if locator.source_module != "files" or locator.resource_type != "file_version":
|
||||
raise RecordContractError("Unsupported Files record source type.")
|
||||
if not str(purpose or "").strip():
|
||||
raise RecordContractError("Files record references require a purpose.")
|
||||
if not hasattr(principal, "has") or not (
|
||||
principal.has("files:file:read") or principal.has("files:file:admin")
|
||||
):
|
||||
raise RecordContractError("Current Files read permission is required.")
|
||||
user = getattr(principal, "user", None)
|
||||
user_id = str(
|
||||
getattr(user, "id", "") or getattr(principal, "membership_id", "") or ""
|
||||
).strip()
|
||||
if not user_id:
|
||||
raise RecordContractError(
|
||||
"Files record references require a tenant user principal."
|
||||
)
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=locator.resource_id,
|
||||
is_admin=principal.has("files:file:admin"),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise RecordContractError(str(exc)) from exc
|
||||
version_query = session.query(FileVersion).filter(
|
||||
FileVersion.tenant_id == tenant_id,
|
||||
FileVersion.file_asset_id == asset.id,
|
||||
)
|
||||
revision = locator.source_revision.strip()
|
||||
version = version_query.filter(FileVersion.id == revision).one_or_none()
|
||||
if version is None:
|
||||
raise RecordContractError("The exact file version does not exist.")
|
||||
blob = session.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != tenant_id:
|
||||
raise RecordContractError(
|
||||
"The exact file version has no managed content object."
|
||||
)
|
||||
if blob.quarantined_at is not None or blob.integrity_status == "failed":
|
||||
raise RecordContractError(
|
||||
"The exact file version failed the current integrity gate."
|
||||
)
|
||||
return RecordSourceReference(
|
||||
locator=locator,
|
||||
label=version.filename_at_upload,
|
||||
authority_mode="external_authoritative",
|
||||
content_sha256=version.checksum_sha256,
|
||||
content_type=version.content_type,
|
||||
size_bytes=version.size_bytes,
|
||||
recorded_at=version.created_at,
|
||||
launch_url=(
|
||||
f"/files?fileId={quote(asset.id, safe='')}&versionId={quote(version.id, safe='')}"
|
||||
),
|
||||
metadata={
|
||||
"display_path": version.display_path_at_upload,
|
||||
"version_number": version.version_number,
|
||||
"integrity_status": blob.integrity_status,
|
||||
"protection": blob.protection_discriminator,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def create_files_record_source(_context: object) -> FilesRecordSource:
|
||||
return FilesRecordSource()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CAPABILITY_RECORD_SOURCE_FILES",
|
||||
"FilesRecordSource",
|
||||
"create_files_record_source",
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,722 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from io import BytesIO
|
||||
from typing import Literal
|
||||
from urllib.parse import quote
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, UploadFile, status
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
from fastapi import APIRouter
|
||||
|
||||
from govoplan_core.auth.dependencies import ApiPrincipal, has_scope, require_scope
|
||||
from govoplan_core.core.optional import reraise_unless_missing_package
|
||||
from govoplan_files.backend.schemas import (
|
||||
ArchiveRequest,
|
||||
BulkFileShareRequest,
|
||||
BulkFileShareResponse,
|
||||
BulkDeleteRequest,
|
||||
BulkDeleteResponse,
|
||||
ConflictResolutionRequest,
|
||||
FileAssetResponse,
|
||||
FileFolderCreateRequest,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderDeleteResponse,
|
||||
FileFolderResponse,
|
||||
FileFoldersResponse,
|
||||
FileListResponse,
|
||||
FileShareRequest,
|
||||
FileShareResponse,
|
||||
FileSpaceResponse,
|
||||
FileSpacesResponse,
|
||||
FileUploadResponse,
|
||||
PatternMatchResponse,
|
||||
PatternResolveRequest,
|
||||
PatternResolveResponse,
|
||||
RenamePreviewItem,
|
||||
RenameRequest,
|
||||
RenameResponse,
|
||||
TransferRequest,
|
||||
TransferResponse,
|
||||
_conflict_resolutions,
|
||||
from govoplan_files.backend.routes.assets import router as assets_router
|
||||
from govoplan_files.backend.routes.connector_io import router as connector_io_router
|
||||
from govoplan_files.backend.routes.connector_profiles import (
|
||||
router as connector_profiles_router,
|
||||
)
|
||||
from govoplan_core.db.models import Group, UserGroupMembership
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError, filename_from_path, normalize_logical_path
|
||||
from govoplan_files.backend.storage.access import ensure_group_access, user_group_ids
|
||||
from govoplan_files.backend.storage.archives import create_zip_file, extract_zip_upload
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
asset_is_audit_relevant,
|
||||
create_file_asset,
|
||||
current_version_and_blob,
|
||||
get_asset_for_user,
|
||||
list_assets_for_user,
|
||||
read_asset_bytes,
|
||||
share_file,
|
||||
share_files,
|
||||
soft_delete_assets,
|
||||
from govoplan_files.backend.routes.connector_settings import (
|
||||
router as connector_settings_router,
|
||||
)
|
||||
from govoplan_files.backend.storage.folders import create_folder, list_folders_for_user, soft_delete_folder
|
||||
from govoplan_files.backend.storage.search import resolve_patterns
|
||||
from govoplan_files.backend.storage.transfers import rename_selection, transfer_selection
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
from govoplan_files.backend.routes.folders import router as folders_router
|
||||
from govoplan_files.backend.routes.form_evidence import router as form_evidence_router
|
||||
from govoplan_files.backend.routes.integrity import router as integrity_router
|
||||
from govoplan_files.backend.routes.lifecycle import router as lifecycle_router
|
||||
from govoplan_files.backend.routes.listing import router as listing_router
|
||||
from govoplan_files.backend.routes.shares import router as shares_router
|
||||
from govoplan_files.backend.routes.spaces import router as spaces_router
|
||||
from govoplan_files.backend.routes.transfers import router as transfers_router
|
||||
from govoplan_files.backend.routes.uploads import router as uploads_router
|
||||
|
||||
|
||||
def _campaign_models():
|
||||
try:
|
||||
from govoplan_campaign.backend.db.models import Campaign, CampaignShare
|
||||
except ModuleNotFoundError as exc:
|
||||
reraise_unless_missing_package(exc, "govoplan_campaign")
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Campaign module is not installed") from exc
|
||||
return Campaign, CampaignShare
|
||||
|
||||
|
||||
def _is_admin(principal: ApiPrincipal) -> bool:
|
||||
return has_scope(principal, "files:file:admin")
|
||||
|
||||
|
||||
async def _read_limited_upload(upload: UploadFile, *, max_bytes: int) -> bytes:
|
||||
data = await upload.read(max_bytes + 1)
|
||||
if len(data) > max_bytes:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
|
||||
detail=f"Upload exceeds limit of {max_bytes} bytes",
|
||||
)
|
||||
return data
|
||||
|
||||
|
||||
|
||||
|
||||
def _cleanup_temp_file(path: str) -> None:
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def _attachment_disposition(filename: str) -> str:
|
||||
safe = filename.replace("\\", "_").replace("/", "_").replace("\r", "_").replace("\n", "_").strip() or "download"
|
||||
ascii_name = "".join(
|
||||
char if 32 <= ord(char) < 127 and char not in {'"', "\\", ";"} else "_"
|
||||
for char in safe
|
||||
).strip() or "download"
|
||||
encoded = quote(safe, safe="")
|
||||
return f'attachment; filename="{ascii_name}"; filename*=UTF-8' + "''" + encoded
|
||||
|
||||
|
||||
def _ensure_campaign_file_access(session: Session, principal: ApiPrincipal, campaign_id: str | None) -> None:
|
||||
if not campaign_id:
|
||||
return
|
||||
if not has_scope(principal, "campaigns:campaign:read"):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Missing scope: campaign:read")
|
||||
Campaign, CampaignShare = _campaign_models()
|
||||
campaign = session.get(Campaign, campaign_id)
|
||||
if not campaign or campaign.tenant_id != principal.tenant_id:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Campaign not found")
|
||||
if has_scope(principal, "tenant:*"):
|
||||
return
|
||||
if campaign.owner_user_id == principal.user.id:
|
||||
return
|
||||
group_ids = {
|
||||
row[0]
|
||||
for row in session.query(UserGroupMembership.group_id)
|
||||
.filter(UserGroupMembership.tenant_id == principal.tenant_id, UserGroupMembership.user_id == principal.user.id)
|
||||
.all()
|
||||
}
|
||||
if campaign.owner_group_id and campaign.owner_group_id in group_ids:
|
||||
return
|
||||
share = (
|
||||
session.query(CampaignShare)
|
||||
.filter(
|
||||
CampaignShare.tenant_id == principal.tenant_id,
|
||||
CampaignShare.campaign_id == campaign.id,
|
||||
CampaignShare.revoked_at.is_(None),
|
||||
or_(
|
||||
(CampaignShare.target_type == "user") & (CampaignShare.target_id == principal.user.id),
|
||||
(CampaignShare.target_type == "group") & (CampaignShare.target_id.in_(group_ids)),
|
||||
),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if share is None:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No access to this campaign")
|
||||
|
||||
|
||||
def _http_error(exc: Exception, *, not_found: bool = False) -> HTTPException:
|
||||
code = status.HTTP_404_NOT_FOUND if not_found else status.HTTP_400_BAD_REQUEST
|
||||
return HTTPException(status_code=code, detail=str(exc))
|
||||
|
||||
|
||||
def _owner_id(asset: FileAsset) -> str:
|
||||
return asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id # type: ignore[return-value]
|
||||
|
||||
|
||||
def _asset_response(session: Session, asset: FileAsset, *, include_shares: bool = False) -> FileAssetResponse:
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
shares: list[FileShareResponse] = []
|
||||
if include_shares:
|
||||
rows = session.query(FileShare).filter(FileShare.file_asset_id == asset.id).order_by(FileShare.created_at.desc()).all()
|
||||
shares = [
|
||||
FileShareResponse(
|
||||
id=row.id,
|
||||
target_type=row.target_type,
|
||||
target_id=row.target_id,
|
||||
permission=row.permission,
|
||||
created_at=row.created_at.isoformat(),
|
||||
revoked_at=row.revoked_at.isoformat() if row.revoked_at else None,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
return FileAssetResponse(
|
||||
id=asset.id,
|
||||
tenant_id=asset.tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_owner_id(asset),
|
||||
display_path=asset.display_path,
|
||||
filename=asset.filename,
|
||||
description=asset.description,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
version_id=version.id,
|
||||
created_at=asset.created_at.isoformat(),
|
||||
updated_at=asset.updated_at.isoformat(),
|
||||
deleted_at=asset.deleted_at.isoformat() if asset.deleted_at else None,
|
||||
audit_relevant=asset_is_audit_relevant(session, asset),
|
||||
metadata=asset.metadata_ or {},
|
||||
shares=shares,
|
||||
)
|
||||
|
||||
|
||||
def _folder_owner_id(folder: FileFolder) -> str:
|
||||
return folder.owner_user_id if folder.owner_type == "user" else folder.owner_group_id # type: ignore[return-value]
|
||||
|
||||
|
||||
def _folder_response(folder: FileFolder) -> FileFolderResponse:
|
||||
return FileFolderResponse(
|
||||
id=folder.id,
|
||||
tenant_id=folder.tenant_id,
|
||||
owner_type=folder.owner_type,
|
||||
owner_id=_folder_owner_id(folder),
|
||||
path=folder.path,
|
||||
created_at=folder.created_at.isoformat(),
|
||||
updated_at=folder.updated_at.isoformat(),
|
||||
deleted_at=folder.deleted_at.isoformat() if folder.deleted_at else None,
|
||||
)
|
||||
|
||||
|
||||
def _ensure_list_owner_access(session: Session, principal: ApiPrincipal, owner_type: str | None, owner_id: str | None) -> None:
|
||||
if not owner_type:
|
||||
return
|
||||
if owner_type == "user" and owner_id and owner_id != principal.user.id and not _is_admin(principal):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No access to this user file space")
|
||||
if owner_type == "group" and owner_id:
|
||||
try:
|
||||
ensure_group_access(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
group_id=owner_id,
|
||||
user_id=principal.user.id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/spaces", response_model=FileSpacesResponse)
|
||||
def list_file_spaces(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
router = APIRouter()
|
||||
for workflow_router in (
|
||||
spaces_router,
|
||||
folders_router,
|
||||
form_evidence_router,
|
||||
integrity_router,
|
||||
lifecycle_router,
|
||||
listing_router,
|
||||
uploads_router,
|
||||
connector_settings_router,
|
||||
connector_io_router,
|
||||
connector_profiles_router,
|
||||
assets_router,
|
||||
shares_router,
|
||||
transfers_router,
|
||||
):
|
||||
spaces = [
|
||||
FileSpaceResponse(
|
||||
id=f"user:{principal.user.id}",
|
||||
label="My files",
|
||||
owner_type="user",
|
||||
owner_id=principal.user.id,
|
||||
description="Files owned by your user account.",
|
||||
)
|
||||
]
|
||||
group_ids = user_group_ids(session, tenant_id=principal.tenant_id, user_id=principal.user.id, include_admin_groups=_is_admin(principal))
|
||||
if group_ids:
|
||||
groups = session.query(Group).filter(Group.tenant_id == principal.tenant_id, Group.id.in_(group_ids)).order_by(Group.name.asc()).all()
|
||||
spaces.extend(
|
||||
FileSpaceResponse(
|
||||
id=f"group:{group.id}",
|
||||
label=f"{group.name} files",
|
||||
owner_type="group",
|
||||
owner_id=group.id,
|
||||
description="Files owned by this group.",
|
||||
)
|
||||
for group in groups
|
||||
)
|
||||
return FileSpacesResponse(spaces=spaces)
|
||||
|
||||
|
||||
@router.get("/folders", response_model=FileFoldersResponse)
|
||||
def list_file_folders(
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
folders = list_folders_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileFoldersResponse(folders=[_folder_response(folder) for folder in folders])
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders", response_model=FileFolderResponse)
|
||||
def create_file_folder(
|
||||
payload: FileFolderCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
folder = create_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return _folder_response(folder)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
|
||||
def delete_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
deleted_folders, deleted_files = soft_delete_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderDeleteResponse(deleted_folders=deleted_folders, deleted_files=deleted_files)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get("", response_model=FileListResponse)
|
||||
def list_files(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(files=[_asset_response(session, asset, include_shares=True) for asset in assets])
|
||||
|
||||
|
||||
@router.post("/upload", response_model=FileUploadResponse)
|
||||
async def upload_files(
|
||||
files: list[UploadFile] = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
unpack_zip: bool = Form(default=False),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(default="reject"),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
uploaded_assets: list[FileAsset] = []
|
||||
try:
|
||||
raw_resolutions = json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
upload_resolutions = _conflict_resolutions([ConflictResolutionRequest(**item) for item in raw_resolutions])
|
||||
for upload in files:
|
||||
filename = upload.filename or "file"
|
||||
content_type = upload.content_type or None
|
||||
upload_limit = settings.file_upload_zip_max_bytes if unpack_zip and filename.lower().endswith(".zip") else settings.file_upload_max_bytes
|
||||
data = await _read_limited_upload(upload, max_bytes=upload_limit)
|
||||
if unpack_zip and filename.lower().endswith(".zip"):
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=data,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
is_admin=_is_admin(principal),
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
uploaded_assets.extend(item.asset for item in extracted)
|
||||
continue
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder=path,
|
||||
content_type=content_type,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
uploaded_assets.append(stored.asset)
|
||||
session.commit()
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(files=[_asset_response(session, asset, include_shares=True) for asset in uploaded_assets])
|
||||
|
||||
|
||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||
async def upload_zip(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(default="reject"),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
data = await _read_limited_upload(file, max_bytes=settings.file_upload_zip_max_bytes)
|
||||
target_owner = owner_id or principal.user.id
|
||||
try:
|
||||
raw_resolutions = json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
upload_resolutions = _conflict_resolutions([ConflictResolutionRequest(**item) for item in raw_resolutions])
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=data,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
is_admin=_is_admin(principal),
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
session.commit()
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(files=[_asset_response(session, item.asset, include_shares=True) for item in extracted])
|
||||
|
||||
|
||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||
def get_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.get("/{file_id}/download")
|
||||
def download_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
data, _, blob = read_asset_bytes(session, asset)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
|
||||
return StreamingResponse(BytesIO(data), media_type=blob.content_type or "application/octet-stream", headers=headers)
|
||||
|
||||
|
||||
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
||||
def delete_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
count = soft_delete_assets(session, [asset])
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
|
||||
def bulk_delete_files(
|
||||
payload: BulkDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
count = soft_delete_assets(session, assets)
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/{file_id}/shares", response_model=FileShareResponse)
|
||||
def create_share(
|
||||
file_id: str,
|
||||
payload: FileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
share = share_file(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset=asset,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
)
|
||||
session.commit()
|
||||
return FileShareResponse(
|
||||
id=share.id,
|
||||
target_type=share.target_type,
|
||||
target_id=share.target_id,
|
||||
permission=share.permission,
|
||||
created_at=share.created_at.isoformat(),
|
||||
revoked_at=share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-shares", response_model=BulkFileShareResponse)
|
||||
def create_bulk_shares(
|
||||
payload: BulkFileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
file_ids = list(dict.fromkeys(payload.file_ids))
|
||||
assets = [
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, require_write=True, is_admin=_is_admin(principal))
|
||||
for file_id in file_ids
|
||||
]
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
assets=assets,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
)
|
||||
session.commit()
|
||||
return BulkFileShareResponse(
|
||||
shared_count=len(shares),
|
||||
shares=[
|
||||
FileShareResponse(
|
||||
id=share.id,
|
||||
target_type=share.target_type,
|
||||
target_id=share.target_id,
|
||||
permission=share.permission,
|
||||
created_at=share.created_at.isoformat(),
|
||||
revoked_at=share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
)
|
||||
for share in shares
|
||||
],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
|
||||
@router.post("/bulk-rename", response_model=RenameResponse)
|
||||
def bulk_rename(
|
||||
payload: RenameRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
plan = rename_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
mode=payload.mode,
|
||||
new_name=payload.new_name,
|
||||
find=payload.find,
|
||||
replacement=payload.replacement,
|
||||
prefix=payload.prefix,
|
||||
suffix=payload.suffix,
|
||||
recursive=payload.recursive,
|
||||
dry_run=payload.dry_run,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if not payload.dry_run:
|
||||
session.commit()
|
||||
return RenameResponse(
|
||||
dry_run=payload.dry_run,
|
||||
items=[
|
||||
RenamePreviewItem(
|
||||
kind=item.kind,
|
||||
id=item.id,
|
||||
file_id=item.id if item.kind == "file" else None,
|
||||
folder_path=item.old_path if item.kind == "folder" else None,
|
||||
old_path=item.old_path,
|
||||
new_path=item.new_path,
|
||||
)
|
||||
for item in plan
|
||||
],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/transfer", response_model=TransferResponse)
|
||||
def transfer_files(
|
||||
payload: TransferRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
files, folders = transfer_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
operation=payload.operation,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
source_owner_type=payload.source_owner_type,
|
||||
source_owner_id=payload.source_owner_id,
|
||||
target_owner_type=payload.target_owner_type,
|
||||
target_owner_id=payload.target_owner_id,
|
||||
target_folder=payload.target_folder,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return TransferResponse(operation=payload.operation, files=files, folders=folders)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/archive.zip")
|
||||
def download_archive(
|
||||
payload: ArchiveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="multimailer-files-", suffix=".zip", delete=False)
|
||||
tmp_path = tmp.name
|
||||
tmp.close()
|
||||
try:
|
||||
create_zip_file(session, assets, tmp_path)
|
||||
except Exception:
|
||||
_cleanup_temp_file(tmp_path)
|
||||
raise
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
filename = filename_from_path(normalize_logical_path(payload.filename, fallback_filename="files.zip"))
|
||||
headers = {"Content-Disposition": _attachment_disposition(filename)}
|
||||
return FileResponse(tmp_path, media_type="application/zip", headers=headers, background=BackgroundTask(_cleanup_temp_file, tmp_path))
|
||||
|
||||
|
||||
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
|
||||
def resolve_file_patterns(
|
||||
payload: PatternResolveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
try:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
campaign_id=payload.campaign_id,
|
||||
path_prefix=payload.path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
resolved, unmatched = resolve_patterns(assets, payload.patterns, base_path=payload.path_prefix, case_sensitive=payload.case_sensitive)
|
||||
return PatternResolveResponse(
|
||||
patterns=[PatternMatchResponse(pattern=item.pattern, matches=[_asset_response(session, asset) for asset in item.matches]) for item in resolved],
|
||||
unmatched=[_asset_response(session, asset) for asset in unmatched] if payload.include_unmatched else [],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
router.routes.extend(workflow_router.routes)
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Focused HTTP route modules for the Files API."""
|
||||
@@ -0,0 +1,175 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from io import BytesIO
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi.responses import StreamingResponse
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
BulkDeleteRequest,
|
||||
BulkDeleteResponse,
|
||||
FileAssetResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
read_asset_version_bytes,
|
||||
soft_delete_assets,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_attachment_disposition,
|
||||
_audit_connector_event,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||
def get_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.get("/{file_id}/download")
|
||||
def download_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
_audit_connector_event(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.accessed",
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
operation="download",
|
||||
commit=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {"Content-Disposition": _attachment_disposition(asset.filename)}
|
||||
return StreamingResponse(
|
||||
BytesIO(data),
|
||||
media_type=blob.content_type or "application/octet-stream",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{file_id}/versions/{version_id}/download")
|
||||
def download_file_version(
|
||||
file_id: str,
|
||||
version_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_version_bytes(session, asset, version_id)
|
||||
_audit_connector_event(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.accessed",
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
operation="download-version",
|
||||
commit=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
headers = {
|
||||
"Content-Disposition": _attachment_disposition(
|
||||
version.filename_at_upload or asset.filename
|
||||
)
|
||||
}
|
||||
return StreamingResponse(
|
||||
BytesIO(data),
|
||||
media_type=blob.content_type or "application/octet-stream",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{file_id}", response_model=BulkDeleteResponse)
|
||||
def delete_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
count = soft_delete_assets(session, [asset])
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-delete", response_model=BulkDeleteResponse)
|
||||
def bulk_delete_files(
|
||||
payload: BulkDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
count = soft_delete_assets(session, assets)
|
||||
session.commit()
|
||||
return BulkDeleteResponse(deleted_count=count)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,631 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorBrowseItem,
|
||||
FileConnectorBrowseResponse,
|
||||
FileConnectorFolderSyncItemResponse,
|
||||
FileConnectorFolderSyncRequest,
|
||||
FileConnectorFolderSyncResponse,
|
||||
FileConnectorFolderSyncSummary,
|
||||
FileConnectorImportRequest,
|
||||
FileConnectorSyncResponse,
|
||||
FileConnectorWriteRequest,
|
||||
FileConnectorWriteResponse,
|
||||
FileUploadResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_imports import (
|
||||
ConnectorImportError,
|
||||
ConnectorImportUnsupported,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_folder_sync import (
|
||||
connector_relative_path,
|
||||
discover_connector_folder,
|
||||
join_connector_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
ConnectorPolicyDenied,
|
||||
connector_policy_decision,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
get_asset_for_user,
|
||||
read_asset_bytes,
|
||||
sync_file_asset_from_source,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_spaces import (
|
||||
connector_space_owner_id,
|
||||
get_connector_space_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_writes import write_connector_file
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_audit_connector_imports,
|
||||
_audit_connector_sync,
|
||||
_connector_browse_next_token,
|
||||
_connector_policy_error,
|
||||
_connector_space_policy_decision,
|
||||
_download_connector_payload,
|
||||
_ensure_campaign_file_access,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
_visible_connector_profile,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/sync",
|
||||
response_model=FileConnectorFolderSyncResponse,
|
||||
)
|
||||
def sync_connector_space_folder(
|
||||
space_id: str,
|
||||
payload: FileConnectorFolderSyncRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
try:
|
||||
if space.sync_mode != "manual":
|
||||
raise FileStorageError("This connector space is not configured for manual sync")
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
remote_path = join_connector_path(space.remote_path, payload.path)
|
||||
target_folder = join_connector_path(payload.target_folder)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=space.library_id,
|
||||
remote_path=remote_path,
|
||||
operation="sync",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
discovery = discover_connector_folder(
|
||||
profile,
|
||||
path=remote_path,
|
||||
library_id=space.library_id,
|
||||
recursive=payload.recursive,
|
||||
max_files=payload.max_files,
|
||||
max_depth=payload.max_depth,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (ConnectorBrowseError, FileStorageError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
owner_id = connector_space_owner_id(space)
|
||||
items: list[FileConnectorFolderSyncItemResponse] = [
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=skipped.path,
|
||||
action="skipped",
|
||||
detail=skipped.reason,
|
||||
)
|
||||
for skipped in discovery.skipped
|
||||
]
|
||||
counts = {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"skipped": len(discovery.skipped),
|
||||
"conflicts": 0,
|
||||
"policy_denied": 0,
|
||||
"failed": 0,
|
||||
}
|
||||
|
||||
for source in discovery.files:
|
||||
source_path = normalize_connector_browse_path(source.path)
|
||||
try:
|
||||
relative_path = connector_relative_path(
|
||||
space_root=space.remote_path,
|
||||
item_path=source_path,
|
||||
)
|
||||
target_path = join_connector_path(target_folder, relative_path)
|
||||
except (ConnectorBrowseError, ValueError) as exc:
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
action="failed",
|
||||
detail=str(exc),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
try:
|
||||
with session.begin_nested():
|
||||
file_payload = FileConnectorImportRequest(
|
||||
library_id=space.library_id or "",
|
||||
path=source_path,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=owner_id,
|
||||
target_path=target_path,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
source_revision=source.etag,
|
||||
metadata={
|
||||
**dict(source.metadata),
|
||||
**payload.metadata,
|
||||
"connector_space_id": space.id,
|
||||
"browse_name": source.name,
|
||||
"browse_path": source_path,
|
||||
"browse_modified_at": source.modified_at,
|
||||
"browse_etag": source.etag,
|
||||
"folder_sync": True,
|
||||
},
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, file_payload, operation="sync"
|
||||
)
|
||||
stored, sync_action, previous_version_id = (
|
||||
sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
display_path=target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
)
|
||||
_audit_connector_sync(
|
||||
session,
|
||||
principal,
|
||||
stored.asset,
|
||||
sync_action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
)
|
||||
file_response = _asset_response(
|
||||
session, stored.asset, include_shares=True
|
||||
)
|
||||
counts[sync_action] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action=sync_action,
|
||||
file=file_response,
|
||||
previous_version_id=previous_version_id,
|
||||
current_version_id=stored.version.id,
|
||||
source_revision=file_response.source_revision,
|
||||
)
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
counts["policy_denied"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action="policy_denied",
|
||||
source_revision=source.etag,
|
||||
detail=str(exc),
|
||||
policy_decision=exc.decision.to_dict(),
|
||||
)
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
detail = str(exc)
|
||||
if detail.startswith("Skipped upload target:"):
|
||||
action = "skipped"
|
||||
counts["skipped"] += 1
|
||||
elif detail.startswith("Target file already exists:"):
|
||||
action = "conflict"
|
||||
counts["conflicts"] += 1
|
||||
else:
|
||||
action = "failed"
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action=action,
|
||||
source_revision=source.etag,
|
||||
detail=detail,
|
||||
)
|
||||
)
|
||||
except (
|
||||
ConnectorImportError,
|
||||
UnsafeFilePathError,
|
||||
OSError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
counts["failed"] += 1
|
||||
items.append(
|
||||
FileConnectorFolderSyncItemResponse(
|
||||
source_path=source_path,
|
||||
target_path=target_path,
|
||||
action="failed",
|
||||
source_revision=source.etag,
|
||||
detail=str(exc),
|
||||
)
|
||||
)
|
||||
|
||||
summary = FileConnectorFolderSyncSummary(
|
||||
discovered=len(discovery.files),
|
||||
**counts,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.folder_synced",
|
||||
object_type="file_connector_space",
|
||||
object_id=space.id,
|
||||
details={
|
||||
"connector_profile_id": profile.id,
|
||||
"provider": profile.provider,
|
||||
"library_id": space.library_id,
|
||||
"remote_path": remote_path,
|
||||
"target_folder": target_folder,
|
||||
"recursive": payload.recursive,
|
||||
"truncated": discovery.truncated,
|
||||
"summary": summary.model_dump(),
|
||||
"results": [
|
||||
{
|
||||
"source_path": item.source_path,
|
||||
"target_path": item.target_path,
|
||||
"action": item.action,
|
||||
"file_id": item.file.id if item.file else None,
|
||||
"current_version_id": item.current_version_id,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorFolderSyncResponse(
|
||||
connector_space_id=space.id,
|
||||
connector_profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
remote_path=remote_path,
|
||||
target_folder=target_folder,
|
||||
recursive=payload.recursive,
|
||||
truncated=discovery.truncated,
|
||||
summary=summary,
|
||||
items=items,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/write-back",
|
||||
response_model=FileConnectorWriteResponse,
|
||||
)
|
||||
def write_back_connector_file(
|
||||
space_id: str,
|
||||
payload: FileConnectorWriteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:connector:write")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if space.read_only:
|
||||
raise FileStorageError("This connector space is read-only")
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
requested_path = normalize_connector_browse_path(payload.remote_path)
|
||||
remote_path = normalize_connector_browse_path(
|
||||
"/".join(part for part in (space.remote_path, requested_path) if part)
|
||||
)
|
||||
decision = connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path=remote_path,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation="write",
|
||||
),
|
||||
profile.policy_sources,
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
asset = get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=payload.file_id,
|
||||
require_write=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
connector_library_id = space.library_id
|
||||
content_type = blob.content_type
|
||||
# Close the read snapshot before the independent recovery transaction
|
||||
# records authority for the external effect. This avoids upgrading an
|
||||
# older SQLite read snapshot after the ledger commit.
|
||||
session.commit()
|
||||
result = write_connector_file(
|
||||
profile,
|
||||
tenant_id=principal.tenant_id,
|
||||
library_id=connector_library_id,
|
||||
remote_path=remote_path,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
expected_revision=payload.expected_revision,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.written",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={
|
||||
"file_version_id": version.id,
|
||||
"file_blob_id": blob.id,
|
||||
"checksum_sha256": blob.checksum_sha256,
|
||||
"connector_space_id": space.id,
|
||||
"connector_profile_id": profile.id,
|
||||
"remote_path": remote_path,
|
||||
"recovery_operation_id": result.recovery_operation_id,
|
||||
"recovery_status": result.status,
|
||||
"revision": result.revision,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorWriteResponse(
|
||||
recovery_operation_id=result.recovery_operation_id,
|
||||
status=result.status,
|
||||
replayed=result.replayed,
|
||||
provider=result.provider,
|
||||
remote_path=result.remote_path,
|
||||
revision=result.revision,
|
||||
checksum_sha256=result.checksum_sha256,
|
||||
size_bytes=result.size_bytes,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ConnectorBrowseError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles/{profile_id}/import", response_model=FileUploadResponse
|
||||
)
|
||||
def import_connector_file(
|
||||
profile_id: str,
|
||||
payload: FileConnectorImportRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
if payload.campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, profile_id, campaign_id=payload.campaign_id
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, payload, operation="import"
|
||||
)
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
folder=payload.target_folder,
|
||||
display_path=payload.target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
campaign_id=payload.campaign_id,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_audit_connector_imports(session, principal, [stored.asset])
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorImportUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (
|
||||
ConnectorImportError,
|
||||
FileStorageError,
|
||||
UnsafeFilePathError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(
|
||||
files=[_asset_response(session, stored.asset, include_shares=True)]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles/{profile_id}/sync", response_model=FileConnectorSyncResponse
|
||||
)
|
||||
def sync_connector_file(
|
||||
profile_id: str,
|
||||
payload: FileConnectorImportRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
try:
|
||||
if payload.campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, profile_id, campaign_id=payload.campaign_id
|
||||
)
|
||||
_source_path, downloaded, metadata = _download_connector_payload(
|
||||
profile, payload, operation="sync"
|
||||
)
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
stored, sync_action, previous_version_id = sync_file_asset_from_source(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=downloaded.filename,
|
||||
data=downloaded.data,
|
||||
folder=payload.target_folder,
|
||||
display_path=payload.target_path,
|
||||
content_type=downloaded.content_type,
|
||||
metadata=metadata,
|
||||
campaign_id=payload.campaign_id,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_audit_connector_sync(
|
||||
session,
|
||||
principal,
|
||||
stored.asset,
|
||||
sync_action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
)
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorImportUnsupported as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (
|
||||
ConnectorImportError,
|
||||
FileStorageError,
|
||||
UnsafeFilePathError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorSyncResponse(
|
||||
file=_asset_response(session, stored.asset, include_shares=True),
|
||||
action=sync_action,
|
||||
previous_version_id=previous_version_id,
|
||||
current_version_id=stored.version.id,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/profiles/{profile_id}/browse",
|
||||
response_model=FileConnectorBrowseResponse,
|
||||
)
|
||||
def browse_connector_profile_items(
|
||||
profile_id: str,
|
||||
path: str | None = None,
|
||||
library_id: str | None = None,
|
||||
continuation_token: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
"system:settings:read",
|
||||
"admin:settings:read",
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, profile_id, campaign_id=campaign_id
|
||||
)
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
decision = connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path=browse_path,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation="browse",
|
||||
),
|
||||
profile.policy_sources,
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
items = browse_connector_profile(
|
||||
profile,
|
||||
path=browse_path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED, detail=str(exc)
|
||||
) from exc
|
||||
except (ConnectorBrowseError, OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorBrowseResponse(
|
||||
profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
path=browse_path,
|
||||
library_id=library_id,
|
||||
next_continuation_token=_connector_browse_next_token(items),
|
||||
has_more=any(bool(item.metadata.get("listing_truncated")) for item in items),
|
||||
decision=decision.to_dict(),
|
||||
items=[FileConnectorBrowseItem(**item.to_response()) for item in items],
|
||||
)
|
||||
@@ -0,0 +1,261 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorProfileResponse,
|
||||
FileConnectorProfileUpdateRequest,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||
delete_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
connector_profile_from_row,
|
||||
get_connector_profile_row,
|
||||
update_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
_can_read_disabled_connector_profiles,
|
||||
_connector_policy_error,
|
||||
_credential_row_for_profile,
|
||||
_ensure_connector_configuration_allowed,
|
||||
_ensure_connector_local_policy_allowed,
|
||||
_http_error,
|
||||
_record_connector_settings_change,
|
||||
_require_connector_profile_write,
|
||||
_visible_connector_profile,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||
)
|
||||
def get_connector_profile(
|
||||
profile_id: str,
|
||||
campaign_id: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
"system:settings:read",
|
||||
"admin:settings:read",
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
profile = _visible_connector_profile(
|
||||
session,
|
||||
principal,
|
||||
profile_id,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorProfileResponse(**profile.to_response())
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||
)
|
||||
def update_connector_profile(
|
||||
profile_id: str,
|
||||
payload: FileConnectorProfileUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_profile_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
profile_id=profile_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_profile_write(principal, row.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
credential_profile_id = (
|
||||
payload.credential_profile_id
|
||||
if payload.credential_profile_id is not None
|
||||
else row.credential_profile_id
|
||||
)
|
||||
provider = payload.provider if payload.provider is not None else row.provider
|
||||
credential_row = _credential_row_for_profile(
|
||||
session,
|
||||
principal,
|
||||
credential_profile_id=credential_profile_id,
|
||||
provider=provider,
|
||||
profile_id=row.id,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=row.id,
|
||||
credential_id=credential_profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=connector_effective_endpoint_url(
|
||||
provider=provider,
|
||||
endpoint_url=payload.endpoint_url
|
||||
if payload.endpoint_url is not None
|
||||
else row.endpoint_url,
|
||||
metadata=payload.metadata
|
||||
if payload.metadata is not None
|
||||
else row.metadata_,
|
||||
),
|
||||
base_path=payload.base_path
|
||||
if payload.base_path is not None
|
||||
else row.base_path,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
operation="configure",
|
||||
)
|
||||
if payload.policy is not None:
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
update_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
enabled=payload.enabled,
|
||||
credential_profile_id=payload.credential_profile_id,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username if credentials else None,
|
||||
password=credentials.password if credentials else None,
|
||||
token=credentials.token if credentials else None,
|
||||
password_env=credentials.password_env if credentials else None,
|
||||
token_env=credentials.token_env if credentials else None,
|
||||
secret_ref=credentials.secret_ref if credentials else None,
|
||||
clear_password=payload.clear_password,
|
||||
clear_token=payload.clear_token,
|
||||
capabilities=payload.capabilities,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return FileConnectorProfileResponse(
|
||||
**connector_profile_from_row(
|
||||
row, credential_row=credential_row
|
||||
).to_response()
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/connectors/profiles/{profile_id}", response_model=FileConnectorProfileResponse
|
||||
)
|
||||
def deactivate_connector_profile(
|
||||
profile_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_profile_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
profile_id=profile_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_profile_write(principal, row.scope_type)
|
||||
try:
|
||||
changed = delete_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason="api_delete",
|
||||
user_id=principal.user.id,
|
||||
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||
)
|
||||
if changed:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="deleted",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return FileConnectorProfileResponse(
|
||||
**connector_profile_from_row(row).to_response()
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
@@ -0,0 +1,852 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorCredentialCreateRequest,
|
||||
FileConnectorCredentialResponse,
|
||||
FileConnectorCredentialsResponse,
|
||||
FileConnectorCredentialUpdateRequest,
|
||||
FileConnectorDiscoveryRequest,
|
||||
FileConnectorDiscoveryResponse,
|
||||
FileConnectorSettingsDeltaResponse,
|
||||
FileConnectorPolicyEvaluateRequest,
|
||||
FileConnectorPolicyEvaluateResponse,
|
||||
FileConnectorPolicyResponse,
|
||||
FileConnectorPolicyUpdateRequest,
|
||||
FileConnectorProfileCreateRequest,
|
||||
FileConnectorProfileResponse,
|
||||
FileConnectorProfilesResponse,
|
||||
FileConnectorProviderResponse,
|
||||
FileConnectorProvidersResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
create_connector_credential_row,
|
||||
get_connector_credential_row,
|
||||
list_database_connector_credentials,
|
||||
update_connector_credential_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||
delete_connector_credential_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
browse_connector_profile,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
reject_api_controlled_deployment_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
connector_profile_from_row,
|
||||
create_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_providers import (
|
||||
connector_provider_descriptors,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
ConnectorPolicyDenied,
|
||||
connector_policy_decision,
|
||||
connector_policy_sources_from_payload,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
connector_policy_response,
|
||||
set_connector_policy,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
FILES_CONNECTOR_POLICY_RESOURCE,
|
||||
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
_audit_connector_discovery_attempt,
|
||||
_can_read_disabled_connector_profiles,
|
||||
_connector_credential_response,
|
||||
_connector_policy_error,
|
||||
_credential_row_for_profile,
|
||||
_discovery_profile_from_payload,
|
||||
_ensure_campaign_file_access,
|
||||
_ensure_connector_configuration_allowed,
|
||||
_ensure_connector_credential_configuration_allowed,
|
||||
_ensure_connector_local_policy_allowed,
|
||||
_file_connector_policy_resource_id,
|
||||
_file_connector_settings_entries,
|
||||
_http_error,
|
||||
_record_connector_settings_change,
|
||||
_require_connector_credential_write,
|
||||
_require_connector_policy_read,
|
||||
_require_connector_profile_write,
|
||||
_same_endpoint,
|
||||
_visible_connector_profiles,
|
||||
_webdav_discovery_candidates,
|
||||
)
|
||||
from govoplan_files.backend.services.connector_settings_delta import (
|
||||
_full_file_connector_settings_delta_response,
|
||||
_incremental_file_connector_settings_delta_response,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-policy/evaluate", response_model=FileConnectorPolicyEvaluateResponse
|
||||
)
|
||||
def evaluate_connector_policy(
|
||||
payload: FileConnectorPolicyEvaluateRequest,
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope("files:file:read", "files:file:upload", "files:file:download")
|
||||
),
|
||||
):
|
||||
del principal
|
||||
sources = connector_policy_sources_from_payload(
|
||||
[item.model_dump(mode="json") for item in payload.policy_sources]
|
||||
)
|
||||
request = ConnectorAccessRequest.from_provenance(
|
||||
payload.source_provenance.model_dump(mode="json", exclude_none=True),
|
||||
operation=payload.operation,
|
||||
)
|
||||
return FileConnectorPolicyEvaluateResponse(
|
||||
decision=connector_policy_decision(request, sources).to_dict()
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/settings/delta", response_model=FileConnectorSettingsDeltaResponse
|
||||
)
|
||||
def connector_settings_delta(
|
||||
scope_type: str = Query(default="tenant"),
|
||||
scope_id: str | None = Query(default=None),
|
||||
provider: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
include_inactive: bool = False,
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
since: str | None = None,
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
scope_type = scope_type.strip().casefold()
|
||||
_require_connector_policy_read(principal, scope_type)
|
||||
try:
|
||||
if since is None:
|
||||
return _full_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
entries, has_more = _file_connector_settings_entries(
|
||||
session, tenant_id=principal.tenant_id, since=since, limit=limit
|
||||
)
|
||||
if entries is None:
|
||||
return _full_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
return _incremental_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
entries=entries,
|
||||
has_more=has_more,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get("/connectors/providers", response_model=FileConnectorProvidersResponse)
|
||||
def list_connector_providers(
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope("files:file:read", "files:file:upload", "files:file:admin")
|
||||
),
|
||||
):
|
||||
del principal
|
||||
return FileConnectorProvidersResponse(
|
||||
providers=[
|
||||
FileConnectorProviderResponse(**item.to_response())
|
||||
for item in connector_provider_descriptors()
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post("/connectors/discover", response_model=FileConnectorDiscoveryResponse)
|
||||
def discover_connector_endpoint(
|
||||
payload: FileConnectorDiscoveryRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=payload.credentials.password_env,
|
||||
token_env=payload.credentials.token_env,
|
||||
secret_ref=payload.credentials.secret_ref,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
if payload.provider not in {"webdav", "nextcloud"}:
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=None,
|
||||
base_path=payload.base_path,
|
||||
status="unsupported",
|
||||
message=f"Discovery is not implemented for {payload.provider} connectors yet",
|
||||
)
|
||||
candidates: list[dict[str, str]] = []
|
||||
for endpoint_url in _webdav_discovery_candidates(payload):
|
||||
profile = _discovery_profile_from_payload(
|
||||
payload, endpoint_url, principal=principal
|
||||
)
|
||||
try:
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=None,
|
||||
credential_id=None,
|
||||
provider=profile.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
scope_type="tenant",
|
||||
scope_id=principal.tenant_id,
|
||||
operation="discover",
|
||||
)
|
||||
_audit_connector_discovery_attempt(
|
||||
session,
|
||||
principal,
|
||||
provider=profile.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
)
|
||||
browse_connector_profile(profile, path=payload.base_path or "")
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
OSError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
message = str(exc)
|
||||
if "credentials were rejected" in message.casefold():
|
||||
if payload.require_valid_credentials:
|
||||
candidates.append(
|
||||
{
|
||||
"endpoint_url": endpoint_url,
|
||||
"status": "credentials_rejected",
|
||||
"message": "The endpoint exists, but the credentials were rejected.",
|
||||
}
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
status="credentials_rejected",
|
||||
message="The endpoint was found, but login failed with these credentials.",
|
||||
candidates=candidates,
|
||||
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||
)
|
||||
candidates.append(
|
||||
{
|
||||
"endpoint_url": endpoint_url,
|
||||
"status": "found",
|
||||
"message": "The endpoint exists, but credentials are required or were rejected.",
|
||||
}
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
status="found",
|
||||
message="The endpoint was found. Add working credentials before saving or testing the connection.",
|
||||
candidates=candidates,
|
||||
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||
)
|
||||
candidates.append(
|
||||
{"endpoint_url": endpoint_url, "status": "failed", "message": message}
|
||||
)
|
||||
continue
|
||||
status_value = (
|
||||
"usable" if _same_endpoint(endpoint_url, payload.endpoint_url) else "found"
|
||||
)
|
||||
message = (
|
||||
"The supplied URL is directly usable."
|
||||
if status_value == "usable"
|
||||
else "A usable connector endpoint was discovered."
|
||||
)
|
||||
candidates.append(
|
||||
{"endpoint_url": endpoint_url, "status": status_value, "message": message}
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
status=status_value,
|
||||
message=message,
|
||||
candidates=candidates,
|
||||
metadata={"discovered_by": "webdav-propfind"},
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
endpoint_url=None,
|
||||
base_path=payload.base_path,
|
||||
status="not_found",
|
||||
message="No usable WebDAV endpoint was found for this server URL.",
|
||||
candidates=candidates,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/connectors/credentials", response_model=FileConnectorCredentialsResponse)
|
||||
def list_connector_credentials(
|
||||
provider: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
provider_norm = provider.strip().casefold() if provider else None
|
||||
credentials = list_database_connector_credentials(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
)
|
||||
return FileConnectorCredentialsResponse(
|
||||
credentials=[
|
||||
FileConnectorCredentialResponse(**credential.to_response())
|
||||
for credential in credentials
|
||||
if provider_norm is None or credential.provider in {None, provider_norm}
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
|
||||
)
|
||||
def read_connector_policy(
|
||||
scope_type: str,
|
||||
scope_id: str | None = Query(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_policy_read(principal, scope_type)
|
||||
try:
|
||||
return FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.put(
|
||||
"/connectors/policies/{scope_type}", response_model=FileConnectorPolicyResponse
|
||||
)
|
||||
def write_connector_policy(
|
||||
scope_type: str,
|
||||
payload: FileConnectorPolicyUpdateRequest,
|
||||
scope_id: str | None = Query(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_profile_write(principal, scope_type)
|
||||
try:
|
||||
set_connector_policy(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_POLICY_RESOURCE,
|
||||
resource_id=_file_connector_policy_resource_id(scope_type, scope_id),
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=None
|
||||
if scope_type.strip().casefold() == "system"
|
||||
else principal.tenant_id,
|
||||
payload={"scope_type": scope_type.strip().casefold(), "scope_id": scope_id},
|
||||
)
|
||||
session.commit()
|
||||
return FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/credentials",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_connector_credential(
|
||||
payload: FileConnectorCredentialCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_credential_write(principal, payload.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
_ensure_connector_credential_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
credential_id=payload.id,
|
||||
provider=payload.provider,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
operation="configure_credentials",
|
||||
)
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
row = create_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
credential_id=payload.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
enabled=payload.enabled,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username,
|
||||
password=credentials.password,
|
||||
token=credentials.token,
|
||||
password_env=credentials.password_env,
|
||||
token_env=credentials.token_env,
|
||||
secret_ref=credentials.secret_ref,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="created",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get(
|
||||
"/connectors/credentials/{credential_id}",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
)
|
||||
def get_connector_credential(
|
||||
credential_id: str,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:read", "admin:settings:read"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
credential_id=credential_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
)
|
||||
return _connector_credential_response(row)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/connectors/credentials/{credential_id}",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
)
|
||||
def update_connector_credential(
|
||||
credential_id: str,
|
||||
payload: FileConnectorCredentialUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
credential_id=credential_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_credential_write(principal, row.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
provider = payload.provider if payload.provider is not None else row.provider
|
||||
_ensure_connector_credential_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
credential_id=row.id,
|
||||
provider=provider,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
operation="configure_credentials",
|
||||
)
|
||||
if payload.policy is not None:
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
update_connector_credential_row(
|
||||
session,
|
||||
row,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
enabled=payload.enabled,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username if credentials else None,
|
||||
password=credentials.password if credentials else None,
|
||||
token=credentials.token if credentials else None,
|
||||
password_env=credentials.password_env if credentials else None,
|
||||
token_env=credentials.token_env if credentials else None,
|
||||
secret_ref=credentials.secret_ref if credentials else None,
|
||||
clear_password=payload.clear_password,
|
||||
clear_token=payload.clear_token,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/connectors/credentials/{credential_id}",
|
||||
response_model=FileConnectorCredentialResponse,
|
||||
)
|
||||
def deactivate_connector_credential(
|
||||
credential_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
row = get_connector_credential_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
credential_id=credential_id,
|
||||
include_disabled=True,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_credential_write(principal, row.scope_type)
|
||||
try:
|
||||
deletion = delete_connector_credential_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason="api_delete",
|
||||
user_id=principal.user.id,
|
||||
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||
)
|
||||
if deletion.changed:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="deleted",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
for profile in deletion.affected_profiles:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=profile.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=profile.tenant_id,
|
||||
payload={
|
||||
"scope_type": profile.scope_type,
|
||||
"scope_id": profile.scope_id,
|
||||
"provider": profile.provider,
|
||||
"reason": "credential_deleted",
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
|
||||
def list_connector_profiles(
|
||||
provider: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
include_disabled: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:download",
|
||||
"files:file:admin",
|
||||
"system:settings:read",
|
||||
"admin:settings:read",
|
||||
)
|
||||
),
|
||||
):
|
||||
try:
|
||||
if campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return FileConnectorProfilesResponse(
|
||||
profiles=[
|
||||
FileConnectorProfileResponse(**profile.to_response())
|
||||
for profile in profiles
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connectors/profiles",
|
||||
response_model=FileConnectorProfileResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_connector_profile(
|
||||
payload: FileConnectorProfileCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(
|
||||
require_any_scope(
|
||||
"files:file:admin", "system:settings:write", "admin:settings:write"
|
||||
)
|
||||
),
|
||||
):
|
||||
_require_connector_profile_write(principal, payload.scope_type)
|
||||
credentials = payload.credentials
|
||||
try:
|
||||
credential_row = _credential_row_for_profile(
|
||||
session,
|
||||
principal,
|
||||
credential_profile_id=payload.credential_profile_id,
|
||||
provider=payload.provider,
|
||||
profile_id=payload.id,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
)
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=payload.id,
|
||||
credential_id=payload.credential_profile_id,
|
||||
provider=payload.provider,
|
||||
endpoint_url=connector_effective_endpoint_url(
|
||||
provider=payload.provider,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
metadata=payload.metadata,
|
||||
),
|
||||
base_path=payload.base_path,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
operation="configure",
|
||||
)
|
||||
_ensure_connector_local_policy_allowed(
|
||||
session,
|
||||
principal,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
policy=payload.policy,
|
||||
)
|
||||
row = create_connector_profile_row(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
profile_id=payload.id,
|
||||
label=payload.label,
|
||||
provider=payload.provider,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
base_path=payload.base_path,
|
||||
enabled=payload.enabled,
|
||||
credential_profile_id=payload.credential_profile_id,
|
||||
credential_mode=payload.credential_mode,
|
||||
username=credentials.username,
|
||||
password=credentials.password,
|
||||
token=credentials.token,
|
||||
password_env=credentials.password_env,
|
||||
token_env=credentials.token_env,
|
||||
secret_ref=credentials.secret_ref,
|
||||
capabilities=payload.capabilities,
|
||||
policy=payload.policy,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=row.id,
|
||||
operation="created",
|
||||
principal=principal,
|
||||
tenant_id=row.tenant_id,
|
||||
payload={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return FileConnectorProfileResponse(
|
||||
**connector_profile_from_row(
|
||||
row, credential_row=credential_row
|
||||
).to_response()
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,151 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileFolderCreateRequest,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderDeleteResponse,
|
||||
FileFolderResponse,
|
||||
FileFoldersResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.paths import UnsafeFilePathError
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.folders import (
|
||||
create_folder,
|
||||
list_folders_for_user,
|
||||
list_folders_for_user_window,
|
||||
soft_delete_folder,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_folder_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.services.list_queries import (
|
||||
FOLDERS_LIST_CURSOR_SCOPE,
|
||||
_cursor_page_size,
|
||||
_files_delta_watermark,
|
||||
_folder_cursor_values,
|
||||
_folders_list_fingerprint,
|
||||
_next_folder_list_cursor,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/folders", response_model=FileFoldersResponse)
|
||||
def list_file_folders(
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
watermark = _files_delta_watermark(session, principal.tenant_id)
|
||||
effective_page_size = _cursor_page_size(
|
||||
FOLDERS_LIST_CURSOR_SCOPE, cursor, page_size
|
||||
)
|
||||
if effective_page_size is None:
|
||||
folders = list_folders_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileFoldersResponse(
|
||||
folders=[_folder_response(folder) for folder in folders],
|
||||
watermark=watermark,
|
||||
)
|
||||
fingerprint = _folders_list_fingerprint(
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
page_size=effective_page_size,
|
||||
)
|
||||
after_path, after_id = _folder_cursor_values(cursor, fingerprint=fingerprint)
|
||||
folders, has_more = list_folders_for_user_window(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
page_size=effective_page_size,
|
||||
after_path=after_path,
|
||||
after_id=after_id,
|
||||
)
|
||||
return FileFoldersResponse(
|
||||
folders=[_folder_response(folder) for folder in folders],
|
||||
cursor=cursor,
|
||||
next_cursor=_next_folder_list_cursor(
|
||||
principal,
|
||||
folders,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
page_size=effective_page_size,
|
||||
has_more=has_more,
|
||||
),
|
||||
watermark=watermark,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders", response_model=FileFolderResponse)
|
||||
def create_file_folder(
|
||||
payload: FileFolderCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
folder = create_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return _folder_response(folder)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/folders/delete", response_model=FileFolderDeleteResponse)
|
||||
def delete_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:delete")),
|
||||
):
|
||||
try:
|
||||
deleted_folders, deleted_files = soft_delete_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderDeleteResponse(
|
||||
deleted_folders=deleted_folders, deleted_files=deleted_files
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,172 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
|
||||
from fastapi import (
|
||||
APIRouter,
|
||||
Depends,
|
||||
File as FastAPIFile,
|
||||
Header,
|
||||
HTTPException,
|
||||
UploadFile,
|
||||
status,
|
||||
)
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.events import (
|
||||
EventActorRef,
|
||||
EventObjectRef,
|
||||
EventTenantRef,
|
||||
PlatformEvent,
|
||||
emit_platform_event,
|
||||
)
|
||||
from govoplan_core.core.institutional import EvidenceReference
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import FileFormEvidenceGrant
|
||||
from govoplan_files.backend.route_support import _read_limited_upload
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import create_file_asset
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/form-evidence/upload",
|
||||
response_model=dict[str, object],
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def upload_form_evidence(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
x_form_evidence_token: str = Header(alias="X-Form-Evidence-Token"),
|
||||
session: Session = Depends(get_session),
|
||||
) -> dict[str, object]:
|
||||
now = datetime.now(UTC)
|
||||
try:
|
||||
grant = (
|
||||
session.query(FileFormEvidenceGrant)
|
||||
.filter(
|
||||
FileFormEvidenceGrant.token_sha256
|
||||
== _token_sha256(x_form_evidence_token)
|
||||
)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if grant is None or grant.status != "issued" or _aware(grant.expires_at) <= now:
|
||||
raise _upload_unavailable()
|
||||
content_type = (
|
||||
str(file.content_type or "application/octet-stream")
|
||||
.split(";", 1)[0]
|
||||
.strip()
|
||||
.casefold()
|
||||
)
|
||||
if grant.allowed_content_types and content_type not in set(
|
||||
grant.allowed_content_types
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
|
||||
detail="This Form does not accept the uploaded document type.",
|
||||
)
|
||||
data = _read_limited_upload(file, max_bytes=grant.max_size_bytes)
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=grant.tenant_id,
|
||||
owner_type="user",
|
||||
owner_id=grant.custodian_user_id,
|
||||
user_id=grant.custodian_user_id,
|
||||
filename=file.filename or "form-attachment",
|
||||
data=data,
|
||||
folder=f"Form submissions/{grant.form_instance_id}",
|
||||
content_type=content_type,
|
||||
description="Managed attachment captured through Forms Runtime.",
|
||||
metadata={
|
||||
"form_evidence": {
|
||||
"grant_id": grant.id,
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"form_definition_id": grant.form_definition_id,
|
||||
"form_definition_revision": grant.form_definition_revision,
|
||||
"purpose": grant.purpose,
|
||||
},
|
||||
"source_provenance": {
|
||||
"source_type": "form_evidence",
|
||||
"connector_id": "forms_runtime.evidence.files",
|
||||
"provider": "forms_runtime",
|
||||
"external_id": grant.id,
|
||||
"revision": grant.form_definition_revision,
|
||||
},
|
||||
},
|
||||
conflict_strategy="rename",
|
||||
is_admin=True,
|
||||
)
|
||||
grant.status = "uploaded"
|
||||
grant.file_asset_id = stored.asset.id
|
||||
grant.file_version_id = stored.version.id
|
||||
grant.uploaded_at = now
|
||||
session.add(grant)
|
||||
evidence = EvidenceReference(
|
||||
kind="document",
|
||||
owner_module="files",
|
||||
evidence_id=stored.asset.id,
|
||||
tenant_id=grant.tenant_id,
|
||||
version=stored.version.id,
|
||||
checksum=stored.version.checksum_sha256,
|
||||
source_ref=(f"form_submission:{grant.form_instance_id}:grant:{grant.id}"),
|
||||
responsible_actor_ref=f"user:{grant.custodian_user_id}",
|
||||
captured_at=now,
|
||||
)
|
||||
emit_platform_event(
|
||||
session,
|
||||
PlatformEvent(
|
||||
type="files.form_evidence.uploaded",
|
||||
module_id="files",
|
||||
payload={
|
||||
"grant_id": grant.id,
|
||||
"form_instance_id": grant.form_instance_id,
|
||||
"file_asset_id": stored.asset.id,
|
||||
"file_version_id": stored.version.id,
|
||||
"checksum_sha256": stored.version.checksum_sha256,
|
||||
"size_bytes": stored.version.size_bytes,
|
||||
},
|
||||
occurred_at=now,
|
||||
actor=EventActorRef(type="user", id=grant.custodian_user_id),
|
||||
tenant=EventTenantRef(id=grant.tenant_id),
|
||||
resource=EventObjectRef(type="file", id=stored.asset.id),
|
||||
classification="confidential",
|
||||
),
|
||||
)
|
||||
session.commit()
|
||||
except HTTPException:
|
||||
session.rollback()
|
||||
raise
|
||||
except (FileStorageError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
return {
|
||||
"grant_id": grant.id,
|
||||
"evidence": evidence.to_dict(include_inspection=True),
|
||||
}
|
||||
|
||||
|
||||
def _token_sha256(token: str) -> str:
|
||||
clean = str(token or "").strip()
|
||||
if len(clean) < 32:
|
||||
raise _upload_unavailable()
|
||||
return hashlib.sha256(clean.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _upload_unavailable() -> HTTPException:
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="The Form evidence upload is unavailable.",
|
||||
)
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = ["router", "upload_form_evidence"]
|
||||
@@ -0,0 +1,387 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileIntegrityActionRequest,
|
||||
FileIntegrityActionResponse,
|
||||
FileIntegrityFindingResponse,
|
||||
FileIntegrityFindingsResponse,
|
||||
FileIntegrityScanCreateRequest,
|
||||
FileIntegrityScanRunRequest,
|
||||
FileIntegrityScanResponse,
|
||||
FileIntegrityScansResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
cleanup_orphan_finding,
|
||||
create_integrity_scan,
|
||||
mark_integrity_scan_failed,
|
||||
recheck_integrity_finding,
|
||||
run_integrity_scan_batch,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files/integrity", tags=["files-integrity"])
|
||||
|
||||
|
||||
@router.get("/scans", response_model=FileIntegrityScansResponse)
|
||||
def list_integrity_scans(
|
||||
limit: int = Query(default=50, ge=1, le=200),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityScansResponse:
|
||||
rows = (
|
||||
session.query(FileIntegrityScan)
|
||||
.filter(FileIntegrityScan.tenant_id == principal.tenant_id)
|
||||
.order_by(FileIntegrityScan.created_at.desc(), FileIntegrityScan.id.desc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return FileIntegrityScansResponse(
|
||||
scans=[_scan_response(row) for row in rows]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/scans",
|
||||
response_model=FileIntegrityScanResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_scan(
|
||||
payload: FileIntegrityScanCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityScanResponse:
|
||||
try:
|
||||
scan = create_integrity_scan(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
verify_checksums=payload.verify_checksums,
|
||||
batch_size=payload.batch_size,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.integrity.scan_created",
|
||||
object_type="file_integrity_scan",
|
||||
object_id=scan.id,
|
||||
details={
|
||||
"storage_backend": scan.storage_backend,
|
||||
"verify_checksums": scan.verify_checksums,
|
||||
"batch_size": scan.batch_size,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _scan_response(scan)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/scans/{scan_id}/run", response_model=FileIntegrityScanResponse)
|
||||
def run_scan_batch(
|
||||
scan_id: str,
|
||||
payload: FileIntegrityScanRunRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityScanResponse:
|
||||
scan = _scan_for_tenant(
|
||||
session,
|
||||
scan_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
_assert_expected_revision(scan.revision, payload.expected_revision)
|
||||
previous_status = scan.status
|
||||
try:
|
||||
run_integrity_scan_batch(session, scan)
|
||||
if scan.status == "completed" and previous_status != "completed":
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.integrity.scan_completed",
|
||||
object_type="file_integrity_scan",
|
||||
object_id=scan.id,
|
||||
details={
|
||||
"verified_blob_count": scan.verified_blob_count,
|
||||
"quarantined_blob_count": scan.quarantined_blob_count,
|
||||
"orphan_object_count": scan.orphan_object_count,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _scan_response(scan)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
scan = _scan_for_tenant(
|
||||
session,
|
||||
scan_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
mark_integrity_scan_failed(scan, error=exc)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.integrity.scan_failed",
|
||||
object_type="file_integrity_scan",
|
||||
object_id=scan.id,
|
||||
details={"error_type": type(exc).__name__},
|
||||
)
|
||||
session.commit()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get(
|
||||
"/scans/{scan_id}/findings",
|
||||
response_model=FileIntegrityFindingsResponse,
|
||||
)
|
||||
def list_integrity_findings(
|
||||
scan_id: str,
|
||||
state_filter: str | None = Query(default=None, alias="state"),
|
||||
limit: int = Query(default=500, ge=1, le=1000),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityFindingsResponse:
|
||||
scan = _scan_for_tenant(session, scan_id, principal.tenant_id)
|
||||
query = session.query(FileIntegrityFinding).filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.tenant_id == principal.tenant_id,
|
||||
)
|
||||
if state_filter:
|
||||
query = query.filter(FileIntegrityFinding.state == state_filter)
|
||||
rows = (
|
||||
query.order_by(
|
||||
FileIntegrityFinding.created_at.asc(),
|
||||
FileIntegrityFinding.id.asc(),
|
||||
)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return FileIntegrityFindingsResponse(
|
||||
findings=[_finding_response(row) for row in rows]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/findings/{finding_id}/recheck",
|
||||
response_model=FileIntegrityActionResponse,
|
||||
)
|
||||
def recheck_finding(
|
||||
finding_id: str,
|
||||
payload: FileIntegrityActionRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityActionResponse:
|
||||
finding = _finding_for_tenant(
|
||||
session,
|
||||
finding_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
_assert_expected_revision(finding.revision, payload.expected_revision)
|
||||
try:
|
||||
result = recheck_integrity_finding(
|
||||
session,
|
||||
finding,
|
||||
user_id=principal.user.id,
|
||||
dry_run=payload.dry_run,
|
||||
)
|
||||
_audit_integrity_action(session, principal, result)
|
||||
session.commit()
|
||||
return _action_response(result)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/findings/{finding_id}/cleanup",
|
||||
response_model=FileIntegrityActionResponse,
|
||||
)
|
||||
def cleanup_finding(
|
||||
finding_id: str,
|
||||
payload: FileIntegrityActionRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:admin")),
|
||||
) -> FileIntegrityActionResponse:
|
||||
finding = _finding_for_tenant(
|
||||
session,
|
||||
finding_id,
|
||||
principal.tenant_id,
|
||||
for_update=True,
|
||||
)
|
||||
_assert_expected_revision(finding.revision, payload.expected_revision)
|
||||
try:
|
||||
result = cleanup_orphan_finding(
|
||||
session,
|
||||
finding,
|
||||
user_id=principal.user.id,
|
||||
dry_run=payload.dry_run,
|
||||
)
|
||||
_audit_integrity_action(session, principal, result)
|
||||
session.commit()
|
||||
return _action_response(result)
|
||||
except (FileStorageError, StorageBackendError) as exc:
|
||||
session.rollback()
|
||||
raise _integrity_http_error(exc) from exc
|
||||
|
||||
|
||||
def _scan_for_tenant(
|
||||
session: Session,
|
||||
scan_id: str,
|
||||
tenant_id: str,
|
||||
*,
|
||||
for_update: bool = False,
|
||||
) -> FileIntegrityScan:
|
||||
query = session.query(FileIntegrityScan).filter(FileIntegrityScan.id == scan_id)
|
||||
if for_update:
|
||||
query = query.populate_existing().with_for_update()
|
||||
scan = query.one_or_none()
|
||||
if scan is None or scan.tenant_id != tenant_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="Integrity scan not found",
|
||||
)
|
||||
return scan
|
||||
|
||||
|
||||
def _finding_for_tenant(
|
||||
session: Session,
|
||||
finding_id: str,
|
||||
tenant_id: str,
|
||||
*,
|
||||
for_update: bool = False,
|
||||
) -> FileIntegrityFinding:
|
||||
query = session.query(FileIntegrityFinding).filter(
|
||||
FileIntegrityFinding.id == finding_id
|
||||
)
|
||||
if for_update:
|
||||
query = query.populate_existing().with_for_update()
|
||||
finding = query.one_or_none()
|
||||
if finding is None or finding.tenant_id != tenant_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="Integrity finding not found",
|
||||
)
|
||||
return finding
|
||||
|
||||
|
||||
def _assert_expected_revision(current: int, expected: int) -> None:
|
||||
if current != expected:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=(
|
||||
"The integrity record changed after it was loaded; reload before "
|
||||
"performing this action."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _audit_integrity_action(session, principal, result) -> None:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=f"files.integrity.{result.action}",
|
||||
object_type="file_integrity_finding",
|
||||
object_id=result.finding.id,
|
||||
details={
|
||||
"scan_id": result.finding.scan_id,
|
||||
"kind": result.finding.kind,
|
||||
"dry_run": result.dry_run,
|
||||
"changed": result.changed,
|
||||
"storage_key_sha256": hashlib.sha256(
|
||||
result.finding.storage_key.encode("utf-8")
|
||||
).hexdigest(),
|
||||
"inspection": result.inspection.kind
|
||||
if result.inspection
|
||||
else None,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _scan_response(scan: FileIntegrityScan) -> FileIntegrityScanResponse:
|
||||
return FileIntegrityScanResponse(
|
||||
id=scan.id,
|
||||
tenant_id=scan.tenant_id,
|
||||
storage_backend=scan.storage_backend,
|
||||
storage_prefix=scan.storage_prefix,
|
||||
status=scan.status,
|
||||
revision=scan.revision,
|
||||
phase=scan.phase,
|
||||
verify_checksums=scan.verify_checksums,
|
||||
batch_size=scan.batch_size,
|
||||
scanned_blob_count=scan.scanned_blob_count,
|
||||
verified_blob_count=scan.verified_blob_count,
|
||||
quarantined_blob_count=scan.quarantined_blob_count,
|
||||
scanned_object_count=scan.scanned_object_count,
|
||||
orphan_object_count=scan.orphan_object_count,
|
||||
created_by_user_id=scan.created_by_user_id,
|
||||
started_at=scan.started_at.isoformat() if scan.started_at else None,
|
||||
completed_at=scan.completed_at.isoformat()
|
||||
if scan.completed_at
|
||||
else None,
|
||||
last_error=scan.last_error,
|
||||
created_at=scan.created_at.isoformat(),
|
||||
updated_at=scan.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
def _finding_response(
|
||||
finding: FileIntegrityFinding,
|
||||
) -> FileIntegrityFindingResponse:
|
||||
return FileIntegrityFindingResponse(
|
||||
id=finding.id,
|
||||
scan_id=finding.scan_id,
|
||||
tenant_id=finding.tenant_id,
|
||||
kind=finding.kind,
|
||||
state=finding.state,
|
||||
revision=finding.revision,
|
||||
blob_id=finding.blob_id,
|
||||
storage_key=finding.storage_key,
|
||||
expected_size_bytes=finding.expected_size_bytes,
|
||||
observed_size_bytes=finding.observed_size_bytes,
|
||||
expected_checksum_sha256=finding.expected_checksum_sha256,
|
||||
observed_checksum_sha256=finding.observed_checksum_sha256,
|
||||
resolved_at=finding.resolved_at.isoformat()
|
||||
if finding.resolved_at
|
||||
else None,
|
||||
resolved_by_user_id=finding.resolved_by_user_id,
|
||||
created_at=finding.created_at.isoformat(),
|
||||
updated_at=finding.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
def _action_response(result) -> FileIntegrityActionResponse:
|
||||
return FileIntegrityActionResponse(
|
||||
action=result.action,
|
||||
changed=result.changed,
|
||||
dry_run=result.dry_run,
|
||||
finding=_finding_response(result.finding),
|
||||
inspection_kind=result.inspection.kind if result.inspection else None,
|
||||
inspection_valid=result.inspection.valid if result.inspection else None,
|
||||
)
|
||||
|
||||
|
||||
def _integrity_http_error(exc: Exception) -> HTTPException:
|
||||
if isinstance(exc, FileStorageError):
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=str(exc),
|
||||
)
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="The configured file storage backend could not complete the integrity operation",
|
||||
)
|
||||
@@ -0,0 +1,297 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_connector_space_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileAssetResponse,
|
||||
FileBlobGcRequest,
|
||||
FileBlobGcResponse,
|
||||
FileConnectorSpaceResponse,
|
||||
FileFolderDeleteRequest,
|
||||
FileFolderRestoreResponse,
|
||||
FileLifecycleUpdateRequest,
|
||||
FilePurgeExecuteRequest,
|
||||
FilePurgePreviewItem,
|
||||
FilePurgePreviewRequest,
|
||||
FilePurgePreviewResponse,
|
||||
FilePurgeResponse,
|
||||
FileRestoreResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.lifecycle import (
|
||||
execute_asset_purge,
|
||||
garbage_collect_unreferenced_blobs,
|
||||
get_asset_for_lifecycle,
|
||||
preview_asset_purge,
|
||||
restore_asset,
|
||||
restore_connector_space,
|
||||
restore_folder,
|
||||
set_asset_lifecycle,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files-lifecycle"])
|
||||
|
||||
|
||||
@router.patch("/{file_id}/lifecycle", response_model=FileAssetResponse)
|
||||
def update_file_lifecycle(
|
||||
file_id: str,
|
||||
payload: FileLifecycleUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:retention")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_lifecycle(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
set_asset_lifecycle(
|
||||
session,
|
||||
asset,
|
||||
retained_until=payload.retained_until,
|
||||
legal_hold=payload.legal_hold,
|
||||
reason=payload.reason,
|
||||
expected_revision=payload.expected_revision,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.lifecycle.updated",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={
|
||||
"retained_until": (
|
||||
asset.retained_until.isoformat() if asset.retained_until else None
|
||||
),
|
||||
"legal_hold": asset.legal_hold,
|
||||
"lifecycle_revision": asset.lifecycle_revision,
|
||||
"reason": asset.lifecycle_reason,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _asset_response(session, asset, include_shares=True)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/assets/{file_id}/restore", response_model=FileRestoreResponse)
|
||||
def restore_file(
|
||||
file_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_lifecycle(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
changed = restore_asset(session, asset)
|
||||
if changed:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.file.restored",
|
||||
object_type="file",
|
||||
object_id=asset.id,
|
||||
details={"display_path": asset.display_path},
|
||||
)
|
||||
session.commit()
|
||||
return FileRestoreResponse(restored_count=1 if changed else 0)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/folders/restore", response_model=FileFolderRestoreResponse)
|
||||
def restore_file_folder(
|
||||
payload: FileFolderDeleteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
restored_folders, restored_files = restore_folder(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
user_id=principal.user.id,
|
||||
path=payload.path,
|
||||
recursive=payload.recursive,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.folder.restored",
|
||||
object_type="file_folder",
|
||||
object_id=payload.path,
|
||||
details={
|
||||
"restored_folders": restored_folders,
|
||||
"restored_files": restored_files,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return FileFolderRestoreResponse(
|
||||
restored_folders=restored_folders, restored_files=restored_files
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces/{space_id}/restore",
|
||||
response_model=FileConnectorSpaceResponse,
|
||||
)
|
||||
def restore_file_connector_space(
|
||||
space_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:restore")),
|
||||
):
|
||||
try:
|
||||
space = restore_connector_space(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector_space.restored",
|
||||
object_type="file_connector_space",
|
||||
object_id=space.id,
|
||||
details={"label": space.label},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
|
||||
|
||||
@router.post("/purge/preview", response_model=FilePurgePreviewResponse)
|
||||
def preview_file_purge(
|
||||
payload: FilePurgePreviewRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
preview = preview_asset_purge(
|
||||
session, tenant_id=principal.tenant_id, file_ids=payload.file_ids
|
||||
)
|
||||
return FilePurgePreviewResponse(
|
||||
preview_sha256=preview.preview_sha256,
|
||||
eligible=preview.eligible,
|
||||
items=[
|
||||
FilePurgePreviewItem(
|
||||
file_id=item.file_id,
|
||||
filename=item.filename,
|
||||
lifecycle_revision=item.lifecycle_revision,
|
||||
deleted_at=(item.deleted_at.isoformat() if item.deleted_at else None),
|
||||
retained_until=(
|
||||
item.retained_until.isoformat() if item.retained_until else None
|
||||
),
|
||||
legal_hold=item.legal_hold,
|
||||
blockers=list(item.blockers),
|
||||
blob_ids=list(item.blob_ids),
|
||||
)
|
||||
for item in preview.items
|
||||
],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/purge/execute", response_model=FilePurgeResponse)
|
||||
def execute_file_purge(
|
||||
payload: FilePurgeExecuteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
result = execute_asset_purge(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
file_ids=payload.file_ids,
|
||||
preview_sha256=payload.preview_sha256,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
approval_reference=payload.approval_reference,
|
||||
before_commit=lambda assets: audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.files.purged",
|
||||
object_type="file_batch",
|
||||
object_id=payload.preview_sha256,
|
||||
details={
|
||||
"file_ids": [asset.id for asset in assets],
|
||||
"approval_reference": payload.approval_reference,
|
||||
"preview_sha256": payload.preview_sha256,
|
||||
},
|
||||
),
|
||||
)
|
||||
return FilePurgeResponse(
|
||||
recovery_operation_id=result.recovery_operation_id,
|
||||
status=result.status,
|
||||
replayed=result.replayed,
|
||||
purged_files=result.purged_files,
|
||||
released_blobs=result.released_blobs,
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/purge/blobs", response_model=FileBlobGcResponse)
|
||||
def garbage_collect_file_blobs(
|
||||
payload: FileBlobGcRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:purge")),
|
||||
):
|
||||
try:
|
||||
result = garbage_collect_unreferenced_blobs(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
limit=payload.limit,
|
||||
approval_reference=payload.approval_reference,
|
||||
)
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.blobs.garbage_collected",
|
||||
object_type="file_blob_batch",
|
||||
object_id=principal.tenant_id,
|
||||
details={
|
||||
"inspected_blobs": result.inspected_blobs,
|
||||
"deleted_blobs": result.deleted_blobs,
|
||||
"unresolved_operation_ids": list(result.unresolved_operation_ids),
|
||||
"approval_reference": payload.approval_reference,
|
||||
},
|
||||
commit=True,
|
||||
)
|
||||
return FileBlobGcResponse(
|
||||
inspected_blobs=result.inspected_blobs,
|
||||
deleted_blobs=result.deleted_blobs,
|
||||
unresolved_operation_ids=list(result.unresolved_operation_ids),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,194 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileDeltaResponse,
|
||||
FileListResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.files import (
|
||||
count_assets_for_user,
|
||||
list_assets_for_user,
|
||||
list_assets_for_user_window,
|
||||
list_recent_assets_for_user,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_list_response,
|
||||
_ensure_campaign_file_access,
|
||||
_ensure_list_owner_access,
|
||||
_is_admin,
|
||||
)
|
||||
from govoplan_files.backend.services.list_queries import (
|
||||
FILES_LIST_CURSOR_SCOPE,
|
||||
_cursor_page_size,
|
||||
_file_cursor_values,
|
||||
_files_delta_response,
|
||||
_files_delta_watermark,
|
||||
_files_list_fingerprint,
|
||||
_full_file_delta_response,
|
||||
_next_file_list_cursor,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/delta", response_model=FileDeltaResponse)
|
||||
def files_delta(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
since: str | None = None,
|
||||
limit: int = Query(default=500, ge=1, le=1000),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
if since is None:
|
||||
return _full_file_delta_response(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
return _files_delta_response(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
since=since,
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=FileListResponse)
|
||||
def list_files(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
sort: Literal["path", "recent"] = "path",
|
||||
page_size: int | None = Query(default=None, ge=1, le=1000),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, owner_type, owner_id)
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
watermark = _files_delta_watermark(session, principal.tenant_id)
|
||||
total = count_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if sort == "recent":
|
||||
if cursor:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="Recent file projections do not accept a path-order cursor.",
|
||||
)
|
||||
recent_limit = page_size or 25
|
||||
assets = list_recent_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
limit=recent_limit,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
watermark=watermark,
|
||||
)
|
||||
effective_page_size = _cursor_page_size(FILES_LIST_CURSOR_SCOPE, cursor, page_size)
|
||||
if effective_page_size is not None:
|
||||
fingerprint = _files_list_fingerprint(
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
page_size=effective_page_size,
|
||||
)
|
||||
after_display_path, after_updated_at, after_id = _file_cursor_values(
|
||||
cursor, fingerprint=fingerprint
|
||||
)
|
||||
assets, has_more = list_assets_for_user_window(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
page_size=effective_page_size,
|
||||
after_display_path=after_display_path,
|
||||
after_updated_at=after_updated_at,
|
||||
after_id=after_id,
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
cursor=cursor,
|
||||
next_cursor=_next_file_list_cursor(
|
||||
principal,
|
||||
assets,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
page_size=effective_page_size,
|
||||
has_more=has_more,
|
||||
),
|
||||
watermark=watermark,
|
||||
)
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileListResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
total=total,
|
||||
watermark=watermark,
|
||||
)
|
||||
@@ -0,0 +1,330 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.api.v1.schemas import (
|
||||
ReferenceOptionListResponse,
|
||||
ReferenceOptionResponse,
|
||||
)
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.core.references import (
|
||||
access_scope_reference_page,
|
||||
access_scope_reference_provider_available,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import get_registry
|
||||
from govoplan_files.backend.schemas import (
|
||||
BulkFileShareRequest,
|
||||
BulkFileShareResponse,
|
||||
FileShareRequest,
|
||||
FileShareResponse,
|
||||
FileSharesResponse,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
current_file_share_for_target,
|
||||
get_asset_for_share_management,
|
||||
list_file_shares,
|
||||
revoke_file_share,
|
||||
share_file,
|
||||
share_files,
|
||||
)
|
||||
from govoplan_files.backend.route_support import (
|
||||
_file_share_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{file_id}/share-target-options",
|
||||
response_model=ReferenceOptionListResponse,
|
||||
)
|
||||
def search_share_targets(
|
||||
file_id: str,
|
||||
target_type: str,
|
||||
q: str = "",
|
||||
selected: list[str] = Query(default=[]),
|
||||
limit: int = Query(default=50, ge=1, le=200),
|
||||
cursor: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
) -> ReferenceOptionListResponse:
|
||||
if target_type not in {"user", "group"}:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="Share target type must be user or group",
|
||||
)
|
||||
try:
|
||||
get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
registry = get_registry()
|
||||
page = access_scope_reference_page(
|
||||
registry,
|
||||
principal,
|
||||
scope_type=target_type,
|
||||
reference_kind="membership" if target_type == "user" else "group",
|
||||
query=q,
|
||||
selected_values=selected,
|
||||
limit=limit,
|
||||
cursor=cursor,
|
||||
administrative=True,
|
||||
session=session,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return ReferenceOptionListResponse(
|
||||
options=[
|
||||
ReferenceOptionResponse(**option.to_dict()) for option in page.options
|
||||
],
|
||||
provider_available=access_scope_reference_provider_available(registry),
|
||||
next_cursor=page.next_cursor,
|
||||
has_more=page.has_more,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{file_id}/shares", response_model=FileSharesResponse)
|
||||
def list_shares(
|
||||
file_id: str,
|
||||
include_inactive: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileSharesResponse(
|
||||
shares=[
|
||||
_file_share_response(share)
|
||||
for share in list_file_shares(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
]
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/{file_id}/shares", response_model=FileShareResponse)
|
||||
def create_share(
|
||||
file_id: str,
|
||||
payload: FileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
previous = current_file_share_for_target(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
)
|
||||
previous_permission = previous.permission if previous else None
|
||||
previous_expiry = previous.expires_at if previous else None
|
||||
share = share_file(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset=asset,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
expires_at=payload.expires_at,
|
||||
)
|
||||
session.flush()
|
||||
action = _share_audit_action(
|
||||
existed=previous is not None,
|
||||
previous_permission=previous_permission,
|
||||
permission=share.permission,
|
||||
previous_expiry=previous_expiry,
|
||||
expiry=share.expires_at,
|
||||
)
|
||||
if action:
|
||||
_audit_share_change(session, principal, share, action=action)
|
||||
session.commit()
|
||||
return _file_share_response(share)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/bulk-shares", response_model=BulkFileShareResponse)
|
||||
def create_bulk_shares(
|
||||
payload: BulkFileShareRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
file_ids = list(dict.fromkeys(payload.file_ids))
|
||||
assets = [
|
||||
get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
for file_id in file_ids
|
||||
]
|
||||
previous_by_asset = {
|
||||
asset.id: current_file_share_for_target(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
)
|
||||
for asset in assets
|
||||
}
|
||||
previous_values = {
|
||||
asset_id: (
|
||||
share.permission if share else None,
|
||||
share.expires_at if share else None,
|
||||
)
|
||||
for asset_id, share in previous_by_asset.items()
|
||||
}
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
assets=assets,
|
||||
target_type=payload.target_type,
|
||||
target_id=payload.target_id,
|
||||
permission=payload.permission,
|
||||
user_id=principal.user.id,
|
||||
expires_at=payload.expires_at,
|
||||
)
|
||||
session.flush()
|
||||
for share in shares:
|
||||
previous_permission, previous_expiry = previous_values[share.file_asset_id]
|
||||
action = _share_audit_action(
|
||||
existed=previous_by_asset[share.file_asset_id] is not None,
|
||||
previous_permission=previous_permission,
|
||||
permission=share.permission,
|
||||
previous_expiry=previous_expiry,
|
||||
expiry=share.expires_at,
|
||||
)
|
||||
if action:
|
||||
_audit_share_change(session, principal, share, action=action)
|
||||
session.commit()
|
||||
return BulkFileShareResponse(
|
||||
shared_count=len(shares),
|
||||
shares=[_file_share_response(share) for share in shares],
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete("/{file_id}/shares/{share_id}", response_model=FileShareResponse)
|
||||
def revoke_share(
|
||||
file_id: str,
|
||||
share_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:share")),
|
||||
):
|
||||
try:
|
||||
asset = get_asset_for_share_management(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
share, changed = revoke_file_share(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
asset_id=asset.id,
|
||||
share_id=share_id,
|
||||
user_id=principal.user.id,
|
||||
)
|
||||
if changed:
|
||||
_audit_share_change(
|
||||
session, principal, share, action="files.share.revoked"
|
||||
)
|
||||
session.commit()
|
||||
return _file_share_response(share)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
def _share_audit_action(
|
||||
*,
|
||||
existed: bool,
|
||||
previous_permission: str | None,
|
||||
permission: str,
|
||||
previous_expiry: datetime | None,
|
||||
expiry: datetime | None,
|
||||
) -> str | None:
|
||||
if not existed:
|
||||
return "files.share.granted"
|
||||
permission_changed = previous_permission != permission
|
||||
expiry_changed = _normalized_expiry(previous_expiry) != _normalized_expiry(expiry)
|
||||
if permission_changed:
|
||||
return "files.share.changed"
|
||||
if expiry_changed:
|
||||
return "files.share.expiry_changed"
|
||||
return None
|
||||
|
||||
|
||||
def _audit_share_change(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
share,
|
||||
*,
|
||||
action: str,
|
||||
) -> None:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=action,
|
||||
object_type="file_share",
|
||||
object_id=share.id,
|
||||
details={
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
"expires_at": _normalized_expiry(share.expires_at),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _normalized_expiry(value: datetime | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(timezone.utc).isoformat()
|
||||
@@ -0,0 +1,308 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorSpaceCreateRequest,
|
||||
FileConnectorSpaceResponse,
|
||||
FileConnectorSpacesResponse,
|
||||
FileConnectorSpaceUpdateRequest,
|
||||
FileSpaceResponse,
|
||||
FileSpacesResponse,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.access import group_refs_for_ids, user_group_ids
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_spaces import (
|
||||
connector_space_owner_id,
|
||||
create_connector_space,
|
||||
get_connector_space_for_user,
|
||||
list_connector_spaces_for_user,
|
||||
soft_delete_connector_space,
|
||||
update_connector_space,
|
||||
validate_connector_space_write_mode,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
_connector_policy_error,
|
||||
_connector_space_file_space_response,
|
||||
_connector_space_policy_decision,
|
||||
_connector_space_response,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
_record_connector_settings_change,
|
||||
_visible_connector_profile,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.get("/spaces", response_model=FileSpacesResponse)
|
||||
def list_file_spaces(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
spaces = [
|
||||
FileSpaceResponse(
|
||||
id=f"user:{principal.user.id}",
|
||||
label="My files",
|
||||
owner_type="user",
|
||||
owner_id=principal.user.id,
|
||||
description="Files owned by your user account.",
|
||||
)
|
||||
]
|
||||
group_ids = user_group_ids(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
include_admin_groups=_is_admin(principal),
|
||||
)
|
||||
if group_ids:
|
||||
groups = group_refs_for_ids(tenant_id=principal.tenant_id, group_ids=group_ids)
|
||||
spaces.extend(
|
||||
FileSpaceResponse(
|
||||
id=f"group:{group.id}",
|
||||
label=f"{group.name} files",
|
||||
owner_type="group",
|
||||
owner_id=group.id,
|
||||
description="Files owned by this group.",
|
||||
)
|
||||
for group in groups
|
||||
)
|
||||
connector_spaces = list_connector_spaces_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
spaces.extend(
|
||||
_connector_space_file_space_response(space) for space in connector_spaces
|
||||
)
|
||||
return FileSpacesResponse(spaces=spaces)
|
||||
|
||||
|
||||
@router.get("/connector-spaces", response_model=FileConnectorSpacesResponse)
|
||||
def list_file_connector_spaces(
|
||||
owner_type: Literal["user", "group"] | None = None,
|
||||
owner_id: str | None = None,
|
||||
include_inactive: bool = False,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
try:
|
||||
spaces = list_connector_spaces_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive and _is_admin(principal),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
return FileConnectorSpacesResponse(
|
||||
spaces=[_connector_space_response(space) for space in spaces]
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/connector-spaces",
|
||||
response_model=FileConnectorSpaceResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_file_connector_space(
|
||||
payload: FileConnectorSpaceCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
if payload.owner_type == "group" and not payload.owner_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail="owner_id is required for group connector spaces",
|
||||
)
|
||||
target_owner = payload.owner_id or principal.user.id
|
||||
try:
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, payload.connector_profile_id
|
||||
)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
operation="link",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
validate_connector_space_write_mode(profile, read_only=payload.read_only)
|
||||
space = create_connector_space(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
profile=profile,
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
sync_mode=payload.sync_mode,
|
||||
read_only=payload.read_only,
|
||||
metadata=payload.metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
resource_id=space.id,
|
||||
operation="created",
|
||||
principal=principal,
|
||||
tenant_id=space.tenant_id,
|
||||
payload={
|
||||
"owner_type": space.owner_type,
|
||||
"owner_id": connector_space_owner_id(space),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.patch("/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse)
|
||||
def update_file_connector_space(
|
||||
space_id: str,
|
||||
payload: FileConnectorSpaceUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
include_inactive=_is_admin(principal),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
try:
|
||||
if (
|
||||
payload.library_id is not None
|
||||
or payload.remote_path is not None
|
||||
or payload.read_only is not None
|
||||
):
|
||||
profile = _visible_connector_profile(
|
||||
session, principal, space.connector_profile_id
|
||||
)
|
||||
decision = _connector_space_policy_decision(
|
||||
profile,
|
||||
library_id=payload.library_id
|
||||
if payload.library_id is not None
|
||||
else space.library_id,
|
||||
remote_path=payload.remote_path
|
||||
if payload.remote_path is not None
|
||||
else space.remote_path,
|
||||
operation="link",
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
validate_connector_space_write_mode(
|
||||
profile,
|
||||
read_only=(
|
||||
payload.read_only
|
||||
if payload.read_only is not None
|
||||
else space.read_only
|
||||
),
|
||||
)
|
||||
update_connector_space(
|
||||
session,
|
||||
space,
|
||||
user_id=principal.user.id,
|
||||
label=payload.label,
|
||||
library_id=payload.library_id,
|
||||
remote_path=payload.remote_path,
|
||||
sync_mode=payload.sync_mode,
|
||||
read_only=payload.read_only,
|
||||
is_active=payload.is_active,
|
||||
metadata=payload.metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
resource_id=space.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=space.tenant_id,
|
||||
payload={
|
||||
"owner_type": space.owner_type,
|
||||
"owner_id": connector_space_owner_id(space),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/connector-spaces/{space_id}", response_model=FileConnectorSpaceResponse
|
||||
)
|
||||
def delete_file_connector_space(
|
||||
space_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
space = get_connector_space_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
space_id=space_id,
|
||||
include_inactive=True,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
soft_delete_connector_space(
|
||||
session, space, user_id=principal.user.id, is_admin=_is_admin(principal)
|
||||
)
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
resource_id=space.id,
|
||||
operation="deleted",
|
||||
principal=principal,
|
||||
tenant_id=space.tenant_id,
|
||||
payload={
|
||||
"owner_type": space.owner_type,
|
||||
"owner_id": connector_space_owner_id(space),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return _connector_space_response(space)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
@@ -0,0 +1,213 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi.responses import FileResponse
|
||||
from starlette.background import BackgroundTask
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_files.backend.schemas import (
|
||||
ArchiveRequest,
|
||||
PatternMatchResponse,
|
||||
PatternResolveRequest,
|
||||
PatternResolveResponse,
|
||||
RenamePreviewItem,
|
||||
RenameRequest,
|
||||
RenameResponse,
|
||||
TransferRequest,
|
||||
TransferResponse,
|
||||
_conflict_resolutions,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
UnsafeFilePathError,
|
||||
filename_from_path,
|
||||
normalize_logical_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.archives import create_zip_file
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import (
|
||||
get_asset_for_user,
|
||||
list_assets_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.search import resolve_patterns
|
||||
from govoplan_files.backend.storage.transfers import (
|
||||
rename_selection,
|
||||
transfer_selection,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_attachment_disposition,
|
||||
_audit_connector_access,
|
||||
_cleanup_temp_file,
|
||||
_ensure_campaign_file_access,
|
||||
_ensure_list_owner_access,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
|
||||
|
||||
@router.post("/bulk-rename", response_model=RenameResponse)
|
||||
def bulk_rename(
|
||||
payload: RenameRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
plan = rename_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
mode=payload.mode,
|
||||
new_name=payload.new_name,
|
||||
find=payload.find,
|
||||
replacement=payload.replacement,
|
||||
prefix=payload.prefix,
|
||||
suffix=payload.suffix,
|
||||
recursive=payload.recursive,
|
||||
dry_run=payload.dry_run,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if not payload.dry_run:
|
||||
session.commit()
|
||||
return RenameResponse(
|
||||
dry_run=payload.dry_run,
|
||||
items=[
|
||||
RenamePreviewItem(
|
||||
kind=item.kind,
|
||||
id=item.id,
|
||||
file_id=item.id if item.kind == "file" else None,
|
||||
folder_path=item.old_path if item.kind == "folder" else None,
|
||||
old_path=item.old_path,
|
||||
new_path=item.new_path,
|
||||
)
|
||||
for item in plan
|
||||
],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/transfer", response_model=TransferResponse)
|
||||
def transfer_files(
|
||||
payload: TransferRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:organize")),
|
||||
):
|
||||
try:
|
||||
files, folders = transfer_selection(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
operation=payload.operation,
|
||||
file_ids=payload.file_ids,
|
||||
folder_paths=payload.folder_paths,
|
||||
source_owner_type=payload.source_owner_type,
|
||||
source_owner_id=payload.source_owner_id,
|
||||
target_owner_type=payload.target_owner_type,
|
||||
target_owner_id=payload.target_owner_id,
|
||||
target_folder=payload.target_folder,
|
||||
conflict_strategy=payload.conflict_strategy,
|
||||
conflict_resolutions=_conflict_resolutions(payload.conflict_resolutions),
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
session.commit()
|
||||
return TransferResponse(
|
||||
operation=payload.operation, files=files, folders=folders
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/archive.zip")
|
||||
def download_archive(
|
||||
payload: ArchiveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:download")),
|
||||
):
|
||||
try:
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
asset_id=file_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
tmp = tempfile.NamedTemporaryFile(
|
||||
prefix="govoplan-files-", suffix=".zip", delete=False
|
||||
)
|
||||
tmp_path = tmp.name
|
||||
tmp.close()
|
||||
try:
|
||||
create_zip_file(session, assets, tmp_path)
|
||||
except Exception:
|
||||
_cleanup_temp_file(tmp_path)
|
||||
raise
|
||||
_audit_connector_access(session, principal, assets, operation="archive")
|
||||
except FileStorageError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
filename = filename_from_path(
|
||||
normalize_logical_path(payload.filename, fallback_filename="files.zip")
|
||||
)
|
||||
headers = {"Content-Disposition": _attachment_disposition(filename)}
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type="application/zip",
|
||||
headers=headers,
|
||||
background=BackgroundTask(_cleanup_temp_file, tmp_path),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/resolve-patterns", response_model=PatternResolveResponse)
|
||||
def resolve_file_patterns(
|
||||
payload: PatternResolveRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:read")),
|
||||
):
|
||||
_ensure_list_owner_access(session, principal, payload.owner_type, payload.owner_id)
|
||||
_ensure_campaign_file_access(session, principal, payload.campaign_id)
|
||||
try:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=payload.owner_type,
|
||||
owner_id=payload.owner_id,
|
||||
campaign_id=payload.campaign_id,
|
||||
path_prefix=payload.path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
resolved, unmatched = resolve_patterns(
|
||||
assets,
|
||||
payload.patterns,
|
||||
base_path=payload.path_prefix,
|
||||
case_sensitive=payload.case_sensitive,
|
||||
)
|
||||
return PatternResolveResponse(
|
||||
patterns=[
|
||||
PatternMatchResponse(
|
||||
pattern=item.pattern,
|
||||
matches=[_asset_response(session, asset) for asset in item.matches],
|
||||
)
|
||||
for item in resolved
|
||||
],
|
||||
unmatched=[_asset_response(session, asset) for asset in unmatched]
|
||||
if payload.include_unmatched
|
||||
else [],
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
@@ -0,0 +1,487 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Literal
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, UploadFile
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal, require_scope
|
||||
from govoplan_core.security.secrets import (
|
||||
TransientPayloadError,
|
||||
open_transient_payload,
|
||||
seal_transient_payload,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
ArchiveEntryResponse,
|
||||
ArchivePreviewResponse,
|
||||
ConflictResolutionRequest,
|
||||
FileUploadResponse,
|
||||
_conflict_resolutions,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_core.db.session import get_session
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
UnsafeFilePathError,
|
||||
normalize_folder,
|
||||
)
|
||||
from govoplan_files.backend.storage.archives import (
|
||||
archive_format_for_filename,
|
||||
extract_archive_upload,
|
||||
extract_zip_upload,
|
||||
inspect_archive,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorPolicyDenied,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_response,
|
||||
_audit_connector_imports,
|
||||
_cleanup_temp_file,
|
||||
_connector_policy_error,
|
||||
_enforce_connector_policy,
|
||||
_http_error,
|
||||
_is_admin,
|
||||
_read_limited_upload,
|
||||
_source_metadata_from_form,
|
||||
_spool_limited_upload_to_temp,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/files", tags=["files"])
|
||||
_ARCHIVE_PREVIEW_PURPOSE = "files.archive-preview.v1"
|
||||
|
||||
|
||||
def _archive_suffix(filename: str) -> str:
|
||||
lowered = filename.casefold()
|
||||
for suffix in (".tar.bz2", ".tar.gz", ".tar.xz", ".tbz2", ".tgz", ".txz", ".tar", ".zip"):
|
||||
if lowered.endswith(suffix):
|
||||
return suffix
|
||||
return ".archive"
|
||||
|
||||
|
||||
def _archive_sha256(path: str) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as source:
|
||||
while chunk := source.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _archive_selected_paths(value: str) -> list[str]:
|
||||
parsed = json.loads(value)
|
||||
if not isinstance(parsed, list) or not parsed:
|
||||
raise FileStorageError("Select at least one archive file or folder")
|
||||
if len(parsed) > settings.file_archive_max_entries:
|
||||
raise FileStorageError(
|
||||
"Archive selection exceeds the configured entry limit"
|
||||
)
|
||||
selected: list[str] = []
|
||||
for item in parsed:
|
||||
if not isinstance(item, str) or not item.strip():
|
||||
raise FileStorageError("Archive selection contains an invalid path")
|
||||
if len(item) > 4096:
|
||||
raise FileStorageError("Archive selection path is too long")
|
||||
selected.append(item)
|
||||
return selected
|
||||
|
||||
|
||||
def _validate_archive_preview_token(
|
||||
token: str,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
path: str,
|
||||
campaign_id: str | None,
|
||||
archive_format: str,
|
||||
archive_sha256: str,
|
||||
) -> None:
|
||||
payload = open_transient_payload(
|
||||
token,
|
||||
ttl_seconds=settings.file_archive_preview_ttl_seconds,
|
||||
)
|
||||
expected = {
|
||||
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||
"tenant_id": tenant_id,
|
||||
"user_id": user_id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"path": path,
|
||||
"campaign_id": campaign_id or "",
|
||||
"archive_format": archive_format,
|
||||
}
|
||||
if any(payload.get(key) != value for key, value in expected.items()):
|
||||
raise FileStorageError(
|
||||
"Archive preview does not match this upload destination"
|
||||
)
|
||||
token_digest = payload.get("archive_sha256")
|
||||
if not isinstance(token_digest, str) or not hmac.compare_digest(
|
||||
token_digest, archive_sha256
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Archive contents changed after preview; preview it again"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/archive-preview", response_model=ArchivePreviewResponse)
|
||||
def preview_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
password: str | None = Form(default=None),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
try:
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
inspection = inspect_archive(
|
||||
archive_path,
|
||||
filename=filename,
|
||||
password=password,
|
||||
max_entries=settings.file_archive_max_entries,
|
||||
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
normalized_path = normalize_folder(path)
|
||||
preview_token = seal_transient_payload(
|
||||
{
|
||||
"purpose": _ARCHIVE_PREVIEW_PURPOSE,
|
||||
"tenant_id": principal.tenant_id,
|
||||
"user_id": principal.user.id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": target_owner,
|
||||
"path": normalized_path,
|
||||
"campaign_id": campaign_id or "",
|
||||
"archive_format": archive_format,
|
||||
"archive_sha256": digest,
|
||||
}
|
||||
)
|
||||
expires_at = datetime.now(UTC) + timedelta(
|
||||
seconds=settings.file_archive_preview_ttl_seconds
|
||||
)
|
||||
return ArchivePreviewResponse(
|
||||
preview_token=preview_token,
|
||||
archive_format=inspection.archive_format,
|
||||
entries=[
|
||||
ArchiveEntryResponse(
|
||||
path=entry.path,
|
||||
kind=entry.kind,
|
||||
size_bytes=entry.size_bytes,
|
||||
compressed_size_bytes=entry.compressed_size_bytes,
|
||||
encrypted=entry.encrypted,
|
||||
)
|
||||
for entry in inspection.entries
|
||||
],
|
||||
file_count=inspection.file_count,
|
||||
directory_count=inspection.directory_count,
|
||||
expanded_size_bytes=inspection.expanded_size_bytes,
|
||||
compressed_size_bytes=inspection.compressed_size_bytes,
|
||||
requires_password=inspection.requires_password,
|
||||
password_verified=inspection.password_verified,
|
||||
expires_at=expires_at.isoformat(),
|
||||
)
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
|
||||
|
||||
@router.post("/archive-confirm", response_model=FileUploadResponse)
|
||||
def confirm_archive_upload(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
preview_token: str = Form(...),
|
||||
selected_paths_json: str = Form(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
password: str | None = Form(default=None),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||
default="reject"
|
||||
),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
source_provenance_json: str | None = Form(default=None),
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
archive_path: str | None = None
|
||||
filename = file.filename or "archive"
|
||||
try:
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json)
|
||||
if conflict_resolutions_json
|
||||
else []
|
||||
)
|
||||
upload_resolutions = _conflict_resolutions(
|
||||
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||
)
|
||||
selected_paths = _archive_selected_paths(selected_paths_json)
|
||||
_enforce_connector_policy(
|
||||
source_provenance_json,
|
||||
connector_policy_json,
|
||||
operation="import",
|
||||
)
|
||||
metadata = _source_metadata_from_form(
|
||||
source_provenance_json, source_revision
|
||||
)
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
normalized_path = normalize_folder(path)
|
||||
archive_path = _spool_limited_upload_to_temp(
|
||||
file,
|
||||
max_bytes=settings.file_upload_zip_max_bytes,
|
||||
suffix=_archive_suffix(filename),
|
||||
)
|
||||
digest = _archive_sha256(archive_path)
|
||||
_validate_archive_preview_token(
|
||||
preview_token,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
path=normalized_path,
|
||||
campaign_id=campaign_id,
|
||||
archive_format=archive_format,
|
||||
archive_sha256=digest,
|
||||
)
|
||||
extracted = extract_archive_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
archive_data=archive_path,
|
||||
filename=filename,
|
||||
folder=normalized_path,
|
||||
campaign_id=campaign_id,
|
||||
selected_paths=selected_paths,
|
||||
password=password,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_entries=settings.file_archive_max_entries,
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_expanded_bytes=settings.file_archive_max_expanded_bytes,
|
||||
max_expansion_ratio=settings.file_archive_max_expansion_ratio,
|
||||
)
|
||||
uploaded_assets = [item.asset for item in extracted]
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
session.commit()
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (
|
||||
FileStorageError,
|
||||
TransientPayloadError,
|
||||
UnsafeFilePathError,
|
||||
ValueError,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if archive_path:
|
||||
_cleanup_temp_file(archive_path)
|
||||
|
||||
|
||||
@router.post("/upload", response_model=FileUploadResponse)
|
||||
def upload_files(
|
||||
files: list[UploadFile] = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
unpack_zip: bool = Form(default=False),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||
default="reject"
|
||||
),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
source_provenance_json: str | None = Form(default=None),
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
uploaded_assets: list[FileAsset] = []
|
||||
try:
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
)
|
||||
upload_resolutions = _conflict_resolutions(
|
||||
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||
)
|
||||
_enforce_connector_policy(
|
||||
source_provenance_json, connector_policy_json, operation="import"
|
||||
)
|
||||
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
|
||||
for upload in files:
|
||||
filename = upload.filename or "file"
|
||||
content_type = upload.content_type or None
|
||||
upload_limit = (
|
||||
settings.file_upload_zip_max_bytes
|
||||
if unpack_zip and filename.lower().endswith(".zip")
|
||||
else settings.file_upload_max_bytes
|
||||
)
|
||||
if unpack_zip and filename.lower().endswith(".zip"):
|
||||
zip_path = _spool_limited_upload_to_temp(
|
||||
upload, max_bytes=upload_limit, suffix=".zip"
|
||||
)
|
||||
try:
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=zip_path,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
finally:
|
||||
_cleanup_temp_file(zip_path)
|
||||
uploaded_assets.extend(item.asset for item in extracted)
|
||||
continue
|
||||
data = _read_limited_upload(upload, max_bytes=upload_limit)
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder=path,
|
||||
content_type=content_type,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
uploaded_assets.append(stored.asset)
|
||||
_audit_connector_imports(session, principal, uploaded_assets)
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, asset, include_shares=True)
|
||||
for asset in uploaded_assets
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post("/upload-zip", response_model=FileUploadResponse)
|
||||
def upload_zip(
|
||||
file: UploadFile = FastAPIFile(...),
|
||||
owner_type: Literal["user", "group"] = Form(default="user"),
|
||||
owner_id: str | None = Form(default=None),
|
||||
path: str = Form(default=""),
|
||||
campaign_id: str | None = Form(default=None),
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename"] = Form(
|
||||
default="reject"
|
||||
),
|
||||
conflict_resolutions_json: str | None = Form(default=None),
|
||||
source_provenance_json: str | None = Form(default=None),
|
||||
source_revision: str | None = Form(default=None),
|
||||
connector_policy_json: str | None = Form(default=None),
|
||||
encryption_vault_id: str | None = Form(default=None),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_scope("files:file:upload")),
|
||||
):
|
||||
target_owner = owner_id or principal.user.id
|
||||
zip_path: str | None = None
|
||||
try:
|
||||
raw_resolutions = (
|
||||
json.loads(conflict_resolutions_json) if conflict_resolutions_json else []
|
||||
)
|
||||
upload_resolutions = _conflict_resolutions(
|
||||
[ConflictResolutionRequest(**item) for item in raw_resolutions]
|
||||
)
|
||||
_enforce_connector_policy(
|
||||
source_provenance_json, connector_policy_json, operation="import"
|
||||
)
|
||||
metadata = _source_metadata_from_form(source_provenance_json, source_revision)
|
||||
zip_path = _spool_limited_upload_to_temp(
|
||||
file, max_bytes=settings.file_upload_zip_max_bytes, suffix=".zip"
|
||||
)
|
||||
extracted = extract_zip_upload(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=target_owner,
|
||||
user_id=principal.user.id,
|
||||
zip_data=zip_path,
|
||||
folder=path,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=upload_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=_is_admin(principal),
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_file_bytes=settings.file_upload_max_bytes,
|
||||
max_total_bytes=settings.file_upload_zip_max_bytes,
|
||||
)
|
||||
_audit_connector_imports(session, principal, [item.asset for item in extracted])
|
||||
session.commit()
|
||||
except ConnectorPolicyDenied as exc:
|
||||
session.rollback()
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (FileStorageError, UnsafeFilePathError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
finally:
|
||||
if zip_path:
|
||||
_cleanup_temp_file(zip_path)
|
||||
return FileUploadResponse(
|
||||
files=[
|
||||
_asset_response(session, item.asset, include_shares=True)
|
||||
for item in extracted
|
||||
]
|
||||
)
|
||||
@@ -1,29 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from govoplan_core.core.runtime import ModuleRuntimeState
|
||||
|
||||
_runtime_settings: object | None = None
|
||||
_runtime = ModuleRuntimeState("Files")
|
||||
|
||||
|
||||
def configure_runtime(*, settings: object | None = None) -> None:
|
||||
global _runtime_settings
|
||||
if settings is not None:
|
||||
_runtime_settings = settings
|
||||
|
||||
|
||||
def get_settings() -> object:
|
||||
if _runtime_settings is not None:
|
||||
return _runtime_settings
|
||||
try:
|
||||
from govoplan_core.settings import settings as legacy_settings
|
||||
except ModuleNotFoundError as exc:
|
||||
raise RuntimeError("GovOPlaN Files runtime settings are not configured") from exc
|
||||
return legacy_settings
|
||||
|
||||
|
||||
class SettingsProxy:
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(get_settings(), name)
|
||||
|
||||
|
||||
settings = SettingsProxy()
|
||||
configure_runtime = _runtime.configure_runtime
|
||||
get_registry = _runtime.get_registry
|
||||
get_settings = _runtime.get_settings
|
||||
settings = _runtime.settings
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution
|
||||
|
||||
|
||||
@@ -13,19 +15,168 @@ class FileSpaceResponse(BaseModel):
|
||||
owner_type: Literal["user", "group"]
|
||||
owner_id: str
|
||||
description: str | None = None
|
||||
space_type: Literal["managed", "connector"] = "managed"
|
||||
connector_space_id: str | None = None
|
||||
connector_profile_id: str | None = None
|
||||
provider: str | None = None
|
||||
library_id: str | None = None
|
||||
remote_path: str | None = None
|
||||
sync_mode: str | None = None
|
||||
read_only: bool = False
|
||||
|
||||
|
||||
class FileSpacesResponse(BaseModel):
|
||||
spaces: list[FileSpaceResponse]
|
||||
|
||||
|
||||
class FileConnectorSpaceCreateRequest(BaseModel):
|
||||
owner_type: Literal["user", "group"] = "user"
|
||||
owner_id: str | None = None
|
||||
label: str
|
||||
connector_profile_id: str
|
||||
library_id: str | None = None
|
||||
remote_path: str = ""
|
||||
sync_mode: Literal["manual"] = "manual"
|
||||
read_only: bool = True
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorSpaceUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
library_id: str | None = None
|
||||
remote_path: str | None = None
|
||||
sync_mode: Literal["manual"] | None = None
|
||||
read_only: bool | None = None
|
||||
is_active: bool | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorSpaceResponse(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
owner_type: Literal["user", "group"]
|
||||
owner_id: str
|
||||
label: str
|
||||
connector_profile_id: str
|
||||
provider: str
|
||||
library_id: str | None = None
|
||||
remote_path: str = ""
|
||||
sync_mode: str
|
||||
read_only: bool
|
||||
is_active: bool
|
||||
created_at: str
|
||||
updated_at: str
|
||||
deleted_at: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorSpacesResponse(BaseModel):
|
||||
spaces: list[FileConnectorSpaceResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileShareResponse(BaseModel):
|
||||
id: str
|
||||
file_asset_id: str
|
||||
target_type: str
|
||||
target_id: str
|
||||
permission: str
|
||||
created_by_user_id: str | None = None
|
||||
created_at: str
|
||||
expires_at: str | None = None
|
||||
revoked_at: str | None = None
|
||||
revoked_by_user_id: str | None = None
|
||||
active: bool
|
||||
|
||||
|
||||
class FileSharesResponse(BaseModel):
|
||||
shares: list[FileShareResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileIntegrityScanCreateRequest(BaseModel):
|
||||
verify_checksums: bool = True
|
||||
batch_size: int = Field(default=100, ge=1, le=1000)
|
||||
|
||||
|
||||
class FileIntegrityScanResponse(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
storage_backend: str
|
||||
storage_prefix: str
|
||||
status: str
|
||||
revision: int
|
||||
phase: str
|
||||
verify_checksums: bool
|
||||
batch_size: int
|
||||
scanned_blob_count: int
|
||||
verified_blob_count: int
|
||||
quarantined_blob_count: int
|
||||
scanned_object_count: int
|
||||
orphan_object_count: int
|
||||
created_by_user_id: str | None = None
|
||||
started_at: str | None = None
|
||||
completed_at: str | None = None
|
||||
last_error: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class FileIntegrityScansResponse(BaseModel):
|
||||
scans: list[FileIntegrityScanResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileIntegrityFindingResponse(BaseModel):
|
||||
id: str
|
||||
scan_id: str
|
||||
tenant_id: str
|
||||
kind: str
|
||||
state: str
|
||||
revision: int
|
||||
blob_id: str | None = None
|
||||
storage_key: str
|
||||
expected_size_bytes: int | None = None
|
||||
observed_size_bytes: int | None = None
|
||||
expected_checksum_sha256: str | None = None
|
||||
observed_checksum_sha256: str | None = None
|
||||
resolved_at: str | None = None
|
||||
resolved_by_user_id: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class FileIntegrityFindingsResponse(BaseModel):
|
||||
findings: list[FileIntegrityFindingResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileIntegrityActionRequest(BaseModel):
|
||||
dry_run: bool = True
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileIntegrityScanRunRequest(BaseModel):
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileIntegrityActionResponse(BaseModel):
|
||||
action: str
|
||||
changed: bool
|
||||
dry_run: bool
|
||||
finding: FileIntegrityFindingResponse
|
||||
inspection_kind: str | None = None
|
||||
inspection_valid: bool | None = None
|
||||
|
||||
|
||||
class FileSourceProvenance(BaseModel):
|
||||
source_type: str | None = None
|
||||
connector_id: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
external_path: str | None = None
|
||||
external_url: str | None = None
|
||||
revision: str | None = None
|
||||
revision_label: str | None = None
|
||||
observed_at: str | None = None
|
||||
imported_at: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileAssetResponse(BaseModel):
|
||||
@@ -43,8 +194,14 @@ class FileAssetResponse(BaseModel):
|
||||
created_at: str
|
||||
updated_at: str
|
||||
deleted_at: str | None = None
|
||||
retained_until: str | None = None
|
||||
legal_hold: bool = False
|
||||
lifecycle_revision: int = 1
|
||||
lifecycle_reason: str | None = None
|
||||
audit_relevant: bool = False
|
||||
metadata: dict[str, Any] | None = None
|
||||
source_provenance: FileSourceProvenance | None = None
|
||||
source_revision: str | None = None
|
||||
shares: list[FileShareResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
@@ -61,6 +218,9 @@ class FileFolderResponse(BaseModel):
|
||||
|
||||
class FileFoldersResponse(BaseModel):
|
||||
folders: list[FileFolderResponse]
|
||||
cursor: str | None = None
|
||||
next_cursor: str | None = None
|
||||
watermark: str | None = None
|
||||
|
||||
|
||||
class FileFolderCreateRequest(BaseModel):
|
||||
@@ -83,12 +243,376 @@ class FileFolderDeleteResponse(BaseModel):
|
||||
|
||||
class FileListResponse(BaseModel):
|
||||
files: list[FileAssetResponse]
|
||||
total: int
|
||||
cursor: str | None = None
|
||||
next_cursor: str | None = None
|
||||
watermark: str | None = None
|
||||
|
||||
|
||||
class FileDeltaResponse(BaseModel):
|
||||
files: list[FileAssetResponse] = Field(default_factory=list)
|
||||
folders: list[FileFolderResponse] = Field(default_factory=list)
|
||||
deleted: list[DeltaDeletedItem] = Field(default_factory=list)
|
||||
watermark: str | None = None
|
||||
has_more: bool = False
|
||||
full: bool = False
|
||||
|
||||
|
||||
class FileUploadResponse(BaseModel):
|
||||
files: list[FileAssetResponse]
|
||||
|
||||
|
||||
class ArchiveEntryResponse(BaseModel):
|
||||
path: str
|
||||
kind: Literal["file", "directory"]
|
||||
size_bytes: int
|
||||
compressed_size_bytes: int | None = None
|
||||
encrypted: bool = False
|
||||
|
||||
|
||||
class ArchivePreviewResponse(BaseModel):
|
||||
preview_token: str
|
||||
archive_format: str
|
||||
entries: list[ArchiveEntryResponse] = Field(default_factory=list)
|
||||
file_count: int
|
||||
directory_count: int
|
||||
expanded_size_bytes: int
|
||||
compressed_size_bytes: int
|
||||
requires_password: bool
|
||||
password_verified: bool
|
||||
expires_at: str
|
||||
|
||||
|
||||
class FileConnectorPolicySource(BaseModel):
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||
scope_id: str | None = None
|
||||
label: str | None = None
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorPolicyEvaluateRequest(BaseModel):
|
||||
source_provenance: FileSourceProvenance
|
||||
operation: str = "access"
|
||||
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorPolicyEvaluateResponse(BaseModel):
|
||||
decision: dict[str, Any]
|
||||
|
||||
|
||||
class FileConnectorPolicyUpdateRequest(BaseModel):
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorPolicyStepResponse(BaseModel):
|
||||
scope_type: str
|
||||
scope_id: str | None = None
|
||||
path: str
|
||||
label: str
|
||||
applied_fields: list[str] = Field(default_factory=list)
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorPolicyResponse(BaseModel):
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"]
|
||||
scope_id: str | None = None
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
effective_policy: dict[str, Any] | None = None
|
||||
parent_policy: dict[str, Any] | None = None
|
||||
effective_policy_sources: list[FileConnectorPolicyStepResponse] = Field(default_factory=list)
|
||||
parent_policy_sources: list[FileConnectorPolicyStepResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorProfileResponse(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: str
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||
scope_id: str | None = None
|
||||
source_path: str
|
||||
credential_profile_id: str | None = None
|
||||
credential_profile_label: str | None = None
|
||||
credential_mode: str = "none"
|
||||
credential_source: str | None = None
|
||||
credentials_configured: bool = False
|
||||
username: str | None = None
|
||||
capabilities: list[str] = Field(default_factory=list)
|
||||
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
source_kind: str = "settings"
|
||||
|
||||
|
||||
class FileConnectorProfilesResponse(BaseModel):
|
||||
profiles: list[FileConnectorProfileResponse]
|
||||
|
||||
|
||||
class FileConnectorCredentialResponse(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: str | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "system"
|
||||
scope_id: str | None = None
|
||||
source_path: str
|
||||
credential_mode: str = "none"
|
||||
credential_secret_source: str | None = None
|
||||
credentials_configured: bool = False
|
||||
username: str | None = None
|
||||
policy_sources: list[FileConnectorPolicySource] = Field(default_factory=list)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
source_kind: str = "database"
|
||||
|
||||
|
||||
class FileConnectorCredentialsResponse(BaseModel):
|
||||
credentials: list[FileConnectorCredentialResponse]
|
||||
|
||||
|
||||
class FileConnectorSettingsDeltaResponse(BaseModel):
|
||||
profiles: list[FileConnectorProfileResponse] = Field(default_factory=list)
|
||||
credentials: list[FileConnectorCredentialResponse] = Field(default_factory=list)
|
||||
spaces: list[FileConnectorSpaceResponse] = Field(default_factory=list)
|
||||
policy: FileConnectorPolicyResponse | None = None
|
||||
changed_sections: list[str] = Field(default_factory=list)
|
||||
deleted: list[DeltaDeletedItem] = Field(default_factory=list)
|
||||
watermark: str | None = None
|
||||
has_more: bool = False
|
||||
full: bool = False
|
||||
|
||||
|
||||
class FileConnectorProfileCredentialsRequest(BaseModel):
|
||||
username: str | None = None
|
||||
password: str | None = None
|
||||
token: str | None = None
|
||||
password_env: str | None = None
|
||||
token_env: str | None = None
|
||||
secret_ref: str | None = None
|
||||
|
||||
|
||||
class FileConnectorDiscoveryCandidate(BaseModel):
|
||||
endpoint_url: str
|
||||
status: Literal["failed", "usable", "found", "credentials_rejected"]
|
||||
message: str
|
||||
|
||||
|
||||
class FileConnectorDiscoveryRequest(BaseModel):
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||
endpoint_url: str
|
||||
base_path: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
require_valid_credentials: bool = False
|
||||
|
||||
|
||||
class FileConnectorDiscoveryResponse(BaseModel):
|
||||
provider: str
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
status: Literal["usable", "found", "credentials_rejected", "not_found", "unsupported"]
|
||||
message: str
|
||||
candidates: list[FileConnectorDiscoveryCandidate] = Field(default_factory=list)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorProfileCreateRequest(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||
scope_id: str | None = None
|
||||
credential_profile_id: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||
capabilities: list[str] = Field(default_factory=lambda: ["browse", "import", "sync"])
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorProfileUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool | None = None
|
||||
credential_profile_id: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||
clear_password: bool = False
|
||||
clear_token: bool = False
|
||||
capabilities: list[str] | None = None
|
||||
policy: dict[str, Any] | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorCredentialCreateRequest(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||
scope_id: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
credentials: FileConnectorProfileCredentialsRequest = Field(default_factory=FileConnectorProfileCredentialsRequest)
|
||||
policy: dict[str, Any] = Field(default_factory=dict)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorCredentialUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
enabled: bool | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||
clear_password: bool = False
|
||||
clear_token: bool = False
|
||||
policy: dict[str, Any] | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorProviderResponse(BaseModel):
|
||||
provider: str
|
||||
label: str
|
||||
protocol: str
|
||||
implemented: bool
|
||||
installed: bool
|
||||
browse_supported: bool
|
||||
import_supported: bool
|
||||
write_supported: bool = False
|
||||
optional_dependency: str | None = None
|
||||
permission_model: str
|
||||
sync_strategy: str
|
||||
conflict_strategy: str
|
||||
preview_strategy: str
|
||||
audit_events: list[str] = Field(default_factory=list)
|
||||
notes: str | None = None
|
||||
|
||||
|
||||
class FileConnectorProvidersResponse(BaseModel):
|
||||
providers: list[FileConnectorProviderResponse]
|
||||
|
||||
|
||||
class FileConnectorBrowseItem(BaseModel):
|
||||
kind: Literal["library", "folder", "file"]
|
||||
name: str
|
||||
path: str
|
||||
external_id: str | None = None
|
||||
external_url: str | None = None
|
||||
size_bytes: int | None = None
|
||||
content_type: str | None = None
|
||||
modified_at: str | None = None
|
||||
etag: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorBrowseResponse(BaseModel):
|
||||
profile_id: str
|
||||
provider: str
|
||||
path: str = ""
|
||||
library_id: str | None = None
|
||||
read_only: bool = True
|
||||
next_continuation_token: str | None = None
|
||||
has_more: bool = False
|
||||
decision: dict[str, Any]
|
||||
items: list[FileConnectorBrowseItem]
|
||||
|
||||
|
||||
class FileConnectorImportRequest(BaseModel):
|
||||
library_id: str
|
||||
path: str
|
||||
owner_type: Literal["user", "group"] = "user"
|
||||
owner_id: str | None = None
|
||||
target_folder: str | None = None
|
||||
target_path: str | None = None
|
||||
campaign_id: str | None = None
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename", "skip"] = "reject"
|
||||
source_revision: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorSyncResponse(BaseModel):
|
||||
file: FileAssetResponse
|
||||
action: Literal["created", "updated", "unchanged"]
|
||||
previous_version_id: str | None = None
|
||||
current_version_id: str
|
||||
|
||||
|
||||
class FileConnectorFolderSyncRequest(BaseModel):
|
||||
path: str = ""
|
||||
target_folder: str | None = None
|
||||
recursive: bool = True
|
||||
conflict_strategy: Literal["reject", "overwrite", "rename", "skip"] = "skip"
|
||||
max_files: int = Field(default=100, ge=1, le=500)
|
||||
max_depth: int = Field(default=12, ge=0, le=50)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FileConnectorFolderSyncItemResponse(BaseModel):
|
||||
source_path: str
|
||||
target_path: str | None = None
|
||||
action: Literal[
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"skipped",
|
||||
"conflict",
|
||||
"policy_denied",
|
||||
"failed",
|
||||
]
|
||||
file: FileAssetResponse | None = None
|
||||
previous_version_id: str | None = None
|
||||
current_version_id: str | None = None
|
||||
source_revision: str | None = None
|
||||
detail: str | None = None
|
||||
policy_decision: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class FileConnectorFolderSyncSummary(BaseModel):
|
||||
discovered: int = 0
|
||||
created: int = 0
|
||||
updated: int = 0
|
||||
unchanged: int = 0
|
||||
skipped: int = 0
|
||||
conflicts: int = 0
|
||||
policy_denied: int = 0
|
||||
failed: int = 0
|
||||
|
||||
|
||||
class FileConnectorFolderSyncResponse(BaseModel):
|
||||
connector_space_id: str
|
||||
connector_profile_id: str
|
||||
provider: str
|
||||
remote_path: str
|
||||
target_folder: str
|
||||
recursive: bool
|
||||
truncated: bool = False
|
||||
summary: FileConnectorFolderSyncSummary
|
||||
items: list[FileConnectorFolderSyncItemResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FileConnectorWriteRequest(BaseModel):
|
||||
file_id: str
|
||||
remote_path: str
|
||||
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||
expected_revision: str | None = Field(default=None, max_length=500)
|
||||
|
||||
|
||||
class FileConnectorWriteResponse(BaseModel):
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool = False
|
||||
provider: str
|
||||
remote_path: str
|
||||
revision: str | None = None
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
|
||||
|
||||
class BulkDeleteRequest(BaseModel):
|
||||
file_ids: list[str]
|
||||
|
||||
@@ -97,6 +621,69 @@ class BulkDeleteResponse(BaseModel):
|
||||
deleted_count: int
|
||||
|
||||
|
||||
class FileLifecycleUpdateRequest(BaseModel):
|
||||
retained_until: datetime | None = None
|
||||
legal_hold: bool = False
|
||||
reason: str = Field(min_length=1, max_length=500)
|
||||
expected_revision: int = Field(ge=1)
|
||||
|
||||
|
||||
class FileRestoreResponse(BaseModel):
|
||||
restored_count: int
|
||||
|
||||
|
||||
class FileFolderRestoreResponse(BaseModel):
|
||||
restored_folders: int
|
||||
restored_files: int
|
||||
|
||||
|
||||
class FilePurgePreviewRequest(BaseModel):
|
||||
file_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class FilePurgePreviewItem(BaseModel):
|
||||
file_id: str
|
||||
filename: str
|
||||
lifecycle_revision: int
|
||||
deleted_at: str | None = None
|
||||
retained_until: str | None = None
|
||||
legal_hold: bool = False
|
||||
blockers: list[str] = Field(default_factory=list)
|
||||
blob_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class FilePurgePreviewResponse(BaseModel):
|
||||
preview_sha256: str
|
||||
eligible: bool
|
||||
items: list[FilePurgePreviewItem]
|
||||
|
||||
|
||||
class FilePurgeExecuteRequest(FilePurgePreviewRequest):
|
||||
preview_sha256: str = Field(min_length=64, max_length=64)
|
||||
idempotency_key: str = Field(min_length=8, max_length=160)
|
||||
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||
confirmation: Literal["PURGE"]
|
||||
|
||||
|
||||
class FilePurgeResponse(BaseModel):
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool = False
|
||||
purged_files: int = 0
|
||||
released_blobs: int = 0
|
||||
|
||||
|
||||
class FileBlobGcRequest(BaseModel):
|
||||
limit: int = Field(default=25, ge=1, le=100)
|
||||
approval_reference: str = Field(min_length=3, max_length=1000)
|
||||
|
||||
|
||||
class FileBlobGcResponse(BaseModel):
|
||||
inspected_blobs: int
|
||||
deleted_blobs: int
|
||||
unresolved_operation_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ConflictResolutionRequest(BaseModel):
|
||||
target_path: str
|
||||
action: Literal["overwrite", "rename", "skip"]
|
||||
@@ -111,6 +698,7 @@ class FileShareRequest(BaseModel):
|
||||
target_type: Literal["user", "group", "campaign", "tenant"]
|
||||
target_id: str
|
||||
permission: Literal["read", "write", "manage"] = "read"
|
||||
expires_at: datetime | None = None
|
||||
|
||||
|
||||
class BulkFileShareRequest(BaseModel):
|
||||
@@ -118,6 +706,7 @@ class BulkFileShareRequest(BaseModel):
|
||||
target_type: Literal["user", "group", "campaign", "tenant"]
|
||||
target_id: str
|
||||
permission: Literal["read", "write", "manage"] = "read"
|
||||
expires_at: datetime | None = None
|
||||
|
||||
|
||||
class BulkFileShareResponse(BaseModel):
|
||||
@@ -128,8 +717,8 @@ class BulkFileShareResponse(BaseModel):
|
||||
class RenameRequest(BaseModel):
|
||||
file_ids: list[str] = Field(default_factory=list)
|
||||
folder_paths: list[str] = Field(default_factory=list)
|
||||
owner_type: Literal["user", "group"] | None = None
|
||||
owner_id: str | None = None
|
||||
owner_type: Literal["user", "group"]
|
||||
owner_id: str
|
||||
mode: Literal["direct", "prefix", "suffix", "replace"]
|
||||
new_name: str | None = None
|
||||
find: str | None = None
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from urllib.parse import quote
|
||||
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.events import PlatformEvent
|
||||
from govoplan_core.core.modules import ModuleContext
|
||||
from govoplan_core.core.search import (
|
||||
SearchAuthorizationRequest,
|
||||
SearchBackfillPage,
|
||||
SearchBackfillRequest,
|
||||
SearchDocument,
|
||||
SearchIndexChange,
|
||||
SearchResourceReference,
|
||||
SearchResourceType,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
PROVIDER_ID = "files.objects"
|
||||
RESOURCE_MODELS = {
|
||||
"file": FileAsset,
|
||||
"folder": FileFolder,
|
||||
}
|
||||
READ_SCOPE = "files:file:read"
|
||||
ADMIN_SCOPE = "files:file:admin"
|
||||
|
||||
|
||||
class FilesSearchSource:
|
||||
def resource_types(self) -> Sequence[SearchResourceType]:
|
||||
return (
|
||||
SearchResourceType(
|
||||
provider_id=PROVIDER_ID,
|
||||
module_id="files",
|
||||
resource_type="file",
|
||||
label="Files",
|
||||
requires_authorization_recheck=True,
|
||||
),
|
||||
SearchResourceType(
|
||||
provider_id=PROVIDER_ID,
|
||||
module_id="files",
|
||||
resource_type="folder",
|
||||
label="File folders",
|
||||
requires_authorization_recheck=True,
|
||||
),
|
||||
)
|
||||
|
||||
def backfill(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
request: SearchBackfillRequest,
|
||||
) -> SearchBackfillPage:
|
||||
db = _session(session)
|
||||
model = _model(request.provider_id, request.resource_type)
|
||||
statement = select(model).where(
|
||||
model.tenant_id == request.tenant_id,
|
||||
model.deleted_at.is_(None),
|
||||
)
|
||||
if request.cursor:
|
||||
statement = statement.where(model.id > request.cursor)
|
||||
rows = list(
|
||||
db.scalars(
|
||||
statement.order_by(model.id).limit(request.limit + 1)
|
||||
)
|
||||
)
|
||||
has_more = len(rows) > request.limit
|
||||
selected = rows[: request.limit]
|
||||
shares = (
|
||||
_shares_by_asset(db, selected)
|
||||
if request.resource_type == "file"
|
||||
else {}
|
||||
)
|
||||
high_watermark = db.scalar(
|
||||
select(func.max(model.updated_at)).where(
|
||||
model.tenant_id == request.tenant_id,
|
||||
model.deleted_at.is_(None),
|
||||
)
|
||||
)
|
||||
return SearchBackfillPage(
|
||||
documents=tuple(
|
||||
_document(
|
||||
row,
|
||||
resource_type=request.resource_type,
|
||||
shares=shares.get(row.id, ()),
|
||||
)
|
||||
for row in selected
|
||||
),
|
||||
next_cursor=selected[-1].id if has_more and selected else None,
|
||||
complete=not has_more,
|
||||
high_watermark=(
|
||||
high_watermark.isoformat()
|
||||
if high_watermark is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def authorize(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
requests: Sequence[SearchAuthorizationRequest],
|
||||
) -> Mapping[str, bool]:
|
||||
decisions = {item.reference.key: False for item in requests}
|
||||
if not isinstance(principal, ApiPrincipal) or not (
|
||||
principal.has(READ_SCOPE) or principal.has(ADMIN_SCOPE)
|
||||
):
|
||||
return decisions
|
||||
db = _session(session)
|
||||
for request in requests:
|
||||
reference = request.reference
|
||||
if (
|
||||
reference.tenant_id != principal.tenant_id
|
||||
or reference.module_id != "files"
|
||||
or reference.resource_type not in RESOURCE_MODELS
|
||||
):
|
||||
continue
|
||||
decisions[reference.key] = _can_read(
|
||||
db,
|
||||
principal,
|
||||
reference=reference,
|
||||
)
|
||||
return decisions
|
||||
|
||||
def index_changes_for_event(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
event: PlatformEvent,
|
||||
delivery_key: str,
|
||||
) -> Sequence[SearchIndexChange]:
|
||||
if (
|
||||
event.module_id != "files"
|
||||
or event.tenant is None
|
||||
or event.resource is None
|
||||
or event.resource.id is None
|
||||
or event.resource.type not in RESOURCE_MODELS
|
||||
):
|
||||
return ()
|
||||
db = _session(session)
|
||||
reference = SearchResourceReference(
|
||||
tenant_id=event.tenant.id,
|
||||
module_id="files",
|
||||
resource_type=event.resource.type,
|
||||
resource_id=event.resource.id,
|
||||
)
|
||||
model = RESOURCE_MODELS[event.resource.type]
|
||||
row = db.get(model, event.resource.id)
|
||||
deleted = row is None or row.tenant_id != event.tenant.id or row.deleted_at is not None
|
||||
cursor = event.event_id
|
||||
document = None
|
||||
if not deleted:
|
||||
shares = (
|
||||
tuple(_active_shares(db, row.id))
|
||||
if event.resource.type == "file"
|
||||
else ()
|
||||
)
|
||||
document = _document(
|
||||
row,
|
||||
resource_type=event.resource.type,
|
||||
shares=shares,
|
||||
change_cursor=cursor,
|
||||
)
|
||||
return (
|
||||
SearchIndexChange(
|
||||
change_id=f"{delivery_key}:{PROVIDER_ID}:{event.resource.type}",
|
||||
provider_id=PROVIDER_ID,
|
||||
kind="delete" if deleted else "upsert",
|
||||
reference=reference,
|
||||
source_revision=(
|
||||
document.source_revision if document is not None else cursor
|
||||
),
|
||||
cursor=cursor,
|
||||
document=document,
|
||||
occurred_at=event.occurred_at,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def create_files_search_source(_context: ModuleContext) -> FilesSearchSource:
|
||||
return FilesSearchSource()
|
||||
|
||||
|
||||
def _model(provider_id: str, resource_type: str):
|
||||
if provider_id != PROVIDER_ID or resource_type not in RESOURCE_MODELS:
|
||||
raise ValueError("Unsupported Files search source.")
|
||||
return RESOURCE_MODELS[resource_type]
|
||||
|
||||
|
||||
def _document(
|
||||
row: FileAsset | FileFolder,
|
||||
*,
|
||||
resource_type: str,
|
||||
shares: Sequence[FileShare] = (),
|
||||
change_cursor: str | None = None,
|
||||
) -> SearchDocument:
|
||||
is_file = isinstance(row, FileAsset)
|
||||
title = row.filename if is_file else (row.path.rsplit("/", 1)[-1] or row.path)
|
||||
path = row.display_path if is_file else row.path
|
||||
owner_id = row.owner_user_id if row.owner_type == "user" else row.owner_group_id
|
||||
tokens = [f"scope:{READ_SCOPE}", f"scope:{ADMIN_SCOPE}"]
|
||||
if owner_id:
|
||||
tokens.append(
|
||||
f"membership:{owner_id}"
|
||||
if row.owner_type == "user"
|
||||
else f"group:{owner_id}"
|
||||
)
|
||||
for share in shares:
|
||||
prefix = "membership" if share.target_type == "user" else share.target_type
|
||||
if prefix in {"membership", "group", "tenant"}:
|
||||
tokens.append(f"{prefix}:{share.target_id}")
|
||||
updated_at = row.updated_at or row.created_at
|
||||
revision = (
|
||||
f"{row.current_version_id or 'none'}:{updated_at.isoformat()}"
|
||||
if is_file
|
||||
else updated_at.isoformat()
|
||||
)
|
||||
return SearchDocument(
|
||||
tenant_id=row.tenant_id,
|
||||
module_id="files",
|
||||
provider_id=PROVIDER_ID,
|
||||
resource_type=resource_type,
|
||||
resource_id=row.id,
|
||||
title=title,
|
||||
url=f"/files?{resource_type}Id={quote(row.id, safe='')}",
|
||||
summary=((row.description or "") if is_file else path)[:4000] or None,
|
||||
body=" ".join(
|
||||
value for value in (path, row.description if is_file else None) if value
|
||||
)[:200_000],
|
||||
keywords=(path[:200], row.owner_type[:200]),
|
||||
visibility="restricted",
|
||||
acl_tokens=tuple(dict.fromkeys(tokens)),
|
||||
metadata={
|
||||
"path": path,
|
||||
"owner_type": row.owner_type,
|
||||
"current_version_id": row.current_version_id if is_file else None,
|
||||
},
|
||||
source_revision=revision,
|
||||
change_cursor=change_cursor,
|
||||
source_updated_at=updated_at,
|
||||
requires_authorization_recheck=True,
|
||||
)
|
||||
|
||||
|
||||
def _can_read(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
reference: SearchResourceReference,
|
||||
) -> bool:
|
||||
model = RESOURCE_MODELS[reference.resource_type]
|
||||
row = session.get(model, reference.resource_id)
|
||||
if row is None or row.tenant_id != principal.tenant_id or row.deleted_at is not None:
|
||||
return False
|
||||
if principal.has(ADMIN_SCOPE):
|
||||
return True
|
||||
user_id = str(getattr(principal.user, "id", "") or principal.membership_id or "")
|
||||
if row.owner_type == "user" and row.owner_user_id == user_id:
|
||||
return True
|
||||
if row.owner_type == "group" and row.owner_group_id in principal.group_ids:
|
||||
return True
|
||||
if reference.resource_type != "file":
|
||||
return False
|
||||
target_clauses = [
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
(FileShare.target_type == "tenant")
|
||||
& (FileShare.target_id == principal.tenant_id),
|
||||
]
|
||||
if principal.group_ids:
|
||||
target_clauses.append(
|
||||
(FileShare.target_type == "group")
|
||||
& (FileShare.target_id.in_(tuple(principal.group_ids)))
|
||||
)
|
||||
return session.scalar(
|
||||
select(FileShare.id).where(
|
||||
FileShare.tenant_id == principal.tenant_id,
|
||||
FileShare.file_asset_id == row.id,
|
||||
effective_file_share_clause(),
|
||||
or_(*target_clauses),
|
||||
).limit(1)
|
||||
) is not None
|
||||
|
||||
|
||||
def _active_shares(session: Session, asset_id: str) -> Sequence[FileShare]:
|
||||
return tuple(
|
||||
session.scalars(
|
||||
select(FileShare).where(
|
||||
FileShare.file_asset_id == asset_id,
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _shares_by_asset(
|
||||
session: Session,
|
||||
rows: Sequence[FileAsset | FileFolder],
|
||||
) -> dict[str, tuple[FileShare, ...]]:
|
||||
asset_ids = [row.id for row in rows if isinstance(row, FileAsset)]
|
||||
grouped: dict[str, list[FileShare]] = {asset_id: [] for asset_id in asset_ids}
|
||||
if not asset_ids:
|
||||
return {}
|
||||
for share in session.scalars(
|
||||
select(FileShare).where(
|
||||
FileShare.file_asset_id.in_(asset_ids),
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
):
|
||||
grouped.setdefault(share.file_asset_id, []).append(share)
|
||||
return {key: tuple(value) for key, value in grouped.items()}
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Files search requires a SQLAlchemy session.")
|
||||
return value
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FilesSearchSource",
|
||||
"PROVIDER_ID",
|
||||
"create_files_search_source",
|
||||
]
|
||||
@@ -0,0 +1 @@
|
||||
"""Query and response assembly services used by Files routes."""
|
||||
@@ -0,0 +1,292 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||
from govoplan_core.core.change_sequence import (
|
||||
ChangeSequenceEntry,
|
||||
)
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
FILES_CONNECTOR_POLICIES_COLLECTION,
|
||||
FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
FILES_CONNECTOR_SPACES_COLLECTION,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileConnectorCredentialResponse,
|
||||
FileConnectorSettingsDeltaResponse,
|
||||
FileConnectorPolicyResponse,
|
||||
FileConnectorProfileResponse,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileConnectorSpace
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
ConnectorCredential,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
connector_policy_response,
|
||||
)
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
FILES_CONNECTOR_CREDENTIAL_RESOURCE,
|
||||
FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
FILES_CONNECTOR_SPACE_RESOURCE,
|
||||
_can_read_disabled_connector_profiles,
|
||||
_connector_deleted_item,
|
||||
_connector_space_response,
|
||||
_ensure_campaign_file_access,
|
||||
_file_connector_settings_response_watermark,
|
||||
_file_connector_settings_watermark,
|
||||
_visible_connector_credentials,
|
||||
_visible_connector_profiles,
|
||||
_visible_connector_spaces,
|
||||
)
|
||||
|
||||
|
||||
def _full_file_connector_settings_delta_response(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
provider: str | None,
|
||||
campaign_id: str | None,
|
||||
include_disabled: bool,
|
||||
include_inactive: bool,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> FileConnectorSettingsDeltaResponse:
|
||||
if campaign_id:
|
||||
_ensure_campaign_file_access(session, principal, campaign_id)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
credentials = _visible_connector_credentials(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
spaces = _visible_connector_spaces(
|
||||
session,
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
return FileConnectorSettingsDeltaResponse(
|
||||
profiles=[
|
||||
FileConnectorProfileResponse(**profile.to_response())
|
||||
for profile in profiles
|
||||
],
|
||||
credentials=[
|
||||
FileConnectorCredentialResponse(**credential.to_response())
|
||||
for credential in credentials
|
||||
],
|
||||
spaces=[_connector_space_response(space) for space in spaces],
|
||||
policy=FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
),
|
||||
changed_sections=["profiles", "credentials", "spaces", "policy"],
|
||||
deleted=[],
|
||||
watermark=_file_connector_settings_watermark(
|
||||
session, tenant_id=principal.tenant_id
|
||||
),
|
||||
has_more=False,
|
||||
full=True,
|
||||
)
|
||||
|
||||
|
||||
def _changed_file_connector_setting_ids(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
) -> tuple[set[str], set[str], set[str], bool]:
|
||||
changed_profile_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION
|
||||
and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
}
|
||||
changed_credential_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION
|
||||
and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
}
|
||||
changed_space_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION
|
||||
and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
}
|
||||
policy_changed = any(
|
||||
entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries
|
||||
)
|
||||
return (
|
||||
changed_profile_ids,
|
||||
changed_credential_ids,
|
||||
changed_space_ids,
|
||||
policy_changed,
|
||||
)
|
||||
|
||||
|
||||
def _file_connector_settings_changed_sections(
|
||||
*,
|
||||
changed_profile_ids: set[str],
|
||||
changed_credential_ids: set[str],
|
||||
changed_space_ids: set[str],
|
||||
policy_changed: bool,
|
||||
profiles: list[ConnectorProfile],
|
||||
) -> list[str]:
|
||||
changed_sections = []
|
||||
if changed_profile_ids or any(
|
||||
profile.credential_profile_id
|
||||
and profile.credential_profile_id in changed_credential_ids
|
||||
for profile in profiles
|
||||
):
|
||||
changed_sections.append("profiles")
|
||||
if changed_credential_ids:
|
||||
changed_sections.append("credentials")
|
||||
if changed_space_ids:
|
||||
changed_sections.append("spaces")
|
||||
if policy_changed:
|
||||
changed_sections.append("policy")
|
||||
return changed_sections
|
||||
|
||||
|
||||
def _file_connector_settings_deleted_items(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
*,
|
||||
visible_profiles: dict[str, ConnectorProfile],
|
||||
visible_credentials: dict[str, ConnectorCredential],
|
||||
visible_spaces: dict[str, FileConnectorSpace],
|
||||
) -> list[DeltaDeletedItem]:
|
||||
return [
|
||||
_connector_deleted_item(entry)
|
||||
for entry in entries
|
||||
if (
|
||||
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
and entry.resource_id not in visible_profiles
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
and entry.resource_id not in visible_credentials
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
and entry.resource_id not in visible_spaces
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _incremental_file_connector_settings_delta_response(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
entries: list[ChangeSequenceEntry],
|
||||
has_more: bool,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
provider: str | None,
|
||||
campaign_id: str | None,
|
||||
include_disabled: bool,
|
||||
include_inactive: bool,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> FileConnectorSettingsDeltaResponse:
|
||||
changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed = (
|
||||
_changed_file_connector_setting_ids(entries)
|
||||
)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled
|
||||
and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
visible_profiles = {
|
||||
profile.id: profile
|
||||
for profile in profiles
|
||||
if profile.id in changed_profile_ids
|
||||
or (
|
||||
profile.credential_profile_id
|
||||
and profile.credential_profile_id in changed_credential_ids
|
||||
)
|
||||
}
|
||||
credentials = _visible_connector_credentials(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
visible_credentials = {
|
||||
credential.id: credential
|
||||
for credential in credentials
|
||||
if credential.id in changed_credential_ids
|
||||
}
|
||||
spaces = _visible_connector_spaces(
|
||||
session,
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
visible_spaces = {
|
||||
space.id: space for space in spaces if space.id in changed_space_ids
|
||||
}
|
||||
return FileConnectorSettingsDeltaResponse(
|
||||
profiles=[
|
||||
FileConnectorProfileResponse(**profile.to_response())
|
||||
for profile in visible_profiles.values()
|
||||
],
|
||||
credentials=[
|
||||
FileConnectorCredentialResponse(**credential.to_response())
|
||||
for credential in visible_credentials.values()
|
||||
],
|
||||
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
|
||||
policy=FileConnectorPolicyResponse(
|
||||
**connector_policy_response(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
)
|
||||
if policy_changed
|
||||
else None,
|
||||
changed_sections=_file_connector_settings_changed_sections(
|
||||
changed_profile_ids=changed_profile_ids,
|
||||
changed_credential_ids=changed_credential_ids,
|
||||
changed_space_ids=changed_space_ids,
|
||||
policy_changed=policy_changed,
|
||||
profiles=profiles,
|
||||
),
|
||||
deleted=_file_connector_settings_deleted_items(
|
||||
entries,
|
||||
visible_profiles=visible_profiles,
|
||||
visible_credentials=visible_credentials,
|
||||
visible_spaces=visible_spaces,
|
||||
),
|
||||
watermark=_file_connector_settings_response_watermark(
|
||||
session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more
|
||||
),
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
)
|
||||
@@ -0,0 +1,620 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.api.v1.schemas import DeltaDeletedItem
|
||||
from govoplan_core.core.change_sequence import (
|
||||
ChangeSequenceEntry,
|
||||
decode_sequence_watermark,
|
||||
encode_sequence_watermark,
|
||||
max_sequence_id,
|
||||
sequence_entries_since,
|
||||
sequence_watermark_is_expired,
|
||||
)
|
||||
from govoplan_core.core.pagination import (
|
||||
KeysetCursorError,
|
||||
decode_keyset_cursor,
|
||||
encode_keyset_cursor,
|
||||
keyset_query_fingerprint,
|
||||
)
|
||||
from govoplan_files.backend.change_tracking import (
|
||||
FILES_ASSETS_COLLECTION,
|
||||
FILES_FOLDERS_COLLECTION,
|
||||
FILES_MODULE_ID,
|
||||
)
|
||||
from govoplan_files.backend.schemas import (
|
||||
FileDeltaResponse,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare
|
||||
from govoplan_files.backend.storage.access import user_group_ids
|
||||
from govoplan_files.backend.storage.files import (
|
||||
list_assets_for_user,
|
||||
)
|
||||
from govoplan_files.backend.storage.folders import list_folders_for_user
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
from govoplan_files.backend.route_support import (
|
||||
_asset_list_response,
|
||||
_folder_response,
|
||||
_is_admin,
|
||||
)
|
||||
|
||||
_FILES_DELTA_COLLECTIONS = (FILES_ASSETS_COLLECTION, FILES_FOLDERS_COLLECTION)
|
||||
FILES_LIST_CURSOR_SCOPE = "files.list.v1"
|
||||
FOLDERS_LIST_CURSOR_SCOPE = "files.folders.list.v1"
|
||||
DEFAULT_FILE_LIST_PAGE_SIZE = 500
|
||||
|
||||
|
||||
def _cursor_http_error(exc: Exception) -> HTTPException:
|
||||
return HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc))
|
||||
|
||||
|
||||
def _cursor_page_size(
|
||||
scope: str, cursor: str | None, explicit_page_size: int | None
|
||||
) -> int | None:
|
||||
if explicit_page_size is not None:
|
||||
return explicit_page_size
|
||||
if not cursor:
|
||||
return DEFAULT_FILE_LIST_PAGE_SIZE
|
||||
try:
|
||||
values = decode_keyset_cursor(scope, cursor)
|
||||
except KeysetCursorError as exc:
|
||||
raise _cursor_http_error(exc) from exc
|
||||
raw_page_size = values.get("page_size")
|
||||
if not isinstance(raw_page_size, int) or raw_page_size < 1 or raw_page_size > 1000:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
)
|
||||
return raw_page_size
|
||||
|
||||
|
||||
def _files_list_fingerprint(
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None,
|
||||
audit_relevant: bool | None,
|
||||
page_size: int,
|
||||
) -> str:
|
||||
return keyset_query_fingerprint(
|
||||
FILES_LIST_CURSOR_SCOPE,
|
||||
{
|
||||
"tenant_id": principal.tenant_id,
|
||||
"actor": "admin" if _is_admin(principal) else principal.user.id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"campaign_id": campaign_id,
|
||||
"path_prefix": path_prefix or "",
|
||||
"campaign_usage": campaign_usage or "",
|
||||
"audit_relevant": audit_relevant,
|
||||
"sort": "display_path.asc,updated_at.desc,id.asc",
|
||||
"page_size": page_size,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _folders_list_fingerprint(
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
page_size: int,
|
||||
) -> str:
|
||||
return keyset_query_fingerprint(
|
||||
FOLDERS_LIST_CURSOR_SCOPE,
|
||||
{
|
||||
"tenant_id": principal.tenant_id,
|
||||
"actor": "admin" if _is_admin(principal) else principal.user.id,
|
||||
"owner_type": owner_type,
|
||||
"owner_id": owner_id,
|
||||
"sort": "path.asc,id.asc",
|
||||
"page_size": page_size,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _file_cursor_values(
|
||||
cursor: str | None, *, fingerprint: str
|
||||
) -> tuple[str | None, datetime | None, str | None]:
|
||||
try:
|
||||
values = decode_keyset_cursor(
|
||||
FILES_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
|
||||
)
|
||||
except KeysetCursorError as exc:
|
||||
raise _cursor_http_error(exc) from exc
|
||||
if values is None:
|
||||
return None, None, None
|
||||
display_path = values.get("display_path")
|
||||
updated_at = values.get("updated_at")
|
||||
asset_id = values.get("id")
|
||||
if (
|
||||
not isinstance(display_path, str)
|
||||
or not isinstance(updated_at, str)
|
||||
or not isinstance(asset_id, str)
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
)
|
||||
try:
|
||||
parsed_updated_at = datetime.fromisoformat(updated_at)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
) from exc
|
||||
return display_path, parsed_updated_at, asset_id
|
||||
|
||||
|
||||
def _folder_cursor_values(
|
||||
cursor: str | None, *, fingerprint: str
|
||||
) -> tuple[str | None, str | None]:
|
||||
try:
|
||||
values = decode_keyset_cursor(
|
||||
FOLDERS_LIST_CURSOR_SCOPE, cursor, fingerprint=fingerprint
|
||||
)
|
||||
except KeysetCursorError as exc:
|
||||
raise _cursor_http_error(exc) from exc
|
||||
if values is None:
|
||||
return None, None
|
||||
folder_path = values.get("path")
|
||||
folder_id = values.get("id")
|
||||
if not isinstance(folder_path, str) or not isinstance(folder_id, str):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid pagination cursor"
|
||||
)
|
||||
return folder_path, folder_id
|
||||
|
||||
|
||||
def _next_file_list_cursor(
|
||||
principal: ApiPrincipal,
|
||||
assets: list[FileAsset],
|
||||
*,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
campaign_usage: Literal["linked", "unlinked"] | None,
|
||||
audit_relevant: bool | None,
|
||||
page_size: int,
|
||||
has_more: bool,
|
||||
) -> str | None:
|
||||
if not has_more or not assets:
|
||||
return None
|
||||
last = assets[-1]
|
||||
return encode_keyset_cursor(
|
||||
FILES_LIST_CURSOR_SCOPE,
|
||||
fingerprint=_files_list_fingerprint(
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
page_size=page_size,
|
||||
),
|
||||
values={
|
||||
"display_path": last.display_path,
|
||||
"updated_at": last.updated_at,
|
||||
"id": last.id,
|
||||
"page_size": page_size,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _next_folder_list_cursor(
|
||||
principal: ApiPrincipal,
|
||||
folders: list[FileFolder],
|
||||
*,
|
||||
owner_type: Literal["user", "group"],
|
||||
owner_id: str,
|
||||
page_size: int,
|
||||
has_more: bool,
|
||||
) -> str | None:
|
||||
if not has_more or not folders:
|
||||
return None
|
||||
last = folders[-1]
|
||||
return encode_keyset_cursor(
|
||||
FOLDERS_LIST_CURSOR_SCOPE,
|
||||
fingerprint=_folders_list_fingerprint(
|
||||
principal, owner_type=owner_type, owner_id=owner_id, page_size=page_size
|
||||
),
|
||||
values={"path": last.path, "id": last.id, "page_size": page_size},
|
||||
)
|
||||
|
||||
|
||||
def _files_delta_watermark(session: Session, tenant_id: str) -> str:
|
||||
return encode_sequence_watermark(
|
||||
max_sequence_id(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collections=_FILES_DELTA_COLLECTIONS,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _full_file_delta_response(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> FileDeltaResponse:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
folders = _visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
return FileDeltaResponse(
|
||||
files=_asset_list_response(session, assets, include_shares=True),
|
||||
folders=[_folder_response(folder) for folder in folders],
|
||||
deleted=[],
|
||||
watermark=_files_delta_watermark(session, principal.tenant_id),
|
||||
has_more=False,
|
||||
full=True,
|
||||
)
|
||||
|
||||
|
||||
def _visible_folders_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> list[FileFolder]:
|
||||
if not owner_type or not owner_id:
|
||||
return []
|
||||
folders = list_folders_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if not path_prefix:
|
||||
return folders
|
||||
normalized = path_prefix.strip().strip("/")
|
||||
if not normalized:
|
||||
return folders
|
||||
return [
|
||||
folder
|
||||
for folder in folders
|
||||
if folder.path == normalized or folder.path.startswith(f"{normalized}/")
|
||||
]
|
||||
|
||||
|
||||
def _entry_path_matches(
|
||||
entry_payload: dict[str, object], path_prefix: str | None
|
||||
) -> bool:
|
||||
if not path_prefix:
|
||||
return True
|
||||
normalized = path_prefix.strip().strip("/")
|
||||
if not normalized:
|
||||
return True
|
||||
for key in ("path", "previous_path"):
|
||||
value = entry_payload.get(key)
|
||||
if isinstance(value, str) and (
|
||||
value == normalized or value.startswith(f"{normalized}/")
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _entry_owner_matches(
|
||||
entry_payload: dict[str, object],
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> bool:
|
||||
if not owner_type or not owner_id:
|
||||
return True
|
||||
return (
|
||||
entry_payload.get("owner_type") == owner_type
|
||||
and entry_payload.get("owner_id") == owner_id
|
||||
)
|
||||
|
||||
|
||||
def _entry_campaign_matches(session: Session, entry, campaign_id: str | None) -> bool:
|
||||
if not campaign_id:
|
||||
return True
|
||||
payload = entry.payload or {}
|
||||
if (
|
||||
payload.get("share_target_type") == "campaign"
|
||||
and payload.get("share_target_id") == campaign_id
|
||||
):
|
||||
return True
|
||||
if entry.resource_type != "file":
|
||||
return False
|
||||
return (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == entry.tenant_id,
|
||||
FileShare.file_asset_id == entry.resource_id,
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
.first()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _principal_group_ids_for_delta(
|
||||
session: Session, principal: ApiPrincipal
|
||||
) -> set[str]:
|
||||
cache = session.info.setdefault("files_delta_group_ids", {})
|
||||
key = (principal.tenant_id, principal.user.id)
|
||||
if key not in cache:
|
||||
cache[key] = set(
|
||||
user_group_ids(
|
||||
session, tenant_id=principal.tenant_id, user_id=principal.user.id
|
||||
)
|
||||
)
|
||||
return cache[key]
|
||||
|
||||
|
||||
def _entry_subject_matches_principal(
|
||||
session: Session, principal: ApiPrincipal, entry_payload: dict[str, object]
|
||||
) -> bool:
|
||||
if _is_admin(principal):
|
||||
return True
|
||||
owner_type = entry_payload.get("owner_type")
|
||||
owner_id = entry_payload.get("owner_id")
|
||||
if owner_type == "user" and owner_id == principal.user.id:
|
||||
return True
|
||||
if owner_type == "group" and isinstance(owner_id, str):
|
||||
if owner_id in _principal_group_ids_for_delta(session, principal):
|
||||
return True
|
||||
share_target_type = entry_payload.get("share_target_type")
|
||||
share_target_id = entry_payload.get("share_target_id")
|
||||
if share_target_type == "user" and share_target_id == principal.user.id:
|
||||
return True
|
||||
if share_target_type == "tenant" and share_target_id == principal.tenant_id:
|
||||
return True
|
||||
if share_target_type == "group" and isinstance(share_target_id, str):
|
||||
if share_target_id in _principal_group_ids_for_delta(session, principal):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _entry_matches_delta_scope(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
entry,
|
||||
*,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> bool:
|
||||
payload = entry.payload or {}
|
||||
if (
|
||||
not owner_type
|
||||
and not campaign_id
|
||||
and not _entry_subject_matches_principal(session, principal, payload)
|
||||
):
|
||||
return False
|
||||
return (
|
||||
_entry_owner_matches(payload, owner_type, owner_id)
|
||||
and _entry_path_matches(payload, path_prefix)
|
||||
and _entry_campaign_matches(session, entry, campaign_id)
|
||||
)
|
||||
|
||||
|
||||
def _decode_files_delta_watermark(since: str) -> int:
|
||||
try:
|
||||
return decode_sequence_watermark(since)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)
|
||||
) from exc
|
||||
|
||||
|
||||
def _changed_delta_resource_ids(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
) -> tuple[list[str], list[str]]:
|
||||
file_ids = list(
|
||||
dict.fromkeys(
|
||||
entry.resource_id for entry in entries if entry.resource_type == "file"
|
||||
)
|
||||
)
|
||||
folder_ids = list(
|
||||
dict.fromkeys(
|
||||
entry.resource_id for entry in entries if entry.resource_type == "folder"
|
||||
)
|
||||
)
|
||||
return file_ids, folder_ids
|
||||
|
||||
|
||||
def _visible_assets_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
changed_file_ids: list[str],
|
||||
) -> dict[str, FileAsset]:
|
||||
return {
|
||||
asset.id: asset
|
||||
for asset in list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if asset.id in changed_file_ids
|
||||
}
|
||||
|
||||
|
||||
def _changed_visible_folders_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
path_prefix: str | None,
|
||||
changed_folder_ids: list[str],
|
||||
) -> dict[str, FileFolder]:
|
||||
return {
|
||||
folder.id: folder
|
||||
for folder in _visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
if folder.id in changed_folder_ids
|
||||
}
|
||||
|
||||
|
||||
def _deleted_delta_items(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
entries: list[ChangeSequenceEntry],
|
||||
visible_assets: dict[str, FileAsset],
|
||||
visible_folders: dict[str, FileFolder],
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> list[DeltaDeletedItem]:
|
||||
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
|
||||
for entry in entries:
|
||||
if entry.resource_type == "file" and entry.resource_id in visible_assets:
|
||||
continue
|
||||
if entry.resource_type == "folder" and entry.resource_id in visible_folders:
|
||||
continue
|
||||
if not _entry_matches_delta_scope(
|
||||
session,
|
||||
principal,
|
||||
entry,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
):
|
||||
continue
|
||||
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
|
||||
id=entry.resource_id,
|
||||
resource_type=entry.resource_type,
|
||||
revision=encode_sequence_watermark(entry.id),
|
||||
deleted_at=entry.created_at if entry.operation == "deleted" else None,
|
||||
)
|
||||
return list(deleted.values())
|
||||
|
||||
|
||||
def _files_delta_response(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
since: str,
|
||||
limit: int,
|
||||
) -> FileDeltaResponse:
|
||||
since_sequence = _decode_files_delta_watermark(since)
|
||||
if sequence_watermark_is_expired(
|
||||
session,
|
||||
since=since_sequence,
|
||||
tenant_id=principal.tenant_id,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collections=_FILES_DELTA_COLLECTIONS,
|
||||
):
|
||||
return _full_file_delta_response(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
|
||||
entries_plus_one = sequence_entries_since(
|
||||
session,
|
||||
since=since_sequence,
|
||||
tenant_id=principal.tenant_id,
|
||||
module_id=FILES_MODULE_ID,
|
||||
collections=_FILES_DELTA_COLLECTIONS,
|
||||
limit=limit + 1,
|
||||
)
|
||||
has_more = len(entries_plus_one) > limit
|
||||
entries = entries_plus_one[:limit]
|
||||
|
||||
changed_file_ids, changed_folder_ids = _changed_delta_resource_ids(entries)
|
||||
visible_assets = _visible_assets_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
changed_file_ids=changed_file_ids,
|
||||
)
|
||||
visible_folders = _changed_visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
changed_folder_ids=changed_folder_ids,
|
||||
)
|
||||
deleted = _deleted_delta_items(
|
||||
session,
|
||||
principal=principal,
|
||||
entries=entries,
|
||||
visible_assets=visible_assets,
|
||||
visible_folders=visible_folders,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
|
||||
watermark = (
|
||||
encode_sequence_watermark(entries[-1].id)
|
||||
if has_more and entries
|
||||
else _files_delta_watermark(session, principal.tenant_id)
|
||||
)
|
||||
return FileDeltaResponse(
|
||||
files=_asset_list_response(
|
||||
session, list(visible_assets.values()), include_shares=True
|
||||
),
|
||||
folders=[_folder_response(folder) for folder in visible_folders.values()],
|
||||
deleted=deleted,
|
||||
watermark=watermark,
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
)
|
||||
@@ -2,33 +2,50 @@ from __future__ import annotations
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.db.models import Group, UserGroupMembership
|
||||
from govoplan_core.core.access import (
|
||||
CAPABILITY_ACCESS_DIRECTORY,
|
||||
AccessDirectory,
|
||||
GroupRef,
|
||||
)
|
||||
from govoplan_core.core.runtime import get_registry
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
def _access_directory() -> AccessDirectory:
|
||||
registry = get_registry()
|
||||
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_ACCESS_DIRECTORY):
|
||||
raise FileStorageError("Access directory capability is not configured")
|
||||
capability = registry.require_capability(CAPABILITY_ACCESS_DIRECTORY)
|
||||
if not isinstance(capability, AccessDirectory):
|
||||
raise FileStorageError("Access directory capability is invalid")
|
||||
return capability
|
||||
|
||||
|
||||
def group_refs_for_ids(*, tenant_id: str, group_ids: list[str]) -> list[GroupRef]:
|
||||
if not group_ids:
|
||||
return []
|
||||
groups = _access_directory().get_groups(group_ids)
|
||||
return sorted(
|
||||
[group for group in groups.values() if group.tenant_id == tenant_id],
|
||||
key=lambda group: group.name.casefold(),
|
||||
)
|
||||
|
||||
|
||||
def user_group_ids(session: Session, *, tenant_id: str, user_id: str, include_admin_groups: bool = False) -> list[str]:
|
||||
del session
|
||||
directory = _access_directory()
|
||||
if include_admin_groups:
|
||||
return [row.id for row in session.query(Group).filter(Group.tenant_id == tenant_id).order_by(Group.name.asc()).all()]
|
||||
return [
|
||||
row.group_id
|
||||
for row in session.query(UserGroupMembership)
|
||||
.filter(UserGroupMembership.tenant_id == tenant_id, UserGroupMembership.user_id == user_id)
|
||||
.all()
|
||||
]
|
||||
return [group.id for group in directory.groups_for_tenant(tenant_id)]
|
||||
return [group.id for group in directory.groups_for_user(user_id, tenant_id=tenant_id)]
|
||||
|
||||
|
||||
def ensure_group_access(session: Session, *, tenant_id: str, group_id: str, user_id: str, is_admin: bool = False) -> None:
|
||||
group = session.get(Group, group_id)
|
||||
group = _access_directory().get_group(group_id)
|
||||
if not group or group.tenant_id != tenant_id:
|
||||
raise FileStorageError("Group not found")
|
||||
if is_admin:
|
||||
return
|
||||
membership = (
|
||||
session.query(UserGroupMembership)
|
||||
.filter(UserGroupMembership.tenant_id == tenant_id, UserGroupMembership.user_id == user_id, UserGroupMembership.group_id == group_id)
|
||||
.one_or_none()
|
||||
)
|
||||
if membership is None:
|
||||
if group_id not in user_group_ids(session, tenant_id=tenant_id, user_id=user_id):
|
||||
raise FileStorageError("No access to this group file space")
|
||||
|
||||
|
||||
@@ -42,3 +59,21 @@ def ensure_owner_access(session: Session, *, tenant_id: str, owner_type: str, ow
|
||||
ensure_group_access(session, tenant_id=tenant_id, group_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
return
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def ensure_share_target_exists(*, tenant_id: str, target_type: str, target_id: str) -> None:
|
||||
target_type = target_type.lower().strip()
|
||||
if target_type == "user":
|
||||
user = _access_directory().get_user(target_id)
|
||||
if not user or user.tenant_id != tenant_id or user.status != "active":
|
||||
raise FileStorageError("User not found")
|
||||
return
|
||||
if target_type == "group":
|
||||
group = _access_directory().get_group(target_id)
|
||||
if not group or group.tenant_id != tenant_id or group.status != "active":
|
||||
raise FileStorageError("Group not found")
|
||||
return
|
||||
if target_type == "tenant":
|
||||
if target_id != tenant_id:
|
||||
raise FileStorageError("Tenant not found")
|
||||
return
|
||||
|
||||
@@ -1,53 +1,93 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import mimetypes
|
||||
import re
|
||||
import stat
|
||||
import tarfile
|
||||
import zipfile
|
||||
from dataclasses import dataclass
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
from os import PathLike
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any, BinaryIO, Iterable, Literal
|
||||
|
||||
import pyzipper
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
||||
from govoplan_files.backend.storage.files import create_file_asset, current_version_and_blob
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, normalize_folder, normalize_logical_path
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import (
|
||||
FileConflictResolution,
|
||||
FileStorageError,
|
||||
UploadedStoredFile,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
create_file_asset,
|
||||
current_versions_and_blobs,
|
||||
)
|
||||
from govoplan_files.backend.storage.paths import (
|
||||
filename_from_path,
|
||||
normalize_folder,
|
||||
normalize_logical_path,
|
||||
)
|
||||
|
||||
|
||||
_ZIP_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_ARCHIVE_READ_CHUNK_SIZE = 1024 * 1024
|
||||
_WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:")
|
||||
ARCHIVE_UPLOAD_MAX_ENTRIES = 10_000
|
||||
# Kept for callers and documentation using the former ZIP-specific name.
|
||||
ZIP_UPLOAD_MAX_FILES = ARCHIVE_UPLOAD_MAX_ENTRIES
|
||||
SUPPORTED_ARCHIVE_SUFFIXES = (
|
||||
".tar.bz2",
|
||||
".tar.gz",
|
||||
".tar.xz",
|
||||
".tbz2",
|
||||
".tgz",
|
||||
".txz",
|
||||
".tar",
|
||||
".zip",
|
||||
)
|
||||
|
||||
|
||||
def _read_zip_member(
|
||||
archive: zipfile.ZipFile,
|
||||
info: zipfile.ZipInfo,
|
||||
*,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
current_total: int,
|
||||
) -> tuple[bytes, int]:
|
||||
parts: list[bytes] = []
|
||||
actual_size = 0
|
||||
with archive.open(info) as source:
|
||||
while True:
|
||||
read_size = min(_ZIP_READ_CHUNK_SIZE, max_file_bytes + 1 - actual_size)
|
||||
chunk = source.read(read_size)
|
||||
if not chunk:
|
||||
break
|
||||
actual_size += len(chunk)
|
||||
if actual_size > max_file_bytes:
|
||||
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
|
||||
if current_total + actual_size > max_total_bytes:
|
||||
raise FileStorageError("ZIP is too large after extraction")
|
||||
parts.append(chunk)
|
||||
return b"".join(parts), current_total + actual_size
|
||||
class ArchivePasswordError(FileStorageError):
|
||||
pass
|
||||
|
||||
|
||||
def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path: str | Path) -> None:
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ArchiveEntry:
|
||||
path: str
|
||||
kind: Literal["file", "directory"]
|
||||
size_bytes: int
|
||||
compressed_size_bytes: int | None = None
|
||||
encrypted: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ArchiveInspection:
|
||||
archive_format: str
|
||||
entries: tuple[ArchiveEntry, ...]
|
||||
file_count: int
|
||||
directory_count: int
|
||||
expanded_size_bytes: int
|
||||
compressed_size_bytes: int
|
||||
requires_password: bool
|
||||
password_verified: bool
|
||||
|
||||
|
||||
def create_zip_file(
|
||||
session: Session, assets: Iterable[FileAsset], output_path: str | Path
|
||||
) -> None:
|
||||
backend = get_storage_backend()
|
||||
with zipfile.ZipFile(output_path, mode="w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for asset in assets:
|
||||
_version, blob = current_version_and_blob(session, asset)
|
||||
asset_list = list(assets)
|
||||
version_blobs = current_versions_and_blobs(session, asset_list)
|
||||
with zipfile.ZipFile(
|
||||
output_path, mode="w", compression=zipfile.ZIP_DEFLATED
|
||||
) as archive:
|
||||
for asset in asset_list:
|
||||
_version, blob = version_blobs[asset.id]
|
||||
info = zipfile.ZipInfo(asset.display_path)
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
with archive.open(info, "w") as member:
|
||||
@@ -59,6 +99,147 @@ def create_zip_file(session: Session, assets: Iterable[FileAsset], output_path:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
|
||||
|
||||
def archive_format_for_filename(filename: str) -> str:
|
||||
lowered = filename.strip().casefold()
|
||||
if lowered.endswith(".zip"):
|
||||
return "zip"
|
||||
if lowered.endswith((".tar.gz", ".tgz")):
|
||||
return "tar.gz"
|
||||
if lowered.endswith((".tar.bz2", ".tbz2")):
|
||||
return "tar.bz2"
|
||||
if lowered.endswith((".tar.xz", ".txz")):
|
||||
return "tar.xz"
|
||||
if lowered.endswith(".tar"):
|
||||
return "tar"
|
||||
raise FileStorageError(
|
||||
"Unsupported archive format. Use ZIP, TAR, TAR.GZ, TAR.BZ2, or TAR.XZ."
|
||||
)
|
||||
|
||||
|
||||
def is_supported_archive_filename(filename: str) -> bool:
|
||||
try:
|
||||
archive_format_for_filename(filename)
|
||||
except FileStorageError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def inspect_archive(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
filename: str,
|
||||
password: str | None = None,
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> ArchiveInspection:
|
||||
archive_format = archive_format_for_filename(filename)
|
||||
compressed_size = _archive_size(archive_data)
|
||||
if archive_format == "zip":
|
||||
entries, requires_password, password_verified = _inspect_zip(
|
||||
archive_data,
|
||||
password=password,
|
||||
)
|
||||
else:
|
||||
entries = _inspect_tar(archive_data)
|
||||
requires_password = False
|
||||
password_verified = True
|
||||
_validate_archive_limits(
|
||||
entries,
|
||||
compressed_size=compressed_size,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
complete_entries = _with_derived_directories(entries)
|
||||
return ArchiveInspection(
|
||||
archive_format=archive_format,
|
||||
entries=tuple(complete_entries),
|
||||
file_count=sum(entry.kind == "file" for entry in complete_entries),
|
||||
directory_count=sum(
|
||||
entry.kind == "directory" for entry in complete_entries
|
||||
),
|
||||
expanded_size_bytes=sum(
|
||||
entry.size_bytes for entry in entries if entry.kind == "file"
|
||||
),
|
||||
compressed_size_bytes=compressed_size,
|
||||
requires_password=requires_password,
|
||||
password_verified=password_verified,
|
||||
)
|
||||
|
||||
|
||||
def extract_archive_upload(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
filename: str,
|
||||
folder: str | None,
|
||||
campaign_id: str | None,
|
||||
selected_paths: Iterable[str] | None = None,
|
||||
password: str | None = None,
|
||||
conflict_strategy: str = "reject",
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
encryption_vault_id: str | None = None,
|
||||
max_entries: int = ARCHIVE_UPLOAD_MAX_ENTRIES,
|
||||
max_file_bytes: int = 50 * 1024 * 1024,
|
||||
max_expanded_bytes: int = 2 * 1024 * 1024 * 1024,
|
||||
max_expansion_ratio: int = 100,
|
||||
) -> list[UploadedStoredFile]:
|
||||
inspection = inspect_archive(
|
||||
archive_data,
|
||||
filename=filename,
|
||||
password=password,
|
||||
max_entries=max_entries,
|
||||
max_expanded_bytes=max_expanded_bytes,
|
||||
max_expansion_ratio=max_expansion_ratio,
|
||||
)
|
||||
if inspection.requires_password and not inspection.password_verified:
|
||||
raise ArchivePasswordError("Archive password is required")
|
||||
selected_files = _selected_file_paths(inspection.entries, selected_paths)
|
||||
if not selected_files:
|
||||
raise FileStorageError("Select at least one archive file to import")
|
||||
actual_total_limit = min(
|
||||
max_expanded_bytes,
|
||||
inspection.compressed_size_bytes * max_expansion_ratio,
|
||||
)
|
||||
if inspection.archive_format == "zip":
|
||||
members = _read_selected_zip_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
password=password,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
else:
|
||||
members = _read_selected_tar_members(
|
||||
archive_data,
|
||||
selected_files=selected_files,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=actual_total_limit,
|
||||
)
|
||||
return _store_archive_members(
|
||||
session,
|
||||
members=members,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
folder=folder,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
|
||||
|
||||
def extract_zip_upload(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -66,59 +247,416 @@ def extract_zip_upload(
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
zip_data: bytes,
|
||||
zip_data: bytes | str | PathLike[str],
|
||||
folder: str | None,
|
||||
campaign_id: str | None,
|
||||
conflict_strategy: str = "reject",
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
max_files: int = 1000,
|
||||
encryption_vault_id: str | None = None,
|
||||
max_files: int = ZIP_UPLOAD_MAX_FILES,
|
||||
max_file_bytes: int = 50 * 1024 * 1024,
|
||||
max_total_bytes: int = 250 * 1024 * 1024,
|
||||
) -> list[UploadedStoredFile]:
|
||||
uploaded: list[UploadedStoredFile] = []
|
||||
total = 0
|
||||
base_folder = normalize_folder(folder)
|
||||
"""Backward-compatible wrapper for the original immediate ZIP endpoint."""
|
||||
|
||||
return extract_archive_upload(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
archive_data=zip_data,
|
||||
filename="archive.zip",
|
||||
folder=folder,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
metadata=metadata,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
max_entries=max_files,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_expanded_bytes=max_total_bytes,
|
||||
)
|
||||
|
||||
|
||||
def _inspect_zip(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
password: str | None,
|
||||
) -> tuple[list[ArchiveEntry], bool, bool]:
|
||||
try:
|
||||
with zipfile.ZipFile(BytesIO(zip_data)) as archive:
|
||||
infos = [info for info in archive.infolist() if not info.is_dir()]
|
||||
if len(infos) > max_files:
|
||||
raise FileStorageError(f"ZIP contains too many files (limit {max_files})")
|
||||
for info in infos:
|
||||
if info.flag_bits & 0x1:
|
||||
raise FileStorageError("Encrypted ZIP uploads are not supported")
|
||||
if info.file_size < 0:
|
||||
raise FileStorageError("Invalid ZIP member")
|
||||
if info.file_size > max_file_bytes:
|
||||
raise FileStorageError(f"ZIP member {info.filename!r} exceeds per-file limit")
|
||||
if total + info.file_size > max_total_bytes:
|
||||
raise FileStorageError("ZIP is too large after extraction")
|
||||
inner_path = normalize_logical_path(info.filename)
|
||||
target_path = f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
data, total = _read_zip_member(
|
||||
archive,
|
||||
info,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
)
|
||||
uploaded.append(
|
||||
create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
filename=filename_from_path(inner_path),
|
||||
data=data,
|
||||
display_path=target_path,
|
||||
content_type=mimetypes.guess_type(inner_path)[0] or "application/octet-stream",
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
is_admin=is_admin,
|
||||
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||
infos = archive.infolist()
|
||||
entries = [_zip_entry(info) for info in infos]
|
||||
encrypted_info = next(
|
||||
(
|
||||
info
|
||||
for info in infos
|
||||
if not info.is_dir() and bool(info.flag_bits & 0x1)
|
||||
),
|
||||
None,
|
||||
)
|
||||
if encrypted_info is None:
|
||||
return entries, False, True
|
||||
if not password:
|
||||
return entries, True, False
|
||||
try:
|
||||
with archive.open(
|
||||
encrypted_info, pwd=password.encode("utf-8")
|
||||
) as member:
|
||||
member.read(1)
|
||||
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise ArchivePasswordError("Archive password is incorrect") from exc
|
||||
return entries, True, True
|
||||
except ArchivePasswordError:
|
||||
raise
|
||||
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise FileStorageError("Invalid ZIP upload") from exc
|
||||
|
||||
|
||||
def _inspect_tar(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
) -> list[ArchiveEntry]:
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
entries: list[ArchiveEntry] = []
|
||||
for member in archive.getmembers():
|
||||
path = _safe_member_path(member.name)
|
||||
if member.isdir():
|
||||
entries.append(
|
||||
ArchiveEntry(path=path, kind="directory", size_bytes=0)
|
||||
)
|
||||
continue
|
||||
if not member.isfile():
|
||||
raise FileStorageError(
|
||||
f"Archive member {member.name!r} is not a regular file or directory"
|
||||
)
|
||||
entries.append(
|
||||
ArchiveEntry(
|
||||
path=path,
|
||||
kind="file",
|
||||
size_bytes=max(0, int(member.size)),
|
||||
)
|
||||
)
|
||||
except zipfile.BadZipFile as exc:
|
||||
raise FileStorageError("Invalid ZIP upload") from exc
|
||||
return entries
|
||||
except FileStorageError:
|
||||
raise
|
||||
except (OSError, tarfile.TarError) as exc:
|
||||
raise FileStorageError("Invalid TAR upload") from exc
|
||||
|
||||
|
||||
def _zip_entry(info: zipfile.ZipInfo) -> ArchiveEntry:
|
||||
path = _safe_member_path(info.filename)
|
||||
unix_mode = (info.external_attr >> 16) & 0xFFFF
|
||||
file_type = stat.S_IFMT(unix_mode)
|
||||
if file_type and not (
|
||||
stat.S_ISREG(unix_mode) or stat.S_ISDIR(unix_mode)
|
||||
):
|
||||
raise FileStorageError(
|
||||
f"Archive member {info.filename!r} is not a regular file or directory"
|
||||
)
|
||||
if info.file_size < 0 or info.compress_size < 0:
|
||||
raise FileStorageError(f"Archive member {info.filename!r} has invalid size metadata")
|
||||
return ArchiveEntry(
|
||||
path=path,
|
||||
kind="directory" if info.is_dir() else "file",
|
||||
size_bytes=0 if info.is_dir() else int(info.file_size),
|
||||
compressed_size_bytes=(
|
||||
None if info.is_dir() else int(info.compress_size)
|
||||
),
|
||||
encrypted=bool(info.flag_bits & 0x1),
|
||||
)
|
||||
|
||||
|
||||
def _validate_archive_limits(
|
||||
entries: list[ArchiveEntry],
|
||||
*,
|
||||
compressed_size: int,
|
||||
max_entries: int,
|
||||
max_expanded_bytes: int,
|
||||
max_expansion_ratio: int,
|
||||
) -> None:
|
||||
if len(entries) > max_entries:
|
||||
raise FileStorageError(
|
||||
f"Archive contains too many entries (limit {max_entries})"
|
||||
)
|
||||
seen: dict[str, str] = {}
|
||||
expanded_size = 0
|
||||
for entry in entries:
|
||||
previous_kind = seen.get(entry.path)
|
||||
if previous_kind is not None:
|
||||
raise FileStorageError(
|
||||
f"Archive contains duplicate path {entry.path!r}"
|
||||
)
|
||||
seen[entry.path] = entry.kind
|
||||
if entry.kind == "file":
|
||||
expanded_size += entry.size_bytes
|
||||
if expanded_size > max_expanded_bytes:
|
||||
raise FileStorageError(
|
||||
"Archive is too large after extraction "
|
||||
f"(limit {max_expanded_bytes} bytes)"
|
||||
)
|
||||
if expanded_size and (
|
||||
compressed_size <= 0
|
||||
or expanded_size > compressed_size * max_expansion_ratio
|
||||
):
|
||||
raise FileStorageError(
|
||||
"Archive expansion ratio exceeds "
|
||||
f"{max_expansion_ratio}:1"
|
||||
)
|
||||
|
||||
|
||||
def _with_derived_directories(
|
||||
entries: list[ArchiveEntry],
|
||||
) -> list[ArchiveEntry]:
|
||||
by_path = {entry.path: entry for entry in entries}
|
||||
for entry in entries:
|
||||
parent = PurePosixPath(entry.path).parent
|
||||
while str(parent) not in {"", "."}:
|
||||
path = str(parent)
|
||||
existing = by_path.get(path)
|
||||
if existing and existing.kind != "directory":
|
||||
raise FileStorageError(
|
||||
f"Archive path {path!r} is both a file and a directory"
|
||||
)
|
||||
by_path.setdefault(
|
||||
path,
|
||||
ArchiveEntry(path=path, kind="directory", size_bytes=0),
|
||||
)
|
||||
parent = parent.parent
|
||||
return sorted(
|
||||
by_path.values(),
|
||||
key=lambda entry: (
|
||||
tuple(entry.path.casefold().split("/")),
|
||||
entry.kind != "directory",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _selected_file_paths(
|
||||
entries: tuple[ArchiveEntry, ...],
|
||||
selected_paths: Iterable[str] | None,
|
||||
) -> set[str]:
|
||||
file_paths = {entry.path for entry in entries if entry.kind == "file"}
|
||||
if selected_paths is None:
|
||||
return file_paths
|
||||
known = {entry.path: entry for entry in entries}
|
||||
normalized = {_safe_member_path(path) for path in selected_paths}
|
||||
unknown = normalized - known.keys()
|
||||
if unknown:
|
||||
raise FileStorageError(
|
||||
f"Archive selection contains unknown path {sorted(unknown)[0]!r}"
|
||||
)
|
||||
selected_files: set[str] = set()
|
||||
for path in normalized:
|
||||
entry = known[path]
|
||||
if entry.kind == "file":
|
||||
selected_files.add(path)
|
||||
continue
|
||||
prefix = f"{path}/"
|
||||
selected_files.update(
|
||||
file_path
|
||||
for file_path in file_paths
|
||||
if file_path.startswith(prefix)
|
||||
)
|
||||
return selected_files
|
||||
|
||||
|
||||
def _read_selected_zip_members(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
selected_files: set[str],
|
||||
password: str | None,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
total = 0
|
||||
try:
|
||||
with pyzipper.AESZipFile(_archive_source(archive_data)) as archive:
|
||||
for info in archive.infolist():
|
||||
if info.is_dir():
|
||||
continue
|
||||
path = _safe_member_path(info.filename)
|
||||
if path not in selected_files:
|
||||
continue
|
||||
pwd = password.encode("utf-8") if password else None
|
||||
try:
|
||||
with archive.open(info, pwd=pwd) as source:
|
||||
data, total = _read_member(
|
||||
source,
|
||||
path=path,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
)
|
||||
except (RuntimeError, ValueError, zipfile.BadZipFile) as exc:
|
||||
if info.flag_bits & 0x1:
|
||||
raise ArchivePasswordError(
|
||||
"Archive password is incorrect"
|
||||
) from exc
|
||||
raise
|
||||
result.append((path, data))
|
||||
except (FileStorageError, ArchivePasswordError):
|
||||
raise
|
||||
except (OSError, ValueError, zipfile.BadZipFile) as exc:
|
||||
raise FileStorageError("ZIP extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_selected_tar_members(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
*,
|
||||
selected_files: set[str],
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
) -> list[tuple[str, bytes]]:
|
||||
result: list[tuple[str, bytes]] = []
|
||||
total = 0
|
||||
try:
|
||||
with _open_tar(archive_data) as archive:
|
||||
for member in archive.getmembers():
|
||||
if not member.isfile():
|
||||
continue
|
||||
path = _safe_member_path(member.name)
|
||||
if path not in selected_files:
|
||||
continue
|
||||
source = archive.extractfile(member)
|
||||
if source is None:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} could not be read"
|
||||
)
|
||||
with source:
|
||||
data, total = _read_member(
|
||||
source,
|
||||
path=path,
|
||||
max_file_bytes=max_file_bytes,
|
||||
max_total_bytes=max_total_bytes,
|
||||
current_total=total,
|
||||
)
|
||||
result.append((path, data))
|
||||
except FileStorageError:
|
||||
raise
|
||||
except (OSError, tarfile.TarError) as exc:
|
||||
raise FileStorageError("TAR extraction failed") from exc
|
||||
return result
|
||||
|
||||
|
||||
def _read_member(
|
||||
source: BinaryIO,
|
||||
*,
|
||||
path: str,
|
||||
max_file_bytes: int,
|
||||
max_total_bytes: int,
|
||||
current_total: int,
|
||||
) -> tuple[bytes, int]:
|
||||
parts: list[bytes] = []
|
||||
actual_size = 0
|
||||
while True:
|
||||
read_size = min(
|
||||
_ARCHIVE_READ_CHUNK_SIZE,
|
||||
max_file_bytes + 1 - actual_size,
|
||||
)
|
||||
chunk = source.read(read_size)
|
||||
if not chunk:
|
||||
break
|
||||
actual_size += len(chunk)
|
||||
if actual_size > max_file_bytes:
|
||||
raise FileStorageError(
|
||||
f"Archive member {path!r} exceeds per-file limit"
|
||||
)
|
||||
if current_total + actual_size > max_total_bytes:
|
||||
raise FileStorageError("Archive is too large after extraction")
|
||||
parts.append(chunk)
|
||||
return b"".join(parts), current_total + actual_size
|
||||
|
||||
|
||||
def _store_archive_members(
|
||||
session: Session,
|
||||
*,
|
||||
members: Iterable[tuple[str, bytes]],
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
folder: str | None,
|
||||
campaign_id: str | None,
|
||||
conflict_strategy: str,
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None,
|
||||
metadata: dict[str, Any] | None,
|
||||
is_admin: bool,
|
||||
encryption_vault_id: str | None,
|
||||
) -> list[UploadedStoredFile]:
|
||||
uploaded: list[UploadedStoredFile] = []
|
||||
base_folder = normalize_folder(folder)
|
||||
for inner_path, data in members:
|
||||
target_path = (
|
||||
f"{base_folder}/{inner_path}" if base_folder else inner_path
|
||||
)
|
||||
uploaded.append(
|
||||
create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
filename=filename_from_path(inner_path),
|
||||
data=data,
|
||||
display_path=target_path,
|
||||
content_type=mimetypes.guess_type(inner_path)[0]
|
||||
or "application/octet-stream",
|
||||
metadata=metadata,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
conflict_resolutions=conflict_resolutions,
|
||||
is_admin=is_admin,
|
||||
encryption_vault_id=encryption_vault_id,
|
||||
)
|
||||
)
|
||||
return uploaded
|
||||
|
||||
|
||||
def _safe_member_path(value: str) -> str:
|
||||
raw = str(value or "").replace("\\", "/").strip()
|
||||
if (
|
||||
not raw
|
||||
or "\x00" in raw
|
||||
or raw.startswith("/")
|
||||
or _WINDOWS_DRIVE_RE.match(raw)
|
||||
):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
if any(part == ".." for part in raw.split("/")):
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}")
|
||||
try:
|
||||
return normalize_logical_path(raw.rstrip("/"))
|
||||
except ValueError as exc:
|
||||
raise FileStorageError(f"Unsafe archive member path {value!r}") from exc
|
||||
|
||||
|
||||
def _archive_size(archive_data: bytes | str | PathLike[str]) -> int:
|
||||
if isinstance(archive_data, bytes):
|
||||
return len(archive_data)
|
||||
try:
|
||||
return Path(archive_data).stat().st_size
|
||||
except OSError as exc:
|
||||
raise FileStorageError("Archive upload could not be read") from exc
|
||||
|
||||
|
||||
def _archive_source(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
) -> BytesIO | str | PathLike[str]:
|
||||
return BytesIO(archive_data) if isinstance(archive_data, bytes) else archive_data
|
||||
|
||||
|
||||
def _open_tar(
|
||||
archive_data: bytes | str | PathLike[str],
|
||||
) -> tarfile.TarFile:
|
||||
if isinstance(archive_data, bytes):
|
||||
return tarfile.open(fileobj=BytesIO(archive_data), mode="r:*")
|
||||
try:
|
||||
return tarfile.open(name=archive_data, mode="r:*")
|
||||
except OSError as exc:
|
||||
raise FileStorageError("Archive upload could not be read") from exc
|
||||
|
||||
@@ -1,160 +1,32 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Protocol
|
||||
from govoplan_core.core.object_storage import (
|
||||
LocalFilesystemStorageBackend,
|
||||
S3StorageBackend,
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectInfo,
|
||||
StorageObjectMissing,
|
||||
StorageObjectPage,
|
||||
configured_storage_backend,
|
||||
)
|
||||
|
||||
from govoplan_files.backend.runtime import settings
|
||||
|
||||
|
||||
class StorageBackendError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class StorageBackend(Protocol):
|
||||
name: str
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None: ...
|
||||
def get_bytes(self, key: str) -> bytes: ...
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]: ...
|
||||
def delete(self, key: str) -> None: ...
|
||||
def exists(self, key: str) -> bool: ...
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class LocalFilesystemStorageBackend:
|
||||
root: Path
|
||||
fallback_roots: tuple[Path, ...] = field(default_factory=tuple)
|
||||
name: str = "local"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.root = self.root.expanduser().resolve()
|
||||
self.fallback_roots = tuple(root.expanduser().resolve() for root in self.fallback_roots if root)
|
||||
self.root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _path_for_root(self, root: Path, key: str) -> Path:
|
||||
path = (root / key).resolve()
|
||||
if not path.is_relative_to(root):
|
||||
raise StorageBackendError("Storage key escapes local storage root")
|
||||
return path
|
||||
|
||||
def _path(self, key: str) -> Path:
|
||||
return self._path_for_root(self.root, key)
|
||||
|
||||
def _readable_path(self, key: str) -> Path:
|
||||
primary = self._path(key)
|
||||
if primary.exists() and primary.is_file():
|
||||
return primary
|
||||
for root in self.fallback_roots:
|
||||
candidate = self._path_for_root(root, key)
|
||||
if candidate.exists() and candidate.is_file():
|
||||
return candidate
|
||||
raise StorageBackendError("Stored object does not exist")
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
||||
path = self._path(key)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
|
||||
def get_bytes(self, key: str) -> bytes:
|
||||
return self._readable_path(key).read_bytes()
|
||||
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
||||
path = self._readable_path(key)
|
||||
with path.open("rb") as handle:
|
||||
while True:
|
||||
chunk = handle.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
yield chunk
|
||||
|
||||
def delete(self, key: str) -> None:
|
||||
path = self._path(key)
|
||||
if path.exists() and path.is_file():
|
||||
path.unlink()
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
try:
|
||||
self._readable_path(key)
|
||||
except StorageBackendError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class S3StorageBackend:
|
||||
bucket: str
|
||||
endpoint_url: str
|
||||
region_name: str
|
||||
access_key_id: str
|
||||
secret_access_key: str
|
||||
name: str = "s3"
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
try:
|
||||
import boto3
|
||||
except ModuleNotFoundError as exc:
|
||||
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=self.endpoint_url,
|
||||
region_name=self.region_name,
|
||||
aws_access_key_id=self.access_key_id,
|
||||
aws_secret_access_key=self.secret_access_key,
|
||||
)
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
||||
kwargs = {"Bucket": self.bucket, "Key": key, "Body": data}
|
||||
if content_type:
|
||||
kwargs["ContentType"] = content_type
|
||||
self.client.put_object(**kwargs)
|
||||
|
||||
def get_bytes(self, key: str) -> bytes:
|
||||
try:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
return obj["Body"].read()
|
||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
||||
try:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
body = obj["Body"]
|
||||
while True:
|
||||
chunk = body.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
yield chunk
|
||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
|
||||
def delete(self, key: str) -> None:
|
||||
self.client.delete_object(Bucket=self.bucket, Key=key)
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
try:
|
||||
self.client.head_object(Bucket=self.bucket, Key=key)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _fallback_roots() -> tuple[Path, ...]:
|
||||
raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
|
||||
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())
|
||||
|
||||
|
||||
def get_storage_backend() -> StorageBackend:
|
||||
backend = settings.file_storage_backend.lower().strip()
|
||||
if backend in {"local", "filesystem", "fs"}:
|
||||
return LocalFilesystemStorageBackend(Path(settings.file_storage_local_root), fallback_roots=_fallback_roots())
|
||||
if backend in {"s3", "garage"}:
|
||||
return S3StorageBackend(
|
||||
bucket=settings.file_storage_s3_bucket or settings.s3_bucket,
|
||||
endpoint_url=settings.file_storage_s3_endpoint_url or settings.s3_endpoint_url,
|
||||
region_name=settings.file_storage_s3_region or settings.s3_region,
|
||||
access_key_id=settings.file_storage_s3_access_key_id or settings.s3_access_key_id,
|
||||
secret_access_key=settings.file_storage_s3_secret_access_key or settings.s3_secret_access_key,
|
||||
)
|
||||
raise StorageBackendError(f"Unsupported file storage backend: {settings.file_storage_backend}")
|
||||
"""Return the deployment-owned backend for Files-managed objects."""
|
||||
|
||||
return configured_storage_backend(settings)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"LocalFilesystemStorageBackend",
|
||||
"S3StorageBackend",
|
||||
"StorageBackend",
|
||||
"StorageBackendError",
|
||||
"StorageObjectInfo",
|
||||
"StorageObjectMissing",
|
||||
"StorageObjectPage",
|
||||
"get_storage_backend",
|
||||
]
|
||||
|
||||
@@ -12,9 +12,17 @@ from typing import Any, Iterator
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob, list_assets_for_user, read_asset_bytes
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
|
||||
from govoplan_files.backend.storage.backends import StorageBackend, get_storage_backend
|
||||
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
ensure_blob_is_readable,
|
||||
read_verified_blob_bytes,
|
||||
)
|
||||
|
||||
|
||||
MANAGED_SOURCE_PREFIX = "managed:"
|
||||
@@ -34,10 +42,17 @@ class ManagedAttachmentFile:
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
content_type: str | None
|
||||
linked_to_campaign: bool = True
|
||||
source_provenance: dict[str, Any] | None = None
|
||||
source_revision: str | None = None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
payload = asdict(self)
|
||||
payload.pop("local_path", None)
|
||||
if payload.get("source_provenance") is None:
|
||||
payload.pop("source_provenance", None)
|
||||
if payload.get("source_revision") is None:
|
||||
payload.pop("source_revision", None)
|
||||
return payload
|
||||
|
||||
|
||||
@@ -47,6 +62,7 @@ class PreparedCampaignSnapshot:
|
||||
raw_json: dict[str, Any]
|
||||
managed_files_by_local_path: dict[str, ManagedAttachmentFile]
|
||||
shared_assets: list[FileAsset]
|
||||
candidate_assets: list[FileAsset]
|
||||
|
||||
|
||||
def parse_managed_source(value: object) -> tuple[str, str] | None:
|
||||
@@ -108,6 +124,99 @@ def _iter_rule_dicts(attachments: dict[str, Any], raw_json: dict[str, Any]):
|
||||
yield rule
|
||||
|
||||
|
||||
def _managed_base_sources(raw_json: dict[str, Any]) -> list[tuple[str, str, str]]:
|
||||
attachments = raw_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
return []
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
return []
|
||||
sources: list[tuple[str, str, str]] = []
|
||||
seen: set[tuple[str, str, str]] = set()
|
||||
for item in base_paths:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
key = (owner_type, owner_id, logical_root)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
sources.append(key)
|
||||
return sources
|
||||
|
||||
|
||||
def _campaign_linked_asset_ids(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
asset_ids: list[str],
|
||||
) -> set[str]:
|
||||
if not asset_ids:
|
||||
return set()
|
||||
linked: set[str] = set()
|
||||
for offset in range(0, len(asset_ids), 900):
|
||||
chunk = asset_ids[offset : offset + 900]
|
||||
rows = (
|
||||
session.query(FileShare.file_asset_id)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id.in_(chunk),
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
linked.update(row[0] for row in rows)
|
||||
return linked
|
||||
|
||||
|
||||
def _candidate_assets_for_managed_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
raw_json: dict[str, Any],
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
shared_assets: list[FileAsset],
|
||||
) -> tuple[list[FileAsset], set[str]]:
|
||||
assets_by_id: dict[str, FileAsset] = {asset.id: asset for asset in shared_assets}
|
||||
for owner_type, owner_id, logical_root in _managed_base_sources(raw_json):
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
for asset in list_assets_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=logical_root or None,
|
||||
is_admin=is_admin,
|
||||
):
|
||||
assets_by_id.setdefault(asset.id, asset)
|
||||
candidate_assets = sorted(assets_by_id.values(), key=lambda asset: (asset.display_path, asset.updated_at, asset.id))
|
||||
linked_ids = _campaign_linked_asset_ids(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
asset_ids=[asset.id for asset in candidate_assets],
|
||||
)
|
||||
return candidate_assets, linked_ids
|
||||
|
||||
|
||||
def _selected_base_path(
|
||||
rule: dict[str, Any],
|
||||
prepared_by_id: dict[str, tuple[str, str]],
|
||||
@@ -127,6 +236,175 @@ def _selected_base_path(
|
||||
return None
|
||||
|
||||
|
||||
def _prepared_attachment_config(raw_json: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any], list[Any]]:
|
||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||
attachments = prepared_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
attachments = {}
|
||||
prepared_json["attachments"] = attachments
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
base_paths = []
|
||||
return prepared_json, attachments, base_paths
|
||||
|
||||
|
||||
def _campaign_snapshot_assets(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
prepared_json: dict[str, Any],
|
||||
include_unlinked_candidates: bool,
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
) -> tuple[list[FileAsset], list[FileAsset], set[str]]:
|
||||
shared_assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id="",
|
||||
campaign_id=campaign_id,
|
||||
is_admin=True,
|
||||
)
|
||||
if not include_unlinked_candidates:
|
||||
return shared_assets, shared_assets, {asset.id for asset in shared_assets}
|
||||
candidate_assets, linked_asset_ids = _candidate_assets_for_managed_sources(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
raw_json=prepared_json,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
shared_assets=shared_assets,
|
||||
)
|
||||
return shared_assets, candidate_assets, linked_asset_ids
|
||||
|
||||
|
||||
def _assets_grouped_by_owner(assets: list[FileAsset]) -> dict[tuple[str, str], list[FileAsset]]:
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||
for asset in assets:
|
||||
owner_id = _asset_owner_id(asset)
|
||||
if owner_id:
|
||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||
return assets_by_owner
|
||||
|
||||
|
||||
def _materialize_managed_asset(
|
||||
asset: FileAsset,
|
||||
*,
|
||||
owner_id: str,
|
||||
logical_root: str,
|
||||
local_root: Path,
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
backend: StorageBackend | None,
|
||||
include_bytes: bool,
|
||||
linked_asset_ids: set[str],
|
||||
) -> tuple[str, ManagedAttachmentFile] | None:
|
||||
relative_path = _relative_asset_path(asset, logical_root)
|
||||
if not relative_path:
|
||||
return None
|
||||
target = _safe_local_target(local_root, relative_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
version, blob = version_blobs[asset.id]
|
||||
ensure_blob_is_readable(blob)
|
||||
if include_bytes:
|
||||
data = read_verified_blob_bytes(blob, backend=backend) if backend else b""
|
||||
target.write_bytes(data)
|
||||
else:
|
||||
target.touch()
|
||||
local_key = str(target.resolve())
|
||||
return local_key, ManagedAttachmentFile(
|
||||
local_path=local_key,
|
||||
asset_id=asset.id,
|
||||
version_id=version.id,
|
||||
blob_id=blob.id,
|
||||
display_path=asset.display_path,
|
||||
relative_path=normalize_logical_path(relative_path),
|
||||
filename=asset.filename,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=owner_id,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
linked_to_campaign=asset.id in linked_asset_ids,
|
||||
source_provenance=source_provenance_from_metadata(asset.metadata_),
|
||||
source_revision=source_revision_from_metadata(asset.metadata_),
|
||||
)
|
||||
|
||||
|
||||
def _prepare_managed_base_paths(
|
||||
base_paths: list[Any],
|
||||
*,
|
||||
materialized_root: Path,
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]],
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
backend: StorageBackend | None,
|
||||
include_bytes: bool,
|
||||
linked_asset_ids: set[str],
|
||||
) -> tuple[
|
||||
dict[str, ManagedAttachmentFile],
|
||||
dict[str, tuple[str, str]],
|
||||
dict[str, list[tuple[str, str]]],
|
||||
tuple[str, str] | None,
|
||||
]:
|
||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||
first_prepared: tuple[str, str] | None = None
|
||||
for index, item in enumerate(base_paths):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||
local_root.mkdir(parents=True, exist_ok=True)
|
||||
local_root_string = str(local_root.resolve())
|
||||
prepared = (base_path_id, local_root_string)
|
||||
prepared_by_id[base_path_id] = prepared
|
||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||
if first_prepared is None:
|
||||
first_prepared = prepared
|
||||
item["path"] = local_root_string
|
||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||
materialized = _materialize_managed_asset(
|
||||
asset,
|
||||
owner_id=owner_id,
|
||||
logical_root=logical_root,
|
||||
local_root=local_root,
|
||||
version_blobs=version_blobs,
|
||||
backend=backend,
|
||||
include_bytes=include_bytes,
|
||||
linked_asset_ids=linked_asset_ids,
|
||||
)
|
||||
if materialized is not None:
|
||||
local_key, managed_file = materialized
|
||||
manifest[local_key] = managed_file
|
||||
return manifest, prepared_by_id, prepared_by_old_path, first_prepared
|
||||
|
||||
|
||||
def _rewrite_managed_attachment_rules(
|
||||
attachments: dict[str, Any],
|
||||
prepared_json: dict[str, Any],
|
||||
*,
|
||||
prepared_by_id: dict[str, tuple[str, str]],
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]],
|
||||
first_prepared: tuple[str, str] | None,
|
||||
) -> None:
|
||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||
if selected is None:
|
||||
continue
|
||||
base_path_id, local_root_string = selected
|
||||
rule["base_path_id"] = base_path_id
|
||||
rule["base_dir"] = local_root_string
|
||||
if first_prepared is not None:
|
||||
attachments["base_path"] = first_prepared[1]
|
||||
|
||||
|
||||
def prepare_campaign_snapshot(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -135,6 +413,9 @@ def prepare_campaign_snapshot(
|
||||
raw_json: dict[str, Any],
|
||||
destination: Path,
|
||||
include_bytes: bool,
|
||||
include_unlinked_candidates: bool = False,
|
||||
user_id: str = "",
|
||||
is_admin: bool = False,
|
||||
) -> PreparedCampaignSnapshot:
|
||||
"""Create a temporary file-oriented campaign snapshot for managed attachments.
|
||||
|
||||
@@ -150,94 +431,35 @@ def prepare_campaign_snapshot(
|
||||
materialized_root = destination / "managed-attachments"
|
||||
materialized_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||
attachments = prepared_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
attachments = {}
|
||||
prepared_json["attachments"] = attachments
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
base_paths = []
|
||||
|
||||
shared_assets = list_assets_for_user(
|
||||
prepared_json, attachments, base_paths = _prepared_attachment_config(raw_json)
|
||||
shared_assets, candidate_assets, linked_asset_ids = _campaign_snapshot_assets(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id="",
|
||||
campaign_id=campaign_id,
|
||||
is_admin=True,
|
||||
prepared_json=prepared_json,
|
||||
include_unlinked_candidates=include_unlinked_candidates,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
assets_by_owner = _assets_grouped_by_owner(candidate_assets)
|
||||
version_blobs = current_versions_and_blobs(session, candidate_assets)
|
||||
backend = get_storage_backend() if include_bytes else None
|
||||
manifest, prepared_by_id, prepared_by_old_path, first_prepared = _prepare_managed_base_paths(
|
||||
base_paths,
|
||||
materialized_root=materialized_root,
|
||||
assets_by_owner=assets_by_owner,
|
||||
version_blobs=version_blobs,
|
||||
backend=backend,
|
||||
include_bytes=include_bytes,
|
||||
linked_asset_ids=linked_asset_ids,
|
||||
)
|
||||
_rewrite_managed_attachment_rules(
|
||||
attachments,
|
||||
prepared_json,
|
||||
prepared_by_id=prepared_by_id,
|
||||
prepared_by_old_path=prepared_by_old_path,
|
||||
first_prepared=first_prepared,
|
||||
)
|
||||
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||
for asset in shared_assets:
|
||||
owner_id = _asset_owner_id(asset)
|
||||
if owner_id:
|
||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||
|
||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||
first_prepared: tuple[str, str] | None = None
|
||||
|
||||
for index, item in enumerate(base_paths):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||
local_root.mkdir(parents=True, exist_ok=True)
|
||||
local_root_string = str(local_root.resolve())
|
||||
|
||||
prepared = (base_path_id, local_root_string)
|
||||
prepared_by_id[base_path_id] = prepared
|
||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||
if first_prepared is None:
|
||||
first_prepared = prepared
|
||||
|
||||
item["path"] = local_root_string
|
||||
|
||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||
relative_path = _relative_asset_path(asset, logical_root)
|
||||
if not relative_path:
|
||||
continue
|
||||
target = _safe_local_target(local_root, relative_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if include_bytes:
|
||||
data, version, blob = read_asset_bytes(session, asset)
|
||||
target.write_bytes(data)
|
||||
else:
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
target.touch()
|
||||
local_key = str(target.resolve())
|
||||
manifest[local_key] = ManagedAttachmentFile(
|
||||
local_path=local_key,
|
||||
asset_id=asset.id,
|
||||
version_id=version.id,
|
||||
blob_id=blob.id,
|
||||
display_path=asset.display_path,
|
||||
relative_path=normalize_logical_path(relative_path),
|
||||
filename=asset.filename,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=owner_id,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
)
|
||||
|
||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||
if selected is None:
|
||||
continue
|
||||
base_path_id, local_root_string = selected
|
||||
rule["base_path_id"] = base_path_id
|
||||
rule["base_dir"] = local_root_string
|
||||
|
||||
if first_prepared is not None:
|
||||
attachments["base_path"] = first_prepared[1]
|
||||
|
||||
snapshot_path = destination / "campaign.json"
|
||||
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
@@ -246,6 +468,7 @@ def prepare_campaign_snapshot(
|
||||
raw_json=prepared_json,
|
||||
managed_files_by_local_path=manifest,
|
||||
shared_assets=shared_assets,
|
||||
candidate_assets=candidate_assets,
|
||||
)
|
||||
|
||||
|
||||
@@ -257,7 +480,10 @@ def prepared_campaign_snapshot(
|
||||
campaign_id: str,
|
||||
raw_json: dict[str, Any],
|
||||
include_bytes: bool,
|
||||
prefix: str = "multimailer-managed-campaign-",
|
||||
prefix: str = "govoplan-managed-campaign-",
|
||||
include_unlinked_candidates: bool = False,
|
||||
user_id: str = "",
|
||||
is_admin: bool = False,
|
||||
) -> Iterator[PreparedCampaignSnapshot]:
|
||||
temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
|
||||
try:
|
||||
@@ -268,6 +494,9 @@ def prepared_campaign_snapshot(
|
||||
raw_json=raw_json,
|
||||
destination=temp_dir,
|
||||
include_bytes=include_bytes,
|
||||
include_unlinked_candidates=include_unlinked_candidates,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
finally:
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
@@ -285,6 +514,53 @@ def managed_match_payloads(
|
||||
return payloads
|
||||
|
||||
|
||||
def share_assets_with_campaign(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
file_ids: list[str],
|
||||
user_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> list[dict[str, Any]]:
|
||||
unique_ids = list(dict.fromkeys(file_id for file_id in file_ids if file_id))
|
||||
if not unique_ids:
|
||||
return []
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
for file_id in unique_ids
|
||||
]
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
assets=assets,
|
||||
target_type="campaign",
|
||||
target_id=campaign_id,
|
||||
permission="read",
|
||||
user_id=user_id,
|
||||
)
|
||||
session.flush()
|
||||
return [
|
||||
{
|
||||
"id": share.id,
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
"created_at": share.created_at.isoformat() if share.created_at else None,
|
||||
"revoked_at": share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
}
|
||||
for share in shares
|
||||
]
|
||||
|
||||
|
||||
def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
|
||||
"""Return an attachment summary without temporary materialization paths.
|
||||
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePosixPath
|
||||
from typing import TYPE_CHECKING, Iterable
|
||||
from typing import Any, Iterable
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileVersion
|
||||
from govoplan_files.backend.storage.common import utcnow
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob, list_assets_for_user
|
||||
from govoplan_files.backend.storage.files import current_versions_and_blobs, list_assets_for_user
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from govoplan_campaign.backend.db.models import CampaignJob
|
||||
CampaignJobLike = Any
|
||||
|
||||
|
||||
def _candidate_match_keys(raw_match: str) -> set[str]:
|
||||
@@ -25,11 +25,20 @@ def _candidate_match_keys(raw_match: str) -> set[str]:
|
||||
AttachmentUseKey = tuple[str, str, str, str]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _AttachmentBatchRefs:
|
||||
managed_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||
asset_ids: set[str] = field(default_factory=set)
|
||||
version_ids: set[str] = field(default_factory=set)
|
||||
blob_ids: set[str] = field(default_factory=set)
|
||||
|
||||
|
||||
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
|
||||
return job_id, file_version_id, filename_used, stage
|
||||
|
||||
|
||||
def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJob], *, stage: str) -> set[AttachmentUseKey]:
|
||||
def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJobLike], *, stage: str) -> set[AttachmentUseKey]:
|
||||
job_ids = {job.id for job in jobs if job.id}
|
||||
if not job_ids:
|
||||
return set()
|
||||
@@ -72,13 +81,13 @@ def _known_use_keys_for_jobs(session: Session, jobs: Iterable[CampaignJob], *, s
|
||||
return keys
|
||||
|
||||
|
||||
def _known_use_keys(session: Session, job: CampaignJob, *, stage: str) -> set[AttachmentUseKey]:
|
||||
def _known_use_keys(session: Session, job: CampaignJobLike, *, stage: str) -> set[AttachmentUseKey]:
|
||||
return _known_use_keys_for_jobs(session, [job], stage=stage)
|
||||
|
||||
|
||||
def _add_use(
|
||||
session: Session,
|
||||
job: CampaignJob,
|
||||
job: CampaignJobLike,
|
||||
*,
|
||||
asset: FileAsset,
|
||||
version: FileVersion,
|
||||
@@ -118,7 +127,7 @@ def _add_use(
|
||||
|
||||
def record_campaign_attachment_uses_for_jobs(
|
||||
session: Session,
|
||||
jobs: Iterable[CampaignJob],
|
||||
jobs: Iterable[CampaignJobLike],
|
||||
*,
|
||||
stage: str = "built",
|
||||
) -> None:
|
||||
@@ -135,62 +144,91 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
if not job_list:
|
||||
return
|
||||
|
||||
managed_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
||||
refs = _collect_attachment_batch_refs(job_list)
|
||||
job_by_id = {job.id: job for job in job_list}
|
||||
asset_ids: set[str] = set()
|
||||
version_ids: set[str] = set()
|
||||
blob_ids: set[str] = set()
|
||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||
assets_by_id, versions_by_id, blobs_by_id = _load_managed_attachment_entities(session, refs)
|
||||
|
||||
for job in job_list:
|
||||
_record_managed_attachment_uses(
|
||||
session,
|
||||
job_by_id=job_by_id,
|
||||
managed_by_job=refs.managed_by_job,
|
||||
assets_by_id=assets_by_id,
|
||||
versions_by_id=versions_by_id,
|
||||
blobs_by_id=blobs_by_id,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
_record_fallback_attachment_uses(
|
||||
session,
|
||||
job_by_id=job_by_id,
|
||||
fallback_attachments_by_job=refs.fallback_attachments_by_job,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def _collect_attachment_batch_refs(jobs: list[CampaignJobLike]) -> _AttachmentBatchRefs:
|
||||
refs = _AttachmentBatchRefs()
|
||||
for job in jobs:
|
||||
attachments = job.resolved_attachments or []
|
||||
if not isinstance(attachments, list):
|
||||
continue
|
||||
for attachment in attachments:
|
||||
if not isinstance(attachment, dict):
|
||||
continue
|
||||
managed_matches = attachment.get("managed_matches")
|
||||
if isinstance(managed_matches, list) and managed_matches:
|
||||
for item in managed_matches:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
asset_id = str(item.get("asset_id") or "")
|
||||
version_id = str(item.get("version_id") or "")
|
||||
blob_id = str(item.get("blob_id") or "")
|
||||
if not asset_id or not version_id or not blob_id:
|
||||
continue
|
||||
managed_by_job[job.id].append(item)
|
||||
asset_ids.add(asset_id)
|
||||
version_ids.add(version_id)
|
||||
blob_ids.add(blob_id)
|
||||
else:
|
||||
fallback_attachments_by_job[job.id].append(attachment)
|
||||
if not _collect_managed_attachment_refs(refs, job.id, attachment):
|
||||
refs.fallback_attachments_by_job[job.id].append(attachment)
|
||||
return refs
|
||||
|
||||
assets_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileAsset).filter(FileAsset.id.in_(asset_ids)).all()
|
||||
} if asset_ids else {}
|
||||
versions_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileVersion).filter(FileVersion.id.in_(version_ids)).all()
|
||||
} if version_ids else {}
|
||||
blobs_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileBlob).filter(FileBlob.id.in_(blob_ids)).all()
|
||||
} if blob_ids else {}
|
||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||
|
||||
def _collect_managed_attachment_refs(refs: _AttachmentBatchRefs, job_id: str, attachment: dict[str, object]) -> bool:
|
||||
managed_matches = attachment.get("managed_matches")
|
||||
if not isinstance(managed_matches, list) or not managed_matches:
|
||||
return False
|
||||
for item in managed_matches:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
asset_id = str(item.get("asset_id") or "")
|
||||
version_id = str(item.get("version_id") or "")
|
||||
blob_id = str(item.get("blob_id") or "")
|
||||
if not asset_id or not version_id or not blob_id:
|
||||
continue
|
||||
refs.managed_by_job[job_id].append(item)
|
||||
refs.asset_ids.add(asset_id)
|
||||
refs.version_ids.add(version_id)
|
||||
refs.blob_ids.add(blob_id)
|
||||
return True
|
||||
|
||||
|
||||
def _load_managed_attachment_entities(
|
||||
session: Session,
|
||||
refs: _AttachmentBatchRefs,
|
||||
) -> tuple[dict[str, FileAsset], dict[str, FileVersion], dict[str, FileBlob]]:
|
||||
assets_by_id = {item.id: item for item in session.query(FileAsset).filter(FileAsset.id.in_(refs.asset_ids)).all()} if refs.asset_ids else {}
|
||||
versions_by_id = {item.id: item for item in session.query(FileVersion).filter(FileVersion.id.in_(refs.version_ids)).all()} if refs.version_ids else {}
|
||||
blobs_by_id = {item.id: item for item in session.query(FileBlob).filter(FileBlob.id.in_(refs.blob_ids)).all()} if refs.blob_ids else {}
|
||||
return assets_by_id, versions_by_id, blobs_by_id
|
||||
|
||||
|
||||
def _record_managed_attachment_uses(
|
||||
session: Session,
|
||||
*,
|
||||
job_by_id: dict[str, CampaignJobLike],
|
||||
managed_by_job: dict[str, list[dict[str, object]]],
|
||||
assets_by_id: dict[str, FileAsset],
|
||||
versions_by_id: dict[str, FileVersion],
|
||||
blobs_by_id: dict[str, FileBlob],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
for job_id, items in managed_by_job.items():
|
||||
job = job_by_id[job_id]
|
||||
for item in items:
|
||||
asset = assets_by_id.get(str(item.get("asset_id") or ""))
|
||||
version = versions_by_id.get(str(item.get("version_id") or ""))
|
||||
blob = blobs_by_id.get(str(item.get("blob_id") or ""))
|
||||
if not asset or not version or not blob:
|
||||
continue
|
||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||
continue
|
||||
if version.file_asset_id != asset.id or version.blob_id != blob.id:
|
||||
if not _managed_attachment_entities_match_job(job, asset, version, blob):
|
||||
continue
|
||||
_add_use(
|
||||
session,
|
||||
@@ -203,53 +241,109 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def _managed_attachment_entities_match_job(
|
||||
job: CampaignJobLike,
|
||||
asset: FileAsset | None,
|
||||
version: FileVersion | None,
|
||||
blob: FileBlob | None,
|
||||
) -> bool:
|
||||
if not asset or not version or not blob:
|
||||
return False
|
||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||
return False
|
||||
return version.file_asset_id == asset.id and version.blob_id == blob.id
|
||||
|
||||
|
||||
def _record_fallback_attachment_uses(
|
||||
session: Session,
|
||||
*,
|
||||
job_by_id: dict[str, CampaignJobLike],
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
|
||||
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {}
|
||||
for job_id, attachments in fallback_attachments_by_job.items():
|
||||
job = job_by_id[job_id]
|
||||
campaign_key = (job.tenant_id, job.campaign_id)
|
||||
by_key = assets_by_campaign.get(campaign_key)
|
||||
if by_key is None:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=job.tenant_id,
|
||||
user_id="",
|
||||
campaign_id=job.campaign_id,
|
||||
is_admin=True,
|
||||
)
|
||||
by_key = {}
|
||||
for asset in assets:
|
||||
by_key[asset.display_path.strip("/")] = asset
|
||||
by_key.setdefault(asset.filename, asset)
|
||||
assets_by_campaign[campaign_key] = by_key
|
||||
|
||||
by_key, version_blobs = _fallback_campaign_assets(
|
||||
session,
|
||||
job,
|
||||
campaign_key=campaign_key,
|
||||
assets_by_campaign=assets_by_campaign,
|
||||
version_blobs_by_campaign=version_blobs_by_campaign,
|
||||
)
|
||||
for attachment in attachments:
|
||||
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
||||
for raw in matches:
|
||||
if not isinstance(raw, str):
|
||||
continue
|
||||
asset = next((by_key[key] for key in _candidate_match_keys(raw) if key in by_key), None)
|
||||
if not asset:
|
||||
continue
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
_add_use(
|
||||
session,
|
||||
job,
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
filename_used=asset.filename,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
_record_fallback_attachment(session, job, attachment, by_key=by_key, version_blobs=version_blobs, stage=stage, known_keys=known_keys)
|
||||
|
||||
|
||||
def record_campaign_attachment_uses_for_job(session: Session, job: CampaignJob, *, stage: str = "built") -> None:
|
||||
def _fallback_campaign_assets(
|
||||
session: Session,
|
||||
job: CampaignJobLike,
|
||||
*,
|
||||
campaign_key: tuple[str, str],
|
||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]],
|
||||
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]],
|
||||
) -> tuple[dict[str, FileAsset], dict[str, tuple[FileVersion, FileBlob]]]:
|
||||
by_key = assets_by_campaign.get(campaign_key)
|
||||
if by_key is None:
|
||||
assets = list_assets_for_user(session, tenant_id=job.tenant_id, user_id="", campaign_id=job.campaign_id, is_admin=True)
|
||||
by_key = _asset_lookup_by_path_and_filename(assets)
|
||||
assets_by_campaign[campaign_key] = by_key
|
||||
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
|
||||
return by_key, version_blobs_by_campaign[campaign_key]
|
||||
|
||||
|
||||
def _asset_lookup_by_path_and_filename(assets: list[FileAsset]) -> dict[str, FileAsset]:
|
||||
by_key: dict[str, FileAsset] = {}
|
||||
for asset in assets:
|
||||
by_key[asset.display_path.strip("/")] = asset
|
||||
by_key.setdefault(asset.filename, asset)
|
||||
return by_key
|
||||
|
||||
|
||||
def _record_fallback_attachment(
|
||||
session: Session,
|
||||
job: CampaignJobLike,
|
||||
attachment: dict[str, object],
|
||||
*,
|
||||
by_key: dict[str, FileAsset],
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
||||
for raw in matches:
|
||||
if not isinstance(raw, str):
|
||||
continue
|
||||
asset = next((by_key[key] for key in _candidate_match_keys(raw) if key in by_key), None)
|
||||
if not asset:
|
||||
continue
|
||||
version_blob = version_blobs.get(asset.id)
|
||||
if not version_blob:
|
||||
continue
|
||||
version, blob = version_blob
|
||||
_add_use(
|
||||
session,
|
||||
job,
|
||||
asset=asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
filename_used=asset.filename,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def record_campaign_attachment_uses_for_job(session: Session, job: CampaignJobLike, *, stage: str = "built") -> None:
|
||||
"""Record immutable managed file versions used by one built/sent job."""
|
||||
|
||||
record_campaign_attachment_uses_for_jobs(session, [job], stage=stage)
|
||||
|
||||
|
||||
def mark_job_attachment_uses_sent(session: Session, job: CampaignJob) -> None:
|
||||
def mark_job_attachment_uses_sent(session: Session, job: CampaignJobLike) -> None:
|
||||
record_campaign_attachment_uses_for_job(session, job, stage="built")
|
||||
# Sessions use autoflush=False. Flush any compatibility-built evidence so
|
||||
# the following query can copy it to the sent stage in the same call.
|
||||
|
||||
@@ -0,0 +1,923 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from email.utils import parsedate_to_datetime
|
||||
from importlib import import_module
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
from urllib.parse import quote, unquote, urljoin, urlsplit
|
||||
import xml.etree.ElementTree as ET # nosec B405 - typing/element creation only; parsing uses defusedxml below.
|
||||
|
||||
from defusedxml import ElementTree as SafeElementTree
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
validate_outbound_host,
|
||||
validate_outbound_http_url,
|
||||
)
|
||||
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
ConnectorDeploymentConfigurationError,
|
||||
connector_ca_bundle_path,
|
||||
connector_secret_env_value,
|
||||
validate_connector_tls_metadata,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.sdk_peer_pinning import (
|
||||
SdkPeerPinningError,
|
||||
create_pinned_s3_client,
|
||||
install_pinned_smb_transport,
|
||||
pinned_smb_connection_cache,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorBrowseError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectorBrowseUnsupported(ConnectorBrowseError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorBrowseItem:
|
||||
kind: str
|
||||
name: str
|
||||
path: str
|
||||
external_id: str | None = None
|
||||
external_url: str | None = None
|
||||
size_bytes: int | None = None
|
||||
content_type: str | None = None
|
||||
modified_at: str | None = None
|
||||
etag: str | None = None
|
||||
metadata: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"name": self.name,
|
||||
"path": self.path,
|
||||
"external_id": self.external_id,
|
||||
"external_url": self.external_url,
|
||||
"size_bytes": self.size_bytes,
|
||||
"content_type": self.content_type,
|
||||
"modified_at": self.modified_at,
|
||||
"etag": self.etag,
|
||||
"metadata": dict(self.metadata),
|
||||
}
|
||||
|
||||
|
||||
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None, continuation_token: str | None = None) -> list[ConnectorBrowseItem]:
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
static_items = _static_listing(profile, path=browse_path, library_id=library_id)
|
||||
if static_items is not None:
|
||||
return static_items
|
||||
if profile.provider == "seafile":
|
||||
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||
return _browse_webdav(profile, path=browse_path)
|
||||
return _browse_seafile(profile, path=browse_path, library_id=library_id)
|
||||
if profile.provider in {"webdav", "nextcloud"} or _metadata_string(profile, "browse_protocol") == "webdav":
|
||||
return _browse_webdav(profile, path=browse_path)
|
||||
if profile.provider == "smb":
|
||||
return _browse_smb(profile, path=browse_path)
|
||||
if profile.provider == "s3":
|
||||
return _browse_s3(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
|
||||
raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet")
|
||||
|
||||
|
||||
def parse_webdav_multistatus(*, root_url: str, current_path: str, payload: str | bytes) -> list[ConnectorBrowseItem]:
|
||||
try:
|
||||
root = SafeElementTree.fromstring(payload)
|
||||
except SafeElementTree.ParseError as exc:
|
||||
raise ConnectorBrowseError("Connector returned invalid WebDAV XML") from exc
|
||||
root_path = _url_path_root(root_url)
|
||||
current = normalize_connector_browse_path(current_path)
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for response in root.findall("{DAV:}response"):
|
||||
href = response.findtext("{DAV:}href")
|
||||
if not href:
|
||||
continue
|
||||
relative_path = _relative_href_path(root_path, href)
|
||||
if relative_path == current:
|
||||
continue
|
||||
if current and not relative_path.startswith(current.rstrip("/") + "/"):
|
||||
continue
|
||||
parent = relative_path.rsplit("/", 1)[0] if "/" in relative_path else ""
|
||||
if parent != current:
|
||||
continue
|
||||
prop = _webdav_prop(response)
|
||||
is_collection = prop.find("{DAV:}resourcetype/{DAV:}collection") is not None if prop is not None else False
|
||||
name = _webdav_display_name(prop) or _path_name(relative_path)
|
||||
if not name:
|
||||
continue
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder" if is_collection else "file",
|
||||
name=name,
|
||||
path=relative_path,
|
||||
external_id=_text(prop, "{DAV:}getetag") or relative_path,
|
||||
size_bytes=None if is_collection else _int(_text(prop, "{DAV:}getcontentlength")),
|
||||
content_type=None if is_collection else _text(prop, "{DAV:}getcontenttype"),
|
||||
modified_at=_http_date(_text(prop, "{DAV:}getlastmodified")),
|
||||
etag=_text(prop, "{DAV:}getetag"),
|
||||
)
|
||||
)
|
||||
return sorted(items, key=lambda item: (item.kind != "library", item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def seafile_libraries_from_payload(payload: object) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(payload, list):
|
||||
raise ConnectorBrowseError("Seafile library response must be a list")
|
||||
libraries = [_seafile_library_item(item) for item in payload if isinstance(item, Mapping)]
|
||||
return sorted(libraries, key=lambda item: item.name.casefold())
|
||||
|
||||
|
||||
def seafile_directory_items_from_payload(payload: object, *, path: str | None = None) -> list[ConnectorBrowseItem]:
|
||||
if payload == "uptodate":
|
||||
return []
|
||||
if not isinstance(payload, list):
|
||||
raise ConnectorBrowseError("Seafile directory response must be a list")
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
items = [_seafile_directory_item(item, parent_path=browse_path) for item in payload if isinstance(item, Mapping)]
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _browse_seafile(profile: ConnectorProfile, *, path: str, library_id: str | None) -> list[ConnectorBrowseItem]:
|
||||
repo_id, dir_path = _seafile_repo_and_path(path=path, library_id=library_id)
|
||||
headers = _seafile_headers(profile)
|
||||
if not repo_id:
|
||||
payload = _request_json("GET", _seafile_url(profile, "api2/repos/"), headers=headers)
|
||||
return seafile_libraries_from_payload(payload)
|
||||
payload = _request_json(
|
||||
"GET",
|
||||
_seafile_url(profile, f"api2/repos/{quote(repo_id, safe='')}/dir/"),
|
||||
headers=headers,
|
||||
params={"p": "/" + dir_path if dir_path else "/"},
|
||||
)
|
||||
return seafile_directory_items_from_payload(payload, path=dir_path)
|
||||
|
||||
|
||||
def _browse_webdav(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||
root_url = _metadata_string(profile, "webdav_endpoint_url") or profile.endpoint_url
|
||||
if not root_url:
|
||||
raise ConnectorBrowseError("Connector profile does not define an endpoint URL")
|
||||
url = _webdav_url(root_url, path)
|
||||
headers = {"Depth": "1", "Content-Type": "application/xml; charset=utf-8"}
|
||||
auth: tuple[str, str] | None = None
|
||||
password = _profile_password(profile)
|
||||
token = _profile_token(profile)
|
||||
if profile.username and password:
|
||||
auth = (profile.username, password)
|
||||
elif token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref connector credentials need a runtime secret resolver before live browsing")
|
||||
body = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<propfind xmlns="DAV:">
|
||||
<prop>
|
||||
<displayname />
|
||||
<resourcetype />
|
||||
<getcontentlength />
|
||||
<getcontenttype />
|
||||
<getlastmodified />
|
||||
<getetag />
|
||||
</prop>
|
||||
</propfind>"""
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
"PROPFIND",
|
||||
url,
|
||||
headers=headers,
|
||||
content=body,
|
||||
auth=auth,
|
||||
timeout=15.0,
|
||||
label="WebDAV browse",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||
if response.status_code not in {200, 207}:
|
||||
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.content)
|
||||
|
||||
|
||||
def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||
location = _smb_location(profile)
|
||||
unc_path = _smb_unc_path(location, path)
|
||||
smbclient = _smbclient_module()
|
||||
try:
|
||||
entries = smbclient.scandir(unc_path, **_smb_client_kwargs(profile, location))
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"SMB connector browse failed: {exc}") from exc
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
try:
|
||||
with entries as iterator:
|
||||
for entry in iterator:
|
||||
name = _clean(getattr(entry, "name", None))
|
||||
if not name or name in {".", ".."}:
|
||||
continue
|
||||
try:
|
||||
is_dir = bool(entry.is_dir())
|
||||
except Exception:
|
||||
is_dir = False
|
||||
stat_result = _smb_entry_stat(entry)
|
||||
item_path = _join_browse_path(path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder" if is_dir else "file",
|
||||
name=name,
|
||||
path=item_path,
|
||||
external_id=f"{location.share}:{item_path}",
|
||||
size_bytes=None if is_dir else _smb_stat_size(stat_result),
|
||||
content_type=None if is_dir else mimetypes.guess_type(name)[0],
|
||||
modified_at=_smb_stat_modified_at(stat_result),
|
||||
etag=_smb_stat_revision(stat_result),
|
||||
metadata={
|
||||
"share": location.share,
|
||||
**({"server": location.server} if _metadata_bool(profile, "expose_server_metadata", default=False) else {}),
|
||||
},
|
||||
)
|
||||
)
|
||||
except ConnectorBrowseError:
|
||||
raise
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"SMB connector browse failed: {exc}") from exc
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _browse_s3(profile: ConnectorProfile, *, path: str, library_id: str | None, continuation_token: str | None) -> list[ConnectorBrowseItem]:
|
||||
client = _s3_client(profile)
|
||||
try:
|
||||
return _browse_s3_with_client(
|
||||
client,
|
||||
profile=profile,
|
||||
path=path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
def _browse_s3_with_client(
|
||||
client: Any,
|
||||
*,
|
||||
profile: ConnectorProfile,
|
||||
path: str,
|
||||
library_id: str | None,
|
||||
continuation_token: str | None,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
if not bucket:
|
||||
try:
|
||||
payload = client.list_buckets()
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||
buckets = payload.get("Buckets") if isinstance(payload, Mapping) else None
|
||||
if not isinstance(buckets, list):
|
||||
raise ConnectorBrowseError("S3 connector returned invalid bucket list")
|
||||
return sorted(
|
||||
(
|
||||
ConnectorBrowseItem(
|
||||
kind="library",
|
||||
name=_clean(item.get("Name")) or "",
|
||||
path=_clean(item.get("Name")) or "",
|
||||
external_id=_clean(item.get("Name")),
|
||||
modified_at=_timestamp(item.get("CreationDate")),
|
||||
metadata={"bucket": _clean(item.get("Name"))},
|
||||
)
|
||||
for item in buckets
|
||||
if isinstance(item, Mapping) and _clean(item.get("Name"))
|
||||
),
|
||||
key=lambda item: item.name.casefold(),
|
||||
)
|
||||
prefix = _s3_object_key(profile, path, directory=True)
|
||||
params: dict[str, object] = {
|
||||
"Bucket": bucket,
|
||||
"Prefix": prefix,
|
||||
"Delimiter": "/",
|
||||
"MaxKeys": _s3_max_keys(profile),
|
||||
}
|
||||
next_page_request = _clean(continuation_token) or _metadata_string(profile, "continuation_token")
|
||||
if next_page_request:
|
||||
params["ContinuationToken"] = next_page_request
|
||||
try:
|
||||
payload = client.list_objects_v2(**params)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||
if not isinstance(payload, Mapping):
|
||||
raise ConnectorBrowseError("S3 connector returned invalid object listing")
|
||||
items = [
|
||||
*_s3_prefix_items(bucket=bucket, browse_path=path, base_prefix=prefix, prefixes=payload.get("CommonPrefixes")),
|
||||
*_s3_object_items(bucket=bucket, browse_path=path, base_prefix=prefix, objects=payload.get("Contents")),
|
||||
]
|
||||
next_token = _clean(payload.get("NextContinuationToken"))
|
||||
if next_token and items:
|
||||
last = items[-1]
|
||||
items[-1] = ConnectorBrowseItem(
|
||||
kind=last.kind,
|
||||
name=last.name,
|
||||
path=last.path,
|
||||
external_id=last.external_id,
|
||||
external_url=last.external_url,
|
||||
size_bytes=last.size_bytes,
|
||||
content_type=last.content_type,
|
||||
modified_at=last.modified_at,
|
||||
etag=last.etag,
|
||||
metadata={**dict(last.metadata), "next_continuation_token": next_token, "listing_truncated": True},
|
||||
)
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _s3_client(profile: ConnectorProfile) -> Any:
|
||||
if profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref S3 credentials need a runtime secret resolver before live browsing")
|
||||
if profile.endpoint_url:
|
||||
try:
|
||||
endpoint_url = validate_outbound_http_url(
|
||||
profile.endpoint_url,
|
||||
label="S3 connector endpoint",
|
||||
)
|
||||
except OutboundHttpError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
else:
|
||||
endpoint_url = None
|
||||
try:
|
||||
config_module = import_module("botocore.config")
|
||||
unsigned = import_module("botocore").UNSIGNED
|
||||
except ImportError as exc:
|
||||
raise ConnectorBrowseUnsupported("S3 connector browsing requires the optional boto3 dependency") from exc
|
||||
kwargs: dict[str, object] = {}
|
||||
if endpoint_url:
|
||||
kwargs["endpoint_url"] = endpoint_url
|
||||
region = _metadata_string(profile, "region") or _metadata_string(profile, "aws_region")
|
||||
if region:
|
||||
kwargs["region_name"] = region
|
||||
access_key = profile.username or _metadata_env(profile, "access_key_id_env") or _metadata_string(profile, "access_key_id")
|
||||
secret_key = _profile_password(profile) or _metadata_env(profile, "secret_access_key_env")
|
||||
session_token = _profile_token(profile) or _metadata_env(profile, "session_token_env")
|
||||
if access_key:
|
||||
kwargs["aws_access_key_id"] = access_key
|
||||
if secret_key:
|
||||
kwargs["aws_secret_access_key"] = secret_key
|
||||
if session_token:
|
||||
kwargs["aws_session_token"] = session_token
|
||||
verify = _s3_verify(profile)
|
||||
if verify is not None:
|
||||
kwargs["verify"] = verify
|
||||
addressing_style = _s3_addressing_style(profile)
|
||||
config_values: dict[str, object] = {
|
||||
"proxies": {},
|
||||
"retries": {"mode": "standard", "max_attempts": 4},
|
||||
}
|
||||
if addressing_style:
|
||||
config_values["s3"] = {"addressing_style": addressing_style}
|
||||
if bool(access_key) != bool(secret_key):
|
||||
raise ConnectorBrowseError("S3 connectors require both an access key and a secret key")
|
||||
if not access_key:
|
||||
config_values["signature_version"] = unsigned
|
||||
kwargs["config"] = config_module.Config(**config_values)
|
||||
try:
|
||||
return create_pinned_s3_client(**kwargs)
|
||||
except SdkPeerPinningError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector could not be initialized: {exc}") from exc
|
||||
|
||||
|
||||
def _s3_bucket(profile: ConnectorProfile, library_id: str | None = None) -> str | None:
|
||||
return _metadata_string(profile, "bucket") or _metadata_string(profile, "bucket_name") or _clean(library_id)
|
||||
|
||||
|
||||
def _s3_object_key(profile: ConnectorProfile, path: str, *, directory: bool = False) -> str:
|
||||
base_prefix = normalize_connector_browse_path(profile.base_path or _metadata_string(profile, "base_prefix"))
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
key = "/".join(part for part in (base_prefix, browse_path) if part)
|
||||
if directory and key:
|
||||
return key.rstrip("/") + "/"
|
||||
return key
|
||||
|
||||
|
||||
def _s3_prefix_items(
|
||||
*,
|
||||
bucket: str,
|
||||
browse_path: str,
|
||||
base_prefix: str,
|
||||
prefixes: object,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(prefixes, list):
|
||||
return []
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for item in prefixes:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
key = _clean(item.get("Prefix"))
|
||||
if not key:
|
||||
continue
|
||||
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||
name = _path_name(relative)
|
||||
if not name:
|
||||
continue
|
||||
path = _join_browse_path(browse_path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder",
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=f"{bucket}:{key}",
|
||||
metadata={"bucket": bucket, "key": key},
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _s3_object_items(
|
||||
*,
|
||||
bucket: str,
|
||||
browse_path: str,
|
||||
base_prefix: str,
|
||||
objects: object,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(objects, list):
|
||||
return []
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for item in objects:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
key = _clean(item.get("Key"))
|
||||
if not key or key == base_prefix:
|
||||
continue
|
||||
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||
name = _path_name(relative)
|
||||
if not name:
|
||||
continue
|
||||
path = _join_browse_path(browse_path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="file",
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=f"{bucket}:{key}",
|
||||
size_bytes=_int(item.get("Size")),
|
||||
content_type=mimetypes.guess_type(name)[0],
|
||||
modified_at=_timestamp(item.get("LastModified")),
|
||||
etag=_clean(item.get("ETag")),
|
||||
metadata={
|
||||
"bucket": bucket,
|
||||
"key": key,
|
||||
**({"storage_class": item["StorageClass"]} if "StorageClass" in item else {}),
|
||||
},
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _s3_relative_key(key: str, *, base_prefix: str) -> str:
|
||||
clean_key = key.rstrip("/")
|
||||
clean_base = base_prefix.rstrip("/")
|
||||
if clean_base and clean_key.startswith(f"{clean_base}/"):
|
||||
return clean_key[len(clean_base) + 1 :]
|
||||
return clean_key
|
||||
|
||||
|
||||
def _s3_max_keys(profile: ConnectorProfile) -> int:
|
||||
configured = _int(profile.metadata.get("max_keys"))
|
||||
if configured is None:
|
||||
return 1000
|
||||
return max(1, min(configured, 1000))
|
||||
|
||||
|
||||
def _s3_verify(profile: ConnectorProfile) -> bool | str | None:
|
||||
try:
|
||||
validate_connector_tls_metadata(profile.metadata)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
ca_bundle = _metadata_string(profile, "ca_bundle")
|
||||
if ca_bundle:
|
||||
try:
|
||||
return connector_ca_bundle_path(ca_bundle)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
if "verify_tls" in profile.metadata:
|
||||
return _metadata_bool(profile, "verify_tls", default=True)
|
||||
if "tls_verify" in profile.metadata:
|
||||
return _metadata_bool(profile, "tls_verify", default=True)
|
||||
return None
|
||||
|
||||
|
||||
def _s3_addressing_style(profile: ConnectorProfile) -> str | None:
|
||||
style = _metadata_string(profile, "addressing_style")
|
||||
if style in {"path", "virtual", "auto"}:
|
||||
return style
|
||||
if _metadata_bool(profile, "path_style", default=False):
|
||||
return "path"
|
||||
return None
|
||||
|
||||
|
||||
def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]:
|
||||
token = _seafile_token(profile)
|
||||
return {"Authorization": f"Token {token}", "Accept": "application/json"}
|
||||
|
||||
|
||||
def _seafile_token(profile: ConnectorProfile) -> str:
|
||||
token = _profile_token(profile)
|
||||
if token:
|
||||
return token
|
||||
password = _profile_password(profile)
|
||||
if profile.username and password:
|
||||
payload = _request_json(
|
||||
"POST",
|
||||
_seafile_url(profile, "api2/auth-token/"),
|
||||
data={"username": profile.username, "password": password},
|
||||
)
|
||||
if not isinstance(payload, Mapping) or not _clean(payload.get("token")):
|
||||
raise ConnectorBrowseError("Seafile did not return an account token")
|
||||
return _clean(payload.get("token")) or ""
|
||||
if profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref Seafile credentials need a runtime secret resolver before live browsing")
|
||||
raise ConnectorBrowseError("Seafile connector profiles require token credentials or username plus password credentials")
|
||||
|
||||
|
||||
def _seafile_url(profile: ConnectorProfile, suffix: str) -> str:
|
||||
if not profile.endpoint_url:
|
||||
raise ConnectorBrowseError("Seafile connector profile does not define endpoint_url")
|
||||
return urljoin(profile.endpoint_url.rstrip("/") + "/", suffix.lstrip("/"))
|
||||
|
||||
|
||||
def _request_json(
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
headers: Mapping[str, str] | None = None,
|
||||
params: Mapping[str, str] | None = None,
|
||||
data: Mapping[str, str] | None = None,
|
||||
) -> object:
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
method,
|
||||
url,
|
||||
headers=dict(headers or {}),
|
||||
params=params,
|
||||
data=data,
|
||||
timeout=15.0,
|
||||
label="Seafile API",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||
if response.status_code not in {200, 201}:
|
||||
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||
try:
|
||||
return json.loads(response.content)
|
||||
except (UnicodeDecodeError, ValueError) as exc:
|
||||
raise ConnectorBrowseError("Connector returned invalid JSON") from exc
|
||||
|
||||
|
||||
def _seafile_repo_and_path(*, path: str, library_id: str | None) -> tuple[str | None, str]:
|
||||
repo_id = _clean(library_id)
|
||||
if repo_id:
|
||||
return repo_id, path
|
||||
if not path:
|
||||
return None, ""
|
||||
first, _, rest = path.partition("/")
|
||||
return first, rest
|
||||
|
||||
|
||||
def _seafile_library_item(value: Mapping[str, Any]) -> ConnectorBrowseItem:
|
||||
repo_id = _clean(value.get("id") or value.get("repo_id"))
|
||||
name = _clean(value.get("name") or value.get("repo_name") or repo_id)
|
||||
if not repo_id or not name:
|
||||
raise ConnectorBrowseError("Seafile library entries require id and name")
|
||||
return ConnectorBrowseItem(
|
||||
kind="library",
|
||||
name=name,
|
||||
path=repo_id,
|
||||
external_id=repo_id,
|
||||
size_bytes=_int(value.get("size") or value.get("repo_size")),
|
||||
modified_at=_timestamp(value.get("mtime")),
|
||||
metadata={
|
||||
key: value[key]
|
||||
for key in ("type", "permission", "encrypted", "owner", "file_count")
|
||||
if key in value
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _seafile_directory_item(value: Mapping[str, Any], *, parent_path: str) -> ConnectorBrowseItem:
|
||||
name = _clean(value.get("name") or value.get("obj_name"))
|
||||
if not name:
|
||||
raise ConnectorBrowseError("Seafile directory entries require name")
|
||||
item_type = str(value.get("type") or ("dir" if value.get("is_dir") else "file")).casefold()
|
||||
kind = "folder" if item_type in {"dir", "folder"} else "file"
|
||||
path = _join_browse_path(parent_path, name)
|
||||
content_type = None if kind == "folder" else mimetypes.guess_type(name)[0]
|
||||
return ConnectorBrowseItem(
|
||||
kind=kind,
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=_clean(value.get("id") or value.get("obj_id")),
|
||||
size_bytes=None if kind == "folder" else _int(value.get("size")),
|
||||
content_type=content_type,
|
||||
modified_at=_timestamp(value.get("mtime") or value.get("modified")),
|
||||
etag=_clean(value.get("id") or value.get("obj_id")),
|
||||
metadata={
|
||||
key: value[key]
|
||||
for key in ("permission", "modifier_email", "modifier_name")
|
||||
if key in value
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _static_listing(profile: ConnectorProfile, *, path: str, library_id: str | None) -> list[ConnectorBrowseItem] | None:
|
||||
listing = profile.metadata.get("static_listing")
|
||||
if listing is None:
|
||||
return None
|
||||
if isinstance(listing, list):
|
||||
raw_items = listing if path == "" else []
|
||||
elif isinstance(listing, Mapping):
|
||||
key = library_id or path
|
||||
raw_items = listing.get(key)
|
||||
if raw_items is None and key == "":
|
||||
raw_items = listing.get("/")
|
||||
else:
|
||||
raise ConnectorBrowseError("Connector static_listing metadata must be a list or object")
|
||||
if raw_items is None:
|
||||
return []
|
||||
if not isinstance(raw_items, list):
|
||||
raise ConnectorBrowseError("Connector static_listing entries must be lists")
|
||||
return sorted(
|
||||
[_static_item(item, parent_path=path) for item in raw_items if isinstance(item, Mapping)],
|
||||
key=lambda item: (item.kind != "library", item.kind != "folder", item.name.casefold(), item.path.casefold()),
|
||||
)
|
||||
|
||||
|
||||
def _static_item(value: Mapping[str, Any], *, parent_path: str) -> ConnectorBrowseItem:
|
||||
kind = str(value.get("kind") or "file").strip().casefold()
|
||||
if kind not in {"library", "folder", "file"}:
|
||||
raise ConnectorBrowseError("Connector browse items must be library, folder or file")
|
||||
name = str(value.get("name") or value.get("path") or "").strip()
|
||||
if not name:
|
||||
raise ConnectorBrowseError("Connector browse items require name")
|
||||
path = normalize_connector_browse_path(value.get("path"))
|
||||
if not path:
|
||||
path = _join_browse_path(parent_path, name)
|
||||
return ConnectorBrowseItem(
|
||||
kind=kind,
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=_clean(value.get("external_id")),
|
||||
external_url=_clean(value.get("external_url")),
|
||||
size_bytes=_int(value.get("size_bytes")),
|
||||
content_type=_clean(value.get("content_type")),
|
||||
modified_at=_clean(value.get("modified_at")),
|
||||
etag=_clean(value.get("etag")),
|
||||
metadata=value.get("metadata") if isinstance(value.get("metadata"), Mapping) else {},
|
||||
)
|
||||
|
||||
|
||||
def _webdav_url(root_url: str, path: str) -> str:
|
||||
base = root_url if root_url.endswith("/") else root_url + "/"
|
||||
if not path:
|
||||
return base
|
||||
quoted = "/".join(quote(part, safe="") for part in path.split("/") if part)
|
||||
return urljoin(base, quoted + "/")
|
||||
|
||||
|
||||
def _url_path_root(root_url: str) -> str:
|
||||
path = unquote(urlsplit(root_url).path or "/")
|
||||
return path if path.endswith("/") else path + "/"
|
||||
|
||||
|
||||
def _relative_href_path(root_path: str, href: str) -> str:
|
||||
href_path = unquote(urlsplit(href).path or href).strip()
|
||||
if href_path.startswith(root_path):
|
||||
return normalize_connector_browse_path(href_path[len(root_path):])
|
||||
return normalize_connector_browse_path(href_path.rsplit("/", 1)[-1])
|
||||
|
||||
|
||||
def _webdav_prop(response: ET.Element) -> ET.Element:
|
||||
prop = response.find("{DAV:}propstat/{DAV:}prop")
|
||||
if prop is not None:
|
||||
return prop
|
||||
prop = response.find(".//{DAV:}prop")
|
||||
return prop if prop is not None else ET.Element("prop")
|
||||
|
||||
|
||||
def _webdav_display_name(prop: ET.Element | None) -> str | None:
|
||||
return _clean(_text(prop, "{DAV:}displayname"))
|
||||
|
||||
|
||||
def _text(prop: ET.Element | None, tag: str) -> str | None:
|
||||
if prop is None:
|
||||
return None
|
||||
child = prop.find(tag)
|
||||
return child.text.strip() if child is not None and child.text else None
|
||||
|
||||
|
||||
def _http_date(value: str | None) -> str | None:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return parsedate_to_datetime(value).isoformat()
|
||||
except (TypeError, ValueError):
|
||||
return value
|
||||
|
||||
|
||||
def _timestamp(value: object) -> str | None:
|
||||
number = _int(value)
|
||||
if number is None:
|
||||
return _clean(value)
|
||||
return datetime.fromtimestamp(number, tz=timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _metadata_string(profile: ConnectorProfile, key: str) -> str | None:
|
||||
return _clean(profile.metadata.get(key))
|
||||
|
||||
|
||||
def _metadata_bool(profile: ConnectorProfile, key: str, *, default: bool = False) -> bool:
|
||||
value = profile.metadata.get(key)
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
return str(value).strip().casefold() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _metadata_env(profile: ConnectorProfile, key: str) -> str | None:
|
||||
env_name = _metadata_string(profile, key)
|
||||
if not env_name:
|
||||
return None
|
||||
return _env_required(env_name, profile)
|
||||
|
||||
|
||||
def _env_required(name: str, profile: ConnectorProfile) -> str:
|
||||
try:
|
||||
return connector_secret_env_value(name, source_kind=profile.source_kind)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(f"Connector profile {profile.id}: {exc}") from exc
|
||||
|
||||
|
||||
def normalize_connector_browse_path(value: object) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
path = str(value).replace("\\", "/").strip().strip("/")
|
||||
parts = [part for part in path.split("/") if part and part not in {"."}]
|
||||
if any(part == ".." for part in parts):
|
||||
raise ConnectorBrowseError("Connector browse paths cannot contain parent directory segments")
|
||||
return "/".join(parts)
|
||||
|
||||
|
||||
def _join_browse_path(parent: str, name: str) -> str:
|
||||
child = normalize_connector_browse_path(name)
|
||||
if not parent:
|
||||
return child
|
||||
return f"{parent.rstrip('/')}/{child}"
|
||||
|
||||
|
||||
def _path_name(path: str) -> str:
|
||||
return path.rstrip("/").rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def _int(value: object) -> int | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _SmbLocation:
|
||||
server: str
|
||||
share: str
|
||||
port: int
|
||||
root_path: str = ""
|
||||
|
||||
|
||||
def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
|
||||
if not profile.endpoint_url:
|
||||
raise ConnectorBrowseError("SMB connector profile does not define endpoint_url")
|
||||
parsed = urlsplit(profile.endpoint_url)
|
||||
if parsed.scheme.casefold() != "smb":
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must use smb://")
|
||||
server = _clean(parsed.hostname)
|
||||
if not server:
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must include a server")
|
||||
port = parsed.port or _int(profile.metadata.get("port")) or 445
|
||||
try:
|
||||
validate_outbound_host(server, port=port, label="SMB connector endpoint")
|
||||
except OutboundHttpError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part]
|
||||
if not path_parts:
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must include a share name")
|
||||
root_parts = path_parts[1:]
|
||||
if profile.base_path:
|
||||
root_parts.extend(normalize_connector_browse_path(profile.base_path).split("/"))
|
||||
return _SmbLocation(
|
||||
server=server,
|
||||
share=path_parts[0],
|
||||
port=port,
|
||||
root_path=normalize_connector_browse_path("/".join(root_parts)),
|
||||
)
|
||||
|
||||
|
||||
def _smb_unc_path(location: _SmbLocation, path: str) -> str:
|
||||
parts = [part for part in (location.root_path, normalize_connector_browse_path(path)) if part]
|
||||
suffix = "\\".join(part.replace("/", "\\") for part in parts)
|
||||
base = f"\\\\{location.server}\\{location.share}"
|
||||
return f"{base}\\{suffix}" if suffix else base
|
||||
|
||||
|
||||
def _smb_client_kwargs(profile: ConnectorProfile, location: _SmbLocation) -> dict[str, object]:
|
||||
kwargs: dict[str, object] = {
|
||||
"port": location.port,
|
||||
"connection_cache": pinned_smb_connection_cache(),
|
||||
"require_signing": _metadata_bool(profile, "require_signing", default=True),
|
||||
"auth_protocol": _metadata_string(profile, "auth_protocol") or "ntlm",
|
||||
}
|
||||
if "encrypt" in profile.metadata:
|
||||
kwargs["encrypt"] = _metadata_bool(profile, "encrypt", default=False)
|
||||
password = _profile_password(profile)
|
||||
token = _profile_token(profile)
|
||||
if profile.username and password:
|
||||
kwargs["username"] = profile.username
|
||||
kwargs["password"] = password
|
||||
elif token:
|
||||
raise ConnectorBrowseError("SMB connector profiles do not support bearer-token credentials")
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref SMB credentials need a runtime secret resolver before live browsing")
|
||||
return kwargs
|
||||
|
||||
|
||||
def _profile_password(profile: ConnectorProfile) -> str | None:
|
||||
if profile.password_value:
|
||||
return profile.password_value
|
||||
if profile.password_env:
|
||||
return _env_required(profile.password_env, profile)
|
||||
return None
|
||||
|
||||
|
||||
def _profile_token(profile: ConnectorProfile) -> str | None:
|
||||
if profile.token_value:
|
||||
return profile.token_value
|
||||
if profile.token_env:
|
||||
return _env_required(profile.token_env, profile)
|
||||
return None
|
||||
|
||||
|
||||
def _smbclient_module() -> Any:
|
||||
try:
|
||||
return install_pinned_smb_transport(import_module("smbclient"))
|
||||
except ImportError as exc:
|
||||
raise ConnectorBrowseUnsupported("SMB connector browsing requires the optional smbprotocol dependency") from exc
|
||||
except SdkPeerPinningError as exc:
|
||||
raise ConnectorBrowseUnsupported(str(exc)) from exc
|
||||
|
||||
|
||||
def _smb_entry_stat(entry: object) -> object | None:
|
||||
try:
|
||||
return entry.stat() # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _smb_stat_size(stat_result: object | None) -> int | None:
|
||||
return _int(getattr(stat_result, "st_size", None))
|
||||
|
||||
|
||||
def _smb_stat_modified_at(stat_result: object | None) -> str | None:
|
||||
value = getattr(stat_result, "st_mtime", None)
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromtimestamp(float(value), tz=timezone.utc).isoformat()
|
||||
except (TypeError, ValueError, OSError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _smb_stat_revision(stat_result: object | None) -> str | None:
|
||||
mtime_ns = getattr(stat_result, "st_mtime_ns", None)
|
||||
size = getattr(stat_result, "st_size", None)
|
||||
if mtime_ns is not None:
|
||||
return f"{mtime_ns}:{size or 0}"
|
||||
modified = getattr(stat_result, "st_mtime", None)
|
||||
if modified is not None:
|
||||
return f"{modified}:{size or 0}"
|
||||
return None
|
||||
@@ -0,0 +1,337 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import inspect
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_event
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorCredentialDeletionResult:
|
||||
changed: bool
|
||||
affected_profiles: tuple[FileConnectorProfile, ...] = ()
|
||||
|
||||
|
||||
def delete_connector_credential_row(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> ConnectorCredentialDeletionResult:
|
||||
"""Scrub a credential tombstone and disable every profile that used it.
|
||||
|
||||
``secret_ref`` predates a Files-owned secret-provider contract. It may be
|
||||
shared or deployment-owned, so it is detached locally and explicitly
|
||||
audited without ever being passed to a provider delete operation.
|
||||
"""
|
||||
|
||||
dependent_profiles = (
|
||||
tuple(
|
||||
session.query(FileConnectorProfile)
|
||||
.filter(FileConnectorProfile.credential_profile_id == row.id)
|
||||
.order_by(FileConnectorProfile.id.asc())
|
||||
.all()
|
||||
)
|
||||
if inspect(session.get_bind()).has_table(FileConnectorProfile.__tablename__)
|
||||
else ()
|
||||
)
|
||||
affected_profiles: list[FileConnectorProfile] = []
|
||||
for profile in dependent_profiles:
|
||||
if _delete_connector_profile_row(
|
||||
session,
|
||||
profile,
|
||||
deletion_reason="credential_deleted",
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
):
|
||||
affected_profiles.append(profile)
|
||||
|
||||
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||
removed_reference_kinds = _credential_reference_kinds(row)
|
||||
removed_metadata = bool(row.metadata_)
|
||||
changed = _credential_row_requires_deletion(row)
|
||||
if changed:
|
||||
_scrub_credential_row(row, user_id=user_id)
|
||||
session.add(row)
|
||||
_audit_credential_deletion(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
deleted_secret_kinds=deleted_secret_kinds,
|
||||
removed_reference_kinds=removed_reference_kinds,
|
||||
removed_metadata=removed_metadata,
|
||||
affected_profile_count=len(affected_profiles),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
session.flush()
|
||||
return ConnectorCredentialDeletionResult(
|
||||
changed=changed or bool(affected_profiles),
|
||||
affected_profiles=tuple(affected_profiles),
|
||||
)
|
||||
|
||||
|
||||
def delete_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> bool:
|
||||
changed = _delete_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
session.flush()
|
||||
return changed
|
||||
|
||||
|
||||
def delete_connector_credentials_for_retirement(session: Session) -> int:
|
||||
"""Scrub and audit stored connector material before destructive retirement.
|
||||
|
||||
Legacy external references are detached and audited as non-owned. Files
|
||||
never sends those arbitrary references to a provider delete operation.
|
||||
"""
|
||||
|
||||
inspector = inspect(session.get_bind())
|
||||
profiles = (
|
||||
session.query(FileConnectorProfile).order_by(FileConnectorProfile.id.asc()).all()
|
||||
if inspector.has_table(FileConnectorProfile.__tablename__)
|
||||
else []
|
||||
)
|
||||
credentials = (
|
||||
session.query(FileConnectorCredential).order_by(FileConnectorCredential.id.asc()).all()
|
||||
if inspector.has_table(FileConnectorCredential.__tablename__)
|
||||
else []
|
||||
)
|
||||
deleted = 0
|
||||
for profile in profiles:
|
||||
if not _profile_row_has_credential_material(profile):
|
||||
continue
|
||||
if _delete_connector_profile_row(
|
||||
session,
|
||||
profile,
|
||||
deletion_reason="module_data_retired",
|
||||
):
|
||||
deleted += 1
|
||||
for credential in credentials:
|
||||
if not _credential_row_has_material(credential):
|
||||
continue
|
||||
result = delete_connector_credential_row(
|
||||
session,
|
||||
credential,
|
||||
deletion_reason="module_data_retired",
|
||||
)
|
||||
if result.changed:
|
||||
deleted += 1
|
||||
session.flush()
|
||||
return deleted
|
||||
|
||||
|
||||
def _delete_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> bool:
|
||||
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||
removed_reference_kinds = _profile_reference_kinds(row)
|
||||
removed_metadata = bool(row.metadata_)
|
||||
changed = _profile_row_requires_deletion(row)
|
||||
if not changed:
|
||||
return False
|
||||
_scrub_profile_row(row, user_id=user_id)
|
||||
session.add(row)
|
||||
_audit_profile_deletion(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
deleted_secret_kinds=deleted_secret_kinds,
|
||||
removed_reference_kinds=removed_reference_kinds,
|
||||
removed_metadata=removed_metadata,
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def _scrub_credential_row(row: FileConnectorCredential, *, user_id: str | None) -> None:
|
||||
row.enabled = False
|
||||
row.credential_mode = "none"
|
||||
row.username = None
|
||||
row.password_encrypted = None
|
||||
row.token_encrypted = None
|
||||
row.password_env = None
|
||||
row.token_env = None
|
||||
row.secret_ref = None
|
||||
row.metadata_ = {}
|
||||
row.updated_by_user_id = user_id
|
||||
|
||||
|
||||
def _scrub_profile_row(row: FileConnectorProfile, *, user_id: str | None) -> None:
|
||||
row.enabled = False
|
||||
row.credential_profile_id = None
|
||||
row.credential_mode = "none"
|
||||
row.username = None
|
||||
row.password_encrypted = None
|
||||
row.token_encrypted = None
|
||||
row.password_env = None
|
||||
row.token_env = None
|
||||
row.secret_ref = None
|
||||
row.metadata_ = {}
|
||||
row.updated_by_user_id = user_id
|
||||
|
||||
|
||||
def _credential_row_requires_deletion(row: FileConnectorCredential) -> bool:
|
||||
return bool(row.enabled or _credential_row_has_material(row))
|
||||
|
||||
|
||||
def _profile_row_requires_deletion(row: FileConnectorProfile) -> bool:
|
||||
return bool(row.enabled or _profile_row_has_credential_material(row))
|
||||
|
||||
|
||||
def _credential_row_has_material(row: FileConnectorCredential) -> bool:
|
||||
return bool(
|
||||
row.username
|
||||
or row.password_encrypted
|
||||
or row.token_encrypted
|
||||
or row.password_env
|
||||
or row.token_env
|
||||
or row.secret_ref
|
||||
or row.metadata_
|
||||
or row.credential_mode not in {"", "none", "anonymous"}
|
||||
)
|
||||
|
||||
|
||||
def _profile_row_has_credential_material(row: FileConnectorProfile) -> bool:
|
||||
return bool(
|
||||
row.credential_profile_id
|
||||
or row.username
|
||||
or row.password_encrypted
|
||||
or row.token_encrypted
|
||||
or row.password_env
|
||||
or row.token_env
|
||||
or row.secret_ref
|
||||
or row.metadata_
|
||||
or row.credential_mode not in {"", "none", "anonymous"}
|
||||
)
|
||||
|
||||
|
||||
def _encrypted_secret_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||
kinds: list[str] = []
|
||||
if row.password_encrypted:
|
||||
kinds.append("password")
|
||||
if row.token_encrypted:
|
||||
kinds.append("token")
|
||||
return kinds
|
||||
|
||||
|
||||
def _credential_reference_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||
kinds: list[str] = []
|
||||
if row.password_env:
|
||||
kinds.append("password_env")
|
||||
if row.token_env:
|
||||
kinds.append("token_env")
|
||||
if row.secret_ref:
|
||||
kinds.append("unowned_external_secret_ref")
|
||||
return kinds
|
||||
|
||||
|
||||
def _profile_reference_kinds(row: FileConnectorProfile) -> list[str]:
|
||||
kinds = _credential_reference_kinds(row)
|
||||
if row.credential_profile_id:
|
||||
kinds.append("credential_profile")
|
||||
return kinds
|
||||
|
||||
|
||||
def _storage_backend(deleted_secret_kinds: list[str], removed_reference_kinds: list[str]) -> str:
|
||||
if "unowned_external_secret_ref" in removed_reference_kinds:
|
||||
return "unowned_external_reference_detached"
|
||||
if deleted_secret_kinds:
|
||||
return "encrypted_database"
|
||||
if removed_reference_kinds:
|
||||
return "reference_only"
|
||||
return "none"
|
||||
|
||||
|
||||
def _audit_credential_deletion(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
deleted_secret_kinds: list[str],
|
||||
removed_reference_kinds: list[str],
|
||||
removed_metadata: bool,
|
||||
affected_profile_count: int,
|
||||
user_id: str | None,
|
||||
api_key_id: str | None,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=row.tenant_id,
|
||||
scope=_audit_scope(row.scope_type),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
action="files.connector_credential_deleted",
|
||||
object_type="file_connector_credential",
|
||||
object_id=row.id,
|
||||
details={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||
"deleted_secret_kinds": deleted_secret_kinds,
|
||||
"removed_reference_kinds": removed_reference_kinds,
|
||||
"removed_metadata": removed_metadata,
|
||||
"affected_profile_count": affected_profile_count,
|
||||
"deletion_reason": deletion_reason,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _audit_profile_deletion(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
deleted_secret_kinds: list[str],
|
||||
removed_reference_kinds: list[str],
|
||||
removed_metadata: bool,
|
||||
user_id: str | None,
|
||||
api_key_id: str | None,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=row.tenant_id,
|
||||
scope=_audit_scope(row.scope_type),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
action="files.connector_profile_deleted",
|
||||
object_type="file_connector_profile",
|
||||
object_id=row.id,
|
||||
details={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||
"deleted_secret_kinds": deleted_secret_kinds,
|
||||
"removed_reference_kinds": removed_reference_kinds,
|
||||
"removed_metadata": removed_metadata,
|
||||
"deletion_reason": deletion_reason,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _audit_scope(scope_type: str) -> str:
|
||||
return "system" if scope_type == "system" else "tenant"
|
||||
@@ -0,0 +1,540 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import inspect
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||
from govoplan_core.security.credential_envelopes import (
|
||||
CredentialAccessContext,
|
||||
CredentialEnvelope,
|
||||
CredentialEnvelopeError,
|
||||
ResolvedCredentialEnvelope,
|
||||
list_credential_envelopes,
|
||||
resolve_credential_envelope,
|
||||
)
|
||||
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
|
||||
|
||||
CORE_CREDENTIAL_ENVELOPE_PREFIX = "credential-envelope:"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorCredential:
|
||||
id: str
|
||||
label: str
|
||||
scope_type: str
|
||||
scope_id: str | None = None
|
||||
provider: str | None = None
|
||||
enabled: bool = True
|
||||
credential_mode: str = "none"
|
||||
username: str | None = None
|
||||
password_env: str | None = None
|
||||
token_env: str | None = None
|
||||
secret_ref: str | None = None
|
||||
password_value: str | None = None
|
||||
token_value: str | None = None
|
||||
policy_sources: tuple[ConnectorPolicySource, ...] = ()
|
||||
metadata: Mapping[str, Any] | None = None
|
||||
source_kind: str = "database"
|
||||
has_secret: bool = False
|
||||
|
||||
@property
|
||||
def source_path(self) -> str:
|
||||
return policy_source_path(self.scope_type, self.scope_id)
|
||||
|
||||
@property
|
||||
def credential_secret_source(self) -> str | None:
|
||||
if self.secret_ref:
|
||||
return "secret_ref"
|
||||
if self.password_value or self.token_value:
|
||||
return "stored"
|
||||
if self.token_env:
|
||||
return "token_env"
|
||||
if self.password_env:
|
||||
return "password_env"
|
||||
return None
|
||||
|
||||
@property
|
||||
def credentials_configured(self) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
|
||||
return True
|
||||
# Environment references are deliberately unavailable to API-managed
|
||||
# credential rows. Legacy rows remain visible but fail closed.
|
||||
return bool(self.has_secret or self.secret_ref or self.password_value or self.token_value)
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
"id": self.id,
|
||||
"label": self.label,
|
||||
"provider": self.provider,
|
||||
"enabled": self.enabled,
|
||||
"scope_type": self.scope_type,
|
||||
"scope_id": self.scope_id,
|
||||
"source_path": self.source_path,
|
||||
"credential_mode": self.credential_mode,
|
||||
"credential_secret_source": self.credential_secret_source,
|
||||
"credentials_configured": self.credentials_configured,
|
||||
"username": self.username,
|
||||
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||
"metadata": dict(self.metadata or {}),
|
||||
"source_kind": self.source_kind,
|
||||
}
|
||||
|
||||
|
||||
def list_database_connector_credentials(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[ConnectorCredential]:
|
||||
local = [
|
||||
connector_credential_from_row(row)
|
||||
for row in list_connector_credential_rows(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
]
|
||||
return [
|
||||
*local,
|
||||
*list_reusable_connector_credentials(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def reusable_credential_reference(credential_id: str) -> str:
|
||||
return f"{CORE_CREDENTIAL_ENVELOPE_PREFIX}{credential_id}"
|
||||
|
||||
|
||||
def reusable_credential_id(credential_ref: str | None) -> str | None:
|
||||
if not credential_ref or not credential_ref.startswith(CORE_CREDENTIAL_ENVELOPE_PREFIX):
|
||||
return None
|
||||
value = credential_ref.removeprefix(CORE_CREDENTIAL_ENVELOPE_PREFIX).strip()
|
||||
return value or None
|
||||
|
||||
|
||||
def file_credential_context(
|
||||
*,
|
||||
tenant_id: str,
|
||||
profile_id: str | None = None,
|
||||
scope_type: str = "tenant",
|
||||
scope_id: str | None = None,
|
||||
administrative: bool = False,
|
||||
) -> CredentialAccessContext:
|
||||
return CredentialAccessContext(
|
||||
tenant_id=tenant_id,
|
||||
user_id=scope_id if scope_type == "user" else None,
|
||||
group_ids=frozenset({scope_id}) if scope_type == "group" and scope_id else frozenset(),
|
||||
target_scope_type=scope_type,
|
||||
target_scope_id=scope_id or tenant_id,
|
||||
module_id="files",
|
||||
server_ref=f"files:{profile_id}" if profile_id else None,
|
||||
administrative=administrative,
|
||||
)
|
||||
|
||||
|
||||
def list_reusable_connector_credentials(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[ConnectorCredential]:
|
||||
if not inspect(session.get_bind()).has_table(CredentialEnvelope.__tablename__):
|
||||
return []
|
||||
context = file_credential_context(tenant_id=tenant_id, administrative=True)
|
||||
return [
|
||||
connector_credential_from_envelope(row)
|
||||
for row in list_credential_envelopes(
|
||||
session,
|
||||
context=context,
|
||||
include_inactive=include_disabled,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def resolve_reusable_connector_credential(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
credential_ref: str,
|
||||
profile_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
) -> ConnectorCredential:
|
||||
credential_id = reusable_credential_id(credential_ref)
|
||||
if credential_id is None:
|
||||
raise FileStorageError("Reusable credential reference is invalid")
|
||||
try:
|
||||
resolved = resolve_credential_envelope(
|
||||
session,
|
||||
credential_id=credential_id,
|
||||
context=file_credential_context(
|
||||
tenant_id=tenant_id,
|
||||
profile_id=profile_id,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
),
|
||||
)
|
||||
except CredentialEnvelopeError as exc:
|
||||
raise FileStorageError("Reusable credential is unavailable to this file connection") from exc
|
||||
return connector_credential_from_resolved_envelope(
|
||||
resolved,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
)
|
||||
|
||||
|
||||
def connector_credential_from_envelope(row: CredentialEnvelope) -> ConnectorCredential:
|
||||
return ConnectorCredential(
|
||||
id=reusable_credential_reference(row.id),
|
||||
label=row.name,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
enabled=row.is_active,
|
||||
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_keys),
|
||||
username=_clean_public_value(row.public_data, "username"),
|
||||
source_kind="credential_envelope",
|
||||
has_secret=bool(row.secret_data_encrypted),
|
||||
metadata={
|
||||
"credential_envelope_id": row.id,
|
||||
"credential_kind": row.credential_kind,
|
||||
"allowed_modules": list(row.allowed_modules or []),
|
||||
"allowed_server_refs": list(row.allowed_server_refs or []),
|
||||
"inherit_to_lower_scopes": bool(row.inherit_to_lower_scopes),
|
||||
"revision": row.revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def connector_credential_from_resolved_envelope(
|
||||
row: ResolvedCredentialEnvelope,
|
||||
*,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
) -> ConnectorCredential:
|
||||
return ConnectorCredential(
|
||||
id=reusable_credential_reference(row.id),
|
||||
label=row.name,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
enabled=True,
|
||||
credential_mode=_envelope_credential_mode(row.credential_kind, row.secret_data),
|
||||
username=_clean_public_value(row.public_data, "username"),
|
||||
password_value=_first_secret(row.secret_data, "password", "secret"),
|
||||
token_value=_first_secret(
|
||||
row.secret_data,
|
||||
"access_token",
|
||||
"bearer_token",
|
||||
"token",
|
||||
"api_key",
|
||||
),
|
||||
source_kind="credential_envelope",
|
||||
has_secret=bool(row.secret_data),
|
||||
metadata={
|
||||
"credential_envelope_id": row.id,
|
||||
"credential_kind": row.credential_kind,
|
||||
"inherit_to_lower_scopes": True,
|
||||
"revision": row.revision,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def list_connector_credential_rows(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[FileConnectorCredential]:
|
||||
query = session.query(FileConnectorCredential).filter(
|
||||
(FileConnectorCredential.scope_type == "system")
|
||||
| (FileConnectorCredential.tenant_id == tenant_id)
|
||||
)
|
||||
if not include_disabled:
|
||||
query = query.filter(FileConnectorCredential.enabled.is_(True))
|
||||
return query.order_by(FileConnectorCredential.scope_type.asc(), FileConnectorCredential.label.asc()).all()
|
||||
|
||||
|
||||
def connector_credential_from_row(row: FileConnectorCredential) -> ConnectorCredential:
|
||||
policy_sources = []
|
||||
if row.policy:
|
||||
policy_sources = connector_policy_sources_from_payload({
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"label": row.label,
|
||||
"policy": row.policy,
|
||||
})
|
||||
return ConnectorCredential(
|
||||
id=row.id,
|
||||
label=row.label,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
provider=row.provider,
|
||||
enabled=row.enabled,
|
||||
credential_mode=row.credential_mode,
|
||||
username=row.username,
|
||||
password_env=row.password_env,
|
||||
token_env=row.token_env,
|
||||
secret_ref=row.secret_ref,
|
||||
password_value=decrypt_secret(row.password_encrypted),
|
||||
token_value=decrypt_secret(row.token_encrypted),
|
||||
policy_sources=tuple(policy_sources),
|
||||
metadata=dict(row.metadata_ or {}),
|
||||
)
|
||||
|
||||
|
||||
def get_connector_credential_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
credential_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> FileConnectorCredential:
|
||||
row = session.get(FileConnectorCredential, credential_id)
|
||||
if row is None or (row.scope_type != "system" and row.tenant_id != tenant_id):
|
||||
raise FileStorageError("Connector credential not found")
|
||||
if not include_disabled and not row.enabled:
|
||||
raise FileStorageError("Connector credential not found")
|
||||
return row
|
||||
|
||||
|
||||
def credential_rows_by_id(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
credential_ids: set[str],
|
||||
include_disabled: bool = False,
|
||||
) -> dict[str, FileConnectorCredential]:
|
||||
if not credential_ids:
|
||||
return {}
|
||||
rows = session.query(FileConnectorCredential).filter(FileConnectorCredential.id.in_(credential_ids)).all()
|
||||
result: dict[str, FileConnectorCredential] = {}
|
||||
for row in rows:
|
||||
if row.scope_type != "system" and row.tenant_id != tenant_id:
|
||||
continue
|
||||
if not include_disabled and not row.enabled:
|
||||
continue
|
||||
result[row.id] = row
|
||||
return result
|
||||
|
||||
|
||||
def create_connector_credential_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str | None,
|
||||
credential_id: str,
|
||||
label: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
provider: str | None = None,
|
||||
enabled: bool = True,
|
||||
credential_mode: str = "none",
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> FileConnectorCredential:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
clean_id = _normalize_id(credential_id)
|
||||
if session.get(FileConnectorCredential, clean_id) is not None:
|
||||
raise FileStorageError(f"Connector credential already exists: {clean_id}")
|
||||
clean_scope_type, clean_scope_id, row_tenant_id = _normalize_scope(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
row = FileConnectorCredential(
|
||||
id=clean_id,
|
||||
tenant_id=row_tenant_id,
|
||||
scope_type=clean_scope_type,
|
||||
scope_id=clean_scope_id,
|
||||
label=_normalize_label(label),
|
||||
provider=_normalize_provider(provider),
|
||||
enabled=bool(enabled),
|
||||
credential_mode=_normalize_credential_mode(credential_mode),
|
||||
username=_clean(username),
|
||||
password_encrypted=encrypt_secret(_clean(password)),
|
||||
token_encrypted=encrypt_secret(_clean(token)),
|
||||
password_env=_clean(password_env),
|
||||
token_env=_clean(token_env),
|
||||
secret_ref=_clean(secret_ref),
|
||||
policy=dict(policy or {}),
|
||||
metadata_=dict(metadata or {}),
|
||||
created_by_user_id=user_id,
|
||||
updated_by_user_id=user_id,
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def update_connector_credential_row(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
user_id: str | None,
|
||||
label: str | None = None,
|
||||
provider: str | None = None,
|
||||
enabled: bool | None = None,
|
||||
credential_mode: str | None = None,
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
clear_password: bool = False,
|
||||
clear_token: bool = False,
|
||||
) -> FileConnectorCredential:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
|
||||
if _clean(row.secret_ref):
|
||||
raise FileStorageError(
|
||||
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
if label is not None:
|
||||
row.label = _normalize_label(label)
|
||||
if provider is not None:
|
||||
row.provider = _normalize_provider(provider)
|
||||
if enabled is not None:
|
||||
row.enabled = bool(enabled)
|
||||
if credential_mode is not None:
|
||||
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||
if username is not None:
|
||||
row.username = _clean(username)
|
||||
if password is not None:
|
||||
row.password_encrypted = encrypt_secret(_clean(password))
|
||||
elif clear_password:
|
||||
row.password_encrypted = None
|
||||
if token is not None:
|
||||
row.token_encrypted = encrypt_secret(_clean(token))
|
||||
elif clear_token:
|
||||
row.token_encrypted = None
|
||||
if password_env is not None:
|
||||
row.password_env = _clean(password_env)
|
||||
if token_env is not None:
|
||||
row.token_env = _clean(token_env)
|
||||
if secret_ref is not None:
|
||||
row.secret_ref = _clean(secret_ref)
|
||||
if policy is not None:
|
||||
row.policy = dict(policy)
|
||||
if metadata is not None:
|
||||
row.metadata_ = dict(metadata)
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return "system", None, None
|
||||
if clean_scope_type == "tenant":
|
||||
return "tenant", tenant_id, tenant_id
|
||||
if clean_scope_type in {"user", "group", "campaign"}:
|
||||
if not clean_scope_id:
|
||||
raise FileStorageError(f"{clean_scope_type.capitalize()} connector credentials require scope_id")
|
||||
return clean_scope_type, clean_scope_id, tenant_id
|
||||
raise FileStorageError("Unsupported connector credential scope")
|
||||
|
||||
|
||||
def _normalize_id(value: str) -> str:
|
||||
clean = _clean(value)
|
||||
if not clean:
|
||||
raise FileStorageError("Connector credential id is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector credential id is too long")
|
||||
if any(char.isspace() for char in clean):
|
||||
raise FileStorageError("Connector credential id cannot contain whitespace")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_label(value: str) -> str:
|
||||
clean = value.strip()
|
||||
if not clean:
|
||||
raise FileStorageError("Connector credential label is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector credential label is too long")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_provider(value: str | None) -> str | None:
|
||||
clean = _clean(value)
|
||||
if clean is None:
|
||||
return None
|
||||
clean = clean.casefold()
|
||||
if clean not in supported_connector_providers():
|
||||
raise FileStorageError(f"Unsupported connector credential provider: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_credential_mode(value: str) -> str:
|
||||
clean = value.strip().casefold() or "none"
|
||||
if clean not in {"none", "anonymous", "basic", "token", "secret_ref"}:
|
||||
raise FileStorageError(f"Unsupported connector credential mode: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||
return {
|
||||
"scope_type": source.scope_type,
|
||||
"scope_id": source.scope_id,
|
||||
"label": source.label,
|
||||
"policy": dict(source.policy or {}),
|
||||
}
|
||||
|
||||
|
||||
def _envelope_credential_mode(kind: str, secret_values: Mapping[str, Any] | list[str]) -> str:
|
||||
keys = {str(key) for key in secret_values}
|
||||
if kind in {"token", "oauth2", "api_key"} or keys.intersection(
|
||||
{"access_token", "bearer_token", "token", "api_key"}
|
||||
):
|
||||
return "token"
|
||||
return "basic"
|
||||
|
||||
|
||||
def _clean_public_value(values: Mapping[str, Any] | None, key: str) -> str | None:
|
||||
return _clean((values or {}).get(key))
|
||||
|
||||
|
||||
def _first_secret(values: Mapping[str, Any], *keys: str) -> str | None:
|
||||
for key in keys:
|
||||
value = _clean(values.get(key))
|
||||
if value is not None:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,241 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
_SECRET_ENV_ALLOWLIST = "GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST" # noqa: S105 # nosec B105 - configuration key.
|
||||
_CA_BUNDLE_ALLOWLIST = "GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST"
|
||||
_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||
_SECRET_ENV_METADATA_KEYS = frozenset(
|
||||
{
|
||||
"access_key_id_env",
|
||||
"secret_access_key_env",
|
||||
"session_token_env",
|
||||
}
|
||||
)
|
||||
_SECRET_VALUE_METADATA_KEYS = frozenset(
|
||||
{
|
||||
"password",
|
||||
"token",
|
||||
"access_token",
|
||||
"auth_token",
|
||||
"bearer_token",
|
||||
"refresh_token",
|
||||
"api_key",
|
||||
"access_key",
|
||||
"access_key_id",
|
||||
"secret_key",
|
||||
"secret_access_key",
|
||||
"session_token",
|
||||
}
|
||||
)
|
||||
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
|
||||
|
||||
|
||||
class ConnectorDeploymentConfigurationError(ValueError):
|
||||
"""Raised when connector data crosses a deployment-owned trust boundary."""
|
||||
|
||||
|
||||
def connector_secret_env_value(name: str, *, source_kind: str) -> str:
|
||||
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||
value = os.environ.get(clean_name)
|
||||
if value is None or value == "":
|
||||
raise ConnectorDeploymentConfigurationError("Connector credential environment variable is not configured")
|
||||
return value
|
||||
|
||||
|
||||
def connector_secret_env_available(name: str | None, *, source_kind: str) -> bool:
|
||||
if not name:
|
||||
return False
|
||||
try:
|
||||
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||
except ConnectorDeploymentConfigurationError:
|
||||
return False
|
||||
return bool(os.environ.get(clean_name))
|
||||
|
||||
|
||||
def validate_deployment_connector_references(
|
||||
*,
|
||||
source_kind: str,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Validate references in deployment-owned connector configuration."""
|
||||
|
||||
for name in (password_env, token_env):
|
||||
if name:
|
||||
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||
for key in _SECRET_ENV_METADATA_KEYS:
|
||||
name = _clean((metadata or {}).get(key))
|
||||
if name:
|
||||
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||
validate_connector_tls_metadata(metadata)
|
||||
|
||||
|
||||
def reject_api_controlled_deployment_references(
|
||||
*,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Reject process-secret selectors controlled through tenant-facing APIs."""
|
||||
|
||||
if _clean(password_env) or _clean(token_env):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||
)
|
||||
if _clean(secret_ref):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"External secret references cannot be managed through the Files API until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
for key, value in _metadata_entries(metadata or {}):
|
||||
clean_key = str(key).strip().casefold()
|
||||
if _clean(value) and (clean_key in _SECRET_ENV_METADATA_KEYS or clean_key.endswith("_env")):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||
)
|
||||
if _clean(value) and (
|
||||
clean_key in _SECRET_VALUE_METADATA_KEYS
|
||||
or clean_key.endswith(("_password", "_secret", "_api_key"))
|
||||
):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Connector credential values must use the dedicated encrypted credential fields, not metadata"
|
||||
)
|
||||
validate_connector_tls_metadata(metadata)
|
||||
|
||||
|
||||
def _metadata_entries(value: object) -> list[tuple[str, object]]:
|
||||
entries: list[tuple[str, object]] = []
|
||||
if isinstance(value, Mapping):
|
||||
for key, item in value.items():
|
||||
entries.append((str(key), item))
|
||||
entries.extend(_metadata_entries(item))
|
||||
elif isinstance(value, (list, tuple)):
|
||||
for item in value:
|
||||
entries.extend(_metadata_entries(item))
|
||||
return entries
|
||||
|
||||
|
||||
def connector_ca_bundle_path(value: str) -> str:
|
||||
clean_value = _clean(value)
|
||||
if not clean_value:
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path is empty")
|
||||
candidate = Path(clean_value)
|
||||
if not candidate.is_absolute():
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle paths must be absolute")
|
||||
try:
|
||||
resolved = candidate.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path does not exist") from exc
|
||||
allowed = _allowed_ca_bundle_paths()
|
||||
if resolved not in allowed:
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
f"Connector CA bundle path is not listed in {_CA_BUNDLE_ALLOWLIST}"
|
||||
)
|
||||
if not resolved.is_file():
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path must be a regular file")
|
||||
return str(resolved)
|
||||
|
||||
|
||||
def validate_connector_tls_metadata(metadata: Mapping[str, Any] | None) -> None:
|
||||
values = metadata or {}
|
||||
ca_bundle = _clean(values.get("ca_bundle"))
|
||||
if ca_bundle:
|
||||
connector_ca_bundle_path(ca_bundle)
|
||||
for key in ("verify_tls", "tls_verify"):
|
||||
if key in values and not _as_bool(values.get(key), default=True) and not _development_runtime():
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Connector TLS certificate verification may only be disabled in dev/test environments"
|
||||
)
|
||||
|
||||
|
||||
def connector_effective_endpoint_url(
|
||||
*,
|
||||
provider: str | None,
|
||||
endpoint_url: str | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> str | None:
|
||||
"""Return the endpoint that connector I/O will actually use."""
|
||||
|
||||
clean_provider = (provider or "").strip().casefold()
|
||||
values = metadata or {}
|
||||
webdav_url = _clean(values.get("webdav_endpoint_url"))
|
||||
browse_protocol = (_clean(values.get("browse_protocol")) or "").casefold()
|
||||
if webdav_url and (clean_provider in {"seafile", "webdav", "nextcloud"} or browse_protocol == "webdav"):
|
||||
return webdav_url
|
||||
return _clean(endpoint_url)
|
||||
|
||||
|
||||
def _validate_secret_env_reference(name: str, *, source_kind: str) -> str:
|
||||
if source_kind.strip().casefold() != "settings":
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be used by deployment-owned connector configuration"
|
||||
)
|
||||
clean_name = name.strip()
|
||||
if not _ENV_NAME.fullmatch(clean_name):
|
||||
raise ConnectorDeploymentConfigurationError("Connector credential environment variable name is invalid")
|
||||
if clean_name not in _allowed_secret_env_names():
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
f"Connector credential environment variable is not listed in {_SECRET_ENV_ALLOWLIST}"
|
||||
)
|
||||
return clean_name
|
||||
|
||||
|
||||
def _allowed_secret_env_names() -> frozenset[str]:
|
||||
raw = os.environ.get(_SECRET_ENV_ALLOWLIST, "")
|
||||
names = frozenset(item.strip() for item in raw.split(",") if item.strip())
|
||||
invalid = sorted(name for name in names if not _ENV_NAME.fullmatch(name))
|
||||
if invalid:
|
||||
raise ConnectorDeploymentConfigurationError(f"{_SECRET_ENV_ALLOWLIST} contains an invalid variable name")
|
||||
return names
|
||||
|
||||
|
||||
def _allowed_ca_bundle_paths() -> frozenset[Path]:
|
||||
raw = os.environ.get(_CA_BUNDLE_ALLOWLIST, "")
|
||||
paths: set[Path] = set()
|
||||
for item in (part.strip() for part in raw.split(",")):
|
||||
if not item:
|
||||
continue
|
||||
path = Path(item)
|
||||
if not path.is_absolute():
|
||||
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} requires absolute paths")
|
||||
try:
|
||||
paths.add(path.resolve(strict=True))
|
||||
except OSError as exc:
|
||||
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} contains a missing path") from exc
|
||||
return frozenset(paths)
|
||||
|
||||
|
||||
def _development_runtime() -> bool:
|
||||
app_env = os.environ.get("APP_ENV", "dev").strip().casefold()
|
||||
install_profile = os.environ.get("GOVOPLAN_INSTALL_PROFILE", "").strip().casefold()
|
||||
return app_env in _DEVELOPMENT_ENVIRONMENTS and (
|
||||
not install_profile or install_profile in _DEVELOPMENT_ENVIRONMENTS
|
||||
)
|
||||
|
||||
|
||||
def _as_bool(value: object, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
clean = str(value).strip().casefold()
|
||||
if clean in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if clean in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
raise ConnectorDeploymentConfigurationError("Connector TLS verification setting must be true or false")
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
clean = str(value).strip()
|
||||
return clean or None
|
||||
@@ -0,0 +1,139 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseItem,
|
||||
browse_connector_profile,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorFolderSkip:
|
||||
path: str
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ConnectorFolderDiscovery:
|
||||
files: list[ConnectorBrowseItem] = field(default_factory=list)
|
||||
skipped: list[ConnectorFolderSkip] = field(default_factory=list)
|
||||
truncated: bool = False
|
||||
|
||||
|
||||
def discover_connector_folder(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
path: str,
|
||||
library_id: str | None,
|
||||
recursive: bool,
|
||||
max_files: int,
|
||||
max_depth: int,
|
||||
) -> ConnectorFolderDiscovery:
|
||||
"""Discover a bounded, deterministic set of files below one connector folder."""
|
||||
|
||||
root_path = normalize_connector_browse_path(path)
|
||||
queue: deque[tuple[str, int]] = deque([(root_path, 0)])
|
||||
visited_folders: set[str] = set()
|
||||
seen_files: set[tuple[str, str]] = set()
|
||||
result = ConnectorFolderDiscovery()
|
||||
|
||||
while queue:
|
||||
folder_path, depth = queue.popleft()
|
||||
if folder_path in visited_folders:
|
||||
continue
|
||||
visited_folders.add(folder_path)
|
||||
|
||||
continuation_token: str | None = None
|
||||
seen_tokens: set[str] = set()
|
||||
while True:
|
||||
items = browse_connector_profile(
|
||||
profile,
|
||||
path=folder_path,
|
||||
library_id=library_id,
|
||||
continuation_token=continuation_token,
|
||||
)
|
||||
for item in items:
|
||||
item_path = normalize_connector_browse_path(item.path)
|
||||
if item.kind == "file":
|
||||
identity = (str(item.external_id or ""), item_path)
|
||||
if identity in seen_files:
|
||||
continue
|
||||
if len(result.files) >= max_files:
|
||||
result.truncated = True
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path or folder_path,
|
||||
reason=f"The manual sync limit of {max_files} files was reached.",
|
||||
)
|
||||
)
|
||||
return result
|
||||
seen_files.add(identity)
|
||||
result.files.append(item)
|
||||
continue
|
||||
|
||||
if item.kind == "folder":
|
||||
if not recursive:
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason="Subfolder skipped because recursive sync is disabled.",
|
||||
)
|
||||
)
|
||||
elif depth >= max_depth:
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason=f"Subfolder skipped at the configured depth limit of {max_depth}.",
|
||||
)
|
||||
)
|
||||
else:
|
||||
queue.append((item_path, depth + 1))
|
||||
continue
|
||||
|
||||
result.skipped.append(
|
||||
ConnectorFolderSkip(
|
||||
path=item_path,
|
||||
reason=f"Connector item kind {item.kind!r} cannot be synchronized as a file.",
|
||||
)
|
||||
)
|
||||
|
||||
next_token = _next_continuation_token(items)
|
||||
if not next_token or next_token in seen_tokens:
|
||||
break
|
||||
seen_tokens.add(next_token)
|
||||
continuation_token = next_token
|
||||
|
||||
result.files.sort(key=lambda item: normalize_connector_browse_path(item.path).casefold())
|
||||
result.skipped.sort(key=lambda item: (item.path.casefold(), item.reason))
|
||||
return result
|
||||
|
||||
|
||||
def connector_relative_path(*, space_root: str, item_path: str) -> str:
|
||||
root = normalize_connector_browse_path(space_root)
|
||||
item = normalize_connector_browse_path(item_path)
|
||||
if not root:
|
||||
return item
|
||||
if item == root:
|
||||
return item.rsplit("/", 1)[-1]
|
||||
prefix = f"{root}/"
|
||||
if not item.startswith(prefix):
|
||||
raise ValueError("Connector returned a file outside the linked connector-space root")
|
||||
return item[len(prefix) :]
|
||||
|
||||
|
||||
def join_connector_path(*parts: str | None) -> str:
|
||||
return normalize_connector_browse_path(
|
||||
"/".join(str(part or "").strip("/\\") for part in parts if str(part or "").strip("/\\"))
|
||||
)
|
||||
|
||||
|
||||
def _next_continuation_token(items: list[ConnectorBrowseItem]) -> str | None:
|
||||
for item in reversed(items):
|
||||
value = item.metadata.get("next_continuation_token")
|
||||
if value is not None and str(value).strip():
|
||||
return str(value).strip()
|
||||
return None
|
||||
@@ -0,0 +1,358 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.security.outbound_http import response_limit
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
_clean,
|
||||
_metadata_string,
|
||||
_profile_password,
|
||||
_profile_token,
|
||||
_request_json,
|
||||
_smb_client_kwargs,
|
||||
_smb_location,
|
||||
_smb_stat_modified_at,
|
||||
_smb_stat_revision,
|
||||
_smb_stat_size,
|
||||
_smb_unc_path,
|
||||
_smbclient_module,
|
||||
_s3_bucket,
|
||||
_s3_client,
|
||||
_s3_object_key,
|
||||
_seafile_headers,
|
||||
_seafile_url,
|
||||
_webdav_url,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.paths import filename_from_path
|
||||
|
||||
|
||||
class ConnectorImportError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectorImportUnsupported(ConnectorImportError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorDownloadedFile:
|
||||
filename: str
|
||||
data: bytes
|
||||
content_type: str | None = None
|
||||
revision: str | None = None
|
||||
external_id: str | None = None
|
||||
external_url: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
def read_connector_file(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
library_id: str,
|
||||
path: str,
|
||||
max_bytes: int,
|
||||
) -> ConnectorDownloadedFile:
|
||||
max_bytes = min(max_bytes, response_limit("file"))
|
||||
if profile.provider == "seafile":
|
||||
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||
return _read_seafile_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||
if profile.provider in {"webdav", "nextcloud"}:
|
||||
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||
if profile.provider == "smb":
|
||||
return _read_smb_file(profile, path=path, max_bytes=max_bytes)
|
||||
if profile.provider == "s3":
|
||||
return _read_s3_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||
raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet")
|
||||
|
||||
|
||||
def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
repo_id = _clean(library_id)
|
||||
if not repo_id:
|
||||
raise ConnectorImportError("Seafile import requires library_id")
|
||||
file_path = "/" + normalize_connector_browse_path(path)
|
||||
if file_path == "/":
|
||||
raise ConnectorImportError("Seafile import requires a file path")
|
||||
try:
|
||||
headers = _seafile_headers(profile)
|
||||
detail = _request_json(
|
||||
"GET",
|
||||
_seafile_url(profile, f"api2/repos/{repo_id}/file/detail/"),
|
||||
headers=headers,
|
||||
params={"p": file_path},
|
||||
)
|
||||
if isinstance(detail, dict):
|
||||
size = _int(detail.get("size"))
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||
download_url = _request_json(
|
||||
"GET",
|
||||
_seafile_url(profile, f"api2/repos/{repo_id}/file/"),
|
||||
headers=headers,
|
||||
params={"p": file_path, "reuse": "1"},
|
||||
)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
if not isinstance(download_url, str) or not download_url.strip():
|
||||
raise ConnectorImportError("Seafile did not return a file download URL")
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
"GET",
|
||||
download_url,
|
||||
timeout=30.0,
|
||||
kind="file",
|
||||
max_bytes=max_bytes,
|
||||
label="Seafile file download",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc
|
||||
if response.status_code != 200:
|
||||
raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}")
|
||||
data = response.content
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"Seafile file exceeds limit of {max_bytes} bytes")
|
||||
detail = detail if isinstance(detail, dict) else {}
|
||||
filename = filename_from_path(str(detail.get("name") or path))
|
||||
content_type = response.headers.get("content-type") or mimetypes.guess_type(filename)[0]
|
||||
external_id = f"{repo_id}:{normalize_connector_browse_path(path)}"
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=_clean(detail.get("id") or detail.get("mtime") or detail.get("last_modified")),
|
||||
external_id=external_id,
|
||||
external_url=download_url,
|
||||
metadata={
|
||||
"library_id": repo_id,
|
||||
"library_path": normalize_connector_browse_path(path),
|
||||
"size": len(data),
|
||||
**({key: detail[key] for key in ("permission", "last_modified", "mtime", "uploader_email", "last_modifier_email") if key in detail}),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
root_url = _metadata_string(profile, "webdav_endpoint_url") or profile.endpoint_url
|
||||
if not root_url:
|
||||
raise ConnectorImportError("WebDAV connector profile does not define endpoint_url")
|
||||
file_path = normalize_connector_browse_path(path)
|
||||
if not file_path:
|
||||
raise ConnectorImportError("WebDAV import requires a file path")
|
||||
url = _webdav_url(root_url, file_path).rstrip("/")
|
||||
headers: dict[str, str] = {}
|
||||
auth: tuple[str, str] | None = None
|
||||
password = _profile_password(profile)
|
||||
token = _profile_token(profile)
|
||||
if profile.username and password:
|
||||
auth = (profile.username, password)
|
||||
elif token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import")
|
||||
try:
|
||||
response = request_connector_bytes(
|
||||
"GET",
|
||||
url,
|
||||
headers=headers,
|
||||
auth=auth,
|
||||
timeout=30.0,
|
||||
kind="file",
|
||||
max_bytes=max_bytes,
|
||||
label="WebDAV file download",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorImportError("Connector credentials were rejected")
|
||||
if response.status_code != 200:
|
||||
raise ConnectorImportError(f"WebDAV file download failed with HTTP {response.status_code}")
|
||||
length = _int(response.headers.get("content-length"))
|
||||
if length is not None and length > max_bytes:
|
||||
raise ConnectorImportError(f"WebDAV file exceeds limit of {max_bytes} bytes")
|
||||
data = response.content
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"WebDAV file exceeds limit of {max_bytes} bytes")
|
||||
filename = filename_from_path(file_path)
|
||||
revision = _clean(response.headers.get("etag") or response.headers.get("last-modified"))
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=response.headers.get("content-type") or mimetypes.guess_type(filename)[0],
|
||||
revision=revision,
|
||||
external_id=file_path,
|
||||
external_url=url,
|
||||
metadata={"path": file_path, "size": len(data)},
|
||||
)
|
||||
|
||||
|
||||
def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
file_path = normalize_connector_browse_path(path)
|
||||
if not file_path:
|
||||
raise ConnectorImportError("SMB import requires a file path")
|
||||
try:
|
||||
location = _smb_location(profile)
|
||||
unc_path = _smb_unc_path(location, file_path)
|
||||
smbclient = _smbclient_module()
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
size = _smb_stat_size(stat_result)
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
with smbclient.open_file(unc_path, mode="rb", **kwargs) as handle:
|
||||
data = handle.read(max_bytes + 1)
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
except ConnectorImportError:
|
||||
raise
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"SMB file download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"SMB file exceeds limit of {max_bytes} bytes")
|
||||
filename = filename_from_path(file_path)
|
||||
revision = _smb_stat_revision(stat_result)
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=mimetypes.guess_type(filename)[0],
|
||||
revision=revision,
|
||||
external_id=f"{location.share}:{file_path}",
|
||||
external_url=f"smb://{location.server}:{location.port}/{location.share}/{file_path}",
|
||||
metadata={
|
||||
"share": location.share,
|
||||
"path": file_path,
|
||||
"size": len(data),
|
||||
"modified_at": _smb_stat_modified_at(stat_result),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _s3_import_client(profile: ConnectorProfile) -> Any:
|
||||
try:
|
||||
return _s3_client(profile)
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
|
||||
|
||||
def _s3_object_detail(client: Any, *, bucket: str, key: str, max_bytes: int) -> dict[str, Any]:
|
||||
try:
|
||||
detail = client.head_object(Bucket=bucket, Key=key)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"S3 object metadata lookup failed: {exc}") from exc
|
||||
if not isinstance(detail, dict):
|
||||
raise ConnectorImportError("S3 connector returned invalid object metadata")
|
||||
size = _int(detail.get("ContentLength"))
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
return detail
|
||||
|
||||
|
||||
def _download_s3_object(
|
||||
client: Any,
|
||||
*,
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
max_bytes: int,
|
||||
) -> tuple[Any, bytes]:
|
||||
request: dict[str, object] = {"Bucket": bucket, "Key": key}
|
||||
if version_id:
|
||||
request["VersionId"] = version_id
|
||||
try:
|
||||
response = client.get_object(**request)
|
||||
body = response.get("Body")
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
return response, data
|
||||
|
||||
|
||||
def _s3_download_metadata(
|
||||
detail: dict[str, Any],
|
||||
*,
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
etag: str | None,
|
||||
size: int,
|
||||
) -> dict[str, Any]:
|
||||
metadata: dict[str, Any] = {
|
||||
"bucket": bucket,
|
||||
"key": key,
|
||||
"version_id": version_id,
|
||||
"etag": etag,
|
||||
"size": size,
|
||||
}
|
||||
for source_key, target_key in (
|
||||
("ChecksumSHA256", "checksum_sha256"),
|
||||
("ChecksumCRC32", "checksum_crc32"),
|
||||
("StorageClass", "storage_class"),
|
||||
):
|
||||
if source_key in detail:
|
||||
metadata[target_key] = detail[source_key]
|
||||
return metadata
|
||||
|
||||
|
||||
def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
if not bucket:
|
||||
raise ConnectorImportError("S3 import requires a bucket in library_id or profile metadata")
|
||||
key = _s3_object_key(profile, path)
|
||||
if not key:
|
||||
raise ConnectorImportError("S3 import requires an object key")
|
||||
client = _s3_import_client(profile)
|
||||
try:
|
||||
detail = _s3_object_detail(client, bucket=bucket, key=key, max_bytes=max_bytes)
|
||||
version_id = _clean(detail.get("VersionId"))
|
||||
response, data = _download_s3_object(
|
||||
client,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
|
||||
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
|
||||
filename = filename_from_path(key)
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=version_id or etag or _clean(detail.get("LastModified")),
|
||||
external_id=f"{bucket}:{key}",
|
||||
external_url=f"s3://{bucket}/{key}",
|
||||
metadata=_s3_download_metadata(
|
||||
detail,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
etag=etag,
|
||||
size=len(data),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _int(value: object) -> int | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
@@ -0,0 +1,310 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from fnmatch import fnmatchcase
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.core.policy import PolicyDecision, PolicySourceStep, normalize_policy_scope_type
|
||||
|
||||
|
||||
_ALLOW_KEYS = ("allow", "allowlist", "whitelist")
|
||||
_DENY_KEYS = ("deny", "denylist", "blacklist")
|
||||
|
||||
_FIELD_ALIASES = {
|
||||
"connectors": ("connectors", "connector_ids", "connector_id"),
|
||||
"credentials": ("credentials", "credential_ids", "credential_id"),
|
||||
"providers": ("providers", "provider"),
|
||||
"external_ids": ("external_ids", "external_id"),
|
||||
"external_paths": ("external_paths", "paths", "path_prefixes", "external_path"),
|
||||
"external_urls": ("external_urls", "urls", "external_url"),
|
||||
}
|
||||
|
||||
CONNECTOR_POLICY_FIELDS = tuple(_FIELD_ALIASES)
|
||||
|
||||
|
||||
class ConnectorPolicyDenied(RuntimeError):
|
||||
def __init__(self, decision: PolicyDecision) -> None:
|
||||
super().__init__(decision.reason or "Connector policy denied")
|
||||
self.decision = decision
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorAccessRequest:
|
||||
connector_id: str | None = None
|
||||
credential_id: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
external_path: str | None = None
|
||||
external_url: str | None = None
|
||||
operation: str = "access"
|
||||
|
||||
@classmethod
|
||||
def from_provenance(cls, value: Mapping[str, Any] | None, *, operation: str = "access") -> "ConnectorAccessRequest":
|
||||
payload = value or {}
|
||||
return cls(
|
||||
connector_id=_clean(payload.get("connector_id")),
|
||||
credential_id=_clean(payload.get("credential_id") or payload.get("connector_credential_id")),
|
||||
provider=_clean(payload.get("provider") or payload.get("source_type")),
|
||||
external_id=_clean(payload.get("external_id")),
|
||||
external_path=_clean(payload.get("external_path")),
|
||||
external_url=_clean(payload.get("external_url")),
|
||||
operation=_clean(operation) or "access",
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict[str, str | None]:
|
||||
return {
|
||||
"connector_id": self.connector_id,
|
||||
"credential_id": self.credential_id,
|
||||
"provider": self.provider,
|
||||
"external_id": self.external_id,
|
||||
"external_path": self.external_path,
|
||||
"external_url": self.external_url,
|
||||
"operation": self.operation,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorPolicySource:
|
||||
scope_type: str
|
||||
label: str
|
||||
scope_id: str | None = None
|
||||
policy: Mapping[str, Any] | None = None
|
||||
|
||||
def source_step(self, applied_fields: Iterable[str]) -> PolicySourceStep:
|
||||
return PolicySourceStep(
|
||||
scope_type=normalize_policy_scope_type(self.scope_type),
|
||||
scope_id=self.scope_id,
|
||||
label=self.label,
|
||||
applied_fields=tuple(dict.fromkeys(applied_fields)),
|
||||
policy=dict(self.policy or {}),
|
||||
)
|
||||
|
||||
|
||||
def connector_policy_sources_from_payload(value: object) -> list[ConnectorPolicySource]:
|
||||
if value is None or value == "":
|
||||
return []
|
||||
if isinstance(value, Mapping):
|
||||
raw_sources = value.get("sources")
|
||||
if raw_sources is None:
|
||||
raw_sources = [value]
|
||||
elif isinstance(value, list):
|
||||
raw_sources = value
|
||||
else:
|
||||
raise ValueError("connector_policy must be a JSON object or list")
|
||||
if not isinstance(raw_sources, list):
|
||||
raise ValueError("connector_policy.sources must be a list")
|
||||
return [_source_from_mapping(item) for item in raw_sources if isinstance(item, Mapping)]
|
||||
|
||||
|
||||
def connector_policy_applied_fields(policy: Mapping[str, Any] | None) -> tuple[str, ...]:
|
||||
return _applied_fields(_policy_mapping(policy))
|
||||
|
||||
|
||||
def filtered_connector_policy_source(source: ConnectorPolicySource, fields: Iterable[str]) -> ConnectorPolicySource:
|
||||
allowed_fields = {field for field in fields if field in _FIELD_ALIASES}
|
||||
if not allowed_fields:
|
||||
return ConnectorPolicySource(scope_type=source.scope_type, scope_id=source.scope_id, label=source.label, policy={})
|
||||
policy = _policy_mapping(source.policy)
|
||||
filtered: dict[str, Any] = {}
|
||||
for prefix, keys in (("allow", _ALLOW_KEYS), ("deny", _DENY_KEYS)):
|
||||
rules = _merged_rule(policy, keys)
|
||||
selected = {field: _string_list(rules.get(field)) for field in allowed_fields}
|
||||
selected = {field: values for field, values in selected.items() if values}
|
||||
if selected:
|
||||
filtered[prefix] = selected
|
||||
return ConnectorPolicySource(scope_type=source.scope_type, scope_id=source.scope_id, label=source.label, policy=filtered)
|
||||
|
||||
|
||||
def connector_policy_sources_for_fields(
|
||||
sources: Iterable[ConnectorPolicySource],
|
||||
fields: Iterable[str],
|
||||
) -> list[ConnectorPolicySource]:
|
||||
return [filtered_connector_policy_source(source, fields) for source in sources]
|
||||
|
||||
|
||||
def connector_policy_decision(
|
||||
request: ConnectorAccessRequest,
|
||||
sources: Iterable[ConnectorPolicySource],
|
||||
) -> PolicyDecision:
|
||||
source_list = list(sources)
|
||||
source_steps: list[PolicySourceStep] = []
|
||||
deny_matches: list[dict[str, Any]] = []
|
||||
allow_misses: list[dict[str, Any]] = []
|
||||
|
||||
for source in source_list:
|
||||
policy = _policy_mapping(source.policy)
|
||||
applied_fields = _applied_fields(policy)
|
||||
if applied_fields:
|
||||
source_steps.append(source.source_step(applied_fields))
|
||||
|
||||
deny_policy = _merged_rule(policy, _DENY_KEYS)
|
||||
for field, patterns in _rules_by_field(deny_policy).items():
|
||||
if _matches_field(request, field, patterns):
|
||||
deny_matches.append({
|
||||
"scope": source.source_step((f"deny.{field}",)).to_dict(),
|
||||
"field": field,
|
||||
"patterns": patterns,
|
||||
})
|
||||
|
||||
allow_policy = _merged_rule(policy, _ALLOW_KEYS)
|
||||
for field, patterns in _rules_by_field(allow_policy).items():
|
||||
if not _matches_field(request, field, patterns):
|
||||
allow_misses.append({
|
||||
"scope": source.source_step((f"allow.{field}",)).to_dict(),
|
||||
"field": field,
|
||||
"patterns": patterns,
|
||||
})
|
||||
|
||||
details = {
|
||||
"request": request.to_dict(),
|
||||
"deny_matches": deny_matches,
|
||||
"allow_misses": allow_misses,
|
||||
}
|
||||
if deny_matches:
|
||||
return PolicyDecision(
|
||||
allowed=False,
|
||||
reason="Connector access is denied by a blacklist rule.",
|
||||
source_path=tuple(source_steps),
|
||||
requirements=("connector_policy_denylist",),
|
||||
details=details,
|
||||
)
|
||||
if allow_misses:
|
||||
return PolicyDecision(
|
||||
allowed=False,
|
||||
reason="Connector access is not allowed by the configured whitelist.",
|
||||
source_path=tuple(source_steps),
|
||||
requirements=("connector_policy_allowlist",),
|
||||
details=details,
|
||||
)
|
||||
return PolicyDecision(
|
||||
allowed=True,
|
||||
reason=None,
|
||||
source_path=tuple(source_steps),
|
||||
requirements=(),
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
def ensure_connector_policy_allows(
|
||||
request: ConnectorAccessRequest,
|
||||
sources: Iterable[ConnectorPolicySource],
|
||||
) -> PolicyDecision:
|
||||
decision = connector_policy_decision(request, sources)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
return decision
|
||||
|
||||
|
||||
def _source_from_mapping(value: Mapping[str, Any]) -> ConnectorPolicySource:
|
||||
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||
scope_id = _clean(value.get("scope_id"))
|
||||
if scope_type == "system":
|
||||
scope_id = None
|
||||
elif not scope_id:
|
||||
raise ValueError(f"{scope_type} connector policy sources require scope_id")
|
||||
label = _clean(value.get("label")) or scope_type.capitalize()
|
||||
policy = value.get("policy")
|
||||
if policy is None:
|
||||
policy = {key: value[key] for key in (*_ALLOW_KEYS, *_DENY_KEYS) if key in value}
|
||||
if policy is not None and not isinstance(policy, Mapping):
|
||||
raise ValueError("connector policy source policy must be a JSON object")
|
||||
return ConnectorPolicySource(scope_type=scope_type, scope_id=scope_id, label=label, policy=policy or {})
|
||||
|
||||
|
||||
def _policy_mapping(value: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||
return value if isinstance(value, Mapping) else {}
|
||||
|
||||
|
||||
def _merged_rule(policy: Mapping[str, Any], keys: Iterable[str]) -> dict[str, Any]:
|
||||
merged: dict[str, Any] = {}
|
||||
for key in keys:
|
||||
raw = policy.get(key)
|
||||
if isinstance(raw, Mapping):
|
||||
merged.update(raw)
|
||||
return merged
|
||||
|
||||
|
||||
def _rules_by_field(value: Mapping[str, Any]) -> dict[str, list[str]]:
|
||||
result: dict[str, list[str]] = {}
|
||||
for field, aliases in _FIELD_ALIASES.items():
|
||||
items: list[str] = []
|
||||
for alias in aliases:
|
||||
items.extend(_string_list(value.get(alias)))
|
||||
if items:
|
||||
result[field] = list(dict.fromkeys(items))
|
||||
return result
|
||||
|
||||
|
||||
def _applied_fields(policy: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
fields: list[str] = []
|
||||
for prefix, keys in (("allow", _ALLOW_KEYS), ("deny", _DENY_KEYS)):
|
||||
rules = _merged_rule(policy, keys)
|
||||
fields.extend(f"{prefix}.{field}" for field in _rules_by_field(rules))
|
||||
return tuple(dict.fromkeys(fields))
|
||||
|
||||
|
||||
def _matches_field(request: ConnectorAccessRequest, field: str, patterns: list[str]) -> bool:
|
||||
if field == "connectors":
|
||||
return _matches_reference(request.connector_id, patterns)
|
||||
if field == "credentials":
|
||||
return _matches_reference(request.credential_id, patterns)
|
||||
if field == "providers":
|
||||
return _matches_reference(request.provider, patterns)
|
||||
if field == "external_ids":
|
||||
return _matches_glob(request.external_id, patterns)
|
||||
if field == "external_paths":
|
||||
return _matches_path(request.external_path, patterns)
|
||||
if field == "external_urls":
|
||||
return _matches_glob(request.external_url, patterns)
|
||||
return False
|
||||
|
||||
|
||||
def _matches_reference(value: str | None, patterns: list[str]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
clean = value.casefold()
|
||||
return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
|
||||
|
||||
|
||||
def _matches_glob(value: str | None, patterns: list[str]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
clean = value.casefold()
|
||||
return any(fnmatchcase(clean, pattern.casefold()) for pattern in patterns)
|
||||
|
||||
|
||||
def _matches_path(value: str | None, patterns: list[str]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
clean = value.replace("\\", "/").strip()
|
||||
for pattern in patterns:
|
||||
normalized = pattern.replace("\\", "/").strip()
|
||||
if normalized == "*":
|
||||
return True
|
||||
if any(token in normalized for token in "*?[]"):
|
||||
if fnmatchcase(clean.casefold(), normalized.casefold()):
|
||||
return True
|
||||
continue
|
||||
if clean.casefold() == normalized.casefold() or clean.casefold().startswith(normalized.rstrip("/").casefold() + "/"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
values = [value]
|
||||
elif isinstance(value, Iterable) and not isinstance(value, (bytes, bytearray, Mapping)):
|
||||
values = [str(item) for item in value]
|
||||
else:
|
||||
values = [str(value)]
|
||||
return [item.strip() for item in values if item.strip()]
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,334 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type
|
||||
from govoplan_files.backend.db.models import FileConnectorPolicy
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
CONNECTOR_POLICY_FIELDS,
|
||||
ConnectorPolicySource,
|
||||
connector_policy_applied_fields,
|
||||
)
|
||||
|
||||
_RULE_GROUPS = ("allow", "deny")
|
||||
_FIELD_ALIASES = {
|
||||
"connectors": ("connectors", "connector_ids", "connector_id"),
|
||||
"credentials": ("credentials", "credential_ids", "credential_id"),
|
||||
"providers": ("providers", "provider"),
|
||||
"external_ids": ("external_ids", "external_id"),
|
||||
"external_paths": ("external_paths", "paths", "path_prefixes", "external_path"),
|
||||
"external_urls": ("external_urls", "urls", "external_url"),
|
||||
}
|
||||
|
||||
|
||||
def get_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
return _normalize_connector_policy(row.policy if row is not None else None)
|
||||
|
||||
|
||||
def set_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str | None,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
normalized = _normalize_connector_policy(policy)
|
||||
parent = parent_connector_policy(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
if parent is not None:
|
||||
_validate_lower_level_limits(parent, normalized)
|
||||
row_tenant_id, row_scope_type, row_scope_id = _policy_scope_key(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=row_scope_type, scope_id=row_scope_id)
|
||||
if row is None:
|
||||
row = FileConnectorPolicy(
|
||||
tenant_id=row_tenant_id,
|
||||
scope_type=row_scope_type,
|
||||
scope_id=row_scope_id,
|
||||
policy=normalized,
|
||||
created_by_user_id=user_id,
|
||||
updated_by_user_id=user_id,
|
||||
)
|
||||
else:
|
||||
row.policy = normalized
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return normalized
|
||||
|
||||
|
||||
def connector_policy_response(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
policy = get_connector_policy(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
effective_sources = effective_connector_policy_sources(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
parent_policy = parent_connector_policy(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
parent_sources = parent_connector_policy_sources(session, tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
return {
|
||||
"scope_type": clean_scope_type,
|
||||
"scope_id": clean_scope_id,
|
||||
"policy": policy,
|
||||
"effective_policy": merge_connector_policies(source.policy for source in effective_sources),
|
||||
"parent_policy": parent_policy,
|
||||
"effective_policy_sources": [_source_step(source) for source in effective_sources],
|
||||
"parent_policy_sources": [_source_step(source) for source in parent_sources],
|
||||
}
|
||||
|
||||
|
||||
def effective_connector_policy_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> list[ConnectorPolicySource]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
sources: list[ConnectorPolicySource] = []
|
||||
for source_scope_type, source_scope_id in _policy_source_chain(tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id):
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=source_scope_type, scope_id=source_scope_id)
|
||||
if row is None:
|
||||
continue
|
||||
policy = _normalize_connector_policy(row.policy)
|
||||
if _policy_is_empty(policy):
|
||||
continue
|
||||
sources.append(
|
||||
ConnectorPolicySource(
|
||||
scope_type=source_scope_type,
|
||||
scope_id=source_scope_id,
|
||||
label=_policy_source_label(source_scope_type),
|
||||
policy=policy,
|
||||
)
|
||||
)
|
||||
return sources
|
||||
|
||||
|
||||
def parent_connector_policy_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> list[ConnectorPolicySource]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
chain = _policy_source_chain(tenant_id=tenant_id, scope_type=clean_scope_type, scope_id=clean_scope_id)
|
||||
parent_chain = chain[:-1] if chain else []
|
||||
sources: list[ConnectorPolicySource] = []
|
||||
for source_scope_type, source_scope_id in parent_chain:
|
||||
row = _connector_policy_row(session, tenant_id=tenant_id, scope_type=source_scope_type, scope_id=source_scope_id)
|
||||
if row is None:
|
||||
continue
|
||||
policy = _normalize_connector_policy(row.policy)
|
||||
if _policy_is_empty(policy):
|
||||
continue
|
||||
sources.append(
|
||||
ConnectorPolicySource(
|
||||
scope_type=source_scope_type,
|
||||
scope_id=source_scope_id,
|
||||
label=_policy_source_label(source_scope_type),
|
||||
policy=policy,
|
||||
)
|
||||
)
|
||||
return sources
|
||||
|
||||
|
||||
def parent_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
sources = parent_connector_policy_sources(session, tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
if not sources:
|
||||
return None
|
||||
return merge_connector_policies(source.policy for source in sources)
|
||||
|
||||
|
||||
def validate_connector_policy_allowed_by_parent(parent_policy: Mapping[str, Any] | None, local_policy: Mapping[str, Any] | None) -> None:
|
||||
if parent_policy is None:
|
||||
return
|
||||
_validate_lower_level_limits(_normalize_connector_policy(parent_policy), _normalize_connector_policy(local_policy))
|
||||
|
||||
|
||||
def merge_connector_policies(policies: Iterable[Mapping[str, Any] | None]) -> dict[str, Any]:
|
||||
result = _empty_connector_policy()
|
||||
for policy in policies:
|
||||
normalized = _normalize_connector_policy(policy)
|
||||
for group in _RULE_GROUPS:
|
||||
rules = normalized.get(group) or {}
|
||||
if not isinstance(rules, Mapping):
|
||||
continue
|
||||
target = result[group]
|
||||
for field in CONNECTOR_POLICY_FIELDS:
|
||||
values = _string_list(rules.get(field))
|
||||
if values:
|
||||
target[field] = _unique([*target.get(field, []), *values])
|
||||
lower = normalized.get("allow_lower_level_limits") or {}
|
||||
if isinstance(lower, Mapping):
|
||||
for key, value in lower.items():
|
||||
if isinstance(value, bool):
|
||||
result["allow_lower_level_limits"][str(key)] = value
|
||||
return _compact_connector_policy(result, keep_lower=True)
|
||||
|
||||
|
||||
def _connector_policy_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
) -> FileConnectorPolicy | None:
|
||||
row_tenant_id, row_scope_type, row_scope_id = _policy_scope_key(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
query = session.query(FileConnectorPolicy).filter(FileConnectorPolicy.scope_type == row_scope_type)
|
||||
query = query.filter(FileConnectorPolicy.tenant_id.is_(None) if row_tenant_id is None else FileConnectorPolicy.tenant_id == row_tenant_id)
|
||||
query = query.filter(FileConnectorPolicy.scope_id.is_(None) if row_scope_id is None else FileConnectorPolicy.scope_id == row_scope_id)
|
||||
return query.order_by(FileConnectorPolicy.created_at.asc()).first()
|
||||
|
||||
|
||||
def _policy_scope_key(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str | None, str, str | None]:
|
||||
clean_scope_type, clean_scope_id = _policy_scope_ref(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return None, "system", None
|
||||
return tenant_id, clean_scope_type, clean_scope_id
|
||||
|
||||
|
||||
def _policy_scope_ref(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return "system", None
|
||||
if clean_scope_type == "tenant":
|
||||
if clean_scope_id and clean_scope_id != tenant_id:
|
||||
raise FileStorageError("Tenant connector policy scope does not belong to the active tenant")
|
||||
return "tenant", tenant_id
|
||||
if clean_scope_type in {"user", "group", "campaign"}:
|
||||
if not clean_scope_id:
|
||||
raise FileStorageError(f"{clean_scope_type.capitalize()} connector policy requires scope_id")
|
||||
return clean_scope_type, clean_scope_id
|
||||
raise FileStorageError("Unsupported connector policy scope")
|
||||
|
||||
|
||||
def _policy_source_chain(*, tenant_id: str, scope_type: str, scope_id: str | None) -> list[tuple[str, str | None]]:
|
||||
chain: list[tuple[str, str | None]] = [("system", None)]
|
||||
if scope_type == "system":
|
||||
return chain
|
||||
chain.append(("tenant", tenant_id))
|
||||
if scope_type == "tenant":
|
||||
return chain
|
||||
chain.append((scope_type, scope_id))
|
||||
return chain
|
||||
|
||||
|
||||
def _normalize_connector_policy(policy: Mapping[str, Any] | None) -> dict[str, Any]:
|
||||
if policy is None:
|
||||
return {}
|
||||
if not isinstance(policy, Mapping):
|
||||
raise FileStorageError("Connector policy must be a JSON object")
|
||||
normalized = _empty_connector_policy()
|
||||
for group in _RULE_GROUPS:
|
||||
raw = policy.get(group)
|
||||
if raw is None:
|
||||
raw = policy.get(f"{group}list")
|
||||
if raw is None:
|
||||
raw = policy.get("whitelist" if group == "allow" else "blacklist")
|
||||
if raw is not None and not isinstance(raw, Mapping):
|
||||
raise FileStorageError(f"Connector policy {group} rules must be an object")
|
||||
for field, aliases in _FIELD_ALIASES.items():
|
||||
values: list[str] = []
|
||||
if isinstance(raw, Mapping):
|
||||
for alias in aliases:
|
||||
values.extend(_string_list(raw.get(alias)))
|
||||
if values:
|
||||
normalized[group][field] = _unique(values)
|
||||
lower = policy.get("allow_lower_level_limits")
|
||||
if lower is not None:
|
||||
if not isinstance(lower, Mapping):
|
||||
raise FileStorageError("Connector policy allow_lower_level_limits must be an object")
|
||||
for key, value in lower.items():
|
||||
clean_key = str(key).strip()
|
||||
if clean_key and isinstance(value, bool):
|
||||
normalized["allow_lower_level_limits"][clean_key] = value
|
||||
return _compact_connector_policy(normalized, keep_lower=True)
|
||||
|
||||
|
||||
def _empty_connector_policy() -> dict[str, Any]:
|
||||
return {"allow": {}, "deny": {}, "allow_lower_level_limits": {}}
|
||||
|
||||
|
||||
def _compact_connector_policy(policy: dict[str, Any], *, keep_lower: bool = False) -> dict[str, Any]:
|
||||
compact: dict[str, Any] = {}
|
||||
for group in _RULE_GROUPS:
|
||||
rules = {field: _unique(_string_list(values)) for field, values in (policy.get(group) or {}).items()}
|
||||
rules = {field: values for field, values in rules.items() if values}
|
||||
if rules:
|
||||
compact[group] = rules
|
||||
lower = policy.get("allow_lower_level_limits") or {}
|
||||
if isinstance(lower, Mapping):
|
||||
lower_compact = {str(key): bool(value) for key, value in lower.items() if isinstance(value, bool)}
|
||||
if lower_compact or keep_lower:
|
||||
compact["allow_lower_level_limits"] = lower_compact
|
||||
return compact
|
||||
|
||||
|
||||
def _policy_is_empty(policy: Mapping[str, Any]) -> bool:
|
||||
return not connector_policy_applied_fields(policy) and not bool(policy.get("allow_lower_level_limits"))
|
||||
|
||||
|
||||
def _source_step(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||
return source.source_step(connector_policy_applied_fields(source.policy)).to_dict()
|
||||
|
||||
|
||||
def _policy_source_label(scope_type: str) -> str:
|
||||
if scope_type == "system":
|
||||
return "System connector policy"
|
||||
if scope_type == "tenant":
|
||||
return "Tenant connector policy"
|
||||
return f"{scope_type.capitalize()} connector policy"
|
||||
|
||||
|
||||
def _validate_lower_level_limits(parent_policy: Mapping[str, Any], local_policy: Mapping[str, Any]) -> None:
|
||||
limits = parent_policy.get("allow_lower_level_limits")
|
||||
if not isinstance(limits, Mapping):
|
||||
return
|
||||
for field in connector_policy_applied_fields(local_policy):
|
||||
allowed = limits.get(field)
|
||||
if allowed is False:
|
||||
raise FileStorageError(f"Parent connector policy does not allow lower-level {field} limits")
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
clean = value.strip()
|
||||
return [clean] if clean else []
|
||||
if isinstance(value, Iterable) and not isinstance(value, (str, bytes, bytearray, Mapping)):
|
||||
return [str(item).strip() for item in value if str(item).strip()]
|
||||
return [str(value).strip()] if str(value).strip() else []
|
||||
|
||||
|
||||
def _unique(values: Iterable[str]) -> list[str]:
|
||||
return list(dict.fromkeys(value for value in values if value))
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,470 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type
|
||||
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
ConnectorCredential,
|
||||
connector_credential_from_row,
|
||||
credential_rows_by_id,
|
||||
resolve_reusable_connector_credential,
|
||||
reusable_credential_id,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
|
||||
|
||||
|
||||
def list_database_connector_profiles(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||
) -> list[ConnectorProfile]:
|
||||
profiles, _profile_ids = select_database_connector_profiles(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
row_visible=row_visible,
|
||||
)
|
||||
return profiles
|
||||
|
||||
|
||||
def select_database_connector_profiles(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
|
||||
) -> tuple[list[ConnectorProfile], set[str]]:
|
||||
"""Return visible profiles and every database id that shadows settings."""
|
||||
|
||||
query = session.query(FileConnectorProfile).filter(
|
||||
(FileConnectorProfile.scope_type == "system")
|
||||
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||
)
|
||||
if not include_disabled:
|
||||
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||
profile_ids = {row.id for row in rows}
|
||||
if row_visible is not None:
|
||||
rows = [row for row in rows if row_visible(row)]
|
||||
credential_ids = {
|
||||
_clean(row.credential_profile_id)
|
||||
for row in rows
|
||||
if _clean(row.credential_profile_id) and not reusable_credential_id(row.credential_profile_id)
|
||||
}
|
||||
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
|
||||
return (
|
||||
[
|
||||
connector_profile_from_row(
|
||||
row,
|
||||
credential_row=(
|
||||
_resolve_profile_reusable_credential(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
row=row,
|
||||
)
|
||||
if reusable_credential_id(row.credential_profile_id)
|
||||
else credentials.get(row.credential_profile_id or "")
|
||||
),
|
||||
)
|
||||
for row in rows
|
||||
],
|
||||
profile_ids,
|
||||
)
|
||||
|
||||
|
||||
def list_connector_profile_rows(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> list[FileConnectorProfile]:
|
||||
query = session.query(FileConnectorProfile).filter(
|
||||
(FileConnectorProfile.scope_type == "system")
|
||||
| (FileConnectorProfile.tenant_id == tenant_id)
|
||||
)
|
||||
if not include_disabled:
|
||||
query = query.filter(FileConnectorProfile.enabled.is_(True))
|
||||
return query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
|
||||
|
||||
|
||||
def connector_profile_from_row(
|
||||
row: FileConnectorProfile,
|
||||
credential_row: FileConnectorCredential | ConnectorCredential | None = None,
|
||||
) -> ConnectorProfile:
|
||||
policy_sources = []
|
||||
if row.policy:
|
||||
policy_sources = connector_policy_sources_from_payload({
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"label": row.label,
|
||||
"policy": row.policy,
|
||||
})
|
||||
credential = (
|
||||
credential_row
|
||||
if isinstance(credential_row, ConnectorCredential)
|
||||
else connector_credential_from_row(credential_row)
|
||||
if credential_row is not None
|
||||
else None
|
||||
)
|
||||
if credential:
|
||||
policy_sources.extend(credential.policy_sources)
|
||||
metadata = dict(row.metadata_ or {})
|
||||
if reusable_credential_id(row.credential_profile_id) and credential is None:
|
||||
metadata["credential_unavailable"] = True
|
||||
return ConnectorProfile(
|
||||
id=row.id,
|
||||
label=row.label,
|
||||
provider=row.provider,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
endpoint_url=row.endpoint_url,
|
||||
base_path=row.base_path,
|
||||
enabled=row.enabled,
|
||||
credential_profile_id=row.credential_profile_id,
|
||||
credential_profile_label=credential.label if credential else None,
|
||||
credential_mode=credential.credential_mode if credential else row.credential_mode,
|
||||
username=credential.username if credential else row.username,
|
||||
password_env=credential.password_env if credential else row.password_env,
|
||||
token_env=credential.token_env if credential else row.token_env,
|
||||
secret_ref=credential.secret_ref if credential else row.secret_ref,
|
||||
password_value=credential.password_value if credential else decrypt_secret(row.password_encrypted),
|
||||
token_value=credential.token_value if credential else decrypt_secret(row.token_encrypted),
|
||||
capabilities=tuple(_string_list(row.capabilities)),
|
||||
policy_sources=tuple(policy_sources),
|
||||
metadata=metadata,
|
||||
source_kind="database",
|
||||
)
|
||||
|
||||
|
||||
def _resolve_profile_reusable_credential(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
row: FileConnectorProfile,
|
||||
) -> ConnectorCredential | None:
|
||||
try:
|
||||
return resolve_reusable_connector_credential(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
credential_ref=row.credential_profile_id or "",
|
||||
profile_id=row.id,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
)
|
||||
except FileStorageError:
|
||||
return None
|
||||
|
||||
|
||||
def get_connector_profile_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
profile_id: str,
|
||||
include_disabled: bool = False,
|
||||
) -> FileConnectorProfile:
|
||||
row = session.get(FileConnectorProfile, profile_id)
|
||||
if row is None or (row.scope_type != "system" and row.tenant_id != tenant_id):
|
||||
raise FileStorageError("Connector profile not found")
|
||||
if not include_disabled and not row.enabled:
|
||||
raise FileStorageError("Connector profile not found")
|
||||
return row
|
||||
|
||||
|
||||
def create_connector_profile_row(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str | None,
|
||||
profile_id: str,
|
||||
label: str,
|
||||
provider: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None = None,
|
||||
endpoint_url: str | None = None,
|
||||
base_path: str | None = None,
|
||||
enabled: bool = True,
|
||||
credential_profile_id: str | None = None,
|
||||
credential_mode: str = "none",
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
capabilities: list[str] | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> FileConnectorProfile:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
clean_id = _normalize_profile_id(profile_id)
|
||||
if session.get(FileConnectorProfile, clean_id) is not None:
|
||||
raise FileStorageError(f"Connector profile already exists: {clean_id}")
|
||||
clean_scope_type, clean_scope_id, row_tenant_id = _normalize_scope(tenant_id=tenant_id, scope_type=scope_type, scope_id=scope_id)
|
||||
row = FileConnectorProfile(
|
||||
id=clean_id,
|
||||
tenant_id=row_tenant_id,
|
||||
scope_type=clean_scope_type,
|
||||
scope_id=clean_scope_id,
|
||||
label=_normalize_label(label),
|
||||
provider=_normalize_provider(provider),
|
||||
endpoint_url=_clean(endpoint_url),
|
||||
base_path=_clean(base_path),
|
||||
enabled=bool(enabled),
|
||||
credential_profile_id=_clean(credential_profile_id),
|
||||
credential_mode=_normalize_credential_mode(credential_mode),
|
||||
username=_clean(username),
|
||||
password_encrypted=encrypt_secret(_clean(password)),
|
||||
token_encrypted=encrypt_secret(_clean(token)),
|
||||
password_env=_clean(password_env),
|
||||
token_env=_clean(token_env),
|
||||
secret_ref=_clean(secret_ref),
|
||||
capabilities=_string_list(capabilities),
|
||||
policy=dict(policy or {}),
|
||||
metadata_=dict(metadata or {}),
|
||||
created_by_user_id=user_id,
|
||||
updated_by_user_id=user_id,
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def update_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
user_id: str | None,
|
||||
label: str | None = None,
|
||||
provider: str | None = None,
|
||||
endpoint_url: str | None = None,
|
||||
base_path: str | None = None,
|
||||
enabled: bool | None = None,
|
||||
credential_profile_id: str | None = None,
|
||||
credential_mode: str | None = None,
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
token: str | None = None,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
capabilities: list[str] | None = None,
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
clear_password: bool = False,
|
||||
clear_token: bool = False,
|
||||
) -> FileConnectorProfile:
|
||||
_validate_profile_update_references(
|
||||
row,
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
_update_profile_connection_fields(
|
||||
row,
|
||||
label=label,
|
||||
provider=provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=base_path,
|
||||
enabled=enabled,
|
||||
credential_profile_id=credential_profile_id,
|
||||
credential_mode=credential_mode,
|
||||
)
|
||||
_update_profile_credential_fields(
|
||||
row,
|
||||
username=username,
|
||||
password=password,
|
||||
token=token,
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
clear_password=clear_password,
|
||||
clear_token=clear_token,
|
||||
)
|
||||
_update_profile_governance_fields(
|
||||
row,
|
||||
capabilities=capabilities,
|
||||
policy=policy,
|
||||
metadata=metadata,
|
||||
)
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def _validate_profile_update_references(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
password_env: str | None,
|
||||
token_env: str | None,
|
||||
secret_ref: str | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> None:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
if secret_ref is None or _clean(secret_ref) == _clean(row.secret_ref):
|
||||
return
|
||||
if _clean(row.secret_ref):
|
||||
raise FileStorageError(
|
||||
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
|
||||
|
||||
def _update_profile_connection_fields(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
label: str | None,
|
||||
provider: str | None,
|
||||
endpoint_url: str | None,
|
||||
base_path: str | None,
|
||||
enabled: bool | None,
|
||||
credential_profile_id: str | None,
|
||||
credential_mode: str | None,
|
||||
) -> None:
|
||||
if label is not None:
|
||||
row.label = _normalize_label(label)
|
||||
if provider is not None:
|
||||
row.provider = _normalize_provider(provider)
|
||||
if endpoint_url is not None:
|
||||
row.endpoint_url = _clean(endpoint_url)
|
||||
if base_path is not None:
|
||||
row.base_path = _clean(base_path)
|
||||
if enabled is not None:
|
||||
row.enabled = bool(enabled)
|
||||
if credential_profile_id is not None:
|
||||
row.credential_profile_id = _clean(credential_profile_id)
|
||||
if credential_mode is not None:
|
||||
row.credential_mode = _normalize_credential_mode(credential_mode)
|
||||
|
||||
|
||||
def _update_profile_credential_fields(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
username: str | None,
|
||||
password: str | None,
|
||||
token: str | None,
|
||||
password_env: str | None,
|
||||
token_env: str | None,
|
||||
secret_ref: str | None,
|
||||
clear_password: bool,
|
||||
clear_token: bool,
|
||||
) -> None:
|
||||
if username is not None:
|
||||
row.username = _clean(username)
|
||||
if password is not None:
|
||||
row.password_encrypted = encrypt_secret(_clean(password))
|
||||
elif clear_password:
|
||||
row.password_encrypted = None
|
||||
if token is not None:
|
||||
row.token_encrypted = encrypt_secret(_clean(token))
|
||||
elif clear_token:
|
||||
row.token_encrypted = None
|
||||
if password_env is not None:
|
||||
row.password_env = _clean(password_env)
|
||||
if token_env is not None:
|
||||
row.token_env = _clean(token_env)
|
||||
if secret_ref is not None:
|
||||
row.secret_ref = _clean(secret_ref)
|
||||
|
||||
|
||||
def _update_profile_governance_fields(
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
capabilities: list[str] | None,
|
||||
policy: Mapping[str, Any] | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> None:
|
||||
if capabilities is not None:
|
||||
row.capabilities = _string_list(capabilities)
|
||||
if policy is not None:
|
||||
row.policy = dict(policy)
|
||||
if metadata is not None:
|
||||
row.metadata_ = dict(metadata)
|
||||
|
||||
|
||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
if clean_scope_type == "system":
|
||||
return "system", None, None
|
||||
if clean_scope_type == "tenant":
|
||||
return "tenant", tenant_id, tenant_id
|
||||
if clean_scope_type in {"user", "group", "campaign"}:
|
||||
if not clean_scope_id:
|
||||
raise FileStorageError(f"{clean_scope_type.capitalize()} connector profiles require scope_id")
|
||||
return clean_scope_type, clean_scope_id, tenant_id
|
||||
raise FileStorageError("Unsupported connector profile scope")
|
||||
|
||||
|
||||
def _normalize_profile_id(value: str) -> str:
|
||||
clean = _clean(value)
|
||||
if not clean:
|
||||
raise FileStorageError("Connector profile id is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector profile id is too long")
|
||||
if any(char.isspace() for char in clean):
|
||||
raise FileStorageError("Connector profile id cannot contain whitespace")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_label(value: str) -> str:
|
||||
clean = value.strip()
|
||||
if not clean:
|
||||
raise FileStorageError("Connector profile label is required")
|
||||
if len(clean) > 255:
|
||||
raise FileStorageError("Connector profile label is too long")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_provider(value: str) -> str:
|
||||
clean = value.strip().casefold()
|
||||
if clean not in supported_connector_providers():
|
||||
raise FileStorageError(f"Unsupported connector provider: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _normalize_credential_mode(value: str) -> str:
|
||||
clean = value.strip().casefold() or "none"
|
||||
if clean not in {"none", "anonymous", "basic", "token", "secret_ref"}:
|
||||
raise FileStorageError(f"Unsupported connector credential mode: {value}")
|
||||
return clean
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
values = value.split(",")
|
||||
elif isinstance(value, (list, tuple, set)):
|
||||
values = value
|
||||
else:
|
||||
values = [value]
|
||||
return [str(item).strip() for item in values if str(item).strip()]
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,343 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_secret_env_available,
|
||||
validate_deployment_connector_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||
|
||||
|
||||
_ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON")
|
||||
_ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE")
|
||||
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"}
|
||||
_INLINE_SECRET_FIELDS = (
|
||||
"password",
|
||||
"token",
|
||||
"api_key",
|
||||
"access_key",
|
||||
"access_key_id",
|
||||
"secret_key",
|
||||
"secret_access_key",
|
||||
"session_token",
|
||||
)
|
||||
|
||||
|
||||
def supported_connector_providers() -> set[str]:
|
||||
return set(_SUPPORTED_PROVIDERS)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorProfile:
|
||||
id: str
|
||||
label: str
|
||||
provider: str
|
||||
scope_type: str = "system"
|
||||
scope_id: str | None = None
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
credential_profile_id: str | None = None
|
||||
credential_profile_label: str | None = None
|
||||
credential_mode: str = "none"
|
||||
username: str | None = None
|
||||
password_env: str | None = None
|
||||
token_env: str | None = None
|
||||
secret_ref: str | None = None
|
||||
password_value: str | None = field(default=None, repr=False)
|
||||
token_value: str | None = field(default=None, repr=False)
|
||||
has_inline_secret: bool = False
|
||||
capabilities: tuple[str, ...] = ()
|
||||
policy_sources: tuple[ConnectorPolicySource, ...] = ()
|
||||
metadata: Mapping[str, Any] = field(default_factory=dict)
|
||||
source_kind: str = "settings"
|
||||
|
||||
@property
|
||||
def source_path(self) -> str:
|
||||
return policy_source_path(self.scope_type, self.scope_id)
|
||||
|
||||
@property
|
||||
def credential_source(self) -> str | None:
|
||||
if self.credential_profile_id:
|
||||
return "credential_profile"
|
||||
if self.secret_ref:
|
||||
return "secret_ref"
|
||||
if self.token_value or self.password_value:
|
||||
return "stored"
|
||||
if self.token_env:
|
||||
return "token_env"
|
||||
if self.password_env:
|
||||
return "password_env"
|
||||
if self.has_inline_secret:
|
||||
return "inline"
|
||||
return None
|
||||
|
||||
@property
|
||||
def credentials_configured(self) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
mode = self.credential_mode.casefold()
|
||||
if mode in {"", "none", "anonymous"}:
|
||||
return True
|
||||
if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value:
|
||||
return True
|
||||
if self.token_env:
|
||||
return connector_secret_env_available(self.token_env, source_kind=self.source_kind)
|
||||
if self.password_env:
|
||||
return connector_secret_env_available(self.password_env, source_kind=self.source_kind)
|
||||
return False
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
"id": self.id,
|
||||
"label": self.label,
|
||||
"provider": self.provider,
|
||||
"endpoint_url": self.endpoint_url,
|
||||
"base_path": self.base_path,
|
||||
"enabled": self.enabled,
|
||||
"scope_type": self.scope_type,
|
||||
"scope_id": self.scope_id,
|
||||
"source_path": self.source_path,
|
||||
"credential_profile_id": self.credential_profile_id,
|
||||
"credential_profile_label": self.credential_profile_label,
|
||||
"credential_mode": self.credential_mode,
|
||||
"credential_source": self.credential_source,
|
||||
"credentials_configured": self.credentials_configured,
|
||||
"username": self.username,
|
||||
"capabilities": list(self.capabilities),
|
||||
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||
"metadata": _response_metadata(self.metadata),
|
||||
"source_kind": self.source_kind,
|
||||
}
|
||||
|
||||
|
||||
def connector_profiles_from_settings(settings: object | None = None) -> list[ConnectorProfile]:
|
||||
raw = _raw_profiles_payload(settings)
|
||||
if raw in (None, ""):
|
||||
return []
|
||||
payload = json.loads(raw) if isinstance(raw, str) else raw
|
||||
return connector_profiles_from_payload(payload)
|
||||
|
||||
|
||||
def connector_profiles_from_payload(value: object) -> list[ConnectorProfile]:
|
||||
if isinstance(value, Mapping):
|
||||
raw_profiles = value.get("profiles", [])
|
||||
elif isinstance(value, list):
|
||||
raw_profiles = value
|
||||
else:
|
||||
raise ValueError("Connector profiles must be a JSON object with profiles or a list")
|
||||
if not isinstance(raw_profiles, list):
|
||||
raise ValueError("Connector profiles payload requires a profiles list")
|
||||
return [_profile_from_mapping(item) for item in raw_profiles if isinstance(item, Mapping)]
|
||||
|
||||
|
||||
def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
|
||||
profile_id = _profile_id_from_mapping(value)
|
||||
provider = _provider_from_mapping(value)
|
||||
scope_type, scope_id = _scope_from_mapping(value)
|
||||
credentials = _credentials_from_mapping(value)
|
||||
mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none"
|
||||
profile = _profile_from_normalized_mapping(
|
||||
value,
|
||||
profile_id=profile_id,
|
||||
provider=provider,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
credential_mode=mode,
|
||||
credentials=credentials,
|
||||
)
|
||||
result = ConnectorProfile(
|
||||
id=profile.id,
|
||||
label=profile.label,
|
||||
provider=profile.provider,
|
||||
scope_type=profile.scope_type,
|
||||
scope_id=profile.scope_id,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
enabled=profile.enabled,
|
||||
credential_profile_id=profile.credential_profile_id,
|
||||
credential_profile_label=profile.credential_profile_label,
|
||||
credential_mode=profile.credential_mode,
|
||||
username=profile.username,
|
||||
password_env=profile.password_env,
|
||||
token_env=profile.token_env,
|
||||
secret_ref=profile.secret_ref,
|
||||
password_value=profile.password_value,
|
||||
token_value=profile.token_value,
|
||||
has_inline_secret=profile.has_inline_secret,
|
||||
capabilities=profile.capabilities,
|
||||
policy_sources=tuple(_policy_sources_from_profile(value, profile)),
|
||||
metadata=profile.metadata,
|
||||
source_kind="settings",
|
||||
)
|
||||
validate_deployment_connector_references(
|
||||
source_kind=result.source_kind,
|
||||
password_env=result.password_env,
|
||||
token_env=result.token_env,
|
||||
metadata=result.metadata,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _profile_id_from_mapping(value: Mapping[str, Any]) -> str:
|
||||
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
|
||||
if not profile_id:
|
||||
raise ValueError("Connector profiles require id")
|
||||
return profile_id
|
||||
|
||||
|
||||
def _provider_from_mapping(value: Mapping[str, Any]) -> str:
|
||||
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
|
||||
if provider not in _SUPPORTED_PROVIDERS:
|
||||
raise ValueError(f"Unsupported connector provider: {provider}")
|
||||
return provider
|
||||
|
||||
|
||||
def _scope_from_mapping(value: Mapping[str, Any]) -> tuple[str, str | None]:
|
||||
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||
scope_id = _clean(value.get("scope_id"))
|
||||
if scope_type == "system":
|
||||
return scope_type, None
|
||||
if not scope_id:
|
||||
raise ValueError(f"{scope_type} connector profiles require scope_id")
|
||||
return scope_type, scope_id
|
||||
|
||||
|
||||
def _credentials_from_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||
credentials = value.get("credentials")
|
||||
return credentials if isinstance(credentials, Mapping) else {}
|
||||
|
||||
|
||||
def _profile_from_normalized_mapping(
|
||||
value: Mapping[str, Any],
|
||||
*,
|
||||
profile_id: str,
|
||||
provider: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
credential_mode: str,
|
||||
credentials: Mapping[str, Any],
|
||||
) -> ConnectorProfile:
|
||||
return ConnectorProfile(
|
||||
id=profile_id,
|
||||
label=_clean(value.get("label") or value.get("name")) or profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
|
||||
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
|
||||
enabled=_bool(value.get("enabled"), default=True),
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
|
||||
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
|
||||
credential_mode=credential_mode,
|
||||
username=_clean(value.get("username") or credentials.get("username")),
|
||||
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
|
||||
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
|
||||
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
|
||||
password_value=_clean(value.get("password") or credentials.get("password")),
|
||||
token_value=_clean(value.get("token") or credentials.get("token")),
|
||||
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
|
||||
capabilities=tuple(_string_list(value.get("capabilities"))),
|
||||
metadata=_public_metadata(value.get("metadata")),
|
||||
)
|
||||
|
||||
|
||||
def _raw_profiles_payload(settings: object | None) -> object:
|
||||
for key in _ENV_JSON_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return value
|
||||
for key in _ENV_FILE_KEYS:
|
||||
path = os.environ.get(key)
|
||||
if path:
|
||||
with open(path, encoding="utf-8") as handle:
|
||||
return handle.read()
|
||||
if settings is not None:
|
||||
value = getattr(settings, "files_connector_profiles_json", None)
|
||||
if value:
|
||||
return value
|
||||
value = getattr(settings, "files_connector_profiles", None)
|
||||
if value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _policy_sources_from_profile(value: Mapping[str, Any], profile: ConnectorProfile) -> list[ConnectorPolicySource]:
|
||||
raw_sources: list[Mapping[str, Any]] = []
|
||||
policy = value.get("policy")
|
||||
if isinstance(policy, Mapping):
|
||||
raw_sources.append({
|
||||
"scope_type": profile.scope_type,
|
||||
"scope_id": profile.scope_id,
|
||||
"label": profile.label,
|
||||
"policy": policy,
|
||||
})
|
||||
configured_sources = value.get("policy_sources") or value.get("connector_policy_sources")
|
||||
if configured_sources is not None:
|
||||
raw = connector_policy_sources_from_payload(configured_sources)
|
||||
raw_sources.extend(_policy_source_response(source) for source in raw)
|
||||
return connector_policy_sources_from_payload(raw_sources)
|
||||
|
||||
|
||||
def _policy_source_response(source: ConnectorPolicySource) -> dict[str, Any]:
|
||||
return {
|
||||
"scope_type": source.scope_type,
|
||||
"scope_id": source.scope_id,
|
||||
"label": source.label,
|
||||
"policy": dict(source.policy or {}),
|
||||
}
|
||||
|
||||
|
||||
def _public_metadata(value: object) -> Mapping[str, Any]:
|
||||
if not isinstance(value, Mapping):
|
||||
return {}
|
||||
return {
|
||||
str(key): item
|
||||
for key, item in value.items()
|
||||
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||
}
|
||||
|
||||
|
||||
def _response_metadata(value: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
str(key): item
|
||||
for key, item in value.items()
|
||||
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||
and not str(key).strip().casefold().endswith("_env")
|
||||
and str(key).strip().casefold() != "ca_bundle"
|
||||
}
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
values: Iterable[object] = value.split(",")
|
||||
elif isinstance(value, Iterable) and not isinstance(value, (bytes, bytearray, Mapping)):
|
||||
values = value
|
||||
else:
|
||||
values = (value,)
|
||||
return [str(item).strip() for item in values if str(item).strip()]
|
||||
|
||||
|
||||
def _bool(value: object, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return value.strip().casefold() not in {"0", "false", "no", "off"}
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,191 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from importlib.util import find_spec
|
||||
|
||||
|
||||
CONNECTOR_PROVIDER_CAPABILITY = "files.connector.providers"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorProviderDescriptor:
|
||||
provider: str
|
||||
label: str
|
||||
protocol: str
|
||||
implemented: bool
|
||||
browse_supported: bool
|
||||
import_supported: bool
|
||||
optional_dependency: str | None
|
||||
permission_model: str
|
||||
sync_strategy: str
|
||||
conflict_strategy: str
|
||||
preview_strategy: str
|
||||
audit_events: tuple[str, ...]
|
||||
notes: str | None = None
|
||||
|
||||
def to_response(self) -> dict[str, object]:
|
||||
return {
|
||||
"provider": self.provider,
|
||||
"label": self.label,
|
||||
"protocol": self.protocol,
|
||||
"implemented": self.implemented,
|
||||
"installed": self.installed,
|
||||
"browse_supported": self.browse_supported,
|
||||
"import_supported": self.import_supported,
|
||||
"write_supported": self.provider == "s3",
|
||||
"optional_dependency": self.optional_dependency,
|
||||
"permission_model": self.permission_model,
|
||||
"sync_strategy": self.sync_strategy,
|
||||
"conflict_strategy": self.conflict_strategy,
|
||||
"preview_strategy": self.preview_strategy,
|
||||
"audit_events": list(self.audit_events),
|
||||
"notes": self.notes,
|
||||
}
|
||||
|
||||
@property
|
||||
def installed(self) -> bool:
|
||||
if self.optional_dependency is None:
|
||||
return self.implemented
|
||||
return find_spec(self.optional_dependency) is not None
|
||||
|
||||
|
||||
def connector_provider_descriptors() -> tuple[ConnectorProviderDescriptor, ...]:
|
||||
freeze_model = "Imported or synced connector files become managed GovOPlaN files with frozen blob/version provenance."
|
||||
governed_permissions = "GovOPlaN profile scope and connector policy are enforced before browse/import/sync; remote ACLs are still evaluated by the upstream service."
|
||||
return (
|
||||
ConnectorProviderDescriptor(
|
||||
provider="seafile",
|
||||
label="Seafile",
|
||||
protocol="seafile-api",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Native account-token API for libraries, directories, file detail, and download-link import; WebDAV opt-in remains available.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="nextcloud",
|
||||
label="Nextcloud",
|
||||
protocol="webdav",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand WebDAV browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Uses the configured Nextcloud WebDAV endpoint, basic auth, or bearer token profile credentials.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="webdav",
|
||||
label="WebDAV",
|
||||
protocol="webdav",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand WebDAV browse/import/sync; sync updates the managed file version when source content changes and never mutates the remote source.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling: reject, overwrite, or rename.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Generic WebDAV provider for standards-compatible stores.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="smb",
|
||||
label="SMB",
|
||||
protocol="smb",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency="smbprotocol",
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand browse/import/sync over administrator-declared shares; sync updates managed versions and never mutates the remote share.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after download.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the share.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Initial sessions, reconnects, aliases, and DFS referral targets use the Files-owned pinned smbprotocol transport and the deployment-wide private-network policy.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="s3",
|
||||
label="S3-compatible object storage",
|
||||
protocol="s3-api",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency="boto3",
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand bucket/prefix browse and object import/sync; sync updates managed versions and never mutates the remote bucket.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after object download.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the bucket.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Every botocore connection pool uses the Files-owned pinned transport, including retries, redirects, endpoint discovery, and virtual-host aliases; outbound proxies and ambient credential discovery are disabled.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="sharepoint",
|
||||
label="SharePoint",
|
||||
protocol="microsoft-graph/sharepoint",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph or SharePoint credentials.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from SharePoint.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Provider/profile key is reserved; live browsing/import still needs Graph/SharePoint authentication and paging implementation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="onedrive",
|
||||
label="OneDrive",
|
||||
protocol="microsoft-graph/onedrive",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph credentials.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from OneDrive.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Provider/profile key is reserved; live browsing/import still needs Graph drive selection, OAuth/app registration, and paging implementation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="nfs",
|
||||
label="NFS",
|
||||
protocol="nfs",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned optional provider over administrator-mounted paths or a sidecar service.",
|
||||
conflict_strategy="Will use managed import conflict_strategy handling after download.",
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from the mount.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Kept optional because NFS availability is deployment and host-kernel dependent.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="local",
|
||||
label="Local filesystem",
|
||||
protocol="file",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model="Local connector access should be restricted to administrator-declared roots plus GovOPlaN profile and policy checks.",
|
||||
sync_strategy="Planned optional provider; managed storage local backend already exists separately.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import or sync.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Separate from the existing managed-file local storage backend.",
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,264 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import false, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileConnectorSpace
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, user_group_ids
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.connector_browse import normalize_connector_browse_path
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
SYNC_MODES = {"manual"}
|
||||
WRITABLE_PROVIDERS = {"s3"}
|
||||
|
||||
|
||||
def connector_space_owner_id(space: FileConnectorSpace) -> str:
|
||||
return space.owner_user_id if space.owner_type == "user" else space.owner_group_id # type: ignore[return-value]
|
||||
|
||||
|
||||
def create_connector_space(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
label: str,
|
||||
profile: ConnectorProfile,
|
||||
library_id: str | None = None,
|
||||
remote_path: str | None = None,
|
||||
sync_mode: str = "manual",
|
||||
read_only: bool = True,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
label = _normalize_label(label)
|
||||
sync_mode = _normalize_sync_mode(sync_mode)
|
||||
validate_connector_space_write_mode(profile, read_only=read_only)
|
||||
remote_path = normalize_connector_browse_path(remote_path)
|
||||
library_id = _clean_optional(library_id)
|
||||
|
||||
existing = _owned_query(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id).filter(
|
||||
FileConnectorSpace.label == label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if existing is not None:
|
||||
raise FileStorageError(f"Connector space already exists: {label}")
|
||||
|
||||
space = FileConnectorSpace(
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_user_id=owner_id if owner_type == "user" else None,
|
||||
owner_group_id=owner_id if owner_type == "group" else None,
|
||||
label=label,
|
||||
connector_profile_id=profile.id,
|
||||
provider=profile.provider,
|
||||
library_id=library_id,
|
||||
remote_path=remote_path,
|
||||
sync_mode=sync_mode,
|
||||
read_only=read_only,
|
||||
is_active=True,
|
||||
created_by_user_id=user_id,
|
||||
metadata_=dict(metadata or {}),
|
||||
)
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def list_connector_spaces_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
include_inactive: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileConnectorSpace]:
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == tenant_id,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
)
|
||||
if not include_inactive:
|
||||
query = query.filter(FileConnectorSpace.is_active.is_(True))
|
||||
|
||||
if owner_type:
|
||||
if not owner_id:
|
||||
raise FileStorageError("owner_id is required when owner_type is set")
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
query = _owner_filter(query, owner_type, owner_id)
|
||||
elif not is_admin:
|
||||
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||
query = query.filter(
|
||||
or_(
|
||||
FileConnectorSpace.owner_user_id == user_id,
|
||||
FileConnectorSpace.owner_group_id.in_(group_ids) if group_ids else false(),
|
||||
)
|
||||
)
|
||||
|
||||
return query.order_by(FileConnectorSpace.owner_type.asc(), FileConnectorSpace.label.asc()).all()
|
||||
|
||||
|
||||
def get_connector_space_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
space_id: str,
|
||||
include_inactive: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.id == space_id,
|
||||
FileConnectorSpace.tenant_id == tenant_id,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
)
|
||||
if not include_inactive:
|
||||
query = query.filter(FileConnectorSpace.is_active.is_(True))
|
||||
space = query.one_or_none()
|
||||
if space is None:
|
||||
raise FileStorageError("Connector space not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return space
|
||||
|
||||
|
||||
def update_connector_space(
|
||||
session: Session,
|
||||
space: FileConnectorSpace,
|
||||
*,
|
||||
user_id: str,
|
||||
label: str | None = None,
|
||||
library_id: str | None = None,
|
||||
remote_path: str | None = None,
|
||||
sync_mode: str | None = None,
|
||||
read_only: bool | None = None,
|
||||
is_active: bool | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=space.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if label is not None:
|
||||
new_label = _normalize_label(label)
|
||||
existing = _owned_query(
|
||||
session,
|
||||
tenant_id=space.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
).filter(
|
||||
FileConnectorSpace.id != space.id,
|
||||
FileConnectorSpace.label == new_label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if existing is not None:
|
||||
raise FileStorageError(f"Connector space already exists: {new_label}")
|
||||
space.label = new_label
|
||||
if library_id is not None:
|
||||
space.library_id = _clean_optional(library_id)
|
||||
if remote_path is not None:
|
||||
space.remote_path = normalize_connector_browse_path(remote_path)
|
||||
if sync_mode is not None:
|
||||
space.sync_mode = _normalize_sync_mode(sync_mode)
|
||||
if read_only is not None:
|
||||
space.read_only = bool(read_only)
|
||||
if is_active is not None:
|
||||
space.is_active = bool(is_active)
|
||||
if metadata is not None:
|
||||
space.metadata_ = dict(metadata)
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def validate_connector_space_write_mode(
|
||||
profile: ConnectorProfile, *, read_only: bool
|
||||
) -> None:
|
||||
if read_only:
|
||||
return
|
||||
if profile.provider not in WRITABLE_PROVIDERS or "write" not in profile.capabilities:
|
||||
raise FileStorageError(
|
||||
"Two-way mode requires an S3 connection with the write capability enabled"
|
||||
)
|
||||
|
||||
|
||||
def soft_delete_connector_space(
|
||||
session: Session,
|
||||
space: FileConnectorSpace,
|
||||
*,
|
||||
user_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> FileConnectorSpace:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=space.tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
space.deleted_at = utcnow()
|
||||
space.is_active = False
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def _owned_query(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == tenant_id,
|
||||
FileConnectorSpace.owner_type == owner_type,
|
||||
)
|
||||
return _owner_filter(query, owner_type, owner_id)
|
||||
|
||||
|
||||
def _owner_filter(query, owner_type: str, owner_id: str):
|
||||
if owner_type == "user":
|
||||
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def _normalize_label(value: str) -> str:
|
||||
label = value.strip()
|
||||
if not label:
|
||||
raise FileStorageError("Connector space label is required")
|
||||
if len(label) > 255:
|
||||
raise FileStorageError("Connector space label is too long")
|
||||
return label
|
||||
|
||||
|
||||
def _normalize_sync_mode(value: str) -> str:
|
||||
mode = value.strip().casefold()
|
||||
if mode not in SYNC_MODES:
|
||||
raise FileStorageError(f"Unsupported connector space sync mode: {value}")
|
||||
return mode
|
||||
|
||||
|
||||
def _clean_optional(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
@@ -0,0 +1,213 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterable
|
||||
from dataclasses import replace
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
select_database_connector_profiles,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import (
|
||||
ConnectorProfile,
|
||||
connector_profiles_from_settings,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import (
|
||||
ConnectorAccessRequest,
|
||||
connector_policy_decision,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy_store import (
|
||||
effective_connector_policy_sources,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_providers import (
|
||||
connector_provider_descriptors,
|
||||
)
|
||||
|
||||
|
||||
CampaignVisibility = Callable[[str], bool]
|
||||
|
||||
|
||||
def visible_connector_profiles_for_actor(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: Iterable[str],
|
||||
settings: object | None,
|
||||
provider: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
campaign_visible: CampaignVisibility | None = None,
|
||||
include_disabled: bool = False,
|
||||
include_admin_scopes: bool = False,
|
||||
include_effective_policy: bool = True,
|
||||
) -> list[ConnectorProfile]:
|
||||
"""Return the same actor-filtered connector set used by API and docs surfaces."""
|
||||
|
||||
provider_norm = provider.strip().casefold() if provider else None
|
||||
actor_group_ids = {str(group_id) for group_id in group_ids if str(group_id)}
|
||||
database_profiles, database_profile_ids = select_database_connector_profiles(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
include_disabled=include_disabled,
|
||||
row_visible=lambda row: (
|
||||
include_admin_scopes
|
||||
and row.scope_type in {"user", "group", "campaign"}
|
||||
) or _scope_visible_to_actor(
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=actor_group_ids,
|
||||
campaign_id=campaign_id,
|
||||
campaign_visible=campaign_visible,
|
||||
),
|
||||
)
|
||||
configured_profiles = connector_profiles_from_settings(settings)
|
||||
profiles_by_id: dict[str, ConnectorProfile] = {}
|
||||
for profile in database_profiles:
|
||||
if profile.id not in profiles_by_id:
|
||||
profiles_by_id[profile.id] = profile
|
||||
for profile in configured_profiles:
|
||||
if profile.id not in database_profile_ids and profile.id not in profiles_by_id:
|
||||
profiles_by_id[profile.id] = profile
|
||||
|
||||
visible: list[ConnectorProfile] = []
|
||||
for profile in profiles_by_id.values():
|
||||
if not (profile.enabled or include_disabled):
|
||||
continue
|
||||
if provider_norm is not None and profile.provider != provider_norm:
|
||||
continue
|
||||
admin_scope_visible = include_admin_scopes and profile.scope_type in {
|
||||
"user",
|
||||
"group",
|
||||
"campaign",
|
||||
}
|
||||
if not admin_scope_visible and not _profile_visible_to_actor(
|
||||
profile,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=actor_group_ids,
|
||||
campaign_id=campaign_id,
|
||||
campaign_visible=campaign_visible,
|
||||
):
|
||||
continue
|
||||
visible.append(
|
||||
_with_effective_connector_policy(
|
||||
session, tenant_id=tenant_id, profile=profile
|
||||
)
|
||||
if include_effective_policy
|
||||
else profile
|
||||
)
|
||||
return visible
|
||||
|
||||
|
||||
def connector_profile_usable_for_import(profile: ConnectorProfile) -> bool:
|
||||
"""Whether a visible profile can safely offer the current live browse/import task."""
|
||||
|
||||
effective_endpoint_url = connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
)
|
||||
if (
|
||||
not profile.enabled
|
||||
or not effective_endpoint_url
|
||||
or not profile.credentials_configured
|
||||
or bool(profile.secret_ref)
|
||||
):
|
||||
return False
|
||||
descriptor = next(
|
||||
(
|
||||
item
|
||||
for item in connector_provider_descriptors()
|
||||
if item.provider == profile.provider
|
||||
),
|
||||
None,
|
||||
)
|
||||
if descriptor is None:
|
||||
return False
|
||||
if not (
|
||||
descriptor.implemented
|
||||
and descriptor.installed
|
||||
and descriptor.browse_supported
|
||||
and descriptor.import_supported
|
||||
):
|
||||
return False
|
||||
# Prove that the initial root browse performed by the current Files UI is
|
||||
# policy-allowed. A later selected remote path/item is checked again.
|
||||
return connector_policy_decision(
|
||||
ConnectorAccessRequest(
|
||||
connector_id=profile.id,
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path="",
|
||||
external_url=effective_endpoint_url,
|
||||
operation="import",
|
||||
),
|
||||
profile.policy_sources,
|
||||
).allowed
|
||||
|
||||
|
||||
def _profile_visible_to_actor(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: set[str],
|
||||
campaign_id: str | None,
|
||||
campaign_visible: CampaignVisibility | None,
|
||||
) -> bool:
|
||||
return _scope_visible_to_actor(
|
||||
scope_type=profile.scope_type,
|
||||
scope_id=profile.scope_id,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
group_ids=group_ids,
|
||||
campaign_id=campaign_id,
|
||||
campaign_visible=campaign_visible,
|
||||
)
|
||||
|
||||
|
||||
def _scope_visible_to_actor(
|
||||
*,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
group_ids: set[str],
|
||||
campaign_id: str | None,
|
||||
campaign_visible: CampaignVisibility | None,
|
||||
) -> bool:
|
||||
if scope_type == "system":
|
||||
return True
|
||||
if scope_type == "tenant":
|
||||
return scope_id == tenant_id
|
||||
if scope_type == "user":
|
||||
return scope_id == user_id
|
||||
if scope_type == "group":
|
||||
return bool(scope_id and scope_id in group_ids)
|
||||
if scope_type != "campaign" or not scope_id:
|
||||
return False
|
||||
if campaign_id and scope_id != campaign_id:
|
||||
return False
|
||||
return bool(campaign_visible and campaign_visible(scope_id))
|
||||
|
||||
|
||||
def _with_effective_connector_policy(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
profile: ConnectorProfile,
|
||||
) -> ConnectorProfile:
|
||||
sources = effective_connector_policy_sources(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
scope_type=profile.scope_type,
|
||||
scope_id=profile.scope_id,
|
||||
)
|
||||
if not sources:
|
||||
return profile
|
||||
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))
|
||||
@@ -0,0 +1,438 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
from typing import Any, Mapping
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
_clean,
|
||||
_s3_bucket,
|
||||
_s3_client,
|
||||
_s3_object_key,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorWriteResult:
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool
|
||||
provider: str
|
||||
remote_path: str
|
||||
revision: str | None
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _HeadProbe:
|
||||
observed: Mapping[str, Any] | None
|
||||
verified: bool
|
||||
exception_type: str | None = None
|
||||
|
||||
|
||||
def write_connector_file(
|
||||
profile: ConnectorProfile,
|
||||
*,
|
||||
tenant_id: str,
|
||||
library_id: str | None,
|
||||
remote_path: str,
|
||||
data: bytes,
|
||||
content_type: str | None,
|
||||
expected_revision: str | None,
|
||||
idempotency_key: str,
|
||||
) -> ConnectorWriteResult:
|
||||
if profile.provider != "s3" or "write" not in profile.capabilities:
|
||||
raise FileStorageError(
|
||||
"Remote writes require an S3 connection with the write capability enabled"
|
||||
)
|
||||
try:
|
||||
normalized_path = normalize_connector_browse_path(remote_path)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if not normalized_path:
|
||||
raise FileStorageError("Remote write requires an object path")
|
||||
try:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
key = _s3_object_key(profile, normalized_path)
|
||||
except ConnectorBrowseError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if not bucket or not key:
|
||||
raise FileStorageError("S3 remote write requires a bucket and object path")
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
target_digest = hashlib.sha256(f"{bucket}:{key}".encode("utf-8")).hexdigest()
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"connector_profile_id": profile.id,
|
||||
"provider": profile.provider,
|
||||
"remote_target_sha256": target_digest,
|
||||
"content_sha256": checksum,
|
||||
"size_bytes": len(data),
|
||||
"expected_revision": expected_revision,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="connector-s3-write",
|
||||
idempotency_key=f"files-connector-write:{idempotency_key}",
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the connector space is explicitly configured for two-way writes",
|
||||
"the connection and inherited policy allow the exact remote path",
|
||||
"a conditional create or expected remote revision prevents blind overwrite",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"inspect provider metadata for the request and content digests",
|
||||
"resolve success only for an exact digest match or retry after verified absence",
|
||||
),
|
||||
verification_steps=(
|
||||
"read S3 object metadata after the write",
|
||||
"match the recorded request and content digests without downloading content",
|
||||
),
|
||||
),
|
||||
precondition_evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"content_sha256": checksum,
|
||||
"size_bytes": len(data),
|
||||
"conditional_write": True,
|
||||
},
|
||||
lease_resource_key=f"files:connector:{tenant_id}:{profile.id}:{target_digest[:40]}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_connector_object",
|
||||
resource_id=target_digest,
|
||||
metadata={
|
||||
"resources": ["postgresql", "s3-connector"],
|
||||
"provider": "s3",
|
||||
"connector_profile_id": profile.id,
|
||||
},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"This remote object is owned by another write or unresolved recovery operation"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; the remote object was not changed"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=started.status,
|
||||
replayed=True,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=None,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
operation = started.operation
|
||||
try:
|
||||
client = _s3_client(profile)
|
||||
except (ConnectorBrowseError, ConnectorBrowseUnsupported) as exc:
|
||||
operation.reject(
|
||||
summary="The S3 client was unavailable before the remote effect",
|
||||
evidence=_verified(
|
||||
{"external_effect_started": False}, exception_type=type(exc).__name__
|
||||
),
|
||||
)
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
try:
|
||||
try:
|
||||
before = _head_object(client, bucket=bucket, key=key)
|
||||
except Exception as exc:
|
||||
operation.reject(
|
||||
summary="Remote preconditions could not be inspected before the effect",
|
||||
evidence=_verified(
|
||||
{"external_effect_started": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise FileStorageError("S3 remote revision lookup failed") from exc
|
||||
expected = _normalize_revision(expected_revision)
|
||||
if before is not None and expected is None:
|
||||
operation.reject(
|
||||
summary="A blind remote overwrite was rejected",
|
||||
evidence=_verified(
|
||||
{"remote_object_exists": True, "expected_revision_supplied": False}
|
||||
),
|
||||
)
|
||||
raise FileStorageError(
|
||||
"The remote object already exists; reload it and supply its expected revision"
|
||||
)
|
||||
if expected is not None and (
|
||||
before is None or expected not in _observed_revisions(before)
|
||||
):
|
||||
operation.reject(
|
||||
summary="The expected remote revision did not match",
|
||||
evidence=_verified(
|
||||
{
|
||||
"remote_object_exists": before is not None,
|
||||
"expected_revision_matches": False,
|
||||
}
|
||||
),
|
||||
)
|
||||
raise FileStorageError("The remote object changed; reload before writing")
|
||||
params: dict[str, object] = {
|
||||
"Bucket": bucket,
|
||||
"Key": key,
|
||||
"Body": data,
|
||||
"Metadata": {
|
||||
"govoplan-sha256": checksum,
|
||||
"govoplan-request-id": started.operation_id,
|
||||
},
|
||||
}
|
||||
if content_type:
|
||||
params["ContentType"] = content_type
|
||||
if before is None:
|
||||
params["IfNoneMatch"] = "*"
|
||||
else:
|
||||
params["IfMatch"] = str(before.get("ETag") or expected_revision or "")
|
||||
try:
|
||||
client.put_object(**params)
|
||||
except Exception as exc:
|
||||
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||
if probe.verified and _matches_effect(
|
||||
probe.observed,
|
||||
checksum=checksum,
|
||||
operation_id=started.operation_id,
|
||||
):
|
||||
operation.succeed(
|
||||
evidence=_success_evidence(
|
||||
probe.observed,
|
||||
checksum=checksum,
|
||||
operation_id=started.operation_id,
|
||||
)
|
||||
)
|
||||
return _result(
|
||||
started.operation_id,
|
||||
normalized_path,
|
||||
checksum,
|
||||
len(data),
|
||||
probe.observed,
|
||||
)
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The S3 write outcome requires reconciliation",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
"exception_type": type(exc).__name__,
|
||||
},
|
||||
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=_revision(probe.observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
probe = _probe_head_object(client, bucket=bucket, key=key)
|
||||
if not probe.verified:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The S3 write returned but provider evidence could not be queried",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
},
|
||||
failure_summary="Inspect the remote object metadata before retrying this path",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=None,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
observed = probe.observed
|
||||
if not _matches_effect(
|
||||
observed, checksum=checksum, operation_id=started.operation_id
|
||||
):
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="The S3 write returned but exact provider evidence did not match",
|
||||
evidence={
|
||||
"remote_target_sha256": target_digest,
|
||||
"observed": _public_observation(probe),
|
||||
},
|
||||
failure_summary="Reconcile the remote object before another write",
|
||||
)
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=normalized_path,
|
||||
revision=_revision(observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=len(data),
|
||||
)
|
||||
operation.succeed(
|
||||
evidence=_success_evidence(
|
||||
observed, checksum=checksum, operation_id=started.operation_id
|
||||
)
|
||||
)
|
||||
return _result(
|
||||
started.operation_id,
|
||||
normalized_path,
|
||||
checksum,
|
||||
len(data),
|
||||
observed,
|
||||
)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
def _head_object(client: Any, *, bucket: str, key: str) -> Mapping[str, Any] | None:
|
||||
try:
|
||||
response = client.head_object(Bucket=bucket, Key=key)
|
||||
except Exception as exc:
|
||||
if _is_not_found(exc):
|
||||
return None
|
||||
raise
|
||||
if not isinstance(response, Mapping):
|
||||
raise FileStorageError("S3 connector returned invalid object metadata")
|
||||
return response
|
||||
|
||||
|
||||
def _probe_head_object(client: Any, *, bucket: str, key: str) -> _HeadProbe:
|
||||
try:
|
||||
return _HeadProbe(
|
||||
observed=_head_object(client, bucket=bucket, key=key), verified=True
|
||||
)
|
||||
except Exception as exc:
|
||||
return _HeadProbe(
|
||||
observed=None,
|
||||
verified=False,
|
||||
exception_type=type(exc).__name__,
|
||||
)
|
||||
|
||||
|
||||
def _is_not_found(exc: Exception) -> bool:
|
||||
response = getattr(exc, "response", None)
|
||||
if not isinstance(response, Mapping):
|
||||
return False
|
||||
error = response.get("Error")
|
||||
code = error.get("Code") if isinstance(error, Mapping) else None
|
||||
status = response.get("ResponseMetadata")
|
||||
http_status = status.get("HTTPStatusCode") if isinstance(status, Mapping) else None
|
||||
return str(code).casefold() in {"404", "nosuchkey", "notfound"} or http_status == 404
|
||||
|
||||
|
||||
def _metadata(observed: Mapping[str, Any] | None) -> Mapping[str, Any]:
|
||||
value = observed.get("Metadata") if observed else None
|
||||
return value if isinstance(value, Mapping) else {}
|
||||
|
||||
|
||||
def _matches_effect(
|
||||
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||
) -> bool:
|
||||
metadata = _metadata(observed)
|
||||
return bool(
|
||||
observed is not None
|
||||
and _clean(metadata.get("govoplan-sha256")) == checksum
|
||||
and _clean(metadata.get("govoplan-request-id")) == operation_id
|
||||
)
|
||||
|
||||
|
||||
def _revision(observed: Mapping[str, Any] | None) -> str | None:
|
||||
if observed is None:
|
||||
return None
|
||||
return _clean(observed.get("VersionId") or observed.get("ETag"))
|
||||
|
||||
|
||||
def _normalize_revision(value: str | None) -> str | None:
|
||||
cleaned = _clean(value)
|
||||
return cleaned.strip('"') if cleaned else None
|
||||
|
||||
|
||||
def _observed_revisions(observed: Mapping[str, Any]) -> set[str]:
|
||||
return {
|
||||
normalized
|
||||
for value in (observed.get("VersionId"), observed.get("ETag"))
|
||||
if (normalized := _normalize_revision(_clean(value))) is not None
|
||||
}
|
||||
|
||||
|
||||
def _success_evidence(
|
||||
observed: Mapping[str, Any] | None, *, checksum: str, operation_id: str
|
||||
) -> dict[str, object]:
|
||||
return _verified(
|
||||
{
|
||||
"remote_object_present": observed is not None,
|
||||
"content_digest_matches": _clean(_metadata(observed).get("govoplan-sha256")) == checksum,
|
||||
"request_marker_matches": _clean(_metadata(observed).get("govoplan-request-id")) == operation_id,
|
||||
},
|
||||
revision=_revision(observed),
|
||||
content_sha256=checksum,
|
||||
)
|
||||
|
||||
|
||||
def _public_observation(probe: _HeadProbe) -> dict[str, object]:
|
||||
observed = probe.observed
|
||||
return {
|
||||
"probe_verified": probe.verified,
|
||||
"probe_exception_type": probe.exception_type,
|
||||
"present": observed is not None,
|
||||
"revision": _revision(observed),
|
||||
"has_content_digest": bool(_clean(_metadata(observed).get("govoplan-sha256"))),
|
||||
"has_request_marker": bool(_clean(_metadata(observed).get("govoplan-request-id"))),
|
||||
}
|
||||
|
||||
|
||||
def _verified(checks: dict[str, object], **details: object) -> dict[str, object]:
|
||||
return {"verified": True, "checks": checks, **details}
|
||||
|
||||
|
||||
def _result(
|
||||
operation_id: str,
|
||||
remote_path: str,
|
||||
checksum: str,
|
||||
size_bytes: int,
|
||||
observed: Mapping[str, Any] | None,
|
||||
) -> ConnectorWriteResult:
|
||||
return ConnectorWriteResult(
|
||||
recovery_operation_id=operation_id,
|
||||
status=RecoveryStatus.SUCCEEDED.value,
|
||||
replayed=False,
|
||||
provider="s3",
|
||||
remote_path=remote_path,
|
||||
revision=_revision(observed),
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=size_bytes,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["ConnectorWriteResult", "write_connector_file"]
|
||||
@@ -0,0 +1,93 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from govoplan_core.core.encryption import (
|
||||
ContentProtectionRequest,
|
||||
ContentUnprotectionRequest,
|
||||
ProtectedContent,
|
||||
encryption_content_cipher,
|
||||
)
|
||||
from govoplan_files.backend.runtime import get_registry
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
FILES_PROTECTION_PROFILE = "files-server-envelope-v1"
|
||||
|
||||
|
||||
def protect_blob_content(
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
blob_id: str,
|
||||
vault_id: str,
|
||||
ciphertext_ref: str,
|
||||
plaintext: bytes,
|
||||
actor_id: str,
|
||||
content_type: str | None,
|
||||
) -> ProtectedContent:
|
||||
capability = encryption_content_cipher(get_registry())
|
||||
if capability is None:
|
||||
raise FileStorageError(
|
||||
"File encryption was requested, but the Encryption module is unavailable."
|
||||
)
|
||||
try:
|
||||
return capability.protect_content(
|
||||
session,
|
||||
request=ContentProtectionRequest(
|
||||
tenant_id=tenant_id,
|
||||
owner_module="files",
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
profile_id=FILES_PROTECTION_PROFILE,
|
||||
vault_id=vault_id,
|
||||
ciphertext_ref=ciphertext_ref,
|
||||
plaintext=plaintext,
|
||||
policy_decision_ref="files:explicit-vault-selection:v1",
|
||||
idempotency_key=f"file-blob:{blob_id}:content:v1",
|
||||
actor_id=actor_id,
|
||||
metadata={
|
||||
"content_type": content_type or "application/octet-stream",
|
||||
},
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
raise FileStorageError(
|
||||
"Managed file content could not be protected by the configured vault."
|
||||
) from exc
|
||||
|
||||
|
||||
def unprotect_blob_content(
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
blob_id: str,
|
||||
envelope_id: str,
|
||||
ciphertext: bytes,
|
||||
) -> bytes:
|
||||
capability = encryption_content_cipher(get_registry())
|
||||
if capability is None:
|
||||
raise FileStorageError(
|
||||
"This file is encrypted and cannot be read while Encryption is unavailable."
|
||||
)
|
||||
try:
|
||||
return capability.unprotect_content(
|
||||
session,
|
||||
request=ContentUnprotectionRequest(
|
||||
tenant_id=tenant_id,
|
||||
owner_module="files",
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
envelope_id=envelope_id,
|
||||
ciphertext=ciphertext,
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
raise FileStorageError(
|
||||
"Managed file content could not be opened with its protection envelope."
|
||||
) from exc
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FILES_PROTECTION_PROFILE",
|
||||
"protect_blob_content",
|
||||
"unprotect_blob_content",
|
||||
]
|
||||
@@ -2,30 +2,47 @@ from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from datetime import datetime
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any, Iterable
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy import and_, exists, func, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.db.models import Group, Tenant, User
|
||||
from govoplan_core.core.optional import reraise_unless_missing_package
|
||||
from govoplan_core.core.campaigns import CAPABILITY_CAMPAIGNS_ACCESS, CampaignAccessProvider
|
||||
from govoplan_files.backend.db.models import CampaignAttachmentUse, FileAsset, FileBlob, FileShare, FileVersion
|
||||
from govoplan_files.backend.runtime import settings
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, user_group_ids
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
||||
from govoplan_files.backend.runtime import get_registry, settings
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access, ensure_share_target_exists, user_group_ids
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError, UploadedStoredFile, utcnow
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path, safe_storage_component
|
||||
from govoplan_files.backend.storage.paths import filename_from_path, join_folder_filename, normalize_folder, normalize_logical_path
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata
|
||||
from govoplan_files.backend.storage.recovery import begin_blob_write_recovery
|
||||
from govoplan_files.backend.storage.integrity import (
|
||||
QUARANTINED_BLOB_STATUSES,
|
||||
read_verified_blob_bytes,
|
||||
)
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
def _campaign_model():
|
||||
try:
|
||||
from govoplan_campaign.backend.db.models import Campaign
|
||||
except ModuleNotFoundError as exc:
|
||||
reraise_unless_missing_package(exc, "govoplan_campaign")
|
||||
raise FileStorageError("Campaign module is not installed") from exc
|
||||
return Campaign
|
||||
def _campaign_access_provider() -> CampaignAccessProvider:
|
||||
registry = get_registry()
|
||||
if registry is None or not hasattr(registry, "has_capability") or not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
|
||||
raise FileStorageError("Campaign module is not installed")
|
||||
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
|
||||
if not isinstance(capability, CampaignAccessProvider):
|
||||
raise FileStorageError("Campaign access capability is invalid")
|
||||
return capability
|
||||
|
||||
|
||||
def _ensure_campaign_exists(session: Session, *, tenant_id: str, campaign_id: str) -> None:
|
||||
if not _campaign_access_provider().campaign_exists(session, tenant_id=tenant_id, campaign_id=campaign_id):
|
||||
raise FileStorageError("Campaign not found")
|
||||
|
||||
|
||||
def _asset_query_for_owner(session: Session, *, tenant_id: str, owner_type: str, owner_id: str):
|
||||
@@ -45,8 +62,9 @@ def _storage_backend_name() -> str:
|
||||
return settings.file_storage_backend.lower().strip()
|
||||
|
||||
|
||||
def _storage_key(*, tenant_id: str, checksum: str, filename: str) -> str:
|
||||
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}-{safe_storage_component(filename)}"
|
||||
def _storage_key(*, tenant_id: str, checksum: str) -> str:
|
||||
# Object locators remain opaque so recovery evidence never persists names.
|
||||
return f"tenants/{tenant_id}/files/{checksum[:2]}/{uuid4().hex}.blob"
|
||||
|
||||
|
||||
def _get_or_create_blob(
|
||||
@@ -56,40 +74,144 @@ def _get_or_create_blob(
|
||||
data: bytes,
|
||||
filename: str,
|
||||
content_type: str | None,
|
||||
actor_id: str,
|
||||
encryption_vault_id: str | None = None,
|
||||
) -> FileBlob:
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
size = len(data)
|
||||
vault_id = str(encryption_vault_id or "").strip() or None
|
||||
protection_discriminator = f"vault:{vault_id}" if vault_id else "plaintext"
|
||||
blob = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size)
|
||||
.filter(FileBlob.tenant_id == tenant_id, FileBlob.checksum_sha256 == checksum, FileBlob.size_bytes == size, FileBlob.protection_discriminator == protection_discriminator)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if blob:
|
||||
backend = get_storage_backend()
|
||||
if not backend.exists(blob.storage_key):
|
||||
repair_required = (
|
||||
blob.integrity_status in QUARANTINED_BLOB_STATUSES
|
||||
or blob.quarantined_at is not None
|
||||
)
|
||||
try:
|
||||
backend.stat(blob.storage_key)
|
||||
except StorageObjectMissing:
|
||||
repair_required = True
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
if repair_required:
|
||||
repair_token = hashlib.sha256(
|
||||
repr(
|
||||
(
|
||||
blob.integrity_status,
|
||||
blob.integrity_checked_at,
|
||||
blob.quarantined_at,
|
||||
blob.storage_checksum_sha256,
|
||||
)
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
recovery = begin_blob_write_recovery(
|
||||
session,
|
||||
backend=backend,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob.id,
|
||||
storage_key=blob.storage_key,
|
||||
semantic_checksum_sha256=checksum,
|
||||
semantic_size_bytes=size,
|
||||
protection_discriminator=protection_discriminator,
|
||||
created_new=False,
|
||||
repair_token=repair_token,
|
||||
)
|
||||
stored_data = data
|
||||
expected_envelope_id = blob.encryption_envelope_id
|
||||
if vault_id:
|
||||
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||
|
||||
protected = protect_blob_content(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob.id,
|
||||
vault_id=vault_id,
|
||||
ciphertext_ref=blob.storage_key,
|
||||
plaintext=data,
|
||||
actor_id=actor_id,
|
||||
content_type=content_type,
|
||||
)
|
||||
if blob.encryption_envelope_id not in {None, protected.envelope.envelope_id}:
|
||||
raise FileStorageError("The existing encrypted blob has another protection envelope.")
|
||||
stored_data = protected.ciphertext
|
||||
blob.encryption_envelope_id = protected.envelope.envelope_id
|
||||
expected_envelope_id = protected.envelope.envelope_id
|
||||
blob.storage_checksum_sha256 = hashlib.sha256(stored_data).hexdigest()
|
||||
blob.storage_size_bytes = len(stored_data)
|
||||
recovery.prepare_stored_bytes(
|
||||
stored_data,
|
||||
envelope_id=expected_envelope_id,
|
||||
)
|
||||
try:
|
||||
backend.put_bytes(blob.storage_key, data, content_type=content_type)
|
||||
backend.put_bytes(blob.storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
blob.integrity_status = "verified"
|
||||
blob.integrity_checked_at = utcnow()
|
||||
blob.integrity_failure = None
|
||||
blob.quarantined_at = None
|
||||
blob.ref_count += 1
|
||||
session.add(blob)
|
||||
return blob
|
||||
|
||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum, filename=filename)
|
||||
blob_id = str(uuid4())
|
||||
storage_key = _storage_key(tenant_id=tenant_id, checksum=checksum)
|
||||
backend = get_storage_backend()
|
||||
recovery = begin_blob_write_recovery(
|
||||
session,
|
||||
backend=backend,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob_id,
|
||||
storage_key=storage_key,
|
||||
semantic_checksum_sha256=checksum,
|
||||
semantic_size_bytes=size,
|
||||
protection_discriminator=protection_discriminator,
|
||||
created_new=True,
|
||||
)
|
||||
stored_data = data
|
||||
envelope_id = None
|
||||
if vault_id:
|
||||
from govoplan_files.backend.storage.content_protection import protect_blob_content
|
||||
|
||||
protected = protect_blob_content(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob_id,
|
||||
vault_id=vault_id,
|
||||
ciphertext_ref=storage_key,
|
||||
plaintext=data,
|
||||
actor_id=actor_id,
|
||||
content_type=content_type,
|
||||
)
|
||||
stored_data = protected.ciphertext
|
||||
envelope_id = protected.envelope.envelope_id
|
||||
recovery.prepare_stored_bytes(stored_data, envelope_id=envelope_id)
|
||||
try:
|
||||
backend.put_bytes(storage_key, data, content_type=content_type)
|
||||
backend.put_bytes(storage_key, stored_data, content_type="application/octet-stream" if vault_id else content_type)
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
blob = FileBlob(
|
||||
id=blob_id,
|
||||
tenant_id=tenant_id,
|
||||
storage_backend=_storage_backend_name(),
|
||||
storage_bucket=_storage_bucket_name(),
|
||||
storage_key=storage_key,
|
||||
checksum_sha256=checksum,
|
||||
size_bytes=size,
|
||||
protection_discriminator=protection_discriminator,
|
||||
encryption_envelope_id=envelope_id,
|
||||
storage_checksum_sha256=hashlib.sha256(stored_data).hexdigest() if vault_id else None,
|
||||
storage_size_bytes=len(stored_data) if vault_id else None,
|
||||
content_type=content_type,
|
||||
ref_count=1,
|
||||
integrity_status="verified",
|
||||
integrity_checked_at=utcnow(),
|
||||
)
|
||||
session.add(blob)
|
||||
session.flush()
|
||||
@@ -114,6 +236,7 @@ def create_file_asset(
|
||||
conflict_strategy: str = "reject",
|
||||
conflict_resolutions: Iterable[FileConflictResolution] | None = None,
|
||||
is_admin: bool = False,
|
||||
encryption_vault_id: str | None = None,
|
||||
) -> UploadedStoredFile:
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
@@ -141,7 +264,7 @@ def create_file_asset(
|
||||
elif action == "rename":
|
||||
logical_path = _next_available_logical_path(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, desired_path=logical_path)
|
||||
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type)
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=encryption_vault_id)
|
||||
asset = FileAsset(
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
@@ -176,6 +299,150 @@ def create_file_asset(
|
||||
return UploadedStoredFile(asset=asset, version=version, blob=blob)
|
||||
|
||||
|
||||
def sync_file_asset_from_source(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
filename: str,
|
||||
data: bytes,
|
||||
metadata: dict[str, Any],
|
||||
folder: str | None = None,
|
||||
display_path: str | None = None,
|
||||
content_type: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
conflict_strategy: str = "rename",
|
||||
is_admin: bool = False,
|
||||
) -> tuple[UploadedStoredFile, str, str | None]:
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
provenance = source_provenance_from_metadata(metadata)
|
||||
if not provenance:
|
||||
raise FileStorageError("Connector sync requires source provenance")
|
||||
existing = find_asset_by_source(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
source_provenance=provenance,
|
||||
)
|
||||
if existing is None:
|
||||
stored = create_file_asset(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
folder=folder,
|
||||
display_path=display_path,
|
||||
content_type=content_type,
|
||||
metadata=metadata,
|
||||
campaign_id=campaign_id,
|
||||
conflict_strategy=conflict_strategy,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return stored, "created", None
|
||||
|
||||
previous_version_id = existing.current_version_id
|
||||
stored, action = update_file_asset_content(
|
||||
session,
|
||||
existing,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
metadata=metadata,
|
||||
)
|
||||
if campaign_id:
|
||||
share_file(session, tenant_id=tenant_id, asset=existing, target_type="campaign", target_id=campaign_id, permission="read", user_id=user_id)
|
||||
return stored, action, previous_version_id
|
||||
|
||||
|
||||
def find_asset_by_source(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
source_provenance: dict[str, Any],
|
||||
) -> FileAsset | None:
|
||||
wanted = _source_identity(source_provenance)
|
||||
if wanted is None:
|
||||
return None
|
||||
assets = (
|
||||
_asset_query_for_owner(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id)
|
||||
.filter(FileAsset.deleted_at.is_(None))
|
||||
.order_by(FileAsset.updated_at.desc())
|
||||
.all()
|
||||
)
|
||||
for asset in assets:
|
||||
if _source_identity(source_provenance_from_metadata(asset.metadata_ or {})) == wanted:
|
||||
return asset
|
||||
return None
|
||||
|
||||
|
||||
def update_file_asset_content(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
filename: str,
|
||||
data: bytes,
|
||||
content_type: str | None,
|
||||
metadata: dict[str, Any],
|
||||
encryption_vault_id: str | None = None,
|
||||
) -> tuple[UploadedStoredFile, str]:
|
||||
if asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||
raise FileStorageError("File not found")
|
||||
safe_filename = filename_from_path(normalize_logical_path(filename, fallback_filename="file"))
|
||||
if not content_type:
|
||||
content_type = mimetypes.guess_type(safe_filename)[0] or "application/octet-stream"
|
||||
current_version, current_blob = current_version_and_blob(session, asset)
|
||||
checksum = hashlib.sha256(data).hexdigest()
|
||||
asset.metadata_ = metadata
|
||||
session.add(asset)
|
||||
inherited_vault_id = encryption_vault_id
|
||||
if inherited_vault_id is None and current_blob.encryption_envelope_id:
|
||||
prefix = "vault:"
|
||||
if current_blob.protection_discriminator.startswith(prefix):
|
||||
inherited_vault_id = current_blob.protection_discriminator[len(prefix) :]
|
||||
inherited_vault_id = str(inherited_vault_id or "").strip() or None
|
||||
target_protection = (
|
||||
f"vault:{inherited_vault_id}" if inherited_vault_id else "plaintext"
|
||||
)
|
||||
if (
|
||||
current_blob.checksum_sha256 == checksum
|
||||
and current_blob.size_bytes == len(data)
|
||||
and current_blob.protection_discriminator == target_protection
|
||||
):
|
||||
return UploadedStoredFile(asset=asset, version=current_version, blob=current_blob), "unchanged"
|
||||
|
||||
blob = _get_or_create_blob(session, tenant_id=tenant_id, data=data, filename=safe_filename, content_type=content_type, actor_id=user_id, encryption_vault_id=inherited_vault_id)
|
||||
version = FileVersion(
|
||||
tenant_id=tenant_id,
|
||||
file_asset_id=asset.id,
|
||||
blob_id=blob.id,
|
||||
version_number=_next_version_number(session, asset.id),
|
||||
filename_at_upload=safe_filename,
|
||||
display_path_at_upload=asset.display_path,
|
||||
content_type=content_type,
|
||||
size_bytes=blob.size_bytes,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
created_by_user_id=user_id,
|
||||
)
|
||||
session.add(version)
|
||||
session.flush()
|
||||
asset.current_version_id = version.id
|
||||
session.add(asset)
|
||||
return UploadedStoredFile(asset=asset, version=version, blob=blob), "updated"
|
||||
|
||||
|
||||
def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_id: str, require_write: bool = False, is_admin: bool = False) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||
@@ -192,7 +459,7 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset.id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
effective_file_share_clause(),
|
||||
FileShare.permission.in_(permission_values),
|
||||
or_(
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
@@ -207,6 +474,32 @@ def get_asset_for_user(session: Session, *, tenant_id: str, user_id: str, asset_
|
||||
return asset
|
||||
|
||||
|
||||
def get_asset_for_share_management(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
asset_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if not asset or asset.tenant_id != tenant_id or asset.deleted_at is not None:
|
||||
raise FileStorageError("File not found")
|
||||
if is_admin:
|
||||
return asset
|
||||
owns_asset = (
|
||||
asset.owner_type == "user"
|
||||
and asset.owner_user_id == user_id
|
||||
) or (
|
||||
asset.owner_type == "group"
|
||||
and asset.owner_group_id
|
||||
in user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||
)
|
||||
if not owns_asset:
|
||||
raise FileStorageError("Only file owners and administrators can manage shares")
|
||||
return asset
|
||||
|
||||
|
||||
def list_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -216,9 +509,119 @@ def list_assets_for_user(
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileAsset]:
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).all()
|
||||
|
||||
|
||||
def list_recent_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
limit: int,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileAsset]:
|
||||
"""Return a bounded recent projection through the normal access query."""
|
||||
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return (
|
||||
query.order_by(FileAsset.updated_at.desc(), FileAsset.id.asc())
|
||||
.limit(max(1, limit))
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def list_assets_for_user_window(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
page_size: int,
|
||||
after_display_path: str | None = None,
|
||||
after_updated_at=None,
|
||||
after_id: str | None = None,
|
||||
) -> tuple[list[FileAsset], bool]:
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if after_display_path is not None and after_updated_at is not None and after_id:
|
||||
query = query.filter(
|
||||
or_(
|
||||
FileAsset.display_path > after_display_path,
|
||||
and_(FileAsset.display_path == after_display_path, FileAsset.updated_at < after_updated_at),
|
||||
and_(FileAsset.display_path == after_display_path, FileAsset.updated_at == after_updated_at, FileAsset.id > after_id),
|
||||
)
|
||||
)
|
||||
rows = query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc(), FileAsset.id.asc()).limit(page_size + 1).all()
|
||||
return rows[:page_size], len(rows) > page_size
|
||||
|
||||
|
||||
def _asset_visibility_query_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
):
|
||||
query = session.query(FileAsset).filter(FileAsset.tenant_id == tenant_id)
|
||||
if not include_deleted:
|
||||
query = query.filter(FileAsset.deleted_at.is_(None))
|
||||
@@ -229,28 +632,95 @@ def list_assets_for_user(
|
||||
if owner_type == "group" and owner_id:
|
||||
query = query.filter(FileAsset.owner_group_id == owner_id)
|
||||
if campaign_id:
|
||||
query = query.join(FileShare, FileShare.file_asset_id == FileAsset.id).filter(
|
||||
campaign_share = exists().where(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == FileAsset.id,
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
query = query.filter(campaign_share)
|
||||
elif not is_admin and not owner_type:
|
||||
group_ids = user_group_ids(session, tenant_id=tenant_id, user_id=user_id)
|
||||
query = query.outerjoin(FileShare, FileShare.file_asset_id == FileAsset.id).filter(
|
||||
active_share = exists().where(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == FileAsset.id,
|
||||
effective_file_share_clause(),
|
||||
or_(
|
||||
(FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
(FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
|
||||
(FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
|
||||
)
|
||||
)
|
||||
query = query.filter(
|
||||
or_(
|
||||
(FileAsset.owner_type == "user") & (FileAsset.owner_user_id == user_id),
|
||||
(FileAsset.owner_type == "group") & (FileAsset.owner_group_id.in_(group_ids)),
|
||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "user") & (FileShare.target_id == user_id),
|
||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "group") & (FileShare.target_id.in_(group_ids)),
|
||||
(FileShare.revoked_at.is_(None)) & (FileShare.target_type == "tenant") & (FileShare.target_id == tenant_id),
|
||||
active_share,
|
||||
)
|
||||
)
|
||||
if path_prefix:
|
||||
prefix = normalize_folder(path_prefix)
|
||||
if prefix:
|
||||
query = query.filter(FileAsset.display_path.like(f"{prefix}/%"))
|
||||
return query.order_by(FileAsset.display_path.asc(), FileAsset.updated_at.desc()).all()
|
||||
campaign_share_exists = exists().where(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == FileAsset.id,
|
||||
FileShare.target_type == "campaign",
|
||||
effective_file_share_clause(),
|
||||
)
|
||||
campaign_use_exists = exists().where(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == FileAsset.id,
|
||||
)
|
||||
if campaign_usage == "linked":
|
||||
query = query.filter(or_(campaign_share_exists, campaign_use_exists))
|
||||
elif campaign_usage == "unlinked":
|
||||
query = query.filter(~or_(campaign_share_exists, campaign_use_exists))
|
||||
|
||||
if audit_relevant is not None:
|
||||
sent_use_exists = exists().where(
|
||||
CampaignAttachmentUse.tenant_id == tenant_id,
|
||||
CampaignAttachmentUse.file_asset_id == FileAsset.id,
|
||||
CampaignAttachmentUse.use_stage == "sent",
|
||||
)
|
||||
query = query.filter(sent_use_exists if audit_relevant else ~sent_use_exists)
|
||||
return query
|
||||
|
||||
|
||||
def count_assets_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str | None = None,
|
||||
owner_id: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
path_prefix: str | None = None,
|
||||
campaign_usage: str | None = None,
|
||||
audit_relevant: bool | None = None,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> int:
|
||||
query = _asset_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
campaign_usage=campaign_usage,
|
||||
audit_relevant=audit_relevant,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return int(
|
||||
query.order_by(None)
|
||||
.with_entities(func.count(func.distinct(FileAsset.id)))
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
|
||||
|
||||
def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVersion, FileBlob]:
|
||||
@@ -265,13 +735,51 @@ def current_version_and_blob(session: Session, asset: FileAsset) -> tuple[FileVe
|
||||
return version, blob
|
||||
|
||||
|
||||
def current_versions_and_blobs(session: Session, assets: Iterable[FileAsset]) -> dict[str, tuple[FileVersion, FileBlob]]:
|
||||
asset_list = list(assets)
|
||||
if not asset_list:
|
||||
return {}
|
||||
version_ids = [asset.current_version_id for asset in asset_list if asset.current_version_id]
|
||||
if len(version_ids) != len(asset_list):
|
||||
raise FileStorageError("File has no current version")
|
||||
|
||||
rows: list[tuple[FileVersion, FileBlob]] = []
|
||||
for chunk in _chunks(version_ids):
|
||||
rows.extend(
|
||||
session.query(FileVersion, FileBlob)
|
||||
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
|
||||
.filter(FileVersion.id.in_(chunk))
|
||||
.all()
|
||||
)
|
||||
by_version_id = {version.id: (version, blob) for version, blob in rows}
|
||||
result: dict[str, tuple[FileVersion, FileBlob]] = {}
|
||||
for asset in asset_list:
|
||||
version_blob = by_version_id.get(asset.current_version_id or "")
|
||||
if not version_blob:
|
||||
raise FileStorageError("File version not found")
|
||||
result[asset.id] = version_blob
|
||||
return result
|
||||
|
||||
|
||||
def read_asset_bytes(session: Session, asset: FileAsset) -> tuple[bytes, FileVersion, FileBlob]:
|
||||
version, blob = current_version_and_blob(session, asset)
|
||||
backend = get_storage_backend()
|
||||
try:
|
||||
return backend.get_bytes(blob.storage_key), version, blob
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||
|
||||
|
||||
def read_asset_version_bytes(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
version_id: str,
|
||||
) -> tuple[bytes, FileVersion, FileBlob]:
|
||||
version = session.get(FileVersion, version_id)
|
||||
if version is None or version.file_asset_id != asset.id:
|
||||
raise FileStorageError("File version not found")
|
||||
blob = session.get(FileBlob, version.blob_id)
|
||||
if blob is None or blob.tenant_id != asset.tenant_id:
|
||||
raise FileStorageError("File blob not found")
|
||||
backend = get_storage_backend()
|
||||
return read_verified_blob_bytes(blob, backend=backend), version, blob
|
||||
|
||||
|
||||
def share_file(
|
||||
@@ -283,6 +791,7 @@ def share_file(
|
||||
target_id: str,
|
||||
permission: str,
|
||||
user_id: str,
|
||||
expires_at: datetime | None = None,
|
||||
) -> FileShare:
|
||||
target_type = target_type.lower().strip()
|
||||
permission = permission.lower().strip()
|
||||
@@ -292,23 +801,11 @@ def share_file(
|
||||
raise FileStorageError("Unsupported share target")
|
||||
if permission not in {"read", "write", "manage"}:
|
||||
raise FileStorageError("Unsupported file permission")
|
||||
if target_type == "user":
|
||||
target_user = session.get(User, target_id)
|
||||
if not target_user or target_user.tenant_id != tenant_id or not target_user.is_active:
|
||||
raise FileStorageError("User not found")
|
||||
if target_type == "group":
|
||||
group = session.get(Group, target_id)
|
||||
if not group or group.tenant_id != tenant_id or not group.is_active:
|
||||
raise FileStorageError("Group not found")
|
||||
if target_type == "tenant":
|
||||
tenant = session.get(Tenant, target_id)
|
||||
if target_id != tenant_id or not tenant or not tenant.is_active:
|
||||
raise FileStorageError("Tenant not found")
|
||||
_validate_share_expiry(expires_at)
|
||||
if target_type in {"user", "group", "tenant"}:
|
||||
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
|
||||
if target_type == "campaign":
|
||||
Campaign = _campaign_model()
|
||||
campaign = session.get(Campaign, target_id)
|
||||
if not campaign or campaign.tenant_id != tenant_id:
|
||||
raise FileStorageError("Campaign not found")
|
||||
_ensure_campaign_exists(session, tenant_id=tenant_id, campaign_id=target_id)
|
||||
existing = (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
@@ -322,6 +819,7 @@ def share_file(
|
||||
)
|
||||
if existing:
|
||||
existing.permission = permission
|
||||
existing.expires_at = expires_at
|
||||
session.add(existing)
|
||||
return existing
|
||||
share = FileShare(
|
||||
@@ -331,6 +829,7 @@ def share_file(
|
||||
target_id=target_id,
|
||||
permission=permission,
|
||||
created_by_user_id=user_id,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
session.add(share)
|
||||
return share
|
||||
@@ -345,6 +844,7 @@ def share_files(
|
||||
target_id: str,
|
||||
permission: str,
|
||||
user_id: str,
|
||||
expires_at: datetime | None = None,
|
||||
) -> list[FileShare]:
|
||||
target_type = target_type.lower().strip()
|
||||
permission = permission.lower().strip()
|
||||
@@ -352,23 +852,11 @@ def share_files(
|
||||
raise FileStorageError("Unsupported share target")
|
||||
if permission not in {"read", "write", "manage"}:
|
||||
raise FileStorageError("Unsupported file permission")
|
||||
if target_type == "user":
|
||||
target_user = session.get(User, target_id)
|
||||
if not target_user or target_user.tenant_id != tenant_id or not target_user.is_active:
|
||||
raise FileStorageError("User not found")
|
||||
if target_type == "group":
|
||||
group = session.get(Group, target_id)
|
||||
if not group or group.tenant_id != tenant_id or not group.is_active:
|
||||
raise FileStorageError("Group not found")
|
||||
if target_type == "tenant":
|
||||
tenant = session.get(Tenant, target_id)
|
||||
if target_id != tenant_id or not tenant or not tenant.is_active:
|
||||
raise FileStorageError("Tenant not found")
|
||||
_validate_share_expiry(expires_at)
|
||||
if target_type in {"user", "group", "tenant"}:
|
||||
ensure_share_target_exists(tenant_id=tenant_id, target_type=target_type, target_id=target_id)
|
||||
if target_type == "campaign":
|
||||
Campaign = _campaign_model()
|
||||
campaign = session.get(Campaign, target_id)
|
||||
if not campaign or campaign.tenant_id != tenant_id:
|
||||
raise FileStorageError("Campaign not found")
|
||||
_ensure_campaign_exists(session, tenant_id=tenant_id, campaign_id=target_id)
|
||||
|
||||
asset_list = list(assets)
|
||||
if not asset_list:
|
||||
@@ -392,6 +880,7 @@ def share_files(
|
||||
existing = existing_by_asset.get(asset.id)
|
||||
if existing:
|
||||
existing.permission = permission
|
||||
existing.expires_at = expires_at
|
||||
session.add(existing)
|
||||
shares.append(existing)
|
||||
continue
|
||||
@@ -402,12 +891,88 @@ def share_files(
|
||||
target_id=target_id,
|
||||
permission=permission,
|
||||
created_by_user_id=user_id,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
session.add(share)
|
||||
shares.append(share)
|
||||
return shares
|
||||
|
||||
|
||||
def list_file_shares(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
asset_id: str,
|
||||
include_inactive: bool = False,
|
||||
) -> list[FileShare]:
|
||||
query = session.query(FileShare).filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset_id,
|
||||
)
|
||||
if not include_inactive:
|
||||
query = query.filter(effective_file_share_clause())
|
||||
return query.order_by(FileShare.created_at.desc(), FileShare.id.desc()).all()
|
||||
|
||||
|
||||
def revoke_file_share(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
asset_id: str,
|
||||
share_id: str,
|
||||
user_id: str,
|
||||
) -> tuple[FileShare, bool]:
|
||||
share = (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
FileShare.id == share_id,
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if share is None:
|
||||
raise FileStorageError("File share not found")
|
||||
if share.revoked_at is not None:
|
||||
return share, False
|
||||
share.revoked_at = utcnow()
|
||||
share.revoked_by_user_id = user_id
|
||||
session.add(share)
|
||||
return share, True
|
||||
|
||||
|
||||
def current_file_share_for_target(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
asset_id: str,
|
||||
target_type: str,
|
||||
target_id: str,
|
||||
) -> FileShare | None:
|
||||
return (
|
||||
session.query(FileShare)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id == asset_id,
|
||||
FileShare.target_type == target_type.lower().strip(),
|
||||
FileShare.target_id == target_id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
)
|
||||
.order_by(FileShare.created_at.desc(), FileShare.id.desc())
|
||||
.first()
|
||||
)
|
||||
|
||||
|
||||
def _validate_share_expiry(expires_at: datetime | None) -> None:
|
||||
if expires_at is None:
|
||||
return
|
||||
from govoplan_files.backend.storage.share_state import file_share_is_active
|
||||
|
||||
candidate = FileShare(expires_at=expires_at)
|
||||
if not file_share_is_active(candidate):
|
||||
raise FileStorageError("File share expiry must be in the future")
|
||||
|
||||
|
||||
|
||||
def soft_delete_assets(session: Session, assets: Iterable[FileAsset]) -> int:
|
||||
count = 0
|
||||
@@ -437,6 +1002,11 @@ def _asset_owner_id(asset: FileAsset) -> str:
|
||||
raise FileStorageError("File has no valid owner")
|
||||
|
||||
|
||||
def _chunks(values: list[str], size: int = 900):
|
||||
for index in range(0, len(values), size):
|
||||
yield values[index:index + size]
|
||||
|
||||
|
||||
def _active_asset_exists(session: Session, *, tenant_id: str, owner_type: str, owner_id: str, path: str, exclude_asset_id: str | None = None) -> bool:
|
||||
return _active_asset_at_path(
|
||||
session,
|
||||
@@ -538,11 +1108,44 @@ def _resolution_by_path(conflict_resolutions: Iterable[FileConflictResolution] |
|
||||
|
||||
def _normalize_conflict_strategy(strategy: str | None) -> str:
|
||||
normalized = (strategy or "reject").lower().strip()
|
||||
if normalized not in {"reject", "overwrite", "rename"}:
|
||||
if normalized not in {"reject", "overwrite", "rename", "skip"}:
|
||||
raise FileStorageError("Unsupported conflict strategy")
|
||||
return normalized
|
||||
|
||||
|
||||
def _next_version_number(session: Session, asset_id: str) -> int:
|
||||
row = (
|
||||
session.query(FileVersion.version_number)
|
||||
.filter(FileVersion.file_asset_id == asset_id)
|
||||
.order_by(FileVersion.version_number.desc())
|
||||
.first()
|
||||
)
|
||||
return (int(row[0]) if row else 0) + 1
|
||||
|
||||
|
||||
def _source_identity(provenance: dict[str, Any] | None) -> tuple[object, ...] | None:
|
||||
if not provenance:
|
||||
return None
|
||||
connector_id = _clean_identity(provenance.get("connector_id"))
|
||||
provider = _clean_identity(provenance.get("provider"))
|
||||
external_id = _clean_identity(provenance.get("external_id"))
|
||||
if connector_id and external_id:
|
||||
return ("external_id", connector_id, provider, external_id)
|
||||
external_path = _clean_identity(provenance.get("external_path"))
|
||||
metadata = provenance.get("metadata") if isinstance(provenance.get("metadata"), dict) else {}
|
||||
library_id = _clean_identity(metadata.get("library_id") or metadata.get("profile_id") or metadata.get("share"))
|
||||
if connector_id and external_path:
|
||||
return ("external_path", connector_id, provider, library_id, external_path)
|
||||
return None
|
||||
|
||||
|
||||
def _clean_identity(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def _copy_asset_to_path(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
|
||||
@@ -68,13 +68,63 @@ def list_folders_for_user(
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
) -> list[FileFolder]:
|
||||
query = _folder_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return query.order_by(FileFolder.path.asc(), FileFolder.id.asc()).all()
|
||||
|
||||
|
||||
def list_folders_for_user_window(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
page_size: int,
|
||||
after_path: str | None = None,
|
||||
after_id: str | None = None,
|
||||
) -> tuple[list[FileFolder], bool]:
|
||||
query = _folder_visibility_query_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_deleted=include_deleted,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if after_path is not None and after_id:
|
||||
query = query.filter(or_(FileFolder.path > after_path, (FileFolder.path == after_path) & (FileFolder.id > after_id)))
|
||||
rows = query.order_by(FileFolder.path.asc(), FileFolder.id.asc()).limit(page_size + 1).all()
|
||||
return rows[:page_size], len(rows) > page_size
|
||||
|
||||
|
||||
def _folder_visibility_query_for_user(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
include_deleted: bool = False,
|
||||
is_admin: bool = False,
|
||||
):
|
||||
owner_type = owner_type.lower().strip()
|
||||
ensure_owner_access(session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id, user_id=user_id, is_admin=is_admin)
|
||||
query = session.query(FileFolder).filter(FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type)
|
||||
query = _owner_filter(query, owner_type, owner_id)
|
||||
if not include_deleted:
|
||||
query = query.filter(FileFolder.deleted_at.is_(None))
|
||||
return query.order_by(FileFolder.path.asc()).all()
|
||||
return query
|
||||
|
||||
|
||||
def soft_delete_folder(
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import ssl
|
||||
import select
|
||||
from collections.abc import Mapping
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterator
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
bounded_chunks_bytes,
|
||||
create_outbound_connection,
|
||||
response_limit,
|
||||
validate_outbound_http_url,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorHttpError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorHttpResponse:
|
||||
status_code: int
|
||||
headers: Mapping[str, str]
|
||||
content: bytes
|
||||
|
||||
|
||||
_HTTPCORE_TRANSPORT_ERRORS = (
|
||||
httpcore.TimeoutException,
|
||||
httpcore.NetworkError,
|
||||
httpcore.ProtocolError,
|
||||
httpcore.ProxyError,
|
||||
httpcore.UnsupportedProtocol,
|
||||
)
|
||||
|
||||
|
||||
class _SocketNetworkStream(httpcore.NetworkStream):
|
||||
"""Public httpcore NetworkStream adapter around an approved socket."""
|
||||
|
||||
def __init__(self, sock: socket.socket) -> None:
|
||||
self._socket = sock
|
||||
|
||||
def read(self, max_bytes: int, timeout: float | None = None) -> bytes:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
return self._socket.recv(max_bytes)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.ReadTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
raise httpcore.ReadError(str(exc)) from exc
|
||||
|
||||
def write(self, buffer: bytes, timeout: float | None = None) -> None:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
self._socket.sendall(buffer)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.WriteTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
raise httpcore.WriteError(str(exc)) from exc
|
||||
|
||||
def close(self) -> None:
|
||||
self._socket.close()
|
||||
|
||||
def start_tls(
|
||||
self,
|
||||
ssl_context: ssl.SSLContext,
|
||||
server_hostname: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> httpcore.NetworkStream:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
tls_socket = ssl_context.wrap_socket(self._socket, server_hostname=server_hostname)
|
||||
except socket.timeout as exc:
|
||||
self.close()
|
||||
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
self.close()
|
||||
raise httpcore.ConnectError(str(exc)) from exc
|
||||
return _SocketNetworkStream(tls_socket)
|
||||
|
||||
def get_extra_info(self, info: str) -> Any:
|
||||
if info == "ssl_object" and isinstance(self._socket, ssl.SSLSocket):
|
||||
return self._socket
|
||||
if info == "client_addr":
|
||||
return self._socket.getsockname()
|
||||
if info == "server_addr":
|
||||
return self._socket.getpeername()
|
||||
if info == "socket":
|
||||
return self._socket
|
||||
if info == "is_readable":
|
||||
try:
|
||||
return bool(select.select([self._socket], [], [], 0)[0])
|
||||
except (OSError, ValueError):
|
||||
return True
|
||||
return None
|
||||
|
||||
|
||||
class _OutboundPolicyNetworkBackend(httpcore.NetworkBackend):
|
||||
def connect_tcp(
|
||||
self,
|
||||
host: str,
|
||||
port: int,
|
||||
timeout: float | None = None,
|
||||
local_address: str | None = None,
|
||||
socket_options: Any = None,
|
||||
) -> httpcore.NetworkStream:
|
||||
source_address = None if local_address is None else (local_address, 0)
|
||||
try:
|
||||
sock = create_outbound_connection(
|
||||
host,
|
||||
port,
|
||||
timeout=timeout,
|
||||
source_address=source_address,
|
||||
socket_options=socket_options,
|
||||
label="File connector HTTP endpoint",
|
||||
)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||
except (OSError, OutboundHttpError) as exc:
|
||||
raise httpcore.ConnectError(str(exc)) from exc
|
||||
return _SocketNetworkStream(sock)
|
||||
|
||||
def connect_unix_socket(self, path: str, timeout: float | None = None, socket_options: Any = None): # type: ignore[no-untyped-def]
|
||||
del path, timeout, socket_options
|
||||
raise httpcore.ConnectError("Unix sockets are not supported for file connectors")
|
||||
|
||||
|
||||
class _HttpcoreResponseStream(httpx.SyncByteStream):
|
||||
def __init__(self, stream: Any, *, request: httpx.Request) -> None:
|
||||
self._stream = stream
|
||||
self._request = request
|
||||
|
||||
def __iter__(self): # type: ignore[no-untyped-def]
|
||||
try:
|
||||
yield from self._stream
|
||||
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||
raise httpx.TransportError(str(exc), request=self._request) from exc
|
||||
|
||||
def close(self) -> None:
|
||||
if hasattr(self._stream, "close"):
|
||||
self._stream.close()
|
||||
|
||||
|
||||
class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
|
||||
def __init__(self) -> None:
|
||||
self._connection_pool = httpcore.ConnectionPool(
|
||||
ssl_context=httpx.create_ssl_context(verify=True, trust_env=False),
|
||||
max_connections=100,
|
||||
max_keepalive_connections=20,
|
||||
keepalive_expiry=5.0,
|
||||
network_backend=_OutboundPolicyNetworkBackend(),
|
||||
)
|
||||
|
||||
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||
core_request = httpcore.Request(
|
||||
method=request.method,
|
||||
url=httpcore.URL(
|
||||
scheme=request.url.raw_scheme,
|
||||
host=request.url.raw_host,
|
||||
port=request.url.port,
|
||||
target=request.url.raw_path,
|
||||
),
|
||||
headers=request.headers.raw,
|
||||
content=request.stream,
|
||||
extensions=request.extensions,
|
||||
)
|
||||
try:
|
||||
response = self._connection_pool.handle_request(core_request)
|
||||
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||
raise httpx.TransportError(str(exc), request=request) from exc
|
||||
return httpx.Response(
|
||||
status_code=response.status,
|
||||
headers=response.headers,
|
||||
stream=_HttpcoreResponseStream(response.stream, request=request),
|
||||
extensions=response.extensions,
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
self._connection_pool.close()
|
||||
|
||||
|
||||
_CONNECTOR_HTTP_CLIENT = httpx.Client(
|
||||
transport=_OutboundPolicyHTTPTransport(),
|
||||
follow_redirects=False,
|
||||
timeout=15.0,
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
|
||||
timeout = kwargs.pop("timeout", 15.0)
|
||||
with _CONNECTOR_HTTP_CLIENT.stream(
|
||||
method,
|
||||
url,
|
||||
timeout=timeout,
|
||||
**kwargs,
|
||||
) as response:
|
||||
yield response
|
||||
|
||||
|
||||
def request_connector_bytes(
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
headers: Mapping[str, str] | None = None,
|
||||
params: Mapping[str, str] | None = None,
|
||||
data: Mapping[str, str] | bytes | str | None = None,
|
||||
content: bytes | str | None = None,
|
||||
auth: Any = None,
|
||||
timeout: float = 15.0,
|
||||
kind: str = "structured",
|
||||
max_bytes: int | None = None,
|
||||
label: str = "File connector",
|
||||
) -> ConnectorHttpResponse:
|
||||
try:
|
||||
validated_url = validate_outbound_http_url(url, label=f"{label} URL")
|
||||
effective_limit = response_limit(kind) if max_bytes is None else min(int(max_bytes), response_limit(kind))
|
||||
with _stream_connector_request(
|
||||
method,
|
||||
validated_url,
|
||||
headers=dict(headers or {}),
|
||||
params=params,
|
||||
data=data,
|
||||
content=content,
|
||||
auth=auth,
|
||||
timeout=timeout,
|
||||
) as response:
|
||||
body = bounded_chunks_bytes(
|
||||
response.iter_bytes(),
|
||||
headers=response.headers,
|
||||
max_bytes=effective_limit,
|
||||
kind=kind,
|
||||
label=f"{label} response",
|
||||
)
|
||||
return ConnectorHttpResponse(
|
||||
status_code=response.status_code,
|
||||
headers=dict(response.headers),
|
||||
content=body,
|
||||
)
|
||||
except (httpx.HTTPError, OutboundHttpError, ValueError) as exc:
|
||||
raise ConnectorHttpError(str(exc)) from exc
|
||||
@@ -0,0 +1,550 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy.orm import Session, object_session
|
||||
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileBlob,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
)
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.recovery import (
|
||||
begin_orphan_cleanup_recovery,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
|
||||
|
||||
TERMINAL_SCAN_STATUSES = {"completed", "cancelled"}
|
||||
QUARANTINED_BLOB_STATUSES = {
|
||||
"missing",
|
||||
"size_mismatch",
|
||||
"checksum_mismatch",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BlobInspection:
|
||||
valid: bool
|
||||
kind: str
|
||||
observed_size_bytes: int | None = None
|
||||
observed_checksum_sha256: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class IntegrityActionResult:
|
||||
action: str
|
||||
changed: bool
|
||||
dry_run: bool
|
||||
finding: FileIntegrityFinding
|
||||
inspection: BlobInspection | None = None
|
||||
|
||||
|
||||
def storage_prefix_for_tenant(tenant_id: str) -> str:
|
||||
return f"tenants/{tenant_id}/files/"
|
||||
|
||||
|
||||
def create_integrity_scan(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
verify_checksums: bool = True,
|
||||
batch_size: int = 100,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> FileIntegrityScan:
|
||||
active_backend = backend or get_storage_backend()
|
||||
scan = FileIntegrityScan(
|
||||
tenant_id=tenant_id,
|
||||
storage_backend=active_backend.name,
|
||||
storage_prefix=storage_prefix_for_tenant(tenant_id),
|
||||
verify_checksums=verify_checksums,
|
||||
batch_size=max(1, min(int(batch_size), 1000)),
|
||||
created_by_user_id=user_id,
|
||||
)
|
||||
session.add(scan)
|
||||
session.flush()
|
||||
return scan
|
||||
|
||||
|
||||
def run_integrity_scan_batch(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
*,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> FileIntegrityScan:
|
||||
if scan.status in TERMINAL_SCAN_STATUSES:
|
||||
return scan
|
||||
active_backend = backend or get_storage_backend()
|
||||
if active_backend.name != scan.storage_backend:
|
||||
raise FileStorageError(
|
||||
"The configured storage backend changed after this integrity scan started"
|
||||
)
|
||||
if scan.started_at is None:
|
||||
scan.started_at = utcnow()
|
||||
scan.status = "running"
|
||||
scan.last_error = None
|
||||
if scan.phase == "blobs":
|
||||
_scan_blob_batch(session, scan, active_backend)
|
||||
elif scan.phase == "objects":
|
||||
_scan_object_batch(session, scan, active_backend)
|
||||
else:
|
||||
scan.phase = "completed"
|
||||
scan.status = "completed"
|
||||
scan.completed_at = utcnow()
|
||||
scan.revision += 1
|
||||
session.add(scan)
|
||||
return scan
|
||||
|
||||
|
||||
def mark_integrity_scan_failed(
|
||||
scan: FileIntegrityScan,
|
||||
*,
|
||||
error: Exception,
|
||||
) -> None:
|
||||
scan.status = "failed"
|
||||
scan.last_error = type(error).__name__[:255]
|
||||
scan.revision += 1
|
||||
|
||||
|
||||
def inspect_blob(
|
||||
blob: FileBlob,
|
||||
*,
|
||||
backend: StorageBackend,
|
||||
verify_checksum: bool = True,
|
||||
) -> BlobInspection:
|
||||
expected_size = blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes
|
||||
expected_checksum = blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256
|
||||
try:
|
||||
info = backend.stat(blob.storage_key)
|
||||
except StorageObjectMissing:
|
||||
return BlobInspection(valid=False, kind="missing")
|
||||
if info.size_bytes != expected_size:
|
||||
return BlobInspection(
|
||||
valid=False,
|
||||
kind="size_mismatch",
|
||||
observed_size_bytes=info.size_bytes,
|
||||
)
|
||||
if not verify_checksum:
|
||||
return BlobInspection(
|
||||
valid=True,
|
||||
kind="verified",
|
||||
observed_size_bytes=info.size_bytes,
|
||||
)
|
||||
digest = hashlib.sha256()
|
||||
observed_size = 0
|
||||
for chunk in backend.iter_bytes(blob.storage_key):
|
||||
observed_size += len(chunk)
|
||||
digest.update(chunk)
|
||||
observed_checksum = digest.hexdigest()
|
||||
if observed_size != expected_size:
|
||||
return BlobInspection(
|
||||
valid=False,
|
||||
kind="size_mismatch",
|
||||
observed_size_bytes=observed_size,
|
||||
observed_checksum_sha256=observed_checksum,
|
||||
)
|
||||
if observed_checksum != expected_checksum:
|
||||
return BlobInspection(
|
||||
valid=False,
|
||||
kind="checksum_mismatch",
|
||||
observed_size_bytes=observed_size,
|
||||
observed_checksum_sha256=observed_checksum,
|
||||
)
|
||||
return BlobInspection(
|
||||
valid=True,
|
||||
kind="verified",
|
||||
observed_size_bytes=observed_size,
|
||||
observed_checksum_sha256=observed_checksum,
|
||||
)
|
||||
|
||||
|
||||
def apply_blob_inspection(
|
||||
session: Session,
|
||||
blob: FileBlob,
|
||||
inspection: BlobInspection,
|
||||
*,
|
||||
checked_at=None,
|
||||
) -> None:
|
||||
timestamp = checked_at or utcnow()
|
||||
blob.integrity_checked_at = timestamp
|
||||
if inspection.valid:
|
||||
blob.integrity_status = "verified"
|
||||
blob.integrity_failure = None
|
||||
blob.quarantined_at = None
|
||||
else:
|
||||
blob.integrity_status = inspection.kind
|
||||
blob.integrity_failure = inspection.kind
|
||||
blob.quarantined_at = blob.quarantined_at or timestamp
|
||||
session.add(blob)
|
||||
|
||||
|
||||
def ensure_blob_is_readable(blob: FileBlob) -> None:
|
||||
if blob.integrity_status in QUARANTINED_BLOB_STATUSES or blob.quarantined_at:
|
||||
raise FileStorageError(
|
||||
"Managed file content is quarantined because integrity verification failed"
|
||||
)
|
||||
|
||||
|
||||
def read_verified_blob_bytes(
|
||||
blob: FileBlob,
|
||||
*,
|
||||
backend: StorageBackend,
|
||||
) -> bytes:
|
||||
ensure_blob_is_readable(blob)
|
||||
try:
|
||||
data = backend.get_bytes(blob.storage_key)
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError("Managed file content is not available") from exc
|
||||
expected_storage_size = blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes
|
||||
expected_storage_checksum = blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256
|
||||
if len(data) != expected_storage_size:
|
||||
raise FileStorageError(
|
||||
"Managed file content failed its recorded size verification"
|
||||
)
|
||||
if hashlib.sha256(data).hexdigest() != expected_storage_checksum:
|
||||
raise FileStorageError(
|
||||
"Managed file content failed its recorded checksum verification"
|
||||
)
|
||||
if blob.encryption_envelope_id:
|
||||
session = object_session(blob)
|
||||
if session is None:
|
||||
raise FileStorageError("Encrypted managed content requires an attached database session")
|
||||
from govoplan_files.backend.storage.content_protection import unprotect_blob_content
|
||||
|
||||
data = unprotect_blob_content(
|
||||
session,
|
||||
tenant_id=blob.tenant_id,
|
||||
blob_id=blob.id,
|
||||
envelope_id=blob.encryption_envelope_id,
|
||||
ciphertext=data,
|
||||
)
|
||||
if len(data) != blob.size_bytes:
|
||||
raise FileStorageError("Decrypted managed file content failed its recorded size verification")
|
||||
if hashlib.sha256(data).hexdigest() != blob.checksum_sha256:
|
||||
raise FileStorageError("Decrypted managed file content failed its recorded checksum verification")
|
||||
return data
|
||||
|
||||
|
||||
def recheck_integrity_finding(
|
||||
session: Session,
|
||||
finding: FileIntegrityFinding,
|
||||
*,
|
||||
user_id: str,
|
||||
dry_run: bool,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> IntegrityActionResult:
|
||||
if finding.kind == "orphan_object" or not finding.blob_id:
|
||||
raise FileStorageError("Only blob integrity findings can be rechecked")
|
||||
blob = session.get(FileBlob, finding.blob_id)
|
||||
if blob is None or blob.tenant_id != finding.tenant_id:
|
||||
raise FileStorageError("Integrity finding blob no longer exists")
|
||||
active_backend = backend or get_storage_backend()
|
||||
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||
if scan is None or scan.storage_backend != active_backend.name:
|
||||
raise FileStorageError(
|
||||
"The configured storage backend does not match the integrity finding"
|
||||
)
|
||||
inspection = inspect_blob(
|
||||
blob,
|
||||
backend=active_backend,
|
||||
verify_checksum=True,
|
||||
)
|
||||
changed = False
|
||||
if not dry_run:
|
||||
previous = (
|
||||
blob.integrity_status,
|
||||
blob.quarantined_at,
|
||||
finding.state,
|
||||
)
|
||||
apply_blob_inspection(session, blob, inspection)
|
||||
_update_finding_from_inspection(finding, inspection)
|
||||
if inspection.valid:
|
||||
finding.state = "resolved"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.resolved_by_user_id = user_id
|
||||
session.add(finding)
|
||||
finding.revision += 1
|
||||
changed = previous != (
|
||||
blob.integrity_status,
|
||||
blob.quarantined_at,
|
||||
finding.state,
|
||||
)
|
||||
return IntegrityActionResult(
|
||||
action="recheck",
|
||||
changed=changed,
|
||||
dry_run=dry_run,
|
||||
finding=finding,
|
||||
inspection=inspection,
|
||||
)
|
||||
|
||||
|
||||
def cleanup_orphan_finding(
|
||||
session: Session,
|
||||
finding: FileIntegrityFinding,
|
||||
*,
|
||||
user_id: str,
|
||||
dry_run: bool,
|
||||
backend: StorageBackend | None = None,
|
||||
) -> IntegrityActionResult:
|
||||
if finding.kind != "orphan_object":
|
||||
raise FileStorageError("Only orphan-object findings can be cleaned up")
|
||||
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||
if scan is None or scan.tenant_id != finding.tenant_id:
|
||||
raise FileStorageError("Integrity scan not found")
|
||||
if not finding.storage_key.startswith(scan.storage_prefix):
|
||||
raise FileStorageError("Orphan cleanup is outside the scan storage scope")
|
||||
referenced = (
|
||||
session.query(FileBlob.id)
|
||||
.filter(
|
||||
FileBlob.tenant_id == finding.tenant_id,
|
||||
FileBlob.storage_backend == scan.storage_backend,
|
||||
FileBlob.storage_key == finding.storage_key,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if referenced:
|
||||
if not dry_run and finding.state != "resolved":
|
||||
finding.state = "resolved"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.resolved_by_user_id = user_id
|
||||
finding.revision += 1
|
||||
session.add(finding)
|
||||
return IntegrityActionResult(
|
||||
action="retained_referenced",
|
||||
changed=True,
|
||||
dry_run=False,
|
||||
finding=finding,
|
||||
)
|
||||
return IntegrityActionResult(
|
||||
action="retained_referenced",
|
||||
changed=False,
|
||||
dry_run=dry_run,
|
||||
finding=finding,
|
||||
)
|
||||
if finding.state == "deleted":
|
||||
return IntegrityActionResult(
|
||||
action="already_deleted",
|
||||
changed=False,
|
||||
dry_run=dry_run,
|
||||
finding=finding,
|
||||
)
|
||||
if dry_run:
|
||||
return IntegrityActionResult(
|
||||
action="would_delete",
|
||||
changed=False,
|
||||
dry_run=True,
|
||||
finding=finding,
|
||||
)
|
||||
active_backend = backend or get_storage_backend()
|
||||
if active_backend.name != scan.storage_backend:
|
||||
raise FileStorageError(
|
||||
"The configured storage backend does not match the integrity finding"
|
||||
)
|
||||
begin_orphan_cleanup_recovery(
|
||||
session,
|
||||
finding,
|
||||
backend=active_backend,
|
||||
user_id=user_id,
|
||||
)
|
||||
try:
|
||||
active_backend.stat(finding.storage_key)
|
||||
except StorageObjectMissing:
|
||||
action = "already_absent"
|
||||
else:
|
||||
active_backend.delete(finding.storage_key)
|
||||
action = "deleted"
|
||||
finding.state = "deleted"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.resolved_by_user_id = user_id
|
||||
finding.revision += 1
|
||||
session.add(finding)
|
||||
return IntegrityActionResult(
|
||||
action=action,
|
||||
changed=True,
|
||||
dry_run=False,
|
||||
finding=finding,
|
||||
)
|
||||
|
||||
|
||||
def _scan_blob_batch(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
backend: StorageBackend,
|
||||
) -> None:
|
||||
query = session.query(FileBlob).filter(
|
||||
FileBlob.tenant_id == scan.tenant_id,
|
||||
FileBlob.storage_backend == scan.storage_backend,
|
||||
)
|
||||
if scan.blob_cursor:
|
||||
query = query.filter(FileBlob.id > scan.blob_cursor)
|
||||
blobs = query.order_by(FileBlob.id.asc()).limit(scan.batch_size).all()
|
||||
for blob in blobs:
|
||||
inspection = inspect_blob(
|
||||
blob,
|
||||
backend=backend,
|
||||
verify_checksum=scan.verify_checksums,
|
||||
)
|
||||
apply_blob_inspection(session, blob, inspection)
|
||||
scan.scanned_blob_count += 1
|
||||
if inspection.valid:
|
||||
scan.verified_blob_count += 1
|
||||
_resolve_scan_blob_findings(session, scan, blob)
|
||||
else:
|
||||
scan.quarantined_blob_count += 1
|
||||
_record_blob_finding(session, scan, blob, inspection)
|
||||
scan.blob_cursor = blob.id
|
||||
if len(blobs) < scan.batch_size:
|
||||
scan.phase = "objects"
|
||||
scan.object_cursor = None
|
||||
|
||||
|
||||
def _scan_object_batch(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
backend: StorageBackend,
|
||||
) -> None:
|
||||
page = backend.list_objects(
|
||||
prefix=scan.storage_prefix,
|
||||
after=scan.object_cursor,
|
||||
limit=scan.batch_size,
|
||||
)
|
||||
keys = [item.key for item in page.objects]
|
||||
referenced_keys = {
|
||||
row[0]
|
||||
for row in session.query(FileBlob.storage_key)
|
||||
.filter(
|
||||
FileBlob.tenant_id == scan.tenant_id,
|
||||
FileBlob.storage_backend == scan.storage_backend,
|
||||
FileBlob.storage_key.in_(keys),
|
||||
)
|
||||
.all()
|
||||
} if keys else set()
|
||||
for item in page.objects:
|
||||
scan.scanned_object_count += 1
|
||||
if item.key in referenced_keys:
|
||||
continue
|
||||
scan.orphan_object_count += 1
|
||||
_record_orphan_finding(
|
||||
session,
|
||||
scan,
|
||||
storage_key=item.key,
|
||||
size_bytes=item.size_bytes,
|
||||
)
|
||||
scan.object_cursor = page.next_cursor
|
||||
if page.next_cursor is None:
|
||||
scan.phase = "completed"
|
||||
scan.status = "completed"
|
||||
scan.completed_at = utcnow()
|
||||
|
||||
|
||||
def _record_blob_finding(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
blob: FileBlob,
|
||||
inspection: BlobInspection,
|
||||
) -> FileIntegrityFinding:
|
||||
finding = (
|
||||
session.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.blob_id == blob.id,
|
||||
FileIntegrityFinding.kind == inspection.kind,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if finding is None:
|
||||
finding = FileIntegrityFinding(
|
||||
scan_id=scan.id,
|
||||
tenant_id=scan.tenant_id,
|
||||
kind=inspection.kind,
|
||||
blob_id=blob.id,
|
||||
storage_key=blob.storage_key,
|
||||
expected_size_bytes=blob.storage_size_bytes if blob.storage_size_bytes is not None else blob.size_bytes,
|
||||
expected_checksum_sha256=blob.storage_checksum_sha256 if blob.storage_checksum_sha256 is not None else blob.checksum_sha256,
|
||||
)
|
||||
else:
|
||||
finding.revision += 1
|
||||
_update_finding_from_inspection(finding, inspection)
|
||||
session.add(finding)
|
||||
return finding
|
||||
|
||||
|
||||
def _record_orphan_finding(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
*,
|
||||
storage_key: str,
|
||||
size_bytes: int,
|
||||
) -> FileIntegrityFinding:
|
||||
finding = (
|
||||
session.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.kind == "orphan_object",
|
||||
FileIntegrityFinding.storage_key == storage_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if finding is None:
|
||||
finding = FileIntegrityFinding(
|
||||
scan_id=scan.id,
|
||||
tenant_id=scan.tenant_id,
|
||||
kind="orphan_object",
|
||||
storage_key=storage_key,
|
||||
observed_size_bytes=size_bytes,
|
||||
)
|
||||
session.add(finding)
|
||||
return finding
|
||||
|
||||
|
||||
def _resolve_scan_blob_findings(
|
||||
session: Session,
|
||||
scan: FileIntegrityScan,
|
||||
blob: FileBlob,
|
||||
) -> None:
|
||||
for finding in (
|
||||
session.query(FileIntegrityFinding)
|
||||
.filter(
|
||||
FileIntegrityFinding.scan_id == scan.id,
|
||||
FileIntegrityFinding.blob_id == blob.id,
|
||||
FileIntegrityFinding.state == "open",
|
||||
)
|
||||
.all()
|
||||
):
|
||||
finding.state = "resolved"
|
||||
finding.resolved_at = utcnow()
|
||||
finding.revision += 1
|
||||
session.add(finding)
|
||||
|
||||
|
||||
def _update_finding_from_inspection(
|
||||
finding: FileIntegrityFinding,
|
||||
inspection: BlobInspection,
|
||||
) -> None:
|
||||
finding.observed_size_bytes = inspection.observed_size_bytes
|
||||
finding.observed_checksum_sha256 = inspection.observed_checksum_sha256
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BlobInspection",
|
||||
"IntegrityActionResult",
|
||||
"QUARANTINED_BLOB_STATUSES",
|
||||
"apply_blob_inspection",
|
||||
"cleanup_orphan_finding",
|
||||
"create_integrity_scan",
|
||||
"ensure_blob_is_readable",
|
||||
"inspect_blob",
|
||||
"mark_integrity_scan_failed",
|
||||
"read_verified_blob_bytes",
|
||||
"recheck_integrity_finding",
|
||||
"run_integrity_scan_batch",
|
||||
"storage_prefix_for_tenant",
|
||||
]
|
||||
@@ -0,0 +1,796 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Callable, Iterable
|
||||
|
||||
from sqlalchemy import func, inspect, or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import process_runtime_identity
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileFormEvidenceGrant,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackendError,
|
||||
get_storage_backend,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError, utcnow
|
||||
from govoplan_files.backend.storage.connector_spaces import connector_space_owner_id
|
||||
from govoplan_files.backend.storage.paths import normalize_folder
|
||||
from govoplan_files.backend.storage.share_state import effective_file_share_clause
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgePreviewItem:
|
||||
file_id: str
|
||||
filename: str
|
||||
lifecycle_revision: int
|
||||
deleted_at: datetime | None
|
||||
retained_until: datetime | None
|
||||
legal_hold: bool
|
||||
blockers: tuple[str, ...]
|
||||
blob_ids: tuple[str, ...]
|
||||
|
||||
def digest_payload(self) -> dict[str, object]:
|
||||
return {
|
||||
"file_id": self.file_id,
|
||||
"filename": self.filename,
|
||||
"lifecycle_revision": self.lifecycle_revision,
|
||||
"deleted_at": _iso(self.deleted_at),
|
||||
"retained_until": _iso(self.retained_until),
|
||||
"legal_hold": self.legal_hold,
|
||||
"blockers": list(self.blockers),
|
||||
"blob_ids": list(self.blob_ids),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgePreview:
|
||||
preview_sha256: str
|
||||
items: tuple[PurgePreviewItem, ...]
|
||||
|
||||
@property
|
||||
def eligible(self) -> bool:
|
||||
return all(not item.blockers for item in self.items)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PurgeResult:
|
||||
recovery_operation_id: str
|
||||
status: str
|
||||
replayed: bool
|
||||
purged_files: int
|
||||
released_blobs: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BlobGcResult:
|
||||
inspected_blobs: int
|
||||
deleted_blobs: int
|
||||
unresolved_operation_ids: tuple[str, ...]
|
||||
|
||||
|
||||
def set_asset_lifecycle(
|
||||
session: Session,
|
||||
asset: FileAsset,
|
||||
*,
|
||||
retained_until: datetime | None,
|
||||
legal_hold: bool,
|
||||
reason: str,
|
||||
expected_revision: int,
|
||||
) -> FileAsset:
|
||||
if asset.lifecycle_revision != expected_revision:
|
||||
raise FileStorageError("File lifecycle settings changed; reload before saving")
|
||||
asset.retained_until = retained_until
|
||||
asset.legal_hold = bool(legal_hold)
|
||||
asset.lifecycle_reason = reason.strip()
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return asset
|
||||
|
||||
|
||||
def get_asset_for_lifecycle(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
asset_id: str,
|
||||
is_admin: bool,
|
||||
) -> FileAsset:
|
||||
asset = session.get(FileAsset, asset_id)
|
||||
if asset is None or asset.tenant_id != tenant_id:
|
||||
raise FileStorageError("File not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_asset_owner_id(asset),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
return asset
|
||||
|
||||
|
||||
def restore_asset(session: Session, asset: FileAsset) -> bool:
|
||||
if asset.deleted_at is None:
|
||||
return False
|
||||
collision = _asset_owner_query(session, asset).filter(
|
||||
FileAsset.id != asset.id,
|
||||
FileAsset.display_path == asset.display_path,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
"The file cannot be restored because its path is already in use"
|
||||
)
|
||||
asset.deleted_at = None
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return True
|
||||
|
||||
|
||||
def restore_folder(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
owner_type: str,
|
||||
owner_id: str,
|
||||
user_id: str,
|
||||
path: str,
|
||||
recursive: bool,
|
||||
is_admin: bool,
|
||||
) -> tuple[int, int]:
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
normalized = normalize_folder(path)
|
||||
if not normalized:
|
||||
raise FileStorageError("Folder path is required")
|
||||
prefix = f"{normalized}/"
|
||||
folders = _folder_owner_query(
|
||||
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||
).filter(FileFolder.deleted_at.is_not(None))
|
||||
assets = _asset_owner_query_by_values(
|
||||
session, tenant_id=tenant_id, owner_type=owner_type, owner_id=owner_id
|
||||
).filter(FileAsset.deleted_at.is_not(None))
|
||||
if recursive:
|
||||
folders = folders.filter(
|
||||
(FileFolder.path == normalized) | FileFolder.path.like(f"{prefix}%")
|
||||
)
|
||||
assets = assets.filter(FileAsset.display_path.like(f"{prefix}%"))
|
||||
else:
|
||||
folders = folders.filter(FileFolder.path == normalized)
|
||||
assets = assets.filter(False)
|
||||
folder_rows = folders.order_by(FileFolder.path.asc()).all()
|
||||
asset_rows = assets.order_by(FileAsset.display_path.asc()).all()
|
||||
if not folder_rows:
|
||||
raise FileStorageError("Deleted folder not found")
|
||||
for folder in folder_rows:
|
||||
collision = _folder_owner_query(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
).filter(
|
||||
FileFolder.id != folder.id,
|
||||
FileFolder.path == folder.path,
|
||||
FileFolder.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
f"The folder cannot be restored because its path is in use: {folder.path}"
|
||||
)
|
||||
for asset in asset_rows:
|
||||
collision = _asset_owner_query(session, asset).filter(
|
||||
FileAsset.id != asset.id,
|
||||
FileAsset.display_path == asset.display_path,
|
||||
FileAsset.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
f"A file cannot be restored because its path is in use: {asset.display_path}"
|
||||
)
|
||||
for folder in folder_rows:
|
||||
folder.deleted_at = None
|
||||
session.add(folder)
|
||||
for asset in asset_rows:
|
||||
asset.deleted_at = None
|
||||
asset.lifecycle_revision += 1
|
||||
session.add(asset)
|
||||
session.flush()
|
||||
return len(folder_rows), len(asset_rows)
|
||||
|
||||
|
||||
def restore_connector_space(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
user_id: str,
|
||||
space_id: str,
|
||||
is_admin: bool,
|
||||
) -> FileConnectorSpace:
|
||||
space = session.get(FileConnectorSpace, space_id)
|
||||
if space is None or space.tenant_id != tenant_id or space.deleted_at is None:
|
||||
raise FileStorageError("Deleted connector space not found")
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=space.owner_type,
|
||||
owner_id=connector_space_owner_id(space),
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
collision = _connector_space_owner_query(session, space).filter(
|
||||
FileConnectorSpace.id != space.id,
|
||||
FileConnectorSpace.label == space.label,
|
||||
FileConnectorSpace.deleted_at.is_(None),
|
||||
).first()
|
||||
if collision is not None:
|
||||
raise FileStorageError(
|
||||
"The connector space cannot be restored because its label is in use"
|
||||
)
|
||||
space.deleted_at = None
|
||||
space.is_active = True
|
||||
session.add(space)
|
||||
session.flush()
|
||||
return space
|
||||
|
||||
|
||||
def preview_asset_purge(
|
||||
session: Session, *, tenant_id: str, file_ids: Iterable[str]
|
||||
) -> PurgePreview:
|
||||
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||
if not normalized_ids or len(normalized_ids) > 100:
|
||||
raise FileStorageError("Purge preview must contain between 1 and 100 files")
|
||||
assets = (
|
||||
session.query(FileAsset)
|
||||
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||
.order_by(FileAsset.id.asc())
|
||||
.all()
|
||||
)
|
||||
by_id = {asset.id: asset for asset in assets}
|
||||
items: list[PurgePreviewItem] = []
|
||||
for file_id in normalized_ids:
|
||||
asset = by_id.get(file_id)
|
||||
if asset is None:
|
||||
items.append(
|
||||
PurgePreviewItem(
|
||||
file_id=file_id,
|
||||
filename="",
|
||||
lifecycle_revision=0,
|
||||
deleted_at=None,
|
||||
retained_until=None,
|
||||
legal_hold=False,
|
||||
blockers=("not_found",),
|
||||
blob_ids=(),
|
||||
)
|
||||
)
|
||||
continue
|
||||
blockers = _asset_purge_blockers(session, asset)
|
||||
blob_ids = tuple(
|
||||
sorted(
|
||||
row[0]
|
||||
for row in session.query(FileVersion.blob_id)
|
||||
.filter(FileVersion.file_asset_id == asset.id)
|
||||
.distinct()
|
||||
.all()
|
||||
)
|
||||
)
|
||||
items.append(
|
||||
PurgePreviewItem(
|
||||
file_id=asset.id,
|
||||
filename=asset.filename,
|
||||
lifecycle_revision=asset.lifecycle_revision,
|
||||
deleted_at=asset.deleted_at,
|
||||
retained_until=asset.retained_until,
|
||||
legal_hold=asset.legal_hold,
|
||||
blockers=tuple(blockers),
|
||||
blob_ids=blob_ids,
|
||||
)
|
||||
)
|
||||
payload = [item.digest_payload() for item in items]
|
||||
digest = hashlib.sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
).hexdigest()
|
||||
return PurgePreview(preview_sha256=digest, items=tuple(items))
|
||||
|
||||
|
||||
def execute_asset_purge(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
file_ids: Iterable[str],
|
||||
preview_sha256: str,
|
||||
idempotency_key: str,
|
||||
approval_reference: str,
|
||||
before_commit: Callable[[list[FileAsset]], None] | None = None,
|
||||
) -> PurgeResult:
|
||||
normalized_ids = sorted(set(str(value).strip() for value in file_ids if str(value).strip()))
|
||||
if not normalized_ids or len(normalized_ids) > 100:
|
||||
raise FileStorageError("Purge must contain between 1 and 100 files")
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"file_ids": normalized_ids,
|
||||
"preview_sha256": preview_sha256,
|
||||
"approval_reference": approval_reference,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="asset-hard-purge",
|
||||
idempotency_key=f"files-asset-purge:{idempotency_key}",
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.IRREVERSIBLE,
|
||||
preconditions=(
|
||||
"the actor holds the dedicated Files purge permission",
|
||||
"the signed-off preview still matches current lifecycle state",
|
||||
"every target is soft-deleted and free of retention or legal-hold blockers",
|
||||
),
|
||||
verification_steps=(
|
||||
"verify every target FileAsset and FileVersion row is absent",
|
||||
"recalculate retained FileBlob reference counts before later garbage collection",
|
||||
),
|
||||
approval_reference=approval_reference,
|
||||
),
|
||||
precondition_evidence={
|
||||
"preview_sha256": preview_sha256,
|
||||
"target_count": len(normalized_ids),
|
||||
},
|
||||
lease_resource_key=f"files:purge:{tenant_id}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_asset_batch",
|
||||
resource_id=preview_sha256,
|
||||
metadata={"resources": ["postgresql"], "bounded_target_count": len(normalized_ids)},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"Another purge or unresolved retry owns the tenant purge fence"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError("The Files recovery ledger is unavailable; nothing was purged") from exc
|
||||
if started.replayed or started.operation is None:
|
||||
return PurgeResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=started.status,
|
||||
replayed=True,
|
||||
purged_files=0,
|
||||
released_blobs=0,
|
||||
)
|
||||
operation = started.operation
|
||||
try:
|
||||
preview = preview_asset_purge(session, tenant_id=tenant_id, file_ids=normalized_ids)
|
||||
if preview.preview_sha256 != preview_sha256:
|
||||
operation.reject(
|
||||
summary="The purge preview became stale before execution",
|
||||
evidence=_verified_evidence(
|
||||
{"preview_matches": False},
|
||||
expected_preview_sha256=preview_sha256,
|
||||
observed_preview_sha256=preview.preview_sha256,
|
||||
),
|
||||
)
|
||||
raise FileStorageError("The purge preview is stale; preview again")
|
||||
if not preview.eligible:
|
||||
operation.reject(
|
||||
summary="The purge was blocked by current lifecycle policy",
|
||||
evidence=_verified_evidence(
|
||||
{"lifecycle_policy_allows": False},
|
||||
blocked_files=[
|
||||
{"file_id": item.file_id, "blockers": list(item.blockers)}
|
||||
for item in preview.items
|
||||
if item.blockers
|
||||
]
|
||||
),
|
||||
)
|
||||
raise FileStorageError("One or more files are not eligible for purge")
|
||||
assets = (
|
||||
session.query(FileAsset)
|
||||
.filter(FileAsset.tenant_id == tenant_id, FileAsset.id.in_(normalized_ids))
|
||||
.with_for_update()
|
||||
.all()
|
||||
)
|
||||
blob_ids = sorted({blob_id for item in preview.items for blob_id in item.blob_ids})
|
||||
if before_commit is not None:
|
||||
before_commit(assets)
|
||||
session.query(FileShare).filter(
|
||||
FileShare.file_asset_id.in_(normalized_ids)
|
||||
).delete(synchronize_session=False)
|
||||
session.query(FileVersion).filter(
|
||||
FileVersion.file_asset_id.in_(normalized_ids)
|
||||
).delete(synchronize_session=False)
|
||||
for asset in assets:
|
||||
session.delete(asset)
|
||||
session.flush()
|
||||
released_blobs = 0
|
||||
for blob_id in blob_ids:
|
||||
references = int(
|
||||
session.query(func.count(FileVersion.id))
|
||||
.filter(FileVersion.blob_id == blob_id)
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
blob = session.get(FileBlob, blob_id)
|
||||
if blob is not None:
|
||||
blob.ref_count = references
|
||||
session.add(blob)
|
||||
if references == 0:
|
||||
released_blobs += 1
|
||||
operation.commit_verified_success(
|
||||
session,
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"assets_absent": True,
|
||||
"versions_absent": True,
|
||||
"blob_reference_counts_recalculated": True,
|
||||
},
|
||||
purged_file_ids=normalized_ids,
|
||||
purged_files=len(assets),
|
||||
released_blobs=released_blobs,
|
||||
preview_sha256=preview_sha256,
|
||||
),
|
||||
)
|
||||
return PurgeResult(
|
||||
recovery_operation_id=started.operation_id,
|
||||
status=RecoveryStatus.SUCCEEDED.value,
|
||||
replayed=False,
|
||||
purged_files=len(assets),
|
||||
released_blobs=released_blobs,
|
||||
)
|
||||
except FileStorageError:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.release_unresolved()
|
||||
raise
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.reject(
|
||||
summary="The purge failed before its database transaction committed",
|
||||
evidence=_verified_evidence(
|
||||
{"database_transaction_committed": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
def garbage_collect_unreferenced_blobs(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
limit: int,
|
||||
approval_reference: str,
|
||||
) -> BlobGcResult:
|
||||
candidate_rows = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.tenant_id == tenant_id)
|
||||
.filter(
|
||||
or_(FileBlob.retained_until.is_(None), FileBlob.retained_until <= utcnow())
|
||||
)
|
||||
.filter(
|
||||
~session.query(FileVersion.id)
|
||||
.filter(FileVersion.blob_id == FileBlob.id)
|
||||
.exists()
|
||||
)
|
||||
.order_by(FileBlob.created_at.asc(), FileBlob.id.asc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
candidates = tuple(
|
||||
(row.id, row.storage_key) for row in candidate_rows
|
||||
)
|
||||
# Close the read snapshot before the independent recovery transaction takes
|
||||
# its durable SQLite/PostgreSQL write lock.
|
||||
session.commit()
|
||||
deleted = 0
|
||||
unresolved: list[str] = []
|
||||
backend = get_storage_backend()
|
||||
for candidate_id, storage_key in candidates:
|
||||
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"blob_id": candidate_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"approval_reference": approval_reference,
|
||||
}
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="blob-garbage-collection",
|
||||
idempotency_key=(
|
||||
f"files-blob-gc:{candidate_id}:{key_digest[:12]}:"
|
||||
f"{hashlib.sha256(approval_reference.encode('utf-8')).hexdigest()[:12]}"
|
||||
),
|
||||
request=request,
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the caller holds dedicated Files purge authority",
|
||||
"a fresh database check shows no FileVersion reference",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"verify the object is absent",
|
||||
"delete the unreferenced FileBlob row after absence is proven",
|
||||
),
|
||||
verification_steps=(
|
||||
"recheck FileVersion references while holding the blob fence",
|
||||
"probe the exact managed storage key after deletion",
|
||||
),
|
||||
approval_reference=approval_reference,
|
||||
),
|
||||
precondition_evidence={
|
||||
"blob_id": candidate_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"observed_reference_count": 0,
|
||||
},
|
||||
lease_resource_key=f"files:blob:{tenant_id}:{candidate_id}",
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=candidate_id,
|
||||
metadata={"resources": ["postgresql", "object-storage"], "storage_backend": backend.name},
|
||||
block_unresolved_resource=True,
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict):
|
||||
continue
|
||||
except (RecoveryGuaranteeError, RuntimeError, ValueError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no blob bytes were deleted"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
continue
|
||||
operation = started.operation
|
||||
try:
|
||||
blob = (
|
||||
session.query(FileBlob)
|
||||
.filter(FileBlob.id == candidate_id, FileBlob.tenant_id == tenant_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
references = int(
|
||||
session.query(func.count(FileVersion.id))
|
||||
.filter(FileVersion.blob_id == candidate_id)
|
||||
.scalar()
|
||||
or 0
|
||||
)
|
||||
blob_retained = bool(
|
||||
blob is not None
|
||||
and blob.retained_until is not None
|
||||
and _as_utc(blob.retained_until) > utcnow()
|
||||
)
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.reject(
|
||||
summary="Blob eligibility could not be rechecked before deletion",
|
||||
evidence=_verified_evidence(
|
||||
{"external_effect_started": False},
|
||||
exception_type=type(exc).__name__,
|
||||
),
|
||||
)
|
||||
raise FileStorageError(
|
||||
"Blob eligibility could not be rechecked; no bytes were deleted"
|
||||
) from exc
|
||||
if blob is None or references or blob_retained:
|
||||
operation.reject(
|
||||
summary="Blob garbage collection was no longer eligible",
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"blob_present": blob is not None,
|
||||
"reference_count_zero": references == 0,
|
||||
"blob_retention_expired": not blob_retained,
|
||||
},
|
||||
reference_count=references,
|
||||
),
|
||||
)
|
||||
session.rollback()
|
||||
continue
|
||||
try:
|
||||
backend.delete(blob.storage_key)
|
||||
object_present = backend.exists(blob.storage_key)
|
||||
except (StorageBackendError, OSError) as exc:
|
||||
try:
|
||||
object_present = backend.exists(blob.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
object_present = None
|
||||
session.rollback()
|
||||
if object_present is True:
|
||||
operation.fail(
|
||||
summary="The unreferenced blob object was retained",
|
||||
evidence={"object_present": True, "exception_type": type(exc).__name__},
|
||||
)
|
||||
else:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The blob deletion outcome could not be verified",
|
||||
evidence={"object_present": object_present, "exception_type": type(exc).__name__},
|
||||
failure_summary="Reconcile the exact blob key before retrying garbage collection",
|
||||
)
|
||||
unresolved.append(started.operation_id)
|
||||
continue
|
||||
if object_present:
|
||||
session.rollback()
|
||||
operation.fail(
|
||||
summary="The object store did not delete the unreferenced blob",
|
||||
evidence={"object_present": True},
|
||||
)
|
||||
continue
|
||||
try:
|
||||
session.delete(blob)
|
||||
session.flush()
|
||||
operation.commit_verified_success(
|
||||
session,
|
||||
evidence=_verified_evidence(
|
||||
{
|
||||
"object_absent": True,
|
||||
"database_blob_absent": True,
|
||||
"reference_count_zero": True,
|
||||
},
|
||||
blob_id=candidate_id,
|
||||
storage_key_sha256=key_digest,
|
||||
object_present=False,
|
||||
database_blob_present=False,
|
||||
reference_count=0,
|
||||
),
|
||||
)
|
||||
deleted += 1
|
||||
except Exception as exc:
|
||||
session.rollback()
|
||||
if not operation.closed:
|
||||
operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="The object is absent but blob metadata still requires forward recovery",
|
||||
evidence={"object_present": False, "exception_type": type(exc).__name__},
|
||||
failure_summary="Delete the FileBlob row only after rechecking all FileVersion references",
|
||||
)
|
||||
unresolved.append(started.operation_id)
|
||||
return BlobGcResult(
|
||||
inspected_blobs=len(candidates),
|
||||
deleted_blobs=deleted,
|
||||
unresolved_operation_ids=tuple(unresolved),
|
||||
)
|
||||
|
||||
|
||||
def _asset_purge_blockers(session: Session, asset: FileAsset) -> list[str]:
|
||||
blockers: list[str] = []
|
||||
if asset.deleted_at is None:
|
||||
blockers.append("not_soft_deleted")
|
||||
if asset.legal_hold:
|
||||
blockers.append("legal_hold")
|
||||
if asset.retained_until is not None and _as_utc(asset.retained_until) > utcnow():
|
||||
blockers.append("retention_active")
|
||||
if _table_exists(session, CampaignAttachmentUse.__tablename__) and session.query(
|
||||
CampaignAttachmentUse.id
|
||||
).filter(CampaignAttachmentUse.file_asset_id == asset.id).first() is not None:
|
||||
blockers.append("campaign_evidence")
|
||||
if _table_exists(session, FileFormEvidenceGrant.__tablename__) and session.query(
|
||||
FileFormEvidenceGrant.id
|
||||
).filter(FileFormEvidenceGrant.file_asset_id == asset.id).first() is not None:
|
||||
blockers.append("form_evidence")
|
||||
if session.query(FileShare.id).filter(
|
||||
FileShare.file_asset_id == asset.id,
|
||||
effective_file_share_clause(),
|
||||
).first() is not None:
|
||||
blockers.append("active_share")
|
||||
return blockers
|
||||
|
||||
|
||||
def _asset_owner_id(asset: FileAsset) -> str:
|
||||
owner_id = asset.owner_user_id if asset.owner_type == "user" else asset.owner_group_id
|
||||
if not owner_id:
|
||||
raise FileStorageError("File has no valid owner")
|
||||
return owner_id
|
||||
|
||||
|
||||
def _asset_owner_query(session: Session, asset: FileAsset):
|
||||
return _asset_owner_query_by_values(
|
||||
session,
|
||||
tenant_id=asset.tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=_asset_owner_id(asset),
|
||||
)
|
||||
|
||||
|
||||
def _asset_owner_query_by_values(
|
||||
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||
):
|
||||
query = session.query(FileAsset).filter(
|
||||
FileAsset.tenant_id == tenant_id, FileAsset.owner_type == owner_type
|
||||
)
|
||||
if owner_type == "user":
|
||||
return query.filter(FileAsset.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileAsset.owner_group_id == owner_id)
|
||||
raise FileStorageError("Files must be owned by a user or group")
|
||||
|
||||
|
||||
def _folder_owner_query(
|
||||
session: Session, *, tenant_id: str, owner_type: str, owner_id: str
|
||||
):
|
||||
query = session.query(FileFolder).filter(
|
||||
FileFolder.tenant_id == tenant_id, FileFolder.owner_type == owner_type
|
||||
)
|
||||
if owner_type == "user":
|
||||
return query.filter(FileFolder.owner_user_id == owner_id)
|
||||
if owner_type == "group":
|
||||
return query.filter(FileFolder.owner_group_id == owner_id)
|
||||
raise FileStorageError("Folders must be owned by a user or group")
|
||||
|
||||
|
||||
def _connector_space_owner_query(session: Session, space: FileConnectorSpace):
|
||||
query = session.query(FileConnectorSpace).filter(
|
||||
FileConnectorSpace.tenant_id == space.tenant_id,
|
||||
FileConnectorSpace.owner_type == space.owner_type,
|
||||
)
|
||||
owner_id = connector_space_owner_id(space)
|
||||
if space.owner_type == "user":
|
||||
return query.filter(FileConnectorSpace.owner_user_id == owner_id)
|
||||
return query.filter(FileConnectorSpace.owner_group_id == owner_id)
|
||||
|
||||
|
||||
def _as_utc(value: datetime) -> datetime:
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
return _as_utc(value).isoformat() if value is not None else None
|
||||
|
||||
|
||||
def _table_exists(session: Session, table_name: str) -> bool:
|
||||
return bool(session.bind is not None and inspect(session.bind).has_table(table_name))
|
||||
|
||||
|
||||
def _verified_evidence(
|
||||
checks: dict[str, object], **details: object
|
||||
) -> dict[str, object]:
|
||||
return {"verified": True, "checks": checks, **details}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BlobGcResult",
|
||||
"PurgePreview",
|
||||
"PurgePreviewItem",
|
||||
"PurgeResult",
|
||||
"execute_asset_purge",
|
||||
"garbage_collect_unreferenced_blobs",
|
||||
"get_asset_for_lifecycle",
|
||||
"preview_asset_purge",
|
||||
"restore_asset",
|
||||
"restore_connector_space",
|
||||
"restore_folder",
|
||||
"set_asset_lifecycle",
|
||||
]
|
||||
@@ -0,0 +1,90 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
|
||||
SOURCE_PROVENANCE_METADATA_KEY = "source_provenance"
|
||||
SOURCE_REVISION_METADATA_KEY = "source_revision"
|
||||
|
||||
_PROVENANCE_STRING_FIELDS = {
|
||||
"source_type",
|
||||
"connector_id",
|
||||
"provider",
|
||||
"external_id",
|
||||
"external_path",
|
||||
"external_url",
|
||||
"revision",
|
||||
"revision_label",
|
||||
"observed_at",
|
||||
"imported_at",
|
||||
}
|
||||
|
||||
|
||||
def normalize_source_revision(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
|
||||
def normalize_source_provenance(value: object, *, source_revision: object = None) -> dict[str, Any] | None:
|
||||
if value is None:
|
||||
revision = normalize_source_revision(source_revision)
|
||||
return {"revision": revision} if revision else None
|
||||
if not isinstance(value, Mapping):
|
||||
raise ValueError("source_provenance must be a JSON object")
|
||||
|
||||
payload: dict[str, Any] = {}
|
||||
for field in _PROVENANCE_STRING_FIELDS:
|
||||
normalized = normalize_source_revision(value.get(field))
|
||||
if normalized:
|
||||
payload[field] = normalized
|
||||
|
||||
extra = value.get("metadata")
|
||||
if isinstance(extra, Mapping) and extra:
|
||||
payload["metadata"] = dict(extra)
|
||||
elif extra is not None and extra != "":
|
||||
raise ValueError("source_provenance.metadata must be a JSON object")
|
||||
|
||||
revision = normalize_source_revision(source_revision)
|
||||
if revision:
|
||||
payload["revision"] = revision
|
||||
return payload or None
|
||||
|
||||
|
||||
def source_metadata(
|
||||
*,
|
||||
existing: Mapping[str, Any] | None = None,
|
||||
source_provenance: object = None,
|
||||
source_revision: object = None,
|
||||
) -> dict[str, Any] | None:
|
||||
payload = dict(existing or {})
|
||||
normalized_provenance = normalize_source_provenance(source_provenance, source_revision=source_revision)
|
||||
normalized_revision = normalize_source_revision(source_revision)
|
||||
if normalized_provenance:
|
||||
payload[SOURCE_PROVENANCE_METADATA_KEY] = normalized_provenance
|
||||
if normalized_revision:
|
||||
payload[SOURCE_REVISION_METADATA_KEY] = normalized_revision
|
||||
elif normalized_provenance and normalized_provenance.get("revision"):
|
||||
payload[SOURCE_REVISION_METADATA_KEY] = str(normalized_provenance["revision"])
|
||||
return payload or None
|
||||
|
||||
|
||||
def source_provenance_from_metadata(metadata: Mapping[str, Any] | None) -> dict[str, Any] | None:
|
||||
if not isinstance(metadata, Mapping):
|
||||
return None
|
||||
value = metadata.get(SOURCE_PROVENANCE_METADATA_KEY)
|
||||
if isinstance(value, Mapping):
|
||||
return dict(value)
|
||||
return None
|
||||
|
||||
|
||||
def source_revision_from_metadata(metadata: Mapping[str, Any] | None) -> str | None:
|
||||
if not isinstance(metadata, Mapping):
|
||||
return None
|
||||
revision = normalize_source_revision(metadata.get(SOURCE_REVISION_METADATA_KEY))
|
||||
if revision:
|
||||
return revision
|
||||
provenance = source_provenance_from_metadata(metadata)
|
||||
return normalize_source_revision(provenance.get("revision") if provenance else None)
|
||||
@@ -0,0 +1,931 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
from threading import Lock
|
||||
from typing import Protocol
|
||||
|
||||
from sqlalchemy import event, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryGuaranteeError,
|
||||
RecoveryMode,
|
||||
RecoveryOperation,
|
||||
RecoveryPlan,
|
||||
RecoveryStatus,
|
||||
plan_recovery_operation,
|
||||
prepare_recovery_operation,
|
||||
start_recovery_operation,
|
||||
verify_recovery_evidence_chain,
|
||||
)
|
||||
from govoplan_core.core.recovery_runtime import (
|
||||
DurableRecoveryOperation,
|
||||
DurableRecoveryStart,
|
||||
RecoveryOperationBusy,
|
||||
RecoveryOperationStateConflict,
|
||||
begin_durable_recovery_operation,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import (
|
||||
RuntimeIdentity,
|
||||
acquire_lease,
|
||||
process_runtime_identity,
|
||||
release_lease,
|
||||
)
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_files.backend.db.models import (
|
||||
FileBlob,
|
||||
FileIntegrityFinding,
|
||||
FileIntegrityScan,
|
||||
)
|
||||
from govoplan_files.backend.storage.backends import (
|
||||
StorageBackend,
|
||||
StorageBackendError,
|
||||
StorageObjectMissing,
|
||||
)
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
|
||||
|
||||
_PENDING_EFFECTS_KEY = "govoplan_files_pending_recovery_effects"
|
||||
_HOOKS_INSTALLED_KEY = "govoplan_files_recovery_hooks_installed"
|
||||
_ROLLBACK_ERRORS_KEY = "govoplan_files_recovery_rollback_errors"
|
||||
_SQLITE_FENCES: set[str] = set()
|
||||
_SQLITE_FENCES_LOCK = Lock()
|
||||
|
||||
|
||||
class _PendingEffect(Protocol):
|
||||
def settle(self, *, committed: bool) -> None: ...
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PendingBlobWrite:
|
||||
operation: DurableRecoveryOperation
|
||||
backend: StorageBackend
|
||||
tenant_id: str
|
||||
blob_id: str
|
||||
storage_key: str
|
||||
semantic_checksum_sha256: str
|
||||
semantic_size_bytes: int
|
||||
protection_discriminator: str
|
||||
created_new: bool
|
||||
expected_storage_checksum_sha256: str | None = None
|
||||
expected_storage_size_bytes: int | None = None
|
||||
expected_envelope_id: str | None = None
|
||||
sqlite_fence_key: str | None = None
|
||||
restart_after_rollback: Callable[[], DurableRecoveryOperation] | None = None
|
||||
|
||||
def prepare_stored_bytes(
|
||||
self,
|
||||
data: bytes,
|
||||
*,
|
||||
envelope_id: str | None,
|
||||
) -> None:
|
||||
"""Retain process-local verification evidence without recording content."""
|
||||
|
||||
self.expected_storage_checksum_sha256 = hashlib.sha256(data).hexdigest()
|
||||
self.expected_storage_size_bytes = len(data)
|
||||
self.expected_envelope_id = envelope_id
|
||||
|
||||
def settle(self, *, committed: bool) -> None:
|
||||
if not committed and self.restart_after_rollback is not None:
|
||||
try:
|
||||
self.operation = self.restart_after_rollback()
|
||||
except Exception:
|
||||
# SQLite cannot make the pre-effect ledger row independent of
|
||||
# the caller transaction. If reconstructing the evidence row
|
||||
# also fails, still avoid leaving a known new orphan behind.
|
||||
if self.created_new:
|
||||
try:
|
||||
self.backend.delete(self.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
pass
|
||||
raise
|
||||
evidence = _blob_write_evidence(self)
|
||||
if _blob_write_complete(evidence):
|
||||
self.operation.succeed(evidence=evidence)
|
||||
return
|
||||
|
||||
if self.created_new and evidence.get("database_blob_present") is False:
|
||||
self._settle_unreferenced_new_object(evidence)
|
||||
return
|
||||
|
||||
if not self.created_new and _object_matches(evidence):
|
||||
if _forward_complete_blob_repair(self):
|
||||
completed = _blob_write_evidence(self)
|
||||
if _blob_write_complete(completed):
|
||||
self.operation.succeed(evidence=completed)
|
||||
return
|
||||
|
||||
if evidence.get("database_blob_present") is True and not _object_matches(
|
||||
evidence
|
||||
):
|
||||
_quarantine_blob_after_failed_verification(self, evidence)
|
||||
evidence = _blob_write_evidence(self)
|
||||
|
||||
status = (
|
||||
RecoveryStatus.OUTCOME_UNKNOWN
|
||||
if not evidence.get("verified")
|
||||
else RecoveryStatus.RECOVERY_REQUIRED
|
||||
)
|
||||
transaction = "committed" if committed else "rolled back"
|
||||
self.operation.unresolved(
|
||||
status=status,
|
||||
summary=f"Managed Files blob write remained unresolved after the database transaction {transaction}",
|
||||
evidence=evidence,
|
||||
failure_summary=(
|
||||
"The managed object and Files blob metadata require reconciliation"
|
||||
),
|
||||
)
|
||||
|
||||
def _settle_unreferenced_new_object(self, evidence: dict[str, object]) -> None:
|
||||
object_present = evidence.get("object_present")
|
||||
if object_present is False:
|
||||
self.operation.reject(
|
||||
summary="The Files blob write left no durable object or database row",
|
||||
evidence=evidence,
|
||||
)
|
||||
return
|
||||
if object_present is not True:
|
||||
self.operation.unresolved(
|
||||
status=RecoveryStatus.OUTCOME_UNKNOWN,
|
||||
summary="The unreferenced Files object could not be probed",
|
||||
evidence=evidence,
|
||||
failure_summary="Object storage availability prevented upload compensation",
|
||||
)
|
||||
return
|
||||
try:
|
||||
self.backend.delete(self.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
self.operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="An unreferenced Files object could not be compensated",
|
||||
evidence=evidence,
|
||||
failure_summary="Delete the unreferenced managed object after verifying that no FileBlob references it",
|
||||
)
|
||||
return
|
||||
recovered = _blob_write_evidence(self)
|
||||
if (
|
||||
recovered.get("verified") is True
|
||||
and recovered.get("database_blob_present") is False
|
||||
and recovered.get("object_present") is False
|
||||
):
|
||||
self.operation.compensate(
|
||||
failure_summary="The Files database transaction did not retain the new blob",
|
||||
failure_evidence=evidence,
|
||||
recovery_evidence=recovered,
|
||||
)
|
||||
return
|
||||
self.operation.unresolved(
|
||||
status=RecoveryStatus.RECOVERY_REQUIRED,
|
||||
summary="Files upload compensation could not be verified",
|
||||
evidence=recovered,
|
||||
failure_summary="The unreferenced managed object requires operator reconciliation",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PendingOrphanCleanup:
|
||||
operation: DurableRecoveryOperation
|
||||
backend: StorageBackend
|
||||
finding_id: str
|
||||
tenant_id: str
|
||||
storage_key: str
|
||||
resolved_by_user_id: str
|
||||
|
||||
def settle(self, *, committed: bool) -> None:
|
||||
del committed
|
||||
evidence = _orphan_cleanup_evidence(self)
|
||||
if _orphan_cleanup_complete(evidence):
|
||||
self.operation.succeed(evidence=evidence)
|
||||
return
|
||||
if (
|
||||
evidence.get("verified") is True
|
||||
and evidence.get("object_present") is True
|
||||
and evidence.get("finding_deleted") is False
|
||||
):
|
||||
self.operation.reject(
|
||||
summary="The orphan object was retained and the cleanup finding stayed open",
|
||||
evidence=evidence,
|
||||
)
|
||||
return
|
||||
if evidence.get("object_present") is False:
|
||||
if _forward_complete_orphan_finding(self):
|
||||
completed = _orphan_cleanup_evidence(self)
|
||||
if _orphan_cleanup_complete(completed):
|
||||
self.operation.succeed(evidence=completed)
|
||||
return
|
||||
status = (
|
||||
RecoveryStatus.OUTCOME_UNKNOWN
|
||||
if not evidence.get("verified")
|
||||
else RecoveryStatus.RECOVERY_REQUIRED
|
||||
)
|
||||
self.operation.unresolved(
|
||||
status=status,
|
||||
summary="Files orphan cleanup requires reconciliation",
|
||||
evidence=evidence,
|
||||
failure_summary="Recheck the object and integrity-finding state before another cleanup attempt",
|
||||
)
|
||||
|
||||
|
||||
def begin_blob_write_recovery(
|
||||
session: Session,
|
||||
*,
|
||||
backend: StorageBackend,
|
||||
tenant_id: str,
|
||||
blob_id: str,
|
||||
storage_key: str,
|
||||
semantic_checksum_sha256: str,
|
||||
semantic_size_bytes: int,
|
||||
protection_discriminator: str,
|
||||
created_new: bool,
|
||||
repair_token: str | None = None,
|
||||
) -> PendingBlobWrite:
|
||||
disposition = "create" if created_new else "repair"
|
||||
state_token = repair_token or blob_id
|
||||
key_digest = hashlib.sha256(storage_key.encode("utf-8")).hexdigest()
|
||||
idempotency_key = (
|
||||
f"files-blob-{disposition}:{blob_id}:{state_token[:48]}"
|
||||
)
|
||||
try:
|
||||
identity = process_runtime_identity()
|
||||
except RuntimeError as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no object was written"
|
||||
) from exc
|
||||
lease_resource_key = f"files:blob:{tenant_id}:{blob_id}"
|
||||
request = {
|
||||
"tenant_id": tenant_id,
|
||||
"blob_id": blob_id,
|
||||
"storage_key": storage_key if created_new else None,
|
||||
"storage_key_sha256": key_digest,
|
||||
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||
"semantic_size_bytes": semantic_size_bytes,
|
||||
"protection_discriminator": protection_discriminator,
|
||||
"disposition": disposition,
|
||||
}
|
||||
recovery_plan = RecoveryPlan(
|
||||
mode=(
|
||||
RecoveryMode.COMPENSATION
|
||||
if created_new
|
||||
else RecoveryMode.FORWARD_RECOVERY
|
||||
),
|
||||
preconditions=(
|
||||
"the caller has Files write authority for the target owner",
|
||||
"the storage key belongs to the tenant Files namespace",
|
||||
"the request records digests rather than file contents",
|
||||
),
|
||||
compensation_steps=(
|
||||
"verify that no FileBlob references the newly reserved key",
|
||||
"delete only that unreferenced key and verify absence",
|
||||
)
|
||||
if created_new
|
||||
else (),
|
||||
forward_recovery_steps=(
|
||||
"verify the expected bytes at the existing blob key",
|
||||
"forward-complete matching integrity metadata or quarantine the blob",
|
||||
)
|
||||
if not created_new
|
||||
else (),
|
||||
verification_steps=(
|
||||
"reload FileBlob metadata through an independent session",
|
||||
"stream and hash the managed object independently",
|
||||
),
|
||||
)
|
||||
precondition_evidence = {
|
||||
"blob_id": blob_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"semantic_checksum_sha256": semantic_checksum_sha256,
|
||||
"semantic_size_bytes": semantic_size_bytes,
|
||||
"created_new": created_new,
|
||||
}
|
||||
metadata = {
|
||||
"resources": ["postgresql", "object-storage"],
|
||||
"storage_backend": backend.name,
|
||||
"durability_mode": "independent_transaction",
|
||||
}
|
||||
session_factory = get_database().SessionLocal
|
||||
sqlite_mode = session.get_bind().dialect.name == "sqlite"
|
||||
sqlite_fence_key: str | None = None
|
||||
|
||||
def start_independent(*, reconstructed_after_rollback: bool = False) -> DurableRecoveryStart:
|
||||
return begin_durable_recovery_operation(
|
||||
session_factory,
|
||||
identity=identity,
|
||||
module_id="files",
|
||||
operation_type=f"blob-{disposition}",
|
||||
idempotency_key=idempotency_key,
|
||||
request=request,
|
||||
recovery_plan=recovery_plan,
|
||||
precondition_evidence={
|
||||
**precondition_evidence,
|
||||
**(
|
||||
{
|
||||
"sqlite_caller_transaction_rolled_back": True,
|
||||
"effect_may_have_preceded_durable_intent": True,
|
||||
}
|
||||
if reconstructed_after_rollback
|
||||
else {}
|
||||
),
|
||||
},
|
||||
lease_resource_key=lease_resource_key,
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
metadata={
|
||||
**metadata,
|
||||
"durability_mode": (
|
||||
"sqlite_post_rollback_reconstruction"
|
||||
if reconstructed_after_rollback
|
||||
else "independent_transaction"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
if sqlite_mode:
|
||||
_reserve_sqlite_fence(lease_resource_key)
|
||||
sqlite_fence_key = lease_resource_key
|
||||
started = _begin_caller_transaction_recovery_operation(
|
||||
session,
|
||||
session_factory=session_factory,
|
||||
identity=identity,
|
||||
module_id="files",
|
||||
operation_type=f"blob-{disposition}",
|
||||
idempotency_key=idempotency_key,
|
||||
request=request,
|
||||
recovery_plan=recovery_plan,
|
||||
precondition_evidence={
|
||||
**precondition_evidence,
|
||||
"sqlite_caller_transaction": True,
|
||||
"reduced_crash_durability": True,
|
||||
},
|
||||
lease_resource_key=lease_resource_key,
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_blob",
|
||||
resource_id=blob_id,
|
||||
metadata={
|
||||
**metadata,
|
||||
"resources": ["sqlite", "object-storage"],
|
||||
"durability_mode": "sqlite_caller_transaction",
|
||||
},
|
||||
)
|
||||
else:
|
||||
started = start_independent()
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
if sqlite_fence_key is not None:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
raise FileStorageError(
|
||||
"This managed blob is already owned by another recovery operation"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||
if sqlite_fence_key is not None:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no object was written"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
if sqlite_fence_key is not None:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
raise FileStorageError(
|
||||
"The matching Files blob operation was already completed; reload before retrying"
|
||||
)
|
||||
pending = PendingBlobWrite(
|
||||
operation=started.operation,
|
||||
backend=backend,
|
||||
tenant_id=tenant_id,
|
||||
blob_id=blob_id,
|
||||
storage_key=storage_key,
|
||||
semantic_checksum_sha256=semantic_checksum_sha256,
|
||||
semantic_size_bytes=semantic_size_bytes,
|
||||
protection_discriminator=protection_discriminator,
|
||||
created_new=created_new,
|
||||
sqlite_fence_key=sqlite_fence_key,
|
||||
restart_after_rollback=(
|
||||
lambda: _require_started_operation(
|
||||
start_independent(reconstructed_after_rollback=True)
|
||||
)
|
||||
if sqlite_mode
|
||||
else None
|
||||
),
|
||||
)
|
||||
_register_pending_effect(session, pending)
|
||||
return pending
|
||||
|
||||
|
||||
def begin_orphan_cleanup_recovery(
|
||||
session: Session,
|
||||
finding: FileIntegrityFinding,
|
||||
*,
|
||||
backend: StorageBackend,
|
||||
user_id: str,
|
||||
) -> PendingOrphanCleanup:
|
||||
key_digest = hashlib.sha256(finding.storage_key.encode("utf-8")).hexdigest()
|
||||
try:
|
||||
started = begin_durable_recovery_operation(
|
||||
get_database().SessionLocal,
|
||||
identity=process_runtime_identity(),
|
||||
module_id="files",
|
||||
operation_type="integrity-orphan-cleanup",
|
||||
idempotency_key=f"files-orphan-cleanup:{finding.id}",
|
||||
request={
|
||||
"tenant_id": finding.tenant_id,
|
||||
"finding_id": finding.id,
|
||||
"storage_key_sha256": key_digest,
|
||||
},
|
||||
recovery_plan=RecoveryPlan(
|
||||
mode=RecoveryMode.FORWARD_RECOVERY,
|
||||
preconditions=(
|
||||
"the integrity finding identifies an unreferenced object",
|
||||
"the object key remains inside the completed scan scope",
|
||||
"a fresh database reference check found no FileBlob",
|
||||
),
|
||||
forward_recovery_steps=(
|
||||
"verify object absence independently",
|
||||
"mark the durable finding deleted only after absence is proven",
|
||||
),
|
||||
verification_steps=(
|
||||
"reload the finding and its scan through an independent session",
|
||||
"probe the original storage key through the configured backend",
|
||||
),
|
||||
),
|
||||
precondition_evidence={
|
||||
"finding_id": finding.id,
|
||||
"scan_id": finding.scan_id,
|
||||
"storage_key_sha256": key_digest,
|
||||
"finding_state": finding.state,
|
||||
},
|
||||
lease_resource_key=(
|
||||
f"files:orphan-cleanup:{finding.tenant_id}:{key_digest[:40]}"
|
||||
),
|
||||
lease_ttl_seconds=15 * 60,
|
||||
resource_type="file_integrity_finding",
|
||||
resource_id=finding.id,
|
||||
metadata={
|
||||
"resources": ["postgresql", "object-storage"],
|
||||
"storage_backend": backend.name,
|
||||
},
|
||||
)
|
||||
except (RecoveryOperationBusy, RecoveryOperationStateConflict) as exc:
|
||||
raise FileStorageError(
|
||||
"This orphan object is already owned by another recovery operation"
|
||||
) from exc
|
||||
except (RecoveryGuaranteeError, RuntimeError) as exc:
|
||||
raise FileStorageError(
|
||||
"The Files recovery ledger is unavailable; no object was deleted"
|
||||
) from exc
|
||||
if started.replayed or started.operation is None:
|
||||
raise FileStorageError(
|
||||
"This orphan cleanup was already completed; reload the finding"
|
||||
)
|
||||
pending = PendingOrphanCleanup(
|
||||
operation=started.operation,
|
||||
backend=backend,
|
||||
finding_id=finding.id,
|
||||
tenant_id=finding.tenant_id,
|
||||
storage_key=finding.storage_key,
|
||||
resolved_by_user_id=user_id,
|
||||
)
|
||||
_register_pending_effect(session, pending)
|
||||
return pending
|
||||
|
||||
|
||||
def _register_pending_effect(session: Session, effect: _PendingEffect) -> None:
|
||||
if not session.in_transaction():
|
||||
session.begin()
|
||||
pending = session.info.setdefault(_PENDING_EFFECTS_KEY, [])
|
||||
pending.append(effect)
|
||||
if session.info.get(_HOOKS_INSTALLED_KEY):
|
||||
return
|
||||
event.listen(session, "after_commit", _after_session_commit)
|
||||
event.listen(session, "after_rollback", _after_session_rollback)
|
||||
session.info[_HOOKS_INSTALLED_KEY] = True
|
||||
|
||||
|
||||
def _after_session_commit(session: Session) -> None:
|
||||
# SQLAlchemy also emits after_commit for a released SAVEPOINT. A Files
|
||||
# batch may open one while creating a later recovery row; settle only when
|
||||
# the outer business transaction has actually become visible.
|
||||
if session.in_nested_transaction():
|
||||
return
|
||||
_settle_pending_effects(session, committed=True)
|
||||
|
||||
|
||||
def _after_session_rollback(session: Session) -> None:
|
||||
if session.in_nested_transaction():
|
||||
return
|
||||
try:
|
||||
_settle_pending_effects(session, committed=False)
|
||||
except RecoveryGuaranteeError as exc:
|
||||
session.info.setdefault(_ROLLBACK_ERRORS_KEY, []).append(str(exc))
|
||||
|
||||
|
||||
def _settle_pending_effects(session: Session, *, committed: bool) -> None:
|
||||
pending = list(session.info.pop(_PENDING_EFFECTS_KEY, []))
|
||||
failures: list[Exception] = []
|
||||
for effect in pending:
|
||||
try:
|
||||
effect.settle(committed=committed)
|
||||
except Exception as exc: # preserve every effect's chance to settle
|
||||
failures.append(exc)
|
||||
operation = getattr(effect, "operation", None)
|
||||
if operation is not None and not operation.closed:
|
||||
try:
|
||||
operation.release_unresolved()
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
sqlite_fence_key = getattr(effect, "sqlite_fence_key", None)
|
||||
if sqlite_fence_key:
|
||||
_release_sqlite_fence(sqlite_fence_key)
|
||||
if failures:
|
||||
raise RecoveryGuaranteeError(
|
||||
f"{len(failures)} Files recovery operation(s) could not be finalized"
|
||||
) from failures[0]
|
||||
|
||||
|
||||
def _reserve_sqlite_fence(resource_key: str) -> None:
|
||||
with _SQLITE_FENCES_LOCK:
|
||||
if resource_key in _SQLITE_FENCES:
|
||||
raise RecoveryOperationBusy(
|
||||
f"Another local SQLite transaction owns {resource_key}"
|
||||
)
|
||||
_SQLITE_FENCES.add(resource_key)
|
||||
|
||||
|
||||
def _release_sqlite_fence(resource_key: str) -> None:
|
||||
with _SQLITE_FENCES_LOCK:
|
||||
_SQLITE_FENCES.discard(resource_key)
|
||||
|
||||
|
||||
def _require_started_operation(started: DurableRecoveryStart) -> DurableRecoveryOperation:
|
||||
if started.replayed or started.operation is None:
|
||||
raise RecoveryOperationStateConflict(started.operation_id, started.status)
|
||||
return started.operation
|
||||
|
||||
|
||||
def _begin_caller_transaction_recovery_operation(
|
||||
session: Session,
|
||||
*,
|
||||
session_factory: Callable[[], Session],
|
||||
identity: RuntimeIdentity,
|
||||
module_id: str,
|
||||
operation_type: str,
|
||||
idempotency_key: str,
|
||||
request: dict[str, object],
|
||||
recovery_plan: RecoveryPlan,
|
||||
precondition_evidence: dict[str, object],
|
||||
lease_resource_key: str,
|
||||
lease_ttl_seconds: int,
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
metadata: dict[str, object],
|
||||
) -> DurableRecoveryStart:
|
||||
"""Start SQLite recovery evidence inside the caller transaction.
|
||||
|
||||
SQLite permits only one writer, so opening the normal independent ledger
|
||||
transaction after an earlier member of a batch has written will deadlock
|
||||
until the busy timeout. The caller-transaction mode preserves fencing,
|
||||
request hashing, checkpoints, commit verification, and rollback
|
||||
compensation, but cannot make the intent survive a hard process loss
|
||||
before the caller commits. PostgreSQL never uses this reduced mode.
|
||||
"""
|
||||
|
||||
claim = acquire_lease(
|
||||
session,
|
||||
installation_id=identity.installation_id,
|
||||
resource_key=lease_resource_key,
|
||||
holder_node_id=identity.node_id,
|
||||
holder_incarnation=identity.incarnation,
|
||||
ttl_seconds=lease_ttl_seconds,
|
||||
metadata={
|
||||
"module_id": module_id,
|
||||
"operation_type": operation_type,
|
||||
"durability_mode": "sqlite_caller_transaction",
|
||||
},
|
||||
)
|
||||
if claim is None:
|
||||
raise RecoveryOperationBusy(
|
||||
f"Another runtime owns the recovery fence for {lease_resource_key}"
|
||||
)
|
||||
existing = session.execute(
|
||||
select(RecoveryOperation).where(
|
||||
RecoveryOperation.installation_id == identity.installation_id,
|
||||
RecoveryOperation.module_id == module_id,
|
||||
RecoveryOperation.idempotency_key == idempotency_key,
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
operation = plan_recovery_operation(
|
||||
session,
|
||||
installation_id=identity.installation_id,
|
||||
module_id=module_id,
|
||||
operation_type=operation_type,
|
||||
idempotency_key=idempotency_key,
|
||||
request=request,
|
||||
recovery_plan=recovery_plan,
|
||||
resource_type=resource_type,
|
||||
resource_id=resource_id,
|
||||
lease_claim=claim,
|
||||
metadata=metadata,
|
||||
)
|
||||
if existing is not None:
|
||||
if operation.status == RecoveryStatus.SUCCEEDED.value:
|
||||
release_lease(session, claim)
|
||||
return DurableRecoveryStart(
|
||||
operation_id=operation.id,
|
||||
status=operation.status,
|
||||
replayed=True,
|
||||
operation=None,
|
||||
)
|
||||
raise RecoveryOperationStateConflict(operation.id, operation.status)
|
||||
prepare_recovery_operation(
|
||||
session,
|
||||
operation,
|
||||
evidence=precondition_evidence,
|
||||
lease_claim=claim,
|
||||
)
|
||||
start_recovery_operation(
|
||||
session,
|
||||
operation,
|
||||
evidence={"lease_resource_key": lease_resource_key},
|
||||
lease_claim=claim,
|
||||
)
|
||||
if not verify_recovery_evidence_chain(session, operation.id):
|
||||
raise RecoveryGuaranteeError(
|
||||
"Recovery checkpoint chain verification failed before side effects"
|
||||
)
|
||||
return DurableRecoveryStart(
|
||||
operation_id=operation.id,
|
||||
status=operation.status,
|
||||
replayed=False,
|
||||
operation=DurableRecoveryOperation(
|
||||
session_factory=session_factory,
|
||||
operation_id=operation.id,
|
||||
lease_claim=claim,
|
||||
lease_ttl_seconds=lease_ttl_seconds,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _blob_write_evidence(effect: PendingBlobWrite) -> dict[str, object]:
|
||||
database_blob_present: bool | None
|
||||
database_matches: bool | None
|
||||
database_integrity_verified: bool | None
|
||||
try:
|
||||
with effect.operation.session_factory() as session:
|
||||
blob = session.get(FileBlob, effect.blob_id)
|
||||
database_blob_present = blob is not None
|
||||
database_matches = bool(
|
||||
blob is not None
|
||||
and blob.tenant_id == effect.tenant_id
|
||||
and blob.storage_key == effect.storage_key
|
||||
and blob.checksum_sha256 == effect.semantic_checksum_sha256
|
||||
and blob.size_bytes == effect.semantic_size_bytes
|
||||
and blob.protection_discriminator
|
||||
== effect.protection_discriminator
|
||||
and blob.encryption_envelope_id == effect.expected_envelope_id
|
||||
and (
|
||||
effect.expected_storage_checksum_sha256
|
||||
== (
|
||||
blob.storage_checksum_sha256
|
||||
or blob.checksum_sha256
|
||||
)
|
||||
)
|
||||
) if blob is not None else False
|
||||
database_integrity_verified = bool(
|
||||
blob is not None
|
||||
and blob.integrity_status == "verified"
|
||||
and blob.quarantined_at is None
|
||||
) if blob is not None else False
|
||||
except Exception:
|
||||
database_blob_present = None
|
||||
database_matches = None
|
||||
database_integrity_verified = None
|
||||
|
||||
object_present: bool | None
|
||||
observed_size: int | None = None
|
||||
observed_checksum: str | None = None
|
||||
try:
|
||||
digest = hashlib.sha256()
|
||||
observed_size = 0
|
||||
for chunk in effect.backend.iter_bytes(effect.storage_key):
|
||||
observed_size += len(chunk)
|
||||
digest.update(chunk)
|
||||
observed_checksum = digest.hexdigest()
|
||||
object_present = True
|
||||
except StorageObjectMissing:
|
||||
object_present = False
|
||||
except (StorageBackendError, OSError):
|
||||
object_present = None
|
||||
|
||||
object_size_matches = (
|
||||
observed_size == effect.expected_storage_size_bytes
|
||||
if object_present is True and effect.expected_storage_size_bytes is not None
|
||||
else False if object_present is False else None
|
||||
)
|
||||
object_checksum_matches = (
|
||||
observed_checksum == effect.expected_storage_checksum_sha256
|
||||
if object_present is True
|
||||
and effect.expected_storage_checksum_sha256 is not None
|
||||
else False if object_present is False else None
|
||||
)
|
||||
verified = database_blob_present is not None and object_present is not None
|
||||
return {
|
||||
"verified": verified,
|
||||
"checks": {
|
||||
"database_reloaded": database_blob_present is not None,
|
||||
"object_probed": object_present is not None,
|
||||
"database_matches_request": database_matches,
|
||||
"database_integrity_verified": database_integrity_verified,
|
||||
"object_size_matches": object_size_matches,
|
||||
"object_checksum_matches": object_checksum_matches,
|
||||
},
|
||||
"blob_id": effect.blob_id,
|
||||
"database_blob_present": database_blob_present,
|
||||
"database_matches_request": database_matches,
|
||||
"database_integrity_verified": database_integrity_verified,
|
||||
"object_present": object_present,
|
||||
"observed_size_bytes": observed_size,
|
||||
"observed_checksum_sha256": observed_checksum,
|
||||
"expected_storage_size_bytes": effect.expected_storage_size_bytes,
|
||||
"expected_storage_checksum_sha256": effect.expected_storage_checksum_sha256,
|
||||
}
|
||||
|
||||
|
||||
def _blob_write_complete(evidence: dict[str, object]) -> bool:
|
||||
return bool(
|
||||
evidence.get("verified") is True
|
||||
and evidence.get("database_matches_request") is True
|
||||
and evidence.get("database_integrity_verified") is True
|
||||
and _object_matches(evidence)
|
||||
)
|
||||
|
||||
|
||||
def _object_matches(evidence: dict[str, object]) -> bool:
|
||||
checks = evidence.get("checks")
|
||||
return bool(
|
||||
isinstance(checks, dict)
|
||||
and evidence.get("object_present") is True
|
||||
and checks.get("object_size_matches") is True
|
||||
and checks.get("object_checksum_matches") is True
|
||||
)
|
||||
|
||||
|
||||
def _forward_complete_blob_repair(effect: PendingBlobWrite) -> bool:
|
||||
try:
|
||||
with effect.operation.session_factory() as session:
|
||||
blob = session.get(FileBlob, effect.blob_id)
|
||||
if (
|
||||
blob is None
|
||||
or blob.tenant_id != effect.tenant_id
|
||||
or blob.storage_key != effect.storage_key
|
||||
or blob.checksum_sha256 != effect.semantic_checksum_sha256
|
||||
or blob.size_bytes != effect.semantic_size_bytes
|
||||
or blob.protection_discriminator
|
||||
!= effect.protection_discriminator
|
||||
or blob.encryption_envelope_id != effect.expected_envelope_id
|
||||
):
|
||||
return False
|
||||
blob.storage_checksum_sha256 = (
|
||||
effect.expected_storage_checksum_sha256
|
||||
if effect.expected_envelope_id
|
||||
else None
|
||||
)
|
||||
blob.storage_size_bytes = (
|
||||
effect.expected_storage_size_bytes
|
||||
if effect.expected_envelope_id
|
||||
else None
|
||||
)
|
||||
blob.integrity_status = "verified"
|
||||
blob.integrity_checked_at = datetime.now(UTC)
|
||||
blob.integrity_failure = None
|
||||
blob.quarantined_at = None
|
||||
session.add(blob)
|
||||
session.commit()
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _quarantine_blob_after_failed_verification(
|
||||
effect: PendingBlobWrite,
|
||||
evidence: dict[str, object],
|
||||
) -> None:
|
||||
try:
|
||||
with effect.operation.session_factory() as session:
|
||||
blob = session.get(FileBlob, effect.blob_id)
|
||||
if blob is None or blob.storage_key != effect.storage_key:
|
||||
return
|
||||
blob.integrity_status = (
|
||||
"missing"
|
||||
if evidence.get("object_present") is False
|
||||
else "checksum_mismatch"
|
||||
)
|
||||
blob.integrity_checked_at = datetime.now(UTC)
|
||||
blob.integrity_failure = "recovery_verification_failed"
|
||||
blob.quarantined_at = datetime.now(UTC)
|
||||
session.add(blob)
|
||||
session.commit()
|
||||
except Exception:
|
||||
return
|
||||
|
||||
|
||||
def _orphan_cleanup_evidence(effect: PendingOrphanCleanup) -> dict[str, object]:
|
||||
try:
|
||||
object_present: bool | None = effect.backend.exists(effect.storage_key)
|
||||
except (StorageBackendError, OSError):
|
||||
object_present = None
|
||||
finding_present: bool | None
|
||||
finding_deleted: bool | None
|
||||
still_unreferenced: bool | None
|
||||
try:
|
||||
with effect.operation.session_factory() as session:
|
||||
finding = session.get(FileIntegrityFinding, effect.finding_id)
|
||||
finding_present = finding is not None
|
||||
finding_deleted = bool(
|
||||
finding is not None and finding.state == "deleted"
|
||||
) if finding is not None else False
|
||||
referenced = (
|
||||
session.query(FileBlob.id)
|
||||
.filter(
|
||||
FileBlob.tenant_id == effect.tenant_id,
|
||||
FileBlob.storage_key == effect.storage_key,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
still_unreferenced = referenced is None
|
||||
except Exception:
|
||||
finding_present = None
|
||||
finding_deleted = None
|
||||
still_unreferenced = None
|
||||
verified = object_present is not None and finding_present is not None
|
||||
return {
|
||||
"verified": verified,
|
||||
"checks": {
|
||||
"database_reloaded": finding_present is not None,
|
||||
"object_probed": object_present is not None,
|
||||
"finding_marked_deleted": finding_deleted,
|
||||
"object_absent": (
|
||||
not object_present if object_present is not None else None
|
||||
),
|
||||
"still_unreferenced": still_unreferenced,
|
||||
},
|
||||
"finding_id": effect.finding_id,
|
||||
"finding_present": finding_present,
|
||||
"finding_deleted": finding_deleted,
|
||||
"object_present": object_present,
|
||||
"still_unreferenced": still_unreferenced,
|
||||
}
|
||||
|
||||
|
||||
def _orphan_cleanup_complete(evidence: dict[str, object]) -> bool:
|
||||
return bool(
|
||||
evidence.get("verified") is True
|
||||
and evidence.get("finding_deleted") is True
|
||||
and evidence.get("object_present") is False
|
||||
and evidence.get("still_unreferenced") is True
|
||||
)
|
||||
|
||||
|
||||
def _forward_complete_orphan_finding(effect: PendingOrphanCleanup) -> bool:
|
||||
try:
|
||||
with effect.operation.session_factory() as session:
|
||||
finding = session.get(FileIntegrityFinding, effect.finding_id)
|
||||
if (
|
||||
finding is None
|
||||
or finding.tenant_id != effect.tenant_id
|
||||
or finding.storage_key != effect.storage_key
|
||||
):
|
||||
return False
|
||||
scan = session.get(FileIntegrityScan, finding.scan_id)
|
||||
if scan is None or not finding.storage_key.startswith(
|
||||
scan.storage_prefix
|
||||
):
|
||||
return False
|
||||
referenced = (
|
||||
session.query(FileBlob.id)
|
||||
.filter(
|
||||
FileBlob.tenant_id == effect.tenant_id,
|
||||
FileBlob.storage_key == effect.storage_key,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if referenced:
|
||||
return False
|
||||
finding.state = "deleted"
|
||||
finding.resolved_at = datetime.now(UTC)
|
||||
finding.resolved_by_user_id = effect.resolved_by_user_id
|
||||
session.add(finding)
|
||||
session.commit()
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PendingBlobWrite",
|
||||
"PendingOrphanCleanup",
|
||||
"begin_blob_write_recovery",
|
||||
"begin_orphan_cleanup_recovery",
|
||||
]
|
||||
@@ -0,0 +1,331 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
import socket
|
||||
import threading
|
||||
from functools import lru_cache
|
||||
from importlib import import_module
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
create_outbound_connection,
|
||||
)
|
||||
|
||||
|
||||
class SdkPeerPinningError(RuntimeError):
|
||||
"""Raised when an optional SDK cannot be bound to the pinned transport."""
|
||||
|
||||
|
||||
_BOTOCORE_CLIENT_CREATION_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def create_pinned_s3_client(**kwargs: Any) -> Any:
|
||||
"""Construct an S3 client whose first and every later socket is pinned.
|
||||
|
||||
Botocore does not expose the HTTP-session class through boto3's public
|
||||
client API. Its endpoint creator does expose that seam, so this function
|
||||
replaces the creator only for the bounded client-construction operation.
|
||||
The lock avoids an unsafe interleaving with another GovOPlaN client
|
||||
construction. Any unrelated botocore client created during the short
|
||||
replacement window also receives the stricter transport.
|
||||
"""
|
||||
|
||||
try:
|
||||
boto3 = import_module("boto3")
|
||||
botocore_args = import_module("botocore.args")
|
||||
except ImportError as exc:
|
||||
raise SdkPeerPinningError(
|
||||
"S3 connector browsing requires the optional boto3 dependency"
|
||||
) from exc
|
||||
|
||||
pinned_session_cls = _botocore_transport_types()[0]
|
||||
original_creator = getattr(botocore_args, "EndpointCreator", None)
|
||||
if original_creator is None or not hasattr(original_creator, "create_endpoint"):
|
||||
raise SdkPeerPinningError(
|
||||
"The installed botocore release does not expose the endpoint transport seam required for peer pinning"
|
||||
)
|
||||
|
||||
class PinnedEndpointCreator(original_creator): # type: ignore[misc, valid-type]
|
||||
def create_endpoint(self, *args: Any, **endpoint_kwargs: Any) -> Any:
|
||||
endpoint_kwargs["http_session_cls"] = pinned_session_cls
|
||||
return super().create_endpoint(*args, **endpoint_kwargs)
|
||||
|
||||
with _BOTOCORE_CLIENT_CREATION_LOCK:
|
||||
current_creator = getattr(botocore_args, "EndpointCreator", None)
|
||||
if current_creator is not original_creator:
|
||||
raise SdkPeerPinningError(
|
||||
"Botocore endpoint construction changed concurrently; refusing to create an unproven S3 client"
|
||||
)
|
||||
botocore_args.EndpointCreator = PinnedEndpointCreator
|
||||
try:
|
||||
session_type = getattr(getattr(boto3, "session", None), "Session", None)
|
||||
if session_type is None:
|
||||
raise SdkPeerPinningError(
|
||||
"The installed boto3 release does not expose its isolated session constructor"
|
||||
)
|
||||
client = session_type().client("s3", **kwargs)
|
||||
finally:
|
||||
if getattr(botocore_args, "EndpointCreator", None) is PinnedEndpointCreator:
|
||||
botocore_args.EndpointCreator = original_creator
|
||||
|
||||
http_session = getattr(getattr(client, "_endpoint", None), "http_session", None)
|
||||
if http_session is None or not isinstance(http_session, pinned_session_cls):
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
raise SdkPeerPinningError(
|
||||
"Botocore did not install the required pinned HTTP transport; the S3 client was discarded"
|
||||
)
|
||||
return client
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _botocore_transport_types() -> tuple[type[Any], type[Any], type[Any]]:
|
||||
try:
|
||||
awsrequest = import_module("botocore.awsrequest")
|
||||
httpsession = import_module("botocore.httpsession")
|
||||
urllib3_exceptions = import_module("urllib3.exceptions")
|
||||
except ImportError as exc:
|
||||
raise SdkPeerPinningError(
|
||||
"S3 connector browsing requires a compatible botocore HTTP transport"
|
||||
) from exc
|
||||
|
||||
required = (
|
||||
"AWSHTTPConnection",
|
||||
"AWSHTTPSConnection",
|
||||
"AWSHTTPConnectionPool",
|
||||
"AWSHTTPSConnectionPool",
|
||||
)
|
||||
if any(not hasattr(awsrequest, name) for name in required) or not hasattr(
|
||||
httpsession, "URLLib3Session"
|
||||
):
|
||||
raise SdkPeerPinningError(
|
||||
"The installed botocore release is missing the connection classes required for peer pinning"
|
||||
)
|
||||
|
||||
def pinned_new_connection(connection: Any) -> socket.socket:
|
||||
hostname = str(getattr(connection, "_dns_host", "") or "").strip()
|
||||
port = int(getattr(connection, "port", 0) or 0)
|
||||
if not hostname or not port:
|
||||
raise urllib3_exceptions.NewConnectionError(
|
||||
connection, "Pinned S3 connection is missing its target authority"
|
||||
)
|
||||
try:
|
||||
return create_outbound_connection(
|
||||
hostname,
|
||||
port,
|
||||
timeout=getattr(connection, "timeout", None),
|
||||
source_address=getattr(connection, "source_address", None),
|
||||
socket_options=getattr(connection, "socket_options", None),
|
||||
label="S3 connector peer",
|
||||
)
|
||||
except socket.timeout as exc:
|
||||
raise urllib3_exceptions.ConnectTimeoutError(
|
||||
connection,
|
||||
f"Connection to {hostname} timed out while selecting an approved peer",
|
||||
) from exc
|
||||
except (OSError, OutboundHttpError, ValueError) as exc:
|
||||
raise urllib3_exceptions.NewConnectionError(
|
||||
connection,
|
||||
f"S3 connector peer was rejected: {exc}",
|
||||
) from exc
|
||||
|
||||
class PinnedAWSHTTPConnection(awsrequest.AWSHTTPConnection):
|
||||
_new_conn = pinned_new_connection
|
||||
|
||||
class PinnedAWSHTTPSConnection(awsrequest.AWSHTTPSConnection):
|
||||
_new_conn = pinned_new_connection
|
||||
|
||||
class PinnedAWSHTTPConnectionPool(awsrequest.AWSHTTPConnectionPool):
|
||||
ConnectionCls = PinnedAWSHTTPConnection
|
||||
|
||||
class PinnedAWSHTTPSConnectionPool(awsrequest.AWSHTTPSConnectionPool):
|
||||
ConnectionCls = PinnedAWSHTTPSConnection
|
||||
|
||||
class PinnedURLLib3Session(httpsession.URLLib3Session):
|
||||
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||
# A proxy would select the final target outside this process. Until
|
||||
# proxy peer delegation is modeled, connector traffic is direct.
|
||||
kwargs["proxies"] = {}
|
||||
super().__init__(*args, **kwargs)
|
||||
self._pool_classes_by_scheme = {
|
||||
"http": PinnedAWSHTTPConnectionPool,
|
||||
"https": PinnedAWSHTTPSConnectionPool,
|
||||
}
|
||||
manager = getattr(self, "_manager", None)
|
||||
if manager is None or not hasattr(manager, "pool_classes_by_scheme"):
|
||||
raise SdkPeerPinningError(
|
||||
"The installed botocore pool manager cannot enforce pinned connection classes"
|
||||
)
|
||||
manager.pool_classes_by_scheme = self._pool_classes_by_scheme
|
||||
|
||||
return PinnedURLLib3Session, PinnedAWSHTTPConnection, PinnedAWSHTTPSConnection
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def pinned_smb_connection_cache() -> dict[str, Any]:
|
||||
"""Return the Files-owned cache; unproven process-global sessions are never reused."""
|
||||
|
||||
return {}
|
||||
|
||||
|
||||
def install_pinned_smb_transport(smbclient: Any) -> Any:
|
||||
"""Install a process-wide fail-closed smbclient session factory.
|
||||
|
||||
smbclient routes initial connections, reconnects, and DFS targets through
|
||||
``smbclient._pool.register_session``. Replacing that single factory with a
|
||||
behavior-compatible implementation ensures every target uses a socket
|
||||
selected by Core's deployment-wide outbound policy.
|
||||
"""
|
||||
|
||||
try:
|
||||
pool = import_module("smbclient._pool")
|
||||
connection_module = import_module("smbprotocol.connection")
|
||||
session_module = import_module("smbprotocol.session")
|
||||
transport_module = import_module("smbprotocol.transport")
|
||||
except ImportError as exc:
|
||||
raise SdkPeerPinningError(
|
||||
"SMB connector browsing requires the optional smbprotocol dependency"
|
||||
) from exc
|
||||
|
||||
current = getattr(pool, "register_session", None)
|
||||
if getattr(current, "__govoplan_peer_pinned__", False):
|
||||
expected_tcp = getattr(current, "__govoplan_pinned_tcp__", None)
|
||||
if expected_tcp is None or getattr(connection_module, "Tcp", None) is not expected_tcp:
|
||||
raise SdkPeerPinningError(
|
||||
"The installed SMB transport changed after peer pinning; refusing to reuse the session factory"
|
||||
)
|
||||
return smbclient
|
||||
required_parameters = {
|
||||
"server",
|
||||
"username",
|
||||
"password",
|
||||
"port",
|
||||
"encrypt",
|
||||
"connection_timeout",
|
||||
"connection_cache",
|
||||
"auth_protocol",
|
||||
"require_signing",
|
||||
}
|
||||
if current is None or not required_parameters.issubset(
|
||||
inspect.signature(current).parameters
|
||||
):
|
||||
raise SdkPeerPinningError(
|
||||
"The installed smbprotocol release does not expose the session seam required for peer pinning"
|
||||
)
|
||||
|
||||
class PinnedTcp(transport_module.Tcp):
|
||||
def connect(self) -> None:
|
||||
with self._sock_lock:
|
||||
if self.connected:
|
||||
return
|
||||
try:
|
||||
self._sock = create_outbound_connection(
|
||||
self.server,
|
||||
int(self.port),
|
||||
timeout=self.timeout,
|
||||
label="SMB connector peer",
|
||||
)
|
||||
except (OSError, OutboundHttpError, ValueError) as exc:
|
||||
raise ValueError(
|
||||
f"SMB connector peer '{self.server}:{self.port}' was rejected: {exc}"
|
||||
) from exc
|
||||
self._sock.settimeout(None)
|
||||
self.connected = True
|
||||
|
||||
# Connection.connect() instantiates the module-level Tcp symbol on every
|
||||
# reconnect. Replacing it process-wide is deliberate: an SMB connection
|
||||
# created by another code path must become stricter, never bypass Files'
|
||||
# peer boundary.
|
||||
if not hasattr(connection_module, "Tcp"):
|
||||
raise SdkPeerPinningError(
|
||||
"The installed smbprotocol release cannot install the pinned TCP transport"
|
||||
)
|
||||
connection_module.Tcp = PinnedTcp
|
||||
|
||||
def pinned_register_session(
|
||||
server: str,
|
||||
username: str | None = None,
|
||||
password: str | None = None,
|
||||
port: int = 445,
|
||||
encrypt: bool | None = None,
|
||||
connection_timeout: float = 60,
|
||||
connection_cache: dict[str, Any] | None = None,
|
||||
auth_protocol: str = "negotiate",
|
||||
require_signing: bool = True,
|
||||
) -> Any:
|
||||
cache = pinned_smb_connection_cache() if connection_cache is None else connection_cache
|
||||
connection_key = f"{server.lower()}:{port}"
|
||||
connection = cache.get(connection_key)
|
||||
transport = getattr(connection, "transport", None)
|
||||
if connection is not None and not isinstance(transport, PinnedTcp):
|
||||
disconnect = getattr(connection, "disconnect", None)
|
||||
if callable(disconnect):
|
||||
try:
|
||||
disconnect(close=True)
|
||||
except Exception:
|
||||
pass
|
||||
cache.pop(connection_key, None)
|
||||
connection = None
|
||||
if connection is None or not getattr(connection.transport, "connected", False):
|
||||
connection = connection_module.Connection(
|
||||
pool.ClientConfig().client_guid,
|
||||
server,
|
||||
port,
|
||||
require_signing=require_signing,
|
||||
)
|
||||
connection.transport = PinnedTcp(server, port)
|
||||
connection.connect(timeout=connection_timeout)
|
||||
if not isinstance(connection.transport, PinnedTcp):
|
||||
disconnect = getattr(connection, "disconnect", None)
|
||||
if callable(disconnect):
|
||||
try:
|
||||
disconnect(close=True)
|
||||
except Exception:
|
||||
pass
|
||||
raise SdkPeerPinningError(
|
||||
"smbprotocol replaced the required pinned TCP transport during connection setup"
|
||||
)
|
||||
cache[connection_key] = connection
|
||||
|
||||
session = next(
|
||||
(
|
||||
item
|
||||
for item in connection.session_table.values()
|
||||
if username is None or item.username == username
|
||||
),
|
||||
None,
|
||||
)
|
||||
if session is None:
|
||||
session = session_module.Session(
|
||||
connection,
|
||||
username=username,
|
||||
password=password,
|
||||
require_encryption=(encrypt is True),
|
||||
auth_protocol=auth_protocol,
|
||||
)
|
||||
session.connect()
|
||||
elif encrypt is not None:
|
||||
if session.encrypt_data and not encrypt:
|
||||
raise ValueError(
|
||||
"Cannot disable encryption on an already negotiated session."
|
||||
)
|
||||
if not session.encrypt_data and encrypt:
|
||||
session.encrypt = True
|
||||
return session
|
||||
|
||||
pinned_register_session.__govoplan_peer_pinned__ = True # type: ignore[attr-defined]
|
||||
pinned_register_session.__govoplan_pinned_tcp__ = PinnedTcp # type: ignore[attr-defined]
|
||||
pool.register_session = pinned_register_session
|
||||
if hasattr(smbclient, "register_session"):
|
||||
smbclient.register_session = pinned_register_session
|
||||
return smbclient
|
||||
|
||||
|
||||
__all__ = [
|
||||
"SdkPeerPinningError",
|
||||
"create_pinned_s3_client",
|
||||
"install_pinned_smb_transport",
|
||||
"pinned_smb_connection_cache",
|
||||
]
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user