14 Commits

52 changed files with 4237 additions and 1278 deletions

View File

@@ -18,8 +18,8 @@ cd /mnt/DATA/git/govoplan-core
For combined checks, run:
```bash
cd /mnt/DATA/git/govoplan-core
./scripts/check-focused.sh
cd /mnt/DATA/git/govoplan
tools/checks/check-focused.sh
```
For WebUI permutation checks that include files:

View File

@@ -1,5 +1,9 @@
# govoplan-files
<!-- govoplan-repository-type:start -->
**Repository type:** module (domain).
<!-- govoplan-repository-type:end -->
GovOPlaN Files is the managed file module. It bundles backend storage APIs and the Files WebUI package so file features can be installed as one module.
## Ownership
@@ -74,7 +78,11 @@ Profiles can be supplied as JSON through
`GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`,
`endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and
credential references such as `password_env`, `token_env`, or `secret_ref`.
deployment-owned credential references such as `password_env`, `token_env`, or
`secret_ref`. Environment references require an exact name in the deployment-wide
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
process environment variables and use encrypted stored credentials or scoped
secret-provider references instead.
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
current principal (system, tenant, user, group, or accessible campaign scope) and
redacts secret values and environment variable names. Use the returned
@@ -98,8 +106,8 @@ conflict handling, source provenance, and connector audit path as direct
uploads. The Seafile provider uses account-token auth and the native file
download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET`
requests against the configured WebDAV endpoint. SMB profiles use
`smb://server[:port]/share[/path]` endpoints and environment-backed credentials
through `smbprotocol`.
`smb://server[:port]/share[/path]` endpoints and deployment-owned or encrypted
stored credentials through `smbprotocol`.
Local connector development assets live in `dev/connectors/`. The compose stack
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and

View File

@@ -17,3 +17,9 @@ SMB_HOST_PORT=1445
SMB_SHARE_NAME=files
SMB_USER=govoplan
SMB_PASSWORD=govoplan-smb
MINIO_API_HOST_PORT=9000
MINIO_CONSOLE_HOST_PORT=9001
MINIO_ROOT_USER=govoplan
MINIO_ROOT_PASSWORD=govoplan-minio
MINIO_BUCKET=govoplan

View File

@@ -9,7 +9,8 @@ binds services to localhost high ports.
```bash
cd /mnt/DATA/git/govoplan-files/dev/connectors
cp .env.example .env
docker compose up -d nextcloud nextcloud-db webdav smb
mkdir -p data/smb data/webdav
docker compose up -d nextcloud nextcloud-db webdav smb minio
docker compose up -d seafile-db seafile-memcached seafile
```
@@ -29,14 +30,17 @@ Endpoints:
`http://127.0.0.1:9082/seafdav/`
- WebDAV: `http://127.0.0.1:9083`
- SMB: `smb://127.0.0.1:1445/files`
- MinIO/S3 API: `http://127.0.0.1:9000`, console
`http://127.0.0.1:9001`
The local fixture data under `data/` is ignored by git.
## GovOPlaN Profile Config
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Credentials should be referenced by
secret refs or environment variables; profile API responses only expose the
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. This deployment-owned configuration
may reference environment variables only when their exact names are listed in
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; profile API responses expose only the
credential source and configured state.
Example local profile file:
@@ -92,6 +96,25 @@ Example local profile file:
"password_env": "SMB_PASSWORD",
"capabilities": ["browse", "import"],
"policy": { "allow": { "providers": ["smb"] } }
},
{
"id": "dev-s3",
"label": "Dev MinIO",
"provider": "s3",
"endpoint_url": "http://127.0.0.1:9000",
"base_path": "",
"scope_type": "system",
"credential_mode": "basic",
"username": "govoplan",
"password_env": "MINIO_ROOT_PASSWORD",
"capabilities": ["browse", "import"],
"metadata": {
"bucket": "govoplan",
"region": "us-east-1",
"path_style": true,
"verify_tls": false
},
"policy": { "allow": { "providers": ["s3"] } }
}
]
}
@@ -101,6 +124,8 @@ Start GovOPlaN with:
```bash
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
export GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST=SEAFILE_ADMIN_PASSWORD,NEXTCLOUD_ADMIN_PASSWORD,WEBDAV_PASSWORD,SMB_PASSWORD,MINIO_ROOT_PASSWORD
export GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true
```
## Smoke Test
@@ -114,9 +139,11 @@ cd /mnt/DATA/git/govoplan-files/dev/connectors
```
The script reads `.env` from this directory when present, seeds tiny WebDAV,
Nextcloud, and SMB fixtures, then browses and imports them through the connector
helper layer. Pass `--require-smb` after recreating the SMB container to fail on
SMB access errors instead of reporting them as an optional skip.
Nextcloud, SMB, and MinIO fixtures, then browses and imports them through the
connector helper layer. Pass `--require-smb` after recreating the SMB container
to fail on SMB access errors instead of reporting them as an optional skip. Pass
`--require-s3` after starting MinIO and installing `govoplan-files[s3]` to fail
on S3 access errors instead of reporting them as an optional skip.
SMB smoke checks need the optional Python dependency in the environment running
the script:
@@ -125,6 +152,20 @@ the script:
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
```
S3 smoke checks need the optional boto3 dependency in the environment running
the script:
```bash
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[s3]
```
The SMB service is built from `dev/connectors/smb/` so the development share is
deterministic: one `files` share backed by `data/smb`, with the credentials from
`.env`.
The SMB image runs as the non-root `govoplan` user with UID/GID `1000` and
listens on unprivileged container port `1445`. The default host endpoint remains
`smb://127.0.0.1:1445/files`. `SMB_USER` must stay `govoplan` unless the image is
rebuilt with a matching user; `SMB_PORT` must be `1024` or higher. `SMB_PASSWORD`
is applied when the image is built, so rebuild the `smb` service after changing
it in `.env`.

View File

@@ -86,21 +86,35 @@ services:
smb:
build:
context: ./smb
args:
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
image: govoplan-files-samba-dev:latest
restart: unless-stopped
environment:
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
SMB_USER: ${SMB_USER:-govoplan}
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
SMB_UID: ${SMB_UID:-1000}
SMB_GID: ${SMB_GID:-1000}
SMB_PORT: ${SMB_PORT:-1445}
ports:
- "127.0.0.1:${SMB_HOST_PORT:-1445}:445"
- "127.0.0.1:${SMB_HOST_PORT:-1445}:${SMB_PORT:-1445}"
volumes:
- ./data/smb:/storage
minio:
image: minio/minio:latest
restart: unless-stopped
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-govoplan}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-govoplan-minio}
ports:
- "127.0.0.1:${MINIO_API_HOST_PORT:-9000}:9000"
- "127.0.0.1:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"
volumes:
- minio:/data
volumes:
nextcloud-db:
nextcloud:
seafile-db:
seafile-data:
minio:

View File

@@ -1,10 +1,29 @@
FROM alpine:3.20
RUN apk add --no-cache samba-server samba-common-tools
ARG SMB_PASSWORD=govoplan-smb
RUN apk add --no-cache samba-server samba-common-tools \
&& addgroup -S -g 1000 govoplan \
&& adduser -S -D -H -h /nonexistent -s /sbin/nologin -u 1000 -G govoplan govoplan \
&& mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba /var/log/samba/cores \
&& printf '%s\n' \
'[global]' \
' passdb backend = tdbsam' \
' private dir = /var/lib/samba/private' \
' lock directory = /run/samba' \
' state directory = /var/lib/samba' \
' cache directory = /var/cache/samba' \
' pid directory = /run/samba' \
> /etc/samba/smb.conf \
&& printf '%s\n%s\n' "$SMB_PASSWORD" "$SMB_PASSWORD" | smbpasswd -s -a govoplan \
&& smbpasswd -e govoplan \
&& chown -R govoplan:govoplan /storage /var/lib/samba /var/cache/samba /run/samba /etc/samba /var/log/samba
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
EXPOSE 445
EXPOSE 1445
USER govoplan:govoplan
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]

View File

@@ -3,21 +3,42 @@ set -eu
share_name="${SMB_SHARE_NAME:-files}"
user_name="${SMB_USER:-govoplan}"
password="${SMB_PASSWORD:-govoplan-smb}"
uid="${SMB_UID:-1000}"
gid="${SMB_GID:-1000}"
smb_port="${SMB_PORT:-1445}"
runtime_user="$(id -un)"
runtime_group="$(id -gn)"
mkdir -p /storage /var/lib/samba/private /run/samba
case "$share_name" in
"" | *[!A-Za-z0-9_.-]*)
printf 'SMB_SHARE_NAME must contain only letters, numbers, dot, dash, or underscore.\n' >&2
exit 1
;;
esac
if ! getent group "$user_name" >/dev/null 2>&1; then
addgroup -g "$gid" "$user_name"
case "$user_name" in
"" | *[!A-Za-z0-9_.-]*)
printf 'SMB_USER must contain only letters, numbers, dot, dash, or underscore.\n' >&2
exit 1
;;
esac
case "$smb_port" in
"" | *[!0-9]*)
printf 'SMB_PORT must be numeric.\n' >&2
exit 1
;;
esac
if [ "$smb_port" -lt 1024 ]; then
printf 'SMB_PORT must be >= 1024 because the dev Samba container runs as a non-root user.\n' >&2
exit 1
fi
if ! id "$user_name" >/dev/null 2>&1; then
adduser -D -H -s /sbin/nologin -u "$uid" -G "$user_name" "$user_name"
if [ "$user_name" != "$runtime_user" ]; then
printf 'SMB_USER=%s is not supported by the non-root dev image; use %s or rebuild the image with a matching user.\n' "$user_name" "$runtime_user" >&2
exit 1
fi
chown -R "$user_name:$user_name" /storage
mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba
cat > /etc/samba/smb.conf <<EOF
[global]
@@ -25,12 +46,20 @@ cat > /etc/samba/smb.conf <<EOF
workgroup = WORKGROUP
security = user
map to guest = Never
guest account = $runtime_user
server min protocol = SMB2
server signing = mandatory
smb ports = $smb_port
load printers = no
printing = bsd
disable spoolss = yes
log level = 1
passdb backend = tdbsam
private dir = /var/lib/samba/private
lock directory = /run/samba
state directory = /var/lib/samba
cache directory = /var/cache/samba
pid directory = /run/samba
[$share_name]
path = /storage
@@ -38,13 +67,16 @@ cat > /etc/samba/smb.conf <<EOF
read only = no
guest ok = no
valid users = $user_name
force user = $user_name
force group = $user_name
force user = $runtime_user
force group = $runtime_group
create mask = 0664
directory mask = 0775
EOF
printf '%s\n%s\n' "$password" "$password" | smbpasswd -s -a "$user_name" >/dev/null
smbpasswd -e "$user_name" >/dev/null
if ! pdbedit -L -u "$user_name" >/dev/null 2>&1; then
printf 'Samba user %s is missing from passdb. Rebuild the smb image so the non-root passdb is seeded.\n' "$user_name" >&2
printf 'Run: docker compose build --no-cache smb && docker compose up -d smb\n' >&2
exit 1
fi
exec smbd --foreground --no-process-group --debug-stdout
exec smbd --foreground --no-process-group --debug-stdout -p "$smb_port"

View File

@@ -18,16 +18,17 @@ ROOT = Path(__file__).resolve().parent
def main() -> int:
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
parser.add_argument("--require-s3", action="store_true", help="Fail if the S3 connector cannot browse/import.")
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
args = parser.parse_args()
_load_dotenv()
_default_env()
preflight_failures = _preflight_services(require_smb=args.require_smb)
preflight_failures = _preflight_services(require_smb=args.require_smb, require_s3=args.require_s3)
if preflight_failures:
for failure in preflight_failures:
print(f"FAIL {failure}")
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb")
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb minio")
return 1
_seed_webdav_fixture()
_seed_smb_fixture()
@@ -36,6 +37,13 @@ def main() -> int:
except httpx.HTTPError as exc:
print(f"FAIL dev-nextcloud seed failed: {exc}")
return 1
try:
_seed_s3_fixture()
except Exception as exc:
if args.require_s3:
print(f"FAIL dev-s3 seed failed: {exc}")
return 1
print(f"SKIP dev-s3 seed failed: {exc}")
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
if args.debug_smb:
@@ -59,6 +67,15 @@ def main() -> int:
else:
print(f"SKIP {message}")
try:
_exercise_profile(profiles["dev-s3"], folder="GovOPlaN", file_path="GovOPlaN/s3-live.txt", expected="s3 live fixture")
except Exception as exc:
message = f"dev-s3: {exc}"
if args.require_s3:
failures.append(message)
else:
print(f"SKIP {message}")
if failures:
for failure in failures:
print(f"FAIL {failure}")
@@ -76,6 +93,9 @@ def _default_env() -> None:
"SMB_SHARE_NAME": "files",
"SMB_USER": "govoplan",
"SMB_PASSWORD": "govoplan-smb",
"MINIO_ROOT_USER": "govoplan",
"MINIO_ROOT_PASSWORD": "govoplan-minio",
"MINIO_BUCKET": "govoplan",
}
for key, value in defaults.items():
os.environ.setdefault(key, value)
@@ -96,7 +116,7 @@ def _load_dotenv() -> None:
os.environ[key] = value.strip().strip("\"'")
def _preflight_services(*, require_smb: bool) -> list[str]:
def _preflight_services(*, require_smb: bool, require_s3: bool) -> list[str]:
failures: list[str] = []
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
try:
@@ -121,6 +141,14 @@ def _preflight_services(*, require_smb: bool) -> list[str]:
pass
except OSError as exc:
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
if require_s3:
minio_url = f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}/minio/health/live"
try:
response = httpx.get(minio_url, timeout=3.0)
if response.status_code != 200:
failures.append(f"dev-s3 expected HTTP 200 at {minio_url}, got HTTP {response.status_code}")
except httpx.HTTPError as exc:
failures.append(f"dev-s3 is not reachable at {minio_url}: {exc}")
return failures
@@ -150,6 +178,20 @@ def _profile_payloads() -> list[dict[str, object]]:
"username": os.getenv("SMB_USER", "govoplan"),
"password_env": "SMB_PASSWORD",
},
{
"id": "dev-s3",
"provider": "s3",
"endpoint_url": f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
"credential_mode": "basic",
"username": os.getenv("MINIO_ROOT_USER", "govoplan"),
"password_env": "MINIO_ROOT_PASSWORD",
"metadata": {
"bucket": os.getenv("MINIO_BUCKET", "govoplan"),
"region": "us-east-1",
"path_style": True,
"verify_tls": False,
},
},
]
@@ -211,5 +253,30 @@ def _seed_nextcloud_fixture() -> None:
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
def _seed_s3_fixture() -> None:
try:
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
except ImportError as exc:
raise RuntimeError("boto3 is not installed; install govoplan-files[s3]") from exc
bucket = os.getenv("MINIO_BUCKET", "govoplan")
client = boto3.client(
"s3",
endpoint_url=f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
aws_access_key_id=os.getenv("MINIO_ROOT_USER", "govoplan"),
aws_secret_access_key=os.getenv("MINIO_ROOT_PASSWORD", "govoplan-minio"),
region_name="us-east-1",
config=Config(s3={"addressing_style": "path"}),
)
try:
client.create_bucket(Bucket=bucket)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code")
if code not in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
raise
client.put_object(Bucket=bucket, Key="GovOPlaN/s3-live.txt", Body=b"s3 live fixture\n", ContentType="text/plain")
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -49,8 +49,9 @@ any remote write behavior.
Every provider must:
- enforce GovOPlaN profile visibility and connector policy before browse/import
- keep credentials as environment variables or secret references, never API
response values
- keep credentials as encrypted values or scoped secret references, never API
response values; process-environment references are allowed only in
deployment-owned profiles with an exact deployment allowlist
- import external files into managed storage before they are used in campaigns
or workflows
- preserve source provenance and revision metadata

View File

@@ -34,8 +34,9 @@ Credential profile:
- provider: a specific provider or any provider
- scope: `system` or `tenant` for administered credentials
- credential mode: anonymous, environment reference, secret reference, or
encrypted stored password/token
- credential mode: anonymous, scoped secret reference, or encrypted stored
password/token; environment references are reserved for deployment-owned JSON
profiles and exact deployment allowlisting
- username and redacted secret configuration
- credential-local policy for allow/deny rules

View File

@@ -1,6 +1,6 @@
{
"name": "@govoplan/files-webui",
"version": "0.1.7",
"version": "0.1.8",
"private": true,
"type": "module",
"main": "webui/src/index.ts",
@@ -19,7 +19,7 @@
"LICENSE"
],
"peerDependencies": {
"@govoplan/core-webui": "^0.1.7",
"@govoplan/core-webui": "^0.1.9",
"lucide-react": "^1.23.0",
"react": "^19.0.0",
"react-dom": "^19.0.0",

View File

@@ -4,18 +4,22 @@ build-backend = "setuptools.build_meta"
[project]
name = "govoplan-files"
version = "0.1.7"
version = "0.1.8"
description = "GovOPlaN files module with backend and WebUI integration."
readme = "README.md"
requires-python = ">=3.12"
license = { file = "LICENSE" }
authors = [{ name = "GovOPlaN" }]
dependencies = [
"govoplan-core>=0.1.7",
"govoplan-core>=0.1.9",
"defusedxml>=0.7,<1",
"python-multipart>=0.0.31,<1",
]
[project.optional-dependencies]
s3 = [
"boto3>=1.34,<2",
]
smb = [
"smbprotocol>=1.13",
]

View File

@@ -17,6 +17,7 @@ from govoplan_files.backend.storage.campaign_attachments import (
managed_match_payloads,
prepared_campaign_snapshot,
public_attachment_summary_payload,
share_assets_with_campaign,
)
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
from govoplan_files.backend.storage.files import current_version_and_blob
@@ -44,6 +45,7 @@ class FilesCampaignCapability:
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
current_version_and_blob = staticmethod(current_version_and_blob)
share_assets_with_campaign = staticmethod(share_assets_with_campaign)
@staticmethod
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:

View File

@@ -1,12 +1,18 @@
from __future__ import annotations
from datetime import datetime
from typing import Any
from sqlalchemy import event, inspect
from sqlalchemy import event
from sqlalchemy.orm import Session as OrmSession
from govoplan_core.core.change_sequence import record_change
from govoplan_core.core.sqlalchemy_change_tracking import (
ensure_object_id,
has_attr_changes,
object_state,
operation_for_soft_deletable,
previous_value,
)
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid
FILES_MODULE_ID = "files"
@@ -39,7 +45,7 @@ def _record_files_changes(session: OrmSession, _flush_context: object, _instance
def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
operation = _operation_for_soft_deletable(
operation = operation_for_soft_deletable(
asset,
changed_attrs=(
"owner_type",
@@ -70,7 +76,7 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
"owner_type": asset.owner_type,
"owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id),
"path": asset.display_path,
"previous_path": _previous_value(asset, "display_path"),
"previous_path": previous_value(asset, "display_path"),
"filename": asset.filename,
"deleted_at": _isoformat(asset.deleted_at),
},
@@ -78,7 +84,7 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
operation = _operation_for_soft_deletable(
operation = operation_for_soft_deletable(
folder,
changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"),
)
@@ -99,17 +105,17 @@ def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
"owner_type": folder.owner_type,
"owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id),
"path": folder.path,
"previous_path": _previous_value(folder, "path"),
"previous_path": previous_value(folder, "path"),
"deleted_at": _isoformat(folder.deleted_at),
},
)
def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None:
state = inspect(share)
state = object_state(share)
if not share.file_asset_id:
return
if not state.pending and not _has_attr_changes(
if not state.pending and not has_attr_changes(
state,
("file_asset_id", "target_type", "target_id", "permission", "revoked_at"),
):
@@ -135,44 +141,8 @@ def _record_share_visibility_change(session: OrmSession, share: FileShare) -> No
)
def _operation_for_soft_deletable(obj: object, *, changed_attrs: tuple[str, ...]) -> str | None:
state = inspect(obj)
if state.pending:
return "created"
if not _has_attr_changes(state, changed_attrs):
return None
if "deleted_at" in state.attrs:
history = state.attrs.deleted_at.history
if history.has_changes():
if any(value is not None for value in history.added):
return "deleted"
if any(value is not None for value in history.deleted):
return "created"
return "updated"
def _has_attr_changes(state: Any, attrs: tuple[str, ...]) -> bool:
return any(name in state.attrs and state.attrs[name].history.has_changes() for name in attrs)
def _previous_value(obj: object, attr_name: str) -> str | None:
state = inspect(obj)
if attr_name not in state.attrs:
return None
history = state.attrs[attr_name].history
if not history.has_changes() or not history.deleted:
return None
value = history.deleted[0]
return str(value) if value is not None else None
def _ensure_id(obj: object) -> str:
resource_id = getattr(obj, "id", None)
if resource_id:
return str(resource_id)
resource_id = new_uuid()
setattr(obj, "id", resource_id)
return resource_id
return ensure_object_id(obj, new_uuid)
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:

View File

@@ -1 +1,25 @@
from govoplan_files.backend.db.models import *
from govoplan_files.backend.db.models import (
CampaignAttachmentUse,
FileAsset,
FileBlob,
FileConnectorCredential,
FileConnectorPolicy,
FileConnectorProfile,
FileConnectorSpace,
FileFolder,
FileShare,
FileVersion,
)
__all__ = [
"CampaignAttachmentUse",
"FileAsset",
"FileBlob",
"FileConnectorCredential",
"FileConnectorPolicy",
"FileConnectorProfile",
"FileConnectorSpace",
"FileFolder",
"FileShare",
"FileVersion",
]

View File

@@ -112,7 +112,7 @@ def _files_router(context: ModuleContext):
manifest = ModuleManifest(
id="files",
name="Files",
version="0.1.7",
version="0.1.8",
required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR),
optional_dependencies=("campaigns",),
provides_interfaces=(

View File

@@ -0,0 +1,158 @@
"""v0.1.7 files baseline
Revision ID: a7b8c9d0e1f3
Revises: None
Create Date: 2026-07-11 00:00:00.000000
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = 'a7b8c9d0e1f3'
down_revision = None
branch_labels = None
depends_on = '4f2a9c8e7b6d'
def upgrade() -> None:
op.create_table('file_connector_credentials',
sa.Column('id', sa.String(length=255), nullable=False),
sa.Column('tenant_id', sa.String(length=36), nullable=True),
sa.Column('scope_type', sa.String(length=20), nullable=False),
sa.Column('scope_id', sa.String(length=36), nullable=True),
sa.Column('label', sa.String(length=255), nullable=False),
sa.Column('provider', sa.String(length=50), nullable=True),
sa.Column('enabled', sa.Boolean(), nullable=False),
sa.Column('credential_mode', sa.String(length=30), nullable=False),
sa.Column('username', sa.String(length=320), nullable=True),
sa.Column('password_encrypted', sa.Text(), nullable=True),
sa.Column('token_encrypted', sa.Text(), nullable=True),
sa.Column('password_env', sa.String(length=255), nullable=True),
sa.Column('token_env', sa.String(length=255), nullable=True),
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
sa.Column('policy', sa.JSON(), nullable=True),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_created_by_user_id_access_users'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_credentials_tenant_id_scopes'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_credentials_updated_by_user_id_access_users'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_credentials'))
)
op.create_index(op.f('ix_file_connector_credentials_created_by_user_id'), 'file_connector_credentials', ['created_by_user_id'], unique=False)
op.create_index(op.f('ix_file_connector_credentials_enabled'), 'file_connector_credentials', ['enabled'], unique=False)
op.create_index(op.f('ix_file_connector_credentials_provider'), 'file_connector_credentials', ['provider'], unique=False)
op.create_index('ix_file_connector_credentials_scope', 'file_connector_credentials', ['scope_type', 'scope_id'], unique=False)
op.create_index(op.f('ix_file_connector_credentials_scope_id'), 'file_connector_credentials', ['scope_id'], unique=False)
op.create_index(op.f('ix_file_connector_credentials_scope_type'), 'file_connector_credentials', ['scope_type'], unique=False)
op.create_index(op.f('ix_file_connector_credentials_tenant_id'), 'file_connector_credentials', ['tenant_id'], unique=False)
op.create_index(op.f('ix_file_connector_credentials_updated_by_user_id'), 'file_connector_credentials', ['updated_by_user_id'], unique=False)
op.create_table('file_connector_policies',
sa.Column('id', sa.String(length=36), nullable=False),
sa.Column('tenant_id', sa.String(length=36), nullable=True),
sa.Column('scope_type', sa.String(length=20), nullable=False),
sa.Column('scope_id', sa.String(length=36), nullable=True),
sa.Column('policy', sa.JSON(), nullable=False),
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_created_by_user_id_access_users'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_policies_tenant_id_scopes'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_policies_updated_by_user_id_access_users'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_policies')),
sa.UniqueConstraint('tenant_id', 'scope_type', 'scope_id', name='uq_file_connector_policies_scope')
)
op.create_index(op.f('ix_file_connector_policies_created_by_user_id'), 'file_connector_policies', ['created_by_user_id'], unique=False)
op.create_index('ix_file_connector_policies_scope', 'file_connector_policies', ['scope_type', 'scope_id'], unique=False)
op.create_index(op.f('ix_file_connector_policies_scope_id'), 'file_connector_policies', ['scope_id'], unique=False)
op.create_index(op.f('ix_file_connector_policies_scope_type'), 'file_connector_policies', ['scope_type'], unique=False)
op.create_index(op.f('ix_file_connector_policies_tenant_id'), 'file_connector_policies', ['tenant_id'], unique=False)
op.create_index(op.f('ix_file_connector_policies_updated_by_user_id'), 'file_connector_policies', ['updated_by_user_id'], unique=False)
op.create_table('file_connector_profiles',
sa.Column('id', sa.String(length=255), nullable=False),
sa.Column('tenant_id', sa.String(length=36), nullable=True),
sa.Column('scope_type', sa.String(length=20), nullable=False),
sa.Column('scope_id', sa.String(length=36), nullable=True),
sa.Column('label', sa.String(length=255), nullable=False),
sa.Column('provider', sa.String(length=50), nullable=False),
sa.Column('endpoint_url', sa.String(length=1000), nullable=True),
sa.Column('base_path', sa.String(length=1000), nullable=True),
sa.Column('enabled', sa.Boolean(), nullable=False),
sa.Column('credential_profile_id', sa.String(length=255), nullable=True),
sa.Column('credential_mode', sa.String(length=30), nullable=False),
sa.Column('username', sa.String(length=320), nullable=True),
sa.Column('password_encrypted', sa.Text(), nullable=True),
sa.Column('token_encrypted', sa.Text(), nullable=True),
sa.Column('password_env', sa.String(length=255), nullable=True),
sa.Column('token_env', sa.String(length=255), nullable=True),
sa.Column('secret_ref', sa.String(length=1000), nullable=True),
sa.Column('capabilities', sa.JSON(), nullable=True),
sa.Column('policy', sa.JSON(), nullable=True),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
sa.Column('updated_by_user_id', sa.String(length=36), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_created_by_user_id_access_users'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_profiles_tenant_id_scopes'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['updated_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_profiles_updated_by_user_id_access_users'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_profiles'))
)
op.create_index(op.f('ix_file_connector_profiles_created_by_user_id'), 'file_connector_profiles', ['created_by_user_id'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_credential_profile_id'), 'file_connector_profiles', ['credential_profile_id'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_enabled'), 'file_connector_profiles', ['enabled'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_provider'), 'file_connector_profiles', ['provider'], unique=False)
op.create_index('ix_file_connector_profiles_scope', 'file_connector_profiles', ['scope_type', 'scope_id'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_scope_id'), 'file_connector_profiles', ['scope_id'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_scope_type'), 'file_connector_profiles', ['scope_type'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_tenant_id'), 'file_connector_profiles', ['tenant_id'], unique=False)
op.create_index(op.f('ix_file_connector_profiles_updated_by_user_id'), 'file_connector_profiles', ['updated_by_user_id'], unique=False)
op.create_table('file_connector_spaces',
sa.Column('id', sa.String(length=36), nullable=False),
sa.Column('tenant_id', sa.String(length=36), nullable=False),
sa.Column('owner_type', sa.String(length=20), nullable=False),
sa.Column('owner_user_id', sa.String(length=36), nullable=True),
sa.Column('owner_group_id', sa.String(length=36), nullable=True),
sa.Column('label', sa.String(length=255), nullable=False),
sa.Column('connector_profile_id', sa.String(length=255), nullable=False),
sa.Column('provider', sa.String(length=50), nullable=False),
sa.Column('library_id', sa.String(length=255), nullable=True),
sa.Column('remote_path', sa.String(length=1000), nullable=False),
sa.Column('sync_mode', sa.String(length=30), nullable=False),
sa.Column('read_only', sa.Boolean(), nullable=False),
sa.Column('is_active', sa.Boolean(), nullable=False),
sa.Column('created_by_user_id', sa.String(length=36), nullable=True),
sa.Column('deleted_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['created_by_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_created_by_user_id_access_users'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['owner_group_id'], ['access_groups.id'], name=op.f('fk_file_connector_spaces_owner_group_id_access_groups'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['owner_user_id'], ['access_users.id'], name=op.f('fk_file_connector_spaces_owner_user_id_access_users'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['tenant_id'], ['core_scopes.id'], name=op.f('fk_file_connector_spaces_tenant_id_scopes'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_file_connector_spaces'))
)
op.create_index(op.f('ix_file_connector_spaces_connector_profile_id'), 'file_connector_spaces', ['connector_profile_id'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_created_by_user_id'), 'file_connector_spaces', ['created_by_user_id'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_deleted_at'), 'file_connector_spaces', ['deleted_at'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_is_active'), 'file_connector_spaces', ['is_active'], unique=False)
op.create_index('ix_file_connector_spaces_owner', 'file_connector_spaces', ['tenant_id', 'owner_type', 'owner_user_id', 'owner_group_id'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_owner_group_id'), 'file_connector_spaces', ['owner_group_id'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_owner_type'), 'file_connector_spaces', ['owner_type'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_owner_user_id'), 'file_connector_spaces', ['owner_user_id'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_provider'), 'file_connector_spaces', ['provider'], unique=False)
op.create_index(op.f('ix_file_connector_spaces_tenant_id'), 'file_connector_spaces', ['tenant_id'], unique=False)
op.create_index('uq_file_connector_spaces_active_group_label', 'file_connector_spaces', ['tenant_id', 'owner_group_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'group' AND deleted_at IS NULL"))
op.create_index('uq_file_connector_spaces_active_user_label', 'file_connector_spaces', ['tenant_id', 'owner_user_id', 'label'], unique=True, sqlite_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"), postgresql_where=sa.text("owner_type = 'user' AND deleted_at IS NULL"))
def downgrade() -> None:
op.drop_table('file_connector_spaces')
op.drop_table('file_connector_profiles')
op.drop_table('file_connector_policies')
op.drop_table('file_connector_credentials')

View File

@@ -7,7 +7,7 @@ from dataclasses import replace
from datetime import datetime
from io import BytesIO
from typing import Any, Literal
from urllib.parse import quote, urljoin
from urllib.parse import quote, urljoin, urlsplit, urlunsplit
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, Query, UploadFile, status
from fastapi.responses import FileResponse, StreamingResponse
from starlette.background import BackgroundTask
@@ -105,6 +105,7 @@ from govoplan_files.backend.storage.access import ensure_group_access, group_ref
from govoplan_files.backend.storage.archives import create_zip_file, extract_zip_upload
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_credential_store import (
ConnectorCredential,
connector_credential_from_row,
create_connector_credential_row,
deactivate_connector_credential_row,
@@ -119,6 +120,10 @@ from govoplan_files.backend.storage.connector_browse import (
normalize_connector_browse_path,
)
from govoplan_files.backend.storage.connector_imports import ConnectorImportError, ConnectorImportUnsupported, read_connector_file
from govoplan_files.backend.storage.connector_deployment import (
connector_effective_endpoint_url,
reject_api_controlled_deployment_references,
)
from govoplan_files.backend.storage.connector_profile_store import (
connector_profile_from_row,
create_connector_profile_row,
@@ -570,11 +575,48 @@ def _discovery_profile_from_payload(
password_value=credentials.password,
token_value=credentials.token,
capabilities=("browse",),
metadata=payload.metadata,
# Discovery metadata is untrusted API input and must not be able to
# replace the candidate endpoint or inject a static/fake listing.
metadata={},
source_kind="discovery",
)
def _audit_connector_discovery_attempt(
session: Session,
principal: ApiPrincipal,
*,
provider: str,
endpoint_url: str,
base_path: str | None,
) -> None:
audit_from_principal(
session,
principal,
action="files.connector.discovery_attempted",
object_type="connector_endpoint",
object_id=provider,
details={
"provider": provider,
"endpoint_origin": _redacted_connector_url(endpoint_url),
"base_path": base_path,
},
commit=True,
)
def _redacted_connector_url(value: str) -> str:
try:
parsed = urlsplit(value)
hostname = parsed.hostname or ""
if ":" in hostname:
hostname = f"[{hostname}]"
netloc = f"{hostname}:{parsed.port}" if parsed.port is not None else hostname
return urlunsplit((parsed.scheme, netloc, "", "", ""))
except ValueError:
return "<invalid connector URL>"
def _visible_connector_profile(
session: Session,
principal: ApiPrincipal,
@@ -767,7 +809,11 @@ def _download_connector_payload(
provider=profile.provider,
external_id=f"{payload.library_id}:{source_path}",
external_path=source_path,
external_url=profile.endpoint_url,
external_url=connector_effective_endpoint_url(
provider=profile.provider,
endpoint_url=profile.endpoint_url,
metadata=profile.metadata,
),
operation=operation,
),
profile.policy_sources,
@@ -802,6 +848,14 @@ def _download_connector_payload(
return source_path, downloaded, metadata or {}
def _connector_browse_next_token(items: list[Any]) -> str | None:
for item in items:
token = item.metadata.get("next_continuation_token") if hasattr(item, "metadata") else None
if token:
return str(token)
return None
def _connector_audit_details(asset: FileAsset, version: FileVersion, blob: FileBlob, *, operation: str) -> dict[str, object] | None:
metadata = asset.metadata_ or {}
provenance = source_provenance_from_metadata(metadata)
@@ -937,38 +991,8 @@ def _asset_response(session: Session, asset: FileAsset, *, include_shares: bool
)
def _asset_list_response(session: Session, assets: list[FileAsset], *, include_shares: bool = False) -> list[FileAssetResponse]:
if not assets:
return []
asset_ids = [asset.id for asset in assets]
version_ids = [asset.current_version_id for asset in assets if asset.current_version_id]
if len(version_ids) != len(assets):
raise FileStorageError("File has no current version")
version_blob_rows: list[tuple[FileVersion, FileBlob]] = []
for chunk in _chunks(version_ids):
version_blob_rows.extend(
session.query(FileVersion, FileBlob)
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
.filter(FileVersion.id.in_(chunk))
.all()
)
versions_by_id = {version.id: version for version, _blob in version_blob_rows}
blobs_by_version_id = {version.id: blob for version, blob in version_blob_rows}
shares_by_asset_id: dict[str, list[FileShareResponse]] = {asset_id: [] for asset_id in asset_ids}
if include_shares:
for chunk in _chunks(asset_ids):
share_rows = (
session.query(FileShare)
.filter(FileShare.file_asset_id.in_(chunk))
.order_by(FileShare.created_at.desc())
.all()
)
for row in share_rows:
shares_by_asset_id.setdefault(row.file_asset_id, []).append(
FileShareResponse(
def _file_share_response(row: FileShare) -> FileShareResponse:
return FileShareResponse(
id=row.id,
target_type=row.target_type,
target_id=row.target_id,
@@ -976,27 +1000,62 @@ def _asset_list_response(session: Session, assets: list[FileAsset], *, include_s
created_at=row.created_at.isoformat(),
revoked_at=row.revoked_at.isoformat() if row.revoked_at else None,
)
)
def _asset_versions_and_blobs(session: Session, assets: list[FileAsset]) -> dict[str, tuple[FileVersion, FileBlob]]:
version_ids = [asset.current_version_id for asset in assets if asset.current_version_id]
if len(version_ids) != len(assets):
raise FileStorageError("File has no current version")
rows: list[tuple[FileVersion, FileBlob]] = []
for chunk in _chunks(version_ids):
rows.extend(
session.query(FileVersion, FileBlob)
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
.filter(FileVersion.id.in_(chunk))
.all()
)
return {version.id: (version, blob) for version, blob in rows}
def _asset_shares_by_id(session: Session, asset_ids: list[str], *, include_shares: bool) -> dict[str, list[FileShareResponse]]:
shares_by_asset_id: dict[str, list[FileShareResponse]] = {asset_id: [] for asset_id in asset_ids}
if not include_shares:
return shares_by_asset_id
for chunk in _chunks(asset_ids):
rows = (
session.query(FileShare)
.filter(FileShare.file_asset_id.in_(chunk))
.order_by(FileShare.created_at.desc())
.all()
)
for row in rows:
shares_by_asset_id.setdefault(row.file_asset_id, []).append(_file_share_response(row))
return shares_by_asset_id
def _sent_campaign_asset_ids(session: Session, asset_ids: list[str]) -> set[str]:
sent_asset_ids: set[str] = set()
for chunk in _chunks(asset_ids):
sent_rows = (
rows = (
session.query(CampaignAttachmentUse.file_asset_id)
.filter(CampaignAttachmentUse.file_asset_id.in_(chunk), CampaignAttachmentUse.use_stage == "sent")
.distinct()
.all()
)
sent_asset_ids.update(row[0] for row in sent_rows)
sent_asset_ids.update(row[0] for row in rows)
return sent_asset_ids
responses: list[FileAssetResponse] = []
for asset in assets:
version = versions_by_id.get(asset.current_version_id or "")
blob = blobs_by_version_id.get(asset.current_version_id or "")
if not version or not blob:
raise FileStorageError("File version not found")
def _loaded_asset_response(
asset: FileAsset,
*,
version: FileVersion,
blob: FileBlob,
shares: list[FileShareResponse],
sent_asset_ids: set[str],
) -> FileAssetResponse:
metadata = asset.metadata_ or {}
responses.append(
FileAssetResponse(
return FileAssetResponse(
id=asset.id,
tenant_id=asset.tenant_id,
owner_type=asset.owner_type,
@@ -1015,7 +1074,32 @@ def _asset_list_response(session: Session, assets: list[FileAsset], *, include_s
metadata=metadata,
source_provenance=source_provenance_from_metadata(metadata),
source_revision=source_revision_from_metadata(metadata),
shares=shares,
)
def _asset_list_response(session: Session, assets: list[FileAsset], *, include_shares: bool = False) -> list[FileAssetResponse]:
if not assets:
return []
asset_ids = [asset.id for asset in assets]
versions_and_blobs = _asset_versions_and_blobs(session, assets)
shares_by_asset_id = _asset_shares_by_id(session, asset_ids, include_shares=include_shares)
sent_asset_ids = _sent_campaign_asset_ids(session, asset_ids)
responses: list[FileAssetResponse] = []
for asset in assets:
version_and_blob = versions_and_blobs.get(asset.current_version_id or "")
if version_and_blob is None:
raise FileStorageError("File version not found")
version, blob = version_and_blob
responses.append(
_loaded_asset_response(
asset,
version=version,
blob=blob,
shares=shares_by_asset_id.get(asset.id, []),
sent_asset_ids=sent_asset_ids,
)
)
return responses
@@ -1099,7 +1183,11 @@ def _connector_space_policy_decision(
provider=profile.provider,
external_id=external_id,
external_path=browse_path,
external_url=profile.endpoint_url,
external_url=connector_effective_endpoint_url(
provider=profile.provider,
endpoint_url=profile.endpoint_url,
metadata=profile.metadata,
),
operation=operation,
),
profile.policy_sources,
@@ -1441,6 +1529,104 @@ def _entry_matches_delta_scope(
)
def _decode_files_delta_watermark(since: str) -> int:
try:
return decode_sequence_watermark(since)
except ValueError as exc:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc
def _changed_delta_resource_ids(entries: list[ChangeSequenceEntry]) -> tuple[list[str], list[str]]:
file_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "file"))
folder_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "folder"))
return file_ids, folder_ids
def _visible_assets_for_delta(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
changed_file_ids: list[str],
) -> dict[str, FileAsset]:
return {
asset.id: asset
for asset in list_assets_for_user(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
is_admin=_is_admin(principal),
)
if asset.id in changed_file_ids
}
def _changed_visible_folders_for_delta(
session: Session,
*,
principal: ApiPrincipal,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
path_prefix: str | None,
changed_folder_ids: list[str],
) -> dict[str, FileFolder]:
return {
folder.id: folder
for folder in _visible_folders_for_delta(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=path_prefix,
)
if folder.id in changed_folder_ids
}
def _deleted_delta_items(
session: Session,
*,
principal: ApiPrincipal,
entries: list[ChangeSequenceEntry],
visible_assets: dict[str, FileAsset],
visible_folders: dict[str, FileFolder],
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
campaign_id: str | None,
path_prefix: str | None,
) -> list[DeltaDeletedItem]:
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
for entry in entries:
if entry.resource_type == "file" and entry.resource_id in visible_assets:
continue
if entry.resource_type == "folder" and entry.resource_id in visible_folders:
continue
if not _entry_matches_delta_scope(
session,
principal,
entry,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
):
continue
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
id=entry.resource_id,
resource_type=entry.resource_type,
revision=encode_sequence_watermark(entry.id),
deleted_at=entry.created_at if entry.operation == "deleted" else None,
)
return list(deleted.values())
def _files_delta_response(
session: Session,
*,
@@ -1452,10 +1638,7 @@ def _files_delta_response(
since: str,
limit: int,
) -> FileDeltaResponse:
try:
since_sequence = decode_sequence_watermark(since)
except ValueError as exc:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc
since_sequence = _decode_files_delta_watermark(since)
if sequence_watermark_is_expired(
session,
since=since_sequence,
@@ -1483,65 +1666,41 @@ def _files_delta_response(
has_more = len(entries_plus_one) > limit
entries = entries_plus_one[:limit]
changed_file_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "file"))
changed_folder_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "folder"))
visible_assets = {
asset.id: asset
for asset in list_assets_for_user(
changed_file_ids, changed_folder_ids = _changed_delta_resource_ids(entries)
visible_assets = _visible_assets_for_delta(
session,
tenant_id=principal.tenant_id,
user_id=principal.user.id,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
is_admin=_is_admin(principal),
changed_file_ids=changed_file_ids,
)
if asset.id in changed_file_ids
}
visible_folders = {
folder.id: folder
for folder in _visible_folders_for_delta(
visible_folders = _changed_visible_folders_for_delta(
session,
principal=principal,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=path_prefix,
changed_folder_ids=changed_folder_ids,
)
if folder.id in changed_folder_ids
}
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
for entry in entries:
is_visible = (
(entry.resource_type == "file" and entry.resource_id in visible_assets)
or (entry.resource_type == "folder" and entry.resource_id in visible_folders)
)
if is_visible:
continue
if not _entry_matches_delta_scope(
deleted = _deleted_delta_items(
session,
principal,
entry,
principal=principal,
entries=entries,
visible_assets=visible_assets,
visible_folders=visible_folders,
owner_type=owner_type,
owner_id=owner_id,
campaign_id=campaign_id,
path_prefix=path_prefix,
):
continue
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
id=entry.resource_id,
resource_type=entry.resource_type,
revision=encode_sequence_watermark(entry.id),
deleted_at=entry.created_at if entry.operation == "deleted" else None,
)
watermark = encode_sequence_watermark(entries[-1].id) if has_more and entries else _files_delta_watermark(session, principal.tenant_id)
return FileDeltaResponse(
files=_asset_list_response(session, list(visible_assets.values()), include_shares=True),
folders=[_folder_response(folder) for folder in visible_folders.values()],
deleted=list(deleted.values()),
deleted=deleted,
watermark=watermark,
has_more=has_more,
full=False,
@@ -2148,6 +2307,148 @@ def _full_file_connector_settings_delta_response(
)
def _changed_file_connector_setting_ids(entries: list[ChangeSequenceEntry]) -> tuple[set[str], set[str], set[str], bool]:
changed_profile_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
}
changed_credential_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
}
changed_space_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
}
policy_changed = any(entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries)
return changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed
def _file_connector_settings_changed_sections(
*,
changed_profile_ids: set[str],
changed_credential_ids: set[str],
changed_space_ids: set[str],
policy_changed: bool,
profiles: list[ConnectorProfile],
) -> list[str]:
changed_sections = []
if changed_profile_ids or any(profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids for profile in profiles):
changed_sections.append("profiles")
if changed_credential_ids:
changed_sections.append("credentials")
if changed_space_ids:
changed_sections.append("spaces")
if policy_changed:
changed_sections.append("policy")
return changed_sections
def _file_connector_settings_deleted_items(
entries: list[ChangeSequenceEntry],
*,
visible_profiles: dict[str, ConnectorProfile],
visible_credentials: dict[str, ConnectorCredential],
visible_spaces: dict[str, FileConnectorSpace],
) -> list[DeltaDeletedItem]:
return [
_connector_deleted_item(entry)
for entry in entries
if (
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
and entry.resource_id not in visible_profiles
)
or (
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
and entry.resource_id not in visible_credentials
)
or (
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
and entry.resource_id not in visible_spaces
)
]
def _incremental_file_connector_settings_delta_response(
session: Session,
principal: ApiPrincipal,
*,
entries: list[ChangeSequenceEntry],
has_more: bool,
scope_type: str,
scope_id: str | None,
provider: str | None,
campaign_id: str | None,
include_disabled: bool,
include_inactive: bool,
owner_type: Literal["user", "group"] | None,
owner_id: str | None,
) -> FileConnectorSettingsDeltaResponse:
changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed = _changed_file_connector_setting_ids(entries)
profiles = _visible_connector_profiles(
session,
principal,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled and _can_read_disabled_connector_profiles(principal),
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
include_effective_policy=False,
)
visible_profiles = {
profile.id: profile
for profile in profiles
if profile.id in changed_profile_ids or (profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids)
}
credentials = _visible_connector_credentials(
session,
principal,
provider=provider,
include_disabled=include_disabled,
)
visible_credentials = {
credential.id: credential
for credential in credentials
if credential.id in changed_credential_ids
}
spaces = _visible_connector_spaces(
session,
principal,
owner_type=owner_type,
owner_id=owner_id,
include_inactive=include_inactive,
)
visible_spaces = {
space.id: space
for space in spaces
if space.id in changed_space_ids
}
return FileConnectorSettingsDeltaResponse(
profiles=[FileConnectorProfileResponse(**profile.to_response()) for profile in visible_profiles.values()],
credentials=[FileConnectorCredentialResponse(**credential.to_response()) for credential in visible_credentials.values()],
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
policy=FileConnectorPolicyResponse(**connector_policy_response(session, tenant_id=principal.tenant_id, scope_type=scope_type, scope_id=scope_id)) if policy_changed else None,
changed_sections=_file_connector_settings_changed_sections(
changed_profile_ids=changed_profile_ids,
changed_credential_ids=changed_credential_ids,
changed_space_ids=changed_space_ids,
policy_changed=policy_changed,
profiles=profiles,
),
deleted=_file_connector_settings_deleted_items(
entries,
visible_profiles=visible_profiles,
visible_credentials=visible_credentials,
visible_spaces=visible_spaces,
),
watermark=_file_connector_settings_response_watermark(session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more),
has_more=has_more,
full=False,
)
@router.get("/connectors/settings/delta", response_model=FileConnectorSettingsDeltaResponse)
def connector_settings_delta(
scope_type: str = Query(default="tenant"),
@@ -2193,94 +2494,19 @@ def connector_settings_delta(
owner_type=owner_type,
owner_id=owner_id,
)
changed_profile_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
}
changed_credential_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
}
changed_space_ids = {
entry.resource_id
for entry in entries
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
}
policy_changed = any(entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries)
profiles = _visible_connector_profiles(
return _incremental_file_connector_settings_delta_response(
session,
principal,
entries=entries,
has_more=has_more,
scope_type=scope_type,
scope_id=scope_id,
provider=provider,
campaign_id=campaign_id,
include_disabled=include_disabled and _can_read_disabled_connector_profiles(principal),
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
include_effective_policy=False,
)
visible_profiles = {
profile.id: profile
for profile in profiles
if profile.id in changed_profile_ids or (profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids)
}
credentials = _visible_connector_credentials(
session,
principal,
provider=provider,
include_disabled=include_disabled,
)
visible_credentials = {
credential.id: credential
for credential in credentials
if credential.id in changed_credential_ids
}
spaces = _visible_connector_spaces(
session,
principal,
include_inactive=include_inactive,
owner_type=owner_type,
owner_id=owner_id,
include_inactive=include_inactive,
)
visible_spaces = {
space.id: space
for space in spaces
if space.id in changed_space_ids
}
changed_sections = []
if changed_profile_ids or any(profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids for profile in profiles):
changed_sections.append("profiles")
if changed_credential_ids:
changed_sections.append("credentials")
if changed_space_ids:
changed_sections.append("spaces")
if policy_changed:
changed_sections.append("policy")
deleted = [
_connector_deleted_item(entry)
for entry in entries
if (
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
and entry.resource_id not in visible_profiles
)
or (
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
and entry.resource_id not in visible_credentials
)
or (
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
and entry.resource_id not in visible_spaces
)
]
return FileConnectorSettingsDeltaResponse(
profiles=[FileConnectorProfileResponse(**profile.to_response()) for profile in visible_profiles.values()],
credentials=[FileConnectorCredentialResponse(**credential.to_response()) for credential in visible_credentials.values()],
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
policy=FileConnectorPolicyResponse(**connector_policy_response(session, tenant_id=principal.tenant_id, scope_type=scope_type, scope_id=scope_id)) if policy_changed else None,
changed_sections=changed_sections,
deleted=deleted,
watermark=_file_connector_settings_response_watermark(session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more),
has_more=has_more,
full=False,
)
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
raise _http_error(exc) from exc
@@ -2297,8 +2523,17 @@ def list_connector_providers(
@router.post("/connectors/discover", response_model=FileConnectorDiscoveryResponse)
def discover_connector_endpoint(
payload: FileConnectorDiscoveryRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_any_scope("files:file:admin", "system:settings:write", "admin:settings:write")),
):
try:
reject_api_controlled_deployment_references(
password_env=payload.credentials.password_env,
token_env=payload.credentials.token_env,
metadata=payload.metadata,
)
except ValueError as exc:
raise _http_error(exc) from exc
if payload.provider not in {"webdav", "nextcloud"}:
return FileConnectorDiscoveryResponse(
provider=payload.provider,
@@ -2311,7 +2546,28 @@ def discover_connector_endpoint(
for endpoint_url in _webdav_discovery_candidates(payload):
profile = _discovery_profile_from_payload(payload, endpoint_url, principal=principal)
try:
_ensure_connector_configuration_allowed(
session,
principal,
connector_id=None,
credential_id=None,
provider=profile.provider,
endpoint_url=endpoint_url,
base_path=profile.base_path,
scope_type="tenant",
scope_id=principal.tenant_id,
operation="discover",
)
_audit_connector_discovery_attempt(
session,
principal,
provider=profile.provider,
endpoint_url=endpoint_url,
base_path=profile.base_path,
)
browse_connector_profile(profile, path=payload.base_path or "")
except ConnectorPolicyDenied as exc:
raise _connector_policy_error(exc) from exc
except (ConnectorBrowseError, ConnectorBrowseUnsupported, OSError, ValueError, json.JSONDecodeError) as exc:
message = str(exc)
if "credentials were rejected" in message.casefold():
@@ -2324,7 +2580,7 @@ def discover_connector_endpoint(
status="credentials_rejected",
message="The endpoint was found, but login failed with these credentials.",
candidates=candidates,
metadata={**payload.metadata, "discovered_by": "webdav-auth-challenge"},
metadata={"discovered_by": "webdav-auth-challenge"},
)
candidates.append({"endpoint_url": endpoint_url, "status": "found", "message": "The endpoint exists, but credentials are required or were rejected."})
return FileConnectorDiscoveryResponse(
@@ -2334,7 +2590,7 @@ def discover_connector_endpoint(
status="found",
message="The endpoint was found. Add working credentials before saving or testing the connection.",
candidates=candidates,
metadata={**payload.metadata, "discovered_by": "webdav-auth-challenge"},
metadata={"discovered_by": "webdav-auth-challenge"},
)
candidates.append({"endpoint_url": endpoint_url, "status": "failed", "message": message})
continue
@@ -2348,7 +2604,7 @@ def discover_connector_endpoint(
status=status_value,
message=message,
candidates=candidates,
metadata={**payload.metadata, "discovered_by": "webdav-propfind"},
metadata={"discovered_by": "webdav-propfind"},
)
return FileConnectorDiscoveryResponse(
provider=payload.provider,
@@ -2664,8 +2920,12 @@ def create_connector_profile(
principal,
connector_id=payload.id,
credential_id=payload.credential_profile_id,
provider=payload.provider,
endpoint_url=connector_effective_endpoint_url(
provider=payload.provider,
endpoint_url=payload.endpoint_url,
metadata=payload.metadata,
),
base_path=payload.base_path,
scope_type=payload.scope_type,
scope_id=payload.scope_id,
@@ -2819,6 +3079,7 @@ def browse_connector_profile_items(
profile_id: str,
path: str | None = None,
library_id: str | None = None,
continuation_token: str | None = None,
campaign_id: str | None = None,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(require_any_scope("files:file:read", "files:file:upload", "files:file:download", "files:file:admin", "system:settings:read", "admin:settings:read")),
@@ -2832,14 +3093,18 @@ def browse_connector_profile_items(
credential_id=profile.credential_profile_id,
provider=profile.provider,
external_path=browse_path,
external_url=profile.endpoint_url,
external_url=connector_effective_endpoint_url(
provider=profile.provider,
endpoint_url=profile.endpoint_url,
metadata=profile.metadata,
),
operation="browse",
),
profile.policy_sources,
)
if not decision.allowed:
raise ConnectorPolicyDenied(decision)
items = browse_connector_profile(profile, path=browse_path, library_id=library_id)
items = browse_connector_profile(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
except ConnectorPolicyDenied as exc:
raise _connector_policy_error(exc) from exc
except ConnectorBrowseUnsupported as exc:
@@ -2851,6 +3116,8 @@ def browse_connector_profile_items(
provider=profile.provider,
path=browse_path,
library_id=library_id,
next_continuation_token=_connector_browse_next_token(items),
has_more=any(bool(item.metadata.get("listing_truncated")) for item in items),
decision=decision.to_dict(),
items=[FileConnectorBrowseItem(**item.to_response()) for item in items],
)
@@ -2906,8 +3173,12 @@ def update_connector_profile(
principal,
connector_id=row.id,
credential_id=credential_profile_id,
provider=provider,
endpoint_url=connector_effective_endpoint_url(
provider=provider,
endpoint_url=payload.endpoint_url if payload.endpoint_url is not None else row.endpoint_url,
metadata=payload.metadata if payload.metadata is not None else row.metadata_,
),
base_path=payload.base_path if payload.base_path is not None else row.base_path,
scope_type=row.scope_type,
scope_id=row.scope_id,
@@ -3230,7 +3501,7 @@ def download_archive(
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
for file_id in payload.file_ids
]
tmp = tempfile.NamedTemporaryFile(prefix="multimailer-files-", suffix=".zip", delete=False)
tmp = tempfile.NamedTemporaryFile(prefix="govoplan-files-", suffix=".zip", delete=False)
tmp_path = tmp.name
tmp.close()
try:

View File

@@ -1,36 +1,10 @@
from __future__ import annotations
from typing import Any
from govoplan_core.core.runtime import ModuleRuntimeState
_runtime_registry: object | None = None
_runtime_settings: object | None = None
_runtime = ModuleRuntimeState("Files")
def configure_runtime(*, registry: object | None = None, settings: object | None = None) -> None:
global _runtime_registry, _runtime_settings
if registry is not None:
_runtime_registry = registry
if settings is not None:
_runtime_settings = settings
def get_registry() -> object | None:
return _runtime_registry
def get_settings() -> object:
if _runtime_settings is not None:
return _runtime_settings
try:
from govoplan_core.settings import settings as legacy_settings
except ModuleNotFoundError as exc:
raise RuntimeError("GovOPlaN Files runtime settings are not configured") from exc
return legacy_settings
class SettingsProxy:
def __getattr__(self, name: str) -> Any:
return getattr(get_settings(), name)
settings = SettingsProxy()
configure_runtime = _runtime.configure_runtime
get_registry = _runtime.get_registry
get_settings = _runtime.get_settings
settings = _runtime.settings

View File

@@ -287,7 +287,7 @@ class FileConnectorDiscoveryCandidate(BaseModel):
class FileConnectorDiscoveryRequest(BaseModel):
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"]
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
endpoint_url: str
base_path: str | None = None
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
@@ -309,7 +309,7 @@ class FileConnectorDiscoveryResponse(BaseModel):
class FileConnectorProfileCreateRequest(BaseModel):
id: str
label: str
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"]
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
endpoint_url: str | None = None
base_path: str | None = None
enabled: bool = True
@@ -325,7 +325,7 @@ class FileConnectorProfileCreateRequest(BaseModel):
class FileConnectorProfileUpdateRequest(BaseModel):
label: str | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
endpoint_url: str | None = None
base_path: str | None = None
enabled: bool | None = None
@@ -342,7 +342,7 @@ class FileConnectorProfileUpdateRequest(BaseModel):
class FileConnectorCredentialCreateRequest(BaseModel):
id: str
label: str
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
enabled: bool = True
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
scope_id: str | None = None
@@ -354,7 +354,7 @@ class FileConnectorCredentialCreateRequest(BaseModel):
class FileConnectorCredentialUpdateRequest(BaseModel):
label: str | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
enabled: bool | None = None
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
credentials: FileConnectorProfileCredentialsRequest | None = None
@@ -404,6 +404,8 @@ class FileConnectorBrowseResponse(BaseModel):
path: str = ""
library_id: str | None = None
read_only: bool = True
next_continuation_token: str | None = None
has_more: bool = False
decision: dict[str, Any]
items: list[FileConnectorBrowseItem]

View File

@@ -4,6 +4,12 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterable, Protocol
from govoplan_core.security.outbound_http import (
OutboundHttpError,
response_limit,
validate_unpinned_sdk_http_url,
)
from govoplan_files.backend.runtime import settings
@@ -96,15 +102,25 @@ class S3StorageBackend:
import boto3
except ModuleNotFoundError as exc:
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
try:
endpoint_url = validate_unpinned_sdk_http_url(
self.endpoint_url,
label="File storage S3 endpoint",
)
return boto3.client(
"s3",
endpoint_url=self.endpoint_url,
endpoint_url=endpoint_url,
region_name=self.region_name,
aws_access_key_id=self.access_key_id,
aws_secret_access_key=self.secret_access_key,
)
except OutboundHttpError as exc:
raise StorageBackendError(str(exc)) from exc
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
max_bytes = response_limit("file")
if len(data) > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
kwargs = {"Bucket": self.bucket, "Key": key, "Body": data}
if content_type:
kwargs["ContentType"] = content_type
@@ -113,19 +129,39 @@ class S3StorageBackend:
def get_bytes(self, key: str) -> bytes:
try:
obj = self.client.get_object(Bucket=self.bucket, Key=key)
return obj["Body"].read()
max_bytes = response_limit("file")
body = obj["Body"]
try:
_reject_declared_object_size(obj, max_bytes=max_bytes)
data = body.read(max_bytes + 1)
if len(data) > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
return data
finally:
if hasattr(body, "close"):
body.close()
except Exception as exc: # pragma: no cover - depends on S3 backend
raise StorageBackendError(str(exc)) from exc
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
try:
obj = self.client.get_object(Bucket=self.bucket, Key=key)
max_bytes = response_limit("file")
body = obj["Body"]
try:
_reject_declared_object_size(obj, max_bytes=max_bytes)
total = 0
while True:
chunk = body.read(chunk_size)
if not chunk:
break
total += len(chunk)
if total > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
yield chunk
finally:
if hasattr(body, "close"):
body.close()
except Exception as exc: # pragma: no cover - depends on S3 backend
raise StorageBackendError(str(exc)) from exc
@@ -140,6 +176,17 @@ class S3StorageBackend:
return False
def _reject_declared_object_size(obj: object, *, max_bytes: int) -> None:
if not isinstance(obj, dict):
return
try:
declared_size = int(obj.get("ContentLength"))
except (TypeError, ValueError):
return
if declared_size > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
def _fallback_roots() -> tuple[Path, ...]:
raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())

View File

@@ -12,10 +12,11 @@ from typing import Any, Iterator
from sqlalchemy.orm import Session
from govoplan_files.backend.db.models import FileAsset
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
from govoplan_files.backend.storage.backends import StorageBackend, StorageBackendError, get_storage_backend
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import current_versions_and_blobs, list_assets_for_user
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
from govoplan_files.backend.storage.access import ensure_owner_access
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
@@ -37,6 +38,7 @@ class ManagedAttachmentFile:
checksum_sha256: str
size_bytes: int
content_type: str | None
linked_to_campaign: bool = True
source_provenance: dict[str, Any] | None = None
source_revision: str | None = None
@@ -56,6 +58,7 @@ class PreparedCampaignSnapshot:
raw_json: dict[str, Any]
managed_files_by_local_path: dict[str, ManagedAttachmentFile]
shared_assets: list[FileAsset]
candidate_assets: list[FileAsset]
def parse_managed_source(value: object) -> tuple[str, str] | None:
@@ -117,6 +120,99 @@ def _iter_rule_dicts(attachments: dict[str, Any], raw_json: dict[str, Any]):
yield rule
def _managed_base_sources(raw_json: dict[str, Any]) -> list[tuple[str, str, str]]:
attachments = raw_json.get("attachments")
if not isinstance(attachments, dict):
return []
base_paths = attachments.get("base_paths")
if not isinstance(base_paths, list):
return []
sources: list[tuple[str, str, str]] = []
seen: set[tuple[str, str, str]] = set()
for item in base_paths:
if not isinstance(item, dict):
continue
parsed_source = parse_managed_source(item.get("source"))
if parsed_source is None:
continue
owner_type, owner_id = parsed_source
old_path = str(item.get("path") or ".").strip() or "."
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
key = (owner_type, owner_id, logical_root)
if key in seen:
continue
seen.add(key)
sources.append(key)
return sources
def _campaign_linked_asset_ids(
session: Session,
*,
tenant_id: str,
campaign_id: str,
asset_ids: list[str],
) -> set[str]:
if not asset_ids:
return set()
linked: set[str] = set()
for offset in range(0, len(asset_ids), 900):
chunk = asset_ids[offset : offset + 900]
rows = (
session.query(FileShare.file_asset_id)
.filter(
FileShare.tenant_id == tenant_id,
FileShare.file_asset_id.in_(chunk),
FileShare.target_type == "campaign",
FileShare.target_id == campaign_id,
FileShare.revoked_at.is_(None),
)
.all()
)
linked.update(row[0] for row in rows)
return linked
def _candidate_assets_for_managed_sources(
session: Session,
*,
tenant_id: str,
campaign_id: str,
raw_json: dict[str, Any],
user_id: str,
is_admin: bool,
shared_assets: list[FileAsset],
) -> tuple[list[FileAsset], set[str]]:
assets_by_id: dict[str, FileAsset] = {asset.id: asset for asset in shared_assets}
for owner_type, owner_id, logical_root in _managed_base_sources(raw_json):
ensure_owner_access(
session,
tenant_id=tenant_id,
owner_type=owner_type,
owner_id=owner_id,
user_id=user_id,
is_admin=is_admin,
)
for asset in list_assets_for_user(
session,
tenant_id=tenant_id,
user_id=user_id,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=logical_root or None,
is_admin=is_admin,
):
assets_by_id.setdefault(asset.id, asset)
candidate_assets = sorted(assets_by_id.values(), key=lambda asset: (asset.display_path, asset.updated_at, asset.id))
linked_ids = _campaign_linked_asset_ids(
session,
tenant_id=tenant_id,
campaign_id=campaign_id,
asset_ids=[asset.id for asset in candidate_assets],
)
return candidate_assets, linked_ids
def _selected_base_path(
rule: dict[str, Any],
prepared_by_id: dict[str, tuple[str, str]],
@@ -136,6 +232,177 @@ def _selected_base_path(
return None
def _prepared_attachment_config(raw_json: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any], list[Any]]:
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
attachments = prepared_json.get("attachments")
if not isinstance(attachments, dict):
attachments = {}
prepared_json["attachments"] = attachments
base_paths = attachments.get("base_paths")
if not isinstance(base_paths, list):
base_paths = []
return prepared_json, attachments, base_paths
def _campaign_snapshot_assets(
session: Session,
*,
tenant_id: str,
campaign_id: str,
prepared_json: dict[str, Any],
include_unlinked_candidates: bool,
user_id: str,
is_admin: bool,
) -> tuple[list[FileAsset], list[FileAsset], set[str]]:
shared_assets = list_assets_for_user(
session,
tenant_id=tenant_id,
user_id="",
campaign_id=campaign_id,
is_admin=True,
)
if not include_unlinked_candidates:
return shared_assets, shared_assets, {asset.id for asset in shared_assets}
candidate_assets, linked_asset_ids = _candidate_assets_for_managed_sources(
session,
tenant_id=tenant_id,
campaign_id=campaign_id,
raw_json=prepared_json,
user_id=user_id,
is_admin=is_admin,
shared_assets=shared_assets,
)
return shared_assets, candidate_assets, linked_asset_ids
def _assets_grouped_by_owner(assets: list[FileAsset]) -> dict[tuple[str, str], list[FileAsset]]:
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
for asset in assets:
owner_id = _asset_owner_id(asset)
if owner_id:
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
return assets_by_owner
def _materialize_managed_asset(
asset: FileAsset,
*,
owner_id: str,
logical_root: str,
local_root: Path,
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
backend: StorageBackend | None,
include_bytes: bool,
linked_asset_ids: set[str],
) -> tuple[str, ManagedAttachmentFile] | None:
relative_path = _relative_asset_path(asset, logical_root)
if not relative_path:
return None
target = _safe_local_target(local_root, relative_path)
target.parent.mkdir(parents=True, exist_ok=True)
version, blob = version_blobs[asset.id]
if include_bytes:
try:
data = backend.get_bytes(blob.storage_key) if backend else b""
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
target.write_bytes(data)
else:
target.touch()
local_key = str(target.resolve())
return local_key, ManagedAttachmentFile(
local_path=local_key,
asset_id=asset.id,
version_id=version.id,
blob_id=blob.id,
display_path=asset.display_path,
relative_path=normalize_logical_path(relative_path),
filename=asset.filename,
owner_type=asset.owner_type,
owner_id=owner_id,
checksum_sha256=blob.checksum_sha256,
size_bytes=blob.size_bytes,
content_type=blob.content_type,
linked_to_campaign=asset.id in linked_asset_ids,
source_provenance=source_provenance_from_metadata(asset.metadata_),
source_revision=source_revision_from_metadata(asset.metadata_),
)
def _prepare_managed_base_paths(
base_paths: list[Any],
*,
materialized_root: Path,
assets_by_owner: dict[tuple[str, str], list[FileAsset]],
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
backend: StorageBackend | None,
include_bytes: bool,
linked_asset_ids: set[str],
) -> tuple[
dict[str, ManagedAttachmentFile],
dict[str, tuple[str, str]],
dict[str, list[tuple[str, str]]],
tuple[str, str] | None,
]:
manifest: dict[str, ManagedAttachmentFile] = {}
prepared_by_id: dict[str, tuple[str, str]] = {}
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
first_prepared: tuple[str, str] | None = None
for index, item in enumerate(base_paths):
if not isinstance(item, dict):
continue
parsed_source = parse_managed_source(item.get("source"))
if parsed_source is None:
continue
owner_type, owner_id = parsed_source
old_path = str(item.get("path") or ".").strip() or "."
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
local_root.mkdir(parents=True, exist_ok=True)
local_root_string = str(local_root.resolve())
prepared = (base_path_id, local_root_string)
prepared_by_id[base_path_id] = prepared
prepared_by_old_path.setdefault(old_path, []).append(prepared)
if first_prepared is None:
first_prepared = prepared
item["path"] = local_root_string
for asset in assets_by_owner.get((owner_type, owner_id), []):
materialized = _materialize_managed_asset(
asset,
owner_id=owner_id,
logical_root=logical_root,
local_root=local_root,
version_blobs=version_blobs,
backend=backend,
include_bytes=include_bytes,
linked_asset_ids=linked_asset_ids,
)
if materialized is not None:
local_key, managed_file = materialized
manifest[local_key] = managed_file
return manifest, prepared_by_id, prepared_by_old_path, first_prepared
def _rewrite_managed_attachment_rules(
attachments: dict[str, Any],
prepared_json: dict[str, Any],
*,
prepared_by_id: dict[str, tuple[str, str]],
prepared_by_old_path: dict[str, list[tuple[str, str]]],
first_prepared: tuple[str, str] | None,
) -> None:
for rule in _iter_rule_dicts(attachments, prepared_json):
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
if selected is None:
continue
base_path_id, local_root_string = selected
rule["base_path_id"] = base_path_id
rule["base_dir"] = local_root_string
if first_prepared is not None:
attachments["base_path"] = first_prepared[1]
def prepare_campaign_snapshot(
session: Session,
*,
@@ -144,6 +411,9 @@ def prepare_campaign_snapshot(
raw_json: dict[str, Any],
destination: Path,
include_bytes: bool,
include_unlinked_candidates: bool = False,
user_id: str = "",
is_admin: bool = False,
) -> PreparedCampaignSnapshot:
"""Create a temporary file-oriented campaign snapshot for managed attachments.
@@ -159,101 +429,35 @@ def prepare_campaign_snapshot(
materialized_root = destination / "managed-attachments"
materialized_root.mkdir(parents=True, exist_ok=True)
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
attachments = prepared_json.get("attachments")
if not isinstance(attachments, dict):
attachments = {}
prepared_json["attachments"] = attachments
base_paths = attachments.get("base_paths")
if not isinstance(base_paths, list):
base_paths = []
shared_assets = list_assets_for_user(
prepared_json, attachments, base_paths = _prepared_attachment_config(raw_json)
shared_assets, candidate_assets, linked_asset_ids = _campaign_snapshot_assets(
session,
tenant_id=tenant_id,
user_id="",
campaign_id=campaign_id,
is_admin=True,
prepared_json=prepared_json,
include_unlinked_candidates=include_unlinked_candidates,
user_id=user_id,
is_admin=is_admin,
)
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
for asset in shared_assets:
owner_id = _asset_owner_id(asset)
if owner_id:
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
version_blobs = current_versions_and_blobs(session, shared_assets)
assets_by_owner = _assets_grouped_by_owner(candidate_assets)
version_blobs = current_versions_and_blobs(session, candidate_assets)
backend = get_storage_backend() if include_bytes else None
manifest: dict[str, ManagedAttachmentFile] = {}
prepared_by_id: dict[str, tuple[str, str]] = {}
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
first_prepared: tuple[str, str] | None = None
for index, item in enumerate(base_paths):
if not isinstance(item, dict):
continue
parsed_source = parse_managed_source(item.get("source"))
if parsed_source is None:
continue
owner_type, owner_id = parsed_source
old_path = str(item.get("path") or ".").strip() or "."
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
local_root.mkdir(parents=True, exist_ok=True)
local_root_string = str(local_root.resolve())
prepared = (base_path_id, local_root_string)
prepared_by_id[base_path_id] = prepared
prepared_by_old_path.setdefault(old_path, []).append(prepared)
if first_prepared is None:
first_prepared = prepared
item["path"] = local_root_string
for asset in assets_by_owner.get((owner_type, owner_id), []):
relative_path = _relative_asset_path(asset, logical_root)
if not relative_path:
continue
target = _safe_local_target(local_root, relative_path)
target.parent.mkdir(parents=True, exist_ok=True)
version, blob = version_blobs[asset.id]
if include_bytes:
try:
data = backend.get_bytes(blob.storage_key) if backend else b""
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
target.write_bytes(data)
else:
target.touch()
local_key = str(target.resolve())
manifest[local_key] = ManagedAttachmentFile(
local_path=local_key,
asset_id=asset.id,
version_id=version.id,
blob_id=blob.id,
display_path=asset.display_path,
relative_path=normalize_logical_path(relative_path),
filename=asset.filename,
owner_type=asset.owner_type,
owner_id=owner_id,
checksum_sha256=blob.checksum_sha256,
size_bytes=blob.size_bytes,
content_type=blob.content_type,
source_provenance=source_provenance_from_metadata(asset.metadata_),
source_revision=source_revision_from_metadata(asset.metadata_),
manifest, prepared_by_id, prepared_by_old_path, first_prepared = _prepare_managed_base_paths(
base_paths,
materialized_root=materialized_root,
assets_by_owner=assets_by_owner,
version_blobs=version_blobs,
backend=backend,
include_bytes=include_bytes,
linked_asset_ids=linked_asset_ids,
)
_rewrite_managed_attachment_rules(
attachments,
prepared_json,
prepared_by_id=prepared_by_id,
prepared_by_old_path=prepared_by_old_path,
first_prepared=first_prepared,
)
for rule in _iter_rule_dicts(attachments, prepared_json):
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
if selected is None:
continue
base_path_id, local_root_string = selected
rule["base_path_id"] = base_path_id
rule["base_dir"] = local_root_string
if first_prepared is not None:
attachments["base_path"] = first_prepared[1]
snapshot_path = destination / "campaign.json"
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
@@ -262,6 +466,7 @@ def prepare_campaign_snapshot(
raw_json=prepared_json,
managed_files_by_local_path=manifest,
shared_assets=shared_assets,
candidate_assets=candidate_assets,
)
@@ -273,7 +478,10 @@ def prepared_campaign_snapshot(
campaign_id: str,
raw_json: dict[str, Any],
include_bytes: bool,
prefix: str = "multimailer-managed-campaign-",
prefix: str = "govoplan-managed-campaign-",
include_unlinked_candidates: bool = False,
user_id: str = "",
is_admin: bool = False,
) -> Iterator[PreparedCampaignSnapshot]:
temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
try:
@@ -284,6 +492,9 @@ def prepared_campaign_snapshot(
raw_json=raw_json,
destination=temp_dir,
include_bytes=include_bytes,
include_unlinked_candidates=include_unlinked_candidates,
user_id=user_id,
is_admin=is_admin,
)
finally:
shutil.rmtree(temp_dir, ignore_errors=True)
@@ -301,6 +512,53 @@ def managed_match_payloads(
return payloads
def share_assets_with_campaign(
session: Session,
*,
tenant_id: str,
campaign_id: str,
file_ids: list[str],
user_id: str,
is_admin: bool = False,
) -> list[dict[str, Any]]:
unique_ids = list(dict.fromkeys(file_id for file_id in file_ids if file_id))
if not unique_ids:
return []
assets = [
get_asset_for_user(
session,
tenant_id=tenant_id,
user_id=user_id,
asset_id=file_id,
require_write=True,
is_admin=is_admin,
)
for file_id in unique_ids
]
shares = share_files(
session,
tenant_id=tenant_id,
assets=assets,
target_type="campaign",
target_id=campaign_id,
permission="read",
user_id=user_id,
)
session.flush()
return [
{
"id": share.id,
"file_asset_id": share.file_asset_id,
"target_type": share.target_type,
"target_id": share.target_id,
"permission": share.permission,
"created_at": share.created_at.isoformat() if share.created_at else None,
"revoked_at": share.revoked_at.isoformat() if share.revoked_at else None,
}
for share in shares
]
def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
"""Return an attachment summary without temporary materialization paths.

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
from collections import defaultdict
from dataclasses import dataclass, field
from pathlib import PurePosixPath
from typing import Any, Iterable
@@ -24,6 +25,15 @@ def _candidate_match_keys(raw_match: str) -> set[str]:
AttachmentUseKey = tuple[str, str, str, str]
@dataclass(slots=True)
class _AttachmentBatchRefs:
managed_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
asset_ids: set[str] = field(default_factory=set)
version_ids: set[str] = field(default_factory=set)
blob_ids: set[str] = field(default_factory=set)
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
return job_id, file_version_id, filename_used, stage
@@ -134,22 +144,48 @@ def record_campaign_attachment_uses_for_jobs(
if not job_list:
return
managed_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
refs = _collect_attachment_batch_refs(job_list)
job_by_id = {job.id: job for job in job_list}
asset_ids: set[str] = set()
version_ids: set[str] = set()
blob_ids: set[str] = set()
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
assets_by_id, versions_by_id, blobs_by_id = _load_managed_attachment_entities(session, refs)
for job in job_list:
_record_managed_attachment_uses(
session,
job_by_id=job_by_id,
managed_by_job=refs.managed_by_job,
assets_by_id=assets_by_id,
versions_by_id=versions_by_id,
blobs_by_id=blobs_by_id,
stage=stage,
known_keys=known_keys,
)
_record_fallback_attachment_uses(
session,
job_by_id=job_by_id,
fallback_attachments_by_job=refs.fallback_attachments_by_job,
stage=stage,
known_keys=known_keys,
)
def _collect_attachment_batch_refs(jobs: list[CampaignJobLike]) -> _AttachmentBatchRefs:
refs = _AttachmentBatchRefs()
for job in jobs:
attachments = job.resolved_attachments or []
if not isinstance(attachments, list):
continue
for attachment in attachments:
if not isinstance(attachment, dict):
continue
if not _collect_managed_attachment_refs(refs, job.id, attachment):
refs.fallback_attachments_by_job[job.id].append(attachment)
return refs
def _collect_managed_attachment_refs(refs: _AttachmentBatchRefs, job_id: str, attachment: dict[str, object]) -> bool:
managed_matches = attachment.get("managed_matches")
if isinstance(managed_matches, list) and managed_matches:
if not isinstance(managed_matches, list) or not managed_matches:
return False
for item in managed_matches:
if not isinstance(item, dict):
continue
@@ -158,38 +194,41 @@ def record_campaign_attachment_uses_for_jobs(
blob_id = str(item.get("blob_id") or "")
if not asset_id or not version_id or not blob_id:
continue
managed_by_job[job.id].append(item)
asset_ids.add(asset_id)
version_ids.add(version_id)
blob_ids.add(blob_id)
else:
fallback_attachments_by_job[job.id].append(attachment)
refs.managed_by_job[job_id].append(item)
refs.asset_ids.add(asset_id)
refs.version_ids.add(version_id)
refs.blob_ids.add(blob_id)
return True
assets_by_id = {
item.id: item
for item in session.query(FileAsset).filter(FileAsset.id.in_(asset_ids)).all()
} if asset_ids else {}
versions_by_id = {
item.id: item
for item in session.query(FileVersion).filter(FileVersion.id.in_(version_ids)).all()
} if version_ids else {}
blobs_by_id = {
item.id: item
for item in session.query(FileBlob).filter(FileBlob.id.in_(blob_ids)).all()
} if blob_ids else {}
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
def _load_managed_attachment_entities(
session: Session,
refs: _AttachmentBatchRefs,
) -> tuple[dict[str, FileAsset], dict[str, FileVersion], dict[str, FileBlob]]:
assets_by_id = {item.id: item for item in session.query(FileAsset).filter(FileAsset.id.in_(refs.asset_ids)).all()} if refs.asset_ids else {}
versions_by_id = {item.id: item for item in session.query(FileVersion).filter(FileVersion.id.in_(refs.version_ids)).all()} if refs.version_ids else {}
blobs_by_id = {item.id: item for item in session.query(FileBlob).filter(FileBlob.id.in_(refs.blob_ids)).all()} if refs.blob_ids else {}
return assets_by_id, versions_by_id, blobs_by_id
def _record_managed_attachment_uses(
session: Session,
*,
job_by_id: dict[str, CampaignJobLike],
managed_by_job: dict[str, list[dict[str, object]]],
assets_by_id: dict[str, FileAsset],
versions_by_id: dict[str, FileVersion],
blobs_by_id: dict[str, FileBlob],
stage: str,
known_keys: set[AttachmentUseKey],
) -> None:
for job_id, items in managed_by_job.items():
job = job_by_id[job_id]
for item in items:
asset = assets_by_id.get(str(item.get("asset_id") or ""))
version = versions_by_id.get(str(item.get("version_id") or ""))
blob = blobs_by_id.get(str(item.get("blob_id") or ""))
if not asset or not version or not blob:
continue
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
continue
if version.file_asset_id != asset.id or version.blob_id != blob.id:
if not _managed_attachment_entities_match_job(job, asset, version, blob):
continue
_add_use(
session,
@@ -202,29 +241,79 @@ def record_campaign_attachment_uses_for_jobs(
known_keys=known_keys,
)
def _managed_attachment_entities_match_job(
job: CampaignJobLike,
asset: FileAsset | None,
version: FileVersion | None,
blob: FileBlob | None,
) -> bool:
if not asset or not version or not blob:
return False
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
return False
return version.file_asset_id == asset.id and version.blob_id == blob.id
def _record_fallback_attachment_uses(
session: Session,
*,
job_by_id: dict[str, CampaignJobLike],
fallback_attachments_by_job: dict[str, list[dict[str, object]]],
stage: str,
known_keys: set[AttachmentUseKey],
) -> None:
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {}
for job_id, attachments in fallback_attachments_by_job.items():
job = job_by_id[job_id]
campaign_key = (job.tenant_id, job.campaign_id)
by_key, version_blobs = _fallback_campaign_assets(
session,
job,
campaign_key=campaign_key,
assets_by_campaign=assets_by_campaign,
version_blobs_by_campaign=version_blobs_by_campaign,
)
for attachment in attachments:
_record_fallback_attachment(session, job, attachment, by_key=by_key, version_blobs=version_blobs, stage=stage, known_keys=known_keys)
def _fallback_campaign_assets(
session: Session,
job: CampaignJobLike,
*,
campaign_key: tuple[str, str],
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]],
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]],
) -> tuple[dict[str, FileAsset], dict[str, tuple[FileVersion, FileBlob]]]:
by_key = assets_by_campaign.get(campaign_key)
if by_key is None:
assets = list_assets_for_user(
session,
tenant_id=job.tenant_id,
user_id="",
campaign_id=job.campaign_id,
is_admin=True,
)
by_key = {}
assets = list_assets_for_user(session, tenant_id=job.tenant_id, user_id="", campaign_id=job.campaign_id, is_admin=True)
by_key = _asset_lookup_by_path_and_filename(assets)
assets_by_campaign[campaign_key] = by_key
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
return by_key, version_blobs_by_campaign[campaign_key]
def _asset_lookup_by_path_and_filename(assets: list[FileAsset]) -> dict[str, FileAsset]:
by_key: dict[str, FileAsset] = {}
for asset in assets:
by_key[asset.display_path.strip("/")] = asset
by_key.setdefault(asset.filename, asset)
assets_by_campaign[campaign_key] = by_key
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
version_blobs = version_blobs_by_campaign[campaign_key]
return by_key
for attachment in attachments:
def _record_fallback_attachment(
session: Session,
job: CampaignJobLike,
attachment: dict[str, object],
*,
by_key: dict[str, FileAsset],
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
stage: str,
known_keys: set[AttachmentUseKey],
) -> None:
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
for raw in matches:
if not isinstance(raw, str):

View File

@@ -1,6 +1,6 @@
from __future__ import annotations
import os
import json
from collections.abc import Mapping
from dataclasses import dataclass, field
from datetime import datetime, timezone
@@ -9,10 +9,24 @@ from importlib import import_module
import mimetypes
from typing import Any
from urllib.parse import quote, unquote, urljoin, urlsplit
import xml.etree.ElementTree as ET
import xml.etree.ElementTree as ET # nosec B405 - typing/element creation only; parsing uses defusedxml below.
import httpx
from defusedxml import ElementTree as SafeElementTree
from govoplan_core.security.outbound_http import (
OutboundHttpError,
outbound_http_policy,
pinned_outbound_hostname,
validate_unpinned_sdk_http_url,
)
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
from govoplan_files.backend.storage.connector_deployment import (
ConnectorDeploymentConfigurationError,
connector_ca_bundle_path,
connector_secret_env_value,
validate_connector_tls_metadata,
)
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
@@ -52,7 +66,7 @@ class ConnectorBrowseItem:
}
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None) -> list[ConnectorBrowseItem]:
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None, continuation_token: str | None = None) -> list[ConnectorBrowseItem]:
browse_path = normalize_connector_browse_path(path)
static_items = _static_listing(profile, path=browse_path, library_id=library_id)
if static_items is not None:
@@ -65,13 +79,15 @@ def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = No
return _browse_webdav(profile, path=browse_path)
if profile.provider == "smb":
return _browse_smb(profile, path=browse_path)
if profile.provider == "s3":
return _browse_s3(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet")
def parse_webdav_multistatus(*, root_url: str, current_path: str, payload: str | bytes) -> list[ConnectorBrowseItem]:
try:
root = ET.fromstring(payload)
except ET.ParseError as exc:
root = SafeElementTree.fromstring(payload)
except SafeElementTree.ParseError as exc:
raise ConnectorBrowseError("Connector returned invalid WebDAV XML") from exc
root_path = _url_path_root(root_url)
current = normalize_connector_browse_path(current_path)
@@ -167,14 +183,22 @@ def _browse_webdav(profile: ConnectorProfile, *, path: str) -> list[ConnectorBro
</prop>
</propfind>"""
try:
response = httpx.request("PROPFIND", url, headers=headers, content=body, auth=auth, timeout=15.0)
except httpx.HTTPError as exc:
response = request_connector_bytes(
"PROPFIND",
url,
headers=headers,
content=body,
auth=auth,
timeout=15.0,
label="WebDAV browse",
)
except ConnectorHttpError as exc:
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
if response.status_code in {401, 403}:
raise ConnectorBrowseError("Connector credentials were rejected")
if response.status_code not in {200, 207}:
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.text)
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.content)
def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
@@ -221,6 +245,244 @@ def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowse
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
def _browse_s3(profile: ConnectorProfile, *, path: str, library_id: str | None, continuation_token: str | None) -> list[ConnectorBrowseItem]:
client = _s3_client(profile)
bucket = _s3_bucket(profile, library_id)
if not bucket:
try:
payload = client.list_buckets()
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
buckets = payload.get("Buckets") if isinstance(payload, Mapping) else None
if not isinstance(buckets, list):
raise ConnectorBrowseError("S3 connector returned invalid bucket list")
return sorted(
(
ConnectorBrowseItem(
kind="library",
name=_clean(item.get("Name")) or "",
path=_clean(item.get("Name")) or "",
external_id=_clean(item.get("Name")),
modified_at=_timestamp(item.get("CreationDate")),
metadata={"bucket": _clean(item.get("Name"))},
)
for item in buckets
if isinstance(item, Mapping) and _clean(item.get("Name"))
),
key=lambda item: item.name.casefold(),
)
prefix = _s3_object_key(profile, path, directory=True)
params: dict[str, object] = {
"Bucket": bucket,
"Prefix": prefix,
"Delimiter": "/",
"MaxKeys": _s3_max_keys(profile),
}
next_page_request = _clean(continuation_token) or _metadata_string(profile, "continuation_token")
if next_page_request:
params["ContinuationToken"] = next_page_request
try:
payload = client.list_objects_v2(**params)
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
if not isinstance(payload, Mapping):
raise ConnectorBrowseError("S3 connector returned invalid object listing")
items = [
*_s3_prefix_items(bucket=bucket, browse_path=path, base_prefix=prefix, prefixes=payload.get("CommonPrefixes")),
*_s3_object_items(bucket=bucket, browse_path=path, base_prefix=prefix, objects=payload.get("Contents")),
]
next_token = _clean(payload.get("NextContinuationToken"))
if next_token and items:
last = items[-1]
items[-1] = ConnectorBrowseItem(
kind=last.kind,
name=last.name,
path=last.path,
external_id=last.external_id,
external_url=last.external_url,
size_bytes=last.size_bytes,
content_type=last.content_type,
modified_at=last.modified_at,
etag=last.etag,
metadata={**dict(last.metadata), "next_continuation_token": next_token, "listing_truncated": True},
)
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
def _s3_client(profile: ConnectorProfile) -> Any:
if profile.secret_ref:
raise ConnectorBrowseError("Secret-ref S3 credentials need a runtime secret resolver before live browsing")
try:
boto3 = import_module("boto3")
config_module = import_module("botocore.config")
except ImportError as exc:
raise ConnectorBrowseUnsupported("S3 connector browsing requires the optional boto3 dependency") from exc
kwargs: dict[str, object] = {}
if profile.endpoint_url:
try:
kwargs["endpoint_url"] = validate_unpinned_sdk_http_url(
profile.endpoint_url,
label="S3 connector endpoint",
)
except OutboundHttpError as exc:
raise ConnectorBrowseError(str(exc)) from exc
elif not outbound_http_policy().allow_private_networks:
raise ConnectorBrowseError(
"S3 connector endpoint discovery cannot pin the SDK connection address while private-network access is disabled"
)
region = _metadata_string(profile, "region") or _metadata_string(profile, "aws_region")
if region:
kwargs["region_name"] = region
access_key = profile.username or _metadata_env(profile, "access_key_id_env") or _metadata_string(profile, "access_key_id")
secret_key = _profile_password(profile) or _metadata_env(profile, "secret_access_key_env")
session_token = _profile_token(profile) or _metadata_env(profile, "session_token_env")
if access_key:
kwargs["aws_access_key_id"] = access_key
if secret_key:
kwargs["aws_secret_access_key"] = secret_key
if session_token:
kwargs["aws_session_token"] = session_token
verify = _s3_verify(profile)
if verify is not None:
kwargs["verify"] = verify
addressing_style = _s3_addressing_style(profile)
if addressing_style:
kwargs["config"] = config_module.Config(s3={"addressing_style": addressing_style})
try:
return boto3.client("s3", **kwargs)
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorBrowseError(f"S3 connector could not be initialized: {exc}") from exc
def _s3_bucket(profile: ConnectorProfile, library_id: str | None = None) -> str | None:
return _metadata_string(profile, "bucket") or _metadata_string(profile, "bucket_name") or _clean(library_id)
def _s3_object_key(profile: ConnectorProfile, path: str, *, directory: bool = False) -> str:
base_prefix = normalize_connector_browse_path(profile.base_path or _metadata_string(profile, "base_prefix"))
browse_path = normalize_connector_browse_path(path)
key = "/".join(part for part in (base_prefix, browse_path) if part)
if directory and key:
return key.rstrip("/") + "/"
return key
def _s3_prefix_items(
*,
bucket: str,
browse_path: str,
base_prefix: str,
prefixes: object,
) -> list[ConnectorBrowseItem]:
if not isinstance(prefixes, list):
return []
items: list[ConnectorBrowseItem] = []
for item in prefixes:
if not isinstance(item, Mapping):
continue
key = _clean(item.get("Prefix"))
if not key:
continue
relative = _s3_relative_key(key, base_prefix=base_prefix)
name = _path_name(relative)
if not name:
continue
path = _join_browse_path(browse_path, name)
items.append(
ConnectorBrowseItem(
kind="folder",
name=name,
path=path,
external_id=f"{bucket}:{key}",
metadata={"bucket": bucket, "key": key},
)
)
return items
def _s3_object_items(
*,
bucket: str,
browse_path: str,
base_prefix: str,
objects: object,
) -> list[ConnectorBrowseItem]:
if not isinstance(objects, list):
return []
items: list[ConnectorBrowseItem] = []
for item in objects:
if not isinstance(item, Mapping):
continue
key = _clean(item.get("Key"))
if not key or key == base_prefix:
continue
relative = _s3_relative_key(key, base_prefix=base_prefix)
name = _path_name(relative)
if not name:
continue
path = _join_browse_path(browse_path, name)
items.append(
ConnectorBrowseItem(
kind="file",
name=name,
path=path,
external_id=f"{bucket}:{key}",
size_bytes=_int(item.get("Size")),
content_type=mimetypes.guess_type(name)[0],
modified_at=_timestamp(item.get("LastModified")),
etag=_clean(item.get("ETag")),
metadata={
"bucket": bucket,
"key": key,
**({"storage_class": item["StorageClass"]} if "StorageClass" in item else {}),
},
)
)
return items
def _s3_relative_key(key: str, *, base_prefix: str) -> str:
clean_key = key.rstrip("/")
clean_base = base_prefix.rstrip("/")
if clean_base and clean_key.startswith(f"{clean_base}/"):
return clean_key[len(clean_base) + 1 :]
return clean_key
def _s3_max_keys(profile: ConnectorProfile) -> int:
configured = _int(profile.metadata.get("max_keys"))
if configured is None:
return 1000
return max(1, min(configured, 1000))
def _s3_verify(profile: ConnectorProfile) -> bool | str | None:
try:
validate_connector_tls_metadata(profile.metadata)
except ConnectorDeploymentConfigurationError as exc:
raise ConnectorBrowseError(str(exc)) from exc
ca_bundle = _metadata_string(profile, "ca_bundle")
if ca_bundle:
try:
return connector_ca_bundle_path(ca_bundle)
except ConnectorDeploymentConfigurationError as exc:
raise ConnectorBrowseError(str(exc)) from exc
if "verify_tls" in profile.metadata:
return _metadata_bool(profile, "verify_tls", default=True)
if "tls_verify" in profile.metadata:
return _metadata_bool(profile, "tls_verify", default=True)
return None
def _s3_addressing_style(profile: ConnectorProfile) -> str | None:
style = _metadata_string(profile, "addressing_style")
if style in {"path", "virtual", "auto"}:
return style
if _metadata_bool(profile, "path_style", default=False):
return "path"
return None
def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]:
token = _seafile_token(profile)
return {"Authorization": f"Token {token}", "Accept": "application/json"}
@@ -260,16 +522,24 @@ def _request_json(
data: Mapping[str, str] | None = None,
) -> object:
try:
response = httpx.request(method, url, headers=dict(headers or {}), params=params, data=data, timeout=15.0)
except httpx.HTTPError as exc:
response = request_connector_bytes(
method,
url,
headers=dict(headers or {}),
params=params,
data=data,
timeout=15.0,
label="Seafile API",
)
except ConnectorHttpError as exc:
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
if response.status_code in {401, 403}:
raise ConnectorBrowseError("Connector credentials were rejected")
if response.status_code not in {200, 201}:
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
try:
return response.json()
except ValueError as exc:
return json.loads(response.content)
except (UnicodeDecodeError, ValueError) as exc:
raise ConnectorBrowseError("Connector returned invalid JSON") from exc
@@ -443,11 +713,18 @@ def _metadata_bool(profile: ConnectorProfile, key: str, *, default: bool = False
return str(value).strip().casefold() in {"1", "true", "yes", "on"}
def _env_required(name: str, profile_id: str) -> str:
value = _clean(os.environ.get(name))
if not value:
raise ConnectorBrowseError(f"Connector profile {profile_id} is missing required credential environment")
return value
def _metadata_env(profile: ConnectorProfile, key: str) -> str | None:
env_name = _metadata_string(profile, key)
if not env_name:
return None
return _env_required(env_name, profile)
def _env_required(name: str, profile: ConnectorProfile) -> str:
try:
return connector_secret_env_value(name, source_kind=profile.source_kind)
except ConnectorDeploymentConfigurationError as exc:
raise ConnectorBrowseError(f"Connector profile {profile.id}: {exc}") from exc
def normalize_connector_browse_path(value: object) -> str:
@@ -504,6 +781,11 @@ def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
server = _clean(parsed.hostname)
if not server:
raise ConnectorBrowseError("SMB connector endpoint_url must include a server")
port = parsed.port or _int(profile.metadata.get("port")) or 445
try:
server = pinned_outbound_hostname(server, port=port, label="SMB connector endpoint")
except OutboundHttpError as exc:
raise ConnectorBrowseError(str(exc)) from exc
path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part]
if not path_parts:
raise ConnectorBrowseError("SMB connector endpoint_url must include a share name")
@@ -513,7 +795,7 @@ def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
return _SmbLocation(
server=server,
share=path_parts[0],
port=parsed.port or _int(profile.metadata.get("port")) or 445,
port=port,
root_path=normalize_connector_browse_path("/".join(root_parts)),
)
@@ -549,7 +831,7 @@ def _profile_password(profile: ConnectorProfile) -> str | None:
if profile.password_value:
return profile.password_value
if profile.password_env:
return _env_required(profile.password_env, profile.id)
return _env_required(profile.password_env, profile)
return None
@@ -557,7 +839,7 @@ def _profile_token(profile: ConnectorProfile) -> str | None:
if profile.token_value:
return profile.token_value
if profile.token_env:
return _env_required(profile.token_env, profile.id)
return _env_required(profile.token_env, profile)
return None

View File

@@ -10,6 +10,7 @@ from govoplan_core.core.policy import normalize_policy_scope_type, policy_source
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
@@ -53,7 +54,9 @@ class ConnectorCredential:
def credentials_configured(self) -> bool:
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
return True
return bool(self.secret_ref or self.password_value or self.token_value or self.password_env or self.token_env)
# Environment references are deliberately unavailable to API-managed
# credential rows. Legacy rows remain visible but fail closed.
return bool(self.secret_ref or self.password_value or self.token_value)
def to_response(self) -> dict[str, Any]:
return {
@@ -182,6 +185,11 @@ def create_connector_credential_row(
policy: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None,
) -> FileConnectorCredential:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
metadata=metadata,
)
clean_id = _normalize_id(credential_id)
if session.get(FileConnectorCredential, clean_id) is not None:
raise FileStorageError(f"Connector credential already exists: {clean_id}")
@@ -231,6 +239,11 @@ def update_connector_credential_row(
clear_password: bool = False,
clear_token: bool = False,
) -> FileConnectorCredential:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
metadata=metadata,
)
if label is not None:
row.label = _normalize_label(label)
if provider is not None:

View File

@@ -0,0 +1,199 @@
from __future__ import annotations
import os
import re
from collections.abc import Mapping
from pathlib import Path
from typing import Any
_SECRET_ENV_ALLOWLIST = "GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST" # noqa: S105 # nosec B105 - configuration key.
_CA_BUNDLE_ALLOWLIST = "GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST"
_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
_SECRET_ENV_METADATA_KEYS = frozenset(
{
"access_key_id_env",
"secret_access_key_env",
"session_token_env",
}
)
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
class ConnectorDeploymentConfigurationError(ValueError):
"""Raised when connector data crosses a deployment-owned trust boundary."""
def connector_secret_env_value(name: str, *, source_kind: str) -> str:
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
value = os.environ.get(clean_name)
if value is None or value == "":
raise ConnectorDeploymentConfigurationError("Connector credential environment variable is not configured")
return value
def connector_secret_env_available(name: str | None, *, source_kind: str) -> bool:
if not name:
return False
try:
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
except ConnectorDeploymentConfigurationError:
return False
return bool(os.environ.get(clean_name))
def validate_deployment_connector_references(
*,
source_kind: str,
password_env: str | None = None,
token_env: str | None = None,
metadata: Mapping[str, Any] | None = None,
) -> None:
"""Validate references in deployment-owned connector configuration."""
for name in (password_env, token_env):
if name:
_validate_secret_env_reference(name, source_kind=source_kind)
for key in _SECRET_ENV_METADATA_KEYS:
name = _clean((metadata or {}).get(key))
if name:
_validate_secret_env_reference(name, source_kind=source_kind)
validate_connector_tls_metadata(metadata)
def reject_api_controlled_deployment_references(
*,
password_env: str | None = None,
token_env: str | None = None,
metadata: Mapping[str, Any] | None = None,
) -> None:
"""Reject process-secret selectors controlled through tenant-facing APIs."""
if _clean(password_env) or _clean(token_env):
raise ConnectorDeploymentConfigurationError(
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
)
for key, value in (metadata or {}).items():
if _clean(value) and (str(key).strip().casefold() in _SECRET_ENV_METADATA_KEYS or str(key).strip().casefold().endswith("_env")):
raise ConnectorDeploymentConfigurationError(
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
)
validate_connector_tls_metadata(metadata)
def connector_ca_bundle_path(value: str) -> str:
clean_value = _clean(value)
if not clean_value:
raise ConnectorDeploymentConfigurationError("Connector CA bundle path is empty")
candidate = Path(clean_value)
if not candidate.is_absolute():
raise ConnectorDeploymentConfigurationError("Connector CA bundle paths must be absolute")
try:
resolved = candidate.resolve(strict=True)
except OSError as exc:
raise ConnectorDeploymentConfigurationError("Connector CA bundle path does not exist") from exc
allowed = _allowed_ca_bundle_paths()
if resolved not in allowed:
raise ConnectorDeploymentConfigurationError(
f"Connector CA bundle path is not listed in {_CA_BUNDLE_ALLOWLIST}"
)
if not resolved.is_file():
raise ConnectorDeploymentConfigurationError("Connector CA bundle path must be a regular file")
return str(resolved)
def validate_connector_tls_metadata(metadata: Mapping[str, Any] | None) -> None:
values = metadata or {}
ca_bundle = _clean(values.get("ca_bundle"))
if ca_bundle:
connector_ca_bundle_path(ca_bundle)
for key in ("verify_tls", "tls_verify"):
if key in values and not _as_bool(values.get(key), default=True) and not _development_runtime():
raise ConnectorDeploymentConfigurationError(
"Connector TLS certificate verification may only be disabled in dev/test environments"
)
def connector_effective_endpoint_url(
*,
provider: str | None,
endpoint_url: str | None,
metadata: Mapping[str, Any] | None,
) -> str | None:
"""Return the endpoint that connector I/O will actually use."""
clean_provider = (provider or "").strip().casefold()
values = metadata or {}
webdav_url = _clean(values.get("webdav_endpoint_url"))
browse_protocol = (_clean(values.get("browse_protocol")) or "").casefold()
if webdav_url and (clean_provider in {"seafile", "webdav", "nextcloud"} or browse_protocol == "webdav"):
return webdav_url
return _clean(endpoint_url)
def _validate_secret_env_reference(name: str, *, source_kind: str) -> str:
if source_kind.strip().casefold() != "settings":
raise ConnectorDeploymentConfigurationError(
"Environment-backed credentials may only be used by deployment-owned connector configuration"
)
clean_name = name.strip()
if not _ENV_NAME.fullmatch(clean_name):
raise ConnectorDeploymentConfigurationError("Connector credential environment variable name is invalid")
if clean_name not in _allowed_secret_env_names():
raise ConnectorDeploymentConfigurationError(
f"Connector credential environment variable is not listed in {_SECRET_ENV_ALLOWLIST}"
)
return clean_name
def _allowed_secret_env_names() -> frozenset[str]:
raw = os.environ.get(_SECRET_ENV_ALLOWLIST, "")
names = frozenset(item.strip() for item in raw.split(",") if item.strip())
invalid = sorted(name for name in names if not _ENV_NAME.fullmatch(name))
if invalid:
raise ConnectorDeploymentConfigurationError(f"{_SECRET_ENV_ALLOWLIST} contains an invalid variable name")
return names
def _allowed_ca_bundle_paths() -> frozenset[Path]:
raw = os.environ.get(_CA_BUNDLE_ALLOWLIST, "")
paths: set[Path] = set()
for item in (part.strip() for part in raw.split(",")):
if not item:
continue
path = Path(item)
if not path.is_absolute():
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} requires absolute paths")
try:
paths.add(path.resolve(strict=True))
except OSError as exc:
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} contains a missing path") from exc
return frozenset(paths)
def _development_runtime() -> bool:
app_env = os.environ.get("APP_ENV", "dev").strip().casefold()
install_profile = os.environ.get("GOVOPLAN_INSTALL_PROFILE", "").strip().casefold()
return app_env in _DEVELOPMENT_ENVIRONMENTS and (
not install_profile or install_profile in _DEVELOPMENT_ENVIRONMENTS
)
def _as_bool(value: object, *, default: bool) -> bool:
if value is None:
return default
if isinstance(value, bool):
return value
clean = str(value).strip().casefold()
if clean in {"1", "true", "yes", "on"}:
return True
if clean in {"0", "false", "no", "off"}:
return False
raise ConnectorDeploymentConfigurationError("Connector TLS verification setting must be true or false")
def _clean(value: object) -> str | None:
if value is None:
return None
clean = str(value).strip()
return clean or None

View File

@@ -4,7 +4,7 @@ from dataclasses import dataclass, field
import mimetypes
from typing import Any
import httpx
from govoplan_core.security.outbound_http import response_limit
from govoplan_files.backend.storage.connector_browse import (
ConnectorBrowseError,
@@ -21,12 +21,16 @@ from govoplan_files.backend.storage.connector_browse import (
_smb_stat_size,
_smb_unc_path,
_smbclient_module,
_s3_bucket,
_s3_client,
_s3_object_key,
_seafile_headers,
_seafile_url,
_webdav_url,
normalize_connector_browse_path,
)
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
from govoplan_files.backend.storage.paths import filename_from_path
@@ -56,6 +60,7 @@ def read_connector_file(
path: str,
max_bytes: int,
) -> ConnectorDownloadedFile:
max_bytes = min(max_bytes, response_limit("file"))
if profile.provider == "seafile":
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
@@ -64,6 +69,8 @@ def read_connector_file(
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
if profile.provider == "smb":
return _read_smb_file(profile, path=path, max_bytes=max_bytes)
if profile.provider == "s3":
return _read_s3_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet")
@@ -97,8 +104,15 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
if not isinstance(download_url, str) or not download_url.strip():
raise ConnectorImportError("Seafile did not return a file download URL")
try:
response = httpx.request("GET", download_url, timeout=30.0)
except httpx.HTTPError as exc:
response = request_connector_bytes(
"GET",
download_url,
timeout=30.0,
kind="file",
max_bytes=max_bytes,
label="Seafile file download",
)
except ConnectorHttpError as exc:
raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc
if response.status_code != 200:
raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}")
@@ -144,8 +158,17 @@ def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import")
try:
response = httpx.request("GET", url, headers=headers, auth=auth, timeout=30.0)
except httpx.HTTPError as exc:
response = request_connector_bytes(
"GET",
url,
headers=headers,
auth=auth,
timeout=30.0,
kind="file",
max_bytes=max_bytes,
label="WebDAV file download",
)
except ConnectorHttpError as exc:
raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc
if response.status_code in {401, 403}:
raise ConnectorImportError("Connector credentials were rejected")
@@ -213,6 +236,114 @@ def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> C
)
def _s3_import_client(profile: ConnectorProfile) -> Any:
try:
return _s3_client(profile)
except ConnectorBrowseUnsupported as exc:
raise ConnectorImportUnsupported(str(exc)) from exc
except ConnectorBrowseError as exc:
raise ConnectorImportError(str(exc)) from exc
def _s3_object_detail(client: Any, *, bucket: str, key: str, max_bytes: int) -> dict[str, Any]:
try:
detail = client.head_object(Bucket=bucket, Key=key)
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorImportError(f"S3 object metadata lookup failed: {exc}") from exc
if not isinstance(detail, dict):
raise ConnectorImportError("S3 connector returned invalid object metadata")
size = _int(detail.get("ContentLength"))
if size is not None and size > max_bytes:
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
return detail
def _download_s3_object(
client: Any,
*,
bucket: str,
key: str,
version_id: str | None,
max_bytes: int,
) -> tuple[Any, bytes]:
request: dict[str, object] = {"Bucket": bucket, "Key": key}
if version_id:
request["VersionId"] = version_id
try:
response = client.get_object(**request)
body = response.get("Body")
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
if len(data) > max_bytes:
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
return response, data
def _s3_download_metadata(
detail: dict[str, Any],
*,
bucket: str,
key: str,
version_id: str | None,
etag: str | None,
size: int,
) -> dict[str, Any]:
metadata: dict[str, Any] = {
"bucket": bucket,
"key": key,
"version_id": version_id,
"etag": etag,
"size": size,
}
for source_key, target_key in (
("ChecksumSHA256", "checksum_sha256"),
("ChecksumCRC32", "checksum_crc32"),
("StorageClass", "storage_class"),
):
if source_key in detail:
metadata[target_key] = detail[source_key]
return metadata
def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
bucket = _s3_bucket(profile, library_id)
if not bucket:
raise ConnectorImportError("S3 import requires a bucket in library_id or profile metadata")
key = _s3_object_key(profile, path)
if not key:
raise ConnectorImportError("S3 import requires an object key")
client = _s3_import_client(profile)
detail = _s3_object_detail(client, bucket=bucket, key=key, max_bytes=max_bytes)
version_id = _clean(detail.get("VersionId"))
response, data = _download_s3_object(
client,
bucket=bucket,
key=key,
version_id=version_id,
max_bytes=max_bytes,
)
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
filename = filename_from_path(key)
return ConnectorDownloadedFile(
filename=filename,
data=data,
content_type=content_type,
revision=version_id or etag or _clean(detail.get("LastModified")),
external_id=f"{bucket}:{key}",
external_url=f"s3://{bucket}/{key}",
metadata=_s3_download_metadata(
detail,
bucket=bucket,
key=key,
version_id=version_id,
etag=etag,
size=len(data),
),
)
def _int(value: object) -> int | None:
if value is None or value == "":
return None

View File

@@ -9,6 +9,7 @@ from govoplan_core.core.policy import normalize_policy_scope_type
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
from govoplan_files.backend.storage.connector_credential_store import connector_credential_from_row, credential_rows_by_id
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
@@ -124,6 +125,11 @@ def create_connector_profile_row(
policy: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None,
) -> FileConnectorProfile:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
metadata=metadata,
)
clean_id = _normalize_profile_id(profile_id)
if session.get(FileConnectorProfile, clean_id) is not None:
raise FileStorageError(f"Connector profile already exists: {clean_id}")
@@ -181,6 +187,11 @@ def update_connector_profile_row(
clear_password: bool = False,
clear_token: bool = False,
) -> FileConnectorProfile:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
metadata=metadata,
)
if label is not None:
row.label = _normalize_label(label)
if provider is not None:

View File

@@ -7,13 +7,26 @@ from dataclasses import dataclass, field
from typing import Any
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
from govoplan_files.backend.storage.connector_deployment import (
connector_secret_env_available,
validate_deployment_connector_references,
)
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
_ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON")
_ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE")
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"}
_INLINE_SECRET_FIELDS = ("password", "token", "api_key", "access_key", "secret_key")
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"}
_INLINE_SECRET_FIELDS = (
"password",
"token",
"api_key",
"access_key",
"access_key_id",
"secret_key",
"secret_access_key",
"session_token",
)
def supported_connector_providers() -> set[str]:
@@ -73,9 +86,9 @@ class ConnectorProfile:
if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value:
return True
if self.token_env:
return bool(os.environ.get(self.token_env))
return connector_secret_env_available(self.token_env, source_kind=self.source_kind)
if self.password_env:
return bool(os.environ.get(self.password_env))
return connector_secret_env_available(self.password_env, source_kind=self.source_kind)
return False
def to_response(self) -> dict[str, Any]:
@@ -97,7 +110,7 @@ class ConnectorProfile:
"username": self.username,
"capabilities": list(self.capabilities),
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
"metadata": dict(self.metadata),
"metadata": _response_metadata(self.metadata),
"source_kind": self.source_kind,
}
@@ -123,45 +136,21 @@ def connector_profiles_from_payload(value: object) -> list[ConnectorProfile]:
def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
if not profile_id:
raise ValueError("Connector profiles require id")
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
if provider not in _SUPPORTED_PROVIDERS:
raise ValueError(f"Unsupported connector provider: {provider}")
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
scope_id = _clean(value.get("scope_id"))
if scope_type == "system":
scope_id = None
elif not scope_id:
raise ValueError(f"{scope_type} connector profiles require scope_id")
credentials = value.get("credentials")
credentials = credentials if isinstance(credentials, Mapping) else {}
profile_id = _profile_id_from_mapping(value)
provider = _provider_from_mapping(value)
scope_type, scope_id = _scope_from_mapping(value)
credentials = _credentials_from_mapping(value)
mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none"
profile = ConnectorProfile(
id=profile_id,
label=_clean(value.get("label") or value.get("name")) or profile_id,
profile = _profile_from_normalized_mapping(
value,
profile_id=profile_id,
provider=provider,
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
enabled=_bool(value.get("enabled"), default=True),
scope_type=scope_type,
scope_id=scope_id,
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
credential_mode=mode,
username=_clean(value.get("username") or credentials.get("username")),
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
password_value=_clean(value.get("password") or credentials.get("password")),
token_value=_clean(value.get("token") or credentials.get("token")),
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
capabilities=tuple(_string_list(value.get("capabilities"))),
metadata=_public_metadata(value.get("metadata")),
credentials=credentials,
)
return ConnectorProfile(
result = ConnectorProfile(
id=profile.id,
label=profile.label,
provider=profile.provider,
@@ -185,6 +174,76 @@ def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
metadata=profile.metadata,
source_kind="settings",
)
validate_deployment_connector_references(
source_kind=result.source_kind,
password_env=result.password_env,
token_env=result.token_env,
metadata=result.metadata,
)
return result
def _profile_id_from_mapping(value: Mapping[str, Any]) -> str:
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
if not profile_id:
raise ValueError("Connector profiles require id")
return profile_id
def _provider_from_mapping(value: Mapping[str, Any]) -> str:
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
if provider not in _SUPPORTED_PROVIDERS:
raise ValueError(f"Unsupported connector provider: {provider}")
return provider
def _scope_from_mapping(value: Mapping[str, Any]) -> tuple[str, str | None]:
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
scope_id = _clean(value.get("scope_id"))
if scope_type == "system":
return scope_type, None
if not scope_id:
raise ValueError(f"{scope_type} connector profiles require scope_id")
return scope_type, scope_id
def _credentials_from_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]:
credentials = value.get("credentials")
return credentials if isinstance(credentials, Mapping) else {}
def _profile_from_normalized_mapping(
value: Mapping[str, Any],
*,
profile_id: str,
provider: str,
scope_type: str,
scope_id: str | None,
credential_mode: str,
credentials: Mapping[str, Any],
) -> ConnectorProfile:
return ConnectorProfile(
id=profile_id,
label=_clean(value.get("label") or value.get("name")) or profile_id,
provider=provider,
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
enabled=_bool(value.get("enabled"), default=True),
scope_type=scope_type,
scope_id=scope_id,
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
credential_mode=credential_mode,
username=_clean(value.get("username") or credentials.get("username")),
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
password_value=_clean(value.get("password") or credentials.get("password")),
token_value=_clean(value.get("token") or credentials.get("token")),
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
capabilities=tuple(_string_list(value.get("capabilities"))),
metadata=_public_metadata(value.get("metadata")),
)
def _raw_profiles_payload(settings: object | None) -> object:
@@ -243,6 +302,16 @@ def _public_metadata(value: object) -> Mapping[str, Any]:
}
def _response_metadata(value: Mapping[str, Any]) -> dict[str, Any]:
return {
str(key): item
for key, item in value.items()
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
and not str(key).strip().casefold().endswith("_env")
and str(key).strip().casefold() != "ca_bundle"
}
def _string_list(value: object) -> list[str]:
if value is None:
return []

View File

@@ -112,6 +112,51 @@ def connector_provider_descriptors() -> tuple[ConnectorProviderDescriptor, ...]:
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Requires the optional smbprotocol dependency and an smb://server[:port]/share[/path] profile endpoint.",
),
ConnectorProviderDescriptor(
provider="s3",
label="S3-compatible object storage",
protocol="s3-api",
implemented=True,
browse_supported=True,
import_supported=True,
optional_dependency="boto3",
permission_model=governed_permissions,
sync_strategy="On-demand bucket/prefix browse and object import/sync; sync updates managed versions and never mutates the remote bucket.",
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after object download.",
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the bucket.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Configure endpoint_url for MinIO or other S3-compatible stores; use profile metadata for bucket, region, addressing_style, verify_tls, and ca_bundle.",
),
ConnectorProviderDescriptor(
provider="sharepoint",
label="SharePoint",
protocol="microsoft-graph/sharepoint",
implemented=False,
browse_supported=False,
import_supported=False,
optional_dependency=None,
permission_model=governed_permissions,
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph or SharePoint credentials.",
conflict_strategy=freeze_model,
preview_strategy="Will preview from frozen managed file after import, not directly from SharePoint.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Provider/profile key is reserved; live browsing/import still needs Graph/SharePoint authentication and paging implementation.",
),
ConnectorProviderDescriptor(
provider="onedrive",
label="OneDrive",
protocol="microsoft-graph/onedrive",
implemented=False,
browse_supported=False,
import_supported=False,
optional_dependency=None,
permission_model=governed_permissions,
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph credentials.",
conflict_strategy=freeze_model,
preview_strategy="Will preview from frozen managed file after import, not directly from OneDrive.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Provider/profile key is reserved; live browsing/import still needs Graph drive selection, OAuth/app registration, and paging implementation.",
),
ConnectorProviderDescriptor(
provider="nfs",
label="NFS",

View File

@@ -0,0 +1,239 @@
from __future__ import annotations
import socket
import ssl
import select
from collections.abc import Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any, Iterator
import httpcore
import httpx
from govoplan_core.security.outbound_http import (
OutboundHttpError,
bounded_chunks_bytes,
create_outbound_connection,
response_limit,
validate_outbound_http_url,
)
class ConnectorHttpError(RuntimeError):
pass
@dataclass(frozen=True, slots=True)
class ConnectorHttpResponse:
status_code: int
headers: Mapping[str, str]
content: bytes
_HTTPCORE_TRANSPORT_ERRORS = (
httpcore.TimeoutException,
httpcore.NetworkError,
httpcore.ProtocolError,
httpcore.ProxyError,
httpcore.UnsupportedProtocol,
)
class _SocketNetworkStream(httpcore.NetworkStream):
"""Public httpcore NetworkStream adapter around an approved socket."""
def __init__(self, sock: socket.socket) -> None:
self._socket = sock
def read(self, max_bytes: int, timeout: float | None = None) -> bytes:
try:
self._socket.settimeout(timeout)
return self._socket.recv(max_bytes)
except socket.timeout as exc:
raise httpcore.ReadTimeout(str(exc)) from exc
except OSError as exc:
raise httpcore.ReadError(str(exc)) from exc
def write(self, buffer: bytes, timeout: float | None = None) -> None:
try:
self._socket.settimeout(timeout)
self._socket.sendall(buffer)
except socket.timeout as exc:
raise httpcore.WriteTimeout(str(exc)) from exc
except OSError as exc:
raise httpcore.WriteError(str(exc)) from exc
def close(self) -> None:
self._socket.close()
def start_tls(
self,
ssl_context: ssl.SSLContext,
server_hostname: str | None = None,
timeout: float | None = None,
) -> httpcore.NetworkStream:
try:
self._socket.settimeout(timeout)
tls_socket = ssl_context.wrap_socket(self._socket, server_hostname=server_hostname)
except socket.timeout as exc:
self.close()
raise httpcore.ConnectTimeout(str(exc)) from exc
except OSError as exc:
self.close()
raise httpcore.ConnectError(str(exc)) from exc
return _SocketNetworkStream(tls_socket)
def get_extra_info(self, info: str) -> Any:
if info == "ssl_object" and isinstance(self._socket, ssl.SSLSocket):
return self._socket
if info == "client_addr":
return self._socket.getsockname()
if info == "server_addr":
return self._socket.getpeername()
if info == "socket":
return self._socket
if info == "is_readable":
try:
return bool(select.select([self._socket], [], [], 0)[0])
except (OSError, ValueError):
return True
return None
class _OutboundPolicyNetworkBackend(httpcore.NetworkBackend):
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options: Any = None,
) -> httpcore.NetworkStream:
source_address = None if local_address is None else (local_address, 0)
try:
sock = create_outbound_connection(
host,
port,
timeout=timeout,
source_address=source_address,
socket_options=socket_options,
label="File connector HTTP endpoint",
)
except socket.timeout as exc:
raise httpcore.ConnectTimeout(str(exc)) from exc
except (OSError, OutboundHttpError) as exc:
raise httpcore.ConnectError(str(exc)) from exc
return _SocketNetworkStream(sock)
def connect_unix_socket(self, path: str, timeout: float | None = None, socket_options: Any = None): # type: ignore[no-untyped-def]
del path, timeout, socket_options
raise httpcore.ConnectError("Unix sockets are not supported for file connectors")
class _HttpcoreResponseStream(httpx.SyncByteStream):
def __init__(self, stream: Any, *, request: httpx.Request) -> None:
self._stream = stream
self._request = request
def __iter__(self): # type: ignore[no-untyped-def]
try:
yield from self._stream
except _HTTPCORE_TRANSPORT_ERRORS as exc:
raise httpx.TransportError(str(exc), request=self._request) from exc
def close(self) -> None:
if hasattr(self._stream, "close"):
self._stream.close()
class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
def __init__(self) -> None:
self._connection_pool = httpcore.ConnectionPool(
ssl_context=httpx.create_ssl_context(verify=True, trust_env=False),
max_connections=100,
max_keepalive_connections=20,
keepalive_expiry=5.0,
network_backend=_OutboundPolicyNetworkBackend(),
)
def handle_request(self, request: httpx.Request) -> httpx.Response:
core_request = httpcore.Request(
method=request.method,
url=httpcore.URL(
scheme=request.url.raw_scheme,
host=request.url.raw_host,
port=request.url.port,
target=request.url.raw_path,
),
headers=request.headers.raw,
content=request.stream,
extensions=request.extensions,
)
try:
response = self._connection_pool.handle_request(core_request)
except _HTTPCORE_TRANSPORT_ERRORS as exc:
raise httpx.TransportError(str(exc), request=request) from exc
return httpx.Response(
status_code=response.status,
headers=response.headers,
stream=_HttpcoreResponseStream(response.stream, request=request),
extensions=response.extensions,
)
def close(self) -> None:
self._connection_pool.close()
@contextmanager
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
with httpx.Client(
transport=_OutboundPolicyHTTPTransport(),
follow_redirects=False,
timeout=kwargs.pop("timeout", 15.0),
) as client:
with client.stream(method, url, **kwargs) as response:
yield response
def request_connector_bytes(
method: str,
url: str,
*,
headers: Mapping[str, str] | None = None,
params: Mapping[str, str] | None = None,
data: Mapping[str, str] | bytes | str | None = None,
content: bytes | str | None = None,
auth: Any = None,
timeout: float = 15.0,
kind: str = "structured",
max_bytes: int | None = None,
label: str = "File connector",
) -> ConnectorHttpResponse:
try:
validated_url = validate_outbound_http_url(url, label=f"{label} URL")
effective_limit = response_limit(kind) if max_bytes is None else min(int(max_bytes), response_limit(kind))
with _stream_connector_request(
method,
validated_url,
headers=dict(headers or {}),
params=params,
data=data,
content=content,
auth=auth,
timeout=timeout,
) as response:
body = bounded_chunks_bytes(
response.iter_bytes(),
headers=response.headers,
max_bytes=effective_limit,
kind=kind,
label=f"{label} response",
)
return ConnectorHttpResponse(
status_code=response.status_code,
headers=dict(response.headers),
content=body,
)
except (httpx.HTTPError, OutboundHttpError, ValueError) as exc:
raise ConnectorHttpError(str(exc)) from exc

View File

@@ -15,21 +15,6 @@ from govoplan_files.backend.storage.common import (
utcnow,
)
from govoplan_files.backend.storage.files import (
_active_asset_at_path,
_active_asset_exists,
_asset_owner_id,
_asset_query_for_owner,
_candidate_renamed_path,
_copy_asset_to_path,
_get_or_create_blob,
_next_available_logical_path,
_normalize_conflict_strategy,
_resolution_by_path,
_soft_delete_conflicting_asset,
_split_logical_path,
_storage_backend_name,
_storage_bucket_name,
_storage_key,
asset_is_audit_relevant,
create_file_asset,
current_version_and_blob,
@@ -42,10 +27,6 @@ from govoplan_files.backend.storage.files import (
soft_delete_assets,
)
from govoplan_files.backend.storage.folders import (
_active_folder_exists,
_ensure_target_folder_hierarchy,
_folder_query_for_owner,
_owner_filter,
create_folder,
list_folders_for_user,
soft_delete_folder,

File diff suppressed because it is too large Load Diff

View File

@@ -21,6 +21,7 @@ GROUP_ID = "group-1"
class FilesAccessProviderTests(unittest.TestCase):
def test_file_access_provider_explains_owner_share_admin_and_missing_resources(self) -> None:
session = _session()
self.addCleanup(_close_session, session)
_seed_access_subjects(session)
owned = FileAsset(id="file-owned", tenant_id=TENANT_ID, owner_type="user", owner_user_id=USER_ID, display_path="owned.pdf", filename="owned.pdf")
shared = FileAsset(id="file-shared", tenant_id=TENANT_ID, owner_type="user", owner_user_id=OTHER_USER_ID, display_path="shared.pdf", filename="shared.pdf")
@@ -46,6 +47,7 @@ class FilesAccessProviderTests(unittest.TestCase):
def test_file_access_provider_explains_persisted_and_virtual_folders(self) -> None:
session = _session()
self.addCleanup(_close_session, session)
_seed_access_subjects(session)
persisted = FileFolder(id="folder-persisted", tenant_id=TENANT_ID, owner_type="group", owner_group_id=GROUP_ID, path="records")
virtual_child = FileAsset(
@@ -80,6 +82,12 @@ def _session():
return sessionmaker(bind=engine, future=True)()
def _close_session(session) -> None:
engine = session.get_bind()
session.close()
engine.dispose()
def _seed_access_subjects(session) -> None:
session.add_all([
Account(id="account-1", email="one@example.test", normalized_email="one@example.test"),

View File

@@ -0,0 +1,196 @@
from __future__ import annotations
import os
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from unittest.mock import MagicMock
from fastapi import HTTPException
from govoplan_files.backend.router import discover_connector_endpoint
from govoplan_files.backend.schemas import FileConnectorDiscoveryRequest
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _profile_password, _s3_verify
from govoplan_files.backend.storage.connector_deployment import (
ConnectorDeploymentConfigurationError,
connector_effective_endpoint_url,
connector_secret_env_value,
reject_api_controlled_deployment_references,
)
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.connector_profile_store import create_connector_profile_row
class ConnectorDeploymentBoundaryTests(unittest.TestCase):
def test_database_profile_cannot_read_arbitrary_process_environment(self) -> None:
profile = ConnectorProfile(
id="tenant-webdav",
label="Tenant WebDAV",
provider="webdav",
password_env="MASTER_KEY_B64",
source_kind="database",
)
with patch.dict(
os.environ,
{
"MASTER_KEY_B64": "must-not-leave-process",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST": "MASTER_KEY_B64",
},
clear=False,
), self.assertRaisesRegex(ConnectorBrowseError, "deployment-owned"):
_profile_password(profile)
def test_deployment_profile_requires_exact_secret_allowlist(self) -> None:
with patch.dict(
os.environ,
{
"GOVOPLAN_FILES_WEBDAV_PASSWORD": "deployment-secret",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST": "GOVOPLAN_FILES_WEBDAV_PASSWORD",
},
clear=False,
):
self.assertEqual(
"deployment-secret",
connector_secret_env_value(
"GOVOPLAN_FILES_WEBDAV_PASSWORD",
source_kind="settings",
),
)
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "not listed"):
connector_secret_env_value("MASTER_KEY_B64", source_kind="settings")
def test_api_profiles_cannot_select_secret_environment_names(self) -> None:
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
reject_api_controlled_deployment_references(password_env="DATABASE_URL")
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
reject_api_controlled_deployment_references(metadata={"secret_access_key_env": "MASTER_KEY_B64"})
session = MagicMock()
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
create_connector_profile_row(
session,
tenant_id="tenant-1",
user_id="user-1",
profile_id="unsafe",
label="Unsafe",
provider="webdav",
scope_type="tenant",
password_env="DATABASE_URL",
)
session.get.assert_not_called()
def test_profile_responses_hide_environment_and_local_ca_references(self) -> None:
profile = ConnectorProfile(
id="deployment-s3",
label="Deployment S3",
provider="s3",
metadata={
"bucket": "documents",
"secret_access_key_env": "GOVOPLAN_FILES_S3_SECRET",
"ca_bundle": "/etc/govoplan/connector-ca.pem",
},
)
self.assertEqual({"bucket": "documents"}, profile.to_response()["metadata"])
def test_ca_bundle_must_be_an_exact_deployment_allowlisted_file(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
allowed = Path(temp_dir, "connector-ca.pem")
other = Path(temp_dir, "other-ca.pem")
allowed.write_text("test CA", encoding="utf-8")
other.write_text("other CA", encoding="utf-8")
with patch.dict(
os.environ,
{
"APP_ENV": "production",
"GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST": str(allowed),
},
clear=False,
):
profile = ConnectorProfile(
id="s3",
label="S3",
provider="s3",
metadata={"ca_bundle": str(allowed)},
)
self.assertEqual(str(allowed.resolve()), _s3_verify(profile))
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "not listed"):
reject_api_controlled_deployment_references(metadata={"ca_bundle": str(other)})
def test_tls_verification_can_only_be_disabled_in_development(self) -> None:
with patch.dict(os.environ, {"APP_ENV": "production"}, clear=False), self.assertRaisesRegex(
ConnectorDeploymentConfigurationError,
"dev/test",
):
reject_api_controlled_deployment_references(metadata={"verify_tls": False})
with patch.dict(os.environ, {"APP_ENV": "test"}, clear=False):
reject_api_controlled_deployment_references(metadata={"verify_tls": False})
def test_effective_endpoint_uses_webdav_override(self) -> None:
self.assertEqual(
"https://dav.example.test/root",
connector_effective_endpoint_url(
provider="seafile",
endpoint_url="https://seafile.example.test",
metadata={"webdav_endpoint_url": "https://dav.example.test/root"},
),
)
class ConnectorDiscoveryBoundaryTests(unittest.TestCase):
@staticmethod
def _principal() -> SimpleNamespace:
return SimpleNamespace(tenant_id="tenant-1", user=SimpleNamespace(id="user-1"), api_key=None)
def test_discovery_rejects_environment_credentials_before_io(self) -> None:
payload = FileConnectorDiscoveryRequest(
provider="webdav",
endpoint_url="https://dav.example.test",
credential_mode="basic",
credentials={"username": "admin", "password_env": "MASTER_KEY_B64"},
)
with patch("govoplan_files.backend.router.browse_connector_profile") as browse, self.assertRaises(
HTTPException
) as raised:
discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
self.assertEqual(400, raised.exception.status_code)
browse.assert_not_called()
def test_discovery_applies_policy_and_audit_before_each_io_candidate(self) -> None:
payload = FileConnectorDiscoveryRequest(
provider="webdav",
endpoint_url="https://dav.example.test/root",
metadata={
"webdav_endpoint_url": "https://bypass.example.test",
"static_listing": {"": []},
},
)
events: list[str] = []
def ensure(*_args: object, **kwargs: object) -> None:
self.assertEqual("https://dav.example.test/root/", kwargs["endpoint_url"])
events.append("policy")
def audit(*_args: object, **_kwargs: object) -> None:
events.append("audit")
def browse(profile: ConnectorProfile, **_kwargs: object) -> list[object]:
self.assertEqual({}, profile.metadata)
events.append("io")
return []
with patch("govoplan_files.backend.router._ensure_connector_configuration_allowed", side_effect=ensure), patch(
"govoplan_files.backend.router._audit_connector_discovery_attempt",
side_effect=audit,
), patch("govoplan_files.backend.router.browse_connector_profile", side_effect=browse):
response = discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
self.assertEqual("usable", response.status)
self.assertEqual(["policy", "audit", "io"], events)
self.assertEqual({"discovered_by": "webdav-propfind"}, response.metadata)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,175 @@
from __future__ import annotations
import unittest
from datetime import UTC, datetime
from unittest.mock import patch
from govoplan_files.backend.storage.connector_browse import _smb_location, browse_connector_profile
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseUnsupported
from govoplan_files.backend.storage.connector_imports import read_connector_file
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, connector_profiles_from_payload
from govoplan_files.backend.storage.connector_providers import connector_provider_descriptors
class FakeBody:
def __init__(self, data: bytes) -> None:
self.data = data
def read(self, _limit: int) -> bytes:
return self.data
class FakeS3Client:
def __init__(self) -> None:
self.list_objects_request: dict[str, object] | None = None
self.head_request: dict[str, object] | None = None
self.get_request: dict[str, object] | None = None
def list_objects_v2(self, **kwargs: object) -> dict[str, object]:
self.list_objects_request = dict(kwargs)
return {
"CommonPrefixes": [{"Prefix": "root/reports/"}],
"Contents": [
{
"Key": "root/report.xlsx",
"Size": 123,
"LastModified": datetime(2026, 7, 12, 10, 0, tzinfo=UTC),
"ETag": '"etag-1"',
"StorageClass": "STANDARD",
}
],
"NextContinuationToken": "next-page",
}
def list_buckets(self) -> dict[str, object]:
return {"Buckets": [{"Name": "archive", "CreationDate": datetime(2026, 7, 12, 9, 0, tzinfo=UTC)}]}
def head_object(self, **kwargs: object) -> dict[str, object]:
self.head_request = dict(kwargs)
return {
"ContentLength": 13,
"ContentType": "text/plain",
"ETag": '"etag-2"',
"VersionId": "version-1",
"ChecksumSHA256": "checksum",
}
def get_object(self, **kwargs: object) -> dict[str, object]:
self.get_request = dict(kwargs)
return {"Body": FakeBody(b"hello s3\n"), "ContentType": "text/plain", "ETag": '"etag-2"'}
def s3_profile(**overrides: object) -> ConnectorProfile:
values = {
"id": "dev-s3",
"label": "Dev S3",
"provider": "s3",
"endpoint_url": "http://127.0.0.1:9000",
"base_path": "root",
"credential_mode": "basic",
"username": "access-key",
"password_value": "secret-key",
"metadata": {"bucket": "govoplan", "region": "eu-central-1", "path_style": True},
}
values.update(overrides)
return ConnectorProfile(**values)
class ConnectorProviderTests(unittest.TestCase):
def test_smb_uses_validated_numeric_target_when_private_networks_are_disabled(self) -> None:
profile = ConnectorProfile(
id="public-smb",
label="Public SMB",
provider="smb",
endpoint_url="smb://files.example.test/share",
)
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("93.184.216.34", 445))],
):
location = _smb_location(profile)
self.assertEqual("93.184.216.34", location.server)
def test_provider_descriptors_include_s3_and_reserved_microsoft_providers(self) -> None:
descriptors = {descriptor.provider: descriptor for descriptor in connector_provider_descriptors()}
self.assertTrue(descriptors["s3"].implemented)
self.assertTrue(descriptors["s3"].browse_supported)
self.assertEqual("boto3", descriptors["s3"].optional_dependency)
self.assertFalse(descriptors["sharepoint"].implemented)
self.assertFalse(descriptors["onedrive"].implemented)
def test_profile_payload_accepts_new_providers_and_redacts_secret_metadata(self) -> None:
profiles = connector_profiles_from_payload(
{
"profiles": [
{
"id": "dev-s3",
"label": "Dev S3",
"provider": "s3",
"username": "access-key",
"password": "secret-key",
"metadata": {"bucket": "govoplan", "secret_access_key": "do-not-return"},
},
{"id": "sharepoint", "provider": "sharepoint"},
{"id": "onedrive", "provider": "onedrive"},
]
}
)
by_id = {profile.id: profile for profile in profiles}
self.assertEqual("s3", by_id["dev-s3"].provider)
self.assertTrue(by_id["dev-s3"].credentials_configured)
self.assertEqual({"bucket": "govoplan"}, dict(by_id["dev-s3"].metadata))
self.assertEqual("sharepoint", by_id["sharepoint"].provider)
self.assertEqual("onedrive", by_id["onedrive"].provider)
def test_s3_browse_lists_prefixes_and_objects_with_provenance_metadata(self) -> None:
client = FakeS3Client()
with patch("govoplan_files.backend.storage.connector_browse._s3_client", return_value=client):
items = browse_connector_profile(s3_profile(), path="")
self.assertEqual({"Bucket": "govoplan", "Prefix": "root/", "Delimiter": "/", "MaxKeys": 1000}, client.list_objects_request)
self.assertEqual(["folder", "file"], [item.kind for item in items])
self.assertEqual("reports", items[0].path)
self.assertEqual("report.xlsx", items[1].path)
self.assertEqual("govoplan:root/report.xlsx", items[1].external_id)
self.assertEqual("root/report.xlsx", items[1].metadata["key"])
self.assertEqual("next-page", items[1].metadata["next_continuation_token"])
def test_s3_browse_lists_buckets_when_profile_has_no_bucket(self) -> None:
client = FakeS3Client()
with patch("govoplan_files.backend.storage.connector_browse._s3_client", return_value=client):
items = browse_connector_profile(s3_profile(metadata={}), path="")
self.assertEqual(["archive"], [item.path for item in items])
def test_s3_browse_reports_missing_optional_dependency(self) -> None:
with patch("govoplan_files.backend.storage.connector_browse.import_module", side_effect=ImportError):
with self.assertRaisesRegex(ConnectorBrowseUnsupported, "boto3"):
browse_connector_profile(s3_profile(), path="")
def test_s3_import_downloads_object_and_preserves_remote_identity(self) -> None:
client = FakeS3Client()
with patch("govoplan_files.backend.storage.connector_imports._s3_client", return_value=client):
downloaded = read_connector_file(s3_profile(), library_id="", path="report.txt", max_bytes=1024)
self.assertEqual({"Bucket": "govoplan", "Key": "root/report.txt"}, client.head_request)
self.assertEqual({"Bucket": "govoplan", "Key": "root/report.txt", "VersionId": "version-1"}, client.get_request)
self.assertEqual("report.txt", downloaded.filename)
self.assertEqual(b"hello s3\n", downloaded.data)
self.assertEqual("version-1", downloaded.revision)
self.assertEqual("govoplan:root/report.txt", downloaded.external_id)
self.assertEqual("s3://govoplan/root/report.txt", downloaded.external_url)
self.assertEqual("checksum", downloaded.metadata["checksum_sha256"])
if __name__ == "__main__":
unittest.main()

126
tests/test_http_client.py Normal file
View File

@@ -0,0 +1,126 @@
from __future__ import annotations
import unittest
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from unittest.mock import MagicMock, patch
import httpcore
from govoplan_core.security.outbound_http import outbound_http_policy, validate_outbound_http_url
from govoplan_files.backend.storage.http_client import (
ConnectorHttpError,
_OutboundPolicyNetworkBackend,
_SocketNetworkStream,
request_connector_bytes,
)
class _TestHandler(BaseHTTPRequestHandler):
def do_GET(self) -> None: # noqa: N802 - stdlib handler contract
body = b"connector-ok"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, _format: str, *_args: object) -> None:
return
class ConnectorHttpClientTests(unittest.TestCase):
def test_public_transport_adapter_performs_a_real_http_request(self) -> None:
server = ThreadingHTTPServer(("127.0.0.1", 0), _TestHandler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
with patch.dict(
"os.environ",
{"APP_ENV": "test", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
clear=False,
):
result = request_connector_bytes(
"GET",
f"http://127.0.0.1:{server.server_port}/object",
)
finally:
server.shutdown()
server.server_close()
thread.join(timeout=2)
self.assertEqual(200, result.status_code)
self.assertEqual(b"connector-ok", result.content)
def test_connector_responses_are_streamed_without_redirects(self) -> None:
response = MagicMock()
response.status_code = 200
response.headers = {"content-length": "6"}
response.iter_bytes.return_value = iter((b"abc", b"def"))
context = MagicMock()
context.__enter__.return_value = response
with patch("govoplan_files.backend.storage.http_client._stream_connector_request", return_value=context):
result = request_connector_bytes("GET", "https://example.test/object", max_bytes=10)
self.assertEqual(b"abcdef", result.content)
def test_connector_response_limit_is_enforced_while_streaming(self) -> None:
response = MagicMock()
response.status_code = 200
response.headers = {}
response.iter_bytes.return_value = iter((b"12345", b"67890", b"!"))
context = MagicMock()
context.__enter__.return_value = response
with patch(
"govoplan_files.backend.storage.http_client._stream_connector_request",
return_value=context,
), self.assertRaisesRegex(
ConnectorHttpError,
"configured limit",
):
request_connector_bytes("GET", "https://example.test/object", max_bytes=10)
def test_private_destination_is_blocked_before_http_connection(self) -> None:
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("10.0.0.5", 443))],
), patch("govoplan_files.backend.storage.http_client._stream_connector_request") as stream, self.assertRaisesRegex(
ConnectorHttpError,
"non-public network",
):
request_connector_bytes("GET", "https://connector.example.test/object")
stream.assert_not_called()
def test_connection_backend_rejects_private_rebinding_before_socket_open(self) -> None:
policy = outbound_http_policy({"APP_ENV": "production"})
public = [(2, 1, 6, "", ("93.184.216.34", 443))]
private = [(2, 1, 6, "", ("127.0.0.1", 443))]
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
side_effect=(public, private),
), patch("govoplan_core.security.outbound_http.socket.socket") as socket_factory:
validate_outbound_http_url("https://connector.example.test/object", policy=policy)
with self.assertRaises(httpcore.ConnectError):
_OutboundPolicyNetworkBackend().connect_tcp("connector.example.test", 443)
socket_factory.assert_not_called()
def test_network_backend_returns_public_network_stream_adapter(self) -> None:
sock = MagicMock()
with patch(
"govoplan_files.backend.storage.http_client.create_outbound_connection",
return_value=sock,
):
stream = _OutboundPolicyNetworkBackend().connect_tcp("connector.example.test", 443)
self.assertIsInstance(stream, _SocketNetworkStream)
self.assertIs(stream.get_extra_info("socket"), sock)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,36 @@
from __future__ import annotations
import unittest
from govoplan_files.backend.router import router
class FilesRouterContractTests(unittest.TestCase):
def test_connector_routes_keep_existing_api_paths(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
expected = {
(("GET",), "/files/connectors/providers"),
(("GET",), "/files/connectors/settings/delta"),
(("GET",), "/files/connectors/profiles"),
(("POST",), "/files/connectors/profiles"),
(("GET",), "/files/connectors/profiles/{profile_id}/browse"),
(("POST",), "/files/connectors/profiles/{profile_id}/import"),
(("POST",), "/files/connectors/profiles/{profile_id}/sync"),
(("GET",), "/files/connectors/credentials"),
(("POST",), "/files/connectors/credentials"),
(("GET",), "/files/connector-spaces"),
(("POST",), "/files/connector-spaces"),
}
self.assertTrue(expected.issubset(routes))
def test_bulk_organize_routes_keep_existing_api_paths(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
self.assertIn((("POST",), "/files/bulk-rename"), routes)
self.assertIn((("POST",), "/files/transfer"), routes)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,64 @@
from __future__ import annotations
import io
import unittest
from unittest.mock import MagicMock, PropertyMock, patch
from govoplan_files.backend.storage.backends import S3StorageBackend, StorageBackendError
def _backend() -> S3StorageBackend:
return S3StorageBackend(
bucket="files",
endpoint_url="https://objects.example.test",
region_name="test",
access_key_id="access",
secret_access_key="secret",
)
class S3StorageBackendTests(unittest.TestCase):
def test_get_bytes_rejects_declared_oversize_object_without_reading(self) -> None:
body = MagicMock()
client = MagicMock()
client.get_object.return_value = {"ContentLength": 6, "Body": body}
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "5"}), patch.object(
S3StorageBackend,
"client",
new_callable=PropertyMock,
return_value=client,
), self.assertRaisesRegex(StorageBackendError, "deployment limit"):
_backend().get_bytes("large.bin")
body.read.assert_not_called()
body.close.assert_called_once_with()
def test_iter_bytes_rejects_undeclared_oversize_object(self) -> None:
client = MagicMock()
client.get_object.return_value = {"Body": io.BytesIO(b"123456")}
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "5"}), patch.object(
S3StorageBackend,
"client",
new_callable=PropertyMock,
return_value=client,
), self.assertRaisesRegex(StorageBackendError, "deployment limit"):
list(_backend().iter_bytes("large.bin", chunk_size=3))
def test_iter_bytes_closes_streaming_body_when_consumer_stops_early(self) -> None:
body = MagicMock()
body.read.side_effect = (b"123", b"456", b"")
client = MagicMock()
client.get_object.return_value = {"ContentLength": 6, "Body": body}
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "10"}), patch.object(
S3StorageBackend,
"client",
new_callable=PropertyMock,
return_value=client,
):
chunks = _backend().iter_bytes("object.bin", chunk_size=3)
self.assertEqual(b"123", next(chunks))
chunks.close()
body.close.assert_called_once_with()
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,67 @@
from __future__ import annotations
import unittest
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError
from govoplan_files.backend.storage.transfers import _normalize_rename_request, _normalize_transfer_request
class TransferHelperTests(unittest.TestCase):
def test_transfer_normalization_deduplicates_folder_paths_and_conflicts(self) -> None:
normalized = _normalize_transfer_request(
operation=" COPY ",
source_owner_type=" USER ",
target_owner_type=" GROUP ",
folder_paths=["Reports", "Reports/2026", "", "./Archive"],
target_folder=" Target ",
conflict_strategy="rename",
conflict_resolutions=[FileConflictResolution(target_path="Target/a.txt", action="skip")],
)
self.assertEqual("copy", normalized.operation)
self.assertEqual("user", normalized.source_owner_type)
self.assertEqual("group", normalized.target_owner_type)
self.assertEqual(["Reports", "Reports/2026", "Archive"], normalized.source_folder_paths)
self.assertEqual("Target", normalized.target_folder)
self.assertEqual("rename", normalized.conflict_strategy)
self.assertEqual("skip", normalized.conflict_resolution_map["Target/a.txt"].action)
def test_transfer_normalization_rejects_unknown_operation(self) -> None:
with self.assertRaisesRegex(FileStorageError, "Unsupported transfer operation"):
_normalize_transfer_request(
operation="link",
source_owner_type="user",
target_owner_type="group",
folder_paths=[],
target_folder="",
conflict_strategy="reject",
conflict_resolutions=None,
)
def test_rename_normalization_collapses_nested_folder_roots(self) -> None:
normalized = _normalize_rename_request(
file_ids=["file-1", "file-1", "file-2"],
folder_paths=["Reports", "Reports/2026", "Archive"],
owner_type=" USER ",
owner_id="owner-1",
mode=" prefix ",
)
self.assertEqual("prefix", normalized.mode)
self.assertEqual(["file-1", "file-2"], normalized.selected_file_ids)
self.assertEqual(["Archive", "Reports"], normalized.selected_folder_roots)
self.assertEqual("user", normalized.owner_type)
def test_rename_normalization_rejects_invalid_direct_multi_select(self) -> None:
with self.assertRaisesRegex(FileStorageError, "Direct rename requires exactly one selected item"):
_normalize_rename_request(
file_ids=["file-1", "file-2"],
folder_paths=[],
owner_type="user",
owner_id="owner-1",
mode="direct",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -1,6 +1,6 @@
{
"name": "@govoplan/files-webui",
"version": "0.1.7",
"version": "0.1.8",
"private": true,
"type": "module",
"main": "src/index.ts",
@@ -13,6 +13,9 @@
},
"./styles/file-manager.css": "./src/styles/file-manager.css"
},
"scripts": {
"test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs"
},
"peerDependencies": {
"@vitejs/plugin-react": "^4.3.4",
"vite": "^6.0.6",
@@ -21,7 +24,7 @@
"react-dom": "^19.0.0",
"react-router-dom": "^7.1.1",
"lucide-react": "^1.23.0",
"@govoplan/core-webui": "^0.1.7"
"@govoplan/core-webui": "^0.1.9"
},
"peerDependenciesMeta": {
"@govoplan/core-webui": {

View File

@@ -0,0 +1,35 @@
import { readFileSync } from "node:fs";
const source = readFileSync(new URL("../src/features/files/FilesPage.tsx", import.meta.url), "utf8");
const normalized = source.replace(/\s+/g, " ");
function assertIncludes(fragment, message) {
if (!normalized.includes(fragment.replace(/\s+/g, " "))) throw new Error(message);
}
assertIncludes(
"const target = options.target ?? currentActionTarget();",
"uploads must prefer the explicit target supplied by the initiating interaction"
);
assertIncludes(
"const targetSpace = target ? findSpace(target.spaceId) : null;",
"the upload owner must be resolved from the selected target"
);
assertIncludes(
"owner_type: targetSpace.owner_type, owner_id: targetSpace.owner_id, path: target.folderPath,",
"the upload request must use the selected target owner and folder"
);
assertIncludes(
"async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) { await handleFilesUpload(fileList, { target }); }",
"external drops must pass their concrete target without asynchronous dialog-state mutation"
);
assertIncludes(
"await uploadExternalFilesToTarget(event.dataTransfer.files, target);",
"drop handling must forward the target on which the files were dropped"
);
assertIncludes(
"onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })}",
"the dialog picker/drop zone must snapshot its selected target for upload"
);
console.log("Files upload target routing structure is intact.");

View File

@@ -1,4 +1,10 @@
import { ApiError, apiFetch, apiUrl, authHeaders, csrfToken, type ApiSettings, type DeltaDeletedItem, type FilesManagedFileLinkTarget } from "@govoplan/core-webui";
export { fetchResourceAccessExplanation } from "@govoplan/core-webui";
export type {
AccessDecisionProvenanceItem,
ResourceAccessExplanationUser as AccessExplanationUser,
ResourceAccessExplanationResponse
} from "@govoplan/core-webui";
export type FileSpace = {
id: string;
@@ -116,8 +122,6 @@ export type FileConnectorCredentialsPayload = {
username?: string | null;
password?: string | null;
token?: string | null;
password_env?: string | null;
token_env?: string | null;
secret_ref?: string | null;
};
@@ -150,7 +154,7 @@ export type FileConnectorDiscoveryResponse = {
export type FileConnectorProfilePayload = {
id: string;
label: string;
provider: "seafile" | "nextcloud" | "webdav" | "smb" | "nfs" | "dms" | "generic";
provider: "seafile" | "nextcloud" | "webdav" | "smb" | "s3" | "sharepoint" | "onedrive" | "nfs" | "dms" | "generic";
endpoint_url?: string | null;
base_path?: string | null;
enabled?: boolean;
@@ -172,7 +176,7 @@ export type FileConnectorProfileUpdatePayload = Partial<Omit<FileConnectorProfil
export type FileConnectorCredentialPayload = {
id: string;
label: string;
provider?: "seafile" | "nextcloud" | "webdav" | "smb" | "nfs" | "dms" | "generic" | null;
provider?: "seafile" | "nextcloud" | "webdav" | "smb" | "s3" | "sharepoint" | "onedrive" | "nfs" | "dms" | "generic" | null;
enabled?: boolean;
scope_type?: "system" | "tenant" | "user" | "group" | "campaign";
scope_id?: string | null;
@@ -206,6 +210,8 @@ export type FileConnectorBrowseResponse = {
path: string;
library_id?: string | null;
read_only: boolean;
next_continuation_token?: string | null;
has_more: boolean;
decision: FileConnectorPolicyDecision;
items: FileConnectorBrowseItem[];
};
@@ -559,44 +565,6 @@ export function bulkDeleteFiles(settings: ApiSettings, fileIds: string[]): Promi
export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;};
export type AccessExplanationUser = {
id: string;
account_id?: string | null;
email?: string | null;
display_name?: string | null;
};
export type AccessDecisionProvenanceItem = {
kind: string;
id?: string | null;
label?: string | null;
tenant_id?: string | null;
source?: string | null;
details?: Record<string, unknown>;
};
export type ResourceAccessExplanationResponse = {
user: AccessExplanationUser;
resource_type: string;
resource_id: string;
action: string;
provenance: AccessDecisionProvenanceItem[];
};
export function fetchResourceAccessExplanation(
settings: ApiSettings,
options: {userId: string;resourceType: string;resourceId: string;action: string;tenantId?: string | null;})
: Promise<ResourceAccessExplanationResponse> {
const params = new URLSearchParams({
user_id: options.userId,
resource_type: options.resourceType,
resource_id: options.resourceId,
action: options.action
});
if (options.tenantId) params.set("tenant_id", options.tenantId);
return apiFetch<ResourceAccessExplanationResponse>(settings, `/api/v1/admin/access/resource-explanation?${params.toString()}`);
}
export function virtualFolderResourceId(options: {tenantId: string;ownerType: "user" | "group";ownerId: string;path: string;}): string {
return `virtual-folder:v1:${options.tenantId}:${options.ownerType}:${options.ownerId}:${base64Url(options.path.replace(/\\/g, "/").replace(/^\/+|\/+$/g, ""))}`;
}
@@ -808,11 +776,12 @@ export function deactivateFileConnectorProfile(settings: ApiSettings, profileId:
export function browseFileConnectorProfile(
settings: ApiSettings,
profileId: string,
params: {path?: string;library_id?: string;campaign_id?: string;} = {})
params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {})
: Promise<FileConnectorBrowseResponse> {
const search = new URLSearchParams();
if (params.path) search.set("path", params.path);
if (params.library_id) search.set("library_id", params.library_id);
if (params.continuation_token) search.set("continuation_token", params.continuation_token);
if (params.campaign_id) search.set("campaign_id", params.campaign_id);
const suffix = search.toString() ? `?${search.toString()}` : "";
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`);

View File

@@ -5,6 +5,8 @@ import {
Card,
ConnectionTree,
ConfirmDialog,
CredentialPanel,
DisabledActionTooltip,
DismissibleAlert,
Dialog,
FormField,
@@ -13,6 +15,8 @@ import {
LoadingFrame,
mergeDeltaRows,
SegmentedControl,
StatusBadge,
TableActionGroup,
ToggleSwitch,
useDeltaWatermarks,
useUnsavedDraftGuard,
@@ -63,8 +67,6 @@ type ConnectorProfileDraft = {
username: string;
password: string;
token: string;
passwordEnv: string;
tokenEnv: string;
secretRef: string;
canBrowse: boolean;
canImport: boolean;
@@ -89,8 +91,6 @@ type ConnectorCredentialDraft = {
username: string;
password: string;
token: string;
passwordEnv: string;
tokenEnv: string;
secretRef: string;
policyMode: PolicyMode;
allowedPaths: string;
@@ -123,6 +123,9 @@ const PROVIDERS: Array<{id: Provider;label: string;}> = [
{ id: "nextcloud", label: "i18n:govoplan-files.nextcloud.aab73a3d" },
{ id: "seafile", label: "i18n:govoplan-files.seafile.600192cc" },
{ id: "smb", label: "i18n:govoplan-files.smb.515d2f88" },
{ id: "s3", label: "S3" },
{ id: "sharepoint", label: "SharePoint" },
{ id: "onedrive", label: "OneDrive" },
{ id: "nfs", label: "i18n:govoplan-files.nfs.db121865" },
{ id: "dms", label: "i18n:govoplan-files.dms.477e5652" },
{ id: "generic", label: "i18n:govoplan-files.generic.ff7613e5" }];
@@ -133,6 +136,9 @@ const ENDPOINT_PLACEHOLDERS: Record<Provider, string> = {
nextcloud: "http://127.0.0.1:9081/remote.php/dav/files/admin/",
seafile: "http://127.0.0.1:9082/",
smb: "smb://127.0.0.1:1445/files",
s3: "http://127.0.0.1:9000",
sharepoint: "https://graph.microsoft.com/v1.0/sites/{site-id}/drives/{drive-id}",
onedrive: "https://graph.microsoft.com/v1.0/drives/{drive-id}",
nfs: "nfs://127.0.0.1/export",
dms: "https://dms.example.test",
generic: "https://files.example.test"
@@ -340,6 +346,17 @@ export default function FileConnectorSettingsPanel({
setCredentialDraft((current) => current ? { ...current, ...patch } : current);
}
function patchCredentialValues(patch: {username?: string | null;password?: string | null;}) {
const next: Partial<ConnectorCredentialDraft> = {};
if (patch.username !== undefined) next.username = String(patch.username ?? "");
if (patch.password !== undefined) next.password = String(patch.password ?? "");
patchCredentialDraft(next);
}
function patchCredentialToken(patch: {password?: string | null;}) {
if (patch.password !== undefined) patchCredentialDraft({ token: String(patch.password ?? "") });
}
function patchPolicyDraft(patch: Partial<CentralConnectorPolicyDraft>) {
setPolicyDraft((current) => ({ ...current, ...patch }));
}
@@ -472,8 +489,6 @@ export default function FileConnectorSettingsPanel({
username: cleanOrNull(credentialDraft.username),
password: cleanOrUndefined(credentialDraft.password),
token: cleanOrUndefined(credentialDraft.token),
password_env: cleanOrNull(credentialDraft.passwordEnv),
token_env: cleanOrNull(credentialDraft.tokenEnv),
secret_ref: cleanOrNull(credentialDraft.secretRef)
};
const sharedPayload = {
@@ -614,8 +629,6 @@ export default function FileConnectorSettingsPanel({
username: cleanOrNull(draft.username),
password: cleanOrUndefined(draft.password),
token: cleanOrUndefined(draft.token),
password_env: cleanOrNull(draft.passwordEnv),
token_env: cleanOrNull(draft.tokenEnv),
secret_ref: cleanOrNull(draft.secretRef)
},
metadata: metadataFromDraft(draft)
@@ -661,8 +674,6 @@ export default function FileConnectorSettingsPanel({
username: cleanOrNull(credentialDraft.username),
password: cleanOrUndefined(credentialDraft.password),
token: cleanOrUndefined(credentialDraft.token),
password_env: cleanOrNull(credentialDraft.passwordEnv),
token_env: cleanOrNull(credentialDraft.tokenEnv),
secret_ref: cleanOrNull(credentialDraft.secretRef)
},
metadata: profile.metadata ?? {},
@@ -737,7 +748,7 @@ export default function FileConnectorSettingsPanel({
width: "120px",
render: (row) => {
const enabled = row.kind === "profile" ? row.profile.enabled : row.credential.enabled;
return <span className={`status-badge ${enabled ? "success" : "neutral"}`}>{enabled ? "i18n:govoplan-files.enabled.df174a3f" : "i18n:govoplan-files.disabled.f4f4473d"}</span>;
return <StatusBadge status={enabled ? "success" : "inactive"} label={enabled ? "i18n:govoplan-files.enabled.df174a3f" : "i18n:govoplan-files.disabled.f4f4473d"} />;
}
}];
@@ -752,21 +763,19 @@ export default function FileConnectorSettingsPanel({
function renderConnectorActions(row: ConnectorTreeRow) {
if (row.kind === "credential") {
const readOnly = row.credential.source_kind !== "database";
return (
<div className="admin-icon-actions">
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label })} aria-label={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label })} onClick={() => startCredentialEdit(row.credential)} disabled={!canWrite || readOnly || saving}><Edit3 size={15} /></Button>
<Button type="button" variant="danger" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label })} aria-label={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label })} onClick={() => setPendingCredentialDeactivate(row.credential)} disabled={!canWrite || readOnly || saving || !row.credential.enabled}><Trash2 size={15} /></Button>
</div>);
return <TableActionGroup actions={[
{ id: "edit", label: i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label }), icon: <Edit3 size={15} />, disabled: !canWrite || readOnly || saving, onClick: () => startCredentialEdit(row.credential) },
{ id: "disable", label: i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label }), icon: <Trash2 size={15} />, variant: "danger", applicable: row.credential.enabled, disabled: !canWrite || readOnly || saving, onClick: () => setPendingCredentialDeactivate(row.credential) }
]} />;
}
const readOnly = row.profile.source_kind !== "database";
return (
<div className="admin-icon-actions">
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label })} onClick={() => void testBrowse(row.profile)} disabled={saving || testingProfileId === row.profile.id || !row.profile.enabled}><RefreshCw size={15} /></Button>
<Button type="button" variant="primary" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label })} onClick={() => startCredentialCreate(row.profile)} disabled={!canWrite || saving}><UserRoundKey size={15} /></Button>
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label })} onClick={() => startEdit(row.profile)} disabled={!canWrite || readOnly || saving}><Edit3 size={15} /></Button>
<Button type="button" variant="danger" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label })} onClick={() => setPendingDeactivate(row.profile)} disabled={!canWrite || readOnly || saving || !row.profile.enabled}><Trash2 size={15} /></Button>
</div>);
return <TableActionGroup actions={[
{ id: "test", label: i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label }), icon: <RefreshCw size={15} />, applicable: row.profile.enabled, disabled: saving || testingProfileId === row.profile.id, onClick: () => void testBrowse(row.profile) },
{ id: "add-credential", label: i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label }), icon: <UserRoundKey size={15} />, variant: "primary", disabled: !canWrite || saving, onClick: () => startCredentialCreate(row.profile) },
{ id: "edit", label: i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label }), icon: <Edit3 size={15} />, disabled: !canWrite || readOnly || saving, onClick: () => startEdit(row.profile) },
{ id: "disable", label: i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label }), icon: <Trash2 size={15} />, variant: "danger", applicable: row.profile.enabled, disabled: !canWrite || readOnly || saving, onClick: () => setPendingDeactivate(row.profile) }
]} />;
}
@@ -801,7 +810,7 @@ export default function FileConnectorSettingsPanel({
{targetSelectionRequired &&
<Card title={i18nMessage("i18n:govoplan-files.value_scope", { value0: targetLabel })}>
<div className="mail-profile-target-row">
<div className="settings-target-row">
<FormField label={targetLabel}>
<select value={selectedTargetId} onChange={(event) => setSelectedTargetId(event.target.value)} disabled={loading || saving || !hasSelectableTarget}>
{!hasSelectableTarget && <option value="">{targetEmptyText}</option>}
@@ -846,7 +855,7 @@ export default function FileConnectorSettingsPanel({
<LoadingFrame loading={loading} label="i18n:govoplan-files.loading_connector_policy.f12ab285">
<>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8">
<textarea value={policyDraft.allowedConnectors} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedConnectors: event.target.value })} rows={3} placeholder={"tenant-webdav\nseafile-*"} />
</FormField>
@@ -906,11 +915,11 @@ export default function FileConnectorSettingsPanel({
footer={credentialDraft ?
<>
<Button onClick={closeCredentialDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button>
<span className="disabled-action-tooltip" data-tooltip={credentialSaveTooltip}>
<DisabledActionTooltip reason={credentialSaveTooltip}>
<Button variant="primary" onClick={() => void saveCredentialDraft()} disabled={credentialSaveDisabled}>
<Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_credential.2c02a6d2"}
</Button>
</span>
</DisabledActionTooltip>
</> :
null}>
@@ -921,7 +930,7 @@ export default function FileConnectorSettingsPanel({
<h3>Credential</h3>
<p>Choose where this credential can be used and how it signs in.</p>
</header>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.credential_id.9432a6e1">
<input className={!editingCredentialId && !credentialDraft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingCredentialId && !credentialDraft.id.trim() || undefined} value={credentialDraft.id} disabled={Boolean(editingCredentialId) || saving} onChange={(event) => patchCredentialDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav-credentials`} />
</FormField>
@@ -983,51 +992,53 @@ export default function FileConnectorSettingsPanel({
actor: "Connector administrator",
target: "Credential mode"
}} /> :
<div className="form-grid two-column-form-grid">
{(credentialDraft.credentialMode === "basic" || credentialDraft.credentialMode === "secret_ref") &&
<FormField label="i18n:govoplan-files.username.84c29015">
<input value={credentialDraft.username} disabled={saving} onChange={(event) => patchCredentialDraft({ username: event.target.value })} autoComplete="username" />
</FormField>
}
<div className="file-connector-secret-fields">
{credentialDraft.credentialMode === "basic" &&
<FormField label="i18n:govoplan-files.password.8be3c943">
<input value={credentialDraft.password} disabled={saving} onChange={(event) => patchCredentialDraft({ password: event.target.value })} type="password" autoComplete="new-password" placeholder={editingCredentialId ? "i18n:govoplan-files.leave_empty_to_keep_saved_password.6ec39f5e" : ""} />
</FormField>
}
{credentialDraft.credentialMode === "token" &&
<FormField label="i18n:govoplan-files.token.a1141eb9">
<input value={credentialDraft.token} disabled={saving} onChange={(event) => patchCredentialDraft({ token: event.target.value })} type="password" placeholder={editingCredentialId ? "i18n:govoplan-files.leave_empty_to_keep_saved_token.e725857b" : ""} />
</FormField>
<CredentialPanel
values={{ username: credentialDraft.username, password: credentialDraft.password }}
onChange={patchCredentialValues}
disabled={saving}
savedPassword={Boolean(editingCredentialId) && !credentialDraft.clearPassword}
savedPasswordPlaceholder="i18n:govoplan-files.leave_empty_to_keep_saved_password.6ec39f5e" />
}
{credentialDraft.credentialMode === "secret_ref" &&
<>
<CredentialPanel
values={{ username: credentialDraft.username }}
onChange={patchCredentialValues}
disabled={saving}
showPassword={false} />
<div className="form-grid two">
<FormField label="i18n:govoplan-files.secret_reference.04ed2221">
<input value={credentialDraft.secretRef} disabled={saving} onChange={(event) => patchCredentialDraft({ secretRef: event.target.value })} />
</FormField>
</div>
</>
}
{credentialDraft.credentialMode === "token" &&
<CredentialPanel
values={{ password: credentialDraft.token }}
onChange={patchCredentialToken}
disabled={saving}
showUsername={false}
passwordLabel="i18n:govoplan-files.token.a1141eb9"
savedPassword={Boolean(editingCredentialId) && !credentialDraft.clearToken}
savedPasswordPlaceholder="i18n:govoplan-files.leave_empty_to_keep_saved_token.e725857b" />
}
</div>
}
<div className="button-row compact-actions">
<span className="disabled-action-tooltip" data-tooltip={credentialTestDisabled ? credentialTestTooltip : ""}>
<DisabledActionTooltip reason={credentialTestDisabled ? credentialTestTooltip : ""}>
<Button type="button" onClick={() => void testCredentialLogin()} disabled={credentialTestDisabled}>
<RefreshCw size={16} aria-hidden="true" /> {testingCredentialLogin ? "Testing login" : "Test login"}
</Button>
</span>
</DisabledActionTooltip>
</div>
{credentialLoginResult &&
<DismissibleAlert tone={credentialLoginResult.ok ? "success" : "warning"} resetKey={credentialLoginResult.message}>
{credentialLoginResult.message}
</DismissibleAlert>
}
<AdvancedOptionsPanel title="Environment and fallback secrets" summary="Use these only when the deployment injects secrets outside the database.">
<div className="form-grid two-column-form-grid">
<FormField label="i18n:govoplan-files.password_environment_variable.0e77673f">
<input value={credentialDraft.passwordEnv} disabled={saving} onChange={(event) => patchCredentialDraft({ passwordEnv: event.target.value })} />
</FormField>
<FormField label="i18n:govoplan-files.token_environment_variable.5af4a79e">
<input value={credentialDraft.tokenEnv} disabled={saving} onChange={(event) => patchCredentialDraft({ tokenEnv: event.target.value })} />
</FormField>
</div>
</AdvancedOptionsPanel>
</section>
<section className="adaptive-config-section">
@@ -1035,7 +1046,7 @@ export default function FileConnectorSettingsPanel({
<h3>Policy</h3>
<p>Limit where lower levels may use this credential.</p>
</header>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.policy.bb9cf141">
<select value={credentialDraft.policyMode} disabled={saving} onChange={(event) => patchCredentialDraft({ policyMode: event.target.value as PolicyMode })}>
<option value="allow-provider">i18n:govoplan-files.can_use_this_credential.f4a41971</option>
@@ -1072,11 +1083,11 @@ export default function FileConnectorSettingsPanel({
footer={draft ?
<>
<Button onClick={closeProfileDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button>
<span className="disabled-action-tooltip" data-tooltip={profileSaveTooltip}>
<DisabledActionTooltip reason={profileSaveTooltip}>
<Button variant="primary" onClick={() => void saveDraft()} disabled={profileSaveDisabled}>
<Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_connection.07796d38"}
</Button>
</span>
</DisabledActionTooltip>
</> :
null}>
@@ -1087,7 +1098,7 @@ export default function FileConnectorSettingsPanel({
<h3>Connection</h3>
<p>Name the file server connection and choose the provider.</p>
</header>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.profile_id.25961d68">
<input className={!editingProfileId && !draft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingProfileId && !draft.id.trim() || undefined} value={draft.id} disabled={Boolean(editingProfileId) || saving} onChange={(event) => patchDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav`} />
</FormField>
@@ -1120,7 +1131,7 @@ export default function FileConnectorSettingsPanel({
<h3>Location and credentials</h3>
<p>Only fields relevant for the selected provider and credential mode are shown.</p>
</header>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.endpoint_url.65aaaa45">
<input value={draft.endpointUrl} disabled={saving} onChange={(event) => patchDraft({ endpointUrl: event.target.value })} placeholder={ENDPOINT_PLACEHOLDERS[draft.provider]} />
</FormField>
@@ -1157,7 +1168,7 @@ export default function FileConnectorSettingsPanel({
</DismissibleAlert>
}
<AdvancedOptionsPanel title="Protocol and compatibility options" summary="Change these only when the provider or network requires an override.">
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.browse_protocol.d1f27c90">
<select value={draft.browseProtocol} disabled={saving} onChange={(event) => patchDraft({ browseProtocol: event.target.value })}>
<option value="">i18n:govoplan-files.native_default.ba32f7b6</option>
@@ -1191,7 +1202,7 @@ export default function FileConnectorSettingsPanel({
<ToggleSwitch label="i18n:govoplan-files.import.d6fbc9d2" checked={draft.canImport} disabled={saving} onChange={(canImport) => patchDraft({ canImport })} />
<ToggleSwitch label="i18n:govoplan-files.sync.905f6309" checked={draft.canSync} disabled={saving} onChange={(canSync) => patchDraft({ canSync })} />
</div>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.policy.bb9cf141">
<select value={draft.policyMode} disabled={saving} onChange={(event) => patchDraft({ policyMode: event.target.value as PolicyMode })}>
<option value="allow-provider">i18n:govoplan-files.can_use_this_connection.5091a74c</option>
@@ -1342,8 +1353,6 @@ function emptyDraft(scopeType: ConnectorScope): ConnectorProfileDraft {
username: "",
password: "",
token: "",
passwordEnv: "",
tokenEnv: "",
secretRef: "",
canBrowse: true,
canImport: true,
@@ -1406,8 +1415,6 @@ function emptyCredentialDraft(scopeType: ConnectorScope): ConnectorCredentialDra
username: "",
password: "",
token: "",
passwordEnv: "",
tokenEnv: "",
secretRef: "",
policyMode: "allow-provider",
allowedPaths: "",

View File

@@ -8,6 +8,7 @@ import {
FileDropZone,
FormField,
LoadingIndicator,
ResourceAccessExplanation,
ToggleSwitch,
hasScope,
type ApiSettings,
@@ -182,6 +183,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
}, [visibleFiles, folders, currentFolder, searchActive, sortColumn, sortDirection]);
const fileListViewportRef = useRef<HTMLDivElement | null>(null);
const fileListMeasureRowRef = useRef<HTMLDivElement | null>(null);
const managedSpaceLoadsInFlightRef = useRef<Set<string>>(new Set());
const [fileListViewportHeight, setFileListViewportHeight] = useState(0);
const [fileListScrollTop, setFileListScrollTop] = useState(0);
const [fileListRowHeight, setFileListRowHeight] = useState(DEFAULT_FILE_LIST_ROW_HEIGHT);
@@ -291,8 +293,9 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
try {
const response = await listFileSpaces(settings);
setSpaces(response.spaces);
setActiveSpaceId((current) => current || response.spaces[0]?.id || "");
await Promise.all(response.spaces.filter((space) => !isConnectorSpace(space)).map((space) => loadSpaceContents(space, { silent: true })));
const nextActiveSpace = response.spaces.find((space) => space.id === activeSpaceId) ?? response.spaces[0] ?? null;
setActiveSpaceId(nextActiveSpace?.id || "");
if (nextActiveSpace && !isConnectorSpace(nextActiveSpace)) await loadSpaceContents(nextActiveSpace, { silent: true });
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
} finally {
@@ -331,6 +334,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
await loadConnectorSpaceContents(space, { silent: options.silent, folderPath: "" });
return;
}
if (managedSpaceLoadsInFlightRef.current.has(space.id)) return;
managedSpaceLoadsInFlightRef.current.add(space.id);
if (!options.silent) {
setBusy(true);
setError("");
@@ -372,6 +377,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} catch (err) {
setError(err instanceof Error ? err.message : String(err));
} finally {
managedSpaceLoadsInFlightRef.current.delete(space.id);
if (!options.silent) setBusy(false);
}
}
@@ -411,6 +417,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const nextSpace = spaces.find((space) => space.id === activeSpaceId);
if (nextSpace && isConnectorSpace(nextSpace)) {
void loadConnectorSpaceContents(nextSpace, { folderPath: "", silent: true });
} else if (nextSpace && filesBySpace[nextSpace.id] === undefined && foldersBySpace[nextSpace.id] === undefined) {
void loadSpaceContents(nextSpace, { silent: true });
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [activeSpaceId, spaces]);
@@ -432,6 +440,16 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
return activeDialogTarget;
}
function uploadRejectedReason(target: FileActionTarget | null): string {
if (busy || uploadActive) return "Another file operation is already running. Wait until it finishes before dropping files.";
if (!canUpload) return "You do not have permission to upload files here.";
if (!target) return "Choose a destination folder before dropping files.";
const targetSpace = findSpace(target.spaceId);
if (!targetSpace) return "The selected upload destination is no longer available.";
if (isConnectorSpace(targetSpace)) return "Files cannot be uploaded into connector spaces from this view.";
return "The dropped item cannot be uploaded here.";
}
function openDialog(kind: DialogKind, target: FileActionTarget | null = null) {
if ((kind === "upload" || kind === "connector-sync") && !canUpload) return;
if (kind === "connector-space" && !canOrganize) return;
@@ -608,10 +626,12 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
async function handleFilesUpload(fileList: FileList | File[], options: {conflictStrategy?: ConflictStrategy;conflictResolutions?: ConflictResolution[];bypassConflictDialog?: boolean;target?: FileActionTarget;} = {}) {
if (!canUpload) return;
const target = options.target ?? currentActionTarget();
const targetSpace = target ? findSpace(target.spaceId) : null;
if (!target || !targetSpace || isConnectorSpace(targetSpace)) return;
if (busy || uploadActive || !canUpload || !target || !targetSpace || isConnectorSpace(targetSpace)) {
setError(uploadRejectedReason(target));
return;
}
const selected = Array.from(fileList);
if (selected.length === 0) return;
if (!options.bypassConflictDialog && !unpackZip) {
@@ -671,13 +691,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
}
async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) {
const previousTarget = dialogTarget;
setDialogTarget(target);
try {
await handleFilesUpload(fileList);
} finally {
setDialogTarget(previousTarget);
}
await handleFilesUpload(fileList, { target });
}
async function openConnectorSyncDialog(target: FileActionTarget | null = toolbarTarget()) {
@@ -1332,8 +1346,20 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setDropTargetKey(reason || noop ? "" : dropTargetId(target));
return;
}
event.dataTransfer.dropEffect = canUpload ? "copy" : "none";
setDropTargetKey(canUpload ? dropTargetId(target) : "");
event.dataTransfer.dropEffect = canUpload && !busy && !uploadActive ? "copy" : "none";
setDropTargetKey(canUpload && !busy && !uploadActive ? dropTargetId(target) : "");
}
function handleCurrentFolderDragOver(event: ReactDragEvent<HTMLElement>) {
const target = toolbarTarget();
if (!target) {
event.preventDefault();
event.stopPropagation();
event.dataTransfer.dropEffect = "none";
setDropTargetKey("");
return;
}
handleDropTargetDragOver(event, target);
}
async function handleDropOnTarget(event: ReactDragEvent<HTMLElement>, target: FileActionTarget) {
@@ -1341,7 +1367,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
event.stopPropagation();
setDragActive(false);
setDropTargetKey("");
if (isConnectorSpace(findSpace(target.spaceId))) return;
const hasDroppedFiles = event.dataTransfer.files.length > 0;
if (isConnectorSpace(findSpace(target.spaceId))) {
if (hasDroppedFiles) setError(uploadRejectedReason(target));
return;
}
if (isInternalDrag(event)) {
if (!canOrganize) return;
const state = internalDrag ?? parseDragState(event.dataTransfer.getData(INTERNAL_DRAG_TYPE));
@@ -1354,9 +1384,27 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
await runTransfer(event.ctrlKey ? "copy" : "move", state.sourceSpaceId, { fileIds: new Set(state.fileIds), folderPaths: new Set(state.folderPaths) }, target);
return;
}
if (canUpload && event.dataTransfer.files.length > 0) {
await uploadExternalFilesToTarget(event.dataTransfer.files, target);
if (hasDroppedFiles) {
if (!canUpload || busy || uploadActive) {
setError(uploadRejectedReason(target));
return;
}
await uploadExternalFilesToTarget(event.dataTransfer.files, target);
return;
}
setError("Drop one or more files from your computer, or drag files and folders from this file space.");
}
async function handleDropOnCurrentFolder(event: ReactDragEvent<HTMLElement>) {
const target = toolbarTarget();
if (!target) {
event.preventDefault();
event.stopPropagation();
setDropTargetKey("");
setError(uploadRejectedReason(null));
return;
}
await handleDropOnTarget(event, target);
}
function clearDropState() {
@@ -1933,8 +1981,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
}
const currentFolderDropTarget = toolbarTarget();
const currentFolderDropActive = currentFolderDropTarget ? dropTargetKey === dropTargetId(currentFolderDropTarget) : false;
return (
<div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen">
<div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen files-page">
{error &&
<DismissibleAlert tone="danger" resetKey={error} floating>{error}</DismissibleAlert>
}
@@ -2044,9 +2095,12 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
renderConnectorSpaceContent() :
<div
className="file-list-drop-target"
className={`file-list-drop-target ${currentFolderDropActive ? "is-drop-target" : ""}`}
ref={fileListViewportRef}
onScroll={(event) => setFileListScrollTop(event.currentTarget.scrollTop)}
onDragOver={handleCurrentFolderDragOver}
onDragLeave={clearDropState}
onDrop={(event) => void handleDropOnCurrentFolder(event)}
onContextMenu={(event) => openContextMenu(event, "empty")}
onClick={(event) => {
if (event.target === event.currentTarget) applySelectionKeys(new Set<EntrySelectionKey>());
@@ -2158,8 +2212,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
}
{accessExplanationTarget &&
<FileDialog title="i18n:govoplan-files.access_explanation.75ee7f62" onClose={() => {if (!resourceAccessLoading) {setAccessExplanationTarget(null);setResourceAccessExplanation(null);}}}>
<ResourceAccessExplanationContent
<FileDialog title="i18n:govoplan-core.access_explanation.75ee7f62" onClose={() => {if (!resourceAccessLoading) {setAccessExplanationTarget(null);setResourceAccessExplanation(null);}}}>
<ResourceAccessExplanation
loading={resourceAccessLoading}
explanation={resourceAccessExplanation}
fallbackResourceLabel={accessExplanationTarget.label} />
@@ -2172,13 +2226,14 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "upload" &&
<FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}>
<FileDropZone
disabled={busy || !activeDialogSpace || !canUpload}
disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace) || !canUpload}
busy={uploadActive}
progress={visibleUploadProgress}
busyLabel={uploadBusyLabel}
progressLabel={uploadProgressLabel}
note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`}
onFiles={(files) => handleFilesUpload(files)} />
onRejectedDrop={(reason) => setError(reason === "disabled" ? uploadRejectedReason(activeDialogTarget) : "The browser did not provide readable file data for this drop. Use the file picker, or drag local files from a file manager that exposes file contents to the browser.")}
onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} />
<ToggleSwitch label="i18n:govoplan-files.unpack_zip_uploads.256fead4" checked={unpackZip} onChange={setUnpackZip} disabled={busy} />
<div className="field-block">
@@ -2379,7 +2434,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "transfer" && transferDialogState &&
<FileDialog title={`${transferMode === "copy" ? "i18n:govoplan-files.copy.af74f7c5" : "i18n:govoplan-files.move.76cdb950"} selection`} onClose={closeDialog}>
<p className="muted">i18n:govoplan-files.choose_a_destination_space_and_folder_folders_ar.45158643</p>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.destination_space.92b63970" help="i18n:govoplan-files.select_the_space_that_will_receive_the_selected_.0a8bea8f">
<select value={transferDialogState.targetSpaceId} onChange={(event) => setTransferDialogState((current) => current ? { ...current, targetSpaceId: event.target.value, targetFolder: "" } : current)}>
{spaces.filter((space) => !isConnectorSpace(space)).map((space) => <option key={space.id} value={space.id}>{space.label}</option>)}
@@ -2418,7 +2473,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "rename" &&
<FileDialog title="i18n:govoplan-files.bulk_rename_selected_items.5e79d694" onClose={closeDialog}>
<p className="muted">i18n:govoplan-files.bulk_rename_changes_managed_display_paths_only_i.8cf34a6b</p>
<div className="form-grid two-column-form-grid">
<div className="form-grid two">
<FormField label="i18n:govoplan-files.mode.a7b93d21" help="i18n:govoplan-files.choose_how_the_selected_names_should_be_changed.0231854f">
<select value={renameMode} onChange={(event) => setRenameMode(event.target.value as RenameMode)}>
<option value="prefix">i18n:govoplan-files.add_prefix.672452bc</option>
@@ -2454,71 +2509,6 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
}
function ResourceAccessExplanationContent({
loading,
explanation,
fallbackResourceLabel
}: {loading: boolean;explanation: ResourceAccessExplanationResponse | null;fallbackResourceLabel: string;}) {
if (loading) return <p className="muted small-note">i18n:govoplan-files.loading_access_explanation.04a7c934</p>;
if (!explanation) return null;
const userLabel = explanation.user.display_name || explanation.user.email || explanation.user.id;
const resourceLabel = explanation.provenance.find((item) => item.kind === "resource")?.label || fallbackResourceLabel || explanation.resource_id;
return (
<>
<div className="form-grid compact responsive-form-grid">
<div><span className="form-label">i18n:govoplan-files.user.9f8a2389</span><p>{userLabel}</p></div>
<div><span className="form-label">i18n:govoplan-files.resource.d1c626a9</span><p>{resourceLabel}</p></div>
<div><span className="form-label">i18n:govoplan-files.action.97c89a4d</span><p><code>{explanation.action}</code></p></div>
<div><span className="form-label">i18n:govoplan-files.evidence.8487d192</span><p>{explanation.provenance.length}</p></div>
</div>
{explanation.provenance.length === 0 ?
<p className="muted small-note">i18n:govoplan-files.no_access_evidence_was_returned.84a21e4e</p> :
<div className="admin-assignment-grid">
{explanation.provenance.map((item, index) =>
<div key={`${item.kind}:${item.id ?? index}`}>
<strong>{provenanceKindLabel(item.kind)}</strong>
<div className="muted small-note">
{item.source || "i18n:govoplan-files.no_source.6dcf9723"}
{item.id && <> · <code>{item.id}</code></>}
</div>
{item.label && <p>{item.label}</p>}
{Object.keys(item.details ?? {}).length > 0 && <p className="muted small-note">{formatProvenanceDetails(item.details ?? {})}</p>}
</div>
)}
</div>
}
</>);
}
function provenanceKindLabel(kind: string): string {
switch (kind) {
case "resource":
return "i18n:govoplan-files.resource.d1c626a9";
case "owner":
return "i18n:govoplan-files.owner.89ff3122";
case "share":
return "i18n:govoplan-files.share.09ca55ca";
case "policy":
return "i18n:govoplan-files.policy.0b779a05";
case "role":
return "i18n:govoplan-files.role.b5b4a5a2";
case "right":
return "i18n:govoplan-files.permission.2f81a22d";
default:
return kind;
}
}
function formatProvenanceDetails(details: Record<string, unknown>): string {
return Object.entries(details).map(([key, value]) => `${key}: ${formatProvenanceValue(value)}`).join("; ");
}
function formatProvenanceValue(value: unknown): string {
if (value === null || value === undefined) return "i18n:govoplan-files.none.6eef6648";
if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") return String(value);
return JSON.stringify(value);
}
function FileSearchRow({
value,
caseSensitive,

View File

@@ -252,7 +252,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.overwrite.0625c54e": "Overwrite",
"i18n:govoplan-files.owner_space.364c4b62": "Owner space",
"i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder",
"i18n:govoplan-files.password_environment_variable.0e77673f": "Password environment variable",
"i18n:govoplan-files.password.8be3c943": "Password",
"i18n:govoplan-files.path_limited.d32cbef0": "Path-limited",
"i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results",
@@ -318,7 +317,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.this_file_credential.695efb90": "this file credential",
"i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.",
"i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.",
"i18n:govoplan-files.token_environment_variable.5af4a79e": "Token environment variable",
"i18n:govoplan-files.token.a1141eb9": "Token",
"i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads",
"i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive",
@@ -609,7 +607,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.overwrite.0625c54e": "Overwrite",
"i18n:govoplan-files.owner_space.364c4b62": "Owner space",
"i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder",
"i18n:govoplan-files.password_environment_variable.0e77673f": "Password environment variable",
"i18n:govoplan-files.password.8be3c943": "Passwort",
"i18n:govoplan-files.path_limited.d32cbef0": "Path-limited",
"i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results",
@@ -675,7 +672,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.this_file_credential.695efb90": "this file credential",
"i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.",
"i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.",
"i18n:govoplan-files.token_environment_variable.5af4a79e": "Token environment variable",
"i18n:govoplan-files.token.a1141eb9": "Token",
"i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads",
"i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive",

View File

@@ -1,85 +1,6 @@
/* Files manager */
.file-manager-page.file-manager-fullscreen {
position: relative;
display: grid;
grid-template-rows: 1fr;
height: calc(100vh - 115px);
padding: 0;
overflow: hidden;
}
.file-manager-toolbar {
display: flex;
align-items: center;
gap: 8px;
flex-wrap: wrap;
padding: 10px 14px;
border-bottom: 1px solid var(--line);
background: var(--panel-header);
}
.file-manager-toolbar .button {
display: inline-flex;
align-items: center;
gap: 7px;
}
.file-manager-shell {
display: grid;
.files-page .file-manager-shell {
grid-template-columns: minmax(240px, 300px) minmax(0, 1fr);
min-height: 0;
height: 100%;
border: 1px solid var(--line);
border-radius: 0;
overflow: hidden;
background: var(--panel);
}
.file-list-panel {
display: flex;
flex-direction: column;
min-width: 0;
min-height: 0;
background: var(--panel);
}
.file-list-sticky {
flex: 0 0 auto;
z-index: 2;
border-bottom: 1px solid var(--line);
background: var(--panel-soft);
}
.file-breadcrumbs {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 6px;
min-height: 32px;
padding: 10px 14px 6px;
}
.file-breadcrumb,
.file-breadcrumb-segment {
display: inline-flex;
align-items: center;
gap: 5px;
}
.file-breadcrumb {
border: 0;
background: transparent;
color: var(--text-strong);
cursor: pointer;
font-weight: 800;
padding: 4px 6px;
border-radius: 7px;
}
.file-breadcrumb:hover:not(:disabled),
.file-breadcrumb:focus-visible {
background: var(--panel);
outline: none;
}
.file-search-row {
@@ -90,53 +11,21 @@
padding: 4px 0 10px 14px;
}
.file-list-meta {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
flex-wrap: wrap;
padding: 8px 14px;
border-top: 1px solid var(--line);
color: var(--muted);
font-size: 12px;
}
.file-list-drop-target {
position: relative;
flex: 1 1 auto;
min-height: 0;
overflow: auto;
transition: background-color .16s ease, box-shadow .16s ease;
}
.file-list-drop-target.is-active,
.file-list-drop-target.is-drop-target {
background: var(--line);
box-shadow: inset 0 0 0 2px var(--line-dark);
}
.file-list-table {
.files-page .file-list-table {
min-width: 760px;
}
.file-list-table-head,
.file-list-row {
display: grid;
.files-page .file-list-table-head,
.files-page .file-list-row,
.managed-pattern-results .file-list-table-head,
.managed-pattern-results .file-list-row {
grid-template-columns: minmax(280px, 1fr) 120px 240px;
gap: 12px;
align-items: center;
}
.file-list-table-head {
padding: 10px 14px;
border-top: 1px solid var(--line);
background: var(--panel);
color: var(--muted);
font-size: 12px;
font-weight: 800;
letter-spacing: .04em;
text-transform: uppercase;
}
.file-list-table-head button {
@@ -158,24 +47,6 @@
outline: none;
}
.file-list-row {
min-height: 58px;
padding: 9px 14px;
border-bottom: 1px solid var(--line);
cursor: default;
user-select: none;
}
.file-list-row:focus-visible {
outline: 2px solid var(--line-dark);
outline-offset: -2px;
}
.file-list-row:hover,
.file-list-row.is-selected {
background: var(--line);
}
.file-list-row.is-drop-target {
background: var(--line);
box-shadow: inset 0 0 0 2px var(--line-dark);
@@ -191,43 +62,6 @@
background: transparent;
}
.file-list-name-cell {
min-width: 0;
}
.file-list-name {
display: inline-flex;
align-items: center;
gap: 10px;
max-width: 100%;
min-width: 0;
border: 0;
background: transparent;
color: var(--text);
cursor: default;
font: inherit;
text-align: left;
padding: 0;
}
.file-list-name > span {
display: grid;
min-width: 0;
gap: 2px;
}
.file-list-name strong,
.file-list-name small {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.file-list-name small {
color: var(--muted);
font-size: 12px;
}
.file-linkage-status {
display: inline-flex;
align-items: center;
@@ -238,29 +72,8 @@
color: var(--text-strong);
}
.file-row-icon {
color: var(--muted);
flex: 0 0 auto;
}
.folder-row .file-row-icon {
color: #b6791d;
}
.file-row-tail {
display: flex;
align-items: center;
justify-content: space-between;
gap: 10px;
min-width: 0;
color: var(--muted);
font-size: 12px;
}
.file-list-empty {
padding: 36px 20px;
color: var(--muted);
text-align: center;
color: var(--warning-deep);
}
.file-list-virtual-spacer {
@@ -274,7 +87,7 @@
display: grid;
min-width: 190px;
overflow: hidden;
border: 1px solid var(--line-dark);
border: var(--border-line-dark);
border-radius: 12px;
background: var(--panel);
box-shadow: var(--shadow-popover);
@@ -299,7 +112,7 @@
.file-context-menu button:hover,
.file-context-menu button:focus-visible {
background: rgba(13, 110, 253, .08);
background: var(--primary-soft);
outline: none;
}
@@ -315,14 +128,14 @@
display: grid;
place-items: center;
padding: 22px;
background: rgba(28, 25, 22, .38);
background: var(--dialog-backdrop);
}
.file-dialog {
width: min(620px, 100%);
max-height: min(720px, calc(100vh - 44px));
overflow: auto;
border: 1px solid var(--line-dark);
border: var(--border-line-dark);
border-radius: 16px;
background: var(--panel);
box-shadow: var(--shadow-popover);
@@ -334,7 +147,7 @@
justify-content: space-between;
gap: 12px;
padding: 15px 18px;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
background: var(--panel-soft);
}
@@ -397,7 +210,7 @@
overflow-y: auto;
scrollbar-gutter: stable;
padding: 8px;
border: 1px solid var(--line);
border: var(--border-line);
border-radius: 12px;
background: var(--panel-soft);
}
@@ -428,13 +241,13 @@
.file-folder-selector-node:hover:not(:disabled),
.file-folder-selector-node:focus-visible,
.file-folder-selector-node.is-selected {
background: rgba(13, 110, 253, .09);
background: var(--primary-soft);
outline: none;
}
.file-folder-selector-node.is-selected {
color: var(--text-strong);
box-shadow: inset 0 0 0 1px rgba(13, 110, 253, .25);
box-shadow: var(--primary-inset-ring);
}
.file-folder-selector-empty {
@@ -462,7 +275,7 @@
grid-template-rows: auto auto minmax(0, 1fr);
min-height: 320px;
overflow: hidden;
border: 1px solid var(--line);
border: var(--border-line);
border-radius: 12px;
background: var(--panel);
}
@@ -473,7 +286,7 @@
gap: 10px;
align-items: center;
padding: 10px;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
background: var(--panel-soft);
}
@@ -577,7 +390,7 @@
gap: 3px;
min-width: 0;
padding: 10px 12px;
border: 1px solid var(--line);
border: var(--border-line);
border-radius: 10px;
background: var(--panel-soft);
}
@@ -625,7 +438,7 @@
gap: 12px;
align-items: center;
padding: 10px 12px;
border: 1px solid var(--line);
border: var(--border-line);
border-radius: 8px;
background: var(--panel-soft);
}
@@ -698,6 +511,11 @@
white-space: nowrap;
}
.file-connector-secret-fields {
display: grid;
gap: 18px;
}
@media (max-width: 760px) {
.file-connector-profile-row {
grid-template-columns: 1fr;
@@ -719,25 +537,27 @@
}
@media (max-width: 1050px) {
.file-manager-shell {
.files-page .file-manager-shell {
grid-template-columns: 1fr;
}
.file-tree-panel {
.files-page .file-tree-panel {
max-height: 260px;
border-right: 0;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
}
.file-list-table-head {
.files-page .file-list-table-head,
.managed-pattern-results .file-list-table-head {
display: none;
}
.file-list-table {
.files-page .file-list-table {
min-width: 0;
}
.file-list-row {
.files-page .file-list-row,
.managed-pattern-results .file-list-row {
grid-template-columns: 1fr;
gap: 8px;
}
@@ -747,28 +567,11 @@
}
}
.file-manager-shell.is-loading .file-tree-panel,
.file-manager-shell.is-loading .file-list-panel {
filter: blur(1.5px);
pointer-events: none;
user-select: none;
}
.file-manager-loading-overlay {
position: absolute;
inset: 0;
z-index: 35;
display: grid;
place-items: center;
background: rgba(255, 255, 255, .12);
backdrop-filter: blur(1px);
}
.file-conflict-summary {
display: grid;
gap: 3px;
padding: 12px 14px;
border: 1px solid var(--line);
border: var(--border-line);
border-radius: 12px;
background: var(--panel-soft);
}
@@ -791,7 +594,7 @@
gap: 10px;
align-items: center;
padding: 10px;
border: 1px solid var(--line);
border: var(--border-line);
border-radius: 12px;
background: var(--panel-soft);
}
@@ -842,8 +645,8 @@
.rename-preview-more:hover,
.rename-preview-more:focus-visible {
border-color: rgba(13, 110, 253, .45);
background: rgba(13, 110, 253, .08);
border-color: var(--primary-border);
background: var(--primary-soft);
color: var(--text-strong);
outline: none;
}
@@ -860,7 +663,7 @@
.managed-file-chooser-dialog > .dialog-header {
padding: 16px 18px;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
background: var(--panel);
}
@@ -891,9 +694,9 @@
.managed-file-chooser-footer {
flex: 0 0 auto;
padding: 12px 18px;
border-top: 1px solid var(--line);
border-top: var(--border-line);
background: var(--panel);
box-shadow: 0 -8px 24px rgba(15, 23, 42, .06);
box-shadow: var(--shadow-footer);
}
.managed-file-chooser-layout {
@@ -911,7 +714,7 @@
.managed-file-chooser-spaces {
min-width: 0;
padding: 16px;
border-right: 1px solid var(--line);
border-right: var(--border-line);
background: var(--panel-soft);
overflow: auto;
}
@@ -955,7 +758,7 @@
border: 1px solid transparent;
border-radius: 9px;
background: transparent;
color: var(--ink);
color: var(--text-strong);
text-align: left;
cursor: pointer;
}
@@ -1014,7 +817,7 @@
gap: 12px;
align-items: center;
padding: 12px 14px;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
}
.managed-file-breadcrumb {
@@ -1033,7 +836,7 @@
border: 0;
border-radius: 6px;
background: transparent;
color: var(--ink);
color: var(--text-strong);
padding: 5px 6px;
cursor: pointer;
}
@@ -1046,7 +849,7 @@
display: grid;
gap: 8px;
padding: 12px 14px;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
background: var(--panel-soft);
}
@@ -1075,7 +878,7 @@
align-items: center;
min-height: 38px;
padding: 0 20px;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
background: var(--panel-soft);
color: var(--muted);
font-size: 12px;
@@ -1132,7 +935,7 @@
}
.managed-file-entry:disabled {
color: var(--ink);
color: var(--text-strong);
opacity: 1;
cursor: default;
}
@@ -1206,7 +1009,7 @@
.managed-file-chooser-note {
margin: 0;
padding: 10px 14px;
border-top: 1px solid var(--line);
border-top: var(--border-line);
background: var(--panel-soft);
}
@@ -1240,7 +1043,7 @@
border-right: 0;
border-left: 0;
background: transparent;
color: var(--ink);
color: var(--text-strong);
text-align: left;
font: inherit;
cursor: pointer;
@@ -1267,27 +1070,6 @@
color: var(--muted);
}
.split-field-action {
gap: 0;
}
.split-field-action > input,
.split-field-action > select,
.split-field-action > textarea {
border-top-right-radius: 0 !important;
border-bottom-right-radius: 0 !important;
}
.split-field-action > button,
.split-field-action > .button,
.split-field-action > .btn {
align-self: stretch;
margin-left: -1px;
border-top-left-radius: 0;
border-bottom-left-radius: 0;
box-shadow: none;
}
@media (max-width: 850px) {
.managed-file-entry-head,
.managed-file-entry {
@@ -1313,7 +1095,7 @@
.managed-file-chooser-spaces {
border-right: 0;
border-bottom: 1px solid var(--line);
border-bottom: var(--border-line);
max-height: 160px;
}