30 Commits

Author SHA1 Message Date
2b34f6e305 fix(files): align adaptive tasks with live access 2026-07-21 19:15:47 +02:00
58af1a20a7 security(files): filter connectors before secret resolution 2026-07-21 19:15:41 +02:00
4722161592 feat(files): surface configured handbook tasks 2026-07-21 18:54:26 +02:00
1444ba80a0 refactor(files): centralize connector visibility 2026-07-21 18:52:33 +02:00
02ef83ecee test: commit Files retirement transaction 2026-07-21 18:18:33 +02:00
1069f85796 Refactor connector profile updates 2026-07-21 18:16:55 +02:00
1401c78c8a docs: add files workflow assurance topics 2026-07-21 17:13:53 +02:00
b6109245a7 docs(files): describe current sharing permission 2026-07-21 17:07:40 +02:00
f5d40b23c2 docs(files): link lifecycle reliability backlog 2026-07-21 16:48:13 +02:00
94ea629635 docs(files): require import workflow permissions 2026-07-21 16:33:06 +02:00
6c8a8c655d docs(files): expose adaptive handbook topics 2026-07-21 16:31:49 +02:00
062ad5ddfb docs(files): add adaptive operations handbook 2026-07-21 16:27:23 +02:00
9adfa91e74 docs(files): define connector credential retirement 2026-07-21 16:16:53 +02:00
65d8ed80b5 security(files): scrub connector credentials on deletion 2026-07-21 16:16:47 +02:00
cffe161f29 Document governed file connector boundaries 2026-07-21 15:44:00 +02:00
5248e7de4a Fail closed for SMB referral transports 2026-07-21 15:43:56 +02:00
d5d0df792b Disable connector transports without peer pinning 2026-07-21 15:38:09 +02:00
15ade8df75 Use shared settings target layout 2026-07-21 13:48:04 +02:00
f3c485ef61 Use Core explorer work-surface styling 2026-07-21 13:19:29 +02:00
0e36b20a14 refactor(webui): use core access explanation 2026-07-21 13:19:11 +02:00
06e6e7191b refactor(files): decompose S3 connector imports 2026-07-21 12:57:44 +02:00
3449cbc8a5 refactor(files): decompose campaign snapshot preparation 2026-07-21 12:57:40 +02:00
92950af6f4 refactor(files): simplify asset and delta responses 2026-07-21 12:57:37 +02:00
8826cf2890 Use managed secret controls for Files connectors 2026-07-21 12:13:00 +02:00
f2dfb6c90e Harden external file connector boundaries 2026-07-21 12:10:23 +02:00
3bc1d3489e Narrow Files backend import surfaces 2026-07-21 03:16:23 +02:00
d1051293b2 feat(files): expose campaign attachment linking 2026-07-20 20:06:04 +02:00
8c5f149f07 fix(files): route drops through the selected target 2026-07-20 16:57:34 +02:00
f3210234d3 intermittent commit 2026-07-14 13:22:11 +02:00
b8b395e8b5 Run SMB dev connector as non-root 2026-07-11 18:37:51 +02:00
55 changed files with 7907 additions and 1394 deletions

View File

@@ -78,7 +78,17 @@ Profiles can be supplied as JSON through
`GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`, `GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`,
`endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and `endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and
credential references such as `password_env`, `token_env`, or `secret_ref`. deployment-owned credential references such as `password_env`, `token_env`, or
`secret_ref`. Environment references require an exact name in the deployment-wide
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
process environment variables and may use only Files-owned encrypted password or
token values. API-created `secret_ref` values fail closed until Files has an
ownership contract that can confirm provider-side deletion. Legacy external
references are treated as non-owned: deleting a profile or credential detaches
and audits the reference but never passes it to an arbitrary secret provider.
Profile and credential deletion immediately clears encrypted values, credential
identities, deployment references, and private metadata in the same transaction
as the non-secret audit record; an audit failure rolls the deletion back.
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the `GET /api/v1/files/connectors/profiles` returns only profiles visible to the
current principal (system, tenant, user, group, or accessible campaign scope) and current principal (system, tenant, user, group, or accessible campaign scope) and
redacts secret values and environment variable names. Use the returned redacts secret values and environment variable names. Use the returned
@@ -102,8 +112,24 @@ conflict handling, source provenance, and connector audit path as direct
uploads. The Seafile provider uses account-token auth and the native file uploads. The Seafile provider uses account-token auth and the native file
download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET` download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET`
requests against the configured WebDAV endpoint. SMB profiles use requests against the configured WebDAV endpoint. SMB profiles use
`smb://server[:port]/share[/path]` endpoints and environment-backed credentials `smb://server[:port]/share[/path]` endpoints and deployment-owned or encrypted
through `smbprotocol`. stored credentials through `smbprotocol`. Because that SDK cannot accept a
preconnected socket and may follow DFS referrals to additional hosts, live SMB
access fails closed in both public-only and private-network deployments. It will
remain disabled until every initial connection and referral target can be
policy-validated and pinned.
The S3 browse/import implementation and S3 managed-storage backend currently
fail closed before creating a `boto3` client. Botocore does not yet use the
GovOPlaN pinned HTTP transport and may manage redirects itself, so neither an
explicit endpoint nor SDK endpoint discovery is allowed until both peer pinning
and redirect revalidation are enforced.
Destructive Files-module retirement applies the same credential lifecycle before
dropping tables. Every remaining Files-owned encrypted connector secret is
scrubbed and audited first, while legacy non-owned external references are
detached and identified as such in the audit record. Retirement does not claim
or attempt provider-side deletion for references Files cannot prove it owns.
Local connector development assets live in `dev/connectors/`. The compose stack Local connector development assets live in `dev/connectors/`. The compose stack
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
@@ -111,6 +137,8 @@ manual interoperability testing.
Connector and collaboration ownership boundaries are documented in Connector and collaboration ownership boundaries are documented in
`docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`. `docs/CONNECTOR_BOUNDARY.md` and `docs/DOCUMENT_COLLABORATION_BOUNDARY.md`.
The role-adaptive user, administration, integration, and operator guide is the
[Files handbook](docs/FILES_HANDBOOK.md).
ZIP uploads are processed without buffering the whole archive in memory. The API ZIP uploads are processed without buffering the whole archive in memory. The API
spools incoming ZIP request bodies to a bounded temporary file, then extracts spools incoming ZIP request bodies to a bounded temporary file, then extracts

View File

@@ -17,3 +17,9 @@ SMB_HOST_PORT=1445
SMB_SHARE_NAME=files SMB_SHARE_NAME=files
SMB_USER=govoplan SMB_USER=govoplan
SMB_PASSWORD=govoplan-smb SMB_PASSWORD=govoplan-smb
MINIO_API_HOST_PORT=9000
MINIO_CONSOLE_HOST_PORT=9001
MINIO_ROOT_USER=govoplan
MINIO_ROOT_PASSWORD=govoplan-minio
MINIO_BUCKET=govoplan

View File

@@ -9,7 +9,8 @@ binds services to localhost high ports.
```bash ```bash
cd /mnt/DATA/git/govoplan-files/dev/connectors cd /mnt/DATA/git/govoplan-files/dev/connectors
cp .env.example .env cp .env.example .env
docker compose up -d nextcloud nextcloud-db webdav smb mkdir -p data/smb data/webdav
docker compose up -d nextcloud nextcloud-db webdav smb minio
docker compose up -d seafile-db seafile-memcached seafile docker compose up -d seafile-db seafile-memcached seafile
``` ```
@@ -29,14 +30,17 @@ Endpoints:
`http://127.0.0.1:9082/seafdav/` `http://127.0.0.1:9082/seafdav/`
- WebDAV: `http://127.0.0.1:9083` - WebDAV: `http://127.0.0.1:9083`
- SMB: `smb://127.0.0.1:1445/files` - SMB: `smb://127.0.0.1:1445/files`
- MinIO/S3 API: `http://127.0.0.1:9000`, console
`http://127.0.0.1:9001`
The local fixture data under `data/` is ignored by git. The local fixture data under `data/` is ignored by git.
## GovOPlaN Profile Config ## GovOPlaN Profile Config
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Credentials should be referenced by `GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. This deployment-owned configuration
secret refs or environment variables; profile API responses only expose the may reference environment variables only when their exact names are listed in
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; profile API responses expose only the
credential source and configured state. credential source and configured state.
Example local profile file: Example local profile file:
@@ -92,6 +96,25 @@ Example local profile file:
"password_env": "SMB_PASSWORD", "password_env": "SMB_PASSWORD",
"capabilities": ["browse", "import"], "capabilities": ["browse", "import"],
"policy": { "allow": { "providers": ["smb"] } } "policy": { "allow": { "providers": ["smb"] } }
},
{
"id": "dev-s3",
"label": "Dev MinIO",
"provider": "s3",
"endpoint_url": "http://127.0.0.1:9000",
"base_path": "",
"scope_type": "system",
"credential_mode": "basic",
"username": "govoplan",
"password_env": "MINIO_ROOT_PASSWORD",
"capabilities": ["browse", "import"],
"metadata": {
"bucket": "govoplan",
"region": "us-east-1",
"path_style": true,
"verify_tls": false
},
"policy": { "allow": { "providers": ["s3"] } }
} }
] ]
} }
@@ -101,6 +124,8 @@ Start GovOPlaN with:
```bash ```bash
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
export GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST=SEAFILE_ADMIN_PASSWORD,NEXTCLOUD_ADMIN_PASSWORD,WEBDAV_PASSWORD,SMB_PASSWORD,MINIO_ROOT_PASSWORD
export GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true
``` ```
## Smoke Test ## Smoke Test
@@ -113,10 +138,13 @@ cd /mnt/DATA/git/govoplan-files/dev/connectors
/mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py /mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py
``` ```
The script reads `.env` from this directory when present, seeds tiny WebDAV, The script reads `.env` from this directory when present and seeds tiny WebDAV,
Nextcloud, and SMB fixtures, then browses and imports them through the connector Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud are browsed and imported
helper layer. Pass `--require-smb` after recreating the SMB container to fail on through the pinned connector transport. SMB and S3 product access deliberately
SMB access errors instead of reporting them as an optional skip. fails closed until their SDK transports support peer pinning, so their fixtures
are retained for transport development and reported as expected optional skips.
The `--require-smb` and `--require-s3` switches are useful only while developing
that transport support and currently make the smoke check fail by design.
SMB smoke checks need the optional Python dependency in the environment running SMB smoke checks need the optional Python dependency in the environment running
the script: the script:
@@ -125,6 +153,20 @@ the script:
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb] /mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
``` ```
S3 smoke checks need the optional boto3 dependency in the environment running
the script:
```bash
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[s3]
```
The SMB service is built from `dev/connectors/smb/` so the development share is The SMB service is built from `dev/connectors/smb/` so the development share is
deterministic: one `files` share backed by `data/smb`, with the credentials from deterministic: one `files` share backed by `data/smb`, with the credentials from
`.env`. `.env`.
The SMB image runs as the non-root `govoplan` user with UID/GID `1000` and
listens on unprivileged container port `1445`. The default host endpoint remains
`smb://127.0.0.1:1445/files`. `SMB_USER` must stay `govoplan` unless the image is
rebuilt with a matching user; `SMB_PORT` must be `1024` or higher. `SMB_PASSWORD`
is applied when the image is built, so rebuild the `smb` service after changing
it in `.env`.

View File

@@ -86,21 +86,35 @@ services:
smb: smb:
build: build:
context: ./smb context: ./smb
args:
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
image: govoplan-files-samba-dev:latest image: govoplan-files-samba-dev:latest
restart: unless-stopped restart: unless-stopped
environment: environment:
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files} SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
SMB_USER: ${SMB_USER:-govoplan} SMB_USER: ${SMB_USER:-govoplan}
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb} SMB_PORT: ${SMB_PORT:-1445}
SMB_UID: ${SMB_UID:-1000}
SMB_GID: ${SMB_GID:-1000}
ports: ports:
- "127.0.0.1:${SMB_HOST_PORT:-1445}:445" - "127.0.0.1:${SMB_HOST_PORT:-1445}:${SMB_PORT:-1445}"
volumes: volumes:
- ./data/smb:/storage - ./data/smb:/storage
minio:
image: minio/minio:latest
restart: unless-stopped
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-govoplan}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-govoplan-minio}
ports:
- "127.0.0.1:${MINIO_API_HOST_PORT:-9000}:9000"
- "127.0.0.1:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"
volumes:
- minio:/data
volumes: volumes:
nextcloud-db: nextcloud-db:
nextcloud: nextcloud:
seafile-db: seafile-db:
seafile-data: seafile-data:
minio:

View File

@@ -1,10 +1,29 @@
FROM alpine:3.20 FROM alpine:3.20
RUN apk add --no-cache samba-server samba-common-tools ARG SMB_PASSWORD=govoplan-smb
RUN apk add --no-cache samba-server samba-common-tools \
&& addgroup -S -g 1000 govoplan \
&& adduser -S -D -H -h /nonexistent -s /sbin/nologin -u 1000 -G govoplan govoplan \
&& mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba /var/log/samba/cores \
&& printf '%s\n' \
'[global]' \
' passdb backend = tdbsam' \
' private dir = /var/lib/samba/private' \
' lock directory = /run/samba' \
' state directory = /var/lib/samba' \
' cache directory = /var/cache/samba' \
' pid directory = /run/samba' \
> /etc/samba/smb.conf \
&& printf '%s\n%s\n' "$SMB_PASSWORD" "$SMB_PASSWORD" | smbpasswd -s -a govoplan \
&& smbpasswd -e govoplan \
&& chown -R govoplan:govoplan /storage /var/lib/samba /var/cache/samba /run/samba /etc/samba /var/log/samba
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
EXPOSE 445 EXPOSE 1445
USER govoplan:govoplan
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"] ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]

View File

@@ -3,21 +3,42 @@ set -eu
share_name="${SMB_SHARE_NAME:-files}" share_name="${SMB_SHARE_NAME:-files}"
user_name="${SMB_USER:-govoplan}" user_name="${SMB_USER:-govoplan}"
password="${SMB_PASSWORD:-govoplan-smb}" smb_port="${SMB_PORT:-1445}"
uid="${SMB_UID:-1000}" runtime_user="$(id -un)"
gid="${SMB_GID:-1000}" runtime_group="$(id -gn)"
mkdir -p /storage /var/lib/samba/private /run/samba case "$share_name" in
"" | *[!A-Za-z0-9_.-]*)
printf 'SMB_SHARE_NAME must contain only letters, numbers, dot, dash, or underscore.\n' >&2
exit 1
;;
esac
if ! getent group "$user_name" >/dev/null 2>&1; then case "$user_name" in
addgroup -g "$gid" "$user_name" "" | *[!A-Za-z0-9_.-]*)
printf 'SMB_USER must contain only letters, numbers, dot, dash, or underscore.\n' >&2
exit 1
;;
esac
case "$smb_port" in
"" | *[!0-9]*)
printf 'SMB_PORT must be numeric.\n' >&2
exit 1
;;
esac
if [ "$smb_port" -lt 1024 ]; then
printf 'SMB_PORT must be >= 1024 because the dev Samba container runs as a non-root user.\n' >&2
exit 1
fi fi
if ! id "$user_name" >/dev/null 2>&1; then if [ "$user_name" != "$runtime_user" ]; then
adduser -D -H -s /sbin/nologin -u "$uid" -G "$user_name" "$user_name" printf 'SMB_USER=%s is not supported by the non-root dev image; use %s or rebuild the image with a matching user.\n' "$user_name" "$runtime_user" >&2
exit 1
fi fi
chown -R "$user_name:$user_name" /storage mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba
cat > /etc/samba/smb.conf <<EOF cat > /etc/samba/smb.conf <<EOF
[global] [global]
@@ -25,12 +46,20 @@ cat > /etc/samba/smb.conf <<EOF
workgroup = WORKGROUP workgroup = WORKGROUP
security = user security = user
map to guest = Never map to guest = Never
guest account = $runtime_user
server min protocol = SMB2 server min protocol = SMB2
server signing = mandatory
smb ports = $smb_port
load printers = no load printers = no
printing = bsd printing = bsd
disable spoolss = yes disable spoolss = yes
log level = 1 log level = 1
passdb backend = tdbsam passdb backend = tdbsam
private dir = /var/lib/samba/private
lock directory = /run/samba
state directory = /var/lib/samba
cache directory = /var/cache/samba
pid directory = /run/samba
[$share_name] [$share_name]
path = /storage path = /storage
@@ -38,13 +67,16 @@ cat > /etc/samba/smb.conf <<EOF
read only = no read only = no
guest ok = no guest ok = no
valid users = $user_name valid users = $user_name
force user = $user_name force user = $runtime_user
force group = $user_name force group = $runtime_group
create mask = 0664 create mask = 0664
directory mask = 0775 directory mask = 0775
EOF EOF
printf '%s\n%s\n' "$password" "$password" | smbpasswd -s -a "$user_name" >/dev/null if ! pdbedit -L -u "$user_name" >/dev/null 2>&1; then
smbpasswd -e "$user_name" >/dev/null printf 'Samba user %s is missing from passdb. Rebuild the smb image so the non-root passdb is seeded.\n' "$user_name" >&2
printf 'Run: docker compose build --no-cache smb && docker compose up -d smb\n' >&2
exit 1
fi
exec smbd --foreground --no-process-group --debug-stdout exec smbd --foreground --no-process-group --debug-stdout -p "$smb_port"

View File

@@ -18,16 +18,17 @@ ROOT = Path(__file__).resolve().parent
def main() -> int: def main() -> int:
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.") parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.") parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
parser.add_argument("--require-s3", action="store_true", help="Fail if the S3 connector cannot browse/import.")
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.") parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
args = parser.parse_args() args = parser.parse_args()
_load_dotenv() _load_dotenv()
_default_env() _default_env()
preflight_failures = _preflight_services(require_smb=args.require_smb) preflight_failures = _preflight_services(require_smb=args.require_smb, require_s3=args.require_s3)
if preflight_failures: if preflight_failures:
for failure in preflight_failures: for failure in preflight_failures:
print(f"FAIL {failure}") print(f"FAIL {failure}")
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb") print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb minio")
return 1 return 1
_seed_webdav_fixture() _seed_webdav_fixture()
_seed_smb_fixture() _seed_smb_fixture()
@@ -36,6 +37,13 @@ def main() -> int:
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
print(f"FAIL dev-nextcloud seed failed: {exc}") print(f"FAIL dev-nextcloud seed failed: {exc}")
return 1 return 1
try:
_seed_s3_fixture()
except Exception as exc:
if args.require_s3:
print(f"FAIL dev-s3 seed failed: {exc}")
return 1
print(f"SKIP dev-s3 seed failed: {exc}")
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})} profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
if args.debug_smb: if args.debug_smb:
@@ -59,6 +67,15 @@ def main() -> int:
else: else:
print(f"SKIP {message}") print(f"SKIP {message}")
try:
_exercise_profile(profiles["dev-s3"], folder="GovOPlaN", file_path="GovOPlaN/s3-live.txt", expected="s3 live fixture")
except Exception as exc:
message = f"dev-s3: {exc}"
if args.require_s3:
failures.append(message)
else:
print(f"SKIP {message}")
if failures: if failures:
for failure in failures: for failure in failures:
print(f"FAIL {failure}") print(f"FAIL {failure}")
@@ -76,6 +93,9 @@ def _default_env() -> None:
"SMB_SHARE_NAME": "files", "SMB_SHARE_NAME": "files",
"SMB_USER": "govoplan", "SMB_USER": "govoplan",
"SMB_PASSWORD": "govoplan-smb", "SMB_PASSWORD": "govoplan-smb",
"MINIO_ROOT_USER": "govoplan",
"MINIO_ROOT_PASSWORD": "govoplan-minio",
"MINIO_BUCKET": "govoplan",
} }
for key, value in defaults.items(): for key, value in defaults.items():
os.environ.setdefault(key, value) os.environ.setdefault(key, value)
@@ -96,7 +116,7 @@ def _load_dotenv() -> None:
os.environ[key] = value.strip().strip("\"'") os.environ[key] = value.strip().strip("\"'")
def _preflight_services(*, require_smb: bool) -> list[str]: def _preflight_services(*, require_smb: bool, require_s3: bool) -> list[str]:
failures: list[str] = [] failures: list[str] = []
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php" nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
try: try:
@@ -121,6 +141,14 @@ def _preflight_services(*, require_smb: bool) -> list[str]:
pass pass
except OSError as exc: except OSError as exc:
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}") failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
if require_s3:
minio_url = f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}/minio/health/live"
try:
response = httpx.get(minio_url, timeout=3.0)
if response.status_code != 200:
failures.append(f"dev-s3 expected HTTP 200 at {minio_url}, got HTTP {response.status_code}")
except httpx.HTTPError as exc:
failures.append(f"dev-s3 is not reachable at {minio_url}: {exc}")
return failures return failures
@@ -150,6 +178,20 @@ def _profile_payloads() -> list[dict[str, object]]:
"username": os.getenv("SMB_USER", "govoplan"), "username": os.getenv("SMB_USER", "govoplan"),
"password_env": "SMB_PASSWORD", "password_env": "SMB_PASSWORD",
}, },
{
"id": "dev-s3",
"provider": "s3",
"endpoint_url": f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
"credential_mode": "basic",
"username": os.getenv("MINIO_ROOT_USER", "govoplan"),
"password_env": "MINIO_ROOT_PASSWORD",
"metadata": {
"bucket": os.getenv("MINIO_BUCKET", "govoplan"),
"region": "us-east-1",
"path_style": True,
"verify_tls": False,
},
},
] ]
@@ -211,5 +253,30 @@ def _seed_nextcloud_fixture() -> None:
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}") raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
def _seed_s3_fixture() -> None:
try:
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
except ImportError as exc:
raise RuntimeError("boto3 is not installed; install govoplan-files[s3]") from exc
bucket = os.getenv("MINIO_BUCKET", "govoplan")
client = boto3.client(
"s3",
endpoint_url=f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
aws_access_key_id=os.getenv("MINIO_ROOT_USER", "govoplan"),
aws_secret_access_key=os.getenv("MINIO_ROOT_PASSWORD", "govoplan-minio"),
region_name="us-east-1",
config=Config(s3={"addressing_style": "path"}),
)
try:
client.create_bucket(Bucket=bucket)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code")
if code not in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
raise
client.put_object(Bucket=bucket, Key="GovOPlaN/s3-live.txt", Body=b"s3 live fixture\n", ContentType="text/plain")
if __name__ == "__main__": if __name__ == "__main__":
raise SystemExit(main()) raise SystemExit(main())

View File

@@ -49,13 +49,22 @@ any remote write behavior.
Every provider must: Every provider must:
- enforce GovOPlaN profile visibility and connector policy before browse/import - enforce GovOPlaN profile visibility and connector policy before browse/import
- keep credentials as environment variables or secret references, never API - keep credentials as encrypted values or scoped secret references, never API
response values response values; process-environment references are allowed only in
deployment-owned profiles with an exact deployment allowlist
- import external files into managed storage before they are used in campaigns - import external files into managed storage before they are used in campaigns
or workflows or workflows
- preserve source provenance and revision metadata - preserve source provenance and revision metadata
- emit connector audit events for imported or accessed files - emit connector audit events for imported or accessed files
- treat remote ACLs as upstream checks, not as a replacement for GovOPlaN policy - treat remote ACLs as upstream checks, not as a replacement for GovOPlaN policy
- use a transport that pins every connection to a policy-validated DNS/IP answer
and revalidates redirects; SDK transports without that guarantee fail closed
Live SMB access is disabled in all modes until `smbprotocol` initial connections
and DFS referral targets can be pinned and policy-validated. An explicit IP is
not sufficient because the server may still issue a referral to another peer.
Live S3 access is likewise disabled until `boto3`/botocore can be bound to the
pinned transport, including SDK-managed redirects and endpoint discovery.
## Non-Goals For Files ## Non-Goals For Files

View File

@@ -34,8 +34,9 @@ Credential profile:
- provider: a specific provider or any provider - provider: a specific provider or any provider
- scope: `system` or `tenant` for administered credentials - scope: `system` or `tenant` for administered credentials
- credential mode: anonymous, environment reference, secret reference, or - credential mode: anonymous, scoped secret reference, or encrypted stored
encrypted stored password/token password/token; environment references are reserved for deployment-owned JSON
profiles and exact deployment allowlisting
- username and redacted secret configuration - username and redacted secret configuration
- credential-local policy for allow/deny rules - credential-local policy for allow/deny rules

782
docs/FILES_HANDBOOK.md Normal file
View File

@@ -0,0 +1,782 @@
# GovOPlaN Files Handbook
This handbook describes the Files module as implemented in version `0.1.9`.
It is the operational source of truth for users, process owners, administrators,
operators, auditors, and module integrators. Statements about future behavior
are marked **planned**; an unmarked statement describes the current code.
Files is a governed snapshot store. It owns managed file content, versions,
logical folders, shares, source provenance, and the evidence that another
GovOPlaN module used a particular file version. It can browse selected external
stores read-only and import a frozen copy. It is not a general remote filesystem,
a document collaboration engine, or a records-management system.
## Choose a reading path
| If you need to... | Start with... |
| --- | --- |
| Upload, find, organize, download, or import a file | [User tasks](#user-tasks) |
| Design a governed process that uses files | [Process perspective](#process-perspective) |
| Decide who can use a file or connector | [Ownership and access](#ownership-and-access) and [Administration and policy](#administration-and-policy) |
| Operate storage, connectors, backups, or recovery | [Operator runbook](#operator-runbook) |
| Integrate another GovOPlaN module | [Capabilities and integration](#capabilities-and-integration) |
| Review evidence, deletion, or security behavior | [Security, provenance, audit, deletion, and retention](#security-provenance-audit-deletion-and-retention) |
| Verify a release or scenario | [Acceptance scenarios](#acceptance-scenarios) |
| Check whether an idea exists today | [Implemented and planned boundary](#implemented-and-planned-boundary) |
## The service contract
A managed file is a tenant-scoped logical asset with exactly one user or group
owner, a normalized path, and a current version. The current version points to a
blob record containing the storage location, SHA-256 checksum, byte size, and
content type. Current service paths append a version when connector sync finds
changed content; they do not mutate the previous version record.
Content with the same tenant, SHA-256 checksum, and size can reuse one blob.
This is storage deduplication and integrity evidence, not proof of authorship or
source authenticity.
The main domain objects are:
| Object | Meaning | Lifecycle today |
| --- | --- | --- |
| File asset | The user-facing file identity, owner, logical path, description, metadata, and current version | Created, organized, shared, and soft-deleted |
| File version | A numbered snapshot of one asset and its blob | Appended by connector sync when bytes change; retained |
| File blob | Stored bytes plus checksum, size, backend, key, reference count, and optional retention timestamp | Reused within a tenant; no automated garbage collection |
| Folder | An explicit logical path in a user or group space | Created, moved/renamed through organize operations, and soft-deleted |
| Share | A grant from an asset to a user, group, tenant, or campaign with `read`, `write`, or `manage` permission | Created or updated; no public revocation endpoint yet |
| Connector profile | A governed external endpoint, scope, optional credential link, policy, and descriptive capabilities | Created, updated, disabled, or credential-scrubbed on deletion |
| Connector credential | Reusable authentication material, optionally limited to a provider and scope | Encrypted when database-managed; immediately scrubbed on deletion |
| Connector policy | Allow and deny rules inherited from system through tenant to one leaf scope | Evaluated before configuration and connector use |
| Connector space | A read-only, manually synchronized remote folder/library linked to a user or group space | Created, updated, disabled, and soft-deleted |
| Campaign attachment use | Evidence connecting a campaign job or entry to an exact asset, version, blob, checksum, and stage | Retained for campaign execution evidence |
## User tasks
The Files page is available at `/files`. Actions appear only when the current
principal has the required permission and resource access.
The configured Help Center projects these sections as independently authorized
tasks, so upload, ZIP import, organization, download, sharing, and deletion do
not disappear merely because an unrelated permission is absent. It states the
deployment's actual upload limits. External import appears as a task only when
the actor has the required permissions and at least one actor-visible profile
is eligible for the current fail-closed browse/import path; endpoint, path, and
item policy are still enforced when the operation runs.
### Choose a space
Every file user sees **My files**. Group spaces are added for active groups of
which the user is a member. The space list also exposes every tenant group to a
Files administrator; governed API operations can administer other tenant-owned
Files resources when their owner is specified. Linked connector spaces appear
beside managed spaces when they are active and visible to the user.
A connector space is intentionally read-only. It is a view of an approved
remote location and a starting point for importing or synchronizing selected
files into managed storage; it is not a mounted write-through filesystem.
### Upload files
The managed-space UI supports upload and drag-and-drop. A caller chooses the
user or group owner and destination folder. The normal per-file deployment
limit defaults to 50 MiB through `FILE_UPLOAD_MAX_BYTES`.
When a target path already exists, the operation must use one of these conflict
strategies:
- `reject` stops instead of silently replacing content;
- `rename` selects the next available `copy` name;
- `overwrite` soft-deletes the conflicting asset and creates the new asset;
- an item-specific conflict resolution can also `skip` that item.
An ordinary upload does not append a new version to an existing asset. The
`overwrite` strategy retires the old asset at that path. Version-preserving
updates currently belong to connector sync.
### Upload and unpack ZIP files
The UI can unpack a ZIP upload. The request body is spooled to a bounded
temporary file rather than buffered wholly in memory. The defaults are:
- 250 MiB compressed request and total extracted data;
- 50 MiB per extracted member;
- 1,000 non-directory members;
- encrypted archives are rejected;
- member paths are normalized and `..` traversal is rejected;
- the actual bytes read are counted, not only the ZIP header declarations.
The byte limits use `FILE_UPLOAD_ZIP_MAX_BYTES` and
`FILE_UPLOAD_MAX_BYTES`. The 1,000-member limit is currently fixed in the
extraction service.
### Organize files and folders
With `files:file:organize`, a user can:
- create logical folders;
- rename one selection directly;
- preview and apply bulk prefix, suffix, or replacement renames;
- move or copy files and folder trees between accessible user/group spaces;
- resolve target conflicts by rejecting, renaming, overwriting, or skipping;
- use drag-and-drop for move operations in the file explorer.
Copies create new assets and versions while reusing the immutable blob bytes.
Moves keep the asset identity and change its owner/path. All source and target
owners are validated; knowing an identifier does not bypass space membership.
Recursive folder deletion is the default. It soft-deletes the selected folder,
its child folders, and files below it. A non-recursive delete fails when the
folder is not empty.
### Find and download files
Files can list by owner and path, use cursor pagination, and consume incremental
changes through a watermark. The UI supports path/name pattern search and
sorting. The pattern API can resolve campaign-style wildcard selections and can
return unmatched files.
A user with download permission and resource access can download one current
version or create a ZIP archive from a selection. Downloads use an attachment
content disposition. The archive is generated in a temporary file and removed
after the response completes.
There is no dedicated content-preview service in the Files API today. File
responses expose content type, size, checksum, and current version metadata.
### Share files
The API can grant or update a share for a user, group, the tenant, or a campaign.
The supported permissions are `read`, `write`, and `manage`. Ownership remains
unchanged. A write operation accepts a `write` or `manage` share; read/download
accepts any of the three.
The current Files page shows campaign linkage but does not offer a general
user/group share editor. The API also has no share-revocation route yet. Treat
share revocation and a complete share-management UI as planned work.
### Browse and import an external file
With a visible connector profile, a user can browse the allowed remote path,
select one file, and import it into an accessible managed space. The imported
asset records the connector, provider, remote identity/path/URL, selected remote
metadata, and source revision when supplied by the provider.
Manual sync looks for an existing managed asset with the same source identity
inside the chosen owner space:
- no match creates a managed asset;
- identical checksum and size updates provenance and returns `unchanged`;
- changed bytes append a version and return `updated`.
Browse, import, and sync never write, rename, or delete the remote source.
## Process perspective
### Managed ingestion
The managed-file flow is:
1. Core authenticates the principal and evaluates the operation permission.
2. Files validates the tenant, owner, group membership, or applicable share.
3. Files normalizes the logical path and resolves conflicts explicitly.
4. Upload or connector response limits are enforced before content is retained.
5. Files calculates SHA-256 and stores or reuses a tenant blob.
6. Files creates the asset/version records and optional campaign share.
7. The database transaction commits and emits change-sequence entries.
8. Connector-originated operations also emit their connector audit evidence.
Blob storage is not part of the database transaction. An object may therefore
be left without committed metadata after a process or database failure. No
automatic orphan reconciliation exists yet; operators must account for this in
integrity checks and retention plans.
### Governed connector import
The connector flow separates four concerns:
1. An administrator defines reusable credential material.
2. An administrator defines a scoped endpoint profile that may reference that
credential.
3. System, tenant, and leaf policy sources narrow the allowed profile,
credential, provider, URL, and remote path.
4. A user optionally links an allowed remote root as a user/group connector
space, then browses and imports selected content.
Before each network operation, Files checks profile visibility, connector
policy, endpoint safety, transport support, and response size. A successful
import becomes an independent managed snapshot. Later source changes have no
effect until an explicit sync.
### Campaign attachment evidence
When Campaign uses Files, the integration follows a freeze-before-send model:
1. The campaign refers to managed user/group sources and attachment patterns.
2. Files verifies access and resolves matching managed assets.
3. A prepared campaign snapshot records the exact asset, version, blob,
checksum, size, relative path, and source provenance.
4. Files materializes those bytes for the campaign build without exposing its
database models to Campaign.
5. Campaign job/entry use is recorded and later marked as sent.
Changing the current file after preparation does not change the version already
recorded as campaign evidence. A file response is marked `audit_relevant` once
the asset has a sent campaign attachment-use record.
### Process ownership
| Concern | Owner |
| --- | --- |
| Authentication, tenants, RBAC evaluation, audit service, change sequence, settings, and module lifecycle | GovOPlaN Core |
| Managed assets, blobs, versions, folders, shares, connector baseline, provenance, and campaign attachment evidence | Files |
| Campaign definition, recipient data, message build/send state, and delivery policy | Campaign |
| Remote ACLs, remote source content, and upstream revision semantics | The external provider |
| Storage durability, egress policy, master key, secret environment, backup, and recovery | Deployment operator |
| Collaborative editing, comments, review, locks, and semantic document workflows | A future Documents/workflow/provider module |
## Ownership and access
Files applies both permission checks and resource checks. A broad operation
permission alone does not make another user's file visible.
### Resource access
- A user owns their personal space.
- A group member can use the group's file space.
- A Files administrator can access all Files resources in the active tenant.
- A file share can grant read or write access to a user, group, or the tenant.
- Campaign shares are resolved only in a verified campaign context and do not
become ordinary user shares.
- Folders are owned by a user or group; they are not independently shared.
- Soft-deleted resources are excluded from normal access and listing.
- Tenant identifiers are checked on every managed object lookup.
The `files.access` capability can explain why a principal has access: resource,
owner, administrator scope, or active share. It also explains virtual folders
that exist through child assets even when there is no explicit folder row.
Deleting an organization/access group is vetoed while it owns Files assets,
folders, connector spaces, or is the target of file shares. Reassign or remove
those relationships first.
### Operation permissions
| Permission | Allows |
| --- | --- |
| `files:file:read` | List and inspect accessible files, folders, spaces, and visible connectors |
| `files:file:download` | Download an accessible current version or ZIP archive |
| `files:file:upload` | Upload managed assets and import/sync selected connector files |
| `files:file:organize` | Create folders, rename, move/copy, and manage linked connector spaces |
| `files:file:share` | Create or update file shares |
| `files:file:delete` | Soft-delete accessible writable files and folders |
| `files:file:admin` | Administer all Files spaces and connector settings in the active tenant |
The `file_manager` role template grants all normal file operations except
`files:file:admin`. The `file_viewer` template grants read and download.
System and tenant settings permissions can also authorize the corresponding
connector administration endpoints.
## Administration and policy
### Profiles, credentials, policies, and spaces
Keep these definitions separate:
- a **credential** holds reusable authentication material and may be restricted
to a provider;
- a **profile** holds the endpoint, scope, base path, credential reference,
local policy, and descriptive operation capabilities;
- a **policy** restricts what a scope may configure or use;
- a **connector space** links one approved profile/library/path to one user or
group and always uses manual, read-only synchronization today.
Profile capability values such as `browse`, `import`, and `sync` are stored and
returned, but they are descriptive today. Provider implementation and policy
checks enforce actual availability; do not use the capability list as the sole
security control.
Profiles, credentials, and policies support `system`, `tenant`, `user`,
`group`, and `campaign` scopes. A normal user sees system and active-tenant
profiles plus leaf profiles that match the user, one of their groups, or an
accessible campaign. Disabled definitions are visible only through authorized
administrative reads.
### Policy evaluation
For a leaf scope, the effective source chain is:
```text
system -> tenant -> user | group | campaign
```
Policy fields are:
- connector/profile IDs;
- credential IDs;
- providers;
- external IDs;
- external path prefixes or glob patterns;
- external URL glob patterns.
Rules use `allow` and `deny` objects. Legacy synonyms `allowlist`, `whitelist`,
`denylist`, and `blacklist` are normalized. A matching deny at any source wins.
Every allow field defined by a source must match, so lower sources can narrow an
inherited set. The effective-policy response includes the contributing source
path and applied fields for explanation.
A parent may set `allow_lower_level_limits` for individual fields such as
`allow.providers` or `deny.external_paths`. An explicit `false` prevents a lower
scope from configuring that field. Absence does not lock the field.
Example tenant policy:
```json
{
"policy": {
"allow": {
"providers": ["webdav", "nextcloud"],
"external_urls": ["https://files.example.edu/*"],
"external_paths": ["departments/finance"]
},
"deny": {
"external_paths": ["departments/finance/private"]
},
"allow_lower_level_limits": {
"allow.providers": true,
"deny.external_paths": true
}
}
}
```
Use `POST /api/v1/files/connector-policy/evaluate` for an explainable preflight.
The normal profile browse/import/sync routes perform their own policy checks;
preflight does not replace enforcement.
### Credential rules
Database-created passwords and tokens use Core's Fernet encryption and require
the deployment `MASTER_KEY_B64` outside development/test/local environments.
Secret values, environment variable names, and local CA paths are never returned
in connector profile responses.
API-managed profiles and credentials cannot select process environment
variables, create an external `secret_ref`, or conceal secret-like values in
nested metadata. Deployment-owned JSON/file profiles may use `password_env` or
`token_env` only when the exact environment variable name appears in
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`.
Custom CA bundles must be absolute existing files in
`GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST`. TLS verification can be disabled only
in a development/test runtime.
### Provider status
| Provider | Current status |
| --- | --- |
| Seafile | Read-only native API browse/download-link import and manual sync implemented using the pinned HTTP transport; WebDAV opt-in supported |
| Nextcloud | Read-only WebDAV browse/import/manual sync implemented using the pinned HTTP transport |
| Generic WebDAV | Read-only browse/import/manual sync implemented using the pinned HTTP transport |
| SMB | Browse/import code and descriptor implemented, but every live connection fails closed until initial connections and DFS referrals can be policy-validated and pinned |
| S3 connector | Browse/import code and descriptor implemented, but every live SDK connection fails closed until botocore connections, redirects, and endpoint discovery can be validated and pinned |
| SharePoint and OneDrive | Provider keys/descriptors reserved; live Microsoft Graph browse/import is planned |
| NFS and local connector | Described as optional future providers; the local managed-storage backend is a different feature |
Provider descriptors are available from
`GET /api/v1/files/connectors/providers`. Use their `implemented`, `installed`,
and support fields for display, but expect a fail-closed transport error where
the table above says live access is disabled.
## Operator runbook
### Storage configuration
| Setting | Default | Purpose |
| --- | --- | --- |
| `FILE_STORAGE_BACKEND` | `local` | Selects `local` or `s3` managed blob storage |
| `FILE_STORAGE_LOCAL_ROOT` | `runtime/files` | Primary local write/read root |
| `FILE_STORAGE_LOCAL_FALLBACK_ROOTS` | empty | Comma-separated older read-only roots checked after the primary root |
| `FILE_STORAGE_S3_ENDPOINT_URL` and related `FILE_STORAGE_S3_*` values | deployment-specific | S3-compatible endpoint, region, credentials, and bucket |
| `FILE_UPLOAD_MAX_BYTES` | 50 MiB | Direct-upload and ZIP-member maximum |
| `FILE_UPLOAD_ZIP_MAX_BYTES` | 250 MiB | ZIP request and extracted-total maximum |
| `MASTER_KEY_B64` | development fallback only | Encrypts database-managed connector secrets |
The local backend is the operational baseline. It resolves every storage key
under the configured root and rejects escape attempts. Fallback roots support a
controlled storage-root migration: new writes go to the primary root while
reads can still find older objects.
The S3 managed-storage adapter currently fails closed before creating a boto3
client because the SDK cannot yet guarantee connection-time DNS/IP pinning and
redirect revalidation. Do not select `FILE_STORAGE_BACKEND=s3` for a live Files
deployment until that boundary is implemented and the status above changes.
Multiple API replicas require the same durable blob namespace. Separate local
container filesystems will produce incomplete reads. Until a pinned shared
object-storage transport is available, use one durable shared mount or constrain
Files traffic to a deployment topology that preserves one consistent local
root.
### Connector egress
Connector access to private networks is a deployment-wide decision:
```text
GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true|false
```
Production-like configuration validation requires an explicit value. Public-only
mode rejects any hostname whose DNS answers include a non-public address.
Private-enabled mode still rejects link-local, multicast, unspecified, and
limited-broadcast addresses.
The built-in HTTP transport:
- resolves and validates every connection attempt;
- connects the socket to the exact approved address while retaining the
original hostname for HTTP Host, TLS SNI, and certificate verification;
- does not inherit proxy settings;
- refuses redirects instead of following a new peer implicitly;
- bounds structured responses to 16 MiB and file transfers to 512 MiB by
default.
Override the connector response limits with
`GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES` and
`GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES`. The smaller applicable limit wins
when an import is also subject to `FILE_UPLOAD_MAX_BYTES`.
Never work around a pinning failure by adding a raw IP, disabling TLS, or
enabling private networks. SMB may redirect through DFS, and S3 SDKs may perform
their own redirects or endpoint discovery; both remain disabled even for an IP
literal until every connection peer can be governed.
### Backup and restore
The database and blob namespace are one logical backup set. A usable backup must
include:
- Files database rows, including asset/version/blob relationships, shares,
connector settings, and campaign attachment-use evidence;
- every object below `FILE_STORAGE_LOCAL_ROOT` and any still-used fallback root;
- the exact `MASTER_KEY_B64` needed to decrypt retained connector credentials;
- deployment-owned connector profile files, referenced CA bundles, and secret
environment configuration where those definitions are in use.
There is no Files backup/restore API and no automated blob integrity or orphan
reconciliation job. Use a write quiesce or coordinated snapshots so database
references and objects represent the same recovery point.
After restore:
1. Verify the active tenant and module migration state.
2. Verify the storage backend and roots before allowing writes.
3. Verify that the original master key is available before testing connectors.
4. Download representative files and compare bytes with their recorded SHA-256.
5. Test one authorized and one unauthorized owner/share path.
6. Test a permitted pinned HTTP connector, if connectors are configured.
7. Review audit and change-sequence continuity around the recovery point.
An inconsistent restore should fail closed for missing objects or undecryptable
credentials. Do not repair it by deleting evidence rows without an approved,
audited data-recovery decision.
### Disable, uninstall, and retire
Disabling a module preserves its persistent data. Ordinary uninstall is guarded
while Files tables contain persistent rows.
Destructive retirement is separate and irreversible at the application level.
The installer records a database snapshot, then Files scrubs and audits
remaining encrypted connector material before its database tables are dropped.
Legacy external secret references are detached and identified as non-owned;
Files never calls a provider delete operation for them.
The retirement executor drops database tables but does not delete corresponding
objects from the configured blob backend. Operators must include those orphaned
objects in the approved retention/destruction plan. Validate the installer
snapshot and independent blob backup before retirement.
### Operational signals
Use these symptoms as routing hints:
| Symptom | Likely boundary |
| --- | --- |
| `Stored object does not exist` | Database/blob restore mismatch, wrong root, or missing shared storage |
| `Stored secret cannot be decrypted` | Wrong or rotated `MASTER_KEY_B64` |
| Private/non-public endpoint blocked | Deployment-wide egress policy is public-only or DNS returned a forbidden answer |
| SDK cannot pin redirects/referrals | Expected fail-closed S3/SMB boundary, not a transient connector outage |
| Connector response exceeds limit | Remote payload exceeds connector or upload limit |
| Profile is not visible | Scope, disabled state, campaign access, or policy mismatch |
| Group removal is vetoed | The group still owns a file/folder/connector space or is a share target |
## Capabilities and integration
Other modules must integrate through Core contracts, Files capabilities, or the
public HTTP API. They must not import Files ORM models or storage helpers.
### Provided capabilities
| Capability | Purpose |
| --- | --- |
| `files.access` (`0.1.6`) | Explain resource access provenance for managed files, explicit folders, and virtual folders |
| `files.campaign_attachments` (`0.1.6`) | Resolve managed attachment matches, prepare frozen campaign snapshots, annotate built messages, share assets with a campaign, and record/mark exact attachment use |
Files requires Core principal resolution and permission evaluation. Campaign is
an optional dependency; when installed, Files consumes the optional
`campaigns.access` interface to verify campaign existence and access. Missing
optional Campaign support fails explicitly rather than bypassing the check.
### API families
All routes below are under `/api/v1/files`.
| Area | Routes |
| --- | --- |
| Spaces and content | `GET /spaces`, `GET /`, `GET /folders`, `GET /delta` |
| Upload and folders | `POST /upload`, `POST /upload-zip`, `POST /folders`, `POST /folders/delete` |
| File access | `GET /{file_id}`, `GET /{file_id}/download`, `DELETE /{file_id}`, `POST /bulk-delete` |
| Organization | `POST /bulk-rename`, `POST /transfer`, `POST /archive.zip`, `POST /resolve-patterns` |
| Sharing | `POST /{file_id}/shares`, `POST /bulk-shares` |
| Connector spaces | `GET/POST /connector-spaces`, `PATCH/DELETE /connector-spaces/{space_id}` |
| Connector catalog/discovery | `GET /connectors/providers`, `POST /connectors/discover` |
| Connector profiles | `GET/POST /connectors/profiles`, `GET/PATCH/DELETE /connectors/profiles/{profile_id}` |
| Browse/import/sync | `GET /connectors/profiles/{profile_id}/browse`, `POST /connectors/profiles/{profile_id}/import`, `POST /connectors/profiles/{profile_id}/sync` |
| Credentials | `GET/POST /connectors/credentials`, `GET/PATCH/DELETE /connectors/credentials/{credential_id}` |
| Policy | `GET/PUT /connectors/policies/{scope_type}`, `POST /connector-policy/evaluate` |
| Incremental connector settings | `GET /connectors/settings/delta` |
Consumers should use cursor/watermark contracts instead of assuming an
unbounded complete list. The default full-list page size is 500 and public page
sizes are capped at 1,000.
### Integration invariants
An integrating module should:
- ask Files/Core for access rather than trusting a submitted file ID;
- retain an exact version/blob/checksum at every governed evidence point;
- import external content before using it in a campaign, report, workflow, or
generated document;
- preserve Files provenance when producing a derived managed snapshot;
- treat source provenance as captured context, not cryptographic attestation;
- avoid writing remote providers through the baseline connector layer;
- define a separate capability when it needs behavior beyond managed snapshots;
- tolerate Files being absent when the integration is declared optional.
Future Postbox, Templates/Reports, BI, Documents, DMS, and workflow modules
should keep their own domain state and use Files for governed input/output
snapshots. Long-running provider sync, OAuth, remote mutation, and
provider-specific health belong in connector modules rather than expanding the
Files baseline indiscriminately.
## Security, provenance, audit, deletion, and retention
### Security controls implemented
- Core authentication, tenant scoping, CSRF/API handling, and RBAC protect the
public routes.
- Owner and share checks protect individual resources after operation-scope
checks.
- Logical paths reject traversal and storage keys cannot escape the local root.
- Upload, ZIP extraction, connector response, and S3 stream code use bounded
reads; encrypted ZIP archives are rejected.
- Connector HTTP sockets use connection-time DNS/IP validation and pinning,
redirects are refused, and unsafe SDK transports fail before client creation.
- Database-managed connector passwords/tokens are encrypted; responses redact
secrets and deployment references.
- API metadata is recursively checked for secret-like values.
- Downloads use sanitized attachment filenames.
- SHA-256 and byte size are recorded for every blob/version.
The module does **not** currently provide malware scanning, content disarm and
reconstruction, a file-type allowlist, per-user quota, at-rest encryption for
local blob bytes, or a dedicated preview sandbox. Deployments that require
these controls must supply them outside Files until explicit module contracts
exist.
### Provenance
Connector-originated assets can retain:
- source type;
- connector/profile ID and provider;
- external ID, path, and URL;
- revision and revision label;
- observation/import timestamps when supplied;
- selected provider metadata.
The normalized provenance and source revision are returned in file responses,
carried into campaign attachment matches, and included in connector audit
events. External metadata is provider/user input and is not a digital signature.
### Audit and change evidence
Files records canonical audit events for:
- connector discovery attempts, before the attempted external I/O;
- connector imports and manual syncs;
- download/archive access to connector-originated managed files;
- immediate connector profile and credential deletion/scrubbing;
- credential scrubbing during destructive module retirement.
Connector audit details include the managed asset/version/blob, checksum, size,
operation, source revision, and provenance where applicable. Deletion audit
details name secret/reference kinds but never the secret values.
Assets, folders, shares, profiles, credentials, policies, and connector spaces
also feed Core's incremental change sequence for UI synchronization. A change
entry is not equivalent to a canonical audit event. Ordinary local upload,
rename, share, and soft-delete operations do not yet all emit dedicated Files
audit events.
Campaign attachment-use records provide separate domain evidence for an exact
file version used in campaign preparation and delivery.
### Deletion semantics
The word "delete" has different meanings by object type:
| Object | Current delete behavior |
| --- | --- |
| File asset | Sets `deleted_at`; content, versions, blob references, and campaign evidence remain |
| Folder | Sets `deleted_at`; recursive deletion also soft-deletes descendants |
| Connector space | Sets `deleted_at` and disables the space |
| Connector credential | Immediately disables the tombstone and clears username, encrypted password/token, environment references, legacy external reference, mode, and private metadata; dependent profiles are disabled and detached |
| Connector profile | Immediately disables the tombstone and clears credential links/material/references and private metadata |
| Legacy `secret_ref` | Detached and audited as an unowned external reference; no provider deletion is attempted or claimed |
| Module retirement | Scrubs/audits credential material, then drops Files database tables; blob-backend cleanup is an operator responsibility |
Credential/profile scrubbing and its audit event use the same database
transaction. If audit creation fails, the deletion rolls back. Repeating a
delete against an already scrubbed tombstone does not recreate secret evidence.
### Retention boundary
File versions and blobs are effectively retained indefinitely today. Although a
blob has `ref_count` and `retained_until` fields, no complete retention-policy,
legal-hold, hard-purge, or garbage-collection service enforces them. There is
also no supported user restore endpoint for soft-deleted assets/folders and no
automated orphan-object reconciliation.
Do not promise erasure, timed retention, legal hold, or self-service recovery
from the current soft-delete behavior. Those require an explicit, auditable
retention/purge design that preserves campaign and other evidence references.
## Acceptance scenarios
These scenarios describe expected behavior at the current boundary.
### Personal and group ownership
Given Alice has `files:file:upload` and belongs to Finance, when she uploads to
her personal or Finance space, then the asset has the selected owner and tenant.
Given Bob is outside Finance and has no share or Files admin permission, the
same asset ID must not make the asset readable to Bob.
### Safe ZIP import
Given a ZIP member contains `../../secret.txt`, is encrypted, exceeds 50 MiB,
or pushes actual extracted bytes above 250 MiB, unpacking must fail without a
committed managed asset. A valid archive must preserve normalized relative paths
below the chosen logical folder.
### Explicit conflicts
Given a target path already exists, `reject` must leave it unchanged, `rename`
must choose a non-conflicting path, and `overwrite` must soft-delete the old
asset before creating the replacement. A per-item `skip` must not create that
item.
### Policy-denied connector
Given the tenant permits WebDAV only below `departments/finance` and denies its
`private` child, a browse/import/sync request below the denied child must return
an explainable policy denial. Directly invoking the import endpoint must not
bypass the same rule.
### Pinned connector transport
Given public-only mode and DNS returns any private address, the connection must
be rejected before a socket opens. Given private mode, the HTTP connection must
still use the validated address and reject redirects. SMB and S3 must fail
before their SDK clients connect until all SDK-managed peers can be pinned.
### Imported evidence and sync
Given a permitted WebDAV file is imported, its managed response and audit event
must carry source identity, revision when available, current version, checksum,
and size. Re-syncing identical bytes must return `unchanged`; changed bytes must
create a higher version while preserving the previous version.
### Campaign freeze
Given a campaign snapshot selected version V1, when the managed asset later
advances to V2, the prepared/sent attachment-use evidence must still identify
V1 and its original blob/checksum.
### Immediate credential deletion
Given a database credential contains an encrypted password and is referenced by
two profiles, deleting it must scrub the credential, disable/detach both
profiles, record non-secret audit evidence, and publish connector-setting
changes in one transaction. If audit creation fails, no part of the deletion
may commit.
### Recovery
Given a coordinated database/blob backup and the original master key, restoring
it must allow representative downloads whose bytes match recorded SHA-256
values. A missing blob or wrong key must surface an error rather than silently
returning different content or credentials.
## Implemented and planned boundary
| Area | Implemented now | Planned or explicitly outside the current boundary |
| --- | --- | --- |
| Managed storage | Local durable-root backend, fallback read roots, tenant blob deduplication, checksums | Operational S3 after pinned SDK transport ([#34](https://git.add-ideas.de/add-ideas/govoplan-files/issues/34)); automated integrity/orphan reconciliation ([#36](https://git.add-ideas.de/add-ideas/govoplan-files/issues/36)) |
| Upload | Bounded direct upload, drag-and-drop UI, ZIP spool/extract limits, explicit conflicts | Malware scanning, quotas, type policy, resumable/chunked upload |
| Organization | Folders, bulk rename preview/apply, move/copy, drag-and-drop, ZIP download, pattern resolution | General file-history UI and user-driven append-version/restore |
| Sharing | User/group/tenant/campaign grant or permission update through API; campaign linkage display | Share revocation/expiry and complete general share-management UI ([#37](https://git.add-ideas.de/add-ideas/govoplan-files/issues/37)) |
| Deletion/retention | Soft-delete assets/folders/spaces; immediate audited connector-secret scrubbing | File restore API, hard purge, retention policy, legal hold, and blob GC ([#38](https://git.add-ideas.de/add-ideas/govoplan-files/issues/38)) |
| Connector governance | Scoped profiles/credentials/policies, effective source explanation, separate credentials, linked user/group spaces | Provider-owned external secret lifecycle; API `secret_ref` remains rejected |
| HTTP connectors | Pinned, bounded, no-redirect Seafile and WebDAV/Nextcloud browse/import/manual sync | Background/folder sync, remote mutation, long-running transfer workers |
| SMB and S3 connectors | Provider descriptors and browse/import logic | Live access only after SDK connections plus DFS referrals/redirects are validated and pinned ([SMB #35](https://git.add-ideas.de/add-ideas/govoplan-files/issues/35), [S3 #34](https://git.add-ideas.de/add-ideas/govoplan-files/issues/34)) |
| Other providers | Reserved SharePoint/OneDrive keys and NFS/local descriptors | Graph/OAuth/provider paging, NFS deployment integration, DMS connectors |
| Connector spaces | User/group link, browse, manual selected-file sync, edit/disable/delete | Background sync, remote writes/deletes, full conflict-reporting jobs |
| Profile capabilities | Stored and displayed | Enforce capability flags as an independent operation gate |
| Audit | Connector discovery/import/sync/access and connector deletion; campaign exact-use evidence | Dedicated canonical audit events for every ordinary Files mutation |
| Preview | File metadata and attachment download | Dedicated safe content-preview service |
| Campaign | Stable capability-based frozen attachments and sent-use evidence | Campaign-specific process state remains in Campaign |
| Collaboration | Governed input/output snapshots | Co-editing, comments, review, presence, locks, and semantic document versions belong to Documents/workflow/provider modules |
## Release and change checklist
Before releasing Files:
1. Keep `pyproject.toml`, root `package.json`, WebUI `package.json`, and the
module manifest version aligned. Version alignment is a release gate.
2. Run the Files Python tests and security/static-analysis gates from the
GovOPlaN meta repository.
3. Build/type-check the WebUI through the Core host application.
4. Verify database migrations on an upgrade copy and a clean database.
5. Exercise an allowed and denied owner/share path.
6. Exercise upload, ZIP bounds, conflict handling, download, and soft deletion.
7. Exercise connector policy explanation and one pinned HTTP provider where
configured; verify S3/SMB still fail closed.
8. Verify credential deletion scrubs dependents and produces audit evidence.
9. Verify a campaign attachment snapshot still identifies its exact version and
checksum after the current file changes.
10. Update the implemented/planned table whenever a boundary changes.
## Related documents
- [Repository overview](../README.md)
- [Connector ownership boundary](CONNECTOR_BOUNDARY.md)
- [Connector spaces design and implementation history](CONNECTOR_SPACES.md)
- [Document collaboration boundary](DOCUMENT_COLLABORATION_BOUNDARY.md)
Where an older planning section conflicts with current code or this handbook's
implemented/planned table, verify the code and update both documents in the same
reviewable documentation slice.

View File

@@ -1,6 +1,6 @@
{ {
"name": "@govoplan/files-webui", "name": "@govoplan/files-webui",
"version": "0.1.8", "version": "0.1.9",
"private": true, "private": true,
"type": "module", "type": "module",
"main": "webui/src/index.ts", "main": "webui/src/index.ts",
@@ -19,7 +19,7 @@
"LICENSE" "LICENSE"
], ],
"peerDependencies": { "peerDependencies": {
"@govoplan/core-webui": "^0.1.8", "@govoplan/core-webui": "^0.1.9",
"lucide-react": "^1.23.0", "lucide-react": "^1.23.0",
"react": "^19.0.0", "react": "^19.0.0",
"react-dom": "^19.0.0", "react-dom": "^19.0.0",

View File

@@ -4,19 +4,22 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "govoplan-files" name = "govoplan-files"
version = "0.1.8" version = "0.1.9"
description = "GovOPlaN files module with backend and WebUI integration." description = "GovOPlaN files module with backend and WebUI integration."
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
license = { file = "LICENSE" } license = { file = "LICENSE" }
authors = [{ name = "GovOPlaN" }] authors = [{ name = "GovOPlaN" }]
dependencies = [ dependencies = [
"govoplan-core>=0.1.8", "govoplan-core>=0.1.9",
"defusedxml>=0.7,<1", "defusedxml>=0.7,<1",
"python-multipart>=0.0.31,<1", "python-multipart>=0.0.31,<1",
] ]
[project.optional-dependencies] [project.optional-dependencies]
s3 = [
"boto3>=1.34,<2",
]
smb = [ smb = [
"smbprotocol>=1.13", "smbprotocol>=1.13",
] ]

View File

@@ -17,6 +17,7 @@ from govoplan_files.backend.storage.campaign_attachments import (
managed_match_payloads, managed_match_payloads,
prepared_campaign_snapshot, prepared_campaign_snapshot,
public_attachment_summary_payload, public_attachment_summary_payload,
share_assets_with_campaign,
) )
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
from govoplan_files.backend.storage.files import current_version_and_blob from govoplan_files.backend.storage.files import current_version_and_blob
@@ -44,6 +45,7 @@ class FilesCampaignCapability:
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files) annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs) record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
current_version_and_blob = staticmethod(current_version_and_blob) current_version_and_blob = staticmethod(current_version_and_blob)
share_assets_with_campaign = staticmethod(share_assets_with_campaign)
@staticmethod @staticmethod
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None: def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:

View File

@@ -1,12 +1,18 @@
from __future__ import annotations from __future__ import annotations
from datetime import datetime from datetime import datetime
from typing import Any
from sqlalchemy import event, inspect from sqlalchemy import event
from sqlalchemy.orm import Session as OrmSession from sqlalchemy.orm import Session as OrmSession
from govoplan_core.core.change_sequence import record_change from govoplan_core.core.change_sequence import record_change
from govoplan_core.core.sqlalchemy_change_tracking import (
ensure_object_id,
has_attr_changes,
object_state,
operation_for_soft_deletable,
previous_value,
)
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid
FILES_MODULE_ID = "files" FILES_MODULE_ID = "files"
@@ -39,7 +45,7 @@ def _record_files_changes(session: OrmSession, _flush_context: object, _instance
def _record_asset_change(session: OrmSession, asset: FileAsset) -> None: def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
operation = _operation_for_soft_deletable( operation = operation_for_soft_deletable(
asset, asset,
changed_attrs=( changed_attrs=(
"owner_type", "owner_type",
@@ -70,7 +76,7 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
"owner_type": asset.owner_type, "owner_type": asset.owner_type,
"owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id), "owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id),
"path": asset.display_path, "path": asset.display_path,
"previous_path": _previous_value(asset, "display_path"), "previous_path": previous_value(asset, "display_path"),
"filename": asset.filename, "filename": asset.filename,
"deleted_at": _isoformat(asset.deleted_at), "deleted_at": _isoformat(asset.deleted_at),
}, },
@@ -78,7 +84,7 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
def _record_folder_change(session: OrmSession, folder: FileFolder) -> None: def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
operation = _operation_for_soft_deletable( operation = operation_for_soft_deletable(
folder, folder,
changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"), changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"),
) )
@@ -99,17 +105,17 @@ def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
"owner_type": folder.owner_type, "owner_type": folder.owner_type,
"owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id), "owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id),
"path": folder.path, "path": folder.path,
"previous_path": _previous_value(folder, "path"), "previous_path": previous_value(folder, "path"),
"deleted_at": _isoformat(folder.deleted_at), "deleted_at": _isoformat(folder.deleted_at),
}, },
) )
def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None: def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None:
state = inspect(share) state = object_state(share)
if not share.file_asset_id: if not share.file_asset_id:
return return
if not state.pending and not _has_attr_changes( if not state.pending and not has_attr_changes(
state, state,
("file_asset_id", "target_type", "target_id", "permission", "revoked_at"), ("file_asset_id", "target_type", "target_id", "permission", "revoked_at"),
): ):
@@ -135,44 +141,8 @@ def _record_share_visibility_change(session: OrmSession, share: FileShare) -> No
) )
def _operation_for_soft_deletable(obj: object, *, changed_attrs: tuple[str, ...]) -> str | None:
state = inspect(obj)
if state.pending:
return "created"
if not _has_attr_changes(state, changed_attrs):
return None
if "deleted_at" in state.attrs:
history = state.attrs.deleted_at.history
if history.has_changes():
if any(value is not None for value in history.added):
return "deleted"
if any(value is not None for value in history.deleted):
return "created"
return "updated"
def _has_attr_changes(state: Any, attrs: tuple[str, ...]) -> bool:
return any(name in state.attrs and state.attrs[name].history.has_changes() for name in attrs)
def _previous_value(obj: object, attr_name: str) -> str | None:
state = inspect(obj)
if attr_name not in state.attrs:
return None
history = state.attrs[attr_name].history
if not history.has_changes() or not history.deleted:
return None
value = history.deleted[0]
return str(value) if value is not None else None
def _ensure_id(obj: object) -> str: def _ensure_id(obj: object) -> str:
resource_id = getattr(obj, "id", None) return ensure_object_id(obj, new_uuid)
if resource_id:
return str(resource_id)
resource_id = new_uuid()
setattr(obj, "id", resource_id)
return resource_id
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None: def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:

View File

@@ -1 +1,25 @@
from govoplan_files.backend.db.models import * from govoplan_files.backend.db.models import (
CampaignAttachmentUse,
FileAsset,
FileBlob,
FileConnectorCredential,
FileConnectorPolicy,
FileConnectorProfile,
FileConnectorSpace,
FileFolder,
FileShare,
FileVersion,
)
__all__ = [
"CampaignAttachmentUse",
"FileAsset",
"FileBlob",
"FileConnectorCredential",
"FileConnectorPolicy",
"FileConnectorProfile",
"FileConnectorSpace",
"FileFolder",
"FileShare",
"FileVersion",
]

View File

@@ -0,0 +1,461 @@
from __future__ import annotations
from sqlalchemy.orm import Session
from govoplan_core.core.campaigns import (
CAPABILITY_CAMPAIGNS_ACCESS,
CampaignAccessProvider,
)
from govoplan_core.core.modules import (
DocumentationCondition,
DocumentationContext,
DocumentationLink,
DocumentationTopic,
)
from govoplan_files.backend.storage.archives import ZIP_UPLOAD_MAX_FILES
from govoplan_files.backend.storage.access import user_group_ids
from govoplan_files.backend.storage.connector_visibility import (
connector_profile_usable_for_import,
visible_connector_profiles_for_actor,
)
_DEFAULT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024
_DEFAULT_ZIP_MAX_BYTES = 250 * 1024 * 1024
_FILES_READ_SCOPE = "files:file:read"
_FILES_UPLOAD_SCOPE = "files:file:upload"
def documentation_topics(
context: DocumentationContext,
) -> tuple[DocumentationTopic, ...]:
if context.documentation_type != "user":
return ()
upload_limit = _configured_positive_int(
context.settings,
"file_upload_max_bytes",
default=_DEFAULT_UPLOAD_MAX_BYTES,
)
zip_limit = _configured_positive_int(
context.settings,
"file_upload_zip_max_bytes",
default=_DEFAULT_ZIP_MAX_BYTES,
)
topics: list[DocumentationTopic] = []
if upload_limit is not None:
topics.append(_upload_topic(upload_limit))
if upload_limit is not None and zip_limit is not None:
topics.append(_zip_topic(upload_limit, zip_limit))
topics.append(_connector_import_topic(context))
return tuple(topics)
def _upload_topic(max_bytes: int) -> DocumentationTopic:
limit = _format_byte_limit(max_bytes)
return DocumentationTopic(
id="files.workflow.upload-managed-files",
title="Upload managed files",
summary=f"Upload files to a personal or accessible group space; each uploaded file may contain at most {limit}.",
body=(
f"The current deployment accepts at most {limit} for each ordinary upload. "
"A name conflict is never resolved silently: reject stops the upload, rename chooses a copy name, and overwrite retires the old asset before creating a new one."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=39,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(
label="Files handbook",
href="govoplan-files/docs/FILES_HANDBOOK.md",
kind="repository",
),
),
related_modules=("campaigns",),
unlocks=(
"Users and connected processes can place governed content in managed storage.",
),
source_module_id="files",
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and upload managed files.",
"The destination personal or group space grants this account write access; upload permission alone does not grant access to every space.",
],
"steps": [
"Open Files and choose My files or an accessible group space.",
"Open the intended destination folder and choose Upload, or drag files into the file list.",
"For every name conflict, explicitly reject, rename, overwrite, or skip the affected item.",
"Wait for the upload to finish, then open the resulting file details.",
],
"outcome": "Each accepted file is stored as a governed managed asset in the selected space.",
"verification": "Confirm the owner, logical path, size, checksum, and current version in Files.",
"constraints": [
{
"id": "ordinary-upload-size",
"label": "Maximum size per file",
"description": f"The current safe upload limit is {limit} per file.",
"values": [limit],
},
],
"related_topic_ids": [
"files.workflow.upload-and-unpack-zip",
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
],
},
)
def _zip_topic(max_file_bytes: int, max_zip_bytes: int) -> DocumentationTopic:
member_limit = _format_byte_limit(max_file_bytes)
archive_limit = _format_byte_limit(max_zip_bytes)
return DocumentationTopic(
id="files.workflow.upload-and-unpack-zip",
title="Upload and unpack a ZIP archive",
summary=(
f"Safely unpack up to {ZIP_UPLOAD_MAX_FILES:,} files from a ZIP whose request and extracted total are each limited to {archive_limit}."
),
body=(
f"The current deployment limits the ZIP request and actual extracted total to {archive_limit}, each member to {member_limit}, "
f"and the archive to {ZIP_UPLOAD_MAX_FILES:,} non-directory members. Encrypted archives and unsafe member paths are rejected. "
"Actual extracted bytes are counted instead of trusting ZIP headers."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=40,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(
label="Files handbook",
href="govoplan-files/docs/FILES_HANDBOOK.md",
kind="repository",
),
),
unlocks=(
"A bounded archive can become governed managed files without trusting archive paths or size declarations.",
),
source_module_id="files",
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and upload managed files.",
"The archive is not encrypted and fits the current configured limits.",
"The destination personal or group space grants this account write access.",
],
"steps": [
"Open Files and choose the managed destination space and folder.",
"Enable Unpack ZIP uploads, then choose or drag the ZIP archive.",
"Resolve every destination conflict explicitly.",
"Wait for extraction and finalization to finish before leaving the page.",
],
"outcome": "Accepted archive members are stored as separate governed managed assets below the selected folder.",
"verification": "Confirm the expected member paths and inspect representative file sizes, checksums, and versions.",
"constraints": [
{
"id": "zip-request-and-total",
"label": "Maximum ZIP request and extracted total",
"description": f"Both the compressed request and the actual extracted total are limited to {archive_limit}.",
"values": [archive_limit],
},
{
"id": "zip-member-size",
"label": "Maximum extracted member size",
"description": f"Each extracted file is limited to {member_limit}.",
"values": [member_limit],
},
{
"id": "zip-member-count",
"label": "Maximum file count",
"description": f"A ZIP may contain at most {ZIP_UPLOAD_MAX_FILES:,} non-directory members.",
"values": [f"{ZIP_UPLOAD_MAX_FILES:,} files"],
},
],
"related_topic_ids": [
"files.workflow.upload-managed-files",
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
],
},
)
def _connector_import_topic(context: DocumentationContext) -> DocumentationTopic:
principal = context.principal
if not _has_all_scopes(principal, (_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE)):
return _connector_import_limitation(
"Connector import is not available to this account because both permission to view Files and permission to upload managed files are required."
)
tenant_id = _safe_text_attribute(principal, "tenant_id")
user_id = _safe_text_attribute(getattr(principal, "user", None), "id")
session = context.session
if not tenant_id or not user_id or not isinstance(session, Session):
return _connector_import_limitation(
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
)
try:
member_group_ids = _actor_group_ids(
session,
principal=principal,
tenant_id=tenant_id,
user_id=user_id,
include_admin_groups=False,
)
connector_group_ids = (
_actor_group_ids(
session,
principal=principal,
tenant_id=tenant_id,
user_id=user_id,
include_admin_groups=True,
)
if _has_all_scopes(principal, ("files:file:admin",))
else member_group_ids
)
profiles = visible_connector_profiles_for_actor(
session,
tenant_id=tenant_id,
user_id=user_id,
group_ids=connector_group_ids,
settings=context.settings,
campaign_visible=_campaign_visibility(
context,
session=session,
tenant_id=tenant_id,
user_id=user_id,
group_ids=member_group_ids,
),
include_effective_policy=True,
)
usable_profiles = tuple(
profile
for profile in profiles
if connector_profile_usable_for_import(profile)
)
except Exception:
return _connector_import_limitation(
"Connector import availability could not be safely evaluated for this request. Try again, or ask a Files administrator to verify an actor-visible connection."
)
if not usable_profiles:
return _connector_import_limitation(
"No connection visible to this account is currently eligible to offer an enabled, credential-ready, policy-allowed browse/import path through a pinning-safe provider. Ask a Files administrator to configure or authorize one."
)
return DocumentationTopic(
id="files.workflow.import-managed-snapshot",
title="Import an external file as a governed snapshot",
summary="Browse an authorized connection read-only and import one selected file into managed storage as a frozen, traceable snapshot.",
body=(
"At least one connection visible to this account is currently eligible to offer the governed browse/import path. "
"The selected remote path and item are re-authorized when used, and browse, import, and sync never mutate the remote source. "
"The managed snapshot stays unchanged until an explicit manual sync."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "campaign_manager", "report_author"),
order=41,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(
label="Files handbook",
href="govoplan-files/docs/FILES_HANDBOOK.md",
kind="repository",
),
),
related_modules=("campaigns",),
unlocks=(
"Campaigns, reports, and workflows can consume a managed snapshot with stable source evidence.",
),
source_module_id="files",
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list", "files.connector-import"],
"prerequisites": [
"You may view and upload managed files.",
"At least one enabled, credential-ready, policy-allowed connection using a pinning-safe provider is visible to this account.",
"The selected remote path and item must pass their operation-time policy checks.",
"The managed destination space grants this account write access.",
],
"steps": [
"Open Files and choose a managed destination space.",
"Choose Sync from connection, select an available connection, and browse to the permitted remote file.",
"Import the selected file and resolve any destination conflict explicitly.",
"Review the managed file's source and current-version details before using it in another task.",
],
"current_configuration": [
"At least one actor-visible connection is eligible to offer a safe browse/import operation; the selected endpoint, path, and item are still checked when used.",
"Every selected remote path and item is re-authorized at operation time.",
],
"outcome": "The external content is a tenant-managed snapshot with a checksum, exact version, and recorded source context.",
"verification": "Reopen the managed file and confirm its recorded source context, source revision when available, checksum, and current version.",
"related_topic_ids": [
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
"files.reference.snapshot-provenance-and-capabilities",
],
},
)
def _connector_import_limitation(message: str) -> DocumentationTopic:
return DocumentationTopic(
id="files.connector-import-unavailable",
title="External file import is not currently available",
summary=message,
body=(
f"{message} Files never exposes connection endpoints, storage paths, credential references, or raw connector policies in user documentation."
),
layer="available",
documentation_types=("user",),
order=41,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(_FILES_READ_SCOPE, _FILES_UPLOAD_SCOPE),
),
),
links=(DocumentationLink(label="Files", href="/files", kind="runtime"),),
source_module_id="files",
metadata={
"kind": "reference",
"screen": "Files",
"help_contexts": ["files.list", "files.connector-import"],
"limitations": [message],
},
)
def _configured_positive_int(
settings: object | None, name: str, *, default: int
) -> int | None:
raw_value = getattr(settings, name, default) if settings is not None else default
try:
value = int(raw_value)
except (TypeError, ValueError):
return None
return value if value > 0 else None
def _format_byte_limit(value: int) -> str:
units = ((1024**3, "GiB"), (1024**2, "MiB"), (1024, "KiB"))
for divisor, label in units:
if value % divisor == 0:
return f"{value // divisor:,} {label} ({value:,} bytes)"
return f"{value:,} bytes"
def _has_all_scopes(principal: object | None, scopes: tuple[str, ...]) -> bool:
checker = getattr(principal, "has", None)
if callable(checker):
try:
return all(bool(checker(scope)) for scope in scopes)
except Exception:
return False
granted = {str(scope) for scope in getattr(principal, "scopes", ())}
return all(scope in granted for scope in scopes)
def _safe_text_attribute(value: object | None, name: str) -> str:
try:
result = getattr(value, name, "")
except Exception:
return ""
return str(result or "")
def _principal_group_ids(principal: object) -> tuple[str, ...]:
try:
values = getattr(principal, "group_ids", ())
except Exception:
return ()
return tuple(str(value) for value in values if str(value))
def _actor_group_ids(
session: Session,
*,
principal: object,
tenant_id: str,
user_id: str,
include_admin_groups: bool,
) -> tuple[str, ...]:
try:
values = user_group_ids(
session,
tenant_id=tenant_id,
user_id=user_id,
include_admin_groups=include_admin_groups,
)
except Exception:
return _principal_group_ids(principal)
return tuple(str(value) for value in values if str(value))
def _campaign_visibility(
context: DocumentationContext,
*,
session: Session,
tenant_id: str,
user_id: str,
group_ids: tuple[str, ...],
):
principal = context.principal
registry = context.registry
if not _has_all_scopes(principal, ("campaigns:campaign:read",)):
return None
try:
if not registry.has_capability(CAPABILITY_CAMPAIGNS_ACCESS):
return None
capability = registry.require_capability(CAPABILITY_CAMPAIGNS_ACCESS)
except Exception:
return None
if not isinstance(capability, CampaignAccessProvider):
return None
def campaign_visible(campaign_id: str) -> bool:
try:
return capability.campaign_exists(
session, tenant_id=tenant_id, campaign_id=campaign_id
) and capability.can_read_campaign(
session,
tenant_id=tenant_id,
campaign_id=campaign_id,
user_id=user_id,
group_ids=group_ids,
tenant_admin=_has_all_scopes(principal, ("tenant:*",)),
)
except Exception:
return False
return campaign_visible

View File

@@ -1,11 +1,17 @@
from __future__ import annotations from __future__ import annotations
from dataclasses import replace
from pathlib import Path from pathlib import Path
from sqlalchemy import inspect
from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER
from govoplan_core.core.files import CAPABILITY_FILES_ACCESS from govoplan_core.core.files import CAPABILITY_FILES_ACCESS
from govoplan_core.core.module_guards import drop_table_retirement_provider, persistent_table_uninstall_guard from govoplan_core.core.module_guards import drop_table_retirement_provider, persistent_table_uninstall_guard
from govoplan_core.core.modules import ( from govoplan_core.core.modules import (
DocumentationCondition,
DocumentationLink,
DocumentationTopic,
FrontendModule, FrontendModule,
MigrationSpec, MigrationSpec,
ModuleContext, ModuleContext,
@@ -19,10 +25,60 @@ from govoplan_core.core.modules import (
from govoplan_core.db.base import Base from govoplan_core.db.base import Base
from govoplan_files.backend.change_tracking import register_files_change_tracking from govoplan_files.backend.change_tracking import register_files_change_tracking
from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata from govoplan_files.backend.db import models as file_models # noqa: F401 - populate Files ORM metadata
from govoplan_files.backend.documentation import documentation_topics
register_files_change_tracking() register_files_change_tracking()
_files_table_retirement_provider = drop_table_retirement_provider(
file_models.FileBlob,
file_models.FileFolder,
file_models.FileAsset,
file_models.FileVersion,
file_models.FileShare,
file_models.FileConnectorCredential,
file_models.FileConnectorPolicy,
file_models.FileConnectorProfile,
file_models.FileConnectorSpace,
file_models.CampaignAttachmentUse,
label="Files",
)
def _files_retirement_provider(session: object | None, module_id: str):
plan = _files_table_retirement_provider(session, module_id)
base_executor = plan.destroy_data_executor
if base_executor is None:
return plan
def executor(execute_session: object, execute_module_id: str) -> None:
if not hasattr(execute_session, "get_bind") or not hasattr(execute_session, "query"):
raise RuntimeError("No database session is available for Files credential retirement.")
live_inspector = inspect(execute_session.get_bind())
if any(
live_inspector.has_table(table_name)
for table_name in (
file_models.FileConnectorCredential.__tablename__,
file_models.FileConnectorProfile.__tablename__,
)
):
from govoplan_files.backend.storage.connector_credential_deletion import (
delete_connector_credentials_for_retirement,
)
delete_connector_credentials_for_retirement(execute_session)
base_executor(execute_session, execute_module_id)
return replace(
plan,
destroy_data_warnings=(
*plan.destroy_data_warnings,
"Files-owned encrypted connector credentials are scrubbed and audited immediately before tables are dropped; legacy non-owned external references are detached without claiming provider-side deletion.",
),
destroy_data_executor=executor,
)
def _permission(scope: str, label: str, description: str) -> PermissionDefinition: def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
module_id, resource, action = scope.split(":", 2) module_id, resource, action = scope.split(":", 2)
return PermissionDefinition( return PermissionDefinition(
@@ -42,7 +98,7 @@ PERMISSIONS = (
_permission("files:file:download", "Download files", "Download managed files and generated archives."), _permission("files:file:download", "Download files", "Download managed files and generated archives."),
_permission("files:file:upload", "Upload files", "Upload new managed file versions."), _permission("files:file:upload", "Upload files", "Upload new managed file versions."),
_permission("files:file:organize", "Organize files", "Create folders, rename, move or copy managed files."), _permission("files:file:organize", "Organize files", "Create folders, rename, move or copy managed files."),
_permission("files:file:share", "Share files", "Grant or revoke managed file shares."), _permission("files:file:share", "Share files", "Grant or update managed file shares; revocation is not available yet."),
_permission("files:file:delete", "Delete files", "Delete or hide managed files and folders where policy allows it."), _permission("files:file:delete", "Delete files", "Delete or hide managed files and folders where policy allows it."),
_permission("files:file:admin", "Administer file spaces", "Administer all file spaces in the tenant."), _permission("files:file:admin", "Administer file spaces", "Administer all file spaces in the tenant."),
) )
@@ -112,7 +168,7 @@ def _files_router(context: ModuleContext):
manifest = ModuleManifest( manifest = ModuleManifest(
id="files", id="files",
name="Files", name="Files",
version="0.1.8", version="0.1.9",
required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR), required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR),
optional_dependencies=("campaigns",), optional_dependencies=("campaigns",),
provides_interfaces=( provides_interfaces=(
@@ -138,24 +194,434 @@ manifest = ModuleManifest(
package_name="@govoplan/files-webui", package_name="@govoplan/files-webui",
nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),), nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),),
), ),
documentation=(
DocumentationTopic(
id="files.workflow.organize-managed-files",
title="Organize managed files and folders",
summary="Create folders and rename, move, or copy accessible managed content with explicit conflict handling.",
body=(
"Organization stays inside governed personal or group spaces. Moves preserve the asset identity, while copies create new assets and versions that reuse immutable blob bytes. "
"Every target conflict must be rejected, renamed, overwritten, or skipped explicitly."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=42,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:organize"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Move or copy API", href="/api/v1/files/transfer", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
unlocks=("Managed content can be placed at stable logical paths without bypassing space access.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and organize managed files.",
"You have write or owner access to every source item and destination space used by the operation; the global organize permission alone does not grant resource access.",
],
"steps": [
"Open Files and select the personal or group space to organize.",
"Create the required destination folders or select the files and folders to rename, move, or copy.",
"Choose the destination and resolve each target conflict explicitly.",
"Apply the operation and reopen the destination folder.",
],
"outcome": "The selected content has the intended governed owner and logical path.",
"verification": "Confirm each resulting path and owner; for a move, also confirm the old path is gone, and for a copy, confirm the source remains.",
"related_topic_ids": [
"files.workflow.upload-managed-files",
"files.workflow.find-and-download-files",
"files.workflow.delete-managed-files",
],
},
),
DocumentationTopic(
id="files.workflow.find-and-download-files",
title="Find and download managed files",
summary="Search accessible managed content and download a current file version or a ZIP archive of a selection.",
body=(
"Files can be sorted and searched by logical path or name pattern. A download always uses the accessible current version; a multi-file selection can be generated as a temporary ZIP archive. "
"There is no dedicated content-preview service yet."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=43,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:download"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
unlocks=("Authorized users can retrieve governed current versions without gaining organization or sharing authority.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": ["You may view and download managed files in the relevant space."],
"steps": [
"Open Files and select the relevant personal or group space.",
"Navigate folders, sort the list, or use a path/name pattern to find the intended content.",
"Review the displayed owner, path, size, checksum, and version details.",
"Download one current file version, or select several files and choose Download ZIP.",
],
"outcome": "The authorized current file bytes or generated archive are downloaded to the local device.",
"verification": "Confirm the downloaded names and, where integrity matters, compare the file bytes with the displayed checksum.",
"related_topic_ids": [
"files.workflow.organize-managed-files",
"files.reference.snapshot-provenance-and-capabilities",
],
},
),
DocumentationTopic(
id="files.workflow.share-managed-files",
title="Grant or update access to managed files",
summary="Grant or update read, write, or manage access through a supporting workflow or API client without changing ownership.",
body=(
"The Files service can grant or update a share for a user, group, tenant, or campaign. The Files page does not yet provide a general share editor, and the API has no share-revocation route. "
"Do not use this task when access must later be revoked until that explicit capability is implemented."
),
layer="available",
documentation_types=("user",),
audience=("file_manager", "process_participant"),
order=44,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:share"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Grant or update a share", href="/api/v1/files/{file_id}/shares", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("campaigns",),
unlocks=("A supporting process can grant governed file access without changing file ownership.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and share the managed file and have write/manage access to that specific asset; the global share permission alone does not grant resource access.",
"A supporting workflow or API client is available; the Files page has no general share editor yet.",
"The process does not require share revocation, because no revocation route exists yet.",
],
"steps": [
"Open Files and verify the file owner, logical path, current version, and checksum.",
"Through the supporting workflow, identify the intended user, group, tenant, or campaign and choose read, write, or manage access.",
"Grant or update the share without changing file ownership.",
"Have the intended recipient verify access and an unrelated account verify denial.",
],
"limitations": [
"The Files page has no general share editor.",
"Shares can be granted or updated, but not revoked through the current API.",
],
"outcome": "The requested access is granted or updated while the managed asset keeps its owner.",
"verification": "Test one intended and one denied access path. Do not claim that the share can later be revoked.",
"related_topic_ids": [
"files.workflow.find-and-download-files",
"files.assurance.process-and-release-readiness",
],
},
),
DocumentationTopic(
id="files.workflow.delete-managed-files",
title="Delete managed files and folders",
summary="Soft-delete accessible managed files or a folder tree where current policy allows it.",
body=(
"Deletion hides the selected managed assets rather than hard-purging their stored evidence. Folder deletion is recursive by default and includes child folders and files; a non-recursive request fails for a non-empty folder. "
"There is no self-service restore or hard-purge workflow today."
),
layer="configured",
documentation_types=("user",),
audience=("file_user", "file_manager", "process_participant"),
order=45,
conditions=(
DocumentationCondition(
required_modules=("files",),
required_scopes=("files:file:read", "files:file:delete"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
unlocks=("Authorized users can remove obsolete content from active file views while preserving the current soft-delete boundary.",),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files",
"help_contexts": ["files.list"],
"prerequisites": [
"You may view and delete the selected managed content and have write or owner access to every affected asset or folder.",
"You have reviewed the complete folder tree when deleting recursively.",
],
"steps": [
"Open Files and select the files or folder to delete.",
"Review the selection and, for a folder, all content below it.",
"Confirm the delete action.",
"Refresh or reopen the space and verify that the selected paths are no longer active.",
],
"limitations": [
"Deletion is soft deletion, not a hard purge.",
"There is no self-service restore or hard-purge workflow.",
],
"outcome": "The selected content is hidden from active Files views under the current soft-delete model.",
"verification": "Confirm the deleted paths no longer appear in the active space; do not treat the action as physical erasure.",
"related_topic_ids": [
"files.workflow.organize-managed-files",
"files.assurance.process-and-release-readiness",
],
},
),
DocumentationTopic(
id="files.governed-connectors-and-provenance",
title="Govern file connections and credential deletion",
summary="Keep endpoint profiles, reusable credentials, and inherited connector policy separate, and understand what DELETE removes immediately.",
body=(
"System, tenant, and one user/group/campaign leaf form the effective policy chain: deny rules win and every configured allow rule must match. "
"Responses redact secret values and deployment references. Deleting a database-managed credential or profile immediately scrubs Files-owned encrypted material and private metadata in the same transaction as a non-secret audit event; dependent profiles are disabled, while legacy non-owned references are only detached and audited."
),
layer="configured",
documentation_types=("admin",),
audience=("file_admin", "tenant_admin", "system_admin", "security_auditor"),
order=50,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(
"files:file:admin",
"admin:settings:read",
"admin:settings:write",
"system:settings:read",
"system:settings:write",
),
),
),
links=(
DocumentationLink(label="Tenant connector administration", href="/admin?section=tenant-file-connectors", kind="runtime"),
DocumentationLink(label="System connector administration", href="/admin?section=system-file-connectors", kind="runtime"),
DocumentationLink(label="Tenant connector policy API", href="/api/v1/files/connectors/policies/tenant", kind="api"),
DocumentationLink(label="Connector credentials API", href="/api/v1/files/connectors/credentials", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("access", "audit", "mail"),
unlocks=("Scoped, explainable external-file access without exposing credentials to consuming modules.",),
configuration_keys=(
"GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON",
"GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST",
"GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST",
"MASTER_KEY_B64",
),
metadata={
"kind": "reference",
"route": "/admin?section=tenant-file-connectors",
"screen": "File connections",
"section": "Profiles, credentials, and effective connector policy",
"security_invariants": [
"New API-managed external secret references fail closed until Files can prove ownership and provider-side deletion.",
"Deletion and destructive retirement scrub Files-owned encrypted connector material before completion and emit non-secret audit evidence.",
"Legacy non-owned external references are detached and audited, never sent to an arbitrary provider delete operation.",
],
"related_topic_ids": [
"files.workflow.import-managed-snapshot",
"files.reference.integrity-recovery-and-fail-closed-transports",
"mail.profiles-and-policy",
],
},
),
DocumentationTopic(
id="files.reference.integrity-recovery-and-fail-closed-transports",
title="Operate Files integrity, recovery, and connector transport safety",
summary="Back up database evidence, blob bytes, and the encryption key as one recovery unit, and keep unsupported SDK transports fail-closed.",
body=(
"Local durable storage is the operational baseline. Recover Files from a coordinated database/blob snapshot with the original master key, then verify representative checksums and access paths. "
"Live S3 managed storage/connectors and SMB connectors fail closed until botocore redirects/endpoint discovery and SMB initial connections/DFS referrals support connection-time DNS/IP pinning. Destructive module retirement drops database tables but does not remove backend blob objects."
),
layer="configured",
documentation_types=("admin",),
audience=("operator", "system_admin", "security_auditor"),
order=51,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(
"files:file:admin",
"admin:settings:read",
"system:settings:read",
"system:audit:read",
),
),
),
links=(
DocumentationLink(label="System file connections", href="/admin?section=system-file-connectors", kind="runtime"),
DocumentationLink(label="Connector provider status", href="/api/v1/files/connectors/providers", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("audit", "ops"),
unlocks=("Recoverable managed-file evidence without weakening outbound peer validation.",),
configuration_keys=(
"FILE_STORAGE_BACKEND",
"FILE_STORAGE_LOCAL_ROOT",
"FILE_STORAGE_LOCAL_FALLBACK_ROOTS",
"FILE_UPLOAD_MAX_BYTES",
"FILE_UPLOAD_ZIP_MAX_BYTES",
"MASTER_KEY_B64",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
"GOVOPLAN_CONNECTOR_MAX_STRUCTURED_RESPONSE_BYTES",
"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES",
),
metadata={
"kind": "reference",
"route": "/admin?section=system-file-connectors",
"screen": "System file connections and deployment operations",
"section": "Storage integrity, backup/recovery, and fail-closed transports",
"recovery_unit": ["Files database rows", "managed blob namespace", "MASTER_KEY_B64", "deployment-owned connector configuration"],
"verification": "After restore, download representative files, compare their bytes with recorded SHA-256 values, verify authorized and denied access, and test one permitted pinned HTTP connector.",
"related_topic_ids": [
"files.governed-connectors-and-provenance",
"files.reference.snapshot-provenance-and-capabilities",
],
},
),
DocumentationTopic(
id="files.reference.snapshot-provenance-and-capabilities",
title="Integrate through managed snapshots and Files capabilities",
summary="Other modules consume stable Files capabilities or HTTP contracts and retain exact version evidence instead of importing Files internals.",
body=(
"Use files.access to explain resource access and files.campaign_attachments to freeze campaign inputs at an exact asset, version, blob, checksum, and source revision. "
"Import external content before a governed use, preserve provenance on derived snapshots, and keep collaboration, provider sync, OAuth, remote mutation, and domain workflow state in their owning modules."
),
layer="available",
documentation_types=("admin", "user"),
audience=("module_integrator", "file_admin", "campaign_admin", "process_designer"),
order=52,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=("files:file:read", "files:file:upload", "files:file:admin"),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files API", href="/api/v1/files", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("campaigns", "docs"),
unlocks=("Campaign, report, template, workflow, and document modules can exchange governed input/output snapshots.",),
metadata={
"kind": "reference",
"route": "/files",
"screen": "Files and module integration",
"section": "Managed snapshot provenance and capability boundaries",
"provided_interfaces": ["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
"provenance_fields": ["connector_id", "provider", "external_id", "external_path", "revision", "metadata"],
"related_topic_ids": [
"files.workflow.import-managed-snapshot",
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
],
},
),
DocumentationTopic(
id="files.assurance.process-and-release-readiness",
title="Assure a Files-backed process and release",
summary="Check a process against the implemented Files boundary, exercise permitted and denied paths, and retain evidence before approving a release or operational use.",
body=(
"A process owner must distinguish implemented controls from planned capabilities before relying on Files. A release is not ready until all package and manifest versions align and representative authorization, upload limits, conflict handling, download, deletion, connector, and recovery paths have been exercised. "
"Current limitations include no general share-management UI or share revocation, no self-service restore or hard purge, no enforced retention or legal hold, and no dedicated canonical audit event for every ordinary Files mutation. Record these limitations in the process assessment instead of treating soft deletion or change-sequence entries as stronger evidence."
),
layer="configured",
documentation_types=("admin", "user"),
audience=("process_owner", "release_manager", "file_admin", "operator", "security_auditor"),
order=53,
conditions=(
DocumentationCondition(
required_modules=("files",),
any_scopes=(
"files:file:read",
"files:file:admin",
"admin:settings:read",
"system:settings:read",
"system:audit:read",
),
),
),
links=(
DocumentationLink(label="Files", href="/files", kind="runtime"),
DocumentationLink(label="Files API", href="/api/v1/files", kind="api"),
DocumentationLink(label="Connector provider status", href="/api/v1/files/connectors/providers", kind="api"),
DocumentationLink(label="Files handbook", href="govoplan-files/docs/FILES_HANDBOOK.md", kind="repository"),
),
related_modules=("campaigns", "audit", "ops"),
unlocks=("Process owners and release managers can approve Files use against explicit controls, evidence, and known gaps.",),
configuration_keys=(
"FILE_STORAGE_BACKEND",
"FILE_STORAGE_LOCAL_ROOT",
"FILE_UPLOAD_MAX_BYTES",
"FILE_UPLOAD_ZIP_MAX_BYTES",
"MASTER_KEY_B64",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
),
metadata={
"kind": "workflow",
"route": "/files",
"screen": "Files process and release assurance",
"help_contexts": ["files.list"],
"prerequisites": [
"A named process owner has defined the intended users, data classification, retention expectations, and integrations.",
"A candidate release is installed on a clean database and an upgrade copy with aligned Python, root package, WebUI package, and module-manifest versions.",
"Representative permitted and denied accounts, bounded test files, and a coordinated database/blob/key recovery set are available.",
],
"steps": [
"Compare the process requirements with the handbook's implemented/planned boundary and record every unsupported requirement or compensating control.",
"Confirm version alignment, apply migrations on clean and upgrade databases, and run the repository plus meta-repository security and static-analysis gates.",
"Exercise allowed and denied personal, group, and share access with representative accounts.",
"Exercise bounded upload and ZIP handling, every required conflict strategy, organization, download, and soft deletion.",
"Where connectors are configured, verify policy explanation and one pinned HTTP provider; verify live S3 and SMB access still fails closed.",
"Restore a coordinated database/blob/key backup and compare representative downloaded bytes with their recorded SHA-256 checksums.",
"Record the tested versions, results, known limitations, evidence locations, residual risks, owner, and approval decision.",
],
"outcome": "The process or release has an explicit approval record tied to aligned versions, representative evidence, known limitations, and owned residual risks.",
"verification": "A reviewer can reproduce the recorded allow/deny, integrity, connector, and recovery checks and can trace each unmet requirement to a documented limitation or accepted compensating control.",
"related_topic_ids": [
"files.workflow.upload-managed-files",
"files.workflow.upload-and-unpack-zip",
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
"files.workflow.share-managed-files",
"files.workflow.delete-managed-files",
"files.workflow.import-managed-snapshot",
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
"files.reference.snapshot-provenance-and-capabilities",
],
},
),
),
documentation_providers=(documentation_topics,),
migration_spec=MigrationSpec( migration_spec=MigrationSpec(
module_id="files", module_id="files",
metadata=Base.metadata, metadata=Base.metadata,
script_location=str(Path(__file__).with_name("migrations") / "versions"), script_location=str(Path(__file__).with_name("migrations") / "versions"),
retirement_supported=True, retirement_supported=True,
retirement_provider=drop_table_retirement_provider( retirement_provider=_files_retirement_provider,
file_models.FileBlob,
file_models.FileFolder,
file_models.FileAsset,
file_models.FileVersion,
file_models.FileShare,
file_models.FileConnectorCredential,
file_models.FileConnectorPolicy,
file_models.FileConnectorProfile,
file_models.FileConnectorSpace,
file_models.CampaignAttachmentUse,
label="Files",
),
retirement_notes="Destructive retirement drops files-owned database tables after the installer captures a database snapshot.", retirement_notes="Destructive retirement drops files-owned database tables after the installer captures a database snapshot.",
), ),
uninstall_guard_providers=( uninstall_guard_providers=(

File diff suppressed because it is too large Load Diff

View File

@@ -1,36 +1,10 @@
from __future__ import annotations from __future__ import annotations
from typing import Any from govoplan_core.core.runtime import ModuleRuntimeState
_runtime_registry: object | None = None _runtime = ModuleRuntimeState("Files")
_runtime_settings: object | None = None
configure_runtime = _runtime.configure_runtime
def configure_runtime(*, registry: object | None = None, settings: object | None = None) -> None: get_registry = _runtime.get_registry
global _runtime_registry, _runtime_settings get_settings = _runtime.get_settings
if registry is not None: settings = _runtime.settings
_runtime_registry = registry
if settings is not None:
_runtime_settings = settings
def get_registry() -> object | None:
return _runtime_registry
def get_settings() -> object:
if _runtime_settings is not None:
return _runtime_settings
try:
from govoplan_core.settings import settings as legacy_settings
except ModuleNotFoundError as exc:
raise RuntimeError("GovOPlaN Files runtime settings are not configured") from exc
return legacy_settings
class SettingsProxy:
def __getattr__(self, name: str) -> Any:
return getattr(get_settings(), name)
settings = SettingsProxy()

View File

@@ -287,7 +287,7 @@ class FileConnectorDiscoveryCandidate(BaseModel):
class FileConnectorDiscoveryRequest(BaseModel): class FileConnectorDiscoveryRequest(BaseModel):
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
endpoint_url: str endpoint_url: str
base_path: str | None = None base_path: str | None = None
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none" credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
@@ -309,7 +309,7 @@ class FileConnectorDiscoveryResponse(BaseModel):
class FileConnectorProfileCreateRequest(BaseModel): class FileConnectorProfileCreateRequest(BaseModel):
id: str id: str
label: str label: str
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
endpoint_url: str | None = None endpoint_url: str | None = None
base_path: str | None = None base_path: str | None = None
enabled: bool = True enabled: bool = True
@@ -325,7 +325,7 @@ class FileConnectorProfileCreateRequest(BaseModel):
class FileConnectorProfileUpdateRequest(BaseModel): class FileConnectorProfileUpdateRequest(BaseModel):
label: str | None = None label: str | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
endpoint_url: str | None = None endpoint_url: str | None = None
base_path: str | None = None base_path: str | None = None
enabled: bool | None = None enabled: bool | None = None
@@ -342,7 +342,7 @@ class FileConnectorProfileUpdateRequest(BaseModel):
class FileConnectorCredentialCreateRequest(BaseModel): class FileConnectorCredentialCreateRequest(BaseModel):
id: str id: str
label: str label: str
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
enabled: bool = True enabled: bool = True
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant" scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
scope_id: str | None = None scope_id: str | None = None
@@ -354,7 +354,7 @@ class FileConnectorCredentialCreateRequest(BaseModel):
class FileConnectorCredentialUpdateRequest(BaseModel): class FileConnectorCredentialUpdateRequest(BaseModel):
label: str | None = None label: str | None = None
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
enabled: bool | None = None enabled: bool | None = None
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
credentials: FileConnectorProfileCredentialsRequest | None = None credentials: FileConnectorProfileCredentialsRequest | None = None
@@ -404,6 +404,8 @@ class FileConnectorBrowseResponse(BaseModel):
path: str = "" path: str = ""
library_id: str | None = None library_id: str | None = None
read_only: bool = True read_only: bool = True
next_continuation_token: str | None = None
has_more: bool = False
decision: dict[str, Any] decision: dict[str, Any]
items: list[FileConnectorBrowseItem] items: list[FileConnectorBrowseItem]

View File

@@ -17,6 +17,7 @@ from govoplan_files.backend.storage.paths import filename_from_path, normalize_f
_ZIP_READ_CHUNK_SIZE = 1024 * 1024 _ZIP_READ_CHUNK_SIZE = 1024 * 1024
ZIP_UPLOAD_MAX_FILES = 1000
def _read_zip_member( def _read_zip_member(
@@ -76,7 +77,7 @@ def extract_zip_upload(
conflict_resolutions: Iterable[FileConflictResolution] | None = None, conflict_resolutions: Iterable[FileConflictResolution] | None = None,
metadata: dict[str, Any] | None = None, metadata: dict[str, Any] | None = None,
is_admin: bool = False, is_admin: bool = False,
max_files: int = 1000, max_files: int = ZIP_UPLOAD_MAX_FILES,
max_file_bytes: int = 50 * 1024 * 1024, max_file_bytes: int = 50 * 1024 * 1024,
max_total_bytes: int = 250 * 1024 * 1024, max_total_bytes: int = 250 * 1024 * 1024,
) -> list[UploadedStoredFile]: ) -> list[UploadedStoredFile]:

View File

@@ -4,6 +4,12 @@ from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from typing import Iterable, Protocol from typing import Iterable, Protocol
from govoplan_core.security.outbound_http import (
OutboundHttpError,
response_limit,
validate_unpinned_sdk_http_url,
)
from govoplan_files.backend.runtime import settings from govoplan_files.backend.runtime import settings
@@ -92,19 +98,29 @@ class S3StorageBackend:
@property @property
def client(self): def client(self):
try:
endpoint_url = validate_unpinned_sdk_http_url(
self.endpoint_url,
label="File storage S3 endpoint",
)
except OutboundHttpError as exc:
raise StorageBackendError(str(exc)) from exc
try: try:
import boto3 import boto3
except ModuleNotFoundError as exc: except ModuleNotFoundError as exc:
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
return boto3.client( return boto3.client(
"s3", "s3",
endpoint_url=self.endpoint_url, endpoint_url=endpoint_url,
region_name=self.region_name, region_name=self.region_name,
aws_access_key_id=self.access_key_id, aws_access_key_id=self.access_key_id,
aws_secret_access_key=self.secret_access_key, aws_secret_access_key=self.secret_access_key,
) )
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None: def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
max_bytes = response_limit("file")
if len(data) > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
kwargs = {"Bucket": self.bucket, "Key": key, "Body": data} kwargs = {"Bucket": self.bucket, "Key": key, "Body": data}
if content_type: if content_type:
kwargs["ContentType"] = content_type kwargs["ContentType"] = content_type
@@ -113,19 +129,39 @@ class S3StorageBackend:
def get_bytes(self, key: str) -> bytes: def get_bytes(self, key: str) -> bytes:
try: try:
obj = self.client.get_object(Bucket=self.bucket, Key=key) obj = self.client.get_object(Bucket=self.bucket, Key=key)
return obj["Body"].read() max_bytes = response_limit("file")
body = obj["Body"]
try:
_reject_declared_object_size(obj, max_bytes=max_bytes)
data = body.read(max_bytes + 1)
if len(data) > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
return data
finally:
if hasattr(body, "close"):
body.close()
except Exception as exc: # pragma: no cover - depends on S3 backend except Exception as exc: # pragma: no cover - depends on S3 backend
raise StorageBackendError(str(exc)) from exc raise StorageBackendError(str(exc)) from exc
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]: def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
try: try:
obj = self.client.get_object(Bucket=self.bucket, Key=key) obj = self.client.get_object(Bucket=self.bucket, Key=key)
max_bytes = response_limit("file")
body = obj["Body"] body = obj["Body"]
try:
_reject_declared_object_size(obj, max_bytes=max_bytes)
total = 0
while True: while True:
chunk = body.read(chunk_size) chunk = body.read(chunk_size)
if not chunk: if not chunk:
break break
total += len(chunk)
if total > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
yield chunk yield chunk
finally:
if hasattr(body, "close"):
body.close()
except Exception as exc: # pragma: no cover - depends on S3 backend except Exception as exc: # pragma: no cover - depends on S3 backend
raise StorageBackendError(str(exc)) from exc raise StorageBackendError(str(exc)) from exc
@@ -140,6 +176,17 @@ class S3StorageBackend:
return False return False
def _reject_declared_object_size(obj: object, *, max_bytes: int) -> None:
if not isinstance(obj, dict):
return
try:
declared_size = int(obj.get("ContentLength"))
except (TypeError, ValueError):
return
if declared_size > max_bytes:
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
def _fallback_roots() -> tuple[Path, ...]: def _fallback_roots() -> tuple[Path, ...]:
raw = getattr(settings, "file_storage_local_fallback_roots", "") or "" raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip()) return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())

View File

@@ -12,10 +12,11 @@ from typing import Any, Iterator
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from govoplan_files.backend.db.models import FileAsset from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend from govoplan_files.backend.storage.backends import StorageBackend, StorageBackendError, get_storage_backend
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.files import current_versions_and_blobs, list_assets_for_user from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
from govoplan_files.backend.storage.access import ensure_owner_access
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
@@ -37,6 +38,7 @@ class ManagedAttachmentFile:
checksum_sha256: str checksum_sha256: str
size_bytes: int size_bytes: int
content_type: str | None content_type: str | None
linked_to_campaign: bool = True
source_provenance: dict[str, Any] | None = None source_provenance: dict[str, Any] | None = None
source_revision: str | None = None source_revision: str | None = None
@@ -56,6 +58,7 @@ class PreparedCampaignSnapshot:
raw_json: dict[str, Any] raw_json: dict[str, Any]
managed_files_by_local_path: dict[str, ManagedAttachmentFile] managed_files_by_local_path: dict[str, ManagedAttachmentFile]
shared_assets: list[FileAsset] shared_assets: list[FileAsset]
candidate_assets: list[FileAsset]
def parse_managed_source(value: object) -> tuple[str, str] | None: def parse_managed_source(value: object) -> tuple[str, str] | None:
@@ -117,6 +120,99 @@ def _iter_rule_dicts(attachments: dict[str, Any], raw_json: dict[str, Any]):
yield rule yield rule
def _managed_base_sources(raw_json: dict[str, Any]) -> list[tuple[str, str, str]]:
attachments = raw_json.get("attachments")
if not isinstance(attachments, dict):
return []
base_paths = attachments.get("base_paths")
if not isinstance(base_paths, list):
return []
sources: list[tuple[str, str, str]] = []
seen: set[tuple[str, str, str]] = set()
for item in base_paths:
if not isinstance(item, dict):
continue
parsed_source = parse_managed_source(item.get("source"))
if parsed_source is None:
continue
owner_type, owner_id = parsed_source
old_path = str(item.get("path") or ".").strip() or "."
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
key = (owner_type, owner_id, logical_root)
if key in seen:
continue
seen.add(key)
sources.append(key)
return sources
def _campaign_linked_asset_ids(
session: Session,
*,
tenant_id: str,
campaign_id: str,
asset_ids: list[str],
) -> set[str]:
if not asset_ids:
return set()
linked: set[str] = set()
for offset in range(0, len(asset_ids), 900):
chunk = asset_ids[offset : offset + 900]
rows = (
session.query(FileShare.file_asset_id)
.filter(
FileShare.tenant_id == tenant_id,
FileShare.file_asset_id.in_(chunk),
FileShare.target_type == "campaign",
FileShare.target_id == campaign_id,
FileShare.revoked_at.is_(None),
)
.all()
)
linked.update(row[0] for row in rows)
return linked
def _candidate_assets_for_managed_sources(
session: Session,
*,
tenant_id: str,
campaign_id: str,
raw_json: dict[str, Any],
user_id: str,
is_admin: bool,
shared_assets: list[FileAsset],
) -> tuple[list[FileAsset], set[str]]:
assets_by_id: dict[str, FileAsset] = {asset.id: asset for asset in shared_assets}
for owner_type, owner_id, logical_root in _managed_base_sources(raw_json):
ensure_owner_access(
session,
tenant_id=tenant_id,
owner_type=owner_type,
owner_id=owner_id,
user_id=user_id,
is_admin=is_admin,
)
for asset in list_assets_for_user(
session,
tenant_id=tenant_id,
user_id=user_id,
owner_type=owner_type,
owner_id=owner_id,
path_prefix=logical_root or None,
is_admin=is_admin,
):
assets_by_id.setdefault(asset.id, asset)
candidate_assets = sorted(assets_by_id.values(), key=lambda asset: (asset.display_path, asset.updated_at, asset.id))
linked_ids = _campaign_linked_asset_ids(
session,
tenant_id=tenant_id,
campaign_id=campaign_id,
asset_ids=[asset.id for asset in candidate_assets],
)
return candidate_assets, linked_ids
def _selected_base_path( def _selected_base_path(
rule: dict[str, Any], rule: dict[str, Any],
prepared_by_id: dict[str, tuple[str, str]], prepared_by_id: dict[str, tuple[str, str]],
@@ -136,6 +232,177 @@ def _selected_base_path(
return None return None
def _prepared_attachment_config(raw_json: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any], list[Any]]:
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
attachments = prepared_json.get("attachments")
if not isinstance(attachments, dict):
attachments = {}
prepared_json["attachments"] = attachments
base_paths = attachments.get("base_paths")
if not isinstance(base_paths, list):
base_paths = []
return prepared_json, attachments, base_paths
def _campaign_snapshot_assets(
session: Session,
*,
tenant_id: str,
campaign_id: str,
prepared_json: dict[str, Any],
include_unlinked_candidates: bool,
user_id: str,
is_admin: bool,
) -> tuple[list[FileAsset], list[FileAsset], set[str]]:
shared_assets = list_assets_for_user(
session,
tenant_id=tenant_id,
user_id="",
campaign_id=campaign_id,
is_admin=True,
)
if not include_unlinked_candidates:
return shared_assets, shared_assets, {asset.id for asset in shared_assets}
candidate_assets, linked_asset_ids = _candidate_assets_for_managed_sources(
session,
tenant_id=tenant_id,
campaign_id=campaign_id,
raw_json=prepared_json,
user_id=user_id,
is_admin=is_admin,
shared_assets=shared_assets,
)
return shared_assets, candidate_assets, linked_asset_ids
def _assets_grouped_by_owner(assets: list[FileAsset]) -> dict[tuple[str, str], list[FileAsset]]:
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
for asset in assets:
owner_id = _asset_owner_id(asset)
if owner_id:
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
return assets_by_owner
def _materialize_managed_asset(
asset: FileAsset,
*,
owner_id: str,
logical_root: str,
local_root: Path,
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
backend: StorageBackend | None,
include_bytes: bool,
linked_asset_ids: set[str],
) -> tuple[str, ManagedAttachmentFile] | None:
relative_path = _relative_asset_path(asset, logical_root)
if not relative_path:
return None
target = _safe_local_target(local_root, relative_path)
target.parent.mkdir(parents=True, exist_ok=True)
version, blob = version_blobs[asset.id]
if include_bytes:
try:
data = backend.get_bytes(blob.storage_key) if backend else b""
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
target.write_bytes(data)
else:
target.touch()
local_key = str(target.resolve())
return local_key, ManagedAttachmentFile(
local_path=local_key,
asset_id=asset.id,
version_id=version.id,
blob_id=blob.id,
display_path=asset.display_path,
relative_path=normalize_logical_path(relative_path),
filename=asset.filename,
owner_type=asset.owner_type,
owner_id=owner_id,
checksum_sha256=blob.checksum_sha256,
size_bytes=blob.size_bytes,
content_type=blob.content_type,
linked_to_campaign=asset.id in linked_asset_ids,
source_provenance=source_provenance_from_metadata(asset.metadata_),
source_revision=source_revision_from_metadata(asset.metadata_),
)
def _prepare_managed_base_paths(
base_paths: list[Any],
*,
materialized_root: Path,
assets_by_owner: dict[tuple[str, str], list[FileAsset]],
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
backend: StorageBackend | None,
include_bytes: bool,
linked_asset_ids: set[str],
) -> tuple[
dict[str, ManagedAttachmentFile],
dict[str, tuple[str, str]],
dict[str, list[tuple[str, str]]],
tuple[str, str] | None,
]:
manifest: dict[str, ManagedAttachmentFile] = {}
prepared_by_id: dict[str, tuple[str, str]] = {}
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
first_prepared: tuple[str, str] | None = None
for index, item in enumerate(base_paths):
if not isinstance(item, dict):
continue
parsed_source = parse_managed_source(item.get("source"))
if parsed_source is None:
continue
owner_type, owner_id = parsed_source
old_path = str(item.get("path") or ".").strip() or "."
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
local_root.mkdir(parents=True, exist_ok=True)
local_root_string = str(local_root.resolve())
prepared = (base_path_id, local_root_string)
prepared_by_id[base_path_id] = prepared
prepared_by_old_path.setdefault(old_path, []).append(prepared)
if first_prepared is None:
first_prepared = prepared
item["path"] = local_root_string
for asset in assets_by_owner.get((owner_type, owner_id), []):
materialized = _materialize_managed_asset(
asset,
owner_id=owner_id,
logical_root=logical_root,
local_root=local_root,
version_blobs=version_blobs,
backend=backend,
include_bytes=include_bytes,
linked_asset_ids=linked_asset_ids,
)
if materialized is not None:
local_key, managed_file = materialized
manifest[local_key] = managed_file
return manifest, prepared_by_id, prepared_by_old_path, first_prepared
def _rewrite_managed_attachment_rules(
attachments: dict[str, Any],
prepared_json: dict[str, Any],
*,
prepared_by_id: dict[str, tuple[str, str]],
prepared_by_old_path: dict[str, list[tuple[str, str]]],
first_prepared: tuple[str, str] | None,
) -> None:
for rule in _iter_rule_dicts(attachments, prepared_json):
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
if selected is None:
continue
base_path_id, local_root_string = selected
rule["base_path_id"] = base_path_id
rule["base_dir"] = local_root_string
if first_prepared is not None:
attachments["base_path"] = first_prepared[1]
def prepare_campaign_snapshot( def prepare_campaign_snapshot(
session: Session, session: Session,
*, *,
@@ -144,6 +411,9 @@ def prepare_campaign_snapshot(
raw_json: dict[str, Any], raw_json: dict[str, Any],
destination: Path, destination: Path,
include_bytes: bool, include_bytes: bool,
include_unlinked_candidates: bool = False,
user_id: str = "",
is_admin: bool = False,
) -> PreparedCampaignSnapshot: ) -> PreparedCampaignSnapshot:
"""Create a temporary file-oriented campaign snapshot for managed attachments. """Create a temporary file-oriented campaign snapshot for managed attachments.
@@ -159,101 +429,35 @@ def prepare_campaign_snapshot(
materialized_root = destination / "managed-attachments" materialized_root = destination / "managed-attachments"
materialized_root.mkdir(parents=True, exist_ok=True) materialized_root.mkdir(parents=True, exist_ok=True)
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {}) prepared_json, attachments, base_paths = _prepared_attachment_config(raw_json)
attachments = prepared_json.get("attachments") shared_assets, candidate_assets, linked_asset_ids = _campaign_snapshot_assets(
if not isinstance(attachments, dict):
attachments = {}
prepared_json["attachments"] = attachments
base_paths = attachments.get("base_paths")
if not isinstance(base_paths, list):
base_paths = []
shared_assets = list_assets_for_user(
session, session,
tenant_id=tenant_id, tenant_id=tenant_id,
user_id="",
campaign_id=campaign_id, campaign_id=campaign_id,
is_admin=True, prepared_json=prepared_json,
include_unlinked_candidates=include_unlinked_candidates,
user_id=user_id,
is_admin=is_admin,
) )
assets_by_owner = _assets_grouped_by_owner(candidate_assets)
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list) version_blobs = current_versions_and_blobs(session, candidate_assets)
for asset in shared_assets:
owner_id = _asset_owner_id(asset)
if owner_id:
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
version_blobs = current_versions_and_blobs(session, shared_assets)
backend = get_storage_backend() if include_bytes else None backend = get_storage_backend() if include_bytes else None
manifest, prepared_by_id, prepared_by_old_path, first_prepared = _prepare_managed_base_paths(
manifest: dict[str, ManagedAttachmentFile] = {} base_paths,
prepared_by_id: dict[str, tuple[str, str]] = {} materialized_root=materialized_root,
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {} assets_by_owner=assets_by_owner,
first_prepared: tuple[str, str] | None = None version_blobs=version_blobs,
backend=backend,
for index, item in enumerate(base_paths): include_bytes=include_bytes,
if not isinstance(item, dict): linked_asset_ids=linked_asset_ids,
continue )
parsed_source = parse_managed_source(item.get("source")) _rewrite_managed_attachment_rules(
if parsed_source is None: attachments,
continue prepared_json,
owner_type, owner_id = parsed_source prepared_by_id=prepared_by_id,
old_path = str(item.get("path") or ".").strip() or "." prepared_by_old_path=prepared_by_old_path,
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path) first_prepared=first_prepared,
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
local_root.mkdir(parents=True, exist_ok=True)
local_root_string = str(local_root.resolve())
prepared = (base_path_id, local_root_string)
prepared_by_id[base_path_id] = prepared
prepared_by_old_path.setdefault(old_path, []).append(prepared)
if first_prepared is None:
first_prepared = prepared
item["path"] = local_root_string
for asset in assets_by_owner.get((owner_type, owner_id), []):
relative_path = _relative_asset_path(asset, logical_root)
if not relative_path:
continue
target = _safe_local_target(local_root, relative_path)
target.parent.mkdir(parents=True, exist_ok=True)
version, blob = version_blobs[asset.id]
if include_bytes:
try:
data = backend.get_bytes(blob.storage_key) if backend else b""
except StorageBackendError as exc:
raise FileStorageError(str(exc)) from exc
target.write_bytes(data)
else:
target.touch()
local_key = str(target.resolve())
manifest[local_key] = ManagedAttachmentFile(
local_path=local_key,
asset_id=asset.id,
version_id=version.id,
blob_id=blob.id,
display_path=asset.display_path,
relative_path=normalize_logical_path(relative_path),
filename=asset.filename,
owner_type=asset.owner_type,
owner_id=owner_id,
checksum_sha256=blob.checksum_sha256,
size_bytes=blob.size_bytes,
content_type=blob.content_type,
source_provenance=source_provenance_from_metadata(asset.metadata_),
source_revision=source_revision_from_metadata(asset.metadata_),
) )
for rule in _iter_rule_dicts(attachments, prepared_json):
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
if selected is None:
continue
base_path_id, local_root_string = selected
rule["base_path_id"] = base_path_id
rule["base_dir"] = local_root_string
if first_prepared is not None:
attachments["base_path"] = first_prepared[1]
snapshot_path = destination / "campaign.json" snapshot_path = destination / "campaign.json"
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8") snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
@@ -262,6 +466,7 @@ def prepare_campaign_snapshot(
raw_json=prepared_json, raw_json=prepared_json,
managed_files_by_local_path=manifest, managed_files_by_local_path=manifest,
shared_assets=shared_assets, shared_assets=shared_assets,
candidate_assets=candidate_assets,
) )
@@ -273,7 +478,10 @@ def prepared_campaign_snapshot(
campaign_id: str, campaign_id: str,
raw_json: dict[str, Any], raw_json: dict[str, Any],
include_bytes: bool, include_bytes: bool,
prefix: str = "multimailer-managed-campaign-", prefix: str = "govoplan-managed-campaign-",
include_unlinked_candidates: bool = False,
user_id: str = "",
is_admin: bool = False,
) -> Iterator[PreparedCampaignSnapshot]: ) -> Iterator[PreparedCampaignSnapshot]:
temp_dir = Path(tempfile.mkdtemp(prefix=prefix)) temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
try: try:
@@ -284,6 +492,9 @@ def prepared_campaign_snapshot(
raw_json=raw_json, raw_json=raw_json,
destination=temp_dir, destination=temp_dir,
include_bytes=include_bytes, include_bytes=include_bytes,
include_unlinked_candidates=include_unlinked_candidates,
user_id=user_id,
is_admin=is_admin,
) )
finally: finally:
shutil.rmtree(temp_dir, ignore_errors=True) shutil.rmtree(temp_dir, ignore_errors=True)
@@ -301,6 +512,53 @@ def managed_match_payloads(
return payloads return payloads
def share_assets_with_campaign(
session: Session,
*,
tenant_id: str,
campaign_id: str,
file_ids: list[str],
user_id: str,
is_admin: bool = False,
) -> list[dict[str, Any]]:
unique_ids = list(dict.fromkeys(file_id for file_id in file_ids if file_id))
if not unique_ids:
return []
assets = [
get_asset_for_user(
session,
tenant_id=tenant_id,
user_id=user_id,
asset_id=file_id,
require_write=True,
is_admin=is_admin,
)
for file_id in unique_ids
]
shares = share_files(
session,
tenant_id=tenant_id,
assets=assets,
target_type="campaign",
target_id=campaign_id,
permission="read",
user_id=user_id,
)
session.flush()
return [
{
"id": share.id,
"file_asset_id": share.file_asset_id,
"target_type": share.target_type,
"target_id": share.target_id,
"permission": share.permission,
"created_at": share.created_at.isoformat() if share.created_at else None,
"revoked_at": share.revoked_at.isoformat() if share.revoked_at else None,
}
for share in shares
]
def public_attachment_summary_payload(value: Any) -> dict[str, Any]: def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
"""Return an attachment summary without temporary materialization paths. """Return an attachment summary without temporary materialization paths.

View File

@@ -1,6 +1,7 @@
from __future__ import annotations from __future__ import annotations
from collections import defaultdict from collections import defaultdict
from dataclasses import dataclass, field
from pathlib import PurePosixPath from pathlib import PurePosixPath
from typing import Any, Iterable from typing import Any, Iterable
@@ -24,6 +25,15 @@ def _candidate_match_keys(raw_match: str) -> set[str]:
AttachmentUseKey = tuple[str, str, str, str] AttachmentUseKey = tuple[str, str, str, str]
@dataclass(slots=True)
class _AttachmentBatchRefs:
managed_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
asset_ids: set[str] = field(default_factory=set)
version_ids: set[str] = field(default_factory=set)
blob_ids: set[str] = field(default_factory=set)
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey: def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
return job_id, file_version_id, filename_used, stage return job_id, file_version_id, filename_used, stage
@@ -134,22 +144,48 @@ def record_campaign_attachment_uses_for_jobs(
if not job_list: if not job_list:
return return
managed_by_job: dict[str, list[dict[str, object]]] = defaultdict(list) refs = _collect_attachment_batch_refs(job_list)
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
job_by_id = {job.id: job for job in job_list} job_by_id = {job.id: job for job in job_list}
asset_ids: set[str] = set() known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
version_ids: set[str] = set() assets_by_id, versions_by_id, blobs_by_id = _load_managed_attachment_entities(session, refs)
blob_ids: set[str] = set()
for job in job_list: _record_managed_attachment_uses(
session,
job_by_id=job_by_id,
managed_by_job=refs.managed_by_job,
assets_by_id=assets_by_id,
versions_by_id=versions_by_id,
blobs_by_id=blobs_by_id,
stage=stage,
known_keys=known_keys,
)
_record_fallback_attachment_uses(
session,
job_by_id=job_by_id,
fallback_attachments_by_job=refs.fallback_attachments_by_job,
stage=stage,
known_keys=known_keys,
)
def _collect_attachment_batch_refs(jobs: list[CampaignJobLike]) -> _AttachmentBatchRefs:
refs = _AttachmentBatchRefs()
for job in jobs:
attachments = job.resolved_attachments or [] attachments = job.resolved_attachments or []
if not isinstance(attachments, list): if not isinstance(attachments, list):
continue continue
for attachment in attachments: for attachment in attachments:
if not isinstance(attachment, dict): if not isinstance(attachment, dict):
continue continue
if not _collect_managed_attachment_refs(refs, job.id, attachment):
refs.fallback_attachments_by_job[job.id].append(attachment)
return refs
def _collect_managed_attachment_refs(refs: _AttachmentBatchRefs, job_id: str, attachment: dict[str, object]) -> bool:
managed_matches = attachment.get("managed_matches") managed_matches = attachment.get("managed_matches")
if isinstance(managed_matches, list) and managed_matches: if not isinstance(managed_matches, list) or not managed_matches:
return False
for item in managed_matches: for item in managed_matches:
if not isinstance(item, dict): if not isinstance(item, dict):
continue continue
@@ -158,38 +194,41 @@ def record_campaign_attachment_uses_for_jobs(
blob_id = str(item.get("blob_id") or "") blob_id = str(item.get("blob_id") or "")
if not asset_id or not version_id or not blob_id: if not asset_id or not version_id or not blob_id:
continue continue
managed_by_job[job.id].append(item) refs.managed_by_job[job_id].append(item)
asset_ids.add(asset_id) refs.asset_ids.add(asset_id)
version_ids.add(version_id) refs.version_ids.add(version_id)
blob_ids.add(blob_id) refs.blob_ids.add(blob_id)
else: return True
fallback_attachments_by_job[job.id].append(attachment)
assets_by_id = {
item.id: item
for item in session.query(FileAsset).filter(FileAsset.id.in_(asset_ids)).all()
} if asset_ids else {}
versions_by_id = {
item.id: item
for item in session.query(FileVersion).filter(FileVersion.id.in_(version_ids)).all()
} if version_ids else {}
blobs_by_id = {
item.id: item
for item in session.query(FileBlob).filter(FileBlob.id.in_(blob_ids)).all()
} if blob_ids else {}
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
def _load_managed_attachment_entities(
session: Session,
refs: _AttachmentBatchRefs,
) -> tuple[dict[str, FileAsset], dict[str, FileVersion], dict[str, FileBlob]]:
assets_by_id = {item.id: item for item in session.query(FileAsset).filter(FileAsset.id.in_(refs.asset_ids)).all()} if refs.asset_ids else {}
versions_by_id = {item.id: item for item in session.query(FileVersion).filter(FileVersion.id.in_(refs.version_ids)).all()} if refs.version_ids else {}
blobs_by_id = {item.id: item for item in session.query(FileBlob).filter(FileBlob.id.in_(refs.blob_ids)).all()} if refs.blob_ids else {}
return assets_by_id, versions_by_id, blobs_by_id
def _record_managed_attachment_uses(
session: Session,
*,
job_by_id: dict[str, CampaignJobLike],
managed_by_job: dict[str, list[dict[str, object]]],
assets_by_id: dict[str, FileAsset],
versions_by_id: dict[str, FileVersion],
blobs_by_id: dict[str, FileBlob],
stage: str,
known_keys: set[AttachmentUseKey],
) -> None:
for job_id, items in managed_by_job.items(): for job_id, items in managed_by_job.items():
job = job_by_id[job_id] job = job_by_id[job_id]
for item in items: for item in items:
asset = assets_by_id.get(str(item.get("asset_id") or "")) asset = assets_by_id.get(str(item.get("asset_id") or ""))
version = versions_by_id.get(str(item.get("version_id") or "")) version = versions_by_id.get(str(item.get("version_id") or ""))
blob = blobs_by_id.get(str(item.get("blob_id") or "")) blob = blobs_by_id.get(str(item.get("blob_id") or ""))
if not asset or not version or not blob: if not _managed_attachment_entities_match_job(job, asset, version, blob):
continue
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
continue
if version.file_asset_id != asset.id or version.blob_id != blob.id:
continue continue
_add_use( _add_use(
session, session,
@@ -202,29 +241,79 @@ def record_campaign_attachment_uses_for_jobs(
known_keys=known_keys, known_keys=known_keys,
) )
def _managed_attachment_entities_match_job(
job: CampaignJobLike,
asset: FileAsset | None,
version: FileVersion | None,
blob: FileBlob | None,
) -> bool:
if not asset or not version or not blob:
return False
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
return False
return version.file_asset_id == asset.id and version.blob_id == blob.id
def _record_fallback_attachment_uses(
session: Session,
*,
job_by_id: dict[str, CampaignJobLike],
fallback_attachments_by_job: dict[str, list[dict[str, object]]],
stage: str,
known_keys: set[AttachmentUseKey],
) -> None:
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {} assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {} version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {}
for job_id, attachments in fallback_attachments_by_job.items(): for job_id, attachments in fallback_attachments_by_job.items():
job = job_by_id[job_id] job = job_by_id[job_id]
campaign_key = (job.tenant_id, job.campaign_id) campaign_key = (job.tenant_id, job.campaign_id)
by_key, version_blobs = _fallback_campaign_assets(
session,
job,
campaign_key=campaign_key,
assets_by_campaign=assets_by_campaign,
version_blobs_by_campaign=version_blobs_by_campaign,
)
for attachment in attachments:
_record_fallback_attachment(session, job, attachment, by_key=by_key, version_blobs=version_blobs, stage=stage, known_keys=known_keys)
def _fallback_campaign_assets(
session: Session,
job: CampaignJobLike,
*,
campaign_key: tuple[str, str],
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]],
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]],
) -> tuple[dict[str, FileAsset], dict[str, tuple[FileVersion, FileBlob]]]:
by_key = assets_by_campaign.get(campaign_key) by_key = assets_by_campaign.get(campaign_key)
if by_key is None: if by_key is None:
assets = list_assets_for_user( assets = list_assets_for_user(session, tenant_id=job.tenant_id, user_id="", campaign_id=job.campaign_id, is_admin=True)
session, by_key = _asset_lookup_by_path_and_filename(assets)
tenant_id=job.tenant_id, assets_by_campaign[campaign_key] = by_key
user_id="", version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
campaign_id=job.campaign_id, return by_key, version_blobs_by_campaign[campaign_key]
is_admin=True,
)
by_key = {} def _asset_lookup_by_path_and_filename(assets: list[FileAsset]) -> dict[str, FileAsset]:
by_key: dict[str, FileAsset] = {}
for asset in assets: for asset in assets:
by_key[asset.display_path.strip("/")] = asset by_key[asset.display_path.strip("/")] = asset
by_key.setdefault(asset.filename, asset) by_key.setdefault(asset.filename, asset)
assets_by_campaign[campaign_key] = by_key return by_key
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
version_blobs = version_blobs_by_campaign[campaign_key]
for attachment in attachments:
def _record_fallback_attachment(
session: Session,
job: CampaignJobLike,
attachment: dict[str, object],
*,
by_key: dict[str, FileAsset],
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
stage: str,
known_keys: set[AttachmentUseKey],
) -> None:
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else [] matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
for raw in matches: for raw in matches:
if not isinstance(raw, str): if not isinstance(raw, str):

View File

@@ -1,6 +1,6 @@
from __future__ import annotations from __future__ import annotations
import os import json
from collections.abc import Mapping from collections.abc import Mapping
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import datetime, timezone from datetime import datetime, timezone
@@ -9,11 +9,23 @@ from importlib import import_module
import mimetypes import mimetypes
from typing import Any from typing import Any
from urllib.parse import quote, unquote, urljoin, urlsplit from urllib.parse import quote, unquote, urljoin, urlsplit
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET # nosec B405 - typing/element creation only; parsing uses defusedxml below.
from defusedxml import ElementTree as SafeElementTree from defusedxml import ElementTree as SafeElementTree
import httpx
from govoplan_core.security.outbound_http import (
OutboundHttpError,
validate_unpinned_sdk_host,
validate_unpinned_sdk_http_url,
)
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
from govoplan_files.backend.storage.connector_deployment import (
ConnectorDeploymentConfigurationError,
connector_ca_bundle_path,
connector_secret_env_value,
validate_connector_tls_metadata,
)
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
@@ -53,7 +65,7 @@ class ConnectorBrowseItem:
} }
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None) -> list[ConnectorBrowseItem]: def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None, continuation_token: str | None = None) -> list[ConnectorBrowseItem]:
browse_path = normalize_connector_browse_path(path) browse_path = normalize_connector_browse_path(path)
static_items = _static_listing(profile, path=browse_path, library_id=library_id) static_items = _static_listing(profile, path=browse_path, library_id=library_id)
if static_items is not None: if static_items is not None:
@@ -66,6 +78,8 @@ def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = No
return _browse_webdav(profile, path=browse_path) return _browse_webdav(profile, path=browse_path)
if profile.provider == "smb": if profile.provider == "smb":
return _browse_smb(profile, path=browse_path) return _browse_smb(profile, path=browse_path)
if profile.provider == "s3":
return _browse_s3(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet") raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet")
@@ -168,14 +182,22 @@ def _browse_webdav(profile: ConnectorProfile, *, path: str) -> list[ConnectorBro
</prop> </prop>
</propfind>""" </propfind>"""
try: try:
response = httpx.request("PROPFIND", url, headers=headers, content=body, auth=auth, timeout=15.0) response = request_connector_bytes(
except httpx.HTTPError as exc: "PROPFIND",
url,
headers=headers,
content=body,
auth=auth,
timeout=15.0,
label="WebDAV browse",
)
except ConnectorHttpError as exc:
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
if response.status_code in {401, 403}: if response.status_code in {401, 403}:
raise ConnectorBrowseError("Connector credentials were rejected") raise ConnectorBrowseError("Connector credentials were rejected")
if response.status_code not in {200, 207}: if response.status_code not in {200, 207}:
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}") raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.text) return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.content)
def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]: def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
@@ -222,6 +244,245 @@ def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowse
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold())) return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
def _browse_s3(profile: ConnectorProfile, *, path: str, library_id: str | None, continuation_token: str | None) -> list[ConnectorBrowseItem]:
client = _s3_client(profile)
bucket = _s3_bucket(profile, library_id)
if not bucket:
try:
payload = client.list_buckets()
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
buckets = payload.get("Buckets") if isinstance(payload, Mapping) else None
if not isinstance(buckets, list):
raise ConnectorBrowseError("S3 connector returned invalid bucket list")
return sorted(
(
ConnectorBrowseItem(
kind="library",
name=_clean(item.get("Name")) or "",
path=_clean(item.get("Name")) or "",
external_id=_clean(item.get("Name")),
modified_at=_timestamp(item.get("CreationDate")),
metadata={"bucket": _clean(item.get("Name"))},
)
for item in buckets
if isinstance(item, Mapping) and _clean(item.get("Name"))
),
key=lambda item: item.name.casefold(),
)
prefix = _s3_object_key(profile, path, directory=True)
params: dict[str, object] = {
"Bucket": bucket,
"Prefix": prefix,
"Delimiter": "/",
"MaxKeys": _s3_max_keys(profile),
}
next_page_request = _clean(continuation_token) or _metadata_string(profile, "continuation_token")
if next_page_request:
params["ContinuationToken"] = next_page_request
try:
payload = client.list_objects_v2(**params)
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
if not isinstance(payload, Mapping):
raise ConnectorBrowseError("S3 connector returned invalid object listing")
items = [
*_s3_prefix_items(bucket=bucket, browse_path=path, base_prefix=prefix, prefixes=payload.get("CommonPrefixes")),
*_s3_object_items(bucket=bucket, browse_path=path, base_prefix=prefix, objects=payload.get("Contents")),
]
next_token = _clean(payload.get("NextContinuationToken"))
if next_token and items:
last = items[-1]
items[-1] = ConnectorBrowseItem(
kind=last.kind,
name=last.name,
path=last.path,
external_id=last.external_id,
external_url=last.external_url,
size_bytes=last.size_bytes,
content_type=last.content_type,
modified_at=last.modified_at,
etag=last.etag,
metadata={**dict(last.metadata), "next_continuation_token": next_token, "listing_truncated": True},
)
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
def _s3_client(profile: ConnectorProfile) -> Any:
if profile.secret_ref:
raise ConnectorBrowseError("Secret-ref S3 credentials need a runtime secret resolver before live browsing")
if profile.endpoint_url:
try:
endpoint_url = validate_unpinned_sdk_http_url(
profile.endpoint_url,
label="S3 connector endpoint",
)
except OutboundHttpError as exc:
raise ConnectorBrowseError(str(exc)) from exc
else:
raise ConnectorBrowseError(
"S3 connector endpoint discovery uses an SDK transport that cannot guarantee connection-time DNS/IP "
"pinning; live S3 access is disabled until that transport supports pinning"
)
try:
boto3 = import_module("boto3")
config_module = import_module("botocore.config")
except ImportError as exc:
raise ConnectorBrowseUnsupported("S3 connector browsing requires the optional boto3 dependency") from exc
kwargs: dict[str, object] = {"endpoint_url": endpoint_url}
region = _metadata_string(profile, "region") or _metadata_string(profile, "aws_region")
if region:
kwargs["region_name"] = region
access_key = profile.username or _metadata_env(profile, "access_key_id_env") or _metadata_string(profile, "access_key_id")
secret_key = _profile_password(profile) or _metadata_env(profile, "secret_access_key_env")
session_token = _profile_token(profile) or _metadata_env(profile, "session_token_env")
if access_key:
kwargs["aws_access_key_id"] = access_key
if secret_key:
kwargs["aws_secret_access_key"] = secret_key
if session_token:
kwargs["aws_session_token"] = session_token
verify = _s3_verify(profile)
if verify is not None:
kwargs["verify"] = verify
addressing_style = _s3_addressing_style(profile)
if addressing_style:
kwargs["config"] = config_module.Config(s3={"addressing_style": addressing_style})
try:
return boto3.client("s3", **kwargs)
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorBrowseError(f"S3 connector could not be initialized: {exc}") from exc
def _s3_bucket(profile: ConnectorProfile, library_id: str | None = None) -> str | None:
return _metadata_string(profile, "bucket") or _metadata_string(profile, "bucket_name") or _clean(library_id)
def _s3_object_key(profile: ConnectorProfile, path: str, *, directory: bool = False) -> str:
base_prefix = normalize_connector_browse_path(profile.base_path or _metadata_string(profile, "base_prefix"))
browse_path = normalize_connector_browse_path(path)
key = "/".join(part for part in (base_prefix, browse_path) if part)
if directory and key:
return key.rstrip("/") + "/"
return key
def _s3_prefix_items(
*,
bucket: str,
browse_path: str,
base_prefix: str,
prefixes: object,
) -> list[ConnectorBrowseItem]:
if not isinstance(prefixes, list):
return []
items: list[ConnectorBrowseItem] = []
for item in prefixes:
if not isinstance(item, Mapping):
continue
key = _clean(item.get("Prefix"))
if not key:
continue
relative = _s3_relative_key(key, base_prefix=base_prefix)
name = _path_name(relative)
if not name:
continue
path = _join_browse_path(browse_path, name)
items.append(
ConnectorBrowseItem(
kind="folder",
name=name,
path=path,
external_id=f"{bucket}:{key}",
metadata={"bucket": bucket, "key": key},
)
)
return items
def _s3_object_items(
*,
bucket: str,
browse_path: str,
base_prefix: str,
objects: object,
) -> list[ConnectorBrowseItem]:
if not isinstance(objects, list):
return []
items: list[ConnectorBrowseItem] = []
for item in objects:
if not isinstance(item, Mapping):
continue
key = _clean(item.get("Key"))
if not key or key == base_prefix:
continue
relative = _s3_relative_key(key, base_prefix=base_prefix)
name = _path_name(relative)
if not name:
continue
path = _join_browse_path(browse_path, name)
items.append(
ConnectorBrowseItem(
kind="file",
name=name,
path=path,
external_id=f"{bucket}:{key}",
size_bytes=_int(item.get("Size")),
content_type=mimetypes.guess_type(name)[0],
modified_at=_timestamp(item.get("LastModified")),
etag=_clean(item.get("ETag")),
metadata={
"bucket": bucket,
"key": key,
**({"storage_class": item["StorageClass"]} if "StorageClass" in item else {}),
},
)
)
return items
def _s3_relative_key(key: str, *, base_prefix: str) -> str:
clean_key = key.rstrip("/")
clean_base = base_prefix.rstrip("/")
if clean_base and clean_key.startswith(f"{clean_base}/"):
return clean_key[len(clean_base) + 1 :]
return clean_key
def _s3_max_keys(profile: ConnectorProfile) -> int:
configured = _int(profile.metadata.get("max_keys"))
if configured is None:
return 1000
return max(1, min(configured, 1000))
def _s3_verify(profile: ConnectorProfile) -> bool | str | None:
try:
validate_connector_tls_metadata(profile.metadata)
except ConnectorDeploymentConfigurationError as exc:
raise ConnectorBrowseError(str(exc)) from exc
ca_bundle = _metadata_string(profile, "ca_bundle")
if ca_bundle:
try:
return connector_ca_bundle_path(ca_bundle)
except ConnectorDeploymentConfigurationError as exc:
raise ConnectorBrowseError(str(exc)) from exc
if "verify_tls" in profile.metadata:
return _metadata_bool(profile, "verify_tls", default=True)
if "tls_verify" in profile.metadata:
return _metadata_bool(profile, "tls_verify", default=True)
return None
def _s3_addressing_style(profile: ConnectorProfile) -> str | None:
style = _metadata_string(profile, "addressing_style")
if style in {"path", "virtual", "auto"}:
return style
if _metadata_bool(profile, "path_style", default=False):
return "path"
return None
def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]: def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]:
token = _seafile_token(profile) token = _seafile_token(profile)
return {"Authorization": f"Token {token}", "Accept": "application/json"} return {"Authorization": f"Token {token}", "Accept": "application/json"}
@@ -261,16 +522,24 @@ def _request_json(
data: Mapping[str, str] | None = None, data: Mapping[str, str] | None = None,
) -> object: ) -> object:
try: try:
response = httpx.request(method, url, headers=dict(headers or {}), params=params, data=data, timeout=15.0) response = request_connector_bytes(
except httpx.HTTPError as exc: method,
url,
headers=dict(headers or {}),
params=params,
data=data,
timeout=15.0,
label="Seafile API",
)
except ConnectorHttpError as exc:
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
if response.status_code in {401, 403}: if response.status_code in {401, 403}:
raise ConnectorBrowseError("Connector credentials were rejected") raise ConnectorBrowseError("Connector credentials were rejected")
if response.status_code not in {200, 201}: if response.status_code not in {200, 201}:
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}") raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
try: try:
return response.json() return json.loads(response.content)
except ValueError as exc: except (UnicodeDecodeError, ValueError) as exc:
raise ConnectorBrowseError("Connector returned invalid JSON") from exc raise ConnectorBrowseError("Connector returned invalid JSON") from exc
@@ -444,11 +713,18 @@ def _metadata_bool(profile: ConnectorProfile, key: str, *, default: bool = False
return str(value).strip().casefold() in {"1", "true", "yes", "on"} return str(value).strip().casefold() in {"1", "true", "yes", "on"}
def _env_required(name: str, profile_id: str) -> str: def _metadata_env(profile: ConnectorProfile, key: str) -> str | None:
value = _clean(os.environ.get(name)) env_name = _metadata_string(profile, key)
if not value: if not env_name:
raise ConnectorBrowseError(f"Connector profile {profile_id} is missing required credential environment") return None
return value return _env_required(env_name, profile)
def _env_required(name: str, profile: ConnectorProfile) -> str:
try:
return connector_secret_env_value(name, source_kind=profile.source_kind)
except ConnectorDeploymentConfigurationError as exc:
raise ConnectorBrowseError(f"Connector profile {profile.id}: {exc}") from exc
def normalize_connector_browse_path(value: object) -> str: def normalize_connector_browse_path(value: object) -> str:
@@ -505,6 +781,11 @@ def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
server = _clean(parsed.hostname) server = _clean(parsed.hostname)
if not server: if not server:
raise ConnectorBrowseError("SMB connector endpoint_url must include a server") raise ConnectorBrowseError("SMB connector endpoint_url must include a server")
port = parsed.port or _int(profile.metadata.get("port")) or 445
try:
validate_unpinned_sdk_host(server, port=port, label="SMB connector endpoint")
except OutboundHttpError as exc:
raise ConnectorBrowseError(str(exc)) from exc
path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part] path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part]
if not path_parts: if not path_parts:
raise ConnectorBrowseError("SMB connector endpoint_url must include a share name") raise ConnectorBrowseError("SMB connector endpoint_url must include a share name")
@@ -514,7 +795,7 @@ def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
return _SmbLocation( return _SmbLocation(
server=server, server=server,
share=path_parts[0], share=path_parts[0],
port=parsed.port or _int(profile.metadata.get("port")) or 445, port=port,
root_path=normalize_connector_browse_path("/".join(root_parts)), root_path=normalize_connector_browse_path("/".join(root_parts)),
) )
@@ -550,7 +831,7 @@ def _profile_password(profile: ConnectorProfile) -> str | None:
if profile.password_value: if profile.password_value:
return profile.password_value return profile.password_value
if profile.password_env: if profile.password_env:
return _env_required(profile.password_env, profile.id) return _env_required(profile.password_env, profile)
return None return None
@@ -558,7 +839,7 @@ def _profile_token(profile: ConnectorProfile) -> str | None:
if profile.token_value: if profile.token_value:
return profile.token_value return profile.token_value
if profile.token_env: if profile.token_env:
return _env_required(profile.token_env, profile.id) return _env_required(profile.token_env, profile)
return None return None

View File

@@ -0,0 +1,337 @@
from __future__ import annotations
from dataclasses import dataclass
from sqlalchemy import inspect
from sqlalchemy.orm import Session
from govoplan_core.audit.logging import audit_event
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
@dataclass(frozen=True, slots=True)
class ConnectorCredentialDeletionResult:
changed: bool
affected_profiles: tuple[FileConnectorProfile, ...] = ()
def delete_connector_credential_row(
session: Session,
row: FileConnectorCredential,
*,
deletion_reason: str,
user_id: str | None = None,
api_key_id: str | None = None,
) -> ConnectorCredentialDeletionResult:
"""Scrub a credential tombstone and disable every profile that used it.
``secret_ref`` predates a Files-owned secret-provider contract. It may be
shared or deployment-owned, so it is detached locally and explicitly
audited without ever being passed to a provider delete operation.
"""
dependent_profiles = (
tuple(
session.query(FileConnectorProfile)
.filter(FileConnectorProfile.credential_profile_id == row.id)
.order_by(FileConnectorProfile.id.asc())
.all()
)
if inspect(session.get_bind()).has_table(FileConnectorProfile.__tablename__)
else ()
)
affected_profiles: list[FileConnectorProfile] = []
for profile in dependent_profiles:
if _delete_connector_profile_row(
session,
profile,
deletion_reason="credential_deleted",
user_id=user_id,
api_key_id=api_key_id,
):
affected_profiles.append(profile)
deleted_secret_kinds = _encrypted_secret_kinds(row)
removed_reference_kinds = _credential_reference_kinds(row)
removed_metadata = bool(row.metadata_)
changed = _credential_row_requires_deletion(row)
if changed:
_scrub_credential_row(row, user_id=user_id)
session.add(row)
_audit_credential_deletion(
session,
row,
deletion_reason=deletion_reason,
deleted_secret_kinds=deleted_secret_kinds,
removed_reference_kinds=removed_reference_kinds,
removed_metadata=removed_metadata,
affected_profile_count=len(affected_profiles),
user_id=user_id,
api_key_id=api_key_id,
)
session.flush()
return ConnectorCredentialDeletionResult(
changed=changed or bool(affected_profiles),
affected_profiles=tuple(affected_profiles),
)
def delete_connector_profile_row(
session: Session,
row: FileConnectorProfile,
*,
deletion_reason: str,
user_id: str | None = None,
api_key_id: str | None = None,
) -> bool:
changed = _delete_connector_profile_row(
session,
row,
deletion_reason=deletion_reason,
user_id=user_id,
api_key_id=api_key_id,
)
session.flush()
return changed
def delete_connector_credentials_for_retirement(session: Session) -> int:
"""Scrub and audit stored connector material before destructive retirement.
Legacy external references are detached and audited as non-owned. Files
never sends those arbitrary references to a provider delete operation.
"""
inspector = inspect(session.get_bind())
profiles = (
session.query(FileConnectorProfile).order_by(FileConnectorProfile.id.asc()).all()
if inspector.has_table(FileConnectorProfile.__tablename__)
else []
)
credentials = (
session.query(FileConnectorCredential).order_by(FileConnectorCredential.id.asc()).all()
if inspector.has_table(FileConnectorCredential.__tablename__)
else []
)
deleted = 0
for profile in profiles:
if not _profile_row_has_credential_material(profile):
continue
if _delete_connector_profile_row(
session,
profile,
deletion_reason="module_data_retired",
):
deleted += 1
for credential in credentials:
if not _credential_row_has_material(credential):
continue
result = delete_connector_credential_row(
session,
credential,
deletion_reason="module_data_retired",
)
if result.changed:
deleted += 1
session.flush()
return deleted
def _delete_connector_profile_row(
session: Session,
row: FileConnectorProfile,
*,
deletion_reason: str,
user_id: str | None = None,
api_key_id: str | None = None,
) -> bool:
deleted_secret_kinds = _encrypted_secret_kinds(row)
removed_reference_kinds = _profile_reference_kinds(row)
removed_metadata = bool(row.metadata_)
changed = _profile_row_requires_deletion(row)
if not changed:
return False
_scrub_profile_row(row, user_id=user_id)
session.add(row)
_audit_profile_deletion(
session,
row,
deletion_reason=deletion_reason,
deleted_secret_kinds=deleted_secret_kinds,
removed_reference_kinds=removed_reference_kinds,
removed_metadata=removed_metadata,
user_id=user_id,
api_key_id=api_key_id,
)
return True
def _scrub_credential_row(row: FileConnectorCredential, *, user_id: str | None) -> None:
row.enabled = False
row.credential_mode = "none"
row.username = None
row.password_encrypted = None
row.token_encrypted = None
row.password_env = None
row.token_env = None
row.secret_ref = None
row.metadata_ = {}
row.updated_by_user_id = user_id
def _scrub_profile_row(row: FileConnectorProfile, *, user_id: str | None) -> None:
row.enabled = False
row.credential_profile_id = None
row.credential_mode = "none"
row.username = None
row.password_encrypted = None
row.token_encrypted = None
row.password_env = None
row.token_env = None
row.secret_ref = None
row.metadata_ = {}
row.updated_by_user_id = user_id
def _credential_row_requires_deletion(row: FileConnectorCredential) -> bool:
return bool(row.enabled or _credential_row_has_material(row))
def _profile_row_requires_deletion(row: FileConnectorProfile) -> bool:
return bool(row.enabled or _profile_row_has_credential_material(row))
def _credential_row_has_material(row: FileConnectorCredential) -> bool:
return bool(
row.username
or row.password_encrypted
or row.token_encrypted
or row.password_env
or row.token_env
or row.secret_ref
or row.metadata_
or row.credential_mode not in {"", "none", "anonymous"}
)
def _profile_row_has_credential_material(row: FileConnectorProfile) -> bool:
return bool(
row.credential_profile_id
or row.username
or row.password_encrypted
or row.token_encrypted
or row.password_env
or row.token_env
or row.secret_ref
or row.metadata_
or row.credential_mode not in {"", "none", "anonymous"}
)
def _encrypted_secret_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
kinds: list[str] = []
if row.password_encrypted:
kinds.append("password")
if row.token_encrypted:
kinds.append("token")
return kinds
def _credential_reference_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
kinds: list[str] = []
if row.password_env:
kinds.append("password_env")
if row.token_env:
kinds.append("token_env")
if row.secret_ref:
kinds.append("unowned_external_secret_ref")
return kinds
def _profile_reference_kinds(row: FileConnectorProfile) -> list[str]:
kinds = _credential_reference_kinds(row)
if row.credential_profile_id:
kinds.append("credential_profile")
return kinds
def _storage_backend(deleted_secret_kinds: list[str], removed_reference_kinds: list[str]) -> str:
if "unowned_external_secret_ref" in removed_reference_kinds:
return "unowned_external_reference_detached"
if deleted_secret_kinds:
return "encrypted_database"
if removed_reference_kinds:
return "reference_only"
return "none"
def _audit_credential_deletion(
session: Session,
row: FileConnectorCredential,
*,
deletion_reason: str,
deleted_secret_kinds: list[str],
removed_reference_kinds: list[str],
removed_metadata: bool,
affected_profile_count: int,
user_id: str | None,
api_key_id: str | None,
) -> None:
audit_event(
session,
tenant_id=row.tenant_id,
scope=_audit_scope(row.scope_type),
user_id=user_id,
api_key_id=api_key_id,
action="files.connector_credential_deleted",
object_type="file_connector_credential",
object_id=row.id,
details={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
"deleted_secret_kinds": deleted_secret_kinds,
"removed_reference_kinds": removed_reference_kinds,
"removed_metadata": removed_metadata,
"affected_profile_count": affected_profile_count,
"deletion_reason": deletion_reason,
},
)
def _audit_profile_deletion(
session: Session,
row: FileConnectorProfile,
*,
deletion_reason: str,
deleted_secret_kinds: list[str],
removed_reference_kinds: list[str],
removed_metadata: bool,
user_id: str | None,
api_key_id: str | None,
) -> None:
audit_event(
session,
tenant_id=row.tenant_id,
scope=_audit_scope(row.scope_type),
user_id=user_id,
api_key_id=api_key_id,
action="files.connector_profile_deleted",
object_type="file_connector_profile",
object_id=row.id,
details={
"scope_type": row.scope_type,
"scope_id": row.scope_id,
"provider": row.provider,
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
"deleted_secret_kinds": deleted_secret_kinds,
"removed_reference_kinds": removed_reference_kinds,
"removed_metadata": removed_metadata,
"deletion_reason": deletion_reason,
},
)
def _audit_scope(scope_type: str) -> str:
return "system" if scope_type == "system" else "tenant"

View File

@@ -10,6 +10,7 @@ from govoplan_core.core.policy import normalize_policy_scope_type, policy_source
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential from govoplan_files.backend.db.models import FileConnectorCredential
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
@@ -51,9 +52,13 @@ class ConnectorCredential:
@property @property
def credentials_configured(self) -> bool: def credentials_configured(self) -> bool:
if not self.enabled:
return False
if self.credential_mode.casefold() in {"", "none", "anonymous"}: if self.credential_mode.casefold() in {"", "none", "anonymous"}:
return True return True
return bool(self.secret_ref or self.password_value or self.token_value or self.password_env or self.token_env) # Environment references are deliberately unavailable to API-managed
# credential rows. Legacy rows remain visible but fail closed.
return bool(self.secret_ref or self.password_value or self.token_value)
def to_response(self) -> dict[str, Any]: def to_response(self) -> dict[str, Any]:
return { return {
@@ -182,6 +187,12 @@ def create_connector_credential_row(
policy: Mapping[str, Any] | None = None, policy: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None, metadata: Mapping[str, Any] | None = None,
) -> FileConnectorCredential: ) -> FileConnectorCredential:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
metadata=metadata,
)
clean_id = _normalize_id(credential_id) clean_id = _normalize_id(credential_id)
if session.get(FileConnectorCredential, clean_id) is not None: if session.get(FileConnectorCredential, clean_id) is not None:
raise FileStorageError(f"Connector credential already exists: {clean_id}") raise FileStorageError(f"Connector credential already exists: {clean_id}")
@@ -231,6 +242,18 @@ def update_connector_credential_row(
clear_password: bool = False, clear_password: bool = False,
clear_token: bool = False, clear_token: bool = False,
) -> FileConnectorCredential: ) -> FileConnectorCredential:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
metadata=metadata,
)
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
if _clean(row.secret_ref):
raise FileStorageError(
"An existing external secret reference cannot be replaced or cleared until Files can prove "
"provider ownership and confirm provider-side deletion"
)
if label is not None: if label is not None:
row.label = _normalize_label(label) row.label = _normalize_label(label)
if provider is not None: if provider is not None:
@@ -265,14 +288,6 @@ def update_connector_credential_row(
return row return row
def deactivate_connector_credential_row(session: Session, row: FileConnectorCredential, *, user_id: str | None) -> FileConnectorCredential:
row.enabled = False
row.updated_by_user_id = user_id
session.add(row)
session.flush()
return row
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]: def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
clean_scope_type = normalize_policy_scope_type(scope_type) clean_scope_type = normalize_policy_scope_type(scope_type)
clean_scope_id = _clean(scope_id) clean_scope_id = _clean(scope_id)

View File

@@ -0,0 +1,241 @@
from __future__ import annotations
import os
import re
from collections.abc import Mapping
from pathlib import Path
from typing import Any
_SECRET_ENV_ALLOWLIST = "GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST" # noqa: S105 # nosec B105 - configuration key.
_CA_BUNDLE_ALLOWLIST = "GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST"
_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
_SECRET_ENV_METADATA_KEYS = frozenset(
{
"access_key_id_env",
"secret_access_key_env",
"session_token_env",
}
)
_SECRET_VALUE_METADATA_KEYS = frozenset(
{
"password",
"token",
"access_token",
"auth_token",
"bearer_token",
"refresh_token",
"api_key",
"access_key",
"access_key_id",
"secret_key",
"secret_access_key",
"session_token",
}
)
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
class ConnectorDeploymentConfigurationError(ValueError):
"""Raised when connector data crosses a deployment-owned trust boundary."""
def connector_secret_env_value(name: str, *, source_kind: str) -> str:
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
value = os.environ.get(clean_name)
if value is None or value == "":
raise ConnectorDeploymentConfigurationError("Connector credential environment variable is not configured")
return value
def connector_secret_env_available(name: str | None, *, source_kind: str) -> bool:
if not name:
return False
try:
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
except ConnectorDeploymentConfigurationError:
return False
return bool(os.environ.get(clean_name))
def validate_deployment_connector_references(
*,
source_kind: str,
password_env: str | None = None,
token_env: str | None = None,
metadata: Mapping[str, Any] | None = None,
) -> None:
"""Validate references in deployment-owned connector configuration."""
for name in (password_env, token_env):
if name:
_validate_secret_env_reference(name, source_kind=source_kind)
for key in _SECRET_ENV_METADATA_KEYS:
name = _clean((metadata or {}).get(key))
if name:
_validate_secret_env_reference(name, source_kind=source_kind)
validate_connector_tls_metadata(metadata)
def reject_api_controlled_deployment_references(
*,
password_env: str | None = None,
token_env: str | None = None,
secret_ref: str | None = None,
metadata: Mapping[str, Any] | None = None,
) -> None:
"""Reject process-secret selectors controlled through tenant-facing APIs."""
if _clean(password_env) or _clean(token_env):
raise ConnectorDeploymentConfigurationError(
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
)
if _clean(secret_ref):
raise ConnectorDeploymentConfigurationError(
"External secret references cannot be managed through the Files API until Files can prove "
"provider ownership and confirm provider-side deletion"
)
for key, value in _metadata_entries(metadata or {}):
clean_key = str(key).strip().casefold()
if _clean(value) and (clean_key in _SECRET_ENV_METADATA_KEYS or clean_key.endswith("_env")):
raise ConnectorDeploymentConfigurationError(
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
)
if _clean(value) and (
clean_key in _SECRET_VALUE_METADATA_KEYS
or clean_key.endswith(("_password", "_secret", "_api_key"))
):
raise ConnectorDeploymentConfigurationError(
"Connector credential values must use the dedicated encrypted credential fields, not metadata"
)
validate_connector_tls_metadata(metadata)
def _metadata_entries(value: object) -> list[tuple[str, object]]:
entries: list[tuple[str, object]] = []
if isinstance(value, Mapping):
for key, item in value.items():
entries.append((str(key), item))
entries.extend(_metadata_entries(item))
elif isinstance(value, (list, tuple)):
for item in value:
entries.extend(_metadata_entries(item))
return entries
def connector_ca_bundle_path(value: str) -> str:
clean_value = _clean(value)
if not clean_value:
raise ConnectorDeploymentConfigurationError("Connector CA bundle path is empty")
candidate = Path(clean_value)
if not candidate.is_absolute():
raise ConnectorDeploymentConfigurationError("Connector CA bundle paths must be absolute")
try:
resolved = candidate.resolve(strict=True)
except OSError as exc:
raise ConnectorDeploymentConfigurationError("Connector CA bundle path does not exist") from exc
allowed = _allowed_ca_bundle_paths()
if resolved not in allowed:
raise ConnectorDeploymentConfigurationError(
f"Connector CA bundle path is not listed in {_CA_BUNDLE_ALLOWLIST}"
)
if not resolved.is_file():
raise ConnectorDeploymentConfigurationError("Connector CA bundle path must be a regular file")
return str(resolved)
def validate_connector_tls_metadata(metadata: Mapping[str, Any] | None) -> None:
values = metadata or {}
ca_bundle = _clean(values.get("ca_bundle"))
if ca_bundle:
connector_ca_bundle_path(ca_bundle)
for key in ("verify_tls", "tls_verify"):
if key in values and not _as_bool(values.get(key), default=True) and not _development_runtime():
raise ConnectorDeploymentConfigurationError(
"Connector TLS certificate verification may only be disabled in dev/test environments"
)
def connector_effective_endpoint_url(
*,
provider: str | None,
endpoint_url: str | None,
metadata: Mapping[str, Any] | None,
) -> str | None:
"""Return the endpoint that connector I/O will actually use."""
clean_provider = (provider or "").strip().casefold()
values = metadata or {}
webdav_url = _clean(values.get("webdav_endpoint_url"))
browse_protocol = (_clean(values.get("browse_protocol")) or "").casefold()
if webdav_url and (clean_provider in {"seafile", "webdav", "nextcloud"} or browse_protocol == "webdav"):
return webdav_url
return _clean(endpoint_url)
def _validate_secret_env_reference(name: str, *, source_kind: str) -> str:
if source_kind.strip().casefold() != "settings":
raise ConnectorDeploymentConfigurationError(
"Environment-backed credentials may only be used by deployment-owned connector configuration"
)
clean_name = name.strip()
if not _ENV_NAME.fullmatch(clean_name):
raise ConnectorDeploymentConfigurationError("Connector credential environment variable name is invalid")
if clean_name not in _allowed_secret_env_names():
raise ConnectorDeploymentConfigurationError(
f"Connector credential environment variable is not listed in {_SECRET_ENV_ALLOWLIST}"
)
return clean_name
def _allowed_secret_env_names() -> frozenset[str]:
raw = os.environ.get(_SECRET_ENV_ALLOWLIST, "")
names = frozenset(item.strip() for item in raw.split(",") if item.strip())
invalid = sorted(name for name in names if not _ENV_NAME.fullmatch(name))
if invalid:
raise ConnectorDeploymentConfigurationError(f"{_SECRET_ENV_ALLOWLIST} contains an invalid variable name")
return names
def _allowed_ca_bundle_paths() -> frozenset[Path]:
raw = os.environ.get(_CA_BUNDLE_ALLOWLIST, "")
paths: set[Path] = set()
for item in (part.strip() for part in raw.split(",")):
if not item:
continue
path = Path(item)
if not path.is_absolute():
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} requires absolute paths")
try:
paths.add(path.resolve(strict=True))
except OSError as exc:
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} contains a missing path") from exc
return frozenset(paths)
def _development_runtime() -> bool:
app_env = os.environ.get("APP_ENV", "dev").strip().casefold()
install_profile = os.environ.get("GOVOPLAN_INSTALL_PROFILE", "").strip().casefold()
return app_env in _DEVELOPMENT_ENVIRONMENTS and (
not install_profile or install_profile in _DEVELOPMENT_ENVIRONMENTS
)
def _as_bool(value: object, *, default: bool) -> bool:
if value is None:
return default
if isinstance(value, bool):
return value
clean = str(value).strip().casefold()
if clean in {"1", "true", "yes", "on"}:
return True
if clean in {"0", "false", "no", "off"}:
return False
raise ConnectorDeploymentConfigurationError("Connector TLS verification setting must be true or false")
def _clean(value: object) -> str | None:
if value is None:
return None
clean = str(value).strip()
return clean or None

View File

@@ -4,7 +4,7 @@ from dataclasses import dataclass, field
import mimetypes import mimetypes
from typing import Any from typing import Any
import httpx from govoplan_core.security.outbound_http import response_limit
from govoplan_files.backend.storage.connector_browse import ( from govoplan_files.backend.storage.connector_browse import (
ConnectorBrowseError, ConnectorBrowseError,
@@ -21,12 +21,16 @@ from govoplan_files.backend.storage.connector_browse import (
_smb_stat_size, _smb_stat_size,
_smb_unc_path, _smb_unc_path,
_smbclient_module, _smbclient_module,
_s3_bucket,
_s3_client,
_s3_object_key,
_seafile_headers, _seafile_headers,
_seafile_url, _seafile_url,
_webdav_url, _webdav_url,
normalize_connector_browse_path, normalize_connector_browse_path,
) )
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
from govoplan_files.backend.storage.paths import filename_from_path from govoplan_files.backend.storage.paths import filename_from_path
@@ -56,6 +60,7 @@ def read_connector_file(
path: str, path: str,
max_bytes: int, max_bytes: int,
) -> ConnectorDownloadedFile: ) -> ConnectorDownloadedFile:
max_bytes = min(max_bytes, response_limit("file"))
if profile.provider == "seafile": if profile.provider == "seafile":
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"): if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
return _read_webdav_file(profile, path=path, max_bytes=max_bytes) return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
@@ -64,6 +69,8 @@ def read_connector_file(
return _read_webdav_file(profile, path=path, max_bytes=max_bytes) return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
if profile.provider == "smb": if profile.provider == "smb":
return _read_smb_file(profile, path=path, max_bytes=max_bytes) return _read_smb_file(profile, path=path, max_bytes=max_bytes)
if profile.provider == "s3":
return _read_s3_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet") raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet")
@@ -97,8 +104,15 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
if not isinstance(download_url, str) or not download_url.strip(): if not isinstance(download_url, str) or not download_url.strip():
raise ConnectorImportError("Seafile did not return a file download URL") raise ConnectorImportError("Seafile did not return a file download URL")
try: try:
response = httpx.request("GET", download_url, timeout=30.0) response = request_connector_bytes(
except httpx.HTTPError as exc: "GET",
download_url,
timeout=30.0,
kind="file",
max_bytes=max_bytes,
label="Seafile file download",
)
except ConnectorHttpError as exc:
raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc
if response.status_code != 200: if response.status_code != 200:
raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}") raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}")
@@ -144,8 +158,17 @@ def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref: elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import") raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import")
try: try:
response = httpx.request("GET", url, headers=headers, auth=auth, timeout=30.0) response = request_connector_bytes(
except httpx.HTTPError as exc: "GET",
url,
headers=headers,
auth=auth,
timeout=30.0,
kind="file",
max_bytes=max_bytes,
label="WebDAV file download",
)
except ConnectorHttpError as exc:
raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc
if response.status_code in {401, 403}: if response.status_code in {401, 403}:
raise ConnectorImportError("Connector credentials were rejected") raise ConnectorImportError("Connector credentials were rejected")
@@ -171,12 +194,12 @@ def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -
def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile: def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
location = _smb_location(profile)
file_path = normalize_connector_browse_path(path) file_path = normalize_connector_browse_path(path)
if not file_path: if not file_path:
raise ConnectorImportError("SMB import requires a file path") raise ConnectorImportError("SMB import requires a file path")
unc_path = _smb_unc_path(location, file_path)
try: try:
location = _smb_location(profile)
unc_path = _smb_unc_path(location, file_path)
smbclient = _smbclient_module() smbclient = _smbclient_module()
kwargs = _smb_client_kwargs(profile, location) kwargs = _smb_client_kwargs(profile, location)
stat_result = smbclient.stat(unc_path, **kwargs) stat_result = smbclient.stat(unc_path, **kwargs)
@@ -213,6 +236,114 @@ def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> C
) )
def _s3_import_client(profile: ConnectorProfile) -> Any:
try:
return _s3_client(profile)
except ConnectorBrowseUnsupported as exc:
raise ConnectorImportUnsupported(str(exc)) from exc
except ConnectorBrowseError as exc:
raise ConnectorImportError(str(exc)) from exc
def _s3_object_detail(client: Any, *, bucket: str, key: str, max_bytes: int) -> dict[str, Any]:
try:
detail = client.head_object(Bucket=bucket, Key=key)
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorImportError(f"S3 object metadata lookup failed: {exc}") from exc
if not isinstance(detail, dict):
raise ConnectorImportError("S3 connector returned invalid object metadata")
size = _int(detail.get("ContentLength"))
if size is not None and size > max_bytes:
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
return detail
def _download_s3_object(
client: Any,
*,
bucket: str,
key: str,
version_id: str | None,
max_bytes: int,
) -> tuple[Any, bytes]:
request: dict[str, object] = {"Bucket": bucket, "Key": key}
if version_id:
request["VersionId"] = version_id
try:
response = client.get_object(**request)
body = response.get("Body")
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
if len(data) > max_bytes:
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
return response, data
def _s3_download_metadata(
detail: dict[str, Any],
*,
bucket: str,
key: str,
version_id: str | None,
etag: str | None,
size: int,
) -> dict[str, Any]:
metadata: dict[str, Any] = {
"bucket": bucket,
"key": key,
"version_id": version_id,
"etag": etag,
"size": size,
}
for source_key, target_key in (
("ChecksumSHA256", "checksum_sha256"),
("ChecksumCRC32", "checksum_crc32"),
("StorageClass", "storage_class"),
):
if source_key in detail:
metadata[target_key] = detail[source_key]
return metadata
def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
bucket = _s3_bucket(profile, library_id)
if not bucket:
raise ConnectorImportError("S3 import requires a bucket in library_id or profile metadata")
key = _s3_object_key(profile, path)
if not key:
raise ConnectorImportError("S3 import requires an object key")
client = _s3_import_client(profile)
detail = _s3_object_detail(client, bucket=bucket, key=key, max_bytes=max_bytes)
version_id = _clean(detail.get("VersionId"))
response, data = _download_s3_object(
client,
bucket=bucket,
key=key,
version_id=version_id,
max_bytes=max_bytes,
)
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
filename = filename_from_path(key)
return ConnectorDownloadedFile(
filename=filename,
data=data,
content_type=content_type,
revision=version_id or etag or _clean(detail.get("LastModified")),
external_id=f"{bucket}:{key}",
external_url=f"s3://{bucket}/{key}",
metadata=_s3_download_metadata(
detail,
bucket=bucket,
key=key,
version_id=version_id,
etag=etag,
size=len(data),
),
)
def _int(value: object) -> int | None: def _int(value: object) -> int | None:
if value is None or value == "": if value is None or value == "":
return None return None

View File

@@ -1,6 +1,6 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Mapping from collections.abc import Callable, Mapping
from typing import Any from typing import Any
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -9,6 +9,7 @@ from govoplan_core.core.policy import normalize_policy_scope_type
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
from govoplan_files.backend.storage.common import FileStorageError from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
from govoplan_files.backend.storage.connector_credential_store import connector_credential_from_row, credential_rows_by_id from govoplan_files.backend.storage.connector_credential_store import connector_credential_from_row, credential_rows_by_id
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
@@ -19,7 +20,26 @@ def list_database_connector_profiles(
*, *,
tenant_id: str, tenant_id: str,
include_disabled: bool = False, include_disabled: bool = False,
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
) -> list[ConnectorProfile]: ) -> list[ConnectorProfile]:
profiles, _profile_ids = select_database_connector_profiles(
session,
tenant_id=tenant_id,
include_disabled=include_disabled,
row_visible=row_visible,
)
return profiles
def select_database_connector_profiles(
session: Session,
*,
tenant_id: str,
include_disabled: bool = False,
row_visible: Callable[[FileConnectorProfile], bool] | None = None,
) -> tuple[list[ConnectorProfile], set[str]]:
"""Return visible profiles and every database id that shadows settings."""
query = session.query(FileConnectorProfile).filter( query = session.query(FileConnectorProfile).filter(
(FileConnectorProfile.scope_type == "system") (FileConnectorProfile.scope_type == "system")
| (FileConnectorProfile.tenant_id == tenant_id) | (FileConnectorProfile.tenant_id == tenant_id)
@@ -27,9 +47,15 @@ def list_database_connector_profiles(
if not include_disabled: if not include_disabled:
query = query.filter(FileConnectorProfile.enabled.is_(True)) query = query.filter(FileConnectorProfile.enabled.is_(True))
rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all() rows = query.order_by(FileConnectorProfile.scope_type.asc(), FileConnectorProfile.label.asc()).all()
profile_ids = {row.id for row in rows}
if row_visible is not None:
rows = [row for row in rows if row_visible(row)]
credential_ids = {_clean(row.credential_profile_id) for row in rows if _clean(row.credential_profile_id)} credential_ids = {_clean(row.credential_profile_id) for row in rows if _clean(row.credential_profile_id)}
credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled) credentials = credential_rows_by_id(session, tenant_id=tenant_id, credential_ids={item for item in credential_ids if item}, include_disabled=include_disabled)
return [connector_profile_from_row(row, credential_row=credentials.get(row.credential_profile_id or "")) for row in rows] return (
[connector_profile_from_row(row, credential_row=credentials.get(row.credential_profile_id or "")) for row in rows],
profile_ids,
)
def list_connector_profile_rows( def list_connector_profile_rows(
@@ -124,6 +150,12 @@ def create_connector_profile_row(
policy: Mapping[str, Any] | None = None, policy: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None, metadata: Mapping[str, Any] | None = None,
) -> FileConnectorProfile: ) -> FileConnectorProfile:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
metadata=metadata,
)
clean_id = _normalize_profile_id(profile_id) clean_id = _normalize_profile_id(profile_id)
if session.get(FileConnectorProfile, clean_id) is not None: if session.get(FileConnectorProfile, clean_id) is not None:
raise FileStorageError(f"Connector profile already exists: {clean_id}") raise FileStorageError(f"Connector profile already exists: {clean_id}")
@@ -181,6 +213,80 @@ def update_connector_profile_row(
clear_password: bool = False, clear_password: bool = False,
clear_token: bool = False, clear_token: bool = False,
) -> FileConnectorProfile: ) -> FileConnectorProfile:
_validate_profile_update_references(
row,
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
metadata=metadata,
)
_update_profile_connection_fields(
row,
label=label,
provider=provider,
endpoint_url=endpoint_url,
base_path=base_path,
enabled=enabled,
credential_profile_id=credential_profile_id,
credential_mode=credential_mode,
)
_update_profile_credential_fields(
row,
username=username,
password=password,
token=token,
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
clear_password=clear_password,
clear_token=clear_token,
)
_update_profile_governance_fields(
row,
capabilities=capabilities,
policy=policy,
metadata=metadata,
)
row.updated_by_user_id = user_id
session.add(row)
session.flush()
return row
def _validate_profile_update_references(
row: FileConnectorProfile,
*,
password_env: str | None,
token_env: str | None,
secret_ref: str | None,
metadata: Mapping[str, Any] | None,
) -> None:
reject_api_controlled_deployment_references(
password_env=password_env,
token_env=token_env,
secret_ref=secret_ref,
metadata=metadata,
)
if secret_ref is None or _clean(secret_ref) == _clean(row.secret_ref):
return
if _clean(row.secret_ref):
raise FileStorageError(
"An existing external secret reference cannot be replaced or cleared until Files can prove "
"provider ownership and confirm provider-side deletion"
)
def _update_profile_connection_fields(
row: FileConnectorProfile,
*,
label: str | None,
provider: str | None,
endpoint_url: str | None,
base_path: str | None,
enabled: bool | None,
credential_profile_id: str | None,
credential_mode: str | None,
) -> None:
if label is not None: if label is not None:
row.label = _normalize_label(label) row.label = _normalize_label(label)
if provider is not None: if provider is not None:
@@ -195,6 +301,20 @@ def update_connector_profile_row(
row.credential_profile_id = _clean(credential_profile_id) row.credential_profile_id = _clean(credential_profile_id)
if credential_mode is not None: if credential_mode is not None:
row.credential_mode = _normalize_credential_mode(credential_mode) row.credential_mode = _normalize_credential_mode(credential_mode)
def _update_profile_credential_fields(
row: FileConnectorProfile,
*,
username: str | None,
password: str | None,
token: str | None,
password_env: str | None,
token_env: str | None,
secret_ref: str | None,
clear_password: bool,
clear_token: bool,
) -> None:
if username is not None: if username is not None:
row.username = _clean(username) row.username = _clean(username)
if password is not None: if password is not None:
@@ -211,24 +331,21 @@ def update_connector_profile_row(
row.token_env = _clean(token_env) row.token_env = _clean(token_env)
if secret_ref is not None: if secret_ref is not None:
row.secret_ref = _clean(secret_ref) row.secret_ref = _clean(secret_ref)
def _update_profile_governance_fields(
row: FileConnectorProfile,
*,
capabilities: list[str] | None,
policy: Mapping[str, Any] | None,
metadata: Mapping[str, Any] | None,
) -> None:
if capabilities is not None: if capabilities is not None:
row.capabilities = _string_list(capabilities) row.capabilities = _string_list(capabilities)
if policy is not None: if policy is not None:
row.policy = dict(policy) row.policy = dict(policy)
if metadata is not None: if metadata is not None:
row.metadata_ = dict(metadata) row.metadata_ = dict(metadata)
row.updated_by_user_id = user_id
session.add(row)
session.flush()
return row
def deactivate_connector_profile_row(session: Session, row: FileConnectorProfile, *, user_id: str | None) -> FileConnectorProfile:
row.enabled = False
row.updated_by_user_id = user_id
session.add(row)
session.flush()
return row
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]: def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:

View File

@@ -7,13 +7,26 @@ from dataclasses import dataclass, field
from typing import Any from typing import Any
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
from govoplan_files.backend.storage.connector_deployment import (
connector_secret_env_available,
validate_deployment_connector_references,
)
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
_ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON") _ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON")
_ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE") _ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE")
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"} _SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"}
_INLINE_SECRET_FIELDS = ("password", "token", "api_key", "access_key", "secret_key") _INLINE_SECRET_FIELDS = (
"password",
"token",
"api_key",
"access_key",
"access_key_id",
"secret_key",
"secret_access_key",
"session_token",
)
def supported_connector_providers() -> set[str]: def supported_connector_providers() -> set[str]:
@@ -67,15 +80,17 @@ class ConnectorProfile:
@property @property
def credentials_configured(self) -> bool: def credentials_configured(self) -> bool:
if not self.enabled:
return False
mode = self.credential_mode.casefold() mode = self.credential_mode.casefold()
if mode in {"", "none", "anonymous"}: if mode in {"", "none", "anonymous"}:
return True return True
if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value: if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value:
return True return True
if self.token_env: if self.token_env:
return bool(os.environ.get(self.token_env)) return connector_secret_env_available(self.token_env, source_kind=self.source_kind)
if self.password_env: if self.password_env:
return bool(os.environ.get(self.password_env)) return connector_secret_env_available(self.password_env, source_kind=self.source_kind)
return False return False
def to_response(self) -> dict[str, Any]: def to_response(self) -> dict[str, Any]:
@@ -97,7 +112,7 @@ class ConnectorProfile:
"username": self.username, "username": self.username,
"capabilities": list(self.capabilities), "capabilities": list(self.capabilities),
"policy_sources": [_policy_source_response(source) for source in self.policy_sources], "policy_sources": [_policy_source_response(source) for source in self.policy_sources],
"metadata": dict(self.metadata), "metadata": _response_metadata(self.metadata),
"source_kind": self.source_kind, "source_kind": self.source_kind,
} }
@@ -123,45 +138,21 @@ def connector_profiles_from_payload(value: object) -> list[ConnectorProfile]:
def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile: def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name")) profile_id = _profile_id_from_mapping(value)
if not profile_id: provider = _provider_from_mapping(value)
raise ValueError("Connector profiles require id") scope_type, scope_id = _scope_from_mapping(value)
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold() credentials = _credentials_from_mapping(value)
if provider not in _SUPPORTED_PROVIDERS:
raise ValueError(f"Unsupported connector provider: {provider}")
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
scope_id = _clean(value.get("scope_id"))
if scope_type == "system":
scope_id = None
elif not scope_id:
raise ValueError(f"{scope_type} connector profiles require scope_id")
credentials = value.get("credentials")
credentials = credentials if isinstance(credentials, Mapping) else {}
mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none" mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none"
profile = ConnectorProfile( profile = _profile_from_normalized_mapping(
id=profile_id, value,
label=_clean(value.get("label") or value.get("name")) or profile_id, profile_id=profile_id,
provider=provider, provider=provider,
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
enabled=_bool(value.get("enabled"), default=True),
scope_type=scope_type, scope_type=scope_type,
scope_id=scope_id, scope_id=scope_id,
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
credential_mode=mode, credential_mode=mode,
username=_clean(value.get("username") or credentials.get("username")), credentials=credentials,
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
password_value=_clean(value.get("password") or credentials.get("password")),
token_value=_clean(value.get("token") or credentials.get("token")),
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
capabilities=tuple(_string_list(value.get("capabilities"))),
metadata=_public_metadata(value.get("metadata")),
) )
return ConnectorProfile( result = ConnectorProfile(
id=profile.id, id=profile.id,
label=profile.label, label=profile.label,
provider=profile.provider, provider=profile.provider,
@@ -185,6 +176,76 @@ def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
metadata=profile.metadata, metadata=profile.metadata,
source_kind="settings", source_kind="settings",
) )
validate_deployment_connector_references(
source_kind=result.source_kind,
password_env=result.password_env,
token_env=result.token_env,
metadata=result.metadata,
)
return result
def _profile_id_from_mapping(value: Mapping[str, Any]) -> str:
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
if not profile_id:
raise ValueError("Connector profiles require id")
return profile_id
def _provider_from_mapping(value: Mapping[str, Any]) -> str:
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
if provider not in _SUPPORTED_PROVIDERS:
raise ValueError(f"Unsupported connector provider: {provider}")
return provider
def _scope_from_mapping(value: Mapping[str, Any]) -> tuple[str, str | None]:
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
scope_id = _clean(value.get("scope_id"))
if scope_type == "system":
return scope_type, None
if not scope_id:
raise ValueError(f"{scope_type} connector profiles require scope_id")
return scope_type, scope_id
def _credentials_from_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]:
credentials = value.get("credentials")
return credentials if isinstance(credentials, Mapping) else {}
def _profile_from_normalized_mapping(
value: Mapping[str, Any],
*,
profile_id: str,
provider: str,
scope_type: str,
scope_id: str | None,
credential_mode: str,
credentials: Mapping[str, Any],
) -> ConnectorProfile:
return ConnectorProfile(
id=profile_id,
label=_clean(value.get("label") or value.get("name")) or profile_id,
provider=provider,
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
enabled=_bool(value.get("enabled"), default=True),
scope_type=scope_type,
scope_id=scope_id,
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
credential_mode=credential_mode,
username=_clean(value.get("username") or credentials.get("username")),
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
password_value=_clean(value.get("password") or credentials.get("password")),
token_value=_clean(value.get("token") or credentials.get("token")),
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
capabilities=tuple(_string_list(value.get("capabilities"))),
metadata=_public_metadata(value.get("metadata")),
)
def _raw_profiles_payload(settings: object | None) -> object: def _raw_profiles_payload(settings: object | None) -> object:
@@ -243,6 +304,16 @@ def _public_metadata(value: object) -> Mapping[str, Any]:
} }
def _response_metadata(value: Mapping[str, Any]) -> dict[str, Any]:
return {
str(key): item
for key, item in value.items()
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
and not str(key).strip().casefold().endswith("_env")
and str(key).strip().casefold() != "ca_bundle"
}
def _string_list(value: object) -> list[str]: def _string_list(value: object) -> list[str]:
if value is None: if value is None:
return [] return []

View File

@@ -110,7 +110,52 @@ def connector_provider_descriptors() -> tuple[ConnectorProviderDescriptor, ...]:
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after download.", conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after download.",
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the share.", preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the share.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"), audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Requires the optional smbprotocol dependency and an smb://server[:port]/share[/path] profile endpoint.", notes="Browse/import logic is implemented, but live smbprotocol access fails closed until the SDK supports connection-time DNS/IP pinning for initial connections and DFS referrals.",
),
ConnectorProviderDescriptor(
provider="s3",
label="S3-compatible object storage",
protocol="s3-api",
implemented=True,
browse_supported=True,
import_supported=True,
optional_dependency="boto3",
permission_model=governed_permissions,
sync_strategy="On-demand bucket/prefix browse and object import/sync; sync updates managed versions and never mutates the remote bucket.",
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after object download.",
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the bucket.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Browse/import logic is implemented, but live boto3 access fails closed until its HTTP transport supports connection-time DNS/IP pinning and redirect revalidation.",
),
ConnectorProviderDescriptor(
provider="sharepoint",
label="SharePoint",
protocol="microsoft-graph/sharepoint",
implemented=False,
browse_supported=False,
import_supported=False,
optional_dependency=None,
permission_model=governed_permissions,
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph or SharePoint credentials.",
conflict_strategy=freeze_model,
preview_strategy="Will preview from frozen managed file after import, not directly from SharePoint.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Provider/profile key is reserved; live browsing/import still needs Graph/SharePoint authentication and paging implementation.",
),
ConnectorProviderDescriptor(
provider="onedrive",
label="OneDrive",
protocol="microsoft-graph/onedrive",
implemented=False,
browse_supported=False,
import_supported=False,
optional_dependency=None,
permission_model=governed_permissions,
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph credentials.",
conflict_strategy=freeze_model,
preview_strategy="Will preview from frozen managed file after import, not directly from OneDrive.",
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
notes="Provider/profile key is reserved; live browsing/import still needs Graph drive selection, OAuth/app registration, and paging implementation.",
), ),
ConnectorProviderDescriptor( ConnectorProviderDescriptor(
provider="nfs", provider="nfs",

View File

@@ -0,0 +1,218 @@
from __future__ import annotations
from collections.abc import Callable, Iterable
from dataclasses import replace
from sqlalchemy.orm import Session
from govoplan_files.backend.storage.connector_deployment import (
connector_effective_endpoint_url,
)
from govoplan_files.backend.storage.connector_profile_store import (
select_database_connector_profiles,
)
from govoplan_files.backend.storage.connector_profiles import (
ConnectorProfile,
connector_profiles_from_settings,
)
from govoplan_files.backend.storage.connector_policy import (
ConnectorAccessRequest,
connector_policy_decision,
)
from govoplan_files.backend.storage.connector_policy_store import (
effective_connector_policy_sources,
)
from govoplan_files.backend.storage.connector_providers import (
connector_provider_descriptors,
)
CampaignVisibility = Callable[[str], bool]
def visible_connector_profiles_for_actor(
session: Session,
*,
tenant_id: str,
user_id: str,
group_ids: Iterable[str],
settings: object | None,
provider: str | None = None,
campaign_id: str | None = None,
campaign_visible: CampaignVisibility | None = None,
include_disabled: bool = False,
include_admin_scopes: bool = False,
include_effective_policy: bool = True,
) -> list[ConnectorProfile]:
"""Return the same actor-filtered connector set used by API and docs surfaces."""
provider_norm = provider.strip().casefold() if provider else None
actor_group_ids = {str(group_id) for group_id in group_ids if str(group_id)}
database_profiles, database_profile_ids = select_database_connector_profiles(
session,
tenant_id=tenant_id,
include_disabled=include_disabled,
row_visible=lambda row: (
include_admin_scopes
and row.scope_type in {"user", "group", "campaign"}
) or _scope_visible_to_actor(
scope_type=row.scope_type,
scope_id=row.scope_id,
tenant_id=tenant_id,
user_id=user_id,
group_ids=actor_group_ids,
campaign_id=campaign_id,
campaign_visible=campaign_visible,
),
)
configured_profiles = connector_profiles_from_settings(settings)
profiles_by_id: dict[str, ConnectorProfile] = {}
for profile in database_profiles:
if profile.id not in profiles_by_id:
profiles_by_id[profile.id] = profile
for profile in configured_profiles:
if profile.id not in database_profile_ids and profile.id not in profiles_by_id:
profiles_by_id[profile.id] = profile
visible: list[ConnectorProfile] = []
for profile in profiles_by_id.values():
if not (profile.enabled or include_disabled):
continue
if provider_norm is not None and profile.provider != provider_norm:
continue
admin_scope_visible = include_admin_scopes and profile.scope_type in {
"user",
"group",
"campaign",
}
if not admin_scope_visible and not _profile_visible_to_actor(
profile,
tenant_id=tenant_id,
user_id=user_id,
group_ids=actor_group_ids,
campaign_id=campaign_id,
campaign_visible=campaign_visible,
):
continue
visible.append(
_with_effective_connector_policy(
session, tenant_id=tenant_id, profile=profile
)
if include_effective_policy
else profile
)
return visible
def connector_profile_usable_for_import(profile: ConnectorProfile) -> bool:
"""Whether a visible profile can safely offer the current live browse/import task."""
effective_endpoint_url = connector_effective_endpoint_url(
provider=profile.provider,
endpoint_url=profile.endpoint_url,
metadata=profile.metadata,
)
if (
not profile.enabled
or not effective_endpoint_url
or not profile.credentials_configured
or bool(profile.secret_ref)
):
return False
descriptor = next(
(
item
for item in connector_provider_descriptors()
if item.provider == profile.provider
),
None,
)
if descriptor is None:
return False
if not (
descriptor.implemented
and descriptor.installed
and descriptor.browse_supported
and descriptor.import_supported
):
return False
# These SDK paths intentionally fail closed until every connection peer and
# SDK-managed redirect/referral can be pinned and revalidated.
if profile.provider in {"s3", "smb"}:
return False
# Prove that the initial root browse performed by the current Files UI is
# policy-allowed. A later selected remote path/item is checked again.
return connector_policy_decision(
ConnectorAccessRequest(
connector_id=profile.id,
credential_id=profile.credential_profile_id,
provider=profile.provider,
external_path="",
external_url=effective_endpoint_url,
operation="import",
),
profile.policy_sources,
).allowed
def _profile_visible_to_actor(
profile: ConnectorProfile,
*,
tenant_id: str,
user_id: str,
group_ids: set[str],
campaign_id: str | None,
campaign_visible: CampaignVisibility | None,
) -> bool:
return _scope_visible_to_actor(
scope_type=profile.scope_type,
scope_id=profile.scope_id,
tenant_id=tenant_id,
user_id=user_id,
group_ids=group_ids,
campaign_id=campaign_id,
campaign_visible=campaign_visible,
)
def _scope_visible_to_actor(
*,
scope_type: str,
scope_id: str | None,
tenant_id: str,
user_id: str,
group_ids: set[str],
campaign_id: str | None,
campaign_visible: CampaignVisibility | None,
) -> bool:
if scope_type == "system":
return True
if scope_type == "tenant":
return scope_id == tenant_id
if scope_type == "user":
return scope_id == user_id
if scope_type == "group":
return bool(scope_id and scope_id in group_ids)
if scope_type != "campaign" or not scope_id:
return False
if campaign_id and scope_id != campaign_id:
return False
return bool(campaign_visible and campaign_visible(scope_id))
def _with_effective_connector_policy(
session: Session,
*,
tenant_id: str,
profile: ConnectorProfile,
) -> ConnectorProfile:
sources = effective_connector_policy_sources(
session,
tenant_id=tenant_id,
scope_type=profile.scope_type,
scope_id=profile.scope_id,
)
if not sources:
return profile
return replace(profile, policy_sources=tuple([*sources, *profile.policy_sources]))

View File

@@ -0,0 +1,239 @@
from __future__ import annotations
import socket
import ssl
import select
from collections.abc import Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any, Iterator
import httpcore
import httpx
from govoplan_core.security.outbound_http import (
OutboundHttpError,
bounded_chunks_bytes,
create_outbound_connection,
response_limit,
validate_outbound_http_url,
)
class ConnectorHttpError(RuntimeError):
pass
@dataclass(frozen=True, slots=True)
class ConnectorHttpResponse:
status_code: int
headers: Mapping[str, str]
content: bytes
_HTTPCORE_TRANSPORT_ERRORS = (
httpcore.TimeoutException,
httpcore.NetworkError,
httpcore.ProtocolError,
httpcore.ProxyError,
httpcore.UnsupportedProtocol,
)
class _SocketNetworkStream(httpcore.NetworkStream):
"""Public httpcore NetworkStream adapter around an approved socket."""
def __init__(self, sock: socket.socket) -> None:
self._socket = sock
def read(self, max_bytes: int, timeout: float | None = None) -> bytes:
try:
self._socket.settimeout(timeout)
return self._socket.recv(max_bytes)
except socket.timeout as exc:
raise httpcore.ReadTimeout(str(exc)) from exc
except OSError as exc:
raise httpcore.ReadError(str(exc)) from exc
def write(self, buffer: bytes, timeout: float | None = None) -> None:
try:
self._socket.settimeout(timeout)
self._socket.sendall(buffer)
except socket.timeout as exc:
raise httpcore.WriteTimeout(str(exc)) from exc
except OSError as exc:
raise httpcore.WriteError(str(exc)) from exc
def close(self) -> None:
self._socket.close()
def start_tls(
self,
ssl_context: ssl.SSLContext,
server_hostname: str | None = None,
timeout: float | None = None,
) -> httpcore.NetworkStream:
try:
self._socket.settimeout(timeout)
tls_socket = ssl_context.wrap_socket(self._socket, server_hostname=server_hostname)
except socket.timeout as exc:
self.close()
raise httpcore.ConnectTimeout(str(exc)) from exc
except OSError as exc:
self.close()
raise httpcore.ConnectError(str(exc)) from exc
return _SocketNetworkStream(tls_socket)
def get_extra_info(self, info: str) -> Any:
if info == "ssl_object" and isinstance(self._socket, ssl.SSLSocket):
return self._socket
if info == "client_addr":
return self._socket.getsockname()
if info == "server_addr":
return self._socket.getpeername()
if info == "socket":
return self._socket
if info == "is_readable":
try:
return bool(select.select([self._socket], [], [], 0)[0])
except (OSError, ValueError):
return True
return None
class _OutboundPolicyNetworkBackend(httpcore.NetworkBackend):
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options: Any = None,
) -> httpcore.NetworkStream:
source_address = None if local_address is None else (local_address, 0)
try:
sock = create_outbound_connection(
host,
port,
timeout=timeout,
source_address=source_address,
socket_options=socket_options,
label="File connector HTTP endpoint",
)
except socket.timeout as exc:
raise httpcore.ConnectTimeout(str(exc)) from exc
except (OSError, OutboundHttpError) as exc:
raise httpcore.ConnectError(str(exc)) from exc
return _SocketNetworkStream(sock)
def connect_unix_socket(self, path: str, timeout: float | None = None, socket_options: Any = None): # type: ignore[no-untyped-def]
del path, timeout, socket_options
raise httpcore.ConnectError("Unix sockets are not supported for file connectors")
class _HttpcoreResponseStream(httpx.SyncByteStream):
def __init__(self, stream: Any, *, request: httpx.Request) -> None:
self._stream = stream
self._request = request
def __iter__(self): # type: ignore[no-untyped-def]
try:
yield from self._stream
except _HTTPCORE_TRANSPORT_ERRORS as exc:
raise httpx.TransportError(str(exc), request=self._request) from exc
def close(self) -> None:
if hasattr(self._stream, "close"):
self._stream.close()
class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
def __init__(self) -> None:
self._connection_pool = httpcore.ConnectionPool(
ssl_context=httpx.create_ssl_context(verify=True, trust_env=False),
max_connections=100,
max_keepalive_connections=20,
keepalive_expiry=5.0,
network_backend=_OutboundPolicyNetworkBackend(),
)
def handle_request(self, request: httpx.Request) -> httpx.Response:
core_request = httpcore.Request(
method=request.method,
url=httpcore.URL(
scheme=request.url.raw_scheme,
host=request.url.raw_host,
port=request.url.port,
target=request.url.raw_path,
),
headers=request.headers.raw,
content=request.stream,
extensions=request.extensions,
)
try:
response = self._connection_pool.handle_request(core_request)
except _HTTPCORE_TRANSPORT_ERRORS as exc:
raise httpx.TransportError(str(exc), request=request) from exc
return httpx.Response(
status_code=response.status,
headers=response.headers,
stream=_HttpcoreResponseStream(response.stream, request=request),
extensions=response.extensions,
)
def close(self) -> None:
self._connection_pool.close()
@contextmanager
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
with httpx.Client(
transport=_OutboundPolicyHTTPTransport(),
follow_redirects=False,
timeout=kwargs.pop("timeout", 15.0),
) as client:
with client.stream(method, url, **kwargs) as response:
yield response
def request_connector_bytes(
method: str,
url: str,
*,
headers: Mapping[str, str] | None = None,
params: Mapping[str, str] | None = None,
data: Mapping[str, str] | bytes | str | None = None,
content: bytes | str | None = None,
auth: Any = None,
timeout: float = 15.0,
kind: str = "structured",
max_bytes: int | None = None,
label: str = "File connector",
) -> ConnectorHttpResponse:
try:
validated_url = validate_outbound_http_url(url, label=f"{label} URL")
effective_limit = response_limit(kind) if max_bytes is None else min(int(max_bytes), response_limit(kind))
with _stream_connector_request(
method,
validated_url,
headers=dict(headers or {}),
params=params,
data=data,
content=content,
auth=auth,
timeout=timeout,
) as response:
body = bounded_chunks_bytes(
response.iter_bytes(),
headers=response.headers,
max_bytes=effective_limit,
kind=kind,
label=f"{label} response",
)
return ConnectorHttpResponse(
status_code=response.status_code,
headers=dict(response.headers),
content=body,
)
except (httpx.HTTPError, OutboundHttpError, ValueError) as exc:
raise ConnectorHttpError(str(exc)) from exc

View File

@@ -15,21 +15,6 @@ from govoplan_files.backend.storage.common import (
utcnow, utcnow,
) )
from govoplan_files.backend.storage.files import ( from govoplan_files.backend.storage.files import (
_active_asset_at_path,
_active_asset_exists,
_asset_owner_id,
_asset_query_for_owner,
_candidate_renamed_path,
_copy_asset_to_path,
_get_or_create_blob,
_next_available_logical_path,
_normalize_conflict_strategy,
_resolution_by_path,
_soft_delete_conflicting_asset,
_split_logical_path,
_storage_backend_name,
_storage_bucket_name,
_storage_key,
asset_is_audit_relevant, asset_is_audit_relevant,
create_file_asset, create_file_asset,
current_version_and_blob, current_version_and_blob,
@@ -42,10 +27,6 @@ from govoplan_files.backend.storage.files import (
soft_delete_assets, soft_delete_assets,
) )
from govoplan_files.backend.storage.folders import ( from govoplan_files.backend.storage.folders import (
_active_folder_exists,
_ensure_target_folder_hierarchy,
_folder_query_for_owner,
_owner_filter,
create_folder, create_folder,
list_folders_for_user, list_folders_for_user,
soft_delete_folder, soft_delete_folder,

File diff suppressed because it is too large Load Diff

View File

@@ -21,6 +21,7 @@ GROUP_ID = "group-1"
class FilesAccessProviderTests(unittest.TestCase): class FilesAccessProviderTests(unittest.TestCase):
def test_file_access_provider_explains_owner_share_admin_and_missing_resources(self) -> None: def test_file_access_provider_explains_owner_share_admin_and_missing_resources(self) -> None:
session = _session() session = _session()
self.addCleanup(_close_session, session)
_seed_access_subjects(session) _seed_access_subjects(session)
owned = FileAsset(id="file-owned", tenant_id=TENANT_ID, owner_type="user", owner_user_id=USER_ID, display_path="owned.pdf", filename="owned.pdf") owned = FileAsset(id="file-owned", tenant_id=TENANT_ID, owner_type="user", owner_user_id=USER_ID, display_path="owned.pdf", filename="owned.pdf")
shared = FileAsset(id="file-shared", tenant_id=TENANT_ID, owner_type="user", owner_user_id=OTHER_USER_ID, display_path="shared.pdf", filename="shared.pdf") shared = FileAsset(id="file-shared", tenant_id=TENANT_ID, owner_type="user", owner_user_id=OTHER_USER_ID, display_path="shared.pdf", filename="shared.pdf")
@@ -46,6 +47,7 @@ class FilesAccessProviderTests(unittest.TestCase):
def test_file_access_provider_explains_persisted_and_virtual_folders(self) -> None: def test_file_access_provider_explains_persisted_and_virtual_folders(self) -> None:
session = _session() session = _session()
self.addCleanup(_close_session, session)
_seed_access_subjects(session) _seed_access_subjects(session)
persisted = FileFolder(id="folder-persisted", tenant_id=TENANT_ID, owner_type="group", owner_group_id=GROUP_ID, path="records") persisted = FileFolder(id="folder-persisted", tenant_id=TENANT_ID, owner_type="group", owner_group_id=GROUP_ID, path="records")
virtual_child = FileAsset( virtual_child = FileAsset(
@@ -80,6 +82,12 @@ def _session():
return sessionmaker(bind=engine, future=True)() return sessionmaker(bind=engine, future=True)()
def _close_session(session) -> None:
engine = session.get_bind()
session.close()
engine.dispose()
def _seed_access_subjects(session) -> None: def _seed_access_subjects(session) -> None:
session.add_all([ session.add_all([
Account(id="account-1", email="one@example.test", normalized_email="one@example.test"), Account(id="account-1", email="one@example.test", normalized_email="one@example.test"),

View File

@@ -0,0 +1,277 @@
from __future__ import annotations
import unittest
from types import SimpleNamespace
from unittest.mock import patch
from sqlalchemy import create_engine, inspect
from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db import models as access_models # noqa: F401 - resolve Files user foreign keys
from govoplan_core.core.change_sequence import ChangeSequenceEntry
from govoplan_core.db.base import Base
from govoplan_core.security.secrets import encrypt_secret
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
from govoplan_files.backend.router import deactivate_connector_credential, deactivate_connector_profile
class Principal:
tenant_id = "tenant-1"
user = SimpleNamespace(id="user-1")
api_key = None
def has(self, scope: str) -> bool:
return scope == "files:file:admin"
class ConnectorCredentialDeletionTests(unittest.TestCase):
def setUp(self) -> None:
self.engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(
self.engine,
tables=[
FileConnectorCredential.__table__,
FileConnectorProfile.__table__,
ChangeSequenceEntry.__table__,
],
)
self.session = sessionmaker(bind=self.engine)()
self.principal = Principal()
def tearDown(self) -> None:
self.session.close()
Base.metadata.drop_all(bind=self.engine)
self.engine.dispose()
def test_delete_credential_scrubs_material_audits_and_disables_dependents(self) -> None:
credential = self._credential()
profile = self._profile(credential_profile_id=credential.id)
self.session.add_all([credential, profile])
self.session.commit()
with patch(
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit:
response = deactivate_connector_credential(
credential.id,
session=self.session,
principal=self.principal, # type: ignore[arg-type]
)
self.assertFalse(response.enabled)
self.assertFalse(response.credentials_configured)
self.assertIsNone(response.credential_secret_source)
self.session.refresh(credential)
self.session.refresh(profile)
self._assert_credential_scrubbed(credential)
self._assert_profile_scrubbed(profile)
calls = {call.kwargs["action"]: call.kwargs for call in audit.call_args_list}
self.assertEqual(
{"files.connector_profile_deleted", "files.connector_credential_deleted"},
set(calls),
)
credential_audit = calls["files.connector_credential_deleted"]
self.assertEqual("encrypted_database", credential_audit["details"]["storage_backend"])
self.assertEqual(["password", "token"], credential_audit["details"]["deleted_secret_kinds"])
self.assertEqual(1, credential_audit["details"]["affected_profile_count"])
self.assertNotIn("do-not-audit", repr(audit.call_args_list))
changes = self.session.query(ChangeSequenceEntry).order_by(ChangeSequenceEntry.id.asc()).all()
self.assertEqual([credential.id, profile.id], [change.resource_id for change in changes])
self.assertEqual(["deleted", "updated"], [change.operation for change in changes])
def test_delete_profile_scrubs_inline_credentials_and_audits(self) -> None:
profile = self._profile(credential_profile_id="shared-credential")
self.session.add(profile)
self.session.commit()
with patch(
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit:
response = deactivate_connector_profile(
profile.id,
session=self.session,
principal=self.principal, # type: ignore[arg-type]
)
self.assertFalse(response.enabled)
self.assertFalse(response.credentials_configured)
self.assertIsNone(response.credential_profile_id)
self.session.refresh(profile)
self._assert_profile_scrubbed(profile)
audit.assert_called_once()
call = audit.call_args.kwargs
self.assertEqual("files.connector_profile_deleted", call["action"])
self.assertEqual("api_delete", call["details"]["deletion_reason"])
self.assertIn("credential_profile", call["details"]["removed_reference_kinds"])
self.assertNotIn("do-not-audit", repr(call))
def test_delete_rolls_back_when_audit_fails(self) -> None:
credential = self._credential()
self.session.add(credential)
self.session.commit()
original_password = credential.password_encrypted
original_token = credential.token_encrypted
with patch(
"govoplan_files.backend.storage.connector_credential_deletion.audit_event",
side_effect=RuntimeError("audit unavailable"),
), self.assertRaisesRegex(RuntimeError, "audit unavailable"):
deactivate_connector_credential(
credential.id,
session=self.session,
principal=self.principal, # type: ignore[arg-type]
)
persisted = self.session.get(FileConnectorCredential, credential.id)
assert persisted is not None
self.assertTrue(persisted.enabled)
self.assertEqual(original_password, persisted.password_encrypted)
self.assertEqual(original_token, persisted.token_encrypted)
self.assertEqual(0, self.session.query(ChangeSequenceEntry).count())
def test_delete_detaches_unowned_legacy_secret_reference_without_claiming_provider_deletion(self) -> None:
credential = self._credential(secret_ref="vault:tenant-1:files:credential")
self.session.add(credential)
self.session.commit()
with patch(
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit:
response = deactivate_connector_credential(
credential.id,
session=self.session,
principal=self.principal, # type: ignore[arg-type]
)
self.assertFalse(response.enabled)
persisted = self.session.get(FileConnectorCredential, credential.id)
assert persisted is not None
self._assert_credential_scrubbed(persisted)
audit.assert_called_once()
details = audit.call_args.kwargs["details"]
self.assertEqual("unowned_external_reference_detached", details["storage_backend"])
self.assertIn("unowned_external_secret_ref", details["removed_reference_kinds"])
self.assertNotIn("vault:tenant-1:files:credential", repr(audit.call_args))
def test_retirement_scrubs_and_audits_before_tables_are_dropped(self) -> None:
from govoplan_files.backend.manifest import manifest
self.session.add_all([self._credential(), self._profile()])
self.session.commit()
retirement_provider = manifest.migration_spec.retirement_provider
assert retirement_provider is not None
plan = retirement_provider(self.session, "files")
assert plan.destroy_data_executor is not None
with patch(
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit:
plan.destroy_data_executor(self.session, "files")
# The retirement executor deliberately enlists secret scrubbing, audit
# writes, and DDL in the caller-owned transaction. Commit that unit of
# work before inspecting through a separate Engine connection.
self.session.commit()
self.assertEqual(2, audit.call_count)
self.assertEqual(
{"module_data_retired"},
{call.kwargs["details"]["deletion_reason"] for call in audit.call_args_list},
)
self.assertNotIn("do-not-audit", repr(audit.call_args_list))
self.assertFalse(inspect(self.engine).has_table(FileConnectorCredential.__tablename__))
self.assertFalse(inspect(self.engine).has_table(FileConnectorProfile.__tablename__))
def test_retirement_detaches_unowned_external_reference_before_drop(self) -> None:
from govoplan_files.backend.manifest import manifest
self.session.add(self._credential(secret_ref="vault:tenant-1:files:credential"))
self.session.commit()
retirement_provider = manifest.migration_spec.retirement_provider
assert retirement_provider is not None
plan = retirement_provider(self.session, "files")
assert plan.destroy_data_executor is not None
with patch(
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
) as audit:
plan.destroy_data_executor(self.session, "files")
self.session.commit()
audit.assert_called_once()
details = audit.call_args.kwargs["details"]
self.assertIn("unowned_external_secret_ref", details["removed_reference_kinds"])
self.assertNotIn("vault:tenant-1:files:credential", repr(audit.call_args))
self.assertFalse(inspect(self.engine).has_table(FileConnectorCredential.__tablename__))
self.assertFalse(inspect(self.engine).has_table(FileConnectorProfile.__tablename__))
@staticmethod
def _credential(*, secret_ref: str | None = None) -> FileConnectorCredential:
return FileConnectorCredential(
id="credential-1",
tenant_id="tenant-1",
scope_type="tenant",
scope_id="tenant-1",
label="Credential",
provider="webdav",
enabled=True,
credential_mode="basic",
username="connector-user",
password_encrypted=encrypt_secret("do-not-audit-password"),
token_encrypted=encrypt_secret("do-not-audit-token"),
password_env="DEPLOYMENT_PASSWORD",
token_env="DEPLOYMENT_TOKEN",
secret_ref=secret_ref,
policy={},
metadata_={"private_hint": "do-not-audit-metadata"},
)
@staticmethod
def _profile(*, credential_profile_id: str | None = None) -> FileConnectorProfile:
return FileConnectorProfile(
id="profile-1",
tenant_id="tenant-1",
scope_type="tenant",
scope_id="tenant-1",
label="Profile",
provider="webdav",
endpoint_url="https://dav.example.test",
enabled=True,
credential_profile_id=credential_profile_id,
credential_mode="basic",
username="connector-user",
password_encrypted=encrypt_secret("do-not-audit-profile-password"),
token_encrypted=encrypt_secret("do-not-audit-profile-token"),
password_env="DEPLOYMENT_PROFILE_PASSWORD",
token_env="DEPLOYMENT_PROFILE_TOKEN",
policy={},
metadata_={"private_hint": "do-not-audit-profile-metadata"},
)
def _assert_credential_scrubbed(self, row: FileConnectorCredential) -> None:
self.assertFalse(row.enabled)
self.assertEqual("none", row.credential_mode)
self.assertIsNone(row.username)
self.assertIsNone(row.password_encrypted)
self.assertIsNone(row.token_encrypted)
self.assertIsNone(row.password_env)
self.assertIsNone(row.token_env)
self.assertIsNone(row.secret_ref)
self.assertEqual({}, row.metadata_)
def _assert_profile_scrubbed(self, row: FileConnectorProfile) -> None:
self.assertFalse(row.enabled)
self.assertIsNone(row.credential_profile_id)
self.assertEqual("none", row.credential_mode)
self.assertIsNone(row.username)
self.assertIsNone(row.password_encrypted)
self.assertIsNone(row.token_encrypted)
self.assertIsNone(row.password_env)
self.assertIsNone(row.token_env)
self.assertIsNone(row.secret_ref)
self.assertEqual({}, row.metadata_)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,227 @@
from __future__ import annotations
import os
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from unittest.mock import MagicMock
from fastapi import HTTPException
from govoplan_files.backend.router import discover_connector_endpoint
from govoplan_files.backend.schemas import FileConnectorDiscoveryRequest
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _profile_password, _s3_verify
from govoplan_files.backend.storage.connector_deployment import (
ConnectorDeploymentConfigurationError,
connector_effective_endpoint_url,
connector_secret_env_value,
reject_api_controlled_deployment_references,
)
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.connector_profile_store import create_connector_profile_row
class ConnectorDeploymentBoundaryTests(unittest.TestCase):
def test_database_profile_cannot_read_arbitrary_process_environment(self) -> None:
profile = ConnectorProfile(
id="tenant-webdav",
label="Tenant WebDAV",
provider="webdav",
password_env="MASTER_KEY_B64",
source_kind="database",
)
with patch.dict(
os.environ,
{
"MASTER_KEY_B64": "must-not-leave-process",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST": "MASTER_KEY_B64",
},
clear=False,
), self.assertRaisesRegex(ConnectorBrowseError, "deployment-owned"):
_profile_password(profile)
def test_deployment_profile_requires_exact_secret_allowlist(self) -> None:
with patch.dict(
os.environ,
{
"GOVOPLAN_FILES_WEBDAV_PASSWORD": "deployment-secret",
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST": "GOVOPLAN_FILES_WEBDAV_PASSWORD",
},
clear=False,
):
self.assertEqual(
"deployment-secret",
connector_secret_env_value(
"GOVOPLAN_FILES_WEBDAV_PASSWORD",
source_kind="settings",
),
)
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "not listed"):
connector_secret_env_value("MASTER_KEY_B64", source_kind="settings")
def test_api_profiles_cannot_select_secret_environment_names(self) -> None:
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
reject_api_controlled_deployment_references(password_env="DATABASE_URL")
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
reject_api_controlled_deployment_references(metadata={"secret_access_key_env": "MASTER_KEY_B64"})
session = MagicMock()
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
create_connector_profile_row(
session,
tenant_id="tenant-1",
user_id="user-1",
profile_id="unsafe",
label="Unsafe",
provider="webdav",
scope_type="tenant",
password_env="DATABASE_URL",
)
session.get.assert_not_called()
def test_api_profiles_cannot_create_unowned_external_secret_references(self) -> None:
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "provider ownership"):
reject_api_controlled_deployment_references(secret_ref="vault:tenant-1:files")
session = MagicMock()
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "provider ownership"):
create_connector_profile_row(
session,
tenant_id="tenant-1",
user_id="user-1",
profile_id="unsafe-secret-ref",
label="Unsafe secret reference",
provider="webdav",
scope_type="tenant",
credential_mode="secret_ref",
secret_ref="vault:tenant-1:files",
)
session.get.assert_not_called()
def test_api_profiles_cannot_hide_plaintext_credentials_in_metadata(self) -> None:
for metadata in (
{"secret_access_key": "plaintext-secret"},
{"credentials": {"password": "plaintext-secret"}},
{"provider": {"refresh_token": "plaintext-secret"}},
):
with self.subTest(metadata=metadata), self.assertRaisesRegex(
ConnectorDeploymentConfigurationError,
"dedicated encrypted credential fields",
):
reject_api_controlled_deployment_references(metadata=metadata)
def test_profile_responses_hide_environment_and_local_ca_references(self) -> None:
profile = ConnectorProfile(
id="deployment-s3",
label="Deployment S3",
provider="s3",
metadata={
"bucket": "documents",
"secret_access_key_env": "GOVOPLAN_FILES_S3_SECRET",
"ca_bundle": "/etc/govoplan/connector-ca.pem",
},
)
self.assertEqual({"bucket": "documents"}, profile.to_response()["metadata"])
def test_ca_bundle_must_be_an_exact_deployment_allowlisted_file(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
allowed = Path(temp_dir, "connector-ca.pem")
other = Path(temp_dir, "other-ca.pem")
allowed.write_text("test CA", encoding="utf-8")
other.write_text("other CA", encoding="utf-8")
with patch.dict(
os.environ,
{
"APP_ENV": "production",
"GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST": str(allowed),
},
clear=False,
):
profile = ConnectorProfile(
id="s3",
label="S3",
provider="s3",
metadata={"ca_bundle": str(allowed)},
)
self.assertEqual(str(allowed.resolve()), _s3_verify(profile))
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "not listed"):
reject_api_controlled_deployment_references(metadata={"ca_bundle": str(other)})
def test_tls_verification_can_only_be_disabled_in_development(self) -> None:
with patch.dict(os.environ, {"APP_ENV": "production"}, clear=False), self.assertRaisesRegex(
ConnectorDeploymentConfigurationError,
"dev/test",
):
reject_api_controlled_deployment_references(metadata={"verify_tls": False})
with patch.dict(os.environ, {"APP_ENV": "test"}, clear=False):
reject_api_controlled_deployment_references(metadata={"verify_tls": False})
def test_effective_endpoint_uses_webdav_override(self) -> None:
self.assertEqual(
"https://dav.example.test/root",
connector_effective_endpoint_url(
provider="seafile",
endpoint_url="https://seafile.example.test",
metadata={"webdav_endpoint_url": "https://dav.example.test/root"},
),
)
class ConnectorDiscoveryBoundaryTests(unittest.TestCase):
@staticmethod
def _principal() -> SimpleNamespace:
return SimpleNamespace(tenant_id="tenant-1", user=SimpleNamespace(id="user-1"), api_key=None)
def test_discovery_rejects_environment_credentials_before_io(self) -> None:
payload = FileConnectorDiscoveryRequest(
provider="webdav",
endpoint_url="https://dav.example.test",
credential_mode="basic",
credentials={"username": "admin", "password_env": "MASTER_KEY_B64"},
)
with patch("govoplan_files.backend.router.browse_connector_profile") as browse, self.assertRaises(
HTTPException
) as raised:
discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
self.assertEqual(400, raised.exception.status_code)
browse.assert_not_called()
def test_discovery_applies_policy_and_audit_before_each_io_candidate(self) -> None:
payload = FileConnectorDiscoveryRequest(
provider="webdav",
endpoint_url="https://dav.example.test/root",
metadata={
"webdav_endpoint_url": "https://bypass.example.test",
"static_listing": {"": []},
},
)
events: list[str] = []
def ensure(*_args: object, **kwargs: object) -> None:
self.assertEqual("https://dav.example.test/root/", kwargs["endpoint_url"])
events.append("policy")
def audit(*_args: object, **_kwargs: object) -> None:
events.append("audit")
def browse(profile: ConnectorProfile, **_kwargs: object) -> list[object]:
self.assertEqual({}, profile.metadata)
events.append("io")
return []
with patch("govoplan_files.backend.router._ensure_connector_configuration_allowed", side_effect=ensure), patch(
"govoplan_files.backend.router._audit_connector_discovery_attempt",
side_effect=audit,
), patch("govoplan_files.backend.router.browse_connector_profile", side_effect=browse):
response = discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
self.assertEqual("usable", response.status)
self.assertEqual(["policy", "audit", "io"], events)
self.assertEqual({"discovered_by": "webdav-propfind"}, response.metadata)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,176 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_access.backend.db import models as access_models # noqa: F401 - resolve Files user foreign keys
from govoplan_core.db.base import Base
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
from govoplan_files.backend.db.models import FileConnectorProfile
from govoplan_files.backend.storage.common import FileStorageError
from govoplan_files.backend.storage.connector_profile_store import (
list_database_connector_profiles,
update_connector_profile_row,
)
class ConnectorProfileStoreUpdateTests(unittest.TestCase):
def setUp(self) -> None:
self.engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(self.engine, tables=[FileConnectorProfile.__table__])
self.session = sessionmaker(bind=self.engine)()
self.row = FileConnectorProfile(
id="profile-1",
tenant_id="tenant-1",
scope_type="tenant",
scope_id="tenant-1",
label="Original profile",
provider="webdav",
endpoint_url="https://dav.example.test/original",
base_path="original",
enabled=True,
credential_mode="basic",
username="original-user",
password_encrypted=encrypt_secret("original-password"),
token_encrypted=encrypt_secret("original-token"),
capabilities=["browse"],
policy={"allow": {"providers": ["webdav"]}},
metadata_={"original": True},
created_by_user_id=None,
updated_by_user_id=None,
)
self.session.add(self.row)
self.session.commit()
def tearDown(self) -> None:
self.session.close()
Base.metadata.drop_all(bind=self.engine)
self.engine.dispose()
def test_update_preserves_normalization_and_flush_only_transaction_boundary(self) -> None:
updated = update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
label=" Updated profile ",
provider="NEXTCLOUD",
endpoint_url=" https://cloud.example.test/dav ",
base_path=" shared/reports ",
enabled=False,
credential_profile_id=" credential-2 ",
credential_mode="TOKEN",
username=" updated-user ",
password="updated-password",
token="updated-token",
capabilities=["browse", " import ", ""],
policy={"deny": {"external_paths": ["private"]}},
metadata={"department": "reports"},
)
self.assertIs(self.row, updated)
self.assertEqual("Updated profile", updated.label)
self.assertEqual("nextcloud", updated.provider)
self.assertEqual("https://cloud.example.test/dav", updated.endpoint_url)
self.assertEqual("shared/reports", updated.base_path)
self.assertFalse(updated.enabled)
self.assertEqual("credential-2", updated.credential_profile_id)
self.assertEqual("token", updated.credential_mode)
self.assertEqual("updated-user", updated.username)
self.assertEqual("updated-password", decrypt_secret(updated.password_encrypted))
self.assertEqual("updated-token", decrypt_secret(updated.token_encrypted))
self.assertEqual(["browse", "import"], updated.capabilities)
self.assertEqual({"deny": {"external_paths": ["private"]}}, updated.policy)
self.assertEqual({"department": "reports"}, updated.metadata_)
self.assertEqual("user-2", updated.updated_by_user_id)
self.session.rollback()
persisted = self.session.get(FileConnectorProfile, self.row.id)
assert persisted is not None
self.assertEqual("Original profile", persisted.label)
self.assertEqual("original-password", decrypt_secret(persisted.password_encrypted))
self.assertTrue(persisted.enabled)
def test_secret_replacement_wins_over_clear_while_clear_removes_an_omitted_token(self) -> None:
update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
password="replacement-password",
clear_password=True,
clear_token=True,
)
self.assertEqual("replacement-password", decrypt_secret(self.row.password_encrypted))
self.assertIsNone(self.row.token_encrypted)
self.assertEqual("https://dav.example.test/original", self.row.endpoint_url)
self.assertEqual("original-user", self.row.username)
def test_legacy_external_secret_reference_cannot_be_cleared_or_partially_mutate_the_row(self) -> None:
self.row.secret_ref = "vault:tenant-1:files:profile"
self.session.commit()
with self.assertRaisesRegex(FileStorageError, "provider-side deletion"):
update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
label="Must not be applied",
secret_ref="",
)
self.assertEqual("Original profile", self.row.label)
self.assertEqual("vault:tenant-1:files:profile", self.row.secret_ref)
def test_flush_failure_remains_rollback_safe(self) -> None:
with patch.object(self.session, "flush", side_effect=RuntimeError("database unavailable")), self.assertRaisesRegex(
RuntimeError,
"database unavailable",
):
update_connector_profile_row(
self.session,
self.row,
user_id="user-2",
label="Uncommitted profile",
password="uncommitted-password",
)
self.session.rollback()
persisted = self.session.get(FileConnectorProfile, self.row.id)
assert persisted is not None
self.assertEqual("Original profile", persisted.label)
self.assertEqual("original-password", decrypt_secret(persisted.password_encrypted))
def test_visibility_filter_runs_before_connector_secrets_are_decrypted(self) -> None:
invisible = FileConnectorProfile(
id="invisible-profile",
tenant_id="tenant-1",
scope_type="user",
scope_id="another-user",
label="Invisible profile",
provider="webdav",
endpoint_url="https://invisible.example.test",
enabled=True,
credential_mode="basic",
password_encrypted="not-a-valid-encrypted-secret",
capabilities=["browse"],
policy={},
metadata_={},
)
self.session.add(invisible)
self.session.commit()
profiles = list_database_connector_profiles(
self.session,
tenant_id="tenant-1",
row_visible=lambda row: row.id == self.row.id,
)
self.assertEqual([self.row.id], [profile.id for profile in profiles])
self.assertEqual("original-password", profiles[0].password_value)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,223 @@
from __future__ import annotations
import unittest
from datetime import UTC, datetime
from unittest.mock import patch
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _smb_location, browse_connector_profile
from govoplan_files.backend.storage.connector_imports import ConnectorImportError, read_connector_file
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, connector_profiles_from_payload
from govoplan_files.backend.storage.connector_providers import connector_provider_descriptors
class FakeBody:
def __init__(self, data: bytes) -> None:
self.data = data
def read(self, _limit: int) -> bytes:
return self.data
class FakeS3Client:
def __init__(self) -> None:
self.list_objects_request: dict[str, object] | None = None
self.head_request: dict[str, object] | None = None
self.get_request: dict[str, object] | None = None
def list_objects_v2(self, **kwargs: object) -> dict[str, object]:
self.list_objects_request = dict(kwargs)
return {
"CommonPrefixes": [{"Prefix": "root/reports/"}],
"Contents": [
{
"Key": "root/report.xlsx",
"Size": 123,
"LastModified": datetime(2026, 7, 12, 10, 0, tzinfo=UTC),
"ETag": '"etag-1"',
"StorageClass": "STANDARD",
}
],
"NextContinuationToken": "next-page",
}
def list_buckets(self) -> dict[str, object]:
return {"Buckets": [{"Name": "archive", "CreationDate": datetime(2026, 7, 12, 9, 0, tzinfo=UTC)}]}
def head_object(self, **kwargs: object) -> dict[str, object]:
self.head_request = dict(kwargs)
return {
"ContentLength": 13,
"ContentType": "text/plain",
"ETag": '"etag-2"',
"VersionId": "version-1",
"ChecksumSHA256": "checksum",
}
def get_object(self, **kwargs: object) -> dict[str, object]:
self.get_request = dict(kwargs)
return {"Body": FakeBody(b"hello s3\n"), "ContentType": "text/plain", "ETag": '"etag-2"'}
def s3_profile(**overrides: object) -> ConnectorProfile:
values = {
"id": "dev-s3",
"label": "Dev S3",
"provider": "s3",
"endpoint_url": "http://127.0.0.1:9000",
"base_path": "root",
"credential_mode": "basic",
"username": "access-key",
"password_value": "secret-key",
"metadata": {"bucket": "govoplan", "region": "eu-central-1", "path_style": True},
}
values.update(overrides)
return ConnectorProfile(**values)
class ConnectorProviderTests(unittest.TestCase):
def test_smb_sdk_transport_fails_closed_before_client_creation_in_all_modes(self) -> None:
profile = ConnectorProfile(
id="smb",
label="SMB",
provider="smb",
endpoint_url="smb://files.example.test/share",
)
for allow_private, address in ((False, "93.184.216.34"), (True, "10.0.0.5")):
with self.subTest(allow_private=allow_private), patch.dict(
"os.environ",
{
"APP_ENV": "production",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": str(allow_private).lower(),
},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", (address, 445))],
), patch("govoplan_files.backend.storage.connector_browse._smbclient_module") as sdk, self.assertRaisesRegex(
ConnectorBrowseError,
"redirects/referrals.*DNS/IP pinning",
):
browse_connector_profile(profile, path="")
sdk.assert_not_called()
def test_smb_explicit_ip_still_fails_closed_because_the_sdk_may_follow_referrals(self) -> None:
profile = ConnectorProfile(id="smb", label="SMB", provider="smb", endpoint_url="smb://10.0.0.5/share")
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("10.0.0.5", 445))],
), self.assertRaisesRegex(ConnectorBrowseError, "redirects/referrals.*DNS/IP pinning"):
_smb_location(profile)
def test_smb_import_surfaces_fail_closed_policy_as_an_import_error(self) -> None:
profile = ConnectorProfile(id="smb", label="SMB", provider="smb", endpoint_url="smb://10.0.0.5/share")
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("10.0.0.5", 445))],
), patch("govoplan_files.backend.storage.connector_imports._smbclient_module") as sdk, self.assertRaisesRegex(
ConnectorImportError,
"redirects/referrals.*DNS/IP pinning",
):
read_connector_file(profile, library_id="", path="notice.txt", max_bytes=1024)
sdk.assert_not_called()
def test_provider_descriptors_include_s3_and_reserved_microsoft_providers(self) -> None:
descriptors = {descriptor.provider: descriptor for descriptor in connector_provider_descriptors()}
self.assertTrue(descriptors["s3"].implemented)
self.assertTrue(descriptors["s3"].browse_supported)
self.assertEqual("boto3", descriptors["s3"].optional_dependency)
self.assertFalse(descriptors["sharepoint"].implemented)
self.assertFalse(descriptors["onedrive"].implemented)
def test_profile_payload_accepts_new_providers_and_redacts_secret_metadata(self) -> None:
profiles = connector_profiles_from_payload(
{
"profiles": [
{
"id": "dev-s3",
"label": "Dev S3",
"provider": "s3",
"username": "access-key",
"password": "secret-key",
"metadata": {"bucket": "govoplan", "secret_access_key": "do-not-return"},
},
{"id": "sharepoint", "provider": "sharepoint"},
{"id": "onedrive", "provider": "onedrive"},
]
}
)
by_id = {profile.id: profile for profile in profiles}
self.assertEqual("s3", by_id["dev-s3"].provider)
self.assertTrue(by_id["dev-s3"].credentials_configured)
self.assertEqual({"bucket": "govoplan"}, dict(by_id["dev-s3"].metadata))
self.assertEqual("sharepoint", by_id["sharepoint"].provider)
self.assertEqual("onedrive", by_id["onedrive"].provider)
def test_s3_browse_lists_prefixes_and_objects_with_provenance_metadata(self) -> None:
client = FakeS3Client()
with patch("govoplan_files.backend.storage.connector_browse._s3_client", return_value=client):
items = browse_connector_profile(s3_profile(), path="")
self.assertEqual({"Bucket": "govoplan", "Prefix": "root/", "Delimiter": "/", "MaxKeys": 1000}, client.list_objects_request)
self.assertEqual(["folder", "file"], [item.kind for item in items])
self.assertEqual("reports", items[0].path)
self.assertEqual("report.xlsx", items[1].path)
self.assertEqual("govoplan:root/report.xlsx", items[1].external_id)
self.assertEqual("root/report.xlsx", items[1].metadata["key"])
self.assertEqual("next-page", items[1].metadata["next_continuation_token"])
def test_s3_browse_lists_buckets_when_profile_has_no_bucket(self) -> None:
client = FakeS3Client()
with patch("govoplan_files.backend.storage.connector_browse._s3_client", return_value=client):
items = browse_connector_profile(s3_profile(metadata={}), path="")
self.assertEqual(["archive"], [item.path for item in items])
def test_s3_sdk_transport_fails_closed_before_client_creation_in_private_mode(self) -> None:
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("127.0.0.1", 9000))],
), patch("govoplan_files.backend.storage.connector_browse.import_module") as importer, self.assertRaisesRegex(
ConnectorBrowseError,
"until that transport supports.*DNS/IP pinning",
):
browse_connector_profile(s3_profile(), path="")
importer.assert_not_called()
def test_s3_sdk_endpoint_discovery_fails_closed(self) -> None:
with patch("govoplan_files.backend.storage.connector_browse.import_module") as importer, self.assertRaisesRegex(
ConnectorBrowseError,
"endpoint discovery.*cannot guarantee.*DNS/IP pinning",
):
browse_connector_profile(s3_profile(endpoint_url=None), path="")
importer.assert_not_called()
def test_s3_import_downloads_object_and_preserves_remote_identity(self) -> None:
client = FakeS3Client()
with patch("govoplan_files.backend.storage.connector_imports._s3_client", return_value=client):
downloaded = read_connector_file(s3_profile(), library_id="", path="report.txt", max_bytes=1024)
self.assertEqual({"Bucket": "govoplan", "Key": "root/report.txt"}, client.head_request)
self.assertEqual({"Bucket": "govoplan", "Key": "root/report.txt", "VersionId": "version-1"}, client.get_request)
self.assertEqual("report.txt", downloaded.filename)
self.assertEqual(b"hello s3\n", downloaded.data)
self.assertEqual("version-1", downloaded.revision)
self.assertEqual("govoplan:root/report.txt", downloaded.external_id)
self.assertEqual("s3://govoplan/root/report.txt", downloaded.external_url)
self.assertEqual("checksum", downloaded.metadata["checksum_sha256"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,265 @@
from __future__ import annotations
import unittest
from dataclasses import replace
from unittest.mock import patch
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
from govoplan_files.backend.storage.connector_visibility import (
connector_profile_usable_for_import,
visible_connector_profiles_for_actor,
)
def _profile(
profile_id: str,
*,
scope_type: str = "system",
scope_id: str | None = None,
provider: str = "webdav",
enabled: bool = True,
) -> ConnectorProfile:
return ConnectorProfile(
id=profile_id,
label=profile_id,
provider=provider,
scope_type=scope_type,
scope_id=scope_id,
endpoint_url=f"https://{profile_id}.example.invalid",
enabled=enabled,
credential_mode="anonymous",
)
class ConnectorVisibilityTests(unittest.TestCase):
@patch(
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings"
)
@patch(
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
)
def test_profiles_are_filtered_to_actor_scopes_and_database_definition_wins(
self,
database_profiles,
configured_profiles,
_policy_sources,
) -> None:
profiles = [
_profile("system"),
_profile("tenant", scope_type="tenant", scope_id="tenant-1"),
_profile("other-tenant", scope_type="tenant", scope_id="tenant-2"),
_profile("user", scope_type="user", scope_id="user-1"),
_profile("other-user", scope_type="user", scope_id="user-2"),
_profile("group", scope_type="group", scope_id="group-1"),
_profile("campaign", scope_type="campaign", scope_id="campaign-1"),
_profile("disabled", enabled=False),
_profile("duplicate", provider="webdav"),
]
database_profiles.return_value = (profiles, {profile.id for profile in profiles})
configured_profiles.return_value = [_profile("duplicate", provider="nextcloud")]
visible = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids={"group-1"},
settings=object(),
campaign_visible=lambda campaign_id: campaign_id == "campaign-1",
)
self.assertEqual(
{"system", "tenant", "user", "group", "campaign", "duplicate"},
{profile.id for profile in visible},
)
duplicate = next(profile for profile in visible if profile.id == "duplicate")
self.assertEqual("webdav", duplicate.provider)
@patch(
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
)
def test_campaign_and_admin_visibility_remain_explicit(
self,
database_profiles,
_configured_profiles,
_policy_sources,
) -> None:
profiles = [
_profile("campaign-a", scope_type="campaign", scope_id="campaign-a"),
_profile("campaign-b", scope_type="campaign", scope_id="campaign-b"),
_profile("other-user", scope_type="user", scope_id="user-2"),
]
database_profiles.return_value = (profiles, {profile.id for profile in profiles})
campaign_only = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids=(),
settings=None,
campaign_id="campaign-a",
campaign_visible=lambda _campaign_id: True,
)
self.assertEqual(["campaign-a"], [profile.id for profile in campaign_only])
admin = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids=(),
settings=None,
campaign_visible=None,
include_admin_scopes=True,
)
self.assertEqual(
{"campaign-a", "campaign-b", "other-user"},
{profile.id for profile in admin},
)
@patch(
"govoplan_files.backend.storage.connector_visibility.connector_profiles_from_settings",
return_value=[],
)
@patch(
"govoplan_files.backend.storage.connector_visibility.select_database_connector_profiles"
)
@patch(
"govoplan_files.backend.storage.connector_visibility.effective_connector_policy_sources"
)
def test_effective_policy_is_attached_without_mutating_profile(
self,
policy_sources,
database_profiles,
_configured_profiles,
) -> None:
profile = _profile("profile")
source = ConnectorPolicySource(
scope_type="system",
label="System",
policy={"allow": {"providers": ["webdav"]}},
)
database_profiles.return_value = ([profile], {profile.id})
policy_sources.return_value = [source]
visible = visible_connector_profiles_for_actor(
object(), # type: ignore[arg-type]
tenant_id="tenant-1",
user_id="user-1",
group_ids=(),
settings=None,
)
self.assertEqual((), profile.policy_sources)
self.assertEqual((source,), visible[0].policy_sources)
def test_import_usability_requires_safe_provider_credentials_endpoint_and_identity_policy(
self,
) -> None:
self.assertTrue(connector_profile_usable_for_import(_profile("webdav")))
self.assertFalse(
connector_profile_usable_for_import(_profile("s3", provider="s3"))
)
self.assertFalse(
connector_profile_usable_for_import(_profile("smb", provider="smb"))
)
self.assertFalse(
connector_profile_usable_for_import(
_profile("sharepoint", provider="sharepoint")
)
)
self.assertFalse(
connector_profile_usable_for_import(
ConnectorProfile(
id="no-endpoint",
label="No endpoint",
provider="webdav",
credential_mode="anonymous",
)
)
)
self.assertFalse(
connector_profile_usable_for_import(
ConnectorProfile(
id="no-credentials",
label="No credentials",
provider="webdav",
endpoint_url="https://files.example.invalid",
credential_mode="basic",
)
)
)
self.assertTrue(
connector_profile_usable_for_import(
ConnectorProfile(
id="metadata-endpoint",
label="Metadata endpoint",
provider="webdav",
credential_mode="anonymous",
metadata={"webdav_endpoint_url": "https://files.example.invalid"},
)
)
)
self.assertFalse(
connector_profile_usable_for_import(
ConnectorProfile(
id="unresolved-secret-reference",
label="Unresolved secret reference",
provider="webdav",
endpoint_url="https://files.example.invalid",
credential_mode="basic",
secret_ref="vault://files/webdav",
)
)
)
denied = replace(
_profile("denied"),
policy_sources=(
ConnectorPolicySource(
scope_type="system",
label="System",
policy={"deny": {"providers": ["webdav"]}},
),
),
)
self.assertFalse(connector_profile_usable_for_import(denied))
path_limited = replace(
_profile("path-limited"),
policy_sources=(
ConnectorPolicySource(
scope_type="system",
label="System",
policy={"allow": {"external_paths": ["/approved/*"]}},
),
),
)
self.assertFalse(connector_profile_usable_for_import(path_limited))
endpoint_allowed = replace(
_profile("endpoint-allowed"),
policy_sources=(
ConnectorPolicySource(
scope_type="system",
label="System",
policy={"allow": {"external_urls": ["https://*.example.invalid"]}},
),
),
)
self.assertTrue(connector_profile_usable_for_import(endpoint_allowed))
if __name__ == "__main__":
unittest.main()

241
tests/test_documentation.py Normal file
View File

@@ -0,0 +1,241 @@
from __future__ import annotations
import unittest
from types import SimpleNamespace
from unittest.mock import patch
from sqlalchemy.orm import Session
from govoplan_core.core.modules import DocumentationContext
from govoplan_files.backend.documentation import documentation_topics
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
class _Principal:
tenant_id = "tenant-1"
user = SimpleNamespace(id="user-1")
group_ids = frozenset({"group-1"})
def __init__(self, scopes: set[str]) -> None:
self.scopes = frozenset(scopes)
def has(self, scope: str) -> bool:
return scope in self.scopes
class FilesRuntimeDocumentationTests(unittest.TestCase):
def setUp(self) -> None:
self.session = Session()
def tearDown(self) -> None:
self.session.close()
def context(
self,
scopes: set[str],
*,
settings: object | None = None,
documentation_type: str = "user",
) -> DocumentationContext:
return DocumentationContext(
registry=object(),
principal=_Principal(scopes),
settings=settings,
session=self.session,
documentation_type=documentation_type, # type: ignore[arg-type]
)
def test_upload_tasks_state_exact_current_safe_limits(self) -> None:
settings = SimpleNamespace(
file_upload_max_bytes=7 * 1024 * 1024,
file_upload_zip_max_bytes=19 * 1024 * 1024,
)
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[],
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context(
{"files:file:read", "files:file:upload"}, settings=settings
)
)
}
upload = topics["files.workflow.upload-managed-files"]
self.assertIn("7 MiB (7,340,032 bytes)", upload.body)
self.assertEqual(["files.list"], upload.metadata["help_contexts"])
self.assertEqual(
{"files:file:read", "files:file:upload"},
set(upload.conditions[0].required_scopes),
)
archive = topics["files.workflow.upload-and-unpack-zip"]
self.assertIn("19 MiB (19,922,944 bytes)", archive.body)
self.assertIn("7 MiB (7,340,032 bytes)", archive.body)
self.assertIn("1,000", archive.body)
self.assertIn("Actual extracted bytes are counted", archive.body)
def test_connector_task_requires_authority_and_a_visible_usable_profile(
self,
) -> None:
usable = ConnectorProfile(
id="private-profile-id",
label="Private profile label",
provider="webdav",
scope_type="tenant",
scope_id="tenant-1",
endpoint_url="https://private.example.invalid/secret-root",
base_path="/secret-root",
credential_mode="anonymous",
)
context = self.context({"files:file:read", "files:file:upload"})
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[usable],
) as visible:
topics = {topic.id: topic for topic in documentation_topics(context)}
self.assertIn("files.workflow.import-managed-snapshot", topics)
self.assertNotIn("files.connector-import-unavailable", topics)
task = topics["files.workflow.import-managed-snapshot"]
self.assertEqual("configured", task.layer)
self.assertEqual(
["files.list", "files.connector-import"], task.metadata["help_contexts"]
)
self.assertIn("re-authorized", task.body)
self.assertNotIn("private-profile-id", repr(task))
self.assertNotIn("private.example.invalid", repr(task))
self.assertNotIn("secret-root", repr(task))
visible.assert_called_once()
self.assertEqual(("group-1",), tuple(visible.call_args.kwargs["group_ids"]))
without_authority = {
topic.id: topic
for topic in documentation_topics(self.context({"files:file:read"}))
}
self.assertNotIn("files.workflow.import-managed-snapshot", without_authority)
self.assertIn(
"both permission to view Files and permission to upload",
without_authority["files.connector-import-unavailable"].body,
)
def test_connector_limitation_is_precise_but_never_exposes_profile_internals(
self,
) -> None:
blocked = ConnectorProfile(
id="sensitive-profile-id",
label="Sensitive label",
provider="s3",
scope_type="tenant",
scope_id="tenant-1",
endpoint_url="https://internal.example.invalid/private",
base_path="/classified",
credential_mode="basic",
password_value="credential-secret",
)
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[blocked],
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context({"files:file:read", "files:file:upload"})
)
}
self.assertNotIn("files.workflow.import-managed-snapshot", topics)
limitation = topics["files.connector-import-unavailable"]
self.assertEqual("available", limitation.layer)
self.assertIn("pinning-safe provider", limitation.body)
payload = repr(limitation)
for secret in (
"sensitive-profile-id",
"Sensitive label",
"internal.example.invalid",
"classified",
"credential-secret",
):
self.assertNotIn(secret, payload)
def test_files_admin_connector_groups_do_not_widen_campaign_membership(self) -> None:
usable = ConnectorProfile(
id="group-profile",
label="Group profile",
provider="webdav",
scope_type="group",
scope_id="admin-visible-group",
endpoint_url="https://files.example.invalid",
credential_mode="anonymous",
)
def groups_for_actor(_session, *, include_admin_groups, **_kwargs):
return ["member-group", "admin-visible-group"] if include_admin_groups else ["member-group"]
with (
patch(
"govoplan_files.backend.documentation.user_group_ids",
side_effect=groups_for_actor,
),
patch(
"govoplan_files.backend.documentation._campaign_visibility",
return_value=None,
) as campaign_visibility,
patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
return_value=[usable],
) as visible,
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context(
{
"files:file:read",
"files:file:upload",
"files:file:admin",
}
)
)
}
self.assertIn("files.workflow.import-managed-snapshot", topics)
self.assertEqual(
("member-group", "admin-visible-group"),
tuple(visible.call_args.kwargs["group_ids"]),
)
self.assertEqual(
("member-group",),
tuple(campaign_visibility.call_args.kwargs["group_ids"]),
)
def test_connector_evaluation_errors_fail_closed_without_error_details(
self,
) -> None:
with patch(
"govoplan_files.backend.documentation.visible_connector_profiles_for_actor",
side_effect=RuntimeError("private endpoint /root credential-id"),
):
topics = {
topic.id: topic
for topic in documentation_topics(
self.context({"files:file:read", "files:file:upload"})
)
}
self.assertNotIn("files.workflow.import-managed-snapshot", topics)
limitation = topics["files.connector-import-unavailable"]
self.assertIn("could not be safely evaluated", limitation.body)
self.assertNotIn("private endpoint", repr(limitation))
self.assertNotIn("credential-id", repr(limitation))
def test_runtime_provider_only_emits_user_documentation(self) -> None:
self.assertEqual(
(), documentation_topics(self.context(set(), documentation_type="admin"))
)
if __name__ == "__main__":
unittest.main()

132
tests/test_http_client.py Normal file
View File

@@ -0,0 +1,132 @@
from __future__ import annotations
import unittest
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from unittest.mock import MagicMock, patch
import httpcore
from govoplan_core.security.outbound_http import outbound_http_policy, validate_outbound_http_url
from govoplan_files.backend.storage.http_client import (
ConnectorHttpError,
_OutboundPolicyNetworkBackend,
_SocketNetworkStream,
request_connector_bytes,
)
class _TestHandler(BaseHTTPRequestHandler):
def do_GET(self) -> None: # noqa: N802 - stdlib handler contract
body = b"connector-ok"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, _format: str, *_args: object) -> None:
return
class ConnectorHttpClientTests(unittest.TestCase):
def test_public_transport_adapter_performs_a_real_http_request(self) -> None:
server = ThreadingHTTPServer(("127.0.0.1", 0), _TestHandler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
with patch.dict(
"os.environ",
{"APP_ENV": "test", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
clear=False,
):
result = request_connector_bytes(
"GET",
f"http://127.0.0.1:{server.server_port}/object",
)
finally:
server.shutdown()
server.server_close()
thread.join(timeout=2)
self.assertEqual(200, result.status_code)
self.assertEqual(b"connector-ok", result.content)
def test_connector_responses_are_streamed_without_redirects(self) -> None:
response = MagicMock()
response.status_code = 200
response.headers = {"content-length": "6"}
response.iter_bytes.return_value = iter((b"abc", b"def"))
context = MagicMock()
context.__enter__.return_value = response
with patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("93.184.216.34", 443))],
), patch("govoplan_files.backend.storage.http_client._stream_connector_request", return_value=context):
result = request_connector_bytes("GET", "https://example.test/object", max_bytes=10)
self.assertEqual(b"abcdef", result.content)
def test_connector_response_limit_is_enforced_while_streaming(self) -> None:
response = MagicMock()
response.status_code = 200
response.headers = {}
response.iter_bytes.return_value = iter((b"12345", b"67890", b"!"))
context = MagicMock()
context.__enter__.return_value = response
with patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("93.184.216.34", 443))],
), patch(
"govoplan_files.backend.storage.http_client._stream_connector_request",
return_value=context,
), self.assertRaisesRegex(
ConnectorHttpError,
"configured limit",
):
request_connector_bytes("GET", "https://example.test/object", max_bytes=10)
def test_private_destination_is_blocked_before_http_connection(self) -> None:
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("10.0.0.5", 443))],
), patch("govoplan_files.backend.storage.http_client._stream_connector_request") as stream, self.assertRaisesRegex(
ConnectorHttpError,
"non-public network",
):
request_connector_bytes("GET", "https://connector.example.test/object")
stream.assert_not_called()
def test_connection_backend_rejects_private_rebinding_before_socket_open(self) -> None:
policy = outbound_http_policy({"APP_ENV": "production"})
public = [(2, 1, 6, "", ("93.184.216.34", 443))]
private = [(2, 1, 6, "", ("127.0.0.1", 443))]
with patch.dict(
"os.environ",
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
side_effect=(public, private),
), patch("govoplan_core.security.outbound_http.socket.socket") as socket_factory:
validate_outbound_http_url("https://connector.example.test/object", policy=policy)
with self.assertRaises(httpcore.ConnectError):
_OutboundPolicyNetworkBackend().connect_tcp("connector.example.test", 443)
socket_factory.assert_not_called()
def test_network_backend_returns_public_network_stream_adapter(self) -> None:
sock = MagicMock()
with patch(
"govoplan_files.backend.storage.http_client.create_outbound_connection",
return_value=sock,
):
stream = _OutboundPolicyNetworkBackend().connect_tcp("connector.example.test", 443)
self.assertIsInstance(stream, _SocketNetworkStream)
self.assertIs(stream.get_extra_info("socket"), sock)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,210 @@
from __future__ import annotations
import unittest
STATIC_TOPIC_IDS = {
"files.workflow.organize-managed-files",
"files.workflow.find-and-download-files",
"files.workflow.share-managed-files",
"files.workflow.delete-managed-files",
"files.governed-connectors-and-provenance",
"files.reference.integrity-recovery-and-fail-closed-transports",
"files.reference.snapshot-provenance-and-capabilities",
"files.assurance.process-and-release-readiness",
}
RUNTIME_TOPIC_IDS = {
"files.workflow.upload-managed-files",
"files.workflow.upload-and-unpack-zip",
"files.workflow.import-managed-snapshot",
"files.connector-import-unavailable",
}
HANDBOOK_HREF = "govoplan-files/docs/FILES_HANDBOOK.md"
class FilesManifestDocumentationTests(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
from govoplan_files.backend.manifest import manifest
cls.manifest = manifest
cls.topics = {topic.id: topic for topic in manifest.documentation}
def topic(self, topic_id: str):
return self.topics[topic_id]
def test_static_topics_have_role_scope_module_and_link_contracts(self) -> None:
self.assertEqual(STATIC_TOPIC_IDS, set(self.topics))
self.assertEqual(
{"admin", "user"},
{
item
for topic in self.topics.values()
for item in topic.documentation_types
},
)
for topic in self.topics.values():
with self.subTest(topic=topic.id):
self.assertTrue(topic.audience)
self.assertTrue(topic.conditions)
self.assertTrue(
any(
"files" in condition.required_modules
for condition in topic.conditions
)
)
self.assertTrue(
any(
condition.any_scopes or condition.required_scopes
for condition in topic.conditions
)
)
self.assertIn("runtime", {link.kind for link in topic.links})
self.assertIn(
HANDBOOK_HREF,
{link.href for link in topic.links if link.kind == "repository"},
)
self.assertIn(
"documentation_topics",
{provider.__name__ for provider in self.manifest.documentation_providers},
)
def test_user_tasks_are_independently_authorized_and_contextual(self) -> None:
expected_scopes = {
"files.workflow.organize-managed-files": {
"files:file:read",
"files:file:organize",
},
"files.workflow.find-and-download-files": {
"files:file:read",
"files:file:download",
},
"files.workflow.share-managed-files": {
"files:file:read",
"files:file:share",
},
"files.workflow.delete-managed-files": {
"files:file:read",
"files:file:delete",
},
}
for topic_id, scopes in expected_scopes.items():
with self.subTest(topic=topic_id):
topic = self.topic(topic_id)
self.assertEqual(("user",), topic.documentation_types)
self.assertEqual("workflow", topic.metadata["kind"])
self.assertEqual(scopes, set(topic.conditions[0].required_scopes))
self.assertEqual(["files.list"], topic.metadata["help_contexts"])
for key in ("prerequisites", "steps", "outcome", "verification"):
self.assertTrue(topic.metadata[key])
def test_share_and_delete_tasks_state_current_boundaries(self) -> None:
share = self.topic("files.workflow.share-managed-files")
self.assertEqual("available", share.layer)
self.assertIn("does not yet provide a general share editor", share.body)
self.assertIn("no share-revocation route", share.body)
self.assertIn(
"/api/v1/files/{file_id}/shares", {link.href for link in share.links}
)
delete = self.topic("files.workflow.delete-managed-files")
self.assertIn("Soft-delete", delete.summary)
self.assertIn("no self-service restore or hard-purge", delete.body)
self.assertTrue(
any("not a hard purge" in item for item in delete.metadata["limitations"])
)
def test_admin_topic_covers_policy_redaction_and_atomic_credential_deletion(
self,
) -> None:
topic = self.topic("files.governed-connectors-and-provenance")
self.assertEqual(("admin",), topic.documentation_types)
self.assertEqual("reference", topic.metadata["kind"])
self.assertEqual("File connections", topic.metadata["screen"])
self.assertTrue(topic.metadata["section"])
self.assertIn("deny rules win", topic.body)
self.assertIn("redact secret values", topic.body)
self.assertIn("same transaction", topic.body)
self.assertTrue(
any(
"non-owned external references" in item
for item in topic.metadata["security_invariants"]
)
)
self.assertIn(
"/api/v1/files/connectors/policies/tenant",
{link.href for link in topic.links},
)
self.assertIn(
"/api/v1/files/connectors/credentials", {link.href for link in topic.links}
)
def test_operator_topic_covers_recovery_and_fail_closed_s3_smb(self) -> None:
topic = self.topic(
"files.reference.integrity-recovery-and-fail-closed-transports"
)
self.assertEqual(("admin",), topic.documentation_types)
self.assertEqual("reference", topic.metadata["kind"])
self.assertIn("S3", topic.body)
self.assertIn("SMB", topic.body)
self.assertIn("fail closed", topic.body)
self.assertIn("DFS referrals", topic.body)
self.assertIn("does not remove backend blob objects", topic.body)
self.assertIn("MASTER_KEY_B64", topic.metadata["recovery_unit"])
self.assertTrue(topic.metadata["verification"])
self.assertIn(
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", topic.configuration_keys
)
def test_integrator_topic_exposes_capability_and_provenance_boundary(self) -> None:
topic = self.topic("files.reference.snapshot-provenance-and-capabilities")
self.assertEqual(("admin", "user"), topic.documentation_types)
self.assertEqual("reference", topic.metadata["kind"])
self.assertEqual(
["files.access@0.1.6", "files.campaign_attachments@0.1.6"],
topic.metadata["provided_interfaces"],
)
self.assertIn("revision", topic.metadata["provenance_fields"])
self.assertIn("exact asset, version, blob, checksum", topic.body)
self.assertIn("collaboration", topic.body)
def test_process_and_release_assurance_exposes_gates_evidence_and_limits(
self,
) -> None:
topic = self.topic("files.assurance.process-and-release-readiness")
self.assertEqual(("admin", "user"), topic.documentation_types)
self.assertEqual("workflow", topic.metadata["kind"])
self.assertEqual(["files.list"], topic.metadata["help_contexts"])
self.assertIn("process_owner", topic.audience)
self.assertIn("release_manager", topic.audience)
self.assertIn("versions align", topic.body)
self.assertIn("no general share-management UI or share revocation", topic.body)
self.assertIn("no enforced retention or legal hold", topic.body)
for key in ("prerequisites", "steps", "outcome", "verification"):
self.assertTrue(topic.metadata[key])
self.assertTrue(
any("meta-repository security" in step for step in topic.metadata["steps"])
)
self.assertTrue(any("fails closed" in step for step in topic.metadata["steps"]))
self.assertTrue(any("SHA-256" in step for step in topic.metadata["steps"]))
def test_internal_related_topic_ids_resolve(self) -> None:
known_ids = STATIC_TOPIC_IDS | RUNTIME_TOPIC_IDS
for topic in self.topics.values():
for related_id in topic.metadata.get("related_topic_ids", []):
if related_id.startswith("files."):
self.assertIn(
related_id,
known_ids,
f"{topic.id} refers to missing topic {related_id}",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,36 @@
from __future__ import annotations
import unittest
from govoplan_files.backend.router import router
class FilesRouterContractTests(unittest.TestCase):
def test_connector_routes_keep_existing_api_paths(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
expected = {
(("GET",), "/files/connectors/providers"),
(("GET",), "/files/connectors/settings/delta"),
(("GET",), "/files/connectors/profiles"),
(("POST",), "/files/connectors/profiles"),
(("GET",), "/files/connectors/profiles/{profile_id}/browse"),
(("POST",), "/files/connectors/profiles/{profile_id}/import"),
(("POST",), "/files/connectors/profiles/{profile_id}/sync"),
(("GET",), "/files/connectors/credentials"),
(("POST",), "/files/connectors/credentials"),
(("GET",), "/files/connector-spaces"),
(("POST",), "/files/connector-spaces"),
}
self.assertTrue(expected.issubset(routes))
def test_bulk_organize_routes_keep_existing_api_paths(self) -> None:
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
self.assertIn((("POST",), "/files/bulk-rename"), routes)
self.assertIn((("POST",), "/files/transfer"), routes)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,78 @@
from __future__ import annotations
import io
import unittest
from unittest.mock import MagicMock, PropertyMock, patch
from govoplan_files.backend.storage.backends import S3StorageBackend, StorageBackendError
def _backend() -> S3StorageBackend:
return S3StorageBackend(
bucket="files",
endpoint_url="https://objects.example.test",
region_name="test",
access_key_id="access",
secret_access_key="secret",
)
class S3StorageBackendTests(unittest.TestCase):
def test_sdk_transport_fails_closed_in_public_and_private_modes(self) -> None:
for allow_private, address in ((False, "93.184.216.34"), (True, "10.0.0.5")):
with self.subTest(allow_private=allow_private), patch.dict(
"os.environ",
{
"APP_ENV": "production",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": str(allow_private).lower(),
},
), patch(
"govoplan_core.security.outbound_http.socket.getaddrinfo",
return_value=[(2, 1, 6, "", (address, 443))],
), self.assertRaisesRegex(StorageBackendError, "until that transport supports.*DNS/IP pinning"):
_backend().client
def test_get_bytes_rejects_declared_oversize_object_without_reading(self) -> None:
body = MagicMock()
client = MagicMock()
client.get_object.return_value = {"ContentLength": 6, "Body": body}
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "5"}), patch.object(
S3StorageBackend,
"client",
new_callable=PropertyMock,
return_value=client,
), self.assertRaisesRegex(StorageBackendError, "deployment limit"):
_backend().get_bytes("large.bin")
body.read.assert_not_called()
body.close.assert_called_once_with()
def test_iter_bytes_rejects_undeclared_oversize_object(self) -> None:
client = MagicMock()
client.get_object.return_value = {"Body": io.BytesIO(b"123456")}
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "5"}), patch.object(
S3StorageBackend,
"client",
new_callable=PropertyMock,
return_value=client,
), self.assertRaisesRegex(StorageBackendError, "deployment limit"):
list(_backend().iter_bytes("large.bin", chunk_size=3))
def test_iter_bytes_closes_streaming_body_when_consumer_stops_early(self) -> None:
body = MagicMock()
body.read.side_effect = (b"123", b"456", b"")
client = MagicMock()
client.get_object.return_value = {"ContentLength": 6, "Body": body}
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "10"}), patch.object(
S3StorageBackend,
"client",
new_callable=PropertyMock,
return_value=client,
):
chunks = _backend().iter_bytes("object.bin", chunk_size=3)
self.assertEqual(b"123", next(chunks))
chunks.close()
body.close.assert_called_once_with()
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,67 @@
from __future__ import annotations
import unittest
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError
from govoplan_files.backend.storage.transfers import _normalize_rename_request, _normalize_transfer_request
class TransferHelperTests(unittest.TestCase):
def test_transfer_normalization_deduplicates_folder_paths_and_conflicts(self) -> None:
normalized = _normalize_transfer_request(
operation=" COPY ",
source_owner_type=" USER ",
target_owner_type=" GROUP ",
folder_paths=["Reports", "Reports/2026", "", "./Archive"],
target_folder=" Target ",
conflict_strategy="rename",
conflict_resolutions=[FileConflictResolution(target_path="Target/a.txt", action="skip")],
)
self.assertEqual("copy", normalized.operation)
self.assertEqual("user", normalized.source_owner_type)
self.assertEqual("group", normalized.target_owner_type)
self.assertEqual(["Reports", "Reports/2026", "Archive"], normalized.source_folder_paths)
self.assertEqual("Target", normalized.target_folder)
self.assertEqual("rename", normalized.conflict_strategy)
self.assertEqual("skip", normalized.conflict_resolution_map["Target/a.txt"].action)
def test_transfer_normalization_rejects_unknown_operation(self) -> None:
with self.assertRaisesRegex(FileStorageError, "Unsupported transfer operation"):
_normalize_transfer_request(
operation="link",
source_owner_type="user",
target_owner_type="group",
folder_paths=[],
target_folder="",
conflict_strategy="reject",
conflict_resolutions=None,
)
def test_rename_normalization_collapses_nested_folder_roots(self) -> None:
normalized = _normalize_rename_request(
file_ids=["file-1", "file-1", "file-2"],
folder_paths=["Reports", "Reports/2026", "Archive"],
owner_type=" USER ",
owner_id="owner-1",
mode=" prefix ",
)
self.assertEqual("prefix", normalized.mode)
self.assertEqual(["file-1", "file-2"], normalized.selected_file_ids)
self.assertEqual(["Archive", "Reports"], normalized.selected_folder_roots)
self.assertEqual("user", normalized.owner_type)
def test_rename_normalization_rejects_invalid_direct_multi_select(self) -> None:
with self.assertRaisesRegex(FileStorageError, "Direct rename requires exactly one selected item"):
_normalize_rename_request(
file_ids=["file-1", "file-2"],
folder_paths=[],
owner_type="user",
owner_id="owner-1",
mode="direct",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -1,6 +1,6 @@
{ {
"name": "@govoplan/files-webui", "name": "@govoplan/files-webui",
"version": "0.1.8", "version": "0.1.9",
"private": true, "private": true,
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
@@ -13,6 +13,9 @@
}, },
"./styles/file-manager.css": "./src/styles/file-manager.css" "./styles/file-manager.css": "./src/styles/file-manager.css"
}, },
"scripts": {
"test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs"
},
"peerDependencies": { "peerDependencies": {
"@vitejs/plugin-react": "^4.3.4", "@vitejs/plugin-react": "^4.3.4",
"vite": "^6.0.6", "vite": "^6.0.6",
@@ -21,7 +24,7 @@
"react-dom": "^19.0.0", "react-dom": "^19.0.0",
"react-router-dom": "^7.1.1", "react-router-dom": "^7.1.1",
"lucide-react": "^1.23.0", "lucide-react": "^1.23.0",
"@govoplan/core-webui": "^0.1.8" "@govoplan/core-webui": "^0.1.9"
}, },
"peerDependenciesMeta": { "peerDependenciesMeta": {
"@govoplan/core-webui": { "@govoplan/core-webui": {

View File

@@ -0,0 +1,35 @@
import { readFileSync } from "node:fs";
const source = readFileSync(new URL("../src/features/files/FilesPage.tsx", import.meta.url), "utf8");
const normalized = source.replace(/\s+/g, " ");
function assertIncludes(fragment, message) {
if (!normalized.includes(fragment.replace(/\s+/g, " "))) throw new Error(message);
}
assertIncludes(
"const target = options.target ?? currentActionTarget();",
"uploads must prefer the explicit target supplied by the initiating interaction"
);
assertIncludes(
"const targetSpace = target ? findSpace(target.spaceId) : null;",
"the upload owner must be resolved from the selected target"
);
assertIncludes(
"owner_type: targetSpace.owner_type, owner_id: targetSpace.owner_id, path: target.folderPath,",
"the upload request must use the selected target owner and folder"
);
assertIncludes(
"async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) { await handleFilesUpload(fileList, { target }); }",
"external drops must pass their concrete target without asynchronous dialog-state mutation"
);
assertIncludes(
"await uploadExternalFilesToTarget(event.dataTransfer.files, target);",
"drop handling must forward the target on which the files were dropped"
);
assertIncludes(
"onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })}",
"the dialog picker/drop zone must snapshot its selected target for upload"
);
console.log("Files upload target routing structure is intact.");

View File

@@ -1,4 +1,10 @@
import { ApiError, apiFetch, apiUrl, authHeaders, csrfToken, type ApiSettings, type DeltaDeletedItem, type FilesManagedFileLinkTarget } from "@govoplan/core-webui"; import { ApiError, apiFetch, apiUrl, authHeaders, csrfToken, type ApiSettings, type DeltaDeletedItem, type FilesManagedFileLinkTarget } from "@govoplan/core-webui";
export { fetchResourceAccessExplanation } from "@govoplan/core-webui";
export type {
AccessDecisionProvenanceItem,
ResourceAccessExplanationUser as AccessExplanationUser,
ResourceAccessExplanationResponse
} from "@govoplan/core-webui";
export type FileSpace = { export type FileSpace = {
id: string; id: string;
@@ -116,8 +122,6 @@ export type FileConnectorCredentialsPayload = {
username?: string | null; username?: string | null;
password?: string | null; password?: string | null;
token?: string | null; token?: string | null;
password_env?: string | null;
token_env?: string | null;
secret_ref?: string | null; secret_ref?: string | null;
}; };
@@ -150,7 +154,7 @@ export type FileConnectorDiscoveryResponse = {
export type FileConnectorProfilePayload = { export type FileConnectorProfilePayload = {
id: string; id: string;
label: string; label: string;
provider: "seafile" | "nextcloud" | "webdav" | "smb" | "nfs" | "dms" | "generic"; provider: "seafile" | "nextcloud" | "webdav" | "smb" | "s3" | "sharepoint" | "onedrive" | "nfs" | "dms" | "generic";
endpoint_url?: string | null; endpoint_url?: string | null;
base_path?: string | null; base_path?: string | null;
enabled?: boolean; enabled?: boolean;
@@ -172,7 +176,7 @@ export type FileConnectorProfileUpdatePayload = Partial<Omit<FileConnectorProfil
export type FileConnectorCredentialPayload = { export type FileConnectorCredentialPayload = {
id: string; id: string;
label: string; label: string;
provider?: "seafile" | "nextcloud" | "webdav" | "smb" | "nfs" | "dms" | "generic" | null; provider?: "seafile" | "nextcloud" | "webdav" | "smb" | "s3" | "sharepoint" | "onedrive" | "nfs" | "dms" | "generic" | null;
enabled?: boolean; enabled?: boolean;
scope_type?: "system" | "tenant" | "user" | "group" | "campaign"; scope_type?: "system" | "tenant" | "user" | "group" | "campaign";
scope_id?: string | null; scope_id?: string | null;
@@ -206,6 +210,8 @@ export type FileConnectorBrowseResponse = {
path: string; path: string;
library_id?: string | null; library_id?: string | null;
read_only: boolean; read_only: boolean;
next_continuation_token?: string | null;
has_more: boolean;
decision: FileConnectorPolicyDecision; decision: FileConnectorPolicyDecision;
items: FileConnectorBrowseItem[]; items: FileConnectorBrowseItem[];
}; };
@@ -559,44 +565,6 @@ export function bulkDeleteFiles(settings: ApiSettings, fileIds: string[]): Promi
export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;}; export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;};
export type AccessExplanationUser = {
id: string;
account_id?: string | null;
email?: string | null;
display_name?: string | null;
};
export type AccessDecisionProvenanceItem = {
kind: string;
id?: string | null;
label?: string | null;
tenant_id?: string | null;
source?: string | null;
details?: Record<string, unknown>;
};
export type ResourceAccessExplanationResponse = {
user: AccessExplanationUser;
resource_type: string;
resource_id: string;
action: string;
provenance: AccessDecisionProvenanceItem[];
};
export function fetchResourceAccessExplanation(
settings: ApiSettings,
options: {userId: string;resourceType: string;resourceId: string;action: string;tenantId?: string | null;})
: Promise<ResourceAccessExplanationResponse> {
const params = new URLSearchParams({
user_id: options.userId,
resource_type: options.resourceType,
resource_id: options.resourceId,
action: options.action
});
if (options.tenantId) params.set("tenant_id", options.tenantId);
return apiFetch<ResourceAccessExplanationResponse>(settings, `/api/v1/admin/access/resource-explanation?${params.toString()}`);
}
export function virtualFolderResourceId(options: {tenantId: string;ownerType: "user" | "group";ownerId: string;path: string;}): string { export function virtualFolderResourceId(options: {tenantId: string;ownerType: "user" | "group";ownerId: string;path: string;}): string {
return `virtual-folder:v1:${options.tenantId}:${options.ownerType}:${options.ownerId}:${base64Url(options.path.replace(/\\/g, "/").replace(/^\/+|\/+$/g, ""))}`; return `virtual-folder:v1:${options.tenantId}:${options.ownerType}:${options.ownerId}:${base64Url(options.path.replace(/\\/g, "/").replace(/^\/+|\/+$/g, ""))}`;
} }
@@ -808,11 +776,12 @@ export function deactivateFileConnectorProfile(settings: ApiSettings, profileId:
export function browseFileConnectorProfile( export function browseFileConnectorProfile(
settings: ApiSettings, settings: ApiSettings,
profileId: string, profileId: string,
params: {path?: string;library_id?: string;campaign_id?: string;} = {}) params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {})
: Promise<FileConnectorBrowseResponse> { : Promise<FileConnectorBrowseResponse> {
const search = new URLSearchParams(); const search = new URLSearchParams();
if (params.path) search.set("path", params.path); if (params.path) search.set("path", params.path);
if (params.library_id) search.set("library_id", params.library_id); if (params.library_id) search.set("library_id", params.library_id);
if (params.continuation_token) search.set("continuation_token", params.continuation_token);
if (params.campaign_id) search.set("campaign_id", params.campaign_id); if (params.campaign_id) search.set("campaign_id", params.campaign_id);
const suffix = search.toString() ? `?${search.toString()}` : ""; const suffix = search.toString() ? `?${search.toString()}` : "";
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`); return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`);

View File

@@ -5,6 +5,8 @@ import {
Card, Card,
ConnectionTree, ConnectionTree,
ConfirmDialog, ConfirmDialog,
CredentialPanel,
DisabledActionTooltip,
DismissibleAlert, DismissibleAlert,
Dialog, Dialog,
FormField, FormField,
@@ -13,6 +15,8 @@ import {
LoadingFrame, LoadingFrame,
mergeDeltaRows, mergeDeltaRows,
SegmentedControl, SegmentedControl,
StatusBadge,
TableActionGroup,
ToggleSwitch, ToggleSwitch,
useDeltaWatermarks, useDeltaWatermarks,
useUnsavedDraftGuard, useUnsavedDraftGuard,
@@ -63,8 +67,6 @@ type ConnectorProfileDraft = {
username: string; username: string;
password: string; password: string;
token: string; token: string;
passwordEnv: string;
tokenEnv: string;
secretRef: string; secretRef: string;
canBrowse: boolean; canBrowse: boolean;
canImport: boolean; canImport: boolean;
@@ -89,8 +91,6 @@ type ConnectorCredentialDraft = {
username: string; username: string;
password: string; password: string;
token: string; token: string;
passwordEnv: string;
tokenEnv: string;
secretRef: string; secretRef: string;
policyMode: PolicyMode; policyMode: PolicyMode;
allowedPaths: string; allowedPaths: string;
@@ -123,6 +123,9 @@ const PROVIDERS: Array<{id: Provider;label: string;}> = [
{ id: "nextcloud", label: "i18n:govoplan-files.nextcloud.aab73a3d" }, { id: "nextcloud", label: "i18n:govoplan-files.nextcloud.aab73a3d" },
{ id: "seafile", label: "i18n:govoplan-files.seafile.600192cc" }, { id: "seafile", label: "i18n:govoplan-files.seafile.600192cc" },
{ id: "smb", label: "i18n:govoplan-files.smb.515d2f88" }, { id: "smb", label: "i18n:govoplan-files.smb.515d2f88" },
{ id: "s3", label: "S3" },
{ id: "sharepoint", label: "SharePoint" },
{ id: "onedrive", label: "OneDrive" },
{ id: "nfs", label: "i18n:govoplan-files.nfs.db121865" }, { id: "nfs", label: "i18n:govoplan-files.nfs.db121865" },
{ id: "dms", label: "i18n:govoplan-files.dms.477e5652" }, { id: "dms", label: "i18n:govoplan-files.dms.477e5652" },
{ id: "generic", label: "i18n:govoplan-files.generic.ff7613e5" }]; { id: "generic", label: "i18n:govoplan-files.generic.ff7613e5" }];
@@ -133,6 +136,9 @@ const ENDPOINT_PLACEHOLDERS: Record<Provider, string> = {
nextcloud: "http://127.0.0.1:9081/remote.php/dav/files/admin/", nextcloud: "http://127.0.0.1:9081/remote.php/dav/files/admin/",
seafile: "http://127.0.0.1:9082/", seafile: "http://127.0.0.1:9082/",
smb: "smb://127.0.0.1:1445/files", smb: "smb://127.0.0.1:1445/files",
s3: "http://127.0.0.1:9000",
sharepoint: "https://graph.microsoft.com/v1.0/sites/{site-id}/drives/{drive-id}",
onedrive: "https://graph.microsoft.com/v1.0/drives/{drive-id}",
nfs: "nfs://127.0.0.1/export", nfs: "nfs://127.0.0.1/export",
dms: "https://dms.example.test", dms: "https://dms.example.test",
generic: "https://files.example.test" generic: "https://files.example.test"
@@ -340,6 +346,17 @@ export default function FileConnectorSettingsPanel({
setCredentialDraft((current) => current ? { ...current, ...patch } : current); setCredentialDraft((current) => current ? { ...current, ...patch } : current);
} }
function patchCredentialValues(patch: {username?: string | null;password?: string | null;}) {
const next: Partial<ConnectorCredentialDraft> = {};
if (patch.username !== undefined) next.username = String(patch.username ?? "");
if (patch.password !== undefined) next.password = String(patch.password ?? "");
patchCredentialDraft(next);
}
function patchCredentialToken(patch: {password?: string | null;}) {
if (patch.password !== undefined) patchCredentialDraft({ token: String(patch.password ?? "") });
}
function patchPolicyDraft(patch: Partial<CentralConnectorPolicyDraft>) { function patchPolicyDraft(patch: Partial<CentralConnectorPolicyDraft>) {
setPolicyDraft((current) => ({ ...current, ...patch })); setPolicyDraft((current) => ({ ...current, ...patch }));
} }
@@ -472,8 +489,6 @@ export default function FileConnectorSettingsPanel({
username: cleanOrNull(credentialDraft.username), username: cleanOrNull(credentialDraft.username),
password: cleanOrUndefined(credentialDraft.password), password: cleanOrUndefined(credentialDraft.password),
token: cleanOrUndefined(credentialDraft.token), token: cleanOrUndefined(credentialDraft.token),
password_env: cleanOrNull(credentialDraft.passwordEnv),
token_env: cleanOrNull(credentialDraft.tokenEnv),
secret_ref: cleanOrNull(credentialDraft.secretRef) secret_ref: cleanOrNull(credentialDraft.secretRef)
}; };
const sharedPayload = { const sharedPayload = {
@@ -614,8 +629,6 @@ export default function FileConnectorSettingsPanel({
username: cleanOrNull(draft.username), username: cleanOrNull(draft.username),
password: cleanOrUndefined(draft.password), password: cleanOrUndefined(draft.password),
token: cleanOrUndefined(draft.token), token: cleanOrUndefined(draft.token),
password_env: cleanOrNull(draft.passwordEnv),
token_env: cleanOrNull(draft.tokenEnv),
secret_ref: cleanOrNull(draft.secretRef) secret_ref: cleanOrNull(draft.secretRef)
}, },
metadata: metadataFromDraft(draft) metadata: metadataFromDraft(draft)
@@ -661,8 +674,6 @@ export default function FileConnectorSettingsPanel({
username: cleanOrNull(credentialDraft.username), username: cleanOrNull(credentialDraft.username),
password: cleanOrUndefined(credentialDraft.password), password: cleanOrUndefined(credentialDraft.password),
token: cleanOrUndefined(credentialDraft.token), token: cleanOrUndefined(credentialDraft.token),
password_env: cleanOrNull(credentialDraft.passwordEnv),
token_env: cleanOrNull(credentialDraft.tokenEnv),
secret_ref: cleanOrNull(credentialDraft.secretRef) secret_ref: cleanOrNull(credentialDraft.secretRef)
}, },
metadata: profile.metadata ?? {}, metadata: profile.metadata ?? {},
@@ -737,7 +748,7 @@ export default function FileConnectorSettingsPanel({
width: "120px", width: "120px",
render: (row) => { render: (row) => {
const enabled = row.kind === "profile" ? row.profile.enabled : row.credential.enabled; const enabled = row.kind === "profile" ? row.profile.enabled : row.credential.enabled;
return <span className={`status-badge ${enabled ? "success" : "neutral"}`}>{enabled ? "i18n:govoplan-files.enabled.df174a3f" : "i18n:govoplan-files.disabled.f4f4473d"}</span>; return <StatusBadge status={enabled ? "success" : "inactive"} label={enabled ? "i18n:govoplan-files.enabled.df174a3f" : "i18n:govoplan-files.disabled.f4f4473d"} />;
} }
}]; }];
@@ -752,21 +763,19 @@ export default function FileConnectorSettingsPanel({
function renderConnectorActions(row: ConnectorTreeRow) { function renderConnectorActions(row: ConnectorTreeRow) {
if (row.kind === "credential") { if (row.kind === "credential") {
const readOnly = row.credential.source_kind !== "database"; const readOnly = row.credential.source_kind !== "database";
return ( return <TableActionGroup actions={[
<div className="admin-icon-actions"> { id: "edit", label: i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label }), icon: <Edit3 size={15} />, disabled: !canWrite || readOnly || saving, onClick: () => startCredentialEdit(row.credential) },
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label })} aria-label={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label })} onClick={() => startCredentialEdit(row.credential)} disabled={!canWrite || readOnly || saving}><Edit3 size={15} /></Button> { id: "disable", label: i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label }), icon: <Trash2 size={15} />, variant: "danger", applicable: row.credential.enabled, disabled: !canWrite || readOnly || saving, onClick: () => setPendingCredentialDeactivate(row.credential) }
<Button type="button" variant="danger" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label })} aria-label={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label })} onClick={() => setPendingCredentialDeactivate(row.credential)} disabled={!canWrite || readOnly || saving || !row.credential.enabled}><Trash2 size={15} /></Button> ]} />;
</div>);
} }
const readOnly = row.profile.source_kind !== "database"; const readOnly = row.profile.source_kind !== "database";
return ( return <TableActionGroup actions={[
<div className="admin-icon-actions"> { id: "test", label: i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label }), icon: <RefreshCw size={15} />, applicable: row.profile.enabled, disabled: saving || testingProfileId === row.profile.id, onClick: () => void testBrowse(row.profile) },
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label })} onClick={() => void testBrowse(row.profile)} disabled={saving || testingProfileId === row.profile.id || !row.profile.enabled}><RefreshCw size={15} /></Button> { id: "add-credential", label: i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label }), icon: <UserRoundKey size={15} />, variant: "primary", disabled: !canWrite || saving, onClick: () => startCredentialCreate(row.profile) },
<Button type="button" variant="primary" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label })} onClick={() => startCredentialCreate(row.profile)} disabled={!canWrite || saving}><UserRoundKey size={15} /></Button> { id: "edit", label: i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label }), icon: <Edit3 size={15} />, disabled: !canWrite || readOnly || saving, onClick: () => startEdit(row.profile) },
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label })} onClick={() => startEdit(row.profile)} disabled={!canWrite || readOnly || saving}><Edit3 size={15} /></Button> { id: "disable", label: i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label }), icon: <Trash2 size={15} />, variant: "danger", applicable: row.profile.enabled, disabled: !canWrite || readOnly || saving, onClick: () => setPendingDeactivate(row.profile) }
<Button type="button" variant="danger" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label })} onClick={() => setPendingDeactivate(row.profile)} disabled={!canWrite || readOnly || saving || !row.profile.enabled}><Trash2 size={15} /></Button> ]} />;
</div>);
} }
@@ -801,7 +810,7 @@ export default function FileConnectorSettingsPanel({
{targetSelectionRequired && {targetSelectionRequired &&
<Card title={i18nMessage("i18n:govoplan-files.value_scope", { value0: targetLabel })}> <Card title={i18nMessage("i18n:govoplan-files.value_scope", { value0: targetLabel })}>
<div className="mail-profile-target-row"> <div className="settings-target-row">
<FormField label={targetLabel}> <FormField label={targetLabel}>
<select value={selectedTargetId} onChange={(event) => setSelectedTargetId(event.target.value)} disabled={loading || saving || !hasSelectableTarget}> <select value={selectedTargetId} onChange={(event) => setSelectedTargetId(event.target.value)} disabled={loading || saving || !hasSelectableTarget}>
{!hasSelectableTarget && <option value="">{targetEmptyText}</option>} {!hasSelectableTarget && <option value="">{targetEmptyText}</option>}
@@ -846,7 +855,7 @@ export default function FileConnectorSettingsPanel({
<LoadingFrame loading={loading} label="i18n:govoplan-files.loading_connector_policy.f12ab285"> <LoadingFrame loading={loading} label="i18n:govoplan-files.loading_connector_policy.f12ab285">
<> <>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8"> <FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8">
<textarea value={policyDraft.allowedConnectors} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedConnectors: event.target.value })} rows={3} placeholder={"tenant-webdav\nseafile-*"} /> <textarea value={policyDraft.allowedConnectors} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedConnectors: event.target.value })} rows={3} placeholder={"tenant-webdav\nseafile-*"} />
</FormField> </FormField>
@@ -906,11 +915,11 @@ export default function FileConnectorSettingsPanel({
footer={credentialDraft ? footer={credentialDraft ?
<> <>
<Button onClick={closeCredentialDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button> <Button onClick={closeCredentialDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button>
<span className="disabled-action-tooltip" data-tooltip={credentialSaveTooltip}> <DisabledActionTooltip reason={credentialSaveTooltip}>
<Button variant="primary" onClick={() => void saveCredentialDraft()} disabled={credentialSaveDisabled}> <Button variant="primary" onClick={() => void saveCredentialDraft()} disabled={credentialSaveDisabled}>
<Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_credential.2c02a6d2"} <Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_credential.2c02a6d2"}
</Button> </Button>
</span> </DisabledActionTooltip>
</> : </> :
null}> null}>
@@ -921,7 +930,7 @@ export default function FileConnectorSettingsPanel({
<h3>Credential</h3> <h3>Credential</h3>
<p>Choose where this credential can be used and how it signs in.</p> <p>Choose where this credential can be used and how it signs in.</p>
</header> </header>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.credential_id.9432a6e1"> <FormField label="i18n:govoplan-files.credential_id.9432a6e1">
<input className={!editingCredentialId && !credentialDraft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingCredentialId && !credentialDraft.id.trim() || undefined} value={credentialDraft.id} disabled={Boolean(editingCredentialId) || saving} onChange={(event) => patchCredentialDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav-credentials`} /> <input className={!editingCredentialId && !credentialDraft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingCredentialId && !credentialDraft.id.trim() || undefined} value={credentialDraft.id} disabled={Boolean(editingCredentialId) || saving} onChange={(event) => patchCredentialDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav-credentials`} />
</FormField> </FormField>
@@ -983,51 +992,53 @@ export default function FileConnectorSettingsPanel({
actor: "Connector administrator", actor: "Connector administrator",
target: "Credential mode" target: "Credential mode"
}} /> : }} /> :
<div className="form-grid two-column-form-grid"> <div className="file-connector-secret-fields">
{(credentialDraft.credentialMode === "basic" || credentialDraft.credentialMode === "secret_ref") &&
<FormField label="i18n:govoplan-files.username.84c29015">
<input value={credentialDraft.username} disabled={saving} onChange={(event) => patchCredentialDraft({ username: event.target.value })} autoComplete="username" />
</FormField>
}
{credentialDraft.credentialMode === "basic" && {credentialDraft.credentialMode === "basic" &&
<FormField label="i18n:govoplan-files.password.8be3c943"> <CredentialPanel
<input value={credentialDraft.password} disabled={saving} onChange={(event) => patchCredentialDraft({ password: event.target.value })} type="password" autoComplete="new-password" placeholder={editingCredentialId ? "i18n:govoplan-files.leave_empty_to_keep_saved_password.6ec39f5e" : ""} /> values={{ username: credentialDraft.username, password: credentialDraft.password }}
</FormField> onChange={patchCredentialValues}
} disabled={saving}
{credentialDraft.credentialMode === "token" && savedPassword={Boolean(editingCredentialId) && !credentialDraft.clearPassword}
<FormField label="i18n:govoplan-files.token.a1141eb9"> savedPasswordPlaceholder="i18n:govoplan-files.leave_empty_to_keep_saved_password.6ec39f5e" />
<input value={credentialDraft.token} disabled={saving} onChange={(event) => patchCredentialDraft({ token: event.target.value })} type="password" placeholder={editingCredentialId ? "i18n:govoplan-files.leave_empty_to_keep_saved_token.e725857b" : ""} />
</FormField>
} }
{credentialDraft.credentialMode === "secret_ref" && {credentialDraft.credentialMode === "secret_ref" &&
<>
<CredentialPanel
values={{ username: credentialDraft.username }}
onChange={patchCredentialValues}
disabled={saving}
showPassword={false} />
<div className="form-grid two">
<FormField label="i18n:govoplan-files.secret_reference.04ed2221"> <FormField label="i18n:govoplan-files.secret_reference.04ed2221">
<input value={credentialDraft.secretRef} disabled={saving} onChange={(event) => patchCredentialDraft({ secretRef: event.target.value })} /> <input value={credentialDraft.secretRef} disabled={saving} onChange={(event) => patchCredentialDraft({ secretRef: event.target.value })} />
</FormField> </FormField>
</div>
</>
}
{credentialDraft.credentialMode === "token" &&
<CredentialPanel
values={{ password: credentialDraft.token }}
onChange={patchCredentialToken}
disabled={saving}
showUsername={false}
passwordLabel="i18n:govoplan-files.token.a1141eb9"
savedPassword={Boolean(editingCredentialId) && !credentialDraft.clearToken}
savedPasswordPlaceholder="i18n:govoplan-files.leave_empty_to_keep_saved_token.e725857b" />
} }
</div> </div>
} }
<div className="button-row compact-actions"> <div className="button-row compact-actions">
<span className="disabled-action-tooltip" data-tooltip={credentialTestDisabled ? credentialTestTooltip : ""}> <DisabledActionTooltip reason={credentialTestDisabled ? credentialTestTooltip : ""}>
<Button type="button" onClick={() => void testCredentialLogin()} disabled={credentialTestDisabled}> <Button type="button" onClick={() => void testCredentialLogin()} disabled={credentialTestDisabled}>
<RefreshCw size={16} aria-hidden="true" /> {testingCredentialLogin ? "Testing login" : "Test login"} <RefreshCw size={16} aria-hidden="true" /> {testingCredentialLogin ? "Testing login" : "Test login"}
</Button> </Button>
</span> </DisabledActionTooltip>
</div> </div>
{credentialLoginResult && {credentialLoginResult &&
<DismissibleAlert tone={credentialLoginResult.ok ? "success" : "warning"} resetKey={credentialLoginResult.message}> <DismissibleAlert tone={credentialLoginResult.ok ? "success" : "warning"} resetKey={credentialLoginResult.message}>
{credentialLoginResult.message} {credentialLoginResult.message}
</DismissibleAlert> </DismissibleAlert>
} }
<AdvancedOptionsPanel title="Environment and fallback secrets" summary="Use these only when the deployment injects secrets outside the database.">
<div className="form-grid two-column-form-grid">
<FormField label="i18n:govoplan-files.password_environment_variable.0e77673f">
<input value={credentialDraft.passwordEnv} disabled={saving} onChange={(event) => patchCredentialDraft({ passwordEnv: event.target.value })} />
</FormField>
<FormField label="i18n:govoplan-files.token_environment_variable.5af4a79e">
<input value={credentialDraft.tokenEnv} disabled={saving} onChange={(event) => patchCredentialDraft({ tokenEnv: event.target.value })} />
</FormField>
</div>
</AdvancedOptionsPanel>
</section> </section>
<section className="adaptive-config-section"> <section className="adaptive-config-section">
@@ -1035,7 +1046,7 @@ export default function FileConnectorSettingsPanel({
<h3>Policy</h3> <h3>Policy</h3>
<p>Limit where lower levels may use this credential.</p> <p>Limit where lower levels may use this credential.</p>
</header> </header>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.policy.bb9cf141"> <FormField label="i18n:govoplan-files.policy.bb9cf141">
<select value={credentialDraft.policyMode} disabled={saving} onChange={(event) => patchCredentialDraft({ policyMode: event.target.value as PolicyMode })}> <select value={credentialDraft.policyMode} disabled={saving} onChange={(event) => patchCredentialDraft({ policyMode: event.target.value as PolicyMode })}>
<option value="allow-provider">i18n:govoplan-files.can_use_this_credential.f4a41971</option> <option value="allow-provider">i18n:govoplan-files.can_use_this_credential.f4a41971</option>
@@ -1072,11 +1083,11 @@ export default function FileConnectorSettingsPanel({
footer={draft ? footer={draft ?
<> <>
<Button onClick={closeProfileDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button> <Button onClick={closeProfileDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button>
<span className="disabled-action-tooltip" data-tooltip={profileSaveTooltip}> <DisabledActionTooltip reason={profileSaveTooltip}>
<Button variant="primary" onClick={() => void saveDraft()} disabled={profileSaveDisabled}> <Button variant="primary" onClick={() => void saveDraft()} disabled={profileSaveDisabled}>
<Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_connection.07796d38"} <Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_connection.07796d38"}
</Button> </Button>
</span> </DisabledActionTooltip>
</> : </> :
null}> null}>
@@ -1087,7 +1098,7 @@ export default function FileConnectorSettingsPanel({
<h3>Connection</h3> <h3>Connection</h3>
<p>Name the file server connection and choose the provider.</p> <p>Name the file server connection and choose the provider.</p>
</header> </header>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.profile_id.25961d68"> <FormField label="i18n:govoplan-files.profile_id.25961d68">
<input className={!editingProfileId && !draft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingProfileId && !draft.id.trim() || undefined} value={draft.id} disabled={Boolean(editingProfileId) || saving} onChange={(event) => patchDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav`} /> <input className={!editingProfileId && !draft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingProfileId && !draft.id.trim() || undefined} value={draft.id} disabled={Boolean(editingProfileId) || saving} onChange={(event) => patchDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav`} />
</FormField> </FormField>
@@ -1120,7 +1131,7 @@ export default function FileConnectorSettingsPanel({
<h3>Location and credentials</h3> <h3>Location and credentials</h3>
<p>Only fields relevant for the selected provider and credential mode are shown.</p> <p>Only fields relevant for the selected provider and credential mode are shown.</p>
</header> </header>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.endpoint_url.65aaaa45"> <FormField label="i18n:govoplan-files.endpoint_url.65aaaa45">
<input value={draft.endpointUrl} disabled={saving} onChange={(event) => patchDraft({ endpointUrl: event.target.value })} placeholder={ENDPOINT_PLACEHOLDERS[draft.provider]} /> <input value={draft.endpointUrl} disabled={saving} onChange={(event) => patchDraft({ endpointUrl: event.target.value })} placeholder={ENDPOINT_PLACEHOLDERS[draft.provider]} />
</FormField> </FormField>
@@ -1157,7 +1168,7 @@ export default function FileConnectorSettingsPanel({
</DismissibleAlert> </DismissibleAlert>
} }
<AdvancedOptionsPanel title="Protocol and compatibility options" summary="Change these only when the provider or network requires an override."> <AdvancedOptionsPanel title="Protocol and compatibility options" summary="Change these only when the provider or network requires an override.">
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.browse_protocol.d1f27c90"> <FormField label="i18n:govoplan-files.browse_protocol.d1f27c90">
<select value={draft.browseProtocol} disabled={saving} onChange={(event) => patchDraft({ browseProtocol: event.target.value })}> <select value={draft.browseProtocol} disabled={saving} onChange={(event) => patchDraft({ browseProtocol: event.target.value })}>
<option value="">i18n:govoplan-files.native_default.ba32f7b6</option> <option value="">i18n:govoplan-files.native_default.ba32f7b6</option>
@@ -1191,7 +1202,7 @@ export default function FileConnectorSettingsPanel({
<ToggleSwitch label="i18n:govoplan-files.import.d6fbc9d2" checked={draft.canImport} disabled={saving} onChange={(canImport) => patchDraft({ canImport })} /> <ToggleSwitch label="i18n:govoplan-files.import.d6fbc9d2" checked={draft.canImport} disabled={saving} onChange={(canImport) => patchDraft({ canImport })} />
<ToggleSwitch label="i18n:govoplan-files.sync.905f6309" checked={draft.canSync} disabled={saving} onChange={(canSync) => patchDraft({ canSync })} /> <ToggleSwitch label="i18n:govoplan-files.sync.905f6309" checked={draft.canSync} disabled={saving} onChange={(canSync) => patchDraft({ canSync })} />
</div> </div>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.policy.bb9cf141"> <FormField label="i18n:govoplan-files.policy.bb9cf141">
<select value={draft.policyMode} disabled={saving} onChange={(event) => patchDraft({ policyMode: event.target.value as PolicyMode })}> <select value={draft.policyMode} disabled={saving} onChange={(event) => patchDraft({ policyMode: event.target.value as PolicyMode })}>
<option value="allow-provider">i18n:govoplan-files.can_use_this_connection.5091a74c</option> <option value="allow-provider">i18n:govoplan-files.can_use_this_connection.5091a74c</option>
@@ -1342,8 +1353,6 @@ function emptyDraft(scopeType: ConnectorScope): ConnectorProfileDraft {
username: "", username: "",
password: "", password: "",
token: "", token: "",
passwordEnv: "",
tokenEnv: "",
secretRef: "", secretRef: "",
canBrowse: true, canBrowse: true,
canImport: true, canImport: true,
@@ -1406,8 +1415,6 @@ function emptyCredentialDraft(scopeType: ConnectorScope): ConnectorCredentialDra
username: "", username: "",
password: "", password: "",
token: "", token: "",
passwordEnv: "",
tokenEnv: "",
secretRef: "", secretRef: "",
policyMode: "allow-provider", policyMode: "allow-provider",
allowedPaths: "", allowedPaths: "",

View File

@@ -8,6 +8,7 @@ import {
FileDropZone, FileDropZone,
FormField, FormField,
LoadingIndicator, LoadingIndicator,
ResourceAccessExplanation,
ToggleSwitch, ToggleSwitch,
hasScope, hasScope,
type ApiSettings, type ApiSettings,
@@ -182,6 +183,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
}, [visibleFiles, folders, currentFolder, searchActive, sortColumn, sortDirection]); }, [visibleFiles, folders, currentFolder, searchActive, sortColumn, sortDirection]);
const fileListViewportRef = useRef<HTMLDivElement | null>(null); const fileListViewportRef = useRef<HTMLDivElement | null>(null);
const fileListMeasureRowRef = useRef<HTMLDivElement | null>(null); const fileListMeasureRowRef = useRef<HTMLDivElement | null>(null);
const managedSpaceLoadsInFlightRef = useRef<Set<string>>(new Set());
const [fileListViewportHeight, setFileListViewportHeight] = useState(0); const [fileListViewportHeight, setFileListViewportHeight] = useState(0);
const [fileListScrollTop, setFileListScrollTop] = useState(0); const [fileListScrollTop, setFileListScrollTop] = useState(0);
const [fileListRowHeight, setFileListRowHeight] = useState(DEFAULT_FILE_LIST_ROW_HEIGHT); const [fileListRowHeight, setFileListRowHeight] = useState(DEFAULT_FILE_LIST_ROW_HEIGHT);
@@ -291,8 +293,9 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
try { try {
const response = await listFileSpaces(settings); const response = await listFileSpaces(settings);
setSpaces(response.spaces); setSpaces(response.spaces);
setActiveSpaceId((current) => current || response.spaces[0]?.id || ""); const nextActiveSpace = response.spaces.find((space) => space.id === activeSpaceId) ?? response.spaces[0] ?? null;
await Promise.all(response.spaces.filter((space) => !isConnectorSpace(space)).map((space) => loadSpaceContents(space, { silent: true }))); setActiveSpaceId(nextActiveSpace?.id || "");
if (nextActiveSpace && !isConnectorSpace(nextActiveSpace)) await loadSpaceContents(nextActiveSpace, { silent: true });
} catch (err) { } catch (err) {
setError(err instanceof Error ? err.message : String(err)); setError(err instanceof Error ? err.message : String(err));
} finally { } finally {
@@ -331,6 +334,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
await loadConnectorSpaceContents(space, { silent: options.silent, folderPath: "" }); await loadConnectorSpaceContents(space, { silent: options.silent, folderPath: "" });
return; return;
} }
if (managedSpaceLoadsInFlightRef.current.has(space.id)) return;
managedSpaceLoadsInFlightRef.current.add(space.id);
if (!options.silent) { if (!options.silent) {
setBusy(true); setBusy(true);
setError(""); setError("");
@@ -372,6 +377,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} catch (err) { } catch (err) {
setError(err instanceof Error ? err.message : String(err)); setError(err instanceof Error ? err.message : String(err));
} finally { } finally {
managedSpaceLoadsInFlightRef.current.delete(space.id);
if (!options.silent) setBusy(false); if (!options.silent) setBusy(false);
} }
} }
@@ -411,6 +417,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
const nextSpace = spaces.find((space) => space.id === activeSpaceId); const nextSpace = spaces.find((space) => space.id === activeSpaceId);
if (nextSpace && isConnectorSpace(nextSpace)) { if (nextSpace && isConnectorSpace(nextSpace)) {
void loadConnectorSpaceContents(nextSpace, { folderPath: "", silent: true }); void loadConnectorSpaceContents(nextSpace, { folderPath: "", silent: true });
} else if (nextSpace && filesBySpace[nextSpace.id] === undefined && foldersBySpace[nextSpace.id] === undefined) {
void loadSpaceContents(nextSpace, { silent: true });
} }
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
}, [activeSpaceId, spaces]); }, [activeSpaceId, spaces]);
@@ -432,6 +440,16 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
return activeDialogTarget; return activeDialogTarget;
} }
function uploadRejectedReason(target: FileActionTarget | null): string {
if (busy || uploadActive) return "Another file operation is already running. Wait until it finishes before dropping files.";
if (!canUpload) return "You do not have permission to upload files here.";
if (!target) return "Choose a destination folder before dropping files.";
const targetSpace = findSpace(target.spaceId);
if (!targetSpace) return "The selected upload destination is no longer available.";
if (isConnectorSpace(targetSpace)) return "Files cannot be uploaded into connector spaces from this view.";
return "The dropped item cannot be uploaded here.";
}
function openDialog(kind: DialogKind, target: FileActionTarget | null = null) { function openDialog(kind: DialogKind, target: FileActionTarget | null = null) {
if ((kind === "upload" || kind === "connector-sync") && !canUpload) return; if ((kind === "upload" || kind === "connector-sync") && !canUpload) return;
if (kind === "connector-space" && !canOrganize) return; if (kind === "connector-space" && !canOrganize) return;
@@ -608,10 +626,12 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
async function handleFilesUpload(fileList: FileList | File[], options: {conflictStrategy?: ConflictStrategy;conflictResolutions?: ConflictResolution[];bypassConflictDialog?: boolean;target?: FileActionTarget;} = {}) { async function handleFilesUpload(fileList: FileList | File[], options: {conflictStrategy?: ConflictStrategy;conflictResolutions?: ConflictResolution[];bypassConflictDialog?: boolean;target?: FileActionTarget;} = {}) {
if (!canUpload) return;
const target = options.target ?? currentActionTarget(); const target = options.target ?? currentActionTarget();
const targetSpace = target ? findSpace(target.spaceId) : null; const targetSpace = target ? findSpace(target.spaceId) : null;
if (!target || !targetSpace || isConnectorSpace(targetSpace)) return; if (busy || uploadActive || !canUpload || !target || !targetSpace || isConnectorSpace(targetSpace)) {
setError(uploadRejectedReason(target));
return;
}
const selected = Array.from(fileList); const selected = Array.from(fileList);
if (selected.length === 0) return; if (selected.length === 0) return;
if (!options.bypassConflictDialog && !unpackZip) { if (!options.bypassConflictDialog && !unpackZip) {
@@ -671,13 +691,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} }
async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) { async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) {
const previousTarget = dialogTarget; await handleFilesUpload(fileList, { target });
setDialogTarget(target);
try {
await handleFilesUpload(fileList);
} finally {
setDialogTarget(previousTarget);
}
} }
async function openConnectorSyncDialog(target: FileActionTarget | null = toolbarTarget()) { async function openConnectorSyncDialog(target: FileActionTarget | null = toolbarTarget()) {
@@ -1332,8 +1346,20 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
setDropTargetKey(reason || noop ? "" : dropTargetId(target)); setDropTargetKey(reason || noop ? "" : dropTargetId(target));
return; return;
} }
event.dataTransfer.dropEffect = canUpload ? "copy" : "none"; event.dataTransfer.dropEffect = canUpload && !busy && !uploadActive ? "copy" : "none";
setDropTargetKey(canUpload ? dropTargetId(target) : ""); setDropTargetKey(canUpload && !busy && !uploadActive ? dropTargetId(target) : "");
}
function handleCurrentFolderDragOver(event: ReactDragEvent<HTMLElement>) {
const target = toolbarTarget();
if (!target) {
event.preventDefault();
event.stopPropagation();
event.dataTransfer.dropEffect = "none";
setDropTargetKey("");
return;
}
handleDropTargetDragOver(event, target);
} }
async function handleDropOnTarget(event: ReactDragEvent<HTMLElement>, target: FileActionTarget) { async function handleDropOnTarget(event: ReactDragEvent<HTMLElement>, target: FileActionTarget) {
@@ -1341,7 +1367,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
event.stopPropagation(); event.stopPropagation();
setDragActive(false); setDragActive(false);
setDropTargetKey(""); setDropTargetKey("");
if (isConnectorSpace(findSpace(target.spaceId))) return; const hasDroppedFiles = event.dataTransfer.files.length > 0;
if (isConnectorSpace(findSpace(target.spaceId))) {
if (hasDroppedFiles) setError(uploadRejectedReason(target));
return;
}
if (isInternalDrag(event)) { if (isInternalDrag(event)) {
if (!canOrganize) return; if (!canOrganize) return;
const state = internalDrag ?? parseDragState(event.dataTransfer.getData(INTERNAL_DRAG_TYPE)); const state = internalDrag ?? parseDragState(event.dataTransfer.getData(INTERNAL_DRAG_TYPE));
@@ -1354,9 +1384,27 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
await runTransfer(event.ctrlKey ? "copy" : "move", state.sourceSpaceId, { fileIds: new Set(state.fileIds), folderPaths: new Set(state.folderPaths) }, target); await runTransfer(event.ctrlKey ? "copy" : "move", state.sourceSpaceId, { fileIds: new Set(state.fileIds), folderPaths: new Set(state.folderPaths) }, target);
return; return;
} }
if (canUpload && event.dataTransfer.files.length > 0) { if (hasDroppedFiles) {
await uploadExternalFilesToTarget(event.dataTransfer.files, target); if (!canUpload || busy || uploadActive) {
setError(uploadRejectedReason(target));
return;
} }
await uploadExternalFilesToTarget(event.dataTransfer.files, target);
return;
}
setError("Drop one or more files from your computer, or drag files and folders from this file space.");
}
async function handleDropOnCurrentFolder(event: ReactDragEvent<HTMLElement>) {
const target = toolbarTarget();
if (!target) {
event.preventDefault();
event.stopPropagation();
setDropTargetKey("");
setError(uploadRejectedReason(null));
return;
}
await handleDropOnTarget(event, target);
} }
function clearDropState() { function clearDropState() {
@@ -1933,8 +1981,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} }
const currentFolderDropTarget = toolbarTarget();
const currentFolderDropActive = currentFolderDropTarget ? dropTargetKey === dropTargetId(currentFolderDropTarget) : false;
return ( return (
<div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen"> <div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen files-page">
{error && {error &&
<DismissibleAlert tone="danger" resetKey={error} floating>{error}</DismissibleAlert> <DismissibleAlert tone="danger" resetKey={error} floating>{error}</DismissibleAlert>
} }
@@ -2044,9 +2095,12 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
renderConnectorSpaceContent() : renderConnectorSpaceContent() :
<div <div
className="file-list-drop-target" className={`file-list-drop-target ${currentFolderDropActive ? "is-drop-target" : ""}`}
ref={fileListViewportRef} ref={fileListViewportRef}
onScroll={(event) => setFileListScrollTop(event.currentTarget.scrollTop)} onScroll={(event) => setFileListScrollTop(event.currentTarget.scrollTop)}
onDragOver={handleCurrentFolderDragOver}
onDragLeave={clearDropState}
onDrop={(event) => void handleDropOnCurrentFolder(event)}
onContextMenu={(event) => openContextMenu(event, "empty")} onContextMenu={(event) => openContextMenu(event, "empty")}
onClick={(event) => { onClick={(event) => {
if (event.target === event.currentTarget) applySelectionKeys(new Set<EntrySelectionKey>()); if (event.target === event.currentTarget) applySelectionKeys(new Set<EntrySelectionKey>());
@@ -2158,8 +2212,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} }
{accessExplanationTarget && {accessExplanationTarget &&
<FileDialog title="i18n:govoplan-files.access_explanation.75ee7f62" onClose={() => {if (!resourceAccessLoading) {setAccessExplanationTarget(null);setResourceAccessExplanation(null);}}}> <FileDialog title="i18n:govoplan-core.access_explanation.75ee7f62" onClose={() => {if (!resourceAccessLoading) {setAccessExplanationTarget(null);setResourceAccessExplanation(null);}}}>
<ResourceAccessExplanationContent <ResourceAccessExplanation
loading={resourceAccessLoading} loading={resourceAccessLoading}
explanation={resourceAccessExplanation} explanation={resourceAccessExplanation}
fallbackResourceLabel={accessExplanationTarget.label} /> fallbackResourceLabel={accessExplanationTarget.label} />
@@ -2172,13 +2226,14 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "upload" && {dialog === "upload" &&
<FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}> <FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}>
<FileDropZone <FileDropZone
disabled={busy || !activeDialogSpace || !canUpload} disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace) || !canUpload}
busy={uploadActive} busy={uploadActive}
progress={visibleUploadProgress} progress={visibleUploadProgress}
busyLabel={uploadBusyLabel} busyLabel={uploadBusyLabel}
progressLabel={uploadProgressLabel} progressLabel={uploadProgressLabel}
note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`} note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`}
onFiles={(files) => handleFilesUpload(files)} /> onRejectedDrop={(reason) => setError(reason === "disabled" ? uploadRejectedReason(activeDialogTarget) : "The browser did not provide readable file data for this drop. Use the file picker, or drag local files from a file manager that exposes file contents to the browser.")}
onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} />
<ToggleSwitch label="i18n:govoplan-files.unpack_zip_uploads.256fead4" checked={unpackZip} onChange={setUnpackZip} disabled={busy} /> <ToggleSwitch label="i18n:govoplan-files.unpack_zip_uploads.256fead4" checked={unpackZip} onChange={setUnpackZip} disabled={busy} />
<div className="field-block"> <div className="field-block">
@@ -2379,7 +2434,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "transfer" && transferDialogState && {dialog === "transfer" && transferDialogState &&
<FileDialog title={`${transferMode === "copy" ? "i18n:govoplan-files.copy.af74f7c5" : "i18n:govoplan-files.move.76cdb950"} selection`} onClose={closeDialog}> <FileDialog title={`${transferMode === "copy" ? "i18n:govoplan-files.copy.af74f7c5" : "i18n:govoplan-files.move.76cdb950"} selection`} onClose={closeDialog}>
<p className="muted">i18n:govoplan-files.choose_a_destination_space_and_folder_folders_ar.45158643</p> <p className="muted">i18n:govoplan-files.choose_a_destination_space_and_folder_folders_ar.45158643</p>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.destination_space.92b63970" help="i18n:govoplan-files.select_the_space_that_will_receive_the_selected_.0a8bea8f"> <FormField label="i18n:govoplan-files.destination_space.92b63970" help="i18n:govoplan-files.select_the_space_that_will_receive_the_selected_.0a8bea8f">
<select value={transferDialogState.targetSpaceId} onChange={(event) => setTransferDialogState((current) => current ? { ...current, targetSpaceId: event.target.value, targetFolder: "" } : current)}> <select value={transferDialogState.targetSpaceId} onChange={(event) => setTransferDialogState((current) => current ? { ...current, targetSpaceId: event.target.value, targetFolder: "" } : current)}>
{spaces.filter((space) => !isConnectorSpace(space)).map((space) => <option key={space.id} value={space.id}>{space.label}</option>)} {spaces.filter((space) => !isConnectorSpace(space)).map((space) => <option key={space.id} value={space.id}>{space.label}</option>)}
@@ -2418,7 +2473,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
{dialog === "rename" && {dialog === "rename" &&
<FileDialog title="i18n:govoplan-files.bulk_rename_selected_items.5e79d694" onClose={closeDialog}> <FileDialog title="i18n:govoplan-files.bulk_rename_selected_items.5e79d694" onClose={closeDialog}>
<p className="muted">i18n:govoplan-files.bulk_rename_changes_managed_display_paths_only_i.8cf34a6b</p> <p className="muted">i18n:govoplan-files.bulk_rename_changes_managed_display_paths_only_i.8cf34a6b</p>
<div className="form-grid two-column-form-grid"> <div className="form-grid two">
<FormField label="i18n:govoplan-files.mode.a7b93d21" help="i18n:govoplan-files.choose_how_the_selected_names_should_be_changed.0231854f"> <FormField label="i18n:govoplan-files.mode.a7b93d21" help="i18n:govoplan-files.choose_how_the_selected_names_should_be_changed.0231854f">
<select value={renameMode} onChange={(event) => setRenameMode(event.target.value as RenameMode)}> <select value={renameMode} onChange={(event) => setRenameMode(event.target.value as RenameMode)}>
<option value="prefix">i18n:govoplan-files.add_prefix.672452bc</option> <option value="prefix">i18n:govoplan-files.add_prefix.672452bc</option>
@@ -2454,71 +2509,6 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
} }
function ResourceAccessExplanationContent({
loading,
explanation,
fallbackResourceLabel
}: {loading: boolean;explanation: ResourceAccessExplanationResponse | null;fallbackResourceLabel: string;}) {
if (loading) return <p className="muted small-note">i18n:govoplan-files.loading_access_explanation.04a7c934</p>;
if (!explanation) return null;
const userLabel = explanation.user.display_name || explanation.user.email || explanation.user.id;
const resourceLabel = explanation.provenance.find((item) => item.kind === "resource")?.label || fallbackResourceLabel || explanation.resource_id;
return (
<>
<div className="form-grid compact responsive-form-grid">
<div><span className="form-label">i18n:govoplan-files.user.9f8a2389</span><p>{userLabel}</p></div>
<div><span className="form-label">i18n:govoplan-files.resource.d1c626a9</span><p>{resourceLabel}</p></div>
<div><span className="form-label">i18n:govoplan-files.action.97c89a4d</span><p><code>{explanation.action}</code></p></div>
<div><span className="form-label">i18n:govoplan-files.evidence.8487d192</span><p>{explanation.provenance.length}</p></div>
</div>
{explanation.provenance.length === 0 ?
<p className="muted small-note">i18n:govoplan-files.no_access_evidence_was_returned.84a21e4e</p> :
<div className="admin-assignment-grid">
{explanation.provenance.map((item, index) =>
<div key={`${item.kind}:${item.id ?? index}`}>
<strong>{provenanceKindLabel(item.kind)}</strong>
<div className="muted small-note">
{item.source || "i18n:govoplan-files.no_source.6dcf9723"}
{item.id && <> · <code>{item.id}</code></>}
</div>
{item.label && <p>{item.label}</p>}
{Object.keys(item.details ?? {}).length > 0 && <p className="muted small-note">{formatProvenanceDetails(item.details ?? {})}</p>}
</div>
)}
</div>
}
</>);
}
function provenanceKindLabel(kind: string): string {
switch (kind) {
case "resource":
return "i18n:govoplan-files.resource.d1c626a9";
case "owner":
return "i18n:govoplan-files.owner.89ff3122";
case "share":
return "i18n:govoplan-files.share.09ca55ca";
case "policy":
return "i18n:govoplan-files.policy.0b779a05";
case "role":
return "i18n:govoplan-files.role.b5b4a5a2";
case "right":
return "i18n:govoplan-files.permission.2f81a22d";
default:
return kind;
}
}
function formatProvenanceDetails(details: Record<string, unknown>): string {
return Object.entries(details).map(([key, value]) => `${key}: ${formatProvenanceValue(value)}`).join("; ");
}
function formatProvenanceValue(value: unknown): string {
if (value === null || value === undefined) return "i18n:govoplan-files.none.6eef6648";
if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") return String(value);
return JSON.stringify(value);
}
function FileSearchRow({ function FileSearchRow({
value, value,
caseSensitive, caseSensitive,

View File

@@ -252,7 +252,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.overwrite.0625c54e": "Overwrite", "i18n:govoplan-files.overwrite.0625c54e": "Overwrite",
"i18n:govoplan-files.owner_space.364c4b62": "Owner space", "i18n:govoplan-files.owner_space.364c4b62": "Owner space",
"i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder", "i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder",
"i18n:govoplan-files.password_environment_variable.0e77673f": "Password environment variable",
"i18n:govoplan-files.password.8be3c943": "Password", "i18n:govoplan-files.password.8be3c943": "Password",
"i18n:govoplan-files.path_limited.d32cbef0": "Path-limited", "i18n:govoplan-files.path_limited.d32cbef0": "Path-limited",
"i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results", "i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results",
@@ -318,7 +317,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.this_file_credential.695efb90": "this file credential", "i18n:govoplan-files.this_file_credential.695efb90": "this file credential",
"i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.", "i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.",
"i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.", "i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.",
"i18n:govoplan-files.token_environment_variable.5af4a79e": "Token environment variable",
"i18n:govoplan-files.token.a1141eb9": "Token", "i18n:govoplan-files.token.a1141eb9": "Token",
"i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads", "i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads",
"i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive", "i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive",
@@ -609,7 +607,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.overwrite.0625c54e": "Overwrite", "i18n:govoplan-files.overwrite.0625c54e": "Overwrite",
"i18n:govoplan-files.owner_space.364c4b62": "Owner space", "i18n:govoplan-files.owner_space.364c4b62": "Owner space",
"i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder", "i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder",
"i18n:govoplan-files.password_environment_variable.0e77673f": "Password environment variable",
"i18n:govoplan-files.password.8be3c943": "Passwort", "i18n:govoplan-files.password.8be3c943": "Passwort",
"i18n:govoplan-files.path_limited.d32cbef0": "Path-limited", "i18n:govoplan-files.path_limited.d32cbef0": "Path-limited",
"i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results", "i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results",
@@ -675,7 +672,6 @@ export const generatedTranslations: PlatformTranslations = {
"i18n:govoplan-files.this_file_credential.695efb90": "this file credential", "i18n:govoplan-files.this_file_credential.695efb90": "this file credential",
"i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.", "i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.",
"i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.", "i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.",
"i18n:govoplan-files.token_environment_variable.5af4a79e": "Token environment variable",
"i18n:govoplan-files.token.a1141eb9": "Token", "i18n:govoplan-files.token.a1141eb9": "Token",
"i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads", "i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads",
"i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive", "i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive",

View File

@@ -1,85 +1,6 @@
/* Files manager */ /* Files manager */
.file-manager-page.file-manager-fullscreen { .files-page .file-manager-shell {
position: relative;
display: grid;
grid-template-rows: 1fr;
height: calc(100vh - 115px);
padding: 0;
overflow: hidden;
}
.file-manager-toolbar {
display: flex;
align-items: center;
gap: 8px;
flex-wrap: wrap;
padding: 10px 14px;
border-bottom: 1px solid var(--line);
background: var(--panel-header);
}
.file-manager-toolbar .button {
display: inline-flex;
align-items: center;
gap: 7px;
}
.file-manager-shell {
display: grid;
grid-template-columns: minmax(240px, 300px) minmax(0, 1fr); grid-template-columns: minmax(240px, 300px) minmax(0, 1fr);
min-height: 0;
height: 100%;
border: 1px solid var(--line);
border-radius: 0;
overflow: hidden;
background: var(--panel);
}
.file-list-panel {
display: flex;
flex-direction: column;
min-width: 0;
min-height: 0;
background: var(--panel);
}
.file-list-sticky {
flex: 0 0 auto;
z-index: 2;
border-bottom: 1px solid var(--line);
background: var(--panel-soft);
}
.file-breadcrumbs {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 6px;
min-height: 32px;
padding: 10px 14px 6px;
}
.file-breadcrumb,
.file-breadcrumb-segment {
display: inline-flex;
align-items: center;
gap: 5px;
}
.file-breadcrumb {
border: 0;
background: transparent;
color: var(--text-strong);
cursor: pointer;
font-weight: 800;
padding: 4px 6px;
border-radius: 7px;
}
.file-breadcrumb:hover:not(:disabled),
.file-breadcrumb:focus-visible {
background: var(--panel);
outline: none;
} }
.file-search-row { .file-search-row {
@@ -90,53 +11,21 @@
padding: 4px 0 10px 14px; padding: 4px 0 10px 14px;
} }
.file-list-meta {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
flex-wrap: wrap;
padding: 8px 14px;
border-top: 1px solid var(--line);
color: var(--muted);
font-size: 12px;
}
.file-list-drop-target {
position: relative;
flex: 1 1 auto;
min-height: 0;
overflow: auto;
transition: background-color .16s ease, box-shadow .16s ease;
}
.file-list-drop-target.is-active, .file-list-drop-target.is-active,
.file-list-drop-target.is-drop-target { .file-list-drop-target.is-drop-target {
background: var(--line); background: var(--line);
box-shadow: inset 0 0 0 2px var(--line-dark); box-shadow: inset 0 0 0 2px var(--line-dark);
} }
.file-list-table { .files-page .file-list-table {
min-width: 760px; min-width: 760px;
} }
.file-list-table-head, .files-page .file-list-table-head,
.file-list-row { .files-page .file-list-row,
display: grid; .managed-pattern-results .file-list-table-head,
.managed-pattern-results .file-list-row {
grid-template-columns: minmax(280px, 1fr) 120px 240px; grid-template-columns: minmax(280px, 1fr) 120px 240px;
gap: 12px;
align-items: center;
}
.file-list-table-head {
padding: 10px 14px;
border-top: 1px solid var(--line);
background: var(--panel);
color: var(--muted);
font-size: 12px;
font-weight: 800;
letter-spacing: .04em;
text-transform: uppercase;
} }
.file-list-table-head button { .file-list-table-head button {
@@ -158,24 +47,6 @@
outline: none; outline: none;
} }
.file-list-row {
min-height: 58px;
padding: 9px 14px;
border-bottom: 1px solid var(--line);
cursor: default;
user-select: none;
}
.file-list-row:focus-visible {
outline: 2px solid var(--line-dark);
outline-offset: -2px;
}
.file-list-row:hover,
.file-list-row.is-selected {
background: var(--line);
}
.file-list-row.is-drop-target { .file-list-row.is-drop-target {
background: var(--line); background: var(--line);
box-shadow: inset 0 0 0 2px var(--line-dark); box-shadow: inset 0 0 0 2px var(--line-dark);
@@ -191,43 +62,6 @@
background: transparent; background: transparent;
} }
.file-list-name-cell {
min-width: 0;
}
.file-list-name {
display: inline-flex;
align-items: center;
gap: 10px;
max-width: 100%;
min-width: 0;
border: 0;
background: transparent;
color: var(--text);
cursor: default;
font: inherit;
text-align: left;
padding: 0;
}
.file-list-name > span {
display: grid;
min-width: 0;
gap: 2px;
}
.file-list-name strong,
.file-list-name small {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.file-list-name small {
color: var(--muted);
font-size: 12px;
}
.file-linkage-status { .file-linkage-status {
display: inline-flex; display: inline-flex;
align-items: center; align-items: center;
@@ -238,29 +72,8 @@
color: var(--text-strong); color: var(--text-strong);
} }
.file-row-icon {
color: var(--muted);
flex: 0 0 auto;
}
.folder-row .file-row-icon { .folder-row .file-row-icon {
color: #b6791d; color: var(--warning-deep);
}
.file-row-tail {
display: flex;
align-items: center;
justify-content: space-between;
gap: 10px;
min-width: 0;
color: var(--muted);
font-size: 12px;
}
.file-list-empty {
padding: 36px 20px;
color: var(--muted);
text-align: center;
} }
.file-list-virtual-spacer { .file-list-virtual-spacer {
@@ -274,7 +87,7 @@
display: grid; display: grid;
min-width: 190px; min-width: 190px;
overflow: hidden; overflow: hidden;
border: 1px solid var(--line-dark); border: var(--border-line-dark);
border-radius: 12px; border-radius: 12px;
background: var(--panel); background: var(--panel);
box-shadow: var(--shadow-popover); box-shadow: var(--shadow-popover);
@@ -299,7 +112,7 @@
.file-context-menu button:hover, .file-context-menu button:hover,
.file-context-menu button:focus-visible { .file-context-menu button:focus-visible {
background: rgba(13, 110, 253, .08); background: var(--primary-soft);
outline: none; outline: none;
} }
@@ -315,14 +128,14 @@
display: grid; display: grid;
place-items: center; place-items: center;
padding: 22px; padding: 22px;
background: rgba(28, 25, 22, .38); background: var(--dialog-backdrop);
} }
.file-dialog { .file-dialog {
width: min(620px, 100%); width: min(620px, 100%);
max-height: min(720px, calc(100vh - 44px)); max-height: min(720px, calc(100vh - 44px));
overflow: auto; overflow: auto;
border: 1px solid var(--line-dark); border: var(--border-line-dark);
border-radius: 16px; border-radius: 16px;
background: var(--panel); background: var(--panel);
box-shadow: var(--shadow-popover); box-shadow: var(--shadow-popover);
@@ -334,7 +147,7 @@
justify-content: space-between; justify-content: space-between;
gap: 12px; gap: 12px;
padding: 15px 18px; padding: 15px 18px;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -397,7 +210,7 @@
overflow-y: auto; overflow-y: auto;
scrollbar-gutter: stable; scrollbar-gutter: stable;
padding: 8px; padding: 8px;
border: 1px solid var(--line); border: var(--border-line);
border-radius: 12px; border-radius: 12px;
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -428,13 +241,13 @@
.file-folder-selector-node:hover:not(:disabled), .file-folder-selector-node:hover:not(:disabled),
.file-folder-selector-node:focus-visible, .file-folder-selector-node:focus-visible,
.file-folder-selector-node.is-selected { .file-folder-selector-node.is-selected {
background: rgba(13, 110, 253, .09); background: var(--primary-soft);
outline: none; outline: none;
} }
.file-folder-selector-node.is-selected { .file-folder-selector-node.is-selected {
color: var(--text-strong); color: var(--text-strong);
box-shadow: inset 0 0 0 1px rgba(13, 110, 253, .25); box-shadow: var(--primary-inset-ring);
} }
.file-folder-selector-empty { .file-folder-selector-empty {
@@ -462,7 +275,7 @@
grid-template-rows: auto auto minmax(0, 1fr); grid-template-rows: auto auto minmax(0, 1fr);
min-height: 320px; min-height: 320px;
overflow: hidden; overflow: hidden;
border: 1px solid var(--line); border: var(--border-line);
border-radius: 12px; border-radius: 12px;
background: var(--panel); background: var(--panel);
} }
@@ -473,7 +286,7 @@
gap: 10px; gap: 10px;
align-items: center; align-items: center;
padding: 10px; padding: 10px;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -577,7 +390,7 @@
gap: 3px; gap: 3px;
min-width: 0; min-width: 0;
padding: 10px 12px; padding: 10px 12px;
border: 1px solid var(--line); border: var(--border-line);
border-radius: 10px; border-radius: 10px;
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -625,7 +438,7 @@
gap: 12px; gap: 12px;
align-items: center; align-items: center;
padding: 10px 12px; padding: 10px 12px;
border: 1px solid var(--line); border: var(--border-line);
border-radius: 8px; border-radius: 8px;
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -698,6 +511,11 @@
white-space: nowrap; white-space: nowrap;
} }
.file-connector-secret-fields {
display: grid;
gap: 18px;
}
@media (max-width: 760px) { @media (max-width: 760px) {
.file-connector-profile-row { .file-connector-profile-row {
grid-template-columns: 1fr; grid-template-columns: 1fr;
@@ -719,25 +537,27 @@
} }
@media (max-width: 1050px) { @media (max-width: 1050px) {
.file-manager-shell { .files-page .file-manager-shell {
grid-template-columns: 1fr; grid-template-columns: 1fr;
} }
.file-tree-panel { .files-page .file-tree-panel {
max-height: 260px; max-height: 260px;
border-right: 0; border-right: 0;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
} }
.file-list-table-head { .files-page .file-list-table-head,
.managed-pattern-results .file-list-table-head {
display: none; display: none;
} }
.file-list-table { .files-page .file-list-table {
min-width: 0; min-width: 0;
} }
.file-list-row { .files-page .file-list-row,
.managed-pattern-results .file-list-row {
grid-template-columns: 1fr; grid-template-columns: 1fr;
gap: 8px; gap: 8px;
} }
@@ -747,28 +567,11 @@
} }
} }
.file-manager-shell.is-loading .file-tree-panel,
.file-manager-shell.is-loading .file-list-panel {
filter: blur(1.5px);
pointer-events: none;
user-select: none;
}
.file-manager-loading-overlay {
position: absolute;
inset: 0;
z-index: 35;
display: grid;
place-items: center;
background: rgba(255, 255, 255, .12);
backdrop-filter: blur(1px);
}
.file-conflict-summary { .file-conflict-summary {
display: grid; display: grid;
gap: 3px; gap: 3px;
padding: 12px 14px; padding: 12px 14px;
border: 1px solid var(--line); border: var(--border-line);
border-radius: 12px; border-radius: 12px;
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -791,7 +594,7 @@
gap: 10px; gap: 10px;
align-items: center; align-items: center;
padding: 10px; padding: 10px;
border: 1px solid var(--line); border: var(--border-line);
border-radius: 12px; border-radius: 12px;
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -842,8 +645,8 @@
.rename-preview-more:hover, .rename-preview-more:hover,
.rename-preview-more:focus-visible { .rename-preview-more:focus-visible {
border-color: rgba(13, 110, 253, .45); border-color: var(--primary-border);
background: rgba(13, 110, 253, .08); background: var(--primary-soft);
color: var(--text-strong); color: var(--text-strong);
outline: none; outline: none;
} }
@@ -860,7 +663,7 @@
.managed-file-chooser-dialog > .dialog-header { .managed-file-chooser-dialog > .dialog-header {
padding: 16px 18px; padding: 16px 18px;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
background: var(--panel); background: var(--panel);
} }
@@ -891,9 +694,9 @@
.managed-file-chooser-footer { .managed-file-chooser-footer {
flex: 0 0 auto; flex: 0 0 auto;
padding: 12px 18px; padding: 12px 18px;
border-top: 1px solid var(--line); border-top: var(--border-line);
background: var(--panel); background: var(--panel);
box-shadow: 0 -8px 24px rgba(15, 23, 42, .06); box-shadow: var(--shadow-footer);
} }
.managed-file-chooser-layout { .managed-file-chooser-layout {
@@ -911,7 +714,7 @@
.managed-file-chooser-spaces { .managed-file-chooser-spaces {
min-width: 0; min-width: 0;
padding: 16px; padding: 16px;
border-right: 1px solid var(--line); border-right: var(--border-line);
background: var(--panel-soft); background: var(--panel-soft);
overflow: auto; overflow: auto;
} }
@@ -955,7 +758,7 @@
border: 1px solid transparent; border: 1px solid transparent;
border-radius: 9px; border-radius: 9px;
background: transparent; background: transparent;
color: var(--ink); color: var(--text-strong);
text-align: left; text-align: left;
cursor: pointer; cursor: pointer;
} }
@@ -1014,7 +817,7 @@
gap: 12px; gap: 12px;
align-items: center; align-items: center;
padding: 12px 14px; padding: 12px 14px;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
} }
.managed-file-breadcrumb { .managed-file-breadcrumb {
@@ -1033,7 +836,7 @@
border: 0; border: 0;
border-radius: 6px; border-radius: 6px;
background: transparent; background: transparent;
color: var(--ink); color: var(--text-strong);
padding: 5px 6px; padding: 5px 6px;
cursor: pointer; cursor: pointer;
} }
@@ -1046,7 +849,7 @@
display: grid; display: grid;
gap: 8px; gap: 8px;
padding: 12px 14px; padding: 12px 14px;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -1075,7 +878,7 @@
align-items: center; align-items: center;
min-height: 38px; min-height: 38px;
padding: 0 20px; padding: 0 20px;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
background: var(--panel-soft); background: var(--panel-soft);
color: var(--muted); color: var(--muted);
font-size: 12px; font-size: 12px;
@@ -1132,7 +935,7 @@
} }
.managed-file-entry:disabled { .managed-file-entry:disabled {
color: var(--ink); color: var(--text-strong);
opacity: 1; opacity: 1;
cursor: default; cursor: default;
} }
@@ -1206,7 +1009,7 @@
.managed-file-chooser-note { .managed-file-chooser-note {
margin: 0; margin: 0;
padding: 10px 14px; padding: 10px 14px;
border-top: 1px solid var(--line); border-top: var(--border-line);
background: var(--panel-soft); background: var(--panel-soft);
} }
@@ -1240,7 +1043,7 @@
border-right: 0; border-right: 0;
border-left: 0; border-left: 0;
background: transparent; background: transparent;
color: var(--ink); color: var(--text-strong);
text-align: left; text-align: left;
font: inherit; font: inherit;
cursor: pointer; cursor: pointer;
@@ -1267,27 +1070,6 @@
color: var(--muted); color: var(--muted);
} }
.split-field-action {
gap: 0;
}
.split-field-action > input,
.split-field-action > select,
.split-field-action > textarea {
border-top-right-radius: 0 !important;
border-bottom-right-radius: 0 !important;
}
.split-field-action > button,
.split-field-action > .button,
.split-field-action > .btn {
align-self: stretch;
margin-left: -1px;
border-top-left-radius: 0;
border-bottom-left-radius: 0;
box-shadow: none;
}
@media (max-width: 850px) { @media (max-width: 850px) {
.managed-file-entry-head, .managed-file-entry-head,
.managed-file-entry { .managed-file-entry {
@@ -1313,7 +1095,7 @@
.managed-file-chooser-spaces { .managed-file-chooser-spaces {
border-right: 0; border-right: 0;
border-bottom: 1px solid var(--line); border-bottom: var(--border-line);
max-height: 160px; max-height: 160px;
} }